One of a series of test instances for migrating the Koha Wiki MediaWiki database.
For the current Koha Wiki, visit https://wiki.koha-community.org .Jenkins maintenance
Jenkins is the tool we picked for our automated tasks.
Jenkins server
We have a master server, currently provided by Theke Solutions. It has a regular Jenkins install using Debian packages (on Ubuntu 16.04). It is using the Jenkins LTS distribution. This server only runs the Jenkins app, no tests are run there.
Jenkins nodes
Jenkins nodes are VMs that have a jenkins user created, and set with an SSH pubkey we have for that. They have Java 8 on them, the Jenkins server needs the java command available on the nodes to launch its shell. Since we containerized the runs using Docker, the jenkins user is required to belong to the docker group on the host machine.
The nodes are provided by different entities.:
- Docker1: ByWater Solutions
- Docker2: Universidad Nacional de Córdoba
- Docker3: KohaAloha
Jenkins tasks
The Jenkins tasks are basically the tasks we run on the nodes. All of them are triggered by (1) polling on some repository (running the full test suite for Koha for example) or (2) by calling them via webhooks (koha-testing-docker builds triggered from gitlab on each commit).
Koha tests
A while back when koha-testing-docker was created, it targeted the Jenkins use case. i.e. it was designed to run the whole test suite and exit. koha-testing-docker evolved into a KohaDevBox replacement, but there's still a RUN_TESTS_AND_EXIT flag that triggers that original behaviour.
Koha testing tasks basically:
- Set some environment variables:
$ export RUN_TESTS_AND_EXIT="yes" # mentioned before
$ export SYNC_REPO=. # use the 'current' checkout as SYNC_REPO
$ export LOCAL_USER_ID=$(id -u) # run as the calling user
$ export COVERAGE=$COVERAGE # calculate tests coverage? off by default because of the x4 running time, should be run on a schedule, maybe weekends
$ export KOHA_IMAGE=master-jessie # what koha-testing-docker image to use
- Download the docker-compose.yml and env files, and run docker-compose.
- Perform some cleanup tasks before and after the tests are run. We faced many issues with Docker filling the disks with old stuffs from each run, and some abnormal terminations leaving running containers/networks and preventing new runs.
Tasks for a new release
When a new release needs to be set on Jenkins, there are a few easy things to consider/do:
- As the tasks per-se got really generic (i.e. 'run this koha-testing-docker image against this git checkout) when a new release needs it, we use the 'Copy from' option in the 'New task' form. So we copy the task for 'master', using the OS we want. After creating the (copied) task, we just change the branch the task will be polling.
CAVEAT: The Koha repo is huge. So cloning it takes loads of time and it always timeouts on jenkins after 10 minutes or so. It also costs money (inbound traffic). So what we do (for now) is having a full kohaclone in /srv/jenkins/workspace/Koha_Source, and create symlinks. This is the nastiest bit I do in Jenkins:
$ cd /srv/jenkins/workspace
$ ln -s Koha_Source Koha_19.05_D8
$ ln -s Koha_Source Koha_19.05_D9
$ ln -s Koha_Source Koha_19.05_U18
This needs to be done on each Jenkins node. To avoid races on the Koha_Source repository, runs are restricted to one running task on each node. This is also a constraint we have because of the fact that koha-testing-docker will load Elasticsearch and Selenium. Running more than one task proved counter productive (system load skyrockets, memory is not enough, etc).
koha-testing-docker builds
koha-testing-docker (master) project has been recently refactored so the three Dockerfiles (for Jessie, Stretch and Bionic) are on a single branch, to ease maintainability. The shipped Jenkinsfile on the koha-testing-docker project has been adjusted to build all images in one run. Before that we had a branch and a task for each. This builds are of type 'Pipeline' and so they are controlled by the Jenkinsfile included on the repository.
Job configuration recommendations
- delete workspace before build starts
- Source Code Management: honor refspec on initial clone
- Source Code Management: Path of the reference repo to use during clone: /srv/jenkins/workspace/Koha_Source
- Source Code Management: shallow clone
- post-build actions: delete workspace when build is done