Workflow build overview

Matrix build launcher

The workflow build in matrix-launcher.groovy is a relatively simple workflow where the actual build is performed by a normal matrix build. The workflow does these things:

  1. The workflow does a normal git checkout to show Changes and other git data on the build page. This happens in the initial node context where the workflow script is loaded.
  2. Also in the initial node context, the Jenkins job calls loadMatrixConfigs() and passes the name of the matrix to use. The workflow build calls prepare_multi_configuration_build() Python function, and reads the build axis into a local variable.
  3. The Jenkins job calls doBuild() with the name of the matrix build to trigger. The workflow triggers the matrix job, forwarding all relevant build parameters to it, and adding the build configuration axis as an additional parameter.
  4. After the matrix build finishes, the workflow calls process_multi_configuration_build_results(), which uses the Jenkins REST API to verify that all configurations were actually built. If not, the build is marked failed.
  5. The workflow also adds a link to the matrix build to the build summary page (while the build is running, the link can be found from the console log). The build status of the matrix build is also propagated to the status of the workflow job.
  6. Also, as the last step in the workflow, it sets the URL to post back to Gerrit to point to the matrix build. This means that the presence of this launcher job is mostly invisible during normal usage. Only if the workflow itself fails before reaching this step, you actually see a link to the launcher job in Gerrit.

Clang static analysis

The workflow build in clang-analyzer.groovy is a simple workflow that performs static analysis using Clang. The main reason for using a workflow build instead of a freestyle job is to make it easy to dynamically decide the node where the analysis runs, depending on which version of the analyzer should be used. The workflow sequence is this:

  1. The workflow does a normal git checkout to show Changes and other git data on the build page. This happens in the initial node context where the workflow script is loaded.
  2. Also in the initial node context, the workflow calls read_build_script_config() Python function to get the build options defined in the clang-analyzer.py build script in the source repo. This is stored in a local variable.
  3. The Jenkins job calls doBuild() without parameters. The workflow allocates a node based on the build options, and runs the clang-analyzer.py build script there with releng.
  4. After the releng script finishes, the workflow publishes a HTML report produced by the analyzer (if it exists), and scans for compiler warnings from the console log to show them on the build page.

Build & test release tarballs

The workflow build in release.groovy is a more complex worflow that coordinates the building and testing of the tarballs for a release. The packaging of the tarballs is handled by two separate, non-workflow Jenkins jobs, one for the source code and one for the regressiontests. The general sequence is this:

  1. The workflow reads the refspecs to use for the build from build parameters, and does some preparatory steps. It also reads a set of configurations to test from release-matrix.txt in the source repo, using prepare_multi_configuration_build() Python function, and reads the configuration into a data structure. It also extracts version information from the source repository (using get-version-info.py build script), since the regressiontests repository does not contain this.
  2. The workflow checks the latest successful builds in the packaging builds, and if these are not built from the correct commit, it triggers new builds for them. The regressiontests tarball is built first, and its MD5 sum is checked against the one specified in the source tarball. For a RELEASE build, a mismatch fails the build, otherwise it only produces a note in the console output. The packaging builds also compute MD5 sums for the tarballs, and these are accessible from Jenkins.
  3. After the tarballs are available, the workflow runs each configuration from the test matrix in parallel, using run_build(), and the standard gromacs.py build script from the source tarball. A summary is posted to the build summary page (for each configuration, on which host it was built and whether it was successful, unstable, or failed), but compiler warnings etc. are currently only available from the console log (available for a single configuration with some browsing under “Pipeline Steps”).
  4. If all tests passed, the workflow then does a final documentation build from the source tarball, which will produce the HTML pages for the documentation website. The generated pages are available from the Jenkins project page, as well as from a link on the build summary page. If the RELEASE build parameter is set, a tarball containing all the documentation is also archived as an artifact.

In addition to the refspecs to build, the workflow uses two additional build parameters:

RELEASE
If set, the -dev suffix is stripped from all the tarballs, and from content within them. Note that currently, the -dev suffixes never appear in the generated website, irrespective of this
FORCE_REPACKAGING
If set, the tarballs are rebuilt, even if ones built from the correct refspecs and with the same value of RELEASE is available. This is useful if only releng or Jenkins configuration has changed in a way that influences the tarballs.

The workflow and the level of testing is still a work-in-progress, but it already covers most of what the earlier builds did, and remaining content should not be too hard to add. Missing functionality is indicated with TODOs in the workflow script or in the build scripts in the source repo.

On-demand launcher

The workflow build in ondemand.groovy handles builds that are triggered with a [JENKINS] comment from Gerrit. The actual builds are done using separate, non-workflow jobs triggered from the workflow. The general sequence is:

  1. In the context of the initial checkout, the workflow uses get_actions_from_triggering_comment() to parse the comment from Gerrit. This function will also read information from the gromacs repository, e.g., to fill out the matrix options into the returned data structure.
  2. In the same context, the workflow does a normal git checkout to show Changes and other git data on the build page.
  3. The Jenkins job calls doBuild() without parameters. The workflow runs the requested builds in parallel, based on the data structure it got in step 1. All relevant build parameters are forwarded. Some actions are handled directly within the workflow instead of triggering a separate build.
  4. After the builds finish, the workflow adds links to the triggered builds to the build summary page (while the build is running, the link can be found from the console log). The workflow then uses do_ondemand_post_build() to construct the message to post back to Gerrit, as well as to perform other actions such as posting cross-verify messages. The combined build status of the builds is also propagated to the status of the workflow job.