Jenkins scripts (releng Python module)¶

The main scripts used for Jenkins build are collected into a releng Python package in the releng repository.

Build overview¶

Folder structure¶

In all builds, the folder structure in the workspace looks like this:

$WORKSPACE/
  releng/
  gromacs/
  [regressiontests/]
  logs/
    [unsuccessful-reason.log]
    [<category>/]*
  [build/]

Python build script¶

Builds using the releng Python scripts use the following sequence:

Jenkins (or the pipeline script) does some preparatory steps (see Jenkins configuration), including checking out the releng repo.
Jenkins imports the releng Python package, and calls run_build().
The releng script checks out the gromacs repo if not yet done by Jenkins.
The releng script locates a Python build script from the gromacs repo based on the build type given to run_build(), and loads the code. The build script provides some configuration settings as global variables, and a do_build() function that provides the actual build steps.
If the build script requires regression tests, the releng script now checks out the regressiontests repo.
The releng script prepares the build environment, such as initializing environment variables and generic CMake options such as the used compilers. The build environment can be influenced by build options passed to run_build(). Some build options passed to run_build() only set parameters that the build script can access to influence how to do the build (not all build scripts use these parameters). See Build options for details on the supported build options. This step also includes setting up a separate build directory for out-of-source builds if so requested by the build script.
The releng script calls do_build() provided by the build script. do_build() receives a build context that it can use to access information about the build environment, build parameters, and the workspace. The context also provides methods to run CMake, to build targets, to copy logs to a common location in the workspace, to mark the build unstable, and other such helper functions to help interacting with Jenkins in an uniform manner. See releng Python API for details on the API available to the build script.
The build script provides the steps to do the actual build, typically calling methods in the build context to interact with the CMake build system or Jenkins where required.
When the build script returns, or raises a BuildError exception to indicate a build failure, the releng script does some final processing to handle reason reporting for unsuccessful (unstable or failed) builds.
Jenkins does various post-build actions, such as publishing or parsing logs from the common log location, and using the unsuccessful reason reported from the script as the failure message to report back to Gerrit.

Pipeline builds¶

The subdirectory workflow/ contains Groovy scripts for use with the Jenkins Pipeline plugin. The general sequence for these builds is as follows:

Jenkins allocates a node for loading the Groovy script.
Jenkins checks out the releng repo using a shell script. We do not use an SCM step here to avoid showing this checkout on the build summary page. The summary page only works reasonably with at most one Git checkout within the pipeline, and the pipeline script should be in control of what this checkout is.
Jenkins loads the desired pipeline script from workflow/.
The pipeline script further loads utils.groovy as its first statement. Any other statements at the top level of the pipeline script are also executed in the context of the node/workspace where the script is being loaded. The pipeline script should do a return this as its last statement.
As part of the statements run in the initial workspace context, the pipeline script typically calls utils.initBuildRevisions() as the first action, which in turn calls releng.get_build_revisions() in Python. This will parse the various *_REFSPEC variables coming from build parameters and accessible as environment variables in the Python script. The Python script returns new values for the project-specific *_REFSPEC and *_HASH variables, where values like GERRIT_REFSPEC have been applied. These are set into the environment for the rest of the build. The returned information is also used to print information on the build summary page about which commits from each repository are built.
In the same context, the pipeline script typically calls utils.checkoutDefaultProject() next. This performs the SCM checkout for the build in a way that shows the changes on the front page. TODO: Consider if we would actually want to skip this part, as it is causing issues like JENKINS-19022.
Depending on the type of the build, Jenkins may also call other functions defined in the pipeline script in this node/workspace context. This is necessary if some values need to be passed from Jenkins configuration to the pipeline script for code that runs in this context.
Jenkins calls doBuild() defined in the pipeline script outside of any node/workspace context. Depending on the pipeline script, some parameters may be passed.
The pipeline script has full control over the build from now on, until the end.

See Jenkins configuration for more details on the configuration.

See Pipeline build overview for more details on what kinds of builds the pipeline scripts are currently used for.

Matrix builds¶

The releng scripts also support creating Jenkins matrix builds that load the configuration matrix from the gromacs repository. These files are located under admin/builds/. The format of such matrix files is one configuration per line. Empty lines are ignored, and comments can be started with #.

The build host assignment happens through a set of labels: build options that affect the possible host for building the configuration map to labels (the mapping is defined in options.py), and the set of labels supported by each build agent is defined in agents.py.

The building is orchestrated by a pipeline build that loads and preprocesses the configuration matrix, and then triggers a matrix build that takes the configuration axis values as a build parameter. The matrix build uses the standard sequence with releng Python scripts.

See Pipeline build overview and Jenkins configuration for more details.

Input environment variables¶

The following environment variables are used by the releng Python scripts for input from the Jenkins job (or from a pipeline build script):

GROMACS_REFSPEC REGRESSIONTESTS_REFSPEC RELENG_REFSPEC: Refspecs for the repositories used for fetching the change to build. Note that they will not always be used for an actual checkout; for example, Jenkins always needs to do the checkout for releng. For pipeline builds, the first two can also have an empty value or auto (the latter is needed since Jenkins cannot set environment variables to an empty value), in which case get_build_revisions() resolves them to actual refspecs for the remainder of the build. The values can also be overridden by GERRIT_REFSPEC or CHECKOUT_REFSPEC.
GROMACS_HASH REGRESSIONTESTS_HASH RELENG_HASH: If set, these provide hashes to check out, corresponding to the refspecs. Thees can be used to build a fixed commit from a refspec such as refs/heads/master, even if multiple checkouts are done at different times. It is assumed that fetching the corresponding refspec will make the commit with the provided hash available.
CHECKOUT_PROJECT: Needs to be set to the project (gromacs, regressiontests, or releng) that Jenkins has checked out. If not set, the scripts assume releng.
CHECKOUT_REFSPEC: Refspec used to checkout CHECKOUT_PROJECT. If set, this will override the project-specific refspec variable.
GERRIT_PROJECT GERRIT_REFSPEC: These are set by Gerrit Trigger, and override values from the project-specific refspec variable. GERRIT_PROJECT is also used in some cases for interpreting GERRIT_EVENT_COMMENT_TEXT.
GERRIT_BRANCH: This is set by Gerrit Trigger to identify the branch that triggered the build. It is used to support the same job triggering from multiple branches, getting the refspecs for associated repos right.
GERRIT_EVENT_COMMENT_TEXT: This is set by Gerrit Trigger when the build is triggered by a comment added in Jenkins. The text is expected to be base64-encoded (the default in Gerrit Trigger). See Triggering builds for recognized ways of triggering builds through this.
MANUAL_COMMENT_TEXT: If GERRIT_EVENT_COMMENT_TEXT is not set, the value from this variable is used instead (without base64-decoding or removing [JENKINS] tags). This allows creating a test Jenkins job that can be manually triggered.
NODE_NAME: Name of the host where the build is running. This is used for some host-specific logic in configuring the compilation. This is set by Jenkins automatically.
WORKSPACE: Path to the root of the Jenkins workspace where the build is running. This is set by Jenkins automatically.
STATUS_FILE: Path to the file to write on completion of the build, containing the build status and the reason for failed builds. Defaults to logs/unsuccessful-reason.log. If the extension is .json, the file is written as JSON, which is used in pipeline builds to allow programmatic processing of the results. For pipeline builds, the .json file can also contain a return value that provides structured information for further processing in the pipeline script.
NO_PROPAGATE_FAILURE: If set to a non-empty value, the build script will exit with a zero exit code even if the build fails because of a BuildError or ConfigurationError. Only unexpected exceptions will cause a non-zero exit code. The information in STATUS_FILE can be used to determine whether the build failed or not.

Output¶

To communicate back to the Jenkins job (or the pipeline build script), the releng scripts use the following mechanisms:

exit code

The script exits with a non-zero exit code if the build fails, unless NO_PROPAGATE_FAILURE is set. If the environment variable is set, only an unexpected exception will cause a non-zero exit code.

status file

In freestyle jobs, STATUS_FILE is not specified, and logs/unsuccessful-reason.log is written if the build fails or is unstable. This is intended to be used as the unsuccessful message for Gerrit Trigger in non-pipeline builds.

In pipeline builds, STATUS_FILE is specified as a JSON file, and contains additional information about the result of the execution. This is used to communicate success/failure back to the pipeline script, as well as reason for failures and in some cases, additional return values in case of success.

A reasonable effort is done to try to delete this file at the start of the script, so that old versions would not be left if the script fails. Even on unexpected errors, a reasonable effort is made to produce the file and include the exception information in it. If producing this file fails, it is treated as an unexpected error.

console outout

If the build is unstable, it also ensures that the word FAILED appears in the console log. This can be used in non-pipeline builds to mark the build unstable.

other files (specific to build scripts)

The build script can produce other relevant output in logs/ folder and in the build folder (which is typically gromacs/ for in-source builds and build/ for out-of-source builds).

Build options¶

Currently, the following build options can be passed from Jenkins to run_build() to influence the build environment (and as part of a configuration line in a matrix specification). These are typically used for multi-configuration jobs; for jobs that only build a single configuration, the configuration is typically hard-coded in the build script. For boolean options, multiple formats are accepted. E.g., an OpenMP build can be specified as openmp or openmp=yes, and no-OpenMP as no-openmp or openmp=no. The defaults that are used if a particular option is not specified are determined by the build script.

build-jobs=N: Use the specified number of parallel jobs for building.
out-of-source: Do the build out-of-source, even if an in-source build would be supported.
cmake-X.Y.Z: Use the specified CMake version to generate the build system.
gcc-X.Y: Use the specified gcc version as the compiler.
armclang-X.Y: Use the specified armclang version as the compiler.
clang-X.Y: Use the specified clang version as the compiler.
libcxx-X.Y: Use the specified libc++ version (currently only supported with exactly clang-X.Y)
clang-static-analyzer-X.Y: Use the specified clang static analyzer as the compiler.
icc-X.Y: Use Intel compiler (version is currently ignored; it is for informational purposes only and should match whatever is installed on the build nodes).
msvc-YYYY: Use the specified MSVC version as the compiler.
doxygen-X.Y.Z: Use the specified Doxygen version for the documentation build.
cuda-X.Y: Use the specified CUDA version.
opencl-X.Y: Use the specified OpenCL version.
amdappsdk-X.Y: Use the specified AMD SDK version. Deprecated.
armhpc-X.Y: Use the specified ARM HPC compiler toolchain with the specified version; this sets up the environment for either armclang with the same version or the bundled gcc.
phi: Build for Xeon Phi.
tsan: Use thread sanitizer for the build.
atlas: Use ATLAS as an external BLAS/LAPACK library.
x11: Build also gmx view (i.e., use GMX_X11=ON).
simd=SIMD: Use the specified SIMD instruction set. If not set, SIMD is not used.
gpuhw=VENDOR: Use a GPU with the “VENDOR” vendor or “none” if no GPU should be used.
mpi: Do an MPI build.

Build scripts can define additional options that only influence the behavior of the build scripts. This is used for matrix builds in gromacs.py for options that do not influence build the build environment or place requirements on the build host. This allows adding new options when the GROMACS build system changes and new combinations need to be tested, without changing releng.

Build system changes¶

This section collects information on how different types of changes to the GROMACS CMake build system, the releng scripts, and/or Jenkins configuration are handled to keep the CI builds working. Critical part in these changes is to try to keep builds working for older changes still pending review in Gerrit. However, the flipside is that if rebases are not forced, some problems may slip past if some older change is not compatible with the new CI builds.

Different cases for changes are below. The distinction may not always be clear-cut, but the general approach should be well covered.

Compatible change in main repo, no change in releng. In this case, all changes are absorbed in the build script in the main repo. Old changes will build with the old build script, new changes with the new, and all builds will pass. Old changes do not trigger the new functionality, so if the new build script contains new tests or such, they may get silently broken by old changes if they are not rebased (in this respect, the case is similar to the third item below).

An example of this type of change is reorganization or renaming of CMake cache variables or build targets, while still keeping the same or similar functionality. Some types of tests can also be added with this approach.
Compatible change in releng, no change in main repo. In this case, all changes are absorbed in the releng script. As soon as the releng change is merged, both old and new changes will build with the changed script, and all builds will pass.

An example of this type of change is software updates or node reconfiguration in Jenkins that affects, e.g., paths to certain programs. Also many bug fixes to the releng scripts fall here.
Breaking change in main repo, backwards-compatible change in releng. In this case, changes in the main repo build scripts require changes in releng that do not break old builds. The main repo change will not build until releng changes are merged; the releng change can be merged safely without breaking old builds. To verify the releng change with its corresponding main repo change, the releng change can be uploaded to Gerrit and then the on-demand cross-verification mechanism used (see Interacting with builds in Jenkins). After the releng change is merged, the main change build can be triggered and it will pass.

Builds for old changes will continue to work throughout this process, but they will ignore possible new build parameters or such, potentially breaking the new change.

An example of this type of change would be additional methods or parameters required in releng to be able to implement new build tasks.
Breaking change in releng, compatible change in main repo. In this case, changes or additional build configurations in the releng and/or Jenkins cause old builds to break. As soon as the changes in releng are merged, all old changes in Gerrit need to be rebased.

Testing releng scripts¶

Currently, there are limited unit tests for some parts of the Python scripts. They require a backport of unittest.mock to be installed, and can be executed with

python -m releng.test

The only way to fully test the releng script is to upload a change to Gerrit and let Jenkins build it. In principle, it is possible to run the script in an environment that exactly matches a Jenkins node (including paths to all required tools and all relevant environment variables that Jenkins sets), but that can be tedious to set up. However, it is possible to execute most of the code from the command line using

python releng <options>

This requires that you have your projects checked out in the same layout as in Jenkins: the gromacs, regressiontests, and releng repositories should be in sibling directories, with directory names matching the repository names.

Please note that even though the command-line mode does not perform most of the actions that the real build script does (unless you run it with --run), it can still write to some files etc.

Refactoring to better support mock execution is in progress, combined with extending the scope of unit tests.