Jenkins scripts (releng Python module) ====================================== The main scripts used for Jenkins build are collected into a ``releng`` Python package in the ``releng`` repository. .. TODO: Some more introductory text. Build overview -------------- Python build script ^^^^^^^^^^^^^^^^^^^ Builds using the releng Python scripts use the following sequence: 1. Jenkins (or the workflow script) does some preparatory steps (see :doc:`jenkins-config`), including checking out the ``releng`` repo. 2. Jenkins imports the releng Python package, and calls run_build(). 3. The releng script checks out the ``gromacs`` repo if not yet done by Jenkins. 4. The releng script locates a Python build script from the ``gromacs`` repo based on the build type given to run_build(), and loads the code. The build script provides some configuration settings as global variables, and a do_build() function that provides the actual build steps. 5. If the build script requires regression tests, the releng script now checks out the ``regressiontests`` repo. 6. The releng script prepares the build environment, such as initializing environment variables and generic CMake options such as the used compilers. The build environment can be influenced by build options passed to run_build(). Some build options passed to run_build() only set parameters that the build script can access to influence how to do the build (not all build scripts use these parameters). See :ref:`releng-jenkins-build-opts` for details on the supported build options. This step also includes setting up a separate build directory for out-of-source builds if so requested by the build script. 7. The releng script calls do_build() provided by the build script. do_build() receives a build context that it can use to access information about the build environment, build parameters, and the workspace. The context also provides methods to run CMake, to build targets, to copy logs to a common location in the workspace, to mark the build unstable, and other such helper functions to help interacting with Jenkins in an uniform manner. See :doc:`releng-api` for details on the API available to the build script. 8. The build script provides the steps to do the actual build, typically calling methods in the build context to interact with the CMake build system or Jenkins where required. 9. When the build script returns, or raises a BuildError exception to indicate a build failure, the releng script does some final processing to handle reason reporting for unsuccessful (unstable or failed) builds. 10. Jenkins does various post-build actions, such as publishing or parsing logs from the common log location, and using the unsuccessful reason reported from the script as the failure message to report back to Gerrit. Workflow builds ^^^^^^^^^^^^^^^ The subdirectory :file:`workflow/` contains Groovy scripts for use with the Jenkins Pipeline plugin. The general sequence for these builds is as follows: 1. Jenkins allocates a node for loading the Groovy script. 2. Jenkins checks out the ``releng`` repo using a shell script. We do not use an SCM step here to avoid showing this checkout on the build summary page. The summary page only works reasonably with at most one Git checkout within the workflow, and the workflow script should be in control of what this checkout is. 3. Jenkins loads the desired workflow script. 4. Typically, the workflow script further loads ``utils.groovy`` as its first statement. Any other statements at the top level of the workflow script are also executed in the context of the node/workspace where the script is being loaded. The workflow script should do a ``return this`` as its last statement. 5. Depending on the workflow script, Jenkins may also call other functions defined in the workflow script in this node/workspace context. This is necessary if some values need to be passed from Jenkins configuration to the workflow script for code that runs in this context. 6. Jenkins calls ``doBuild()`` defined by the workflow script outside of any node/workspace context. Depending on the workflow script, some parameters may be passed. 7. The workflow script has full control over the build from now on, until the end. See :doc:`jenkins-config` for more details on the configuration. See :doc:`workflow` for more details on what kinds of builds the workflow scripts are currently used for. Matrix builds ^^^^^^^^^^^^^ The releng scripts also support creating Jenkins matrix builds that load the configuration matrix from the ``gromacs`` repository. These files are located under :file:`admin/builds/`. The format of such matrix files is one configuration per line. Empty lines are ignored, and comments can be started with ``#``. The build host assignment happens through a set of labels: build options that affect the possible host for building the configuration map to labels (the mapping is defined in :file:`options.py`), and the set of labels supported by each build slave is defined in :file:`slaves.py`. The building is orchestrated by a workflow build that loads and preprocesses the configuration matrix, and then triggers a matrix build that takes the configuration axis values as a build parameter. The matrix build uses the standard sequence with releng Python scripts. See :doc:`workflow` and :doc:`jenkins-config` for more details. .. _releng-input-env-vars: Input environment variables --------------------------- The following environment variables are used by the releng Python scripts for input from the Jenkins job (or from a workflow build script): ``GROMACS_REFSPEC`` ``REGRESSIONTESTS_REFSPEC`` ``RELENG_REFSPEC`` Refspecs for the repositories used for fetching the change to build. Note that they will not always be used for an actual checkout; for example, Jenkins always needs to do the checkout for ``releng``. ``GROMACS_HASH`` ``REGRESSIONTESTS_HASH`` ``RELENG_HASH`` If set, these provide hashes to check out, corresponding to the refspecs. Thees can be used to build a fixed commit from a refspec such as ``refs/heads/master``, even if multiple checkouts are done at different times. It is assumed that fetching the corresponding refspec will make the commit with the provided hash available. ``CHECKOUT_PROJECT`` Needs to be set to the project (``gromacs``, ``regressiontests``, or ``releng``) that Jenkins has checked out. Needs to be set, unless ``GERRIT_PROJECT`` is set. ``CHECKOUT_REFSPEC`` Refspec used to checkout ``CHECKOUT_PROJECT``. This will override the project-specific refspec for that project. ``GERRIT_PROJECT`` ``GERRIT_REFSPEC`` These are set by Gerrit Trigger, and can be used for simplicity instead of ``CHECKOUT_PROJECT`` and ``CHECKOUT_REFSPEC``. ``GERRIT_PROJECT`` is also used in some cases for interpreting ``GERRIT_EVENT_COMMENT_TEXT``. ``GERRIT_BRANCH`` This is set by Gerrit Trigger to identify the branch that triggered the build. It is used to support the same job triggering from multiple branches, getting the refspecs for associated repos right. ``GERRIT_EVENT_COMMENT_TEXT`` This is set by Gerrit Trigger when the build is triggered by a comment added in Jenkins. The text is expected to be base64-encoded (the default in Gerrit Trigger). See :ref:`releng-triggering-builds` for recognized ways of triggering builds through this. ``MANUAL_COMMENT_TEXT`` If ``GERRIT_EVENT_COMMENT_TEXT`` is not set, the value from this variable is used instead (without base64-decoding or removing ``[JENKINS]`` tags). This allows creating a test Jenkins job that can be manually triggered. ``NODE_NAME`` Name of the host where the build is running. This is used for some host-specific logic in configuring the compilation. This is set by Jenkins automatically. ``WORKSPACE`` Path to the root of the Jenkins workspace where the build is running. This is set by Jenkins automatically, except for workflow builds. ``STATUS_FILE`` Path to the file to write on completion of the build, containing the build status and the reason for failed builds. Defaults to :file:`logs/unsuccessful-reason.log`. If the extension is :file:`.json`, the file is written as JSON, which is useful for further use from a workflow build. ``NO_PROPAGATE_FAILURE`` If set to a non-empty value, the build script will exit with a zero exit code even if the build fails because of a BuildError or ConfigurationError. Only unexpected exceptions will cause a non-zero exit code. The information in ``STATUS_FILE`` can be used to determine whether the build failed or not. Output ------ To communicate back to the Jenkins job (or the workflow build script), the releng scripts use the following mechanisms: exit code The script exits with a non-zero exit code if the build fails, unless ``NO_PROPAGATE_FAILURE`` is set. If the environment variable is set, only an unexpected exception will cause a non-zero exit code. status file In freestyle jobs, ``STATUS_FILE`` is not specified, and :file:`logs/unsuccessful-reason.log` is written if the build fails or is unstable. This is intended to be used as the unsuccessful message for Gerrit Trigger in non-workflow builds. In workflow builds, ``STATUS_FILE`` is specified as a JSON file, and contains additional information about the result of the execution. This is used to communicate success/failure back to the workflow script, as well as reason for failures and in some cases, additional return values in case of success. A reasonable effort is done to try to delete this file at the start of the script, so that old versions would not be left if the script fails. Even on unexpected errors, a reasonable effort is made to produce the file and include the exception information in it. If producing this file fails, it is treated as an unexpected error. console outout If the build is unstable, it also ensures that the word ``FAILED`` appears in the console log. This can be used in non-workflow builds to mark the build unstable. other files (specific to build scripts) The build script can produce other relevant output in :file:`logs/` folder and in the build folder (which is typically :file:`gromacs/` for in-source builds and :file:`build/` for out-of-source builds). .. _releng-jenkins-build-opts: Build options ------------- Currently, the following build options can be passed from Jenkins to run_build() to influence the build environment (and as part of a configuration line in a matrix specification). These are typically used for multi-configuration jobs; for jobs that only build a single configuration, the configuration is typically hard-coded in the build script. For boolean options, multiple formats are accepted. E.g., an OpenMP build can be specified as ``openmp`` or ``openmp=yes``, and no-OpenMP as ``no-openmp`` or ``openmp=no``. The defaults that are used if a particular option is not specified are determined by the build script. build-jobs=N Use the specified number of parallel jobs for building. out-of-source Do the build out-of-source, even if an in-source build would be supported. cmake-X.Y.Z Use the specified CMake version to generate the build system. gcc-X.Y Use the specified gcc version as the compiler. armclang-X.Y Use the specified armclang version as the compiler. clang-X.Y Use the specified clang version as the compiler. clang-static-analyzer-X.Y Use the specified clang static analyzer as the compiler. icc-X.Y Use Intel compiler (version is currently ignored; it is for informational purposes only and should match whatever is installed on the build nodes). msvc-YYYY Use the specified MSVC version as the compiler. cuda-X.Y Use the specified CUDA version. amdappsdk-X.Y Use the specified AMD SDK version. armhpc-X.Y Use the specified ARM HPC compiler toolchain with the specified version; this sets up the environment for either armclang with the same version or the bundled gcc. phi Build for Xeon Phi. tsan Use thread sanitizer for the build. atlas Use ATLAS as an external BLAS/LAPACK library. x11 Build also ``gmx view`` (i.e., use ``GMX_X11=ON``). simd=SIMD Use the specified SIMD instruction set. If not set, SIMD is not used. mpi Do an MPI build. Build scripts can define additional options that only influence the behavior of the build scripts. This is used for matrix builds in :file:`gromacs.py` for options that do not influence build the build environment or place requirements on the build host. This allows adding new options when the |Gromacs| build system changes and new combinations need to be tested, without changing releng. Build system changes -------------------- This section collects information on how different types of changes to the |Gromacs| CMake build system, the releng scripts, and/or Jenkins configuration are handled to keep the CI builds working. Critical part in these changes is to try to keep builds working for older changes still pending review in Gerrit. However, the flipside is that if rebases are not forced, some problems may slip past if some older change is not compatible with the new CI builds. Different cases for changes are below. The distinction may not always be clear-cut, but the general approach should be well covered. 1. *Compatible change in main repo, no change in releng.* In this case, all changes are absorbed in the build script in the main repo. Old changes will build with the old build script, new changes with the new, and all builds will pass. Old changes do not trigger the new functionality, so if the new build script contains new tests or such, they may get silently broken by old changes if they are not rebased (in this respect, the case is similar to the third item below). An example of this type of change is reorganization or renaming of CMake cache variables or build targets, while still keeping the same or similar functionality. Some types of tests can also be added with this approach. 2. *Compatible change in releng, no change in main repo.* In this case, all changes are absorbed in the releng script. As soon as the releng change is merged, both old and new changes will build with the changed script, and all builds will pass. An example of this type of change is software updates or node reconfiguration in Jenkins that affects, e.g., paths to certain programs. Also many bug fixes to the releng scripts fall here. 3. *Breaking change in main repo, backwards-compatible change in releng.* In this case, changes in the main repo build scripts require changes in releng that do not break old builds. The main repo change will not build until releng changes are merged; the releng change can be merged safely without breaking old builds. To verify the releng change with its corresponding main repo change, the releng change can be uploaded to Gerrit and then the on-demand cross-verification mechanism used (see :doc:`jenkins-ui`). After the releng change is merged, the main change build can be triggered and it will pass. Builds for old changes will continue to work throughout this process, but they will ignore possible new build parameters or such, potentially breaking the new change. An example of this type of change would be additional methods or parameters required in releng to be able to implement new build tasks. 4. *Breaking change in releng, compatible change in main repo.* In this case, changes or additional build configurations in the releng and/or Jenkins cause old builds to break. As soon as the changes in releng are merged, all old changes in Gerrit need to be rebased. .. TODO: The matrix-in-source-repo approach makes the example below incorrect, move it elsewhere and find a new one here. An example of this type of change would be introduction of a new build parameter that does not compile cleanly without a corresponding change in the main repo (e.g., introduction of a new compiler version that produces warnings). There is currently no special mechanism for this case. Older builds in Gerrit will fail in unpredictable ways. .. TODO: Identify possible cases that do not fall into any of the above categories, and/or that are distinct enough from the examples above to be worth mentioning. .. TODO: Do we need some mechanism to detect rebasing needs for some of the above, and, e.g., have this indicated in the build failure message (or skip the build or something similar)? Testing releng scripts ---------------------- Currently, there are limited unit tests for some parts of the Python scripts. They require a backport of ``unittest.mock`` to be installed, and can be executed with :: python -m releng.test The only way to fully test the releng script is to upload a change to Gerrit and let Jenkins build it. In principle, it is possible to run the script in an environment that exactly matches a Jenkins node (including paths to all required tools and all relevant environment variables that Jenkins sets), but that can be tedious to set up. However, it is possible to execute most of the code from the command line using :: python releng This requires that you have your projects checked out in the same layout as in Jenkins: the gromacs, regressiontests, and releng repositories should be in sibling directories, with directory names matching the repository names. Please note that even though the command-line mode does not perform most of the actions that the real build script does (unless you run it with ``--run``), it can still write to some files etc. Refactoring to better support mock execution is in progress, combined with extending the scope of unit tests.