releng Python API¶
Build script definition¶
The build script is required to provide one function:
-
do_build
(context)¶ Called to run the actual build. The context parameter is an instance of BuildContext, and can be used to access the build environment and to interact with Jenkins. The function can signal fatal build errors by raising BuildError directly; typically, this is done by methods in BuildContext if they fail to execute the requested commands.
When the function is called, the current working directory is set to the build directory (whether the build is in- or out-of-source).
The build script can also set a few global variables to influence the behavior of the build:
-
build_options
¶ If this list value is set to a non-empty list, then these build options are used to initialize the build environment. Useful for non-matrix builds that want to, e.g., specify the compiler to use.
-
build_out_of_source
¶ If this boolean value is set to
True
, the build will be executed out-of-source. By default, the build will be in-source.
-
extra_options
¶ If this dictionary is set, it declares additional build options that the script understands. This can be used to declare options that only influence the build script; releng declares only options that affect the build environment or the build host assignment. Syntax is as follows:
extra_options = { 'opt': Option.simple, 'opt-bool': Option.bool, 'opt-str': Option.string }
The values of the build options can be read from
context.opts
in do_build(). SeeOptionTypes
documentation for the available option types: in the build script,Option
is bound toOptionTypes
.Technically, the value in the dictionary is a callable that gets called with the name of the option to create an internal handler class for processing the option.
-
extra_projects
¶ If this list value is set to a non-empty list, then these repositories are also checked out before executing the build.
releng
andgromacs
repositories are always checked out. Currently, onlyProject.REGRESSIONTESTS
makes sense to specify here.
When the build script is loaded, various enums from the releng package are injected into the global scope to make them easy to access.
API for build scripts¶
The build script gets input and perfoms most tasks by using data and methods in a BuildContext instance:
-
class
releng.context.
BuildContext
(factory, job_type, opts, script_settings)¶ Top-level interface for build scripts to the releng package.
-
job_type
¶ JobType – Type/scope of the build job (e.g., per-patchset, nightly).
-
env
¶ BuildEnvironment – Access to build environment details like paths to executables. Many of the environment properties, such as selecting the compiler, are handled by the build context transparently, without the build script needing to access this.
-
opts
¶ BuildOptions – Access to all build options.
-
params
¶ BuildParameters – Access to Jenkins build parameters (through environment variables).
-
workspace
¶ Workspace – Access to the build workspace. Can be used to get paths to various parts in the workspace for changing directories and for producing log files.
-
build_target
(target=None, parallel=True, keep_going=False, target_descr=None, failure_string=None, continue_on_failure=False)¶ Builds a given target.
run_cmake() must have been called to generate the build system.
Parameters: - target (Optional[str]) – Name of the target to build.
If
None
, the default (all) target is built. - parallel (Optional[bool]) – Whether parallel building is supported.
- keep_going (Optional[bool]) – Whether to continue building after first error.
- target_descr (str or None) – If given, customizes the error message
when the target fails to build. Should fit the initial part of
the sentence “… failed to build”.
Ignored if
failure_string
is specified. - failure_string (str or None) – If given, this message is used as the failure message if the target fails to build.
- continue_on_failure (Optional[bool]) – If
True
and the target fails to build, the failure is only reported andself.failed
is set toTrue
.
Raises: BuildError
– If the target fails to build, andcontinue_on_failure
is not specified.- target (Optional[str]) – Name of the target to build.
If
-
chdir
(path)¶ Changes the working directory for subsequent run_cmd() calls.
-
compute_md5
(path)¶ Computes MD5 hash of a file.
Parameters: path (str) – Path to the file to compute the hash for. Returns: String with the computed hash in hexadecimal. Return type: str
-
failed
¶ Whether the build has already failed.
This can be used in combination with build_target() argument continue_on_failure if the build script needs to test whether some previous target already has built. If the build script wants to stop the build in such a case, it can simply return; the build will always be marked failed if this property is
True
.
-
get_doc_cmake_options
(doxygen_version, sphinx_version)¶ Returns non-GROMACS-specific CMake options to set for documentation builds.
-
make_archive
(path, root_dir=None, use_git=False, prefix=None)¶ Creates a tar.gz archive.
Parameters:
-
mark_unstable
(reason, details=None)¶ Marks the build unstable.
Parameters:
-
process_clang_analyzer_results
()¶ Processes results from clang analyzer.
-
process_coverage_results
(exclude=None)¶ Processes results from coverage runs.
Uses gcovr to process all coverage files found in the workspace (from running a build compiled with –coverage).
Parameters: exclude (List[str]) – Exclusions to pass to gcovr -e (regexs).
-
process_cppcheck_results
(xml_pattern)¶ Processes results from cppcheck.
This method massages the XML output into a form the Jenkins CppCheck plugin can handle. It could also contain logic that would report additional information about the issues back to Jenkins.
Parameters: - xml_pattern (str) – Pattern that matches all XML files produced by
- cppcheck. –
-
publish_logs
(logs, category=None)¶ Copies provided log(s) to Jenkins.
This should be used for any files that are produced during the build and that need to be parsed by Jenkins (except for special cases where a separate method is provided, such as process_cppcheck_results()). This allows Jenkins configuration stay the same even if the files are relocated because of build system or repository reorganization. Alternatively, the build script can use workspace.get_path_for_logfile() to directly produce the log files into an invariant location.
Parameters:
-
read_cmake_variable_file
(path)¶ Reads a file with CMake variable declarations (set commands).
Parameters: path (str) – Path to the file to read. Returns: variables found from the file, with their values. Return type: Dict
-
replace_in_file
(path, pattern, repl)¶ Performs re.sub() on contents of a file.
Parameters:
-
run_cmake
(options)¶ Runs CMake with the provided options.
Options from the environment, such as for selecting compilers, are added automatically.
The working directory should be the build directory. Currently, does not support running CMake multiple times.
Parameters: options (Dict[str,str]) – Dictionary of macro definitions to pass to CMake using -D
.Raises: BuildError
– If CMake fails to configure the build system.
-
run_cmd
(cmd, ignore_failure=False, use_return_code=False, use_output=False, failure_message=None, **kwargs)¶ Runs a command via subprocess.
This wraps subprocess.call() and check_call() with error-handling code and other generic handling such as ensuring proper output flushing and using bash as the shell on Unix.
Any arguments accepted by subprocess.call() or check_call() can also be passed, e.g. cwd or env to make such calls in stateless ways.
Parameters: - cmd (str/list) – Command to execute (as for subprocess.call()).
- ignore_failure (Optional[bool]) – If
True
, failure to run the command is ignored. - use_return_code (Optional[bool]) – If
True
, exit code from the command is returned. Otherwise, non-zero return code fails the build unless ignore_failure is set. - use_output (Optional[bool]) – If
True
, the output from the command is returned. Mutually exclusive with use_return_code. - failure_message (Optional[str]) – If set, provides a friendly message about what in the build fails if this command fails. This will be reported back to Gerrit.
Returns: Command return code (if
use_return_code=True
).Return type:
-
run_ctest
(args, memcheck=False, failure_string=None)¶ Runs tests using CTest.
The build is marked unstable if any test fails.
Parameters:
-
set_version_info
(version, regtest_md5sum)¶ Provides source version information from a build script.
This method supports the get-version-info.py build script and allows that script to pass out the version information without writing it into a file that would then be read back.
-
write_package_info
(project, file_name, version)¶ Writes an information file for a tar package.
The file has a specific format that allows reading the information in a downstream build, and using the package instead of a git checkout.
Parameters:
-
The build context contains attributes of the following classes to access additional information:
-
class
releng.environment.
BuildEnvironment
(factory)¶ Provides access to the build environment.
Most details of the build environment are handled transparently based on the provided build options, and the build script does not need to consider this. For build scripts, the main interface this class provides is to find locations of some special executables (such as cppcheck) that may be needed for the build. Compiler selection is handled without special action from build scripts.
In rare cases, the build scripts may benefit from inspecting the attributes in this class to determine, e.g., the operating system running the build or the compiler being used.
-
system
¶ System – Operating system of the build node.
-
compiler
¶ Compiler or None – Selected compiler.
-
compiler_version
¶ string – Version number for the selected compiler.
-
c_compiler
¶ str or None – Name of the C compiler executable.
-
cxx_compiler
¶ str or None – Name of the C++ compiler executable.
-
libcxx_version
¶ str or None – Version of libc++ to use
-
doxygen_command
¶ str – Name of the doxygen executable.
-
gcov_command
¶ str – Name of the gcov executable.
-
cmake_command
¶ str – Name of the CMake executable.
-
ctest_command
¶ str – Name of the CTest executable.
-
cmake_version
¶ str – Version of the CMake executable.
-
cmake_generator
¶ str or None – CMake generator being used.
-
armhpc_version
¶ str or None – The version of the ARM HPC toolchain.
-
armpl_dir
¶ str or None – the ARM Perf Libraries directory
-
cuda_root
¶ str or None – Root of the CUDA toolkit being used (for passing to CUDA_TOOLKIT_ROOT_DIR CMake option).
-
libhwloc_root
¶ str or None – Root of the hwloc library being used (for adding to CMake search path).
-
opencl_version
¶ str or None – OpenCL API version expected to be detected.
-
cuda_host_compiler
¶ str or None – Full path to the host compiler used with CUDA (for passing to CUDA_HOST_COMPILER CMake option).
-
amdappsdk_root
¶ str or None – Root of the AMD SDK being used (for using as AMDAPPSDKROOT environment variable).
-
extra_cmake_options
¶ Dict[str, str] – Additional options to pass to CMake.
-
append_path_env
(path)¶ Appends a path to the executable search path (PATH).
-
get_clang_format_command
(version)¶ Returns path to clang-format executable.
-
get_cppcheck_command
(version)¶ Returns path to the cppcheck executable of given version.
Parameters: version (str) – cppcheck version to use.
-
get_doxygen_command
(version)¶ Returns path to the Doxygen executable of given version.
Parameters: version (str) – Doxygen version to use.
-
get_uncrustify_command
()¶ Returns path to the uncrustify executable.
-
prepend_path_env
(path)¶ Prepends a path to the executable search path (PATH).
-
set_env_var
(variable, value)¶ Sets environment variable to be used for further commands.
All subsequent commands run with BuildContext.run_cmd() etc. will use the environment variable.
Parameters:
-
-
class
releng.options.
BuildOptions
(handlers, opts)¶ Values for all build options.
This class provides read-only access to the values of all build options. A build option named
mdrun-only
is accessible asopts.mdrun_only
andopts['mdrun-only']
, whichever is more convenient. The keys for all build options are always available. If an option is not specified, the corresponding value isNone
.For simple flag options that can only be set or unset, the stored value is
True
if the option is set.For boolean options, the stored value is
False
orTrue
. For example,no-mpi
andmpi=no
are both stored asopts.mpi == True
.For options like
gcc-4.8
, the value is stored asopts.gcc == '4.8'
. Similarly,build-jobs=2
is stored asopts.build_jobs == '2'
.
-
class
releng.integration.
BuildParameters
(factory)¶ Access to build parameters.
-
class
releng.workspace.
Workspace
(factory)¶ Provides access to set up, query, and act within the build workspace, particularly involving operations on the git repositories associated with the projects.
Methods are provided for accessing the build directory (whether in- or out-of-source), as well as the root directories of all checked-out projects. Also, methods to access a common log directory (for logs that need to be post-processed in Jenkins) are provided. Implements functionality for updating commits with new files.
-
root
¶ str – Root directory of the workspace.
-
install_dir
¶ str – Directory for test installation.
-
build_dir
¶ Build directory for building gromacs.
Returns either the gromacs project directory or a separate build directory, depending on whether the build is in- or out-of-source.
-
clean_build_dir
()¶ Ensures that the current build dir is in the initial state (empty).
-
get_log_dir
(category=None)¶ Returns directory for log files.
The directory is created if necessary.
Parameters: category (Optional[str]) – Category for the log directory. Log files in the same category are put into a common subdirectory (with the name of the category), allowing Jenkins to glob them for, e.g., parsing warnings. Returns: Absolute path for the requested log directory. Return type: str
-
get_path_for_logfile
(name, category=None)¶ Returns path for producing a log file in a common location.
Directories are created as necessary, but the caller is responsible of creating the actual file.
Parameters: Returns: Absolute path for the requested log.
Return type:
-
get_project_dir
(project)¶ Returns project directory of a given project.
Parameters: project (Project) – Project whose directory should be returned. Returns: Absolute path to the project directory. Return type: str
-
upload_revision
(project, file_glob=’*’)¶ Upload a new version of the patch that triggered this build, but only if files in the glob changed and it came from the specified project.
Parameters: - project (Project) – Enum value to choose which project might be updated
- file_glob (str) – glob describing the files to add to the patch
-
API for Jenkins¶
The following functions from the releng
package are intended to be called
from scripts in Jenkins build configuration or from pipeline scripts
(see Jenkins configuration).
-
releng.
run_build
(build, job_type, opts, project=’gromacs’)¶ Main entry point for Jenkins builds.
Runs the build with the given build script and build parameters. Before calling this script, the job should have checked out the releng repository to a
releng/
subdirectory of the workspace, and the repository that triggered the build, similarly in a subdirectory of the workspace.See Jenkins scripts (releng Python module) for more details on the general build organization.
Parameters: - build (str) – Build type identifying the build script to use.
Names without directory separators are interpreted as
gromacs/admin/builds/build.py
, i.e., loaded from the main repository. - job_type (JobType) – Type/scope of the job that can, e.g., influence the scope of testing. Not all build scripts use the value.
- opts (List[str]) – This is mainly intended for multi-configuration builds. Build scripts not intended for such builds may simply ignore most of the parameters that can be influenced by these options.
- build (str) – Build type identifying the build script to use.
Names without directory separators are interpreted as
-
releng.
read_build_script_config
(script_name)¶ Reads build options specified in a build script.
Parameters: script_name (str) – Name of the build script (see run_build()).
-
releng.
prepare_multi_configuration_build
(configfile)¶ Main entry point for preparing matrix builds.
Reads a file with configurations to use (one configuration per line, with a list of space-separated build options on each line; comments starting with # and empty lines ignored).
Parameters: configfile (str) – File that contains the configurations to use. Names without directory separators are interpreted as gromacs/admin/builds/configfile.txt
.
-
releng.
get_actions_from_triggering_comment
()¶ Processes Gerrit comment that triggered the build.
Parses the comment that triggered an on-demand build and returns a structure that tells the workflow build what it needs to do.
-
releng.
do_ondemand_post_build
(inputfile)¶ Does processing after on-demand builds have finished.
Reads a JSON file that provides information about the builds (and things forwarded from the output of get_actions_from_triggering_comment()), and returns a structure that specifies what to post back to Gerrit.
Can also perform other actions related to processing the build results, such as posting cross-verify messages.
Parameters: inputfile (str) – File to read the input from, relative to working dir.
-
releng.
get_build_revisions
()¶ Provides information about revisions used in the build.
Returns a structure that provides a list of projects and their revisions used in this build.
-
releng.
read_source_version_info
()¶ Reads version info from the source repository.
Returns a structure that provides version information from the source repository.