observablesreducer.h File Reference

#include <cstdint>
#include <functional>
#include <memory>
#include <vector>

Include dependency graph for observablesreducer.h:

Description

Declares gmx::ObservablesReducer and builder.

Periodically modules implementing MD simulations need to communicate with all collaborating ranks to do things like compute observables like total energies, signal conditions, and check internal consistency. This communication synchronizes all participating ranks, which limits scaling and performance, so it is done rarely (typically once per step) and only when required.

Modules may provide data of type double to be reduced across all ranks via an MPI all-reduce with MPI_SUM. Double-precision floating-point is chosen so that no meaningful precision is lost e.g. in computing energies, while also permitting integral or boolean messages to be passed as double-precision floating-point values.

Different modules typically need to communicate on different MD steps, so in principle one might optimize by filling a std::vector<double> with the values required on the current step. However, that requires that each module produce and then copy to the reduction buffer the data for this step. The typical amount of data required even if all modules need to participate (ie. hundreds of doubles) is smaller than the message headers that are used by the underlying network transport protocol. So optimizing for minimum message size is not particularly effective because it does not meaningfully reduce the total time taken to communicate.

Instead, we always reduce a buffer of the size that would be needed if all active modules required communication this step. Then no module needs to copy data merely to achieve reduction. To achieve this, each module needs a stable view of memory into which it can store data for which reduction is desired. It also means that modules not active in the current simulation do not contribute to the workload at run time. Also, modules that are active but don't need communication at any particular MD step can passively opt out and that incurs no overhead.

The functionality is separated two main components, one that does work during the simulation, and a builder that is used only during setup time. This separates the responsibilities of

• allowing subscription and building the communication buffer, from
• orchestrating the minimum activity needed for this MD step.

The interaction diagrams for those two workflows are depicted below.

Once the observablesReducer is built, the builder may be destructed.

The observablesReducer and its modules operate entirely by passing callbacks.

Three callbacks are produced and called per participating module:

1. One produced by the module and passed to the builder so that later the ObservablesReducer can call it to notify the module that reduction is complete.
2. One produced by the builder and returned to the module so the latter can call it to require reduction when it wishes
3. One produced by the module and passed to the builder so the latter can call it to notify the former of the buffer view it should use in the first callback and receive a copy of the second callback.

Modules often request that reduction occur "soon" ie. this step or next step, depending whether reduction has already take place this MD step. However they are also able to request reduction to occur "eventually" ie. only whenever some other module requires it, so the total number of reductions is minimized. Naturally, the callback to such a module happens only after the eventual reduction, which may happen on the same step or a later one. If a module makes more than one "eventually" reduction request before reduction takes place, the callback to that module will be called multiple times when eventually reduction does take place. It is the responsibility of the module to refrain from making those requests if the multiple callbacks would be a problem (e.g. maintain an internal record of whether a reduction request has been made). Modules are not required to set any value for reduction unless they are requesting reduction.

An ObservablesReducer object is intended to replace the use of compute_globals() by simulations, as https://gitlab.com/gromacs/gromacs/-/issues/3887 progresses. When no modules using the legacy style communication remain, it is anticipated that this class will change to contain an MPI communicator to use to implement the MPI_Allreduce internally. At that time, communicationBuffer() and reductionComplete() will likely change into a doReduction() method, or similar. The flow of the whole propagator loop will now be less clear inasmuch as the responsibility for requesting reduction now lies with each module, however this is probably still more clear than the large forest of flags that resulted from all modules having to have their control logic in the propagator loop.

Classes
class	gmx::ArrayRef< typename >
	STL-like interface to a C array of T (or part of a std container of T). More...
class	gmx::ObservablesReducerBuilder
	Builder for ObservablesReducer. More...
class	gmx::ObservablesReducer
	Manage reduction of observables for registered subscribers. More...

Typedefs
using	gmx::Step = int64_t
	Step number.

Enumerations
enum	gmx::ReductionRequirement : int { gmx::ReductionRequirement::Soon, gmx::ReductionRequirement::Eventually }
	Control whether reduction is required soon. More...
enum	gmx::ObservablesReducerStatus : int { gmx::ObservablesReducerStatus::ReadyToReduce, gmx::ObservablesReducerStatus::AlreadyReducedThisStep }
	Report whether the reduction has happened this step. More...