gmx rotmat plots the rotation matrix required for least squares fitting a conformation onto the reference conformation provided with -s. Translation is removed before fitting. The output are the three vectors that give the new directions of the x, y and z directions of the reference conformation, for example: (zx,zy,zz) is the orientation of the reference z-axis in the trajectory frame.
This tool is useful for, for instance, determining the orientation of a molecule at an interface, possibly on a trajectory produced with gmx trjconv -fit rotxy+transxy to remove the rotation in the x-y plane.
Option -ref determines a reference structure for fitting, instead of using the structure from -s. The structure with the lowest sum of RMSD's to all other structures is used. Since the computational cost of this procedure grows with the square of the number of frames, the -skip option can be useful. A full fit or only a fit in the x-y plane can be performed.
Option -fitxy fits in the x-y plane before determining the rotation matrix.