GEM Considerations

This section discusses some specific considerations for the forthcoming integration of the GEM dynamical core into CanCPL and CanESM.

The current communication strategy used in the GEM-NEMO coupled model used at MRD differs from the current CanESM coupling. GEM and NEMO exist in their own MPI_COMM_WORLDs and communicate over TCP/IP using the GOSSIP communication layer (as illustrated in the following figure). This differs from the CanESM coupling paradigm which splits a single MPI_COMM_WORLD into communicators for each model component.

GEM-NEMO communication topology using TCP/IP based GOSSIP communication layer. Each component exists in its own separate MPI_COMM_WORLD not just separate MPI communicators.

Additionally, GEM uses the RPN_COMM_LIBRARY which wraps much of the MPI API and deals with most of the MPI communications within GEM. This includes the splitting of the MPI_COMM_WORLD into separate communicators for the Yin and Yang submodels which form the global overset grid.

The MPI initialization of GEM will likely need to be modified to conform to the CanESM communication paradigm which requires a single MPI_COMM_WORLD for all components within the coupled model. Two ways forward have been proposed.

1. During the initialization of GEM, GEM is responsible for splitting the MPI_COMM_WORLD into a GEM and non-GEM set of communicators. Both of those are then passed into define_groups in CanCPL. CanCPL and NEMO during their call to define_groups splits the GEM communicator between the two.

2. GEM calls define_groups before the Yin-Yang grid communicators are split off. This requires relinquishing control from GEM over the entire MPI_COMM_WORLD. This route requires less refactoring of CanCPL than the first option, but is slightly more invasive into the GEM/RPN initialization.