Running Tsunami HySEA inside a singularity container on Leonardo
These instructions describe how to run Tsunami HySEA simulations inside a singularity container on Leonardo.
To run Tsunami HySEA inside a singularity container on Leonardo, you need:
1. The singularity image of HySEA and
2. A Spack environment with the same versions of openmpi and cuda used in the image
1. Get the singularity image of HySEA
The singularity image of HySEA was created with the eFlows4HPC image service creation and can downloaded from their website, if you have an account.
Alternatively, the image can be found on Leonardo in the shared directory /leonardo_work/DTGEO_T1_2/images
with the name dt-geo_t-hysea_x86_64_openmpi_4.1.4_cuda_11.8_v_dt-geo.sif
.
Note that this image was created with specific versions of openmpi (4.1.4) and cuda (11.8) to match what is available on Leonardo. If you want to use different versions or use the image in a HPC cluster with a different architecture, then you need to create a new image.
2. Create and activate the Spack environment
Now we need to create a Spack environment with openmpi@4.1.4 and cuda@11.8. If you already have created a Spack environment to run HysEA on Leonardo outside a container, then you can simply activate that environment and move to the next point. If you don't have a Spack environment yet, you can either use an existing one, following the instructions in section 1 here or create a new one.
To create a new Spack environment:
module load spack
mkdir envhysea
cd envhysea
spack env create -d .
spack env activate -p .
spack add openmpi@4.1.4+cuda
spack concretize
spack -d install
spack find
. At the moment of writing, the version of cuda installed with this method is 11.8 (cuda@11.8.89 in the output of spack find
). If the version is different, then you have to specify the version of cuda you want to use (11.8) when creating the spack environment to fit the one that is installed in the singularity image.
Now you have a spack environment that you can activate any time you want with the command spack env activate -p .
.
Info
Here you can find some useful Spack commands.
3. Run HySEA inside the singularity container
Spack environment
Any time you want to run HySEA you need to activate the environment envhysea and load the openmpi module (which should automatically load also cuda).
module load spack
spack env activate -p PATH-TO/envhysea
spack load openmpi
To run HySEA inside a container we want to use SingularityPRO.
On Leonardo, both SingularityPRO and the normal version of Singularity are available. SingularityPRO is already installed in the system, so no action is needed to be able to use it, whereas the normal version of Singularity can be loaded as a module (
module load singularity
). The benefit of using SingularityPRO instead of the normal Singularity is that SingularityPRO does not need a lot of memory to create the container from the image. Because the image we have of HySEA is quite large (>8GB), when using the normal version of Singularity, you would need to reserve an entire node when submitting the job, even though you might want to use only 1 GPU inside that node. Therefore, it is better, and faster, to use SingularityPRO.
If you type
singularity --version
you should see this output SingularityPRO version 3.9-9.el8
.
Copy test folder
Copy the directory /leonardo_work/DTGEO_T1_2/testruns_hysea
into your home directory. Here you can find input files for HySEA and shell scripts to run some tests.
Set the right paths
Open the file runHySEA_singularity.sh
and write the full path to the testruns_hysea directory in this line:
mpirun singularity exec --nv --no-home --bind <FULLPATH-TO-ROOT-OF>/testruns_hysea:/testruns_hysea /leonardo_work/DTGEO_T1_2/images/dt-geo_t-hysea_x86_64_openmpi_4.1.4_cuda_11.8_v_dt-geo.sif testruns_hysea/run-inside-container.sh
Run 1 simulation
The first example is to run one simulation inside the singularity container. The input files for this simulation are inside the directory BS_Scenario0001
.
To run the simulation, make sure you stand in the directory where the file runHySEA_singularity.sh
is and have the spack environment active, then submit the job to the queue:
sbatch runHySEA_singularity.sh
mpirun singularity exec ... run-inside-container.sh
. This command executes the script run-inside-container.sh inside the container. This is where the commands get_load_balancing
to create the grid and TsunamiHySEA
are called. The
--nv
flag is necessary because we want to use GPUs. The --bind
flag is to mount the folder needed to run the simulation inside the container.
When the job is done, you should be able to see a file called out_ts.nc
inside BS_Scenario0001
.
Run 4 simulations
To run more than one simulation we use the files inside Step2_BS
. Here there are 4 directories, each of them contains all the input files needed by HySEA. Note that in this example the grid created by get_load_balancing
is already there.
Open the file runHySEA_singularity.sh
and follow the instructions at the bottom of the file to ask for 4 GPUs instead of one and to use the right mpirun singularity exec ...
call that mounts the Step2_BS directory.
Then submit the job to the queue:
sbatch runHySEA_singularity.sh
When the job is done, you should be able to see a file called out_ts.nc
inside each subdirectory in Step2_BS
.
Problems?
Check our troubleshooting guide.