I'm running an OpenFOAM simulation on a cluster. I have used the Scotch decomposition method and my decomposeParDict looks like this:

FoamFile { version 2.0; format ascii; class dictionary; object decomposeParDict; } numberOfSubdomains 6; method scotch;

checkMesh and decomposePar finish with no issues. I have assigned 6 nodes to the slurm by

srun -N6 -l sonicFoam

and the solver runs smoothly without any errors.

The issue is the solution speed is not improved in comparison to the non-parallel simulation I ran before. I want to monitor the CPU usage to see if all of the 6 nodes I have assigned are similarly loaded. The squeue --user=foobar command return the jobNumber and list of nodes assigned ( NODELIST(REASON) ) which looks like this:

foo,bar[061-065]

from sinfo command these nodes are both in debug and main* PARTITION s (which I have absolutely no idea what it means!).

This post says that you can use the sacct or sstat commands to monitor CPU time and memory usage of a slurm job. But when I run

sacct --format="CPUTime,MaxRSS"

it gives me:

CPUTime MaxRSS ---------- ---------- 00:00:00 00:00:00 00:07:36 00:00:56 00:00:26 15:26:24

which I can not understand. And when I specify the job number by

sacct --job=<jobNumber> --format="UserCPU"

The return is empty. So my questions are

Is my simulation loading all nodes or is it running on one or two and the rest are free?

am I running the right commands? if yes what those numbers mean? how they represent the CPU usage per node?

If not then what are the right --format="..." s for sacct and/or sstat (or maybe other slurm commands) to get the CPU usage/load?

P.S.1. I have followed the OpenFOAM compiling following the official instructions. I did not do anything with OpenMPI and it's mpicc compiler for that matter though.

P.S.2 For those of you who might end up here. Maybe I'm running the wrong command apparently one can first allocate some resources by:

srun -N 1 --ntasks-per-node=7 --pty bash

where 7 is the number of cores you want and bash is just a name. and then run the solver with:

mpirun -np 7 sonicFoam -parallel -fileHandler uncollated

I'm not sure yet though.