OpenFOAM

OpenFOAM v6 is available on Mesabi. Several working examples are posted, which illustrate how to run interactive with graphics using ParaFOAM, and how to run non-interactively in the PBS batch queue.

Run Interactively with Graphics
In a terminal window on an MSI NICE or NX desktop
ssh -X mesabi
/home/dhp/public/sing/openfoam6/run

This will start  an instance of the openfoam6-paraview54  singularity container. At the prompt in this container, enter you can  get and run backward facing step with simpleFoam with:
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
cd pitzDaily
blockMesh
simpleFoam
 

View results with Paraview

paraFoam

Exit container
exit

Run in PBS batch
From a login shell on Mesabi, get a copy of the  OpenFOAM v6 tutorial pitzDaily example, PBS run script, and submit it to the queue:
cp -r /panfs/roc/msisoft/openfoam/6.0/pitzDaily_v6 .
cd pitzDaily_v6
cp /panfs/roc/msisoft/openfoam/6.0/pbs_run .
qsub pbs_run

NOTES

Problem size

This is a tiny example which runs for a very short time.  The PBS resource request only asks for 2 cores and 10 min of run time.    In fact, it’s running OpenFOAM single-threaded, and only asks for 2 cores to ensure it has enough memory.  To run larger problems, request all 24 cores on a node and a longer wall clock time. For example, to run for as long as 6 hours and have the full memory of the Mesabi compute node, use a resource request like

#PBS -l nodes=1:ppn=24,pmem=2580mb,walltime=06:00:00

Customize problem

This example runs   blockMesh,   simpleFoam, and foamToVTK   non-interactively in a PBS job, using a singularity container.  It does so by first creating a bash script (sing_input.sh) and then running the container on this script with

   singularity exec $container_file $wrkdir/sing_input.sh

You can modify run_pbs to run your own OpenFOAM commands for your own use case.

Working directory

This example is setup to work as-is for any user, so you do not need to make any modifications to see it run.  It has you copy the OpenFOAM v6 tutorial pitzDaily directory to a location anywhere under your home directory.  You then use this directory as a working directory (i.e.,. where you run your job, and OpenFOAM does its work). The PBS script (see the file pbs_run copied below) generates a path to the working directory in the shell variable   wrkdir   with

    wrkdir=$HOME/$(echo $PWD | cut -d'/' -f8-)

You can leave this as is if you want to have your PBS run script in the OpenFOAM project directory.

Example PBS Batch File  (pbs_run)

#!/bin/bash -l
#PBS -l nodes=1:ppn=2,pmem=2580mb,walltime=00:10:00
#PBS -j oe

cd $PBS_O_WORKDIR

# Construct working directory path that will be valid in container
wrkdir=$HOME/$(echo $PWD | cut -d'/' -f8-)

# Create bash shell scripti with commands you want to run in container
echo "#!/bin/bash -l

# Set OpenFOAM6 environment, and working directory, inside of container
source /opt/openfoam6/etc/bashrc
cd $wrkdir

# Your OpenFOAM commands go here (must be non-interactive)
blockMesh
simpleFoam
foamToVTK
" > sing_input.sh
chmod 755 sing_input.sh

module load singularity
container_file=/panfs/roc/msisoft/openfoam/6.0/openfoam6-paraview54.simg
singularity exec $container_file $wrkdir/sing_input.sh