Partitions

Slurm Partitions

MSI uses the Slurm scheduler to fairly allocate compute resources to users of our systems. Slurm uses different partitions to organize jobs with similar characteristics. The job partitions on our systems manage different sets of hardware, and have different limits for quantities such as wall clock time (walltime), available processors, and available memory. When submitting a calculation it is important to choose a partition where the job is suited to the hardware and resource limitations.

Selecting a Partition

Each MSI system contains job partitions managing sets of hardware with different resource and policy limitations. MSI currently has three primary systems: the Agate cluster, the Mesabi cluster, and the Mesabi expansion Mangi. These systems have high-performance hardware and a wide variety of partitions suitable for many different job types. Agate is the latest cluster with the best hardware, and should be your first choice for submitting jobs. Which system to choose depends highly on which system has partitions appropriate for your software/script. More information about selecting a partitions and the different partition parameters can be found on the Choosing A Partition (Slurm) page.

Below is a summary of the available partitions organized by system, and the associated limitations. The quantities listed are totals or upper limits.

Federated Partitions

Paritions for jobs that can run on either Agate or Mesabi/Mangi.
Job arrays will only run on the cluster you submit the job on.

Partition name	Node sharing?	Cores per node	Walltime limit	Total node memory	Advised memory per core	Local scratch per node	Maximum nodes per job
msismall	Yes	24-128	96:00:00	40-499 GB	1900 MB	380-850 GB	1
msilarge	No	24-128	24:00:00	40-499 GB	1900 MB	380-850 GB	32
msibigmem	Yes	24-128	24:00:00	499-1995 GB	3999 MB	380-850 GB	1
msigpu	Yes	24-128	24:00:00	40-499 GB	1900 MB	380-850 GB	1
interactive	Yes	24-128	24:00:00	60-499 GB	2 GB	228-850 GB	2
interactive-gpu	Yes	24-128	24:00:00	60-499 GB	2 GB	228-850 GB	2
preempt	Yes	24-64	24:00:00	60-499 GB	2 GB	228-850 GB	1-2
preempt-gpu	Yes	24-64	24:00:00	60-499 GB	2 GB	228-850 GB	1-2

Note: Partitions with the same name across clusters are federated (interactive, interactive-gpu, preempt, preempt-gpu).

Agate Partitions

Partition name	Node sharing?	Cores per node	Walltime limit	Total node memory	Advised memory per core	Local scratch per node	Maximum nodes per job
agsmall	Yes	128	96:00:00	499 GB	3999 MB	850 GB	1
aglarge	No	128	24:00:00	499 GB	3999 MB	850 GB	32
ag2tb	Yes	128	96:00:00	1995 GB	15.5 GB	850 GB	1
a100-4⁽¹⁾	Yes	64	96:00:00	499 GB	3999 MB	850 GB	4
a100-8⁽¹⁾	Yes	128	24:00:00	1002 GB	7.5 GB	850 GB	1
interactive	Yes	128	24:00:00	499 GB	3999 MB	850 GB	2
interactive-long⁽²⁾	Yes	128	37-0	499 GB	16 GB	850 GB	1
interactive-gpu⁽³⁾	Yes	64	24:00:00	499 GB	8000 MB	850 GB	2
preempt	Yes	128	24:00:00	499 GB	3999 MB	850 GB	1
preempt-gpu⁽³⁾	Yes	64	24:00:00	499 GB	8000 MB	850 GB	1

⁽¹⁾Note: Jobs using an Agate partition must be submitted from an Agate host (ex. Agate login node).

(2)Note: May only use 2 CPU and 32 GB of memory per user.

⁽³⁾Note: In addition to selecting a100-4 or a100-8 GPU partitions, GPUs need to be requested for all GPU jobs.

One or more A100 GPU can be requested by including the following two lines in your submission script:
This example asks for a single A100 GPU, using the a100-4 partition

#SBATCH -p a100-4
#SBATCH --gres=gpu:a100:1

⁽³⁾Note: The interactive-gpu and preempt-gpu partitions contain A40 GPUs, so include the following two lines in your submission script:
This example asks for a single A40 GPU, using the interactive-gpu partition

#SBATCH -p interactive-gpu
#SBATCH --gres=gpu:a40:1

Mangi and Mesabi

Partition name	Node sharing?	Cores per node	Walltime limit	Total node memory	Advised memory per core	Local scratch per node	Maximum nodes per job
amdsmall	Yes	128	96:00:00	248 GB	1900 MB	415 GB	1
amdlarge	No	128	24:00:00	248 GB	1900 MB	415 GB	32
amd512	Yes	128	96:00:00	499 GB	4000 MB	415 GB	1
amd2tb	Yes	128	96:00:00	1995 GB	15 GB	415 GB	1
v100 ⁽¹⁾	Yes	24	24:00:00	374 GB	15 GB	859 GB	1
small	Yes	24	96:00:00	60 GB	2500 MB	429 GB	10
large	No	24	24:00:00	60 GB	2500 MB	429 GB	48
max	Yes	24	696:00:00	60 GB	2500 MB	429 GB	1
ram256g	Yes	24	96:00:00	248 GB	10 GB	429 GB	2
ram1t ⁽²⁾	Yes	32	96:00:00	1002 GB	31 GB	380 GB	2
k40 ⁽¹⁾	Yes	24	24:00:00	123 GB	5 GB	429 GB	40
interactive ⁽³⁾	Yes	24	24:00:00	60 GB	2 GB	228 GB	2
interactive-gpu ⁽³⁾	Yes	24	24:00:00	60 GB	2 GB	228 GB	2
preempt ⁽⁴⁾	Yes	24	24:00:00	60 GB	2 GB	228 GB	2
preempt-gpu ⁽⁴⁾	Yes	24	24:00:00	60 GB	2 GB	228 GB	2

⁽¹⁾Note: In addition to selecting a GPU partition, GPUs need to be requested for all GPU jobs. A k40 GPU can be requested by including the following two lines in your submission script:

#SBATCH -p k40
#SBATCH --gres=gpu:k40:1

A V100 GPU can be requested by including the following two lines in your submission script:

#SBATCH -p v100
#SBATCH --gres=gpu:v100:1

⁽²⁾Note: The ram1t nodes contain Intel Ivy Bridge processors, which do not support all of the optimized instructions of the Haswell processors. Programs compiled using the Haswell instructions will only run on the Haswell processors.

⁽³⁾Note: Users are limited to 2 jobs in the interactive and interactive-gpu partitions.

⁽³⁾Note: Jobs in the preempt and preempt-gpu partitions may be killed at any time to make room for jobs in the interactive or interactive-gpu partitions.