Skip to content

Slurm overview

Slurm is an open source cluster management and job scheduling system for Linux clusters. As a cluster workload manager, Slurm has three key functions:

  1. It provides access to computing resources for computational workloads.
  2. It runs and tracks work on the assigned resources.
  3. It manages a queue to fairly share resources.

Terminology

Before exploring job submission, users should familiarise themselves with the terminology and concepts used throughout this site, including the main components of the system hardware.

  • Job: A job is a unit of work submitted to Slurm. It represents the computation or workflow you want to run on the cluster, such as a simulation, data analysis, or code execution. Each job is assigned a unique job number by Slurm on submission, which you can use to track, monitor, or cancel the job.

  • Node: A node is a single physical server in the cluster. Each node has CPUs (cores), memory, and sometimes GPUs. Jobs run on one or more nodes depending on the resources requested.

  • Partition: A partition is a group of nodes with similar hardware or special features (for example: GPU nodes, high memory nodes, or parallel nodes). Partitions act like "queues" that organise nodes and control where jobs can run.

  • Core: A core is a single processing unit within a CPU (central processing unit) on a node. Apocrita nodes contain multiple cores, allowing for multiple tasks to run simultaneously.

  • Task: A task is a single process or thread within a job. Jobs can request multiple tasks running in parallel, either on one node or across multiple nodes. For simplicity, cores and tasks should be considered equivalent on Apocrita for most use cases.

  • Job script: A text file containing the required information to run a job.

Job management

To submit a job script, use the sbatch command from a login node, for example:

sbatch job_script.sh

The job will be assigned a unique job number, and added to the queue. When resources become available, the job will become eligible to run.

Jobs in the queue can be monitored using the squeue command. To display only the jobs you have submitted, use squeue --me, for example:

$ squeue --me
  JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
    456   compute job_scri   abc123 PD       0:00      1 (Resources)
    123   compute job_scri   abc123  R       1:43      1 ddy100

In this example, job 123 is currently running (state R), and job 456 is being held in the queue in state pending (PD) awaiting resources.

To cancel a running or queued job, use scancel JODIB. For example, to cancel the above running job number 123:

scancel 123

The scancel command also accepts the --me parameter, but be aware this will cancel all your running and queued jobs, so should be used with caution.