Skip to content

Job statistics

In addition to the standard Slurm command-line utilities, we have written a utility script that displays job information in a more convenient way.

Personal job history utility

You can check your personal job history (including resources requested vs. resources used) using the jobstats utility. By default, it will show a list of your 25 most recent jobs that completed within the last 7 days. Usage information can be seen with the -h flag:

$ jobstats -h

Utility for displaying recent jobs. By default, the 25 most recent jobs from
the last 7 days are shown.

USAGE: jobstats [ -h ] [ -a ACCOUNT(S) ] [ -A ] [ -b START_DATE ] [ -c ]
                [ -e END_DATE ] [ -f | -s ] [ -j JOB_NUMBER(S) ] [ -l ]
                [ -n NODE(S) ] [ -N NUMBER ] [ -p PARTITION(S) ]
                [ -u USER(S) ]

OPTIONS:
 -a Account(s) to show jobs for (multiple accounts should be comma-separated)
 -A Show all recent jobs (default limit: 25 jobs)
 -b Show jobs that started after midnight on DATE (DATE format is "YY-MM-DD")
 -c Strip colours from output
 -e Show jobs that ended before midnight on DATE (DATE format is "YY-MM-DD")
 -f Show only failed jobs. Can not be used together with the -s option
 -h Displays this help prompt and exits
 -j Show the given JOB_NUMBER(s)
 -l Show long output (with extra fields: Submit Date, Start Date etc.)
 -n Show only nodes that match the given pattern (i.e. node1, node[1,3-5])
 -N Show the most recent NUMBER of jobs
 -p Show job(s) for given partition(s)
 -s Show only successful jobs. Can not be used together with the -f option
 -u Filter jobs by username (default: current user - USERNAME)

Looking into the output, you will be able to find out how resources were used during job execution.

$ jobstats

JOBS STATISTICS FOR USER: ABC123 (25 MOST RECENT JOBS FROM LAST 7 DAYS)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|     JOB ID     |   NAME   |   USER   |     ACCOUNT     |       ENDED       |  TIME REQ.  |  DURATION   | MEM R  | MEM U  | CPU | G |  PARTITION   |    NODE(S)  |   STATE   |
+----------------+----------+----------+-----------------+-------------------+-------------+-------------+--------+--------+-----+---+--------------+-------------+-----------+
|         123456 | interact |   abc123 | billing-account | 26-03-03 17:28:19 |    10:00:00 |    05:10:05 |   1.0G |   7.4M |   1 | - |      compute |       ddy87 |    FAILED |
|         123457 | ood_rstu |   abc123 | billing-account | 26-03-04 13:18:57 |    10:00:00 |    09:03:33 |  16.0G |   1.0G |   1 | - |       comput |      ddy131 | CANCELLED |
|         123458 | ood_rstu |   abc123 | billing-account | 26-03-04 13:22:51 |    01:00:00 |    00:03:27 |  16.0G | 174.7M |   1 | - | computeshort |      ddy131 | CANCELLED |
|         123459 | interact |   abc123 | billing-account | 26-03-05 09:19:45 |    01:00:00 |    00:00:09 |   1.0G |   7.3M |   1 | - |      compute |       ddy79 | COMPLETED |
|         123460 | interact |   abc123 | billing-account | 26-03-05 10:25:37 |    01:00:00 |    00:35:15 |  16.0G |   1.7G |   4 | - |      compute |      ddy116 | COMPLETED |
|         123461 | interact |   abc123 | billing-account | 26-03-05 10:50:43 |    01:00:00 |    00:17:05 |  16.0G | 548.9M |   4 | - |      compute |      ddy110 | COMPLETED |
|         123462 | ood_rstu |   abc123 | billing-account | 26-03-05 11:26:21 |    01:00:00 |    00:04:45 |  16.0G |   1.0G |   1 | - | computeshort |       ddy74 | CANCELLED |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

If your terminal is set to display output in smaller size fonts, you may use -l option to extend the output table width (fields SUBMITTED and STARTED will additionally be shown).

$ jobstats -l -N 5

LAST 5 JOBS STATISTICS FOR USER: ABC123
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|     JOB ID     |   NAME   |   USER   |     ACCOUNT     |     SUBMITTED     |      STARTED      |       ENDED       |  TIME REQ.  |  DURATION   | MEM R  | MEM U  | CPU | G |  PARTITION   |    NODE(S)  |   STATE   |
+----------------+----------+----------+-----------------+-------------------+-------------------+-------------------+-------------+-------------+--------+--------+-----+---+--------------+-------------+-----------+
|         123458 | ood_rstu |   abc123 | billing-account | 26-03-04 13:19:20 | 26-03-04 13:19:24 | 26-03-04 13:22:51 |    01:00:00 |    00:03:27 |  16.0G | 174.7M |   1 | - | computeshort |      ddy131 | CANCELLED |
|         123459 | interact |   abc123 | billing-account | 26-03-05 09:19:05 | 26-03-05 09:19:36 | 26-03-05 09:19:45 |    01:00:00 |    00:00:09 |   1.0G |   7.3M |   1 | - |      compute |       ddy79 | COMPLETED |
|         123460 | interact |   abc123 | billing-account | 26-03-05 09:50:20 | 26-03-05 09:50:22 | 26-03-05 10:25:37 |    01:00:00 |    00:35:15 |  16.0G |   1.7G |   4 | - |      compute |      ddy116 | COMPLETED |
|         123461 | interact |   abc123 | billing-account | 26-03-05 10:33:30 | 26-03-05 10:33:38 | 26-03-05 10:50:43 |    01:00:00 |    00:17:05 |  16.0G | 548.9M |   4 | - |      compute |      ddy110 | COMPLETED |
|         123462 | ood_rstu |   abc123 | billing-account | 26-03-05 11:21:30 | 26-03-05 11:21:36 | 26-03-05 11:26:21 |    01:00:00 |    00:04:45 |  16.0G |   1.0G |   1 | - | computeshort |       ddy74 | CANCELLED |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Short fields reference

  • JOB ID - Job ID number (plus task ID for array jobs)
  • NAME - Job name
  • USER - Username of the job owner
  • ACCOUNT - Account the job is billed to
  • SUBMITTED - Date and time when job was submitted
  • STARTED - Date and time when job started execution
  • ENDED - Date and time when job completed execution
  • TIME REQ - Job runtime requested
  • DURATION - Actual job runtime (wallclock)
  • MEM R - Total memory requested
  • MEM U - Maximum amount of memory used during execution
  • CPU - Number of tasks requested
  • G - Number of GPU(s) requested (if no GPU is requested, - will be shown)
  • PARTITION - The partition that executed the job
  • NODE - The node that executed the job
  • STATE - The job exit state

Checking the statistics of an individual job

The sacct command can give useful resource usage information on running and completed jobs:

$ sacct
JobID           JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 
123456             test    compute billing-a+          1 CANCELLED+      0:0 
123456.batch      batch            billing-a+          1  CANCELLED     0:15 
123456.exte+     extern            billing-a+          1  COMPLETED      0:0 
123457             test    compute billing-a+          1  COMPLETED      0:0 
123457.batch      batch            billing-a+          1  COMPLETED      0:0 
123457.exte+     extern            billing-a+          1  COMPLETED      0:0 
123458       interacti+    compute billing-a+          1  COMPLETED      0:0 
123458.inte+ interacti+            billing-a+          1  COMPLETED      0:0 
123458.exte+     extern            billing-a+          1  COMPLETED      0:0

To show more information such as the submit/start/end date, the requested memory and the elapsed time, provide additional formatting to the salloc command, for example:

$ sacct -j 719891 --format=jobid,jobname,partition,submit,start,end,account,alloccpus,state,reqmem,elapsed
JobID           JobName  Partition              Submit               Start                 End    Account  AllocCPUS      State     ReqMem     Elapsed
------------ ---------- ---------- ------------------- ------------------- ------------------- ---------- ---------- ---------- ----------  ---------- 
719891             test    compute 2026-03-10T15:40:15 2026-03-10T15:40:43 2026-03-10T15:41:49 billing-a+          1 CANCELLED+         1G    00:01:06
719891.batch      batch            2026-03-10T15:40:43 2026-03-10T15:40:43 2026-03-10T15:41:50 billing-a+          1  CANCELLED               00:01:07
719891.exte+     extern            2026-03-10T15:40:43 2026-03-10T15:40:43 2026-03-10T15:41:50 billing-a+          1  COMPLETED               00:01:07

Run sacct --helpformat to view a list of available formatting options.

For more detailed information about using the sacct command, please refer to its man page.

Job runtime

See this page for more information.

Memory usage

Jobs running over their memory limit will be killed by the scheduler. The maximum limit is defined by the physical memory on a compute node.

If your job is killed for breaching the requested memory limit it is important to understand why. See the Tuning job requests page for guidance on optimising your memory requests.

See this page for further information about requesting and utilising memory on the cluster.

If you are unsure about an error or exit code, you may contact us for assistance.

Job exit state

All completed jobs will be assigned with an exit state and an exit code.

Successful jobs will exit with the COMPLETED state, and code 0. All other exit states and codes indicate a failure during execution, otherwise known as a non-zero exit code.

A few common exit states have been listed below, with their recommended remedial action before re-submission:

State
 Code Error Description Recommended Action
CANCELLED 0 Cancelled by user or administrator N/A
COMPLETED 0 Job completed successfully N/A
FAILED 1 Job completed unsuccessfully; non-zero exit code or other failure condition Miscellaneous errors, such as "divide by zero" and other impermissible operations. Check the job output file for errors e.g. invalid parameter
FAILED 127 Command not found You are trying to execute a command that cannot be found. Check the output file for errors
OUT_OF_MEMORY 125 Memory requested was exceeded Increase the memory request
TIMEOUT 0 Job runtime was exceeded Increase the runtime request, or split the job if possible

See the official Slurm documentation for more information about job exit states and exit codes.