Tuning job requests

Tuning your jobs can make a big difference to the amount of time they spend on the cluster, while ensuring that resources are not wasted.

Tasks

Asking for more tasks may mean a longer wait for the job to finish - for example, a particular job will run for three hours if it has four tasks, and one hour if it has twelve tasks. But waiting for twelve tasks could mean it sits in the queue for eight hours before the job starts, meaning the results don't come back for nine hours rather than three.

It is also important to make sure your job can use more than one task, otherwise your job may be queued for days for no reason at all, and waste resources when it runs.

Memory tuning

Asking for the right amount of memory is also important to ensure the job runs on the cluster as soon as it can. Ask for too much and the job can queue for a long time and reserve the whole memory allocation exclusively while the job runs - ask for too little, and the job will be terminated by the scheduler. Either way, resources are wasted.

Memory requirements for exclusive node access

For jobs using full and multiple nodes, such as parallel jobs, and serial jobs requesting --exclusive, the memory requirement still needs to be specified, as this exclusive option only requests all the CPUs per node.

Try with an initial sensible value, based on the data size and previous experience with the application. If you aren't sure, the standard way to work out the memory needed is to try the default (1GB per task), and enable email notifications in your job submission script with --mail-type=ALL. The requested memory can be easily changed in the job script. When the job finishes, an email will be sent providing the Memory Utilized and Memory Efficiency values, which displays the memory used by the job. Depending on whether the memory was under or over-estimated, you can use this information to refine your memory request for the next job.

Memory requests with array jobs

When running an array job, please run a single task from the array first to retrieve the memory usage requirements. Submitting large array jobs which subsequently fail due to exceeding memory requirements, or jobs which only use a small fraction of the requested memory are a waste of valuable cluster resources.

Analysing job history using jobstats utility

You can view details of your recent jobs by running the jobstats command. By default it will show a list of your 25 most recent jobs that completed within the last 7 days. Usage information can be seen with the -h flag:

$ jobstats -h

Utility for displaying recent jobs. By default, the 25 most recent jobs from
the last 7 days are shown.

USAGE: jobstats [ -h ] [ -a ACCOUNT(S) ] [ -A ] [ -b START_DATE ] [ -c ]
                [ -e END_DATE ] [ -f | -s ] [ -j JOB_NUMBER(S) ] [ -l ]
                [ -n NODE(S) ] [ -N NUMBER ] [ -p PARTITION(S) ]
                [ -u USER(S) ]

OPTIONS:
 -a Account(s) to show jobs for (multiple accounts should be comma-separated)
 -A Show all recent jobs (default limit: 25 jobs)
 -b Show jobs that started after midnight on DATE (DATE format is "YY-MM-DD")
 -c Strip colours from output
 -e Show jobs that ended before midnight on DATE (DATE format is "YY-MM-DD")
 -f Show only failed jobs. Can not be used together with the -s option
 -h Displays this help prompt and exits
 -j Show the given JOB_NUMBER(s)
 -l Show long output (with extra fields: Submit Date, Start Date etc.)
 -n Show only nodes that match the given pattern (i.e. node1, node[1,3-5])
 -N Show the most recent NUMBER of jobs
 -p Show job(s) for given partition(s)
 -s Show only successful jobs. Can not be used together with the -f option
 -u Filter jobs by username (default: current user - USERNAME)

At this point, the easiest tuning you can do is to look at the requested memory (MEM R) and maximum memory used (MEM U) fields. If there's a discrepancy between the memory you requested and the amount you used, you can then reduce the amount of requested memory for the next job iteration. This will ensure that your job has a good chance of running sooner than it otherwise would have done, and does not waste resources.

Analysing resource usage via job emails

Enabling job completion emails will provide information about your completed job, including runtime duration and exit state.

Below shows an example job output email:

Job info
---------------------------------------------------------
Job ID: 123456
User: abc123
Nodes: ddy1
Partition: compute

Job efficiency:
------------------------------------------------------
Please review the CPU and Memory efficiency data
below to optimise future job resource requests
------------------------------------------------------
Job ID: 123456
Cluster: apocrita
User/Group: abc123/qmul
State: COMPLETED (exit code 0)
Cores: 1
CPU Utilized: 00:10:00
CPU Efficiency: 0.90% of 00:09:03 core-walltime
Job Wall-clock time: 00:09:03
Memory Utilized: 836.00 MB
Memory Efficiency: 84% of 1.00 GB (1.00 GB/core)

Storage

Temporary local node storage is significantly faster than accessing networked storage (GPFS), such as your home directory, or a lab share.

When a job starts, the scheduler will provision a temporary storage location on the node executing your job, under /tmp. This may be accessed using the $TMPDIR variable, also set by the scheduler when the job starts:

TMPDIR=/tmp

Copying datasets to these directories before use will speed up access. To take full advantage of this speed up you can:

# Copy all datafiles to ${TMPDIR}
cp /data/Example/data ${TMPDIR}

# Run script outputting to ${TMPDIR}
./script -i ${TMPDIR}/data -o ${TMPDIR}/output_data

# Copy output data back
cp ${TMPDIR}/output_data /data/Example/

See the Using TMPDIR page for more information.