Slurm batch scheduler allows users to associate to jobs optional fields in text format, such as the name of the job or a comment. These fields are not directly used by Slurm, their value has no influence on its behavior. They are available to store any information related to the jobs and can be manipulated through standard Slurm commands. For example, these fields can be useful for differentiating, filtering, classifying or annotating jobs.

We will discover in this post an approach to use the comment field to associate structured metadata to the jobs and enrich accounting reports with custom information.

The job comment can be defined when submitting jobs with --comment argument:

$ sbatch --comment hello --wrap "sleep 60"
Submitted batch job 943

It is then visible with the scontrol show job $ID command:

$ scontrol show job 943
JobId=943 JobName=wrap

It is also possible to define the comment after the job is submitted with scontrol update job $ID command.

By default, the comment is saved by Slurm as long as the job is in scheduling queue (pending or running). Once the job is finished, the comment is lost. However, it is possible to configure Slurm to save the comment in SlurmDBD accounting database, by enabling this setting in slurm.conf configuration file:

AccountingStoreFlags = job_comment

Job comments then become persistent after jobs end and they can be retrieved with the standard Slurm accounting command sacct.

The comment field can store any form of text, without constraint on its format. In particular, this text can be a serialized representation of a data structure, such as JSON.

Here is an example of a minimalist batch shell script that sets a comment with an associative array containing a mesh key and a random integer value at the end of its execution:


# compute stuff here

MESH=$(shuf -i 0-1000 -n 1)
scontrol update job $SLURM_JOB_ID comment="{\"mesh\": ${MESH}}"

For example, by submitting two jobs with this batch script:

$ sbatch
Submitted batch job 894
$ sbatch
Submitted batch job 895

It is then possible to extract this metadata from the accounting database, by coupling the output of the sacct command to the jq utility:

$ sacct --json | jq '.jobs | map({id: .job_id, user: .user, cores: .required.CPUs, meta: .comment.job|fromjson })'
    "id": 894,
    "user": "remi",
    "cores": 1,
    "meta": {
      "mesh": 760
    "id": 895,
    "user": "remi",
    "cores": 1,
    "meta": {
      "mesh": 720

Here is a more advanced Python example script that updates the comment at the end of its execution with an associative array of 3 keys and values of different types:

 2import signal
 3import time
 4import atexit
 5import sys
 6import os
 7import subprocess
 8import random
 9import json
11def save_metadata():
12    """Save computation metadata in Slurm job's comment."""
13    job_id = os.getenv('SLURM_JOB_ID')
14    metadata = {
15        'mesh': random.randrange(0, 1000),
16        'complexity': random.random(),
17        'tag': random.choice(['choose', 'among', 'three']),
18    }
19    cmd = ['scontrol', 'update', 'job', job_id, f"comment={json.dumps(metadata)}"]
20    print(f"Saving metadata in Slurm job {job_id} comment field")
24def handle_timeout(signum, frame):
25    """Signal handler which stops the computation."""
26    signame = signal.Signals(signum).name
27    print(f"Signal {signame} ({signum}) received due to job timeout, saving "
28           "metadata and exiting properly")
29    sys.exit(0)
32def main():
33    # Bind SIGUSR1 sent by Slurm to notify of approaching job's timelimit
34    signal.signal(signal.SIGUSR1, handle_timeout)
35    # Register save_metadata() function to run just before exiting the program
36    atexit.register(save_metadata)
38    # Start fake computation for 5 minutes
39    print("Starting computation")
40    time.sleep(300.)  # simulating long interruptible computation
43if name == '__main__':
44    main()

The script registers the save_metadata() function (l11) with the atexit module (l36) to properly handle error cases and interruption by Slurm, typically in case of reached time limit or preemption.

This script has an approximate execution time of 5 minutes. It is submitted to Slurm a first time with a 10 minutes limit to let it enough time to end normally, and a second time with a 3 minutes limit with instruction to Slurm to send a SIGUSR1 signal 60 seconds before its termination:

$ sbatch --time 10 --wrap "srun python3 -u"
Submitted batch job 10773
$ sbatch --time 3 --signal USR1@60 --wrap "srun python3 -u"
Submitted batch job 10774

Here are the job outputs in both cases:

$ cat slurm-10773.out
Starting computation
Saving metadata in Slurm job 10773 comment field
$ cat slurm-10774.out
Starting computation
Signal SIGUSR1 (10) received due to job timeout, saving metadata and exiting properly
Saving metadata in Slurm job 10774 comment field

In the first case where the job ended normally, the save_metadata() function was executed at the end of the job. In the second case where the job had was interrupted because of its time limit, the script received SIGUSR1 signal, executed handle_timeout() signal handler which caused the script to stop and eventually triggered execution of save_metadata() function.

The generated metadata can then be extracted from Slurm accounting database with this command:

$ sacct --json | jq '.jobs | map({id: .job_id, user: .user, cores: .required.CPUs, meta: .comment.job|fromjson })'
    "id": 10773,
    "user": "remi",
    "cores": 1,
    "meta": {
      "mesh": 376,
      "complexity": 0.2924316126744422,
      "tag": "among"
    "id": 10774,
    "user": "remi",
    "cores": 1,
    "meta": {
      "mesh": 157,
      "complexity": 0.7724739043178511,
      "tag": "three"

This feature can be useful to associate various metadata to Slurm jobs, typically to generate additional and custom metrics in clusters accounting reports.