Slurm¶
Use Slurm to schedule jobs on the neuro cluster. The scheduler assigns jobs to run (in parallel) where there are resources available.
Please request only the resources that your jobs need. Use the ‘shist’ command (described below) to print the resources your jobs used, so you can more accurately define resources your future jobs need.
Slurm jobs are prioritized by fair-share and queue wait time. To troubleshoot why a job is waiting in the queue, I use many commands described below, including shost, sjobs, squota, and scontrol.
For new Slurm users, please use the tutorial (below), and email me with questions: support-neuro@berkeley.edu
Commands¶
Below are useful commands to administer your jobs.
Run ‘shelp’ from the command-line to view them in your terminal window.
shost¶
show host utilization
For example:
$ shost Host CPU Reserved Free/Total RAM RAM reserved nc1 10/24 42% 215/248 GB 13.2% nc2 0/24 0% 248/248 GB 0.0% nc3 20/20 100% 21/62 GB 66.1% nc4 0/20 0% 126/126 GB 0.0% nc5 12/20 23% 62/62 GB 38.0% nc6 0/20 20% 512/512 GB 43.0% nc7 15/20 56% 126/126 GB 44.0% nc8 0/20 0% 126/126 GB 0.0% nc9 0/20 0% 248/248 GB 0.0% nc10 16/20 77% 248/248 GB 81.0% nc11 18/20 82% 248/248 GB 42.0% nc12 0/20 0% 248/248 GB 0.0% nc13 12/20 99% 150/150 GB 24.0%
sjobs¶
show all running/queued jobs
For example:
$ sjobs JOBID USER STATE CPUS NODELIST(REASON) SUBMIT_TIME TIME NAME 8378_173 jgg RUNNING 1 nc1 2026-07-28T10:14:51 1:14:00 norm_head 8377_92 jgg RUNNING 1 nc1 2026-07-28T10:11:46 1:17:05 norm_head 8362_66 jgg RUNNING 2 nc3 2026-07-28T09:25:42 2:03:08 norm_head 8362_67 jgg RUNNING 2 nc3 2026-07-28T09:25:42 2:03:08 norm_head
shist¶
show your job history (including CPU, RAM usage)
$ shist --help Usage: shist [ -a ] [ -u USER ] [ -d N ] -a Include all job states (failed, cancelled, running, etc.). Default is only completed. -u username. Default is your username -d Days of output. Default is 60.For example:
$ shist -a End JobID Node JobName State CPU(res/avg) Mem(res/peak) 2026-07-23T23:20:47 7336 nc11 rocketship_D CANCELLED 8/7.70 32G/10.97G 2026-07-23T22:53:45 7335 nc11 rocketship_D FAILED 8/1.16 32G/2.95G 2026-07-23T22:47:51 7333 nc11 rocketship_D FAILED 8/1.18 32G/2.94G 2026-07-23T22:40:00 7332 nc11 rocketship_D FAILED 8/1.19 32G/2.96G
squota¶
show your slurm quotas (CPU/RAM)
For example:
$ squota User: jgg CPU (running/quota): 30/120 RAM (reserved/quota): 72G/1500G
scontrol¶
show configuration information about a pending/running job. This is useful to troubleshoot why a job is pending.
$ scontrol show job <jobID>For example:
$ scontrol show job 8362_59 JobId=8363 ArrayJobId=8362 ArrayTaskId=59 JobName=head_norm JobState=RUNNING Reason=None Dependency=(null) RunTime=02:10:05 TimeLimit=1-00:00:00 TimeMin=N/A SubmitTime=2026-07-28T09:25:42 EligibleTime=2026-07-28T09:25:42 StartTime=2026-07-28T09:25:42 EndTime=2026-07-29T09:25:42 Deadline=N/A NodeList=nc3 BatchHost=nc3 NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=2 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=2,mem=4G,node=1,billing=2 AllocTRES=cpu=2,mem=4G,node=1,billing=2 MinCPUsNode=2 MinMemoryNode=4G MinTmpDiskNode=0 Command=/home/jgg/scripts/norm_head.sh WorkDir=/home/jgg/scripts/ StdErr=/home/jgg/slurm/logs/fs_8363_59.err StdOut=/home/jgg/slurm/logs/fs_8363_59.out TresPerTask=cpu=2 ... (some output not included)
sbatch¶
To submit a batch script
$ sbatch <scriptname>For example:
$ sbatch script.sh
scancel¶
terminate a job
$ scancel <jobID>
tutorial¶
Download scripts: tutorial.tar
$ tar xf tutorial.tar
$ chmod 755 *.sh
Example 1: Sequential batch job¶
Below is an example of batch script that runs 3 programs sequentially: test1.sh, test2.sh, and test3.sh.
sequential.sh¶
1 #!/bin/bash
2 #SBATCH --job-name=sequential_test
3 #SBATCH --ntasks=1 # 1 task per subject
4 #SBATCH --mem-per-cpu=4G # Allocates 4G RAM to each individual script
5
6 srun ~/slurm/test1.sh
7 srun ~/slurm/test2.sh
8 srun ~/slurm/test3.sh
$ sbatch sequential.sh
Submitted batch job 90
$ shist -j 90
JobID JobName Elapsed NCPUS State MaxRSS ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
90 sequentia+ 00:00:21 1 COMPLETED 0:0
90.batch batch 00:00:21 1 COMPLETED 5248K 0:0
90.extern extern 00:00:21 1 COMPLETED 0:0
90.0 test1.sh 00:00:10 1 COMPLETED 856K 0:0
90.1 test2.sh 00:00:05 1 COMPLETED 856K 0:0
90.2 test3.sh 00:00:05 1 COMPLETED 852K 0:0
The script finished in 21 seconds, and each test script (step) is identified by ‘.0’, ‘.1’ and ‘.2’:
90.0 (test1.sh) ran 10 seconds
90.1 (test2.sh) ran 5 seconds
90.2 (test3.sh) ran 5 seconds
Example 2: Parallel batch job¶
Below is an example to run the same 3 programs in parallel:
parallel.sh¶
1 #!/bin/bash
2 #SBATCH --job-name=parallel_test
3 #SBATCH --ntasks=3 # reserve 3 slots for 3 parallel scripts
4 #SBATCH --cpus-per-task=1 # 1 CPU per job
5 #SBATCH --mem-per-cpu=4G # Allocates 4G RAM to each individual script
6
7 # Launching each job step into the background using '&'
8 srun --ntasks=1 ~/slurm/test1.sh &
9 srun --ntasks=1 ~/slurm/test2.sh &
10 srun --ntasks=1 ~/slurm/test3.sh &
11
12 # Pauses the main script so it doesn't close before the tests finish
13 wait
$ sbatch parallel-test.sh
Submitted batch job 97
$ shist -j 97
JobID JobName Elapsed NCPUS State MaxRSS ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
97 parallel_+ 00:00:11 3 COMPLETED 0:0
97.batch batch 00:00:11 3 COMPLETED 14192K 0:0
97.extern extern 00:00:11 3 COMPLETED 0:0
97.0 test3.sh 00:00:05 1 COMPLETED 896K 0:0
97.1 test1.sh 00:00:10 1 COMPLETED 900K 0:0
97.2 test2.sh 00:00:05 1 COMPLETED 904K 0:0
The parallel script completes in 11 seconds, with each test script taking the same amount of time (10 seconds, 5 seconds, 5 seconds).
Example 3: Array batch job¶
Below is an example using an array definition to run the test scripts parallel:
parallel.sh¶
1 #!/bin/bash
2 #SBATCH --job-name=array_test
3 #SBATCH --ntasks=1 # 1 task per subject
4 #SBATCH --cpus-per-task=1 # 1 physical core per subject
5 #SBATCH --mem=4G # 4G RAM per subject
6 #SBATCH --array=1-3 # Creates 3 parallel sub-jobs (1, 2, and 3)
7
8 # Slurm automatically updates $SLURM_ARRAY_TASK_ID for each sub-job
9 ~/slurm/test${SLURM_ARRAY_TASK_ID}.sh
$ sbatch array.sh
Submitted batch job 116
$ shist -j 116
JobID JobName Elapsed NCPUS State MaxRSS ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
116_1 array_test 00:00:11 1 COMPLETED 0:0
116_1.batch batch 00:00:11 1 COMPLETED 1400K 0:0
116_1.extern extern 00:00:11 1 COMPLETED 0:0
116_2 array_test 00:00:05 1 COMPLETED 0:0
116_2.batch batch 00:00:05 1 COMPLETED 1532K 0:0
116_2.extern extern 00:00:05 1 COMPLETED 0:0
116_3 array_test 00:00:05 1 COMPLETED 0:0
116_3.batch batch 00:00:05 1 COMPLETED 1440K 0:0
116_3.extern extern 00:00:05 1 COMPLETED 0:0
Documentation¶
For more documentation, see the official Slurm Workload Manager Documentation.