Submitting jobs¶
This page is still under construction. While the information here is mostly correct, there might be gaps and badly explained parts. We are sorry, and are working on this.
Preface¶
This section covers submitting jobs to UTHPC Center's Rocket cluster using Simple Linux Utility for Resource Management (Slurm). Following sections cover monitoring jobs, monitoring resource consumption, cancelling jobs, investigating job failures and more.
A computer cluster is a set of two or more computers (nodes) that run in parallel to solve computational tasks. Computer cluster acts as a single system. A computer cluster offers its resources as one cohesive unit rather that addressing each node's resources separately - at least in most cases. Cluster's resources include CPU cores, GPUs, RAM, available disk space and interconnect connecting it all. Cluster requires a resource manager and job scheduler for optimal resource usage and distribution. UTHPC uses Simple Linux Utility for Resource Management (Slurm) to govern resource allocation and job scheduling. To harness the available computational power the user submits a job to a cluster and from there Slurm checks if the required resources are currently available so that the jobs could start running immediately or if the job should go into a queue (chosen by the user based on the needs of the user) for later execution when the resources become available. Slurm also allows the user to monitor their active jobs status. Additionally, users can look up statistics about their finished jobs - how much resources the job required.
The next section covers submitting jobs in a how-to fashion. Supported by simple example scripts that UTHPC encourages you to copy to your working directory in Rocket cluster to get you started.
Submitting a simple job¶
Copy the following python script into your home directory with a name print_time.py:
import time
print ("Time before : %s" % time.ctime())
time.sleep(10)
print ("Time after : %s" % time.ctime())
simple.job #!/bin/bash
#SBATCH --job-name="Simple python example with SLURM"
#SBATCH --time=00:01:00
#SBATCH --mem-per-cpu=128MB
module load python
srun python ./print_time.py
#!/bin/bash
#SBATCH --job-name="Simple python example with SLURM"
#SBATCH --allocation="ealloc_905b0_something"
#SBATCH --time=00:01:00
#SBATCH --mem-per-cpu=128MB
module load python
srun python ./print_time.py
Note
ETAIS users can only submit jobs when they use the proper allocation. The allocation specifies which ETAIS organization and project is billed for the job.
You can get the information about which allocation name to use from the minu.etais.ee , by going to the appropriate organization's resource, where it says how to submit a job.
Submit the job to Slurm¶
Template command:
$ sbatch <job_script_name>
$ sbatch simple.job
sbatch command Slurm returns one line output: Submitted batch job <job_ID>
Submitted batch job 19110128
Check job status¶
Use squeue command with -j flag to see job's status. The <job_ID> is numerical value:
$ squeue -j <job_ID>
$ squeue -j 19110621
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
19110621 main Simple p andersm9 R 0:09 1 stage4
After the job's finished, the job output appears in the user's home directory in the form of a file. The output file is distinguishable based on it's name. The output file has a prefix slurm- followed by the ’<job_ID>’ and suffixed with .out.
Template output name:
slurm-<job_ID>.out
slurm-19110621.out
Note
If there is no output from the squeue command then it means the submitted job has already finished. And the output is in user's home directory.
Requesting a partition¶
The default partition is ’main’. If you don't explicitly request a partition, your job runs in the default partition. To request a different partition, you must use the Slurm option --partition in your job script:
#SBATCH --partition=testing
To choose an appropriate partition that fits your needs, look info about Cluster Partitions
Parallel jobs¶
A parallel job can either run on multiple CPU cores on a single compute node, or on multiple CPU cores distributed over multiple compute nodes. With Slurm you can request tasks, and CPUs per tasks (meaning CPU cores for a task). Different tasks of a job allocation may run on different compute nodes, however, all threads belonging to a certain process are always executed on the same node.
Shared Memory Jobs (SMP) are parallel jobs that run on a single compute node. You can request a single task and a certain number of CPU cores for that task:
#SBATCH --cpus-per-task=2
MPI jobs are parallel jobs that may run over multiple compute nodes. You may request a certain number of tasks and certain number of nodes:
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=16
Warning
The requested node, task and CPU resources must match. For instance, you request one node --nodes=1 and more tasks --ntasks-per-node than CPU cores are available on a single node in this particular partition. In this case you get an error message: sbatch: error: Batch job submission failed: Requested node configuration is not available.
Open MPI¶
Open MPI has Slurm support, which means that you don't need to specify the number of processes and the execution hosts using -np and the -hostfile options. Slurm automatically provides this information to mpirun based on the allocated tasks:
#!/bin/bash
#SBATCH --mail-user=test.user@ut.ee
(...)
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=16
module load openmpi/4.1.0
mpirun <options> <binary>