Lab 4 - Running jobs: how to request resources for programs¶

Introduction¶

So far you have logged in, submitted a "Hello World" job and managed software with modules. In this lab we look at the thing an HPC cluster is actually built for: running the same computation in parallel on many cores / many nodes / GPUs, and seeing how that changes the runtime.

We will be looking at three common methods of speeding up computation:

To make the differences concrete we will use one running example for the whole lab: matrix multiplication C = A*B, for N*N matrices. It is the classic HPC benchmark because it is simple, purely compute-bound, scales as O(N^3), and is "embarrassingly parallel" — every element of C can be computed independently. This makes scaling performance easy to observe.

For each paradigm we will: read a tiny example program, run it with different parameters (thread counts, process counts, matrix sizes), and record the timings the program prints.

The example problem: matrix multiplication¶

Get the code¶

We have prepared four small programs — a serial baseline plus the OpenMP, MPI and CUDA versions — and a Makefile that builds them. They live in the course scripts folder.

cd /gpfs/space/projects/hpc-course/<your_hpc_username>/lab4
cp -r /gpfs/space/projects/hpc-course/scripts/lab4/matmul-variants .
ls matmul-variants

Read the programs¶

Each program does the same thing: fill two N*N matrices A and B deterministically (fixed seed, so every run produces the same numbers), compute C = A * B, and print one line like:

[OpenMP ]  N=1024   threads=8    time=  123.456 ms  GFLOPS=  17.30  verify=PASS

The line always contains the matrix size N, the parallelism it used (threads / processes), the wall-clock time, a throughput figure in GFLOPS, and a verify= check that compares the result against a serial reference.

Have a quick look at the sources. The serial one is the whole algorithm in three nested loops; the OpenMP one is the same code plus a single #pragma; the MPI one scatters rows of A to ranks and gathers C back; the CUDA one launches one GPU thread per element of C. Understanding the code here is not necessary or expected, but if you are interested in parallel programming, this can be a starting point for further exploration.

Build the CPU programs¶

Threading (OpenMP) and the serial baseline only need a C++ compiler. The MPI program needs the MPI compiler wrapper mpicxx, which comes from the openmpi module. Build all three on the login node (compiling is quick — only running heavy work belongs in a job).

cd /gpfs/space/projects/hpc-course/<your_hpc_username>/lab4/matmul-variants
module load openmpi/5.0.9
make serial openmp mpi
ls -l matmul_serial matmul_openmp matmul_mpi

We will build the CUDA program later, in the GPU section.

Run the serial baseline¶

Run the single-threaded version once to establish the number everything else is measured against. A 256x256 matrix multiplication is a few seconds on one core, so this is fine on the login node for a single quick run — but for the sweeps we will use proper jobs.

cd /gpfs/space/projects/hpc-course/<your_hpc_username>/lab4
./matmul-variants/matmul_serial 256

Requesting parallel resources¶

The single most important idea in this lab is the difference between these two Slurm flags, because they map exactly onto the two CPU methods:

• --cpus-per-task=N — give one task N cores that share memory. This is what an OpenMP program wants: one process, many threads, one node.
• --ntasks=P (-n P) — launch P independent processes (MPI ranks), each with its own memory. Add --nodes and --ntasks-per-node to spread them across nodes. This is what MPI wants.

Concretely:

The other lesson, which the sweeps below will drive home, is: requesting resources is not the same as using them. If you ask for 16 cores but your program only uses 1, the other 15 sit idle and you have wasted an allocation. That is exactly why we measure.

Threading with OpenMP (shared memory)¶

Threading is the lightest way to parallelise: you keep a single program with a single shared memory space. Using OpenMP it is easy to parallelize work. With OpenMP you just tell the compiler "create threads and spread this loop over the threads." OpenMP will do the work of creating threads and allocating computation to them itself. The programmer simply has to tell the compiler where and how to do this. In our code the entire change from serial to threaded is one line:

#pragma omp parallel for
for (int i = 0; i < N; i++)      // each row of C handled by a different thread
    ...

Each iteration of the outer i loop is independent, so OpenMP hands different rows to different threads automatically. Nothing is copied, no messages are sent — the threads just read the shared A and B and write their slice of C.

The catch: this only works within one node, because the threads must share memory. You tell Slurm how many cores to give the one task, and you tell OpenMP how many of them to use. The number of threads OpenMP creates can be set within the code or using the environment variable OMP_NUM_THREADS.

Example: one run by hand¶

cd /gpfs/space/projects/hpc-course/<your_hpc_username>/lab4
srun -p main -n 1 --cpus-per-task 8 --hint=nomultithread \
    ./matmul-variants/matmul_openmp 1024

Note that in the output, it says we used 8 threads. OpenMP will usually select all the cores available to it, however this is not always the case.

  export SLURM_ACCOUNT=ealloc_high-per-...
  export SALLOC_ACCOUNT=ealloc_high-per-...
  export SBATCH_ACCOUNT=ealloc_high-per-...

Exercise: a thread-count sweep¶

We want to see what happens to the runtime as we throw more threads at the same problem. Below is a batch script that requests 16 cores and we will use subsets of these cores up to 32 to see how beneficial they are for our problem.

Save the following as openmp/openmp_sweep.sh (first mkdir openmp):

#!/bin/bash
#SBATCH --partition=main
#SBATCH --time=10:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --hint=nomultithread
#SBATCH --job-name=lab4_openmp

cd /gpfs/space/projects/hpc-course/$USER/lab4 # your lab4 directory — submit from here
RESULTS=openmp/results.txt
: > "$RESULTS"                # start from an empty file each run

for N in 256 1024; do # Matrix sizes 256x256 and 1024x1024
  for T in 1 2 4 8 16 32; do # Thread counts
    export OMP_NUM_THREADS=$T # Set thread count through environment variable
    ./matmul-variants/matmul_openmp "$N" >> "$RESULTS" # Output of scenario in result file
  done
done

Multiprocessing with MPI (distributed memory)¶

OpenMP is stuck on one node. If your problem is big enough to need two, four or a hundred nodes, you reach for MPI (Message Passing Interface). Here you run separate processes (called ranks), each with its own private memory, that cooperate by sending messages to each other.

Our MPI program decomposes the work by rows: rank 0 holds the full matrices, broadcasts (sends) B to everyone, scatters (splits and sends) the rows of A so each rank owns a contiguous block, every rank computes its rows × B, and the blocks of C are gathered back to rank 0. The serial reference is computed on rank 0 to verify.

The Slurm knob is --ntasks (the number of ranks), and at runtime you launch them with mpirun -np <n> or srun -n <n>. Ranks can live on one node or be spread across many — that is the whole point.

Exercise: scaling sweep¶

We will do a similar sweep here. However, this time we vary the count of tasks that we create. Save the following as mpi/mpi_sweep.sh (first mkdir mpi). Then you will need to make some modifications to the batch script below at the "TODO" parts. You need to load a module for OpenMPI, use module spider and load the latest version. Also, looking at the previous OpenMP runs, finish the loop to compute on matrices of dimensions 256x256 and 1024x1024 and process counts 1,2,4,8. Also, add the flag to set the process count mpirun launches (mentioned just above) to the variable we have defined in the loop.

#!/bin/bash
#SBATCH --partition=main
#SBATCH --time=10:00
#SBATCH --ntasks=8
#SBATCH --cpus-per-task=1
#SBATCH --hint=nomultithread
#SBATCH --job-name=lab4_mpi

# TODO: Find and load the latest OpenMPI

cd /gpfs/space/projects/hpc-course/$USER/lab4 # your lab4 directory — submit from here
RESULTS=mpi/results.txt
: > "$RESULTS"

for N in ...; do # TODO: Matrix sizes 256x256 and 1024x1024
  for NP in ...; do  # TODO: Process counts 1, 2, 4, 8
    mpirun ... ./matmul-variants/matmul_mpi "$N" >> "$RESULTS" # TODO: Add the right flag for launching multiple tasks using the environment variable `NP`
  done
done

We request --ntasks=8 once (the maximum) and reuse the same allocation for every step of the sweep — the unused ranks simply stay idle during the smaller-np runs. This is something you can do with both srun and mpirun. MPI is built such that it reads the parameters of the Slurm job and creates the corresponding number of processes with allocated cores.

The parameters you set with #SBATCH --param will be the resources available to whatever you do in that job. Using srun, steps are created within that job using the parameters specified to the srun, which default to the #SBATCH values, but can be changed as has been shown. Anything you write in the batch script will be executed on a compute node though. Thus, srun and mpirun are just ways to create additional processes on the node (or multiple) that you have selected.

Using Multiple Nodes¶

The sweep above ran entirely on one node. For solving bigger problems more nodes are necessary. We can spread the ranks across two nodes as is shown below and more by simply changing the --nodes value in the parameters. Save as mpi/mpi_2node.sh:

#!/bin/bash
#SBATCH --partition=main
#SBATCH --time=10:00
#SBATCH --ntasks=8
#SBATCH --ntasks-per-node=4
#SBATCH --nodes=2
#SBATCH --cpus-per-task=1
#SBATCH --hint=nomultithread
#SBATCH --job-name=lab4_mpi_2node

# TODO: Find and load the latest OpenMPI
cd /gpfs/space/projects/hpc-course/$USER/lab4 # your lab4 directory — submit from here

mpirun ...  hostname >> mpi/results_2node.txt # TODO: mpirun flag. In the output, you can see what node each rank was running on

echo "# two-node run" >> mpi/results_2node.txt
mpirun ... ./matmul-variants/matmul_mpi 1024 >> mpi/results_2node.txt # TODO: mpirun flag

Our example above has a flaw. When running real jobs, you should not branch out to multiple nodes until the resources on a single one have been exhausted. For the CPU nodes on Rocket, there are 128 physical cores available, of which we used 4 per node here. Sending messages across nodes is expensive, since the distance is larger and the connections often slower. This communication can add a significant overhead to your programs, leading to diminishing returns if the problem is not large enough. This is something you might have observed already in these two examples: the N=1024 cases scale better than the N=256 cases. The last two lectures cover the scaling laws describing the maximal benefits we can gain from parallelizing our programs and how this overhead from communication affects it.

GPU computing with CUDA¶

A GPU is a separate accelerator with its own memory and thousands of small cores. The CPU offloads the heavy kernel to it: copy the data to the GPU, launch a huge number of tiny threads, copy the result back. Our naive kernel launches one thread per element of C — for a 1024×1024 matrix multiplication that is over a million threads at once. Note that this is not an optimized matrix multiplication kernel that would be used in cuBLAS, but a simple naive implementation.

GPUs live in their own partition and are requested with --gres=gpu. Asking for a GPU is exclusive — once Slurm gives you one, nobody else can use it until your job ends.

Build the CUDA program¶

Compiling needs the CUDA compiler nvcc (from the cuda module), but does not need a GPU — you can build on the login node and run on the GPU partition.

cd /gpfs/space/projects/hpc-course/$USER/lab4/matmul-variants
module load cuda/12
make cuda
ls -l matmul_cuda

Example: an interactive GPU session¶

Interactive sessions can be created as is shown below. You specify the time limit, since the default 1 minute is quite short, then add that you want an interpreter --pty and set the executable to bash, which is a shell. Look into our documentation on GPU jobs and finish the GPU request line --gres=... by adding a Tesla GPU request. Then run the following test.

srun -p gpu --gres=gpu... -n 1 -c 8 \
     --hint=nomultithread --time=00:05:00 --pty bash # 5-minute interactive session
module load cuda/12
cd /gpfs/space/projects/hpc-course/$USER/lab4
./matmul-variants/matmul_cuda 1024
nvidia-smi # Display GPU counters.
exit

Exercise: a matrix-size sweep¶

This time the "parameter" we vary is N, because that is what decides whether the GPU can pay back its transfer overhead. We do not vary any thread counts, since by requesting a GPU, we also request all the threads and memory available to it. Thread counts and other parameters can be changed in the code though. If you have an interest in programming GPUs, you can look into how CUDA kernels are made and how basic optimizations can be done to matrix multiplication for instance. This, however, is out of the scope of this course.

Save the following as cuda/cuda_sweep.sh (first mkdir cuda):

#!/bin/bash
#SBATCH --partition=gpu
#SBATCH --time=10:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --gres=...
#SBATCH --hint=nomultithread
#SBATCH --job-name=lab4_cuda

module load cuda/12

cd /gpfs/space/projects/hpc-course/<your_hpc_username>/lab4 # your lab4 directory — submit from here
RESULTS=cuda/results.txt
: > "$RESULTS"

echo "# GPU:" >> "$RESULTS"
nvidia-smi --query-gpu=name --format=csv,noheader >> "$RESULTS"

for N in 512 1024 2048 4096; do
  echo "# N=$N" >> "$RESULTS"
  ./matmul-variants/matmul_cuda "$N" >> "$RESULTS"
done

Putting everything together¶

All three techniques used here are not exclusive. MPI can be used with OpenMP to combine threading and multiprocessing. To try this, you can simply take the MPI code and add a #pragma omp parallel for before the main multiplication loop. MPI can also be used with GPUs to use many GPUs for solving a single problem. Furthermore, OpenMP has similar directives for telling the compiler to offload computation to the GPU.

Since all of these methods require different configuration from the side of your batch jobs, you need to understand how your software works at least on a basic level. Adding a lot of cores to an MPI (or GPU) job will likely not help your performance at all. Adding tasks to an OpenMP job will just duplicate your results and if you do not configure the thread parameters correctly, even adding cores might not do anything. For most scientific software, you can find guides regarding good parameters for launching jobs in Slurm. These are a good starting point, but some benchmarking should still be done. In some cases, the problem solved can be too small to benefit from the additional resources, since the overhead from thread/process creation and synchronization or copying to/from the GPU can dominate and lead to a slower result.

Keeping an eye on your jobs¶

A big part of "running jobs" is knowing whether you actually used what you asked for. Two tools:

• CPU / RAM — the Elastic dashboard at elk.hpc.ut.ee. A quickstart guide is in our docs. If your OpenMP job asked for 16 cores but the CPU graph sits at ~6%, you are wasting an allocation — bumping OMP_NUM_THREADS further is pointless and you should find out why (memory bandwidth, serial fraction, …).
• GPU — nvidia-smi inside a running GPU job. You can take the example here and extend it to run longer by increasing the N value or having it run multiple times. Steps on how to get a broad overview are shown here.

Run one of your sweep jobs, then look it up in ELK and compare the measured utilisation to the timings you recorded.

Best practices to take away¶

• Request what you use, use what you request. Cores and GPUs are exclusive while your job runs; idle ones are wasted allocation. Measure with ELK / nvidia-smi.
• Match the Slurm flag to the paradigm: --cpus-per-task for threads, --ntasks for ranks, --gres=gpu for accelerators. Mixing them up is the most common reason a "parallel" job runs serial.
• Don't run heavy work on the login node. A single quick baseline run is fine; anything that takes minutes belongs in a job (like the sweeps above).