Introduction¶

Welcome to the sixth lab. We will cover environments and containers. You can consider this lab to be somewhat of an advanced approach to using software

In this lab we will be going over the practical side of creating environments and containers.

Conda environments¶

In your lab6 directory create another one called conda. We will be installing our environments there.

To work with conda environments, we actually need to have it. Run module load miniconda3 . Do note that when using conda environments in jobs, you need to load the module before you can activate them.

Creating an environment¶

The syntax to create an environment is quite simple: conda create -n <name> package1=version package2=version. To get more options you can run conda create --help.

There are a few things to notice here. First, we didn't use -n, but --prefix instead. This allows you to install the environment in a predefined location instead of the default. This enables more control but you have to manage the location yourself. Usually you should use -n because then you can just use source activate <name> instead of specifying the full path.

We installed two packages to the environment, python and conda. These are not necessary but they are a good thing to install to increase independency from the outside environment. If you don't specify python for example, the same python that you have loaded will be linked into the environment. If something should happen to the underlying python, then your env will also break. Adding a separate conda installation to your environment makes managing the environment simpler. When using an outside conda, it might try to install packages in weird locations. Conda will also create the activate script that we will discuss later.

You can explore the firstenv directory further. You can also run a little script find . -type f | wc -l to list how many files are in that directory.

Managing and updating an environment¶

Technically, conda always has an environment that it's using. When you don't have one loaded it's called Base. Run conda env list to see what environments are available and loaded.

When in an environment, run conda list to see what packages are available.

Run the file count script again to see how many files this action created.

Exporting an environment¶

You can export environments with exact package specifications for others to use. This is achieved by the conda env export command.

You can create environments from export files with the -f flag. Create one with: conda env create --prefix=/gpfs/space/projects/hpc-course/hpc_yourname/lab6/conda/secondenv -f=/gpfs/space/projects/hpc-course/hpc_yourname/lab6/conda/firstenv-export.yaml

Usually when a software package needs an environment, they provide one of these export files. Hopefully now you know how to better manage them with conda.

virtualenv environment¶

Start this part with creating the directory lab6/virtenv.

Run conda deactivate and module purge to clear your environment and then module load python/3.9.12.

We won't go too much in depth here as virtual environments have less functionality than conda (access to only pip packages) and the high level concepts already got introduced with it.

Creating an environment¶

Like export files, pip has something called requirements.txt . We will see how to use it in an environment by installing an open diffusion software.

You now have a working virtualenv environment. You can run the source .../bin/activate command again without having to load any modules. You can follow the guide in the GitHub link to generate your own AI images but those are not part of the course.

Spack environments¶

Create a directory lab6/spackenv to get started. We will be using the same Spack installation initialize your Spack that we used in lab2. Activate the setup-env.sh script if you haven't done so in your .bashrc file.

spack env --help will give you an overview of what you can do.

When you run spack find you will see that you have 0 installed packages. You can install packages to the environment by using spack install but we will be modifying the spack.yaml file instead.

In spack.yaml find the line starting with specs:, change it to:

  specs:
    - python@3.8
    - tcl

Now run spack install. Python 3.8 will be downloaded and built, but Tcl finishes almost instantly — you already installed tcl with Spack in Lab 2, and Spack reuses what is already in its central package store instead of building it again. Note that the environment directory itself does not contain the packages: Spack installs everything into its central store, and the environment only holds a view — the hidden .spack-env directory — full of links to that store. The view is what makes the environment behave like a regular installation prefix.

The Spack environments documentation has tons of more information on what you can achieve with environments.

Singularity containers¶

You now know several ways to get software onto the cluster: the module system, conda, virtual environments and Spack. Containers solve a different problem than all of those. A module, a conda env or a spack env is something you assemble out of packages — a container is a finished environment (the program, its libraries, runtime, everything) that someone else has already built and published. When your tool exists as a container, you run exactly what they ran, with zero dependency handling.

The example in this lab is Ollama, a tool for running large language models. There is no ollama module on the cluster, yet we can gather the whole environment needed to run it in two commands.

module load singularity

Where containers come from¶

Containers are published to registries, and you download ("pull") them from there:

• Docker Hub — the largest one, most software publishes here
• quay.io — hosts BioContainers, ready-made containers for nearly every bioinformatics tool
• GitHub Container Registry and NVIDIA NGC — common for GPU and AI software

Most published images are Docker images. The docker:// prefix tells Singularity to pull from a Docker registry and convert the image into a single .sif file.

Pulling an image and running things: exec¶

singularity pull ollama.sif docker://ollama/ollama:latest

The command you will use most is exec: it runs a single command inside the container. So the pattern is singularity exec <image.sif> <command> <args>. Let's see how we can use the software inside the container.

ollama --version

singularity exec ollama.sif ollama --version

Look at what the image actually carries around with it:

singularity exec ollama.sif ls /usr/lib/ollama/

You will see pre-built runners for a dozen CPU generations and two CUDA generations (cuda_v12 and cuda_v13). This is why one image can run efficiently on any node of the cluster and on your laptop — the container bundles its own compatibility matrix, and you install none of it. However, it does not add onto what you have on the system. Compare the two outputs:

ls /usr/lib/
singularity exec ollama.sif ls /usr/lib/

The two are different. This means that inside the container, you will not have access to all the tools from the outside. This is by design to avoid conflicts with the container's environment.

There is also singularity run, which executes the image's default command. For this image the default is the Ollama server, which keeps running until you stop it. For almost all cases exec is the tool you want to use to have control over what exactly is executed. singularity shell ollama.sif opens an interactive shell inside the container for looking around (leave with exit).

Your data: bind mounts¶

As we saw, the container only sees parts of the filesystem. Singularity automatically binds your home directory, /tmp and the directory you are currently standing in — everything else is invisible to the container.

With ollama, you can easily pull models to run with ollama pull <model_name>. However, for an exercise in binding data to a container, the course has staged an LLM for you at /gpfs/space/projects/hpc-course/models (the quantized qwen2.5:7b, about 4.7 GB). That path is not your home and not your current directory, so it does not exist inside a container.

singularity exec ollama.sif ls /gpfs/space/projects/hpc-course/models/

singularity exec --bind /gpfs/space/projects/hpc-course/models:/models ollama.sif ls /models

--bind <path> mounts a directory into the container at the same path; --bind <path>:/other/path mounts the data at <path> such that in the container it exists at /other/path. You can also set binds once for all following commands with export SINGULARITY_BIND=<path>.

Exercise: run an LLM in a batch job¶

Ollama works as a small server + client pair: ollama serve runs the server, and commands like ollama run talk to it over localhost. The server reads its models from the directory that the OLLAMA_MODELS variable points to. Put the pieces together in a job (everything you learned in Lab 4 applies):

Save the following as lab6/containers/ollama_job.sh:

#!/bin/bash
#SBATCH --partition=gpu
#SBATCH --time=00:15:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --gres=gpu:tesla:1
#SBATCH --job-name=lab6_ollama
#SBATCH --output=ollama_job_%j.out

module load singularity

cd /gpfs/space/projects/hpc-course/$USER/lab6/containers

# The server looks for models here (the bind-mounted course models)
export OLLAMA_MODELS=/models

# Start the server in the background. We add the --nv flag so that the container has access to the GPUs
# We direct the output from the server to "server.log" using ">"
singularity exec --nv ollama.sif ollama serve > server.log 2>&1 & # TODO: add the bind flag we used previously here
sleep 30

# Since we added a "&" symbol to the end of our last serve command, it will keep running in the background.

# List available models
singularity exec ollama.sif ollama list

# We can ask the model something (prints the answer and exits)
singularity exec ollama.sif ollama run qwen2.5:7b "Explain in two sentences what an HPC cluster is."

# Proof it ran on the GPU
nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv

# Stop the server before the job ends
pkill ollama

How containers are made¶

Images are built from a recipe — a Dockerfile for Docker images, a Singularity definition file for native images. They look roughly like you would expect:

FROM ubuntu:24.04
RUN curl -fsSL https://ollama.com/install.sh | sh

A base image, plus the commands that turn it into the environment you want. Building images needs root rights, which you do not have on the cluster — so in practice HPC users pull finished images instead of building their own. Should you ever need your own (for example, to containerize a conda environment), look at cotainr — and if you want containers to show up as modules, look at SHPC.

Extra notes¶

Containers complete the set of ways to get software onto the cluster: modules for the centrally installed stack, conda/venv for your own Python-centric environments, Spack for building anything from source, and containers for running finished environments as-is. In practice you will mix and match — but before you fight a painful install, check whether the tool already exists as a container. For LLM and AI work, Docker Hub and NVIDIA NGC very often have it.