Lab 5 - HPC Center services¶

Overview¶

• Jupyter
  1. What is Jupyter and how to access it?
  2. Understanding the Jupyter architecture
  3. User profiles and their management.
  4. Creating custom Python environments tailored to your needs.
• RStudio
  1. Introduction to RStudio and accessing it.
  2. Common errors and their troubleshooting.
  3. Executing R code on the cluster using Slurm jobs.

Jupyter¶

What is Jupyter? How to gain access?¶

Jupyter is a web-based interactive computational environment for creating notebook documents. It is a powerful tool that allows you to visualize your code, run it in blocks and see the results all in the same place. Jupyterlab and Jupyter notebooks allow you to run blocks in R alongside Python and support separating environments via kernels.

If you're a UT user, you get access to Jupyter via ondemand.hpc.ut.ee when you have access to the HPC cluster. Jupyter Notebook only offers a very simple interface. Users can open notebooks, terminals, and text files. Although Jupyter Notebook doesn't have all the functionalities of JupyterLab, it is enough for the purposes of this lab and for the introduction into Jupyter.

Jupyter architecture¶

When starting the Jupyter instance, you are presented with a web page, where you select what you want to run. You have the option of running terminals in either bash or python, viewing and editing files and running a kernel. A kernel is an execution environment, that defines the context that your notebook runs in. The kernels usually refer to a virtual environment, one of which we are going to create in this lab.

As for Jupyter Notebook, it is run within a batch job on the compute node. The user will then be able to launch a Jupyter Notebook Server from the Open Ondemand interface.

Creating Custom Python Environments¶

If you have your own Conda or Python environment that you'd like to use within JupyterLab, you're in for an interesting adventure of creating a custom IPython kernel, particularly when it comes to customizing it to fit your needs.

RStudio¶

What is RStudio and How to Gain Access?¶

Access RStudio at ondemand.hpc.ut.ee. Start a RStudio app. RStudio serves as a compact development environment, offering a user-friendly interface for coding and light testing. It closely resembles the RStudio environment you may typically use on your laptop.

Common Errors and Troubleshooting¶

A problem that often affects RStudio users is the user state directory located at ~/.rstudio becoming filled. In our experience, if this directory approaches 5 GB in size, RStudio may become sluggish or even freeze during login. If you plan to use RStudio for your studies or research, please keep in mind that the only solution in both cases is to contact us by writing an email to support@hpc.ut.ee.

Running R Code on the HPC Cluster using Slurm Jobs¶

To run R code outside development environment, we encourage you to use the computational power of the HPC cluster, Slurm jobs can be employed. This is particularly useful when dealing with computationally intensive R scripts or analyses that require significant resources.

Here's a step-by-step guide to running your R code using Slurm jobs:

1. Create a Slurm Script: Start by creating a Slurm script (for example, my_r_job.slurm) that outlines the resources your job needs, such as the number of CPUs, memory, and estimated runtime. Specify the R script you want to run within the script as well.

2. Submit the Job: Use the sbatch command to submit your Slurm script to the cluster. For instance: sbatch my_r_job.slurm

3. Monitor Job Progress: After submission, you can monitor the status of your job using squeue -u your_username. This will show you the queue status, job ID, and other relevant information.

4. Retrieve Results: Once the job completes, the results (output/error logs in slurm-*******.out) will be generated. You can then access these files to analyze the outcomes of your R script.

A sample Slurm script (my_r_job.slurm) to run an R script named my_analysis.R might look like this:

#!/bin/bash
#SBATCH --job-name=my_r_job
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
#SBATCH --time=2:00:00

module load r/4.1.3
module load r-bench/1.1.3
module load r-pillar/1.9.0
Rscript my_analysis.R

Using Slurm jobs not only optimizes the utilization of cluster resources but also allows you to run multiple tasks in parallel, saving you valuable time during your data analysis or simulations.

Please tailor this script according to your specific requirements. This approach not only optimizes resource utilization by the job but also allows you to run multiple tasks in parallel, saving you valuable time during data analysis or simulations.

R tasks¶

Let's compare the computational performance of basic mathematical functions between RStudio and running a job on the cluster. To facilitate this comparison, we'll utilize the R library 'bench' for benchmarking. This package employs a high-precision timer, allowing us to compare the execution time of operations that are nearly instantaneous.

Here is the R code for running example calculations and benchmarking their runtime.

library("bench") # loads bench library

x <- runif(100) # creates input vector
lb <- bench::mark(
    sqrt(x),
    x ^ 0.5,
    x ^ (1/2),
    exp(log(x) / 2)
)
lb[c("expression","total_time")] # prints only two essential columns for our test

References¶

• JupyterLab Documentation
• “Advanced R”, a book in Chapman & Hall’s R Series