Skip to content

vLLM

The OnDemand dashboard can launch a vLLM server to serve your models. To use vLLM, click the icon on the dashboard. This opens a form in which you can submit parameters to launch a vLLM server on a GPU node.

After submitting the form, the vLLM server launch will be queued as a Slurm job and you will be redirected to the session card.

Note

Launching a vLLM server with a larger model will take 10-30 or more minutes; the session will wait for up to an hour for the server to become available.

Accessing the server

Once the server is launched, the session card will show:

  • The direct URL to access the vLLM server (http://host:port/v1)
  • The server's API key
  • Some ready-to-paste code examples for accessing the server

The vLLM server can be accessed via the direct URL from outside the cluster with a UT VPN. It is also required for programmatic access, such as with Python or curl. The VPN is not required for access from within the Rocket cluster, for example in a script or another Slurm job.

Requests to the server are authenticated with the header Authorization: Bearer.

Access via SSH tunnel

To access your server from your own device without an UT VPN, you can use an SSH tunnel. To do this, open a terminal, paste the respective command from the session card and keep that terminal window open. Example:

ssh -N -L port:example.hpc.ut.ee:port user@rocket.hpc.ut.ee

Job submission parameters

Note

Unless you are running a large model or know your needs specifically, the default values for GPU count, cores and memory should work well.

  • Model

    Path to your model files. This can either be...

    • the path to your model files on the cluster, e.g. /home/gpfs/helios/username/llama80/,
    • or a HuggingFace repo ID, e.g Qwen/Qwen2.5-7B-Instruct.

  • Served model name

    Optional. Clients passing API requests will have to set this in the field model. By default this will be the name of the model directory or HuggingFace repository ID.

  • vLLM container

    If you are going to be using a Tesla GPU, set this to /gpfs/helios/software/cluster_software/containers/vllm/vllm-openai_v0.8.5.sif. - Containers, models and GPUs are not all compatible with each other, so watch out for errors!

  • HuggingFace cache

    Optional. Location of the cache used when downloading a model from HuggingFace.

  • HuggingFace token

    Optional. This is necessary only for gated models on HuggingFace. Passed into the container as HF_TOKEN.

  • Account

    University department for billing purposes. Select one that is related to your work with vLLM.

  • Tensor parallel size

    The model will be split across a number of GPUs. If you select 0 (auto), the number will match the GPU count.

  • Extra vLLM arguments

    Optional. Additional command line arguments which are appended verbatim to vllm serve. Add --trust-remote-code for models that require it.

  • API key

    This options decides how to create and handle the API key for the server. The key can be auto-generated, pre-generated or open-access (no API key).

  • Custom API key

    Optional. Pre-generated API key, in case you select the Use my own key option in the previous menu.

  • Number of hours

    How long the server should be available once the job starts in the queue.

Output and troubleshooting

Once you queue your server launch, it will be given a session ID (for example, d07cca9c-11fc-4cd7-bfe4-36dbbfda409d) and scripts related to its Slurm job are created in this directory (example):

/gpfs/helios/home/$USER/ondemand/data/sys/dashboard/batch_connect/sys/bc_vllm/output/d07cca9c-11fc-4cd7-bfe4-36dbbfda409d/

The directory will also contain your server's output log (for troubleshooting etc.), in output.log.

Additional resources and support