File transfer to/out of the cluster¶

Unix-like OS¶

Rsync¶

Remote sync (’rsync’) is a flexible network-enabled syncing tool. The syntax is very similar to ’cp’.

To sync the content of ’dir1’ to ’dir2’ on the same machine, -r means recursive necessary for dir syncing:

rsync -r dir1/ dir2

Important

Note the ’/’ after dir1. It means that you are synchronizing the content of the dir1, not the directory itself.

The -a option is a combination flag. It's a quick way of saying you want recursion and want to preserve almost everything, except for hard links. The only exception to the preceding equivalence is when there is --files-from, in which case -r isn't implied.

The -vor --verbose increases the amount of information the daemon logs during its startup phase.

The -n or --dry-run makes rsync perform a trial run that doesn't make any changes but produces mostly the same out‐put as a real run. It's most commonly used in combination with the --verbose, -v and/or --itemize- changes, -i options to see what an rsync command is going to do before one actually runs it.

Sync with a remote machine¶

Push operation to push a dir to a remote machine:

rsync -av source_dir test_user@rocket.hpc.ut.ee:dest_dir

Pull operation to pull a dir from a remote machine:

rsync -av test_user@rocket.hpc.ut.ee:source_dir ~/dest_dir

Useful options for rsync¶

The -z option reduces traffic by adding compression.
The -P option combines the flags --progress & --partial.

SCP¶

Secure copy (SCP) is a utility that allows you to securely copy files and directories between two locations.

You're able to copy files or directory:

From/to your local machine to/out of a remote machine
Between two remote systems

SCP encrypts both the files and passwords, so they're safe from any traffic snooping. The SCP depends on ssh for data transfer, therefore it requires an ssh key or password for the authentication. Below are basic examples.

Warning

Be extra careful copying files which share the same name and location on both machines, SCP overwrites files without any warnings.

Info

Transferring large files, it's recommended to execute the SCP inside a ’screen’ session

The SCP syntax is:

scp [OPTION] user@SRC_HOST:path/to/file user@DEST_HOST:path/to/file

[OPTION] - scp option :
- -c - ’cipher’,
- -F - ’ssh_config’
- -i - ’identity_file’
- -J - ’destination’
- -l - ’limit’
- -o - ’ssh_option’
- -P - ’port’
- -S - ’program’
user@SRC_HOST:path/to/source/file - source file
user@DEST_HOST:path/to/destination/file - destination file

Copy a local file to a remote machine¶

scp path/to/local/file test_user@rocket.hpc.ut.ee:path/to/remote/dir

To copy a dir from local to remote machine, use the -r option:

scp -r path/to/local/dir test_user@rocket.hpc.ut.ee:path/to/remote/dir

Copy a remote file to your local machine¶

scp test_user@rocket.hpc.ut.ee:path/to/local/file local/dir

Copy a file between two remote machines¶

Unlike ’rsync’, with ’scp’ you don't have to log into one of the servers to transfer file from one to another machine

scp test_user1@host1.com:path/to/file test_user2@host2.com:path/to/file

Tip

To route the traffic though your local machine, use -3 option.

SFTP¶

Secure or SSH File Transfer Protocol (SFTP) is method of transferring file between two remote system over a secure connection.

SFTP is inside of various graphical tools (see FileZilla for Windows below).

Important

For a temporary access to certain files, you can request an SFTP account - see sftp.hpc.ut.ee documentation

In order to start file transfer, you need to establish an SFTP session with the cluster login node or sftp.hpc.ut.ee:

sftp myusername@login.hpc.ut.ee

In case you are connecting to sftp.hpc.ut.ee, the username will be provided to you by HPC and is not the same as your central University of Tartu username.

Once the session is active you can move on to downloading or uploading files or directories. The SFTP session allows you to change directories on both local machine and remote one.

To see the syntax and how to use SFTP options, run help in the session:

sftp> help

Navigating inside SFTP session¶

To navigate on your local machine, l stands for local in front of a command. To get the current directory on the local machine:

sftp> lpwd

To list the current directory content on the local change:

sftp> lls

To change the current directory on the local machine:

sftp> lcd

To navigate on the remote machine, these are very familiar Unix commands. To get your current directory on the remote machine:

sftp> pwd

To list the content on the remote machine:

sftp> ls

To change the directory on the remote machine:

sftp> cd

Transferring remote files to your local machine¶

To download a remote file:

sftp> get testfile

To download a remote directory:

sftp> get -Pr testdirectory

Transferring local files to your remote machine¶

To download a remote file:

sftp> put testfile

To download a remote directory:

sftp> put -Pr testdirectory

Windows¶

FileZilla¶

How to connect using the FileZilla client¶

Having downloaded and installed the FileZilla client, enter the following and click ’Quickstart’ button.

Host: rocket.hpc.ut.ee
Username: ’<your username>’
Password: ’<your password>’
Port: 22

Note

If you have a passwordless connection configured with the server, you don't have to insert the password.

Insert the credentials and press ’Quickstart’ button

Accept the ’Unknown host key’

Files on your local machine are on the left hand side and remote files are on the right hand side. By clicking ’right’ mouse button on the file of interest, you can upload it to the other machine.