Skip to content

Upload from the Rocket cluster

Login into UTHPC Rocket cluster

If you need to apply for a Rocket cluster account or require any other kind of assistance, please take a look at UTHPC documentation .

File encryption

Before upload the file must be encrypted using the crypt4gh tool .

Activate software for data encryption on the Rocket cluster.

On the terminal run the command:

module load crypt4gh/1.8.5

Copy UT HPC crypt4gh public key.

The primary function of the key is to encrypt files before transmission to the archive. This encryption process is vital to maintain confidentiality and integrity of the information being transmitted. It's important to note that the key is unique to the particular FEGA instance, not the entire federation.

  • In the folder, where your data to be uploaded is located create a file uthpc-c4gh.pub.pem with a text editor (Vim, Nano) and copy the public key into the file: 
    -----BEGIN CRYPT4GH PUBLIC KEY-----
0F59LNcZ5wNSoKyQg0UXGok8i+Pt6r7qCfMhd5WSNmk=
-----END CRYPT4GH PUBLIC KEY-----
  • Save and close the file.

Encrypt a file with the public key.

  • While in the folder, where your data is located run command: 
    crypt4gh encrypt --file=<my_data_file> --pubkey=uthpc-c4gh.pub.pem
    Where:
    • <my_data_file> is the name of your file that you want to encrypt,
    • uthpc-c4gh.pub.pem is the crypt4gh public key file name (from the previous step).
  • You will be asked whether you want to continue and for the default choice Yes just press Enter.
  • You should see a Success! message.
  • You should have a new file with the ending .c4gh - this is your encrypted file.

File upload

File upload is done with the s3cmd tool .

Activate software for data upload on the Rocket cluster.

On the terminal run the command:

module load py-s3cmd/2.3.0

Get Server configuration

  • In order to submit data, you have to have an account at EGA test web page .
    • From right top select Register, fill out the form and click Create My Account.
    • You will see a message saying that your account is under review.
    • Contact ´Vlad´, ´Ulvi´ or ´Tommy´, who can add all necessary roles to your account (submitter role).
  • Login into auth.fega.etais.ee with your EGA account .
  • At the bottom of the page click Download inbox s3cmd credentials.
  • You will be given the file s3cmd.conf for download. Save it to the same folder, where your encrypted file is located.

Info

Your upload credentials will be valid for a restricted period only and therefore please take a note of the expiry date displayed upon creation. Once credentials have expired they can be renewed by repeating the procedure described above.

Upload encrypted file

Info

Large files are automatically split into 50 MB chunks during upload and each chunk is uploaded separately.

  • Open terminal and navigate to the folder, where s3cmd.conf and your encrypted file are located.
  • Run command: 
    s3cmd -c s3cmd.conf put <my_data_file> s3://<access_key>/<my_data_file_server>
    Where:
    • <my_data_file> is the name of your encrypted file (must end with .c4gh),
    • <access_key> is your access key, that is located inside ´s3cmd.conf´ file:
      • open s3cmd.conf file,
      • find the line, that starts with access_key =,
      • copy all the letters and numbers after the equal sign.
    • <my_data_file_server> is the name that file will have on the server (must end with .c4gh) and can be the same as <my_data_file>.

Info

You can ignore WARNING and ERROR messages at the beginning of output. The last line shows the upload information.

Next steps

Next, please enter the uploaded files metadata in FEGA (FEGA user submitter portal manual).