Use of OpenCAYLE in HPC Environments

✅s3cmd

1. Connection

• Generate the access key if you have not already generated it. You can consult the steps in the OpenCayle user manual.
• Having completed the Python user manual, you can check the connection by using the script s3CheckConnection.py.

• If the connection is correct, you will get the following message:

  [user_1_1@frontend1 ~]$ python s3Scripts/s3CheckConnection.py <access_key>
      Your accessKey connects successfully to OpenCayle
  >>>>>
  Where user_1_1 would be your connection username.

• If, on the contrary, you get an error, please contact the support team to review your case.

2. Bucket creation

There are different options for generating buckets (if the bucket already exists it is not necessary to create it):

• Generating the bucket on your personal machine:

  • See s3Browser usage.
  • See s3cmd usage.

• Generating a bucket from the frontends

  • Create the "s3test.cfg" file.

  • Run "ls", a command that allows listing the contents of a directory, to verify that nothing has been created in OpenCayle yet.
    [user_1_1@frontend1 ~]$ s3cmd -c s3test.cfg ls

  • Create the bucket with the following command:
    [user_1_1@frontend1 ~]$ s3cmd -c s3test.cfg mb s3://

  Example:
  [user_1_1@frontend1 ~]$ s3cmd -c s3test.cfg mb s3://your_first_bucket
  Bucket 's3://your_first_bucket/' created

  • When running "ls" again, your bucket should already appear.
    [user_1_1@frontend1 ~]$ s3cmd -c s3test.cfg ls
    2020-06-16 09:32 s3://your_first_bucket

✅rclone

You can find more useful rclone commands at the following link: https://rclone.org/commands/

1. Connection

A configuration file must exist in the path .config/rclone/rclone.conf with the following information:

[user_1_1@frontend1 ~]$ cat .config/rclone/rclone.conf
[oc]
type = s3
provider = Other
env_auth = false
access_key_id =
secret_access_key =
endpoint = https://ss3.scayle.es

2. List OpenCayle

• List directories
  [user_1_1@frontend1 ~]$ rclone lsd oc:
  -1 2020-06-16 11:32:59 -1 my_first_bucket

• List the contents inside a directory
  [user_1_1@frontend1 ~]$ rclone ls oc:my_first_bucket
  or
  [user_1_1@frontend1 ~]$ rclone lsl oc:my_first_bucket

• List tree
  [user_1_1@frontend1 ~]$ rclone tree oc:m

3. Create/Modify/Delete

• Create a file
  [user_1_1@frontend1 ~]$ rclone touch oc:my_first_bucket/file_created_with_rclone.txt
  [user_1_1@frontend1 ~]$ rclone lsl oc:my_first_bucket
  0 2021-08-30 13:17:50.212381601 file_created_with_rclone.txt

• Delete a file
  [user_1_1@frontend1 ~]$ rclone delete oc:my_first_bucket/file_created_with_rclone.txt
  [user_1_1@frontend1 ~]$ rclone lsl oc:my_first_bucket

• Create a directory
  [user_1_1@frontend1 ~]$ rclone mkdir oc:my_second_bucket
  [user_1_1@frontend1 ~]$ rclone lsd oc:
  -1 2020-06-16 11:32:59 -1 my_first_bucket
  -1 2021-08-30 13:21:10 -1 my_second_bucket

• Delete a directory
  [user_1_1@frontend1 ~]$ rclone rmdir oc:my_second_bucket
  [user_1_1@frontend1 ~]$ rclone lsd oc:
  -1 2020-06-16 11:32:59 -1 my_first_bucket

4. Move/Copy

• Copy a local file to OC
  [user_1_1@frontend1 ~]$ cat rclone.txt
  Hello there
  [user_1_1@frontend1 ~]$ rclone copy rclone.txt oc:my_first_bucket
  [user_1_1@frontend1 ~]$ rclone lsl oc:my_first_bucket
  14 2021-08-30 13:14:46.000000000 rclone.txt

• Copy a file from OC to local
  [user_1_1@frontend1 ~]$ cat rclone.txt
  Hello there
  [user_1_1@frontend1 ~]$ rclone copy oc:my_first_bucket/file_created_with_rclone.txt .
  [user_1_1@frontend1 ~]$ ls -l
  ...
  -rw-r--r-- 1 user_1_1 hpc_scayle 0 Aug 30 13:17 file_created_with_rclone.txt
  ...

• Move a file
  [user_1_1@frontend1 ~]$ rclone lsl oc:my_first_bucket
  0 2020-06-16 11:36:02.000000000 my_first_object
  14 2021-08-30 13:14:46.000000000 rclone.txt
  [user_1_1@frontend1 ~]$ rclone move file_to_move_to_oc.csv oc:my_first_bucket
  [user_1_1@frontend1 ~]$ rclone lsl oc:my_first_bucket
  0 2021-09-02 10:08:19.000000000 file_to_move_to_oc.csv
  0 2020-06-16 11:36:02.000000000 my_first_object
  14 2021-08-30 13:14:46.000000000 rclone.txt

File synchronization

Sync