Rclone

Rclone is a CLI program for accessing S3 buckets.

Installation

On Ubuntu

apt install rclone

Configuration

Copy your keys from ICE Connect

Services ➡ S3 Storage ➡ Keys

Add the configuration file

~/.config/rclone/rclone.conf

[s3]
type = s3
provider = Ceph
env_auth = false
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_SECRET_KEY
endpoint = s3.ice.ri.se
acl = private
bucket_acl = private
upload_cutoff = 1G

List objects

rclone ls s3:project-bucket/

120900442 go1.15.5.linux-amd64.tar.gz

Download

rclone copy s3:project-bucket/go1.15.5.linux-amd64.tar.gz ./
ls -al

total 118096
drwxrwxr-x  2 daniel daniel      4096 Oct 20 11:47 .
drwxrwxr-x 12 daniel daniel      4096 Oct 20 11:12 ..
-rw-rw-r--  1 daniel daniel 120900442 Oct  1 11:27 go1.15.5.linux-amd64.tar.gz

Upload

rclone copy testfile.txt s3:project-bucket/
rclone ls s3:project-bucket/

120900442 go1.15.5.linux-amd64.tar.gz
        5 testfile.txt

Sync bucket to folder

rclone sync s3:project-bucket/ localfolder/

Copy from SFTP to S3

You can copy files directly from an SFTP server to an S3 bucket.

If your SFTP server is accessed using a password, first generate an obscured hash:

rclone obscure mypassword

vfjXA7DihqfJf0BRz4YCcXRw54zD5A38Tvo

Add the SFTP configuration to the rclone configuration file.

~/.config/rclone/rclone.conf

[s3]
type = s3
...
no_check_bucket = true

[sftp]
type = sftp
host = 194.28.122.117
user = sftpuser
pass = vfjXA7DihqfJf0BRz4YCcXRw54zD5A38Tvo
shell_type = unix
md5sum_command = none
sha1sum_command = none

Copy a file from the SFTP server to the S3 bucket.

rclone copy sftp:/files/test.txt s3:project-bucket/