s3cmd Basic Usage#
There are various clients that can be used to interact with the object storage at EODC. We can offer only limited support and guidance on using third party tools.
These clients make use of the S3 interface to the EODC Object storage.
It is expected that you have already generated your EC2 Credentials. For more details, see the introduction to Object Storage page.
s3cmd#
Getting Started#
s3cmd is a popular powerful python based open source CLI utility for interacting with object storage. It is available in the default repository for most Linux distributions and also available for Mac.
Install it in the appropriate way for your environment https://github.com/s3tools/s3cmd
First s3cmd must be configured for the environment Be sure to have your EC2 credentials at hand! This can be handled interactively:
[host] $ s3cmd --configure
# Follow the prompts - Your settings will look similar
New settings:
Access Key: $accessKey
Secret Key: $secretKey
Default Region: US
S3 Endpoint: objectstore.eodc.eu:2222
DNS-style bucket+hostname:port template for accessing a bucket: objectstore.eodc.eu:2222
Encryption password: $optional
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
Alternatively create your own s3cfg file with the following template
[default]
access_key = $access
host_base = objectstore.eodc.eu:2222
host_bucket = objectstore.eodc.eu:2222
use_https = true
secret_key = $secret
s3cmd -c s3cfg $command
Now that s3cmd is configured, we can explore some basic usage
Basic Usage#
To list all available buckets#
[host] $ s3cmd ls
2023-12-14 21:48 s3://Example
2023-12-15 16:20 s3://Example1
2023-12-05 12:08 s3://Example2
2023-12-20 17:11 s3://Example3
To list contents of a bucket#
[host] $ s3cmd ls s3://bucket
2023-12-20 17:44 0 s3://bucket/example.file
To make a new bucket#
[host] $ s3cmd mb s3://bucket
Bucket 's3://bucket/' created
[host] $ s3cmd ls
2023-12-14 21:48 s3://Example
2023-12-15 16:20 s3://Example1
2023-12-05 12:08 s3://Example2
2023-12-20 17:11 s3://Example3
2023-12-20 17:42 s3://bucket
To put a file in a bucket#
[host] $ s3cmd put example.file s3://bucket
upload: 'example.file' -> 's3://bucket/example.file' [1 of 1]
0 of 0 0% in 0s 0.00 B/s done
To delete a file in a bucket#
[host] $ s3cmd ls s3://bucket
2023-12-20 17:44 0 s3://bucket/example.file
[host] $ s3cmd del s3://bucket/example.file
delete: 's3://bucket/example.file'
[host] $ s3cmd ls s3://bucket
[host] $
To remove a bucket#
Note that a bucket must be empty before it can be removed.
[host] $ s3cmd rb s3://bucket
Bucket 's3://bucket/' removed
References https://doc.swift.surfsara.nl/en/latest/Pages/Advanced/advanced_usecases.html https://clouddocs.web.cern.ch/object_store/s3cmd.html
https://openmetal.io/docs/manuals/openstack-admin/swift-s3-api-access-with-s3cmd