s3cmd Basic Usage#

There are various clients that can be used to interact with the object storage at EODC. We can offer only limited support and guidance on using third party tools.

These clients make use of the S3 interface to the EODC Object storage.

It is expected that you have already generated your EC2 Credentials. For more details, see the introduction to Object Storage page.

s3cmd#

Getting Started#

s3cmd is a popular powerful python based open source CLI utility for interacting with object storage. It is available in the default repository for most Linux distributions and also available for Mac.

Install it in the appropriate way for your environment https://github.com/s3tools/s3cmd

First s3cmd must be configured for the environment Be sure to have your EC2 credentials at hand! This can be handled interactively:

[host] $ s3cmd --configure
 # Follow the prompts - Your settings will look similar

New settings:
  Access Key: $accessKey
  Secret Key: $secretKey
  Default Region: US
  S3 Endpoint: objectstore.eodc.eu:2222
  DNS-style bucket+hostname:port template for accessing a bucket: objectstore.eodc.eu:2222
  Encryption password: $optional
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Alternatively create your own s3cfg file with the following template

[default]
access_key = $access
host_base = objectstore.eodc.eu:2222
host_bucket = objectstore.eodc.eu:2222
use_https = true
secret_key = $secret

s3cmd -c s3cfg $command

Now that s3cmd is configured, we can explore some basic usage

Basic Usage#

To list all available buckets#

[host] $ s3cmd ls
2023-12-14 21:48  s3://Example
2023-12-15 16:20  s3://Example1
2023-12-05 12:08  s3://Example2
2023-12-20 17:11  s3://Example3

To list contents of a bucket#

[host] $ s3cmd ls s3://bucket
2023-12-20 17:44            0  s3://bucket/example.file

To make a new bucket#

[host] $ s3cmd mb s3://bucket
Bucket 's3://bucket/' created

[host] $ s3cmd ls
2023-12-14 21:48  s3://Example
2023-12-15 16:20  s3://Example1
2023-12-05 12:08  s3://Example2
2023-12-20 17:11  s3://Example3
2023-12-20 17:42  s3://bucket

To put a file in a bucket#

[host] $ s3cmd put example.file s3://bucket
upload: 'example.file' -> 's3://bucket/example.file'  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done

To delete a file in a bucket#

[host] $ s3cmd ls s3://bucket
2023-12-20 17:44            0  s3://bucket/example.file
[host] $ s3cmd del s3://bucket/example.file
delete: 's3://bucket/example.file'
[host] $ s3cmd ls s3://bucket
[host] $

To remove a bucket#

Note that a bucket must be empty before it can be removed.

[host] $ s3cmd rb s3://bucket
Bucket 's3://bucket/' removed

References https://doc.swift.surfsara.nl/en/latest/Pages/Advanced/advanced_usecases.html https://clouddocs.web.cern.ch/object_store/s3cmd.html

https://openmetal.io/docs/manuals/openstack-admin/swift-s3-api-access-with-s3cmd

On this page
Show Source