s3cmd Advanced Usage#

s3cmd is a popular powerful python based open source CLI utility for interacting with object storage. It is available in the default repository for most Linux distributions and also available for Mac.

This page assumes that you already have configured s3cmd for usage at EODC. For further details, see the “Getting Started with s3cmd”

Bucket Lifecycle - Expiration#

Buckets can be configured with an lifecycle or expiration policy. This policy allows for automatic removal of objects based on desired criteria. These policies can be configured on a per-bucket or per-object basis.

Example Policy#

Relative: Here a policy is set that objects are automatically expired 14 days after they are created.

[host]$ s3cmd expire s3://bucket --expiry-days 14
Bucket 's3://bucket/': expiration configuration is set.

[host]$ s3cmd getlifecycle s3://bucket
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
        <Rule>
                <ID>4sgzpl003evke1w66xl6czjixxq0wfvq7vhq89mu72v273tb</ID>
                <Prefix/>
                <Status>Enabled</Status>
                <Expiration>
                        <Days>14</Days>
                </Expiration>
        </Rule>
</LifecycleConfiguration>

Note that attempting to view the policy of a bucket without a configured policy will return a 404.

[host] $  s3cmd -c clouddemo getlifecycle s3://bucket
ERROR: S3 error: 404 (NoSuchLifecycleConfiguration)

Alternative and additional options for automatic expiration:

       --expiry-date=EXPIRY_DATE
              Indicates when the expiration rule takes effect. (only for [expire] command)

       --expiry-days=EXPIRY_DAYS
              Indicates the number of days after object creation the expiration rule takes effect. (only for [expire] command)

       --expiry-prefix=EXPIRY_PREFIX
              Identifying one or more objects with the prefix to which the expiration rule applies. (only for [expire] command)

Advanced Policies#

Lifecycle Policies can be much more advanced

See which actions are possible here.

https://creodias.docs.cloudferro.com/en/latest/s3/Bucket-sharing-using-s3-bucket-policy-on-Creodias.html