s3cmd Advanced Usage#
s3cmd is a popular powerful python based open source CLI utility for interacting with object storage. It is available in the default repository for most Linux distributions and also available for Mac.
This page assumes that you already have configured s3cmd for usage at EODC. For further details, see the “Getting Started with s3cmd”
Bucket Lifecycle - Expiration#
Buckets can be configured with an lifecycle or expiration policy. This policy allows for automatic removal of objects based on desired criteria. These policies can be configured on a per-bucket or per-object basis.
Example Policy#
Relative: Here a policy is set that objects are automatically expired 14 days after they are created.
[host]$ s3cmd expire s3://bucket --expiry-days 14
Bucket 's3://bucket/': expiration configuration is set.
[host]$ s3cmd getlifecycle s3://bucket
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>4sgzpl003evke1w66xl6czjixxq0wfvq7vhq89mu72v273tb</ID>
<Prefix/>
<Status>Enabled</Status>
<Expiration>
<Days>14</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>
Note that attempting to view the policy of a bucket without a configured policy will return a 404.
[host] $ s3cmd -c clouddemo getlifecycle s3://bucket
ERROR: S3 error: 404 (NoSuchLifecycleConfiguration)
Alternative and additional options for automatic expiration:
--expiry-date=EXPIRY_DATE
Indicates when the expiration rule takes effect. (only for [expire] command)
--expiry-days=EXPIRY_DAYS
Indicates the number of days after object creation the expiration rule takes effect. (only for [expire] command)
--expiry-prefix=EXPIRY_PREFIX
Identifying one or more objects with the prefix to which the expiration rule applies. (only for [expire] command)
Advanced Policies#
Lifecycle Policies can be much more advanced
See which actions are possible here.