S3

Amazon S3 (Simple Storage Service) is an object storage service offering industry-leading scalability, data availability, security, and performance.

Operation	Command
List Buckets	aws s3 ls
Create a Bucket	aws s3 mb s3://my-new-bucket
Upload a File	aws s3 cp localfile.txt s3://my-bucket/
Download a File	aws s3 cp s3://my-bucket/remotefile.txt .
Sync Files	aws s3 sync <source> <target> [--options]

Note: For the sync command, replace <source> and <target> with appropriate local or S3 paths, and add any necessary options.

Options include:

Option	Description
--exclude	Exclude files matching the pattern
--include	Include files matching the pattern
--dryrun	Simulate the sync operation
--delete	Remove files in target not present in source
--quiet	Suppress non-error output

Example:

aws s3 sync . s3://my-bucket/backup --exclude "*.tmp" --delete

Working with S3 Objects

Rename/Move Objects

While mv exists, it's not atomic. Use cp followed by rm for better control:

aws s3 cp s3://my-bucket/oldname s3://my-bucket/newname \
    && aws s3 rm s3://my-bucket/oldname

List Objects with a Specific Prefix

aws s3 ls s3://my-bucket/prefix/

Delete Multiple Objects

aws s3 rm s3://my-bucket/prefix --recursive

S3 Storage Classes

S3 offers various storage classes for different use cases:

• Standard
• Intelligent-Tiering
• Glacier Instant Retrieval
• Glacier Deep Archive

Working with Glacier Storage Class

To restore a Glacier object:

aws s3api restore-object --bucket my-bucket --key path/to/object --restore-request '{"Days":5,"GlacierJobParameters":{"Tier":"Standard"}}'

documentation

Check restore status:

aws s3api head-object --bucket my-bucket --key path/to/object

When downloading restored Glacier objects:

aws s3 cp s3://my-bucket/path/to/object . --force-glacier-transfer

Mounting S3 as a Filesystem

Using mountpoint-s3:

1. Install:

On Ubuntu:

wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb
sudo apt-get install ./mount-s3.deb -y && rm mount-s3.deb

On macOS:

brew install --cask macfuse # install the prerequisite FUSE library
wget https://s3.amazonaws.com/mountpoint-s3-release/latest/arm64/mount-s3.tar.gz
sudo mkdir -p /opt/aws/mountpoint-s3 && sudo tar -C /opt/aws/mountpoint-s3 -xzf ./mount-s3.tar.gz
export PATH=$PATH:/opt/aws/mountpoint-s3/bin # add to .zshrc

1. Mount:

mkdir -vp /mnt/s3/my-bucket
mount-s3 my-bucket /mnt/s3/my-bucket

For more information, refer to the Mountpoint S3 documentation.

Quick start

Create mountpoints for each bucket you have access to:

for bucket in $(aws s3 ls | awk '{print $3}'); do
    mkdir -vp ./s3/$bucket
    mount-s3 $bucket ./s3/$bucket
done

Created by Ryan D. Najac for the Palomero Lab at the Institute for Cancer Genetics.
Page last updated on 2024-10-17.