S3 Object Storage

The S3 storage service is intended for storing "cold" data — i.e., large amounts of information that are accessed very infrequently and must remain "frozen" for long periods. S3 should not be used as scratch space or for routine work with files stored there, since files cannot be modified once uploaded. To change a file, you must download it, make the changes locally, and upload it again.

CLIENT CONFIGURATION

Although there are other alternatives, this documentation is based on using the AWS CLI, downloadable at: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

Once credentials are provided to the user, configure the client tool with:

aws configure --profile=profile_name

It is recommended to work with profiles so you can configure different sets of credentials to access different S3 storages.

You will be prompted for:

AWS Access Key ID
AWS Secret Access Key

and you can leave the following options at their defaults:

Default region name [None]:
Default output format [None]:

BASIC COMMANDS

List buckets:

aws s3 ls --endpoint=http://endpoint_url --profile=profile_name

List objects in a bucket:

aws s3 ls bucket_name --endpoint=http://endpoint_url --profile=profile_name

Upload a file to a bucket:

aws s3 cp file_name s3://bucket_name/ --endpoint=http://endpoint_url --profile=profile_name

IMPORTANT NOTE: If the file size exceeds 50 GB, you must include the following parameter:

--expected-size num_bytes

otherwise the upload process may fail.
https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html

Download a file from a bucket to your local machine:

aws s3 cp s3://bucket_name/file_name local_folder --endpoint=http://endpoint_url --profile=profile_name

Delete an object from a bucket:

aws s3 rm s3://bucket_name/file_name --endpoint=http://endpoint_url --profile=profile_name

ADVANCED COMMANDS

See: https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html

Upload a file providing its MD5 for verification:

  1. Compute the MD5 (hex):
openssl md5 file_name
  1. Compute the MD5 in Base64 format:
openssl md5 --binary file_name | base64
  1. Upload the file specifying the value for verification. The system will verify that the MD5 value after the upload matches the one you provided:
aws s3api put-object --bucket bucket_name --key file_name --body file_name --content-md5 "base64_md5" --endpoint=https://endpoint_url --profile=profile_name

If an error occurs, the system will report it; otherwise it will show information about the file's MD5, although not in Base64. It will correspond to the value calculated in step 1.

{
    "ETag": "\"value_calculated_in_step_one\""
}

Get the MD5s (ETags) of objects in a bucket:

aws s3api list-objects --bucket bucket_name --endpoint=http://endpoint_url --profile=profile_name

Get the MD5 (ETag) of a single object:

aws s3api head-object --bucket bucket_name --key file_name --query ETag --output text --endpoint=http://endpoint_url --profile=profile_name

NOTE: The ETag value may differ from the original file's MD5 if the object was uploaded using multipart upload. To use MD5 verification, prefer aws s3api put-object.