AWS

Using AWS-CLI to interact with AWS S3

Froilan Miranda

Mar 25, 2020 • 2 min read

Overview

This article discusses some of the common commands of AWS-CLI to communicate with the AWS S3 service. The command line tool is a quick and easy way to manage S3 buckets. It is not a complicate interface, with only a hand full of commands. The following will use these commands to create, read, update and remove objects from S3

Anatomy

aws s3 <Command> [<Arg> ...]

About Paths

Every S3 command consist of at least one path argument. A path can be represented in two ways:

LocalPath - represents a path on the local file system. This can be relative or absolute
S3Uri - represents a S3 bucket, object, or prefix.

The S3 directories are refered to as prefixes

S3 resource paths

An S3 URI path is formatted like so:

s3://SomeBucket/ObjectKey

Lets break it down piece by piece.

s3:// : the path to a resource on S3 must begin with this prefix. This denotes that path aurgument refers to a S3 resource.

SomeBucket/ : this refers to the unique bucket name to access

ObjectKey : this is the specified key name value for the object within the bucket

Order Matters

All AWS-CLI S3 commands take one or two uri path arguments. The first argument will be the source path. This could be a local resource or a an S3 resource. When a second path argument is found, this will represent the destination path. This too can be either a local or S3 path. If a command only calls for one path, this is because the command operates on the source resource alone, and ther is no need of a destination path

S3 Operations

The AWS-CLI S3 commands can operate on single files or on file directories

Single File Operations

Here are commands that will operate on a single file:

cp - copy a resource from source path to destination
mv - move a resource from source path to destination
rm - rm a resource at source path

If the --recursive flag is used the operation may affect more than one file.

Slashes Matter

When creating path arguments for the source and the destination, the direction of the slashed matter. When representing a path on the local file system use the slash seperator used by the operating system. When representing a S3 resource use forward slashes.

When configuring the destination resource path, having a slash on end or not can have different behaviors

aws s3 cp src/file.txt s3://bucketname/src

aws s3 cp src/file.txt s3://bucketname/src/

Directory & Prefix Operations

Here are some commands that operate on a directories and/or the contents:

sync - Syncs directories and S3 prefixes
mb - create/make a bucket
rb - remove a bucket
ls - the directory content

Slashes Don't Matter

Unlike single file operatioins, post fixing slashes doesn't affect how directory/prefix operations work

Filters

Most commands allow for filtering using the --exclude <value> and --include <value> parameters. Pattern matching is achieved with the following symbols.

*: Matches everything
?: Matches any single character
[sequence]: Matches any character in sequence
[!sequence]: Matches any character not in sequence

The exclude and include parameters can be used multiple time is a single command

--exclude "*" --include "*.txt"

When multiple filter parameters are present the latter will override the former.

--include "*.txt" --exclude "*"

But reversing the order leads to a different outcome

Filters are applied to the source directory

`aws s3 sync ./ s3://bucket.on.s3 --exclude "" --include ".mov" --include "*.ogg"

This command will perform a sync using the current directory as the source and will exclude all files except .ogg and .mov files.

Summary

We have looked at some of the common ways to interacte with S3 using the AWS-CLI. Breaking down the format of path arguments we are able to connect with S3 buckets and objects with a local file system. We explored several ways to configure s3 commands and filter results. We are now ready to administer S3 through the command line interface!