Comments

Have you ever needed to rename a file on Amazon S3? Officially, the only way to do so is to download the file, change it's name, upload it again, then delete the original file.

Well, it turns out there is a way to do it using the AWS CLI without the need of any third party tools, and it is quite simple. Moreover, this technique can also be used to rename a bunch of files with little trouble.

EDIT: As Reddit user Neres28 pointed out, the download operation does not need to be done to your computer: it can be done from S3 to S3 directly with the PUT copy API operation. This was not clear in the first version of this post but was, actually, the point in it. Thanks again to Neres28 for the feedback

Assuming you have the AWS Command Line Interface correctly installed and set up with your credentials, you should be able to list your S3 buckets like this:

$ aws s3 ls 2015-06-16 12:47:13 sample-bucket-1 2015-06-16 17:36:05 sample-bucket-2

Renaming a single file

Now, let us assume there is a file on sample-bucket-1 called incorrectly-named-file.txt and that you would like to rename it to correctly-named-file.txt . In this scenario, a simple mv operation between two S3 objects (the original file and the new one) would suffice:

$ aws s3 ls sample-bucket-1 2016-04-12 15:13:20 0 incorrectly-named-file.txt $ aws s3 mv s3://sample-bucket-1/incorrectly-named-file.txt s3://sample-bucket-1/correctly-named-file.txt move: s3://sample-bucket-1/incorrectly-named-file.txt to s3://sample-bucket-1/correctly-named-file.txt $ aws s3 ls sample-bucket-1/ 2016-04-12 15:14:39 0 correctly-named-file.txt

Renaming multiple files

Sometimes, you will have a list of files that follow some kind of pattern that you need to replace. Let us see an example:

$ aws s3 ls sample-bucket-1/ 2016-04-12 15:21:04 0 file-replaceme-1.txt 2016-04-12 15:21:05 0 file-replaceme-2.txt 2016-04-12 15:21:06 0 file-replaceme-3.txt 2016-04-12 15:21:07 0 file-replaceme-4.txt 2016-04-12 15:21:08 0 file-replaceme-5.txt

Here, we would like to change replaceme for replaced for all files. First, we are going to need a list of just the filenames:

$ aws s3api list-objects --bucket sample-bucket-1 --prefix "" --delimiter "/" | grep replaceme | cut -f 3 file-replaceme-1.txt file-replaceme-2.txt file-replaceme-3.txt file-replaceme-4.txt file-replaceme-5.txt

Wait, what have we done here?

We have used the s3api subcommand to interact directly with bucket API operations as opposed to file operations.

subcommand to interact directly with bucket API operations as opposed to file operations. We have asked for the list-objects operation because we want a list of S3 objects.

operation because we want a list of S3 objects. We have set the prefix to empty ("") because our objects happen to be in the root of the bucket. If objects were located under a prefix (S3's concept of folders, so to speak), we would have used the prefix name instead: --prefix parentfolder/ or --prefix parentfolder1/parentfolder2 , etc.

to empty ("") because our objects happen to be in the root of the bucket. We have set a / delimiter to prevent recursion. Since prefixes always end with a / , we are effectively discarding them from the search results.

to prevent recursion. Since prefixes always end with a , we are effectively discarding them from the search results. We grep the results to match our search string ("replaceme"). This is done to filter out some output lines that we do not need right now.

the results to match our search string ("replaceme"). This is done to filter out some output lines that we do not need right now. We finally cut that output to select the third tab-separated field of each line, because there's where the file name is.

Now we are ready to create a for loop that iterates over these file names in order to execute an s3 mv operation:

$ for f in $(aws s3api list-objects --bucket sample-bucket-1 --prefix "" --delimiter "/" | grep replaceme | cut -f 3); do aws s3 mv s3://sample-bucket-1/$f s3://sample-bucket-1/${f/replaceme/replaced} done

Again, what have we done here?

We have created a for loop in bash that iterates over the results of the command before

loop in bash that iterates over the results of the command before On each iteration, we call aws s3 because we want to operate at the S3 file level

because we want to operate at the S3 file level We issue mv commands to rename the files just as we learned at the beginning of this post

commands to rename the files just as we learned at the beginning of this post We use bash's native substitution syntax to replace replaceme with replaced : ${f/replaceme/replaced} , where: ${var/str1/str2} is the replace command syntax var ( f ) is the variable name that comes from the for loop str1 ( replaceme ) is the name of the substring in var that we want to replace str2 ( replaced ) is the value we want to have instead of str1 when it is found

with : , where:

And the results are as expected:

$ aws s3 ls sample-bucket-1 2016-04-12 15:53:42 0 file-replaced-1.txt 2016-04-12 15:53:43 0 file-replaced-2.txt 2016-04-12 15:53:45 0 file-replaced-3.txt 2016-04-12 15:53:46 0 file-replaced-4.txt 2016-04-12 15:53:48 0 file-replaced-5.txt

I hope this is useful to you.