--- title: "Upload Files to S3 with AWS CLI: A Comprehensive Guide" description: "Master the AWS CLI tool for uploading, downloading, and synchronizing files between local locations and AWS S3 buckets." canonical: "https://adamtheautomator.com/upload-file-to-s3/" --- # Upload Files to S3 with AWS CLI: A Comprehensive Guide > Master the AWS CLI tool for uploading, downloading, and synchronizing files between local locations and AWS S3 buckets. Source: https://adamtheautomator.com/upload-file-to-s3/ --- ATA Learning Tap to hide [ ATA Learning ](/) * [Home](/) * [Tutorials](/tutorials/) * [Instructors](/author/) * [Advertising](/advertising/) * [Recommended Resources](/resources/) * [About Adam](/about-adam/) Search for: * [](https://twitter.com/adbertram) * [](https://github.com/Adam-the-Automator) * [](https://www.linkedin.com/company/adam-the-automator-llc) * [](/feed/) ![Upload Files to S3 with AWS CLI: A Comprehensive Guide](https://adamtheautomator.com/wp-content/uploads/2020/07/How-To-Upload-Local-Files-to-AWS-S3-with-the-AWS-CLI.png) # Upload Files to S3 with AWS CLI: A Comprehensive Guide [![](https://secure.gravatar.com/avatar/9a14f10ff1b1ec7d790d34f5b559e4d3de2d31b172e6ef266dfd8b479174d97b?s=192&d=mm&r=g)June Castillote](https://adamtheautomator.com/author/june/)28 July 202010 min. read Categories: [Cloud](/category/cloud/) Tags:[AWS CLI](/tag/aws-cli/)[AWS S3](/tag/aws-s3/) Table of Contents * [Prerequisites](#h-prerequisites) * [Preparing Your AWS S3 Access](#h-preparing-your-aws-s3-access) * [Creating an IAM User with S3 Access Permission](#h-creating-an-iam-user-with-s3-access-permission) * [Setting Up an AWS Profile On Your Computer](#h-setting-up-an-aws-profile-on-your-computer) * [Testing AWS CLI Access](#h-testing-aws-cli-access) * [Managing Files in S3](#h-managing-files-in-s3) * [Uploading Individual Files to S3](#h-uploading-individual-files-to-s3) * [Uploading Multiple Files and Folders to S3 Recursively](#h-uploading-multiple-files-and-folders-to-s3-recursively) * [Uploading Multiple Files and Folders to S3 Selectively](#h-uploading-multiple-files-and-folders-to-s3-selectively) * [Downloading Objects from S3](#h-downloading-objects-from-s3) * [Copying Objects Between S3 Locations](#h-copying-objects-between-s3-locations) * [Synchronizing Files and Folders with S3](#h-synchronizing-files-and-folders-with-s3) * [Synchronizing New and Updated Files with S3](#h-synchronizing-new-and-updated-files-with-s3) * [Synchronizing Deletions with S3](#h-synchronizing-deletions-with-s3) * [Summary](#h-summary) * [Further Reading](#h-further-reading) When working with Amazon S3 (Simple Storage Service), you’re probably using the _[S3 web console](https://console.aws.amazon.com/s3)_ to download, copy, or upload file to S3 buckets. Using the console is perfectly fine, that’s what it was designed for, to begin with. Not a reader? Watch this related video tutorial! **_Not seeing the video? Make sure your ad blocker is disabled._** Especially for admins who are used to more mouse-click than keyboard commands, the web console is probably the easiest. However, admins will eventually encounter the need to perform bulk file operations with Amazon S3, like an unattended file upload. The GUI is not the best tool for that. For such automation requirements with Amazon Web Services, including Amazon S3, the AWS CLI tool provides admins with command-line options for managing Amazon S3 buckets and objects. In this article, you will learn how to use the AWS CLI command-line tool to upload, copy, download, and synchronize files with Amazon S3. You will also learn the basics of providing access to your S3 bucket and configure that access profile to work with the AWS CLI tool. ## Prerequisites Since this a how-to article, there will be examples and demonstrations in the succeeding sections. For you to follow along successfully, you will need to meet several requirements. * An AWS account. If you don’t have an existing AWS subscription, you can sign up for an _[AWS Free Tier](https://aws.amazon.com/free/?all-free-tier.sort-by=item.additionalFields.SortRank&all-free-tier.sort-order=asc&awsf.Free%20Tier%20Types=*all&awsf.Free%20Tier%20Categories=*all)._ * An AWS S3 bucket. You can use an existing bucket if you’d prefer. Still, it is recommended to create an empty bucket instead. Please refer to _[Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html)_. * A Windows 10 computer with at least Windows PowerShell 5.1. In this article, PowerShell 7.0.2 will be used. * The [AWS CLI version 2](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-windows.html#cliv2-windows-install) tool must be installed on your computer. * Local folders and files that you will upload or synchronize with Amazon S3 ## Preparing Your AWS S3 Access Suppose that you already have the requirements in place. You’d think you can already go and start operating AWS CLI with your S3 bucket. I mean, wouldn’t it be nice if it were that simple? For those of you who are just beginning to work with Amazon S3 or AWS in general, this section aims to help you set up access to S3 and configure an AWS CLI profile. > _The full documentation for creating an IAM user in AWS can be found in this link below. [Creating an IAM User in Your AWS Account](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html)_ ### Creating an IAM User with S3 Access Permission When accessing AWS using the CLI, you will need to create one or more IAM users with enough access to the resources you intend to work with. In this section, you will create an IAM user with access to Amazon S3. To create an IAM user with access to Amazon S3, you first need to login to your _[AWS IAM console](https://console.aws.amazon.com/iam)_. Under the **Access management** group, click on **Users**. Next, click on **Add user**. ![IAM Users Menu](https://adamtheautomator.com/wp-content/uploads/2020/07/Untitled-1024x313.png) IAM Users Menu Type in the IAM user’s name you are creating inside the **User name**\* box such as _s3Admin_. In the **Access type**\* selection, put a check on **Programmatic access.** Then, click the **Next: Permissions** button. ![Set IAM user details](https://adamtheautomator.com/wp-content/uploads/2020/07/Untitled-1-1024x528.png) Set IAM user details Next, click on **Attach existing policies directly**. Then, search for the _AmazonS3FullAccess_ policy name and put a check on it. When done, click on **Next: Tags**. ![Assign IAM user permissions](https://adamtheautomator.com/wp-content/uploads/2020/07/Untitled-2-1024x528.png) Assign IAM user permissions Creating tags is optional in the **Add tags** page, and you can just skip this and click on the **Next: Review button**. ![IAM user tags](https://adamtheautomator.com/wp-content/uploads/2020/07/Untitled-3-1024x528.png) IAM user tags In the **Review** page, you are presented with a summary of the new account being created. Click **Create user**. ![IAM user summary](https://adamtheautomator.com/wp-content/uploads/2020/07/Untitled-4-1-1024x528.png) IAM user summary Finally, once the user is created, you must copy the **Access key ID** and the **Secret access key** values and save them for later user. Note that this is the only time that you can see these values. ![IAM user key credentials](https://adamtheautomator.com/wp-content/uploads/2020/07/Untitled-5-1024x528.png) IAM user key credentials ### Setting Up an AWS Profile On Your Computer Now that you’ve created the IAM user with the appropriate access to Amazon S3, the next step is to set up the AWS CLI profile on your computer. This section assumes that you already _[installed the AWS CLI version 2](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-windows.html#cliv2-windows-install)_ tool as required. For the profile creation, you will need the following information: * The **Access key ID** of the IAM user. * The **Secret access key** associated with the IAM user. * The **Default region name** is corresponding to the location of your AWS S3 bucket. You can check out the list of endpoints using this _[link](https://docs.aws.amazon.com/general/latest/gr/rande.html)_. In this article, the AWS S3 bucket is located in the _Asia Pacific (Sydney)_ region, and the corresponding endpoint is _ap-southeast-2_. * The default output format. Use _JSON_ for this. To create the profile, open PowerShell, and type the command below and follow the prompts. ```powershell aws configure ``` Enter the _Access key ID, Secret access key, Default region name,_ and _default output name_. Refer to the demonstration below. ![Configure an AWS CLI profile](https://adamtheautomator.com/wp-content/uploads/2020/07/configure-profile.gif) Configure an AWS CLI profile ### Testing AWS CLI Access After configuring the AWS CLI profile, you can confirm that the profile is working by running this command below in PowerShell. ```powershell aws s3 ls ``` The command above should list the Amazon S3 buckets that you have in your account. The demonstration below shows the command in action. The result shows that list of available S3 buckets indicates that the profile configuration was successful. ![List S3 buckets](https://adamtheautomator.com/wp-content/uploads/2020/07/s3-list.gif) List S3 buckets > _To learn about the AWS CLI commands specific to Amazon S3, you can visit the [AWS CLI Command Reference S3](https://docs.aws.amazon.com/cli/latest/reference/s3/#) page._ ## Managing Files in S3 With AWS CLI, typical file management operations can be done like upload files to S3, download files from S3, delete objects in S3, and copy S3 objects to another S3 location. It’s all just a matter of knowing the right command, syntax, parameters, and options. In the following sections, the environment used is consists of the following. * Two S3 buckets, namely _atasync1_and _atasync2._ The screenshot below shows the existing S3 buckets in the _[Amazon S3 console](https://console.aws.amazon.com/s3)_. ![List of available S3 bucket names in the Amazon S3 console](https://adamtheautomator.com/wp-content/uploads/2020/07/Untitled-6.png) List of available S3 bucket names in the Amazon S3 console * Local directory and files located under _c:\\sync._ ![Local Directory](https://adamtheautomator.com/wp-content/uploads/2020/07/Untitled-7.png) Local Directory ### Uploading Individual Files to S3 When you upload files to S3, you can upload one file at a time, or by uploading multiple files and folders recursively. Depending on your requirements, you may choose one over the other that you deem appropriate. To upload a file to S3, you’ll need to provide two arguments (source and destination) to the `aws s3 cp` command. For example, to upload the file _c:\\sync\\logs\\log1.xml_ to the root of the _atasync1_ bucket, you can use the command below. ```powershell aws s3 cp c:\sync\logs\log1.xml s3://atasync1/ ``` > _Note: S3 bucket names are always prefixed with S3:// when used with AWS CLI_ Run the above command in PowerShell, but change the source and destination that fits your environment first. The output should look similar to the demonstration below. ![Upload file to S3](https://adamtheautomator.com/wp-content/uploads/2020/07/upload-single-file.gif) Upload file to S3 The demo above shows that the file named _c:\\sync\\logs\\log1.xml_ was uploaded without errors to the S3 destination _s3://atasync1/_. Use the command below to list the objects at the root of the S3 bucket. ```powershell aws s3 ls s3://atasync1/ ``` Running the command above in PowerShell would result in a similar output, as shown in the demo below. As you can see in the output below, the file _log1.xml_ is present in the root of the S3 location. ![List the uploaded file in S3](https://adamtheautomator.com/wp-content/uploads/2020/07/list-files1.gif) List the uploaded file in S3 ### Uploading Multiple Files and Folders to S3 Recursively The previous section showed you how to copy a single file to an S3 location. What if you need to upload multiple files from a folder and sub-folders? Surely you wouldn’t want to run the same command multiple times for different filenames, right? The `aws s3 cp` command has an option to process files and folders recursively, and this is the `--recursive` option. As an example, the directory _c:\\sync_ contains 166 objects (files and sub-folders). ![The folder containing multiple files and sub-folders](https://adamtheautomator.com/wp-content/uploads/2020/07/Untitled-8.png) The folder containing multiple files and sub-folders Using the `--recursive` option, all the contents of the _c:\\sync_ folder will be uploaded to S3 while also retaining the folder structure. To test, use the example code below, but make sure to change the source and destination appropriate to your environment. You’ll notice from the code below, the source is c:\\sync, and the destination is _s3://atasync1/sync_. The _/sync_ key that follows the S3 bucket name indicates to AWS CLI to upload the files in the _/sync_ folder in S3. If the _/sync_ folder does not exist in S3, it will be automatically created. ```powershell aws s3 cp c:\sync s3://atasync1/sync --recursive ``` The code above will result in the output, as shown in the demonstration below. ![Upload multiple files and folders to S3](https://adamtheautomator.com/wp-content/uploads/2020/07/upload-recursive.gif) Upload multiple files and folders to S3 ### Uploading Multiple Files and Folders to S3 Selectively In some cases, uploading ALL types of files is not the best option. Like, when you only need to upload files with specific file extensions (e.g., \*.ps1). Another two options available to the `cp` command is the `--include` and `--exclude`. While using the command in the previous section includes all files in the recursive upload, the command below will include only the files that match \*_.ps1_ file extension and exclude every other file from the upload. ```powershell aws s3 cp c:\sync s3://atasync1/sync --recursive --exclude * --include *.ps1 ``` The demonstration below shows how the code above works when executed. ![Upload files that matched a specific file extension](https://adamtheautomator.com/wp-content/uploads/2020/07/upload-include-exclude-1.gif) Upload files that matched a specific file extension Another example is if you want to include multiple different file extensions, you will need to specify the `--include` option multiple times. The example command below will include only the \*_.csv_ and \*_.png_ files to the copy command. ```powershell aws s3 cp c:\sync s3://atasync1/sync --recursive --exclude * --include *.csv --include *.png ``` Running the code above in PowerShell would present you with a similar result, as shown below. ![Upload files with multiple include options](https://adamtheautomator.com/wp-content/uploads/2020/07/upload-multiple-include.gif) Upload files with multiple include options ### Downloading Objects from S3 Based on the examples you’ve learned in this section, you can also perform the copy operations in reverse. Meaning, you can download objects from the S3 bucket location to the local machine. Copying from S3 to local would require you to switch the positions of the source and the destination. The source being the S3 location, and the destination is the local path, like the one shown below. ```powershell aws s3 cp s3://atasync1/sync c:\sync ``` Note that the same options used when uploading files to S3 are also applicable when downloading objects from S3 to local. For example, downloading all objects using the command below with the `--recursive` option. ```powershell aws s3 cp s3://atasync1/sync c:\sync --recursive ``` ### Copying Objects Between S3 Locations Apart from uploading and downloading files and folders, using AWS CLI, you can also copy or move files between two S3 bucket locations. You’ll notice the command below using one S3 location as the source, and another S3 location as the destination. ```powershell aws s3 cp s3://atasync1/Log1.xml s3://atasync2/ ``` The demonstration below shows you the source file being copied to another S3 location using the command above. ![Copy objects from one S3 location to another S3 location](https://adamtheautomator.com/wp-content/uploads/2020/07/copy-s3-to-s3.gif) Copy objects from one S3 location to another S3 location ## Synchronizing Files and Folders with S3 You’ve learned how to upload, download, and copy files in S3 using the AWS CLI commands so far. In this section, you’ll learn about one more file operation command available in AWS CLI for S3, which is the `sync` command. The `sync` command only processes the updated, new, and deleted files. There are some cases where you need to keep the contents of an S3 bucket updated and synchronized with a local directory on a server. For example, you may have a requirement to keep transaction logs on a server synchronized to S3 at an interval. Using the command below, \*.XML log files located under the _c:\\sync_ folder on the local server will be synced to the S3 location at _s3://atasync1_. ```powershell aws s3 sync C:\sync\ s3://atasync1/ --exclude * --include *.xml ``` The demonstration below shows that after running the command above in PowerShell, all \*.XML files were uploaded to the S3 destination _s3://atasync1/_. ![Synchronizing local files to S3](https://adamtheautomator.com/wp-content/uploads/2020/07/initial-sync-up.gif) Synchronizing local files to S3 ### Synchronizing New and Updated Files with S3 In this next example, it is assumed that the contents of the log file _Log1.xml_ were modified. The `sync` command should pick up that modification and upload the changes done on the local file to S3, as shown in the demo below. The command to use is still the same as the previous example. ![Synchronizing changes to S3](https://adamtheautomator.com/wp-content/uploads/2020/07/sync-change1.gif) Synchronizing changes to S3 As you can see from the output above, since only the file _Log1.xml_ was changed locally, it was also the only file synchronized to S3. ### Synchronizing Deletions with S3 By default, the `sync` command does not process deletions. Any file deleted from the source location is not removed at the destination. Well, not unless you use the `--delete` option. In this next example, the file named _Log5.xml_ has been deleted from the source. The command to synchronize the files will be appended with the `--delete` option, as shown in the code below. ```powershell aws s3 sync C:\sync\ s3://atasync1/ --exclude * --include *.xml --delete ``` When you run the command above in PowerShell, the deleted file named _Log5.xml_ should also be deleted at the destination S3 location. The sample result is shown below. ![Synchronize file deletions to S3](https://adamtheautomator.com/wp-content/uploads/2020/07/sync-delete.gif) Synchronize file deletions to S3 ## Summary Amazon S3 is an excellent resource for storing files in the cloud. With the use of the AWS CLI tool, the way you utilize Amazon S3 is further expanded and opens the opportunity to automate your processes. In this article, you’ve learned how to use the AWS CLI tool to upload, download, and synchronize files and folders between local locations and S3 buckets. You’ve also learned that S3 buckets’ contents can also be copied or moved to other S3 locations, too. There can be many more use-case scenarios for using the AWS CLI tool to automate file management with Amazon S3. You can even try to combine it with PowerShell scripting and _build your own tools_ or _[modules](https://adamtheautomator.com/powershell-modules/)_ that are reusable. It is up to you to find those opportunities and show off your skills. ## Further Reading * **_[What Is the AWS Command Line Interface?](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html)_** * **_[What is Amazon S3?](https://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html)_** * **_[How To Sync Local Files And Folders To AWS S3 With The AWS CLI](https://www.youtube.com/watch?v=pfo6ZD2fE8w)_** Share this article [Share on X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fadamtheautomator.com%2Fupload-file-to-s3%2F&text=Upload%20Files%20to%20S3%20with%20AWS%20CLI%3A%20A%20Comprehensive%20Guide)[Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fadamtheautomator.com%2Fupload-file-to-s3%2F)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fadamtheautomator.com%2Fupload-file-to-s3%2F) ## Related Posts ![](https://adamtheautomator.com/wp-content/uploads/2021/08/pankaj-patel-Ylk5n_nd9dA-unsplash-scaled.jpg) ### [How to Set Up An AWS S3 Static SSL Website](/aws-s3-static-ssl-website/) Learn how to set up an AWS S3 static SSL website served securely using AWS Cloudfront in this step-by-step guide! ![](https://adamtheautomator.com/wp-content/uploads/2020/02/0.png) ### [Deploy AWS CLI and CloudFormation for Infrastructure](/aws-cli-and-cloudformation/) Master AWS CLI and CloudFormation deployment for efficient infrastructure management. Learn the advantages over manual provisioning. ![](https://adamtheautomator.com/wp-content/uploads/2019/07/photo-1555949963-aa79dcee981c.jpg) ### [Rename S3 Folders Swiftly with Python and Boto SDK](/rename-s3-folder/) Discover how to quickly rename S3 folder objects using Python and the powerful boto SDK. ## Categories * [IT Ops](/category/it-ops/) * [Cloud](/category/cloud/) * [DevOps](/category/devops/) * [Home Ops](/category/home-ops/) * [Information Security](/category/infosec/) * [Software Development](/category/software-development/) ## Site * [Home](/) * [Tutorials](/tutorials/) * [Instructors](/author/) * [Advertising](/advertising/) * [Recommended Resources](/resources/) * [About Adam](/about-adam/) Copyright 2026© ATA Learning | [Privacy Policy](/privacy/)