How To Manage Files Between Local And Azure Storage With AZCopy

June Castillote

June Castillote

Read more posts by this author.

Automation is everywhere nowadays in IT. From virtual machine provisioning to web site deployment, automation is key especially when you need to deal with multiple items. Even file management between Azure storage and your local machine can be automated using a utility program called AzCopy.

AzCopy is a standalone utility that allows the management of Azure storage. With AzCopy, several operations can be performed. In this article, we’re going to focus on:

  • Creating storage containers
  • Copying local files to containers
  • Copying files between containers

Let’s begin!

If you learn better via video, you can also check out a TechSnips video closely related to what you’ll be learning in this article.

Prerequisites

This article will a walkthrough demonstrating various tasks. If you’d like to follow along, be sure you have the below list of requirements met before starting.

Creating New Storage Containers

First up, let’s use AzCopy to create new storage containers in an existing storage account. In the example you’ll be using, you’ll create two new containers named container1 and container2.

Note: Before creating a new container, you must know the container’s endpoint URL. The endpoint URL adheres to the format: https://[storageaccountname].blob.core.windows.net/[container_name].

First, define the endpoint URL. In this example, the containers will be created in a storage account called storageaccount1207 creating two endpoint URLs:

Now that you know the endpoint URLs, use the azcopy make command passing each endpoint URL as a parameter as shown below.

> azcopy make https://storageaccount1207.blob.core.windows.net/container1
> azcopy make https://storageaccount1207.blob.core.windows.net/container2
Creating new containers in the storageaccount1207 storage account
Creating new containers in the storageaccount1207 storage account

Copying Files to/from Azure Containers

Once you have a storage container created, you’re probably asking “How do I upload to Azure Blob Storage?”.  You can make it happen with AzCopy using the azcopy copy command. You can use azcopy to copy PSTs, VHDs and more.

Using OAuth Authentication

When using AzCopy to copy files to blobs, you have a couple of options for authentication. Since it is presumed you’ve already authenticated to Azure Storage (since it was in the prerequisites), you can simply run azcopy copy specifying a local file path followed by a container endpoint URL as shown below.

By not specifying any other parameter, you are using the OAuth authentication.

> azcopy copy c:\myDir\file1.txt https://storageaccount1207.blob.core.windows.net/container1
Upload file to Azure Storage using OAuth
Upload file to Azure Storage using OAuth

How do you know to confirm the file is copied to the Azure blob container? You can use the azcopy list [container url] command. See the example below showing that the file1.txt file exists.

> azcopy list https://storageaccount1207.blob.core.windows.net/container1
List the contents of the Azure blob container
List the contents of the Azure blob container

To download a file from the Azure blob to the local directory, you only need to switch the places of the source and destination arguments. See the example below.

> azcopy copy https://storageaccount1207.blob.core.windows.net/container1/file1.txt>c:\myDir
Download file from Azure Storage using OAuth authentication
Download file from Azure Storage using OAuth authentication

Using a Shared Access Signature (SAS) Token

To use a SAS token, you must first generate one. If you don’t know how to generate a SAS token, check out the How to Generate an Azure SAS Token to Access Storage Accounts article.

Once you have a SAS token available, you can append the token to the destination container’s URL as an HTTP parameter as shown below.

> azcopy copy c:\myDir\file1.txt 'https://storageaccount1207.blob.core.windows.net/container1?sv=2019-02-02&sr=c&sig=LgFsqIRHZovsSjpAVNAyyyy4tnDJ%2BxNh%2B1Cq3rY4Gbk%3D&se=2019-12-09T06%3A34%3A30Z&sp=rw'

If you are using PowerShell and stored the token to a variable (eg. $token), you can pass that variable as instead of the full token string like the command below.

> azcopy copy c:\myDir\file1.txt "https://storageaccount1207.blob.core.windows.net/container1?$token"
Copy file to Azure Container using SAS Token Authentication
Copy file to Azure Container using SAS Token Authentication

Using the same token (assuming it is still within its validity period), you can also download files from the Azure storage to the local storage. The example below copies the file1.txt file from the Azure storage with a SAS token.

In the code below, you can see that this time instead of appending the token after the container name, the name of the file is added first. This instructs AzCopy to copy a specific file using the SAS authentication.

> azcopy copy ('https://storageaccount1207.blob.core.windows.net/container1/file1.txt'+$token) C:\MyDir\
Copy file from Azure Container with SAS Token
Copy file from Azure Container with SAS Token

Copying a Directory to/from Azure Storage

If you have many files to transfer or download from Azure, you can also copy entire directories at once using azcopy copy. The command to do so is not that different from what you use to copy single files.

Below you can see a command that copies the C:\\MyDir recursively to the blob container container1. The --recursive parameter is the key here. The —recursive flag at the end of the command indicates that all contents from the parent directory and sub-directories that may exist will be copied.

> azcopy copy C:\MyDir\ https://storageaccount1207.blob.core.windows.net/container1/ --recursive

You can see below ten files inside of C:\MyDir folder were all copied successfully based on the job summary.

Copying a directory and its contents
Copying a directory and its contents

Copying Files Between Azure Storage Containers

Throughout this article, you’ve been using the azcopy copy command quite a bit. There’s no need to stop now! Not only can you copy directories and files to/from on-prem, you can also copy blobs across storage containers.

Rather than using a local file path, you’d change up the parameters to use one blob container URL as the source and another blob container URL as the destination.

You’ve already created two storage containers called container1 and container2. In this section, let’s copy files, directories and even the entire blob container from container1 to container2.

To copy files from one container to another, first, generate a new SAS token. Since the two containers belong to the same storage account, the same token can be used for both the source and destination.

Assuming you’ve saved the token string to a variable called $token (if running azcopy with PowerShell), you can easily copy all data from one storage container to the other as shown below.

Reminder: Take a look at the How to Generate an Azure SAS Token to Access Storage Accounts article if you don’t know how to generate a SAS token.

> azcopy copy ('https://storageaccount1207.blob.core.windows.net/container1'+$token) ('https://storageaccount1207.blob.core.windows.net/container2'+$token) --recursive
Copying data between containers using SAS Token authentication
Copying data between containers using SAS Token authentication

Other Useful AzCopy Operations

Aside from copying files and folders, there are other copy operations you can perform with AzCopy. Some of these are:

  • Copying specific files (filtered).
  • Copying a directory into another directory in the blob container.
  • Copying from Amazon AWS to Azure Storage.

If you’d like to learn about everything AzCopy can do, check out the help content by running azcopy -h. You’ll find many other useful commands aside from the copy command you’ve focused solely on in this article. You can see a screenshot of these commands below.

List of AzCopy Commands
List of AzCopy Commands

To learn more, visit the AzCopy Copy page. Then you can try these on your own using the techniques you learned in this article.

Summary

In this article, you learned how to use AzCopy to transfer files between the local and Azure storage. You also learned how to perform these tasks while using OAuth and Shared Access Signature Token for authentication. You’ve seen what other AzCopy commands are available and how to find them.

I hope this article has helped you get started with AzCopy and sparked your interest in using it more as part of your administration or development tasks. Thank you!

Subscribe to Adam the Automator

Get the latest posts delivered right to your inbox

Looks like you're offline!