--- title: "Getting Started with Azure Data Factory" description: "Struggling to migrate your data to the cloud? Learn how to utilize Azure Data Factory and store data to the cloud via the Copy Data Tool in this step-by-step tutorial!" canonical: "https://adamtheautomator.com/azure-data-factory/" --- # Getting Started with Azure Data Factory > Struggling to migrate your data to the cloud? Learn how to utilize Azure Data Factory and store data to the cloud via the Copy Data Tool in this step-by-step tutorial! Source: https://adamtheautomator.com/azure-data-factory/ --- ATA Learning Tap to hide [ ATA Learning ](/) * [Home](/) * [Tutorials](/tutorials/) * [Instructors](/author/) * [Advertising](/advertising/) * [Recommended Resources](/resources/) * [About Adam](/about-adam/) Search for: * [](https://twitter.com/adbertram) * [](https://github.com/Adam-the-Automator) * [](https://www.linkedin.com/company/adam-the-automator-llc) * [](/feed/) ![Getting Started with Azure Data Factory](https://adamtheautomator.com/wp-content/uploads/2021/11/Getting-Started-with-Azure-Data-Factory.jpg) # Getting Started with Azure Data Factory [![](https://secure.gravatar.com/avatar/bbb3aa33d8c35820d2892462d21634beb25120c1b1e62e96974f69814715f6bd?s=192&d=mm&r=g)Victor Silva](https://adamtheautomator.com/author/victorsilva/)16 November 20217 min. read Categories: [IT Ops](/category/it-ops/) Tags:[Azure](/tag/azure/)[Azure Data Factory](/tag/azure-data-factory/)[Microsoft Azure](/tag/microsoft-azure/) Table of Contents * [Prerequisites](#prerequisites) * [What is Azure Data Factory?](#what-is-azure-data-factory) * [Creating an Azure Data Factory](#creating-an-azure-data-factory) * [Creating an Azure Data Lake Storage Account](#creating-an-azure-data-lake-storage-account) * [Performing a Data Ingest with the Copy Data Tool](#performing-a-data-ingest-with-the-copy-data-tool) * [Conclusion](#conclusion) Nowadays, data is the new oil, but how does this affect a business transitioning to the cloud? or what happens when you migrate to the cloud with the raw and disorganized data stored in the different storage systems? Fortunately, Microsoft Azure Data Factory has answered these questions. Microsoft Azure Data Factory offers an [extract-transform-load (ETL)](https://docs.microsoft.com/en-us/azure/architecture/example-scenario/data/hybrid-etl-with-adf) and data integration services. These services allow you to build data-driven workflows called [data pipelines](https://docs.microsoft.com/en-us/azure/data-factory/concepts-pipelines-activities). Sounds good? Let’s explore more about how awesome Azure Data Factory is! ## Prerequisites If you’d like to follow along with the step-by-step tutorial, be sure you have an Azure subscription. If you don’t have one, you can register for a [free Azure account](https://azure.microsoft.com/en-us/free/). ## What is Azure Data Factory? In a nutshell, Azure Data Factory provides a cloud-based data integration service that orchestrates the migration and transformation of data between different data stores and compute resources. One of the interesting actions that you make with Azure Data Factory is to copy data to and from different [Software-as-a-Service (SaaS)](https://azure.microsoft.com/en-us/overview/what-is-saas/) applications, on-premises data stores and, cloud data stores. And, during the copying process, you can even convert file formats. After getting the data, you continue working on the transformation side, called [Data Flows](https://docs.microsoft.com/en-us/azure/data-factory/concepts-data-flow-overview#data-flow-data-types) provided by the [Integration Runtime (IR)](https://docs.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime), taking the ETL process seriously. ## Creating an Azure Data Factory An introduction wouldn’t be enough to understand how Azure Data Factory works, so you’ll create an actual Azure Data Factory! 1\. First, open your favorite web browser and navigate to the [Azure Portal](https://portal.azure.com/). 2\. Next, click on the portal menu icon at the top-left of the page and select **All services** from the menu panel. Doing so lets you view all services available to you. ![Accessing All Services](https://adamtheautomator.com/wp-content/uploads/2021/11/image-204.png) Accessing All Services 3\. Click on **Analytics** —> **Data factories** to view the overview page where you’ll see all of your previous data factories’ resources. ![Accessing Data Factories](https://adamtheautomator.com/wp-content/uploads/2021/11/image-205.png) Accessing Data Factories 4\. Now, click **Create** to open the assistant to create a new data factory resource. ![Starting to create an Azure Data Factory resource](https://adamtheautomator.com/wp-content/uploads/2021/11/image-206.png) Starting to create an Azure Data Factory resource 5\. Select the subscription and resource group to place the new resources. But if you prefer to create a new resource group, click the **Create new** link, enter your preferred name and click **OK**. ![Creating a New Resource Group for Deploying Azure Data Factory](https://adamtheautomator.com/wp-content/uploads/2021/11/image-207.png) Creating a New Resource Group for Deploying Azure Data Factory 6\. Now, configure the instance details with the following: * A region to create the resource, typically choosing the location closest to you. * A name you prefer for the resource. * Check that the version is V2, to get the latest features and improvements (recommended by default). * Click on the **Next: Git configuration** button to continue the creation wizard. ![Defining the Instance Details](https://adamtheautomator.com/wp-content/uploads/2021/11/image-208.png) Defining the Instance Details 7\. Check the box **Configure Git later** because now you are only creating the instance. Click on **Next: Networking** to access the **Networking** configuration tab. Related:[How to Get Started to Git Bash on Windows](https://adamtheautomator.com/git-bash/) ![Postponing the Git configuration](https://adamtheautomator.com/wp-content/uploads/2021/11/image-209.png) Postponing the Git configuration 8\. Click on **Review + create** to finish the config steps. By default, the configuration is set to **Public endpoint**, which is okay. > _The Advanced config tab is necessary if you need to specify a custom key to encrypt your data and not use the Microsoft-managed key. While the Tags config tab is required, for example, when you use tags to identify resources within a project. For this demo, these tabs are not necessary._ ![Setting Up Network Configuration](https://adamtheautomator.com/wp-content/uploads/2021/11/image-210.png) Setting Up Network Configuration 9\. Once all the final validation passes, click on the **Create** button to instruct Azure to begin creating the instance of the Azure Data Factory. ![Passing Validations and Creating Azure Data Factory](https://adamtheautomator.com/wp-content/uploads/2021/11/image-211.png) Passing Validations and Creating Azure Data Factory ## Creating an Azure Data Lake Storage Account Now you have an Azure Data Factory, you can now start copying data. But first, create the [Azure Data Lake Storage Account](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) to copy data into using the Azure Data Factory. 1\. On the Azure Portal, hover your mouse on the **Storage accounts** button, and click **Create** on the pop-up window that appears, as shown below. Doing so redirects the page to the “Storage Account” page. ![Creating a Storage Account from Azure Portal](https://adamtheautomator.com/wp-content/uploads/2021/11/image-212.png) Creating a Storage Account from Azure Portal 2\. Under the **Basics** tab, choose the same resource group and region you selected in the “Creating an Azure Data Factory” section (step six). ![Setting up Resource Group and Region for Storage Account](https://adamtheautomator.com/wp-content/uploads/2021/11/image-213.png) Setting up Resource Group and Region for Storage Account 3\. Give your storage account a **unique name**, but for this example, the storage account is named _demodfstorageaccount_. Choose **Standard** performance option and change redundancy to [Locally-redundant storage (LRS)](https://docs.microsoft.com/en-us/azure/storage/common/storage-redundancy#redundancy-in-the-primary-region) to keep costs down for these demos. Click **Next: Advanced** to get to the **Advanced** tab. ![Defining the Name of the Storage Account and Redundancy](https://adamtheautomator.com/wp-content/uploads/2021/11/image-214.png) Defining the Name of the Storage Account and Redundancy 4\. Over the **Advanced** tab, under **Data Lake Storage Gen2**, check the **enable hierarchical namespace** option. Selecting this option turns your storage account into a data lake instead of a regular [blob storage account](https://azure.microsoft.com/es-es/services/storage/blobs/). Click on **Review + create** to review your storage account settings. ![Enabling Hierarchical Namespace to the Storage Account](https://adamtheautomator.com/wp-content/uploads/2021/11/image-215.png) Enabling Hierarchical Namespace to the Storage Account 5\. After the validation, click on the **Create** button to finish the creation of your Azure Data Lake Storage Account: ![Validating and Finalizing Storage Account Creation](https://adamtheautomator.com/wp-content/uploads/2021/11/image-216.png) Validating and Finalizing Storage Account Creation ## Performing a Data Ingest with the Copy Data Tool You now have an active storage account, so all that’s left now is to test if the storage account is working. You’re going to copy a sample CSV file into a blob container called weather in your Azure Data Lake Storage Account. > _The [selected datasets](https://www1.ncdc.noaa.gov/pub/data/cdo/samples/GHCND_sample_csv.csv) for this demo are from the National Centers for Environmental Information / National Oceanic and Atmospheric Administration (NCDD-NOAA). They also have a sample database for the daily summaries of the Global Historical Climate Network that you can [download for free as CSV files](https://www.ncdc.noaa.gov/cdo-web/datasets)._ 1\. In your browser, navigate to the [Azure Data Factory](https://adf.azure.com/) home page, and then click **Ingest** to open the Copy Data Tool where you’ll create a copy job. ![Accessing Copy Data Tool](https://adamtheautomator.com/wp-content/uploads/2021/11/image-217.png) Accessing Copy Data Tool 2\. On the **Properties** page, choose the **Built-in copy task** option to create a single pipeline. Keep the default **Run once now** option selected for the task schedule, and click **Next**. Selecting the **Run once now** option runs the copy job immediately after you set it up. ![Selecting the Built-in Copy Task](https://adamtheautomator.com/wp-content/uploads/2021/11/image-218.png) Selecting the Built-in Copy Task 3\. Next, on the **Source data store** page, choose **HTTP** from the **Source type** dropdown to use the selected dataset for this demo. Click **New connection** to specify the source URL. ![Adding a New Connection on the Source Data Store](https://adamtheautomator.com/wp-content/uploads/2021/11/image-219.png) Adding a New Connection on the Source Data Store Configure the new connection with the following: * Give a name to the connection (the definition of the data source) * Copy and paste the dataset URL below for this demo to the **Base URL** field: https://www1.ncdc.noaa.gov/pub/data/cdo/samples/GHCND\_sample\_csv.csv * Change the **Authentication type** to **Anonymous** * Keep the other default options, and click on ![Configuring the New Connection](https://adamtheautomator.com/wp-content/uploads/2021/11/image-220.png) Configuring the New Connection 5\. Click on **Next** since you don’t need to change or specify anything for the source data store in this demo. ![Configuring source data store](https://adamtheautomator.com/wp-content/uploads/2021/11/image-221.png) Configuring source data store 6\. Now, check the **First row as header** option to specify whether to consider the first row as a header. Set the **Column delimiter** as **Comma (,)** because the format of the origin’s dataset is a CSV file. Click **Preview data** to view the sample dataset shown in the next step. ![File format settings page](https://adamtheautomator.com/wp-content/uploads/2021/11/image-222.png) File format settings page 7\. Scroll through the dataset to see its content, then close the window and click **Next**. ![Preview data from the dataset](https://adamtheautomator.com/wp-content/uploads/2021/11/image-223.png) Preview data from the dataset 8\. On the **Destination data store** page, choose your previously created storage account (**Azure Data Lake Storage Gen2**) from the **Target type** dropdown. Click the **New connection** button to open the assistant to create a new connection in the next step. ![Configuring Target Destination Data Store](https://adamtheautomator.com/wp-content/uploads/2021/11/image-224.png) Configuring Target Destination Data Store 9\. Configure the new connection with the following: * Enter your preferred name for the new connection in the **Name** field. * Click on the **Storage account name** dropdown and choose the storage account you previously created (demodfstorageaccount) — “Creating an Azure Data Lake Storage Account” section (step three). * Click on **Create** to create the new connection. ![Configuring New Connection for the Azure Data Lake Storage](https://adamtheautomator.com/wp-content/uploads/2021/11/image-225.png) Configuring New Connection for the Azure Data Lake Storage 10\. Now, enter your preferred **Folder path** and **File name.** But for this demo, the dataset with the file name **GHCN-daily** will be copied to the **weather** folder. Keep the other default options, and click **Next**. ![Setting up Destination Folder path and File name](https://adamtheautomator.com/wp-content/uploads/2021/11/image-226.png) Setting up Destination Folder path and File name 11\. In the **File format settings** page, confirm the **File format** is set to **Text format** and the Column delimiter is set to **Comma (,)**, then click **Next**. Setting the delimiter to **Comma (,)** is crucial since you’re creating a copy job for a CSV file. ![File format settings page](https://adamtheautomator.com/wp-content/uploads/2021/11/image-227.png) File format settings page 12\. Configure the Settings page with the following: * Write a pipeline name in the **Task name** field. * Change the [fault tolerance](https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-fault-tolerance) to **Skip incompatible rows** to avoid errors and check the **Enable logging** option to save logs. * Specify _weather/errors_ (these folders will be created) as the **Folder path** to save the logs. * Click **Next** to review the overall settings of the copy job you’re creating shown in the next step. ![Configuring Pipeline Settings for the Deployment Process](https://adamtheautomator.com/wp-content/uploads/2021/11/image-228.png) Configuring Pipeline Settings for the Deployment Process 13\. On the **Summary** page, review the overall settings for the copy job, and click **Next** to start the deployment process. Below, you can see that you are ingesting data from an **HTTP** source to an **Azure Data Lake Storage Gen2** target. Since you chose the “run once now” option in step two, the pipeline executes immediately. ![Viewing Summary of the Copy Data Tool Process](https://adamtheautomator.com/wp-content/uploads/2021/11/image-229.png) Viewing Summary of the Copy Data Tool Process 14\. Once the deployment is complete, click on the **Monitor** button to view the execution on the “Monitor” page. ![Click on the Monitor button to open the Monitor page ](https://adamtheautomator.com/wp-content/uploads/2021/11/image-230.png) Click on the Monitor button to open the Monitor page After starting the Deployment and clicking Monitor, you will be directed to the Pipeline runs page as shown below. ![Deployment executed succeeded](https://adamtheautomator.com/wp-content/uploads/2021/11/image-231.png) Deployment executed succeeded 15\. Finally, open your Azure Storage Explorer, and you’ll see your newly moved data like the one below. ![Viewing Newly Moved Data in Azure Storage Explorer](https://adamtheautomator.com/wp-content/uploads/2021/11/image-232.png) Viewing Newly Moved Data in Azure Storage Explorer ## Conclusion In this tutorial, you learned how to navigate the Copy Data Tool to download a CSV file from your demo datasets and load it into your Azure data lake storage account. Microsoft Azure Data Factory is a great tool to store your data in the cloud efficiently and more rapidly. What’s next? Why not explore different data sources and datasets? Or transform your data using the file format settings, and save them to a different target data store? Share this article [Share on X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fadamtheautomator.com%2Fazure-data-factory%2F&text=Getting%20Started%20with%20Azure%20Data%20Factory)[Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fadamtheautomator.com%2Fazure-data-factory%2F)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fadamtheautomator.com%2Fazure-data-factory%2F) ## Related Posts ![](https://adamtheautomator.com/wp-content/uploads/2022/09/Discover-How-to-Use-and-Manage-Azure-Portal.jpg) ### [Discover How to Use and Manage Azure Portal](/manage-azure-portal/) Discover the ins-and-outs to use and manage the Azure Portal and take control of your Microsoft Azure resources! ![](https://adamtheautomator.com/wp-content/uploads/2026/06/featured_image-21.png) ### [Automate SOC 2 Compliance with PowerShell](/automate-soc-2-compliance-powershell/) A comprehensive walkthrough on using PowerShell and Azure Policy as Code to automate evidence collection and enforce SOC 2 compliance controls across enterprise cloud environments. ![](https://adamtheautomator.com/wp-content/uploads/2026/06/ditch-gpos-intune-featured.webp) ### [Ditch the GPOs: Migrate to Microsoft Intune](/ditch-gpos-migrate-microsoft-intune-2/) Use Group Policy Analytics to migrate GPOs to Intune Settings Catalog profiles, handle unsupported Group Policy Preferences, and manage hybrid AD-to-MDM transitions. ## Categories * [IT Ops](/category/it-ops/) * [Cloud](/category/cloud/) * [DevOps](/category/devops/) * [Home Ops](/category/home-ops/) * [Information Security](/category/infosec/) * [Software Development](/category/software-development/) ## Site * [Home](/) * [Tutorials](/tutorials/) * [Instructors](/author/) * [Advertising](/advertising/) * [Recommended Resources](/resources/) * [About Adam](/about-adam/) Copyright 2026© ATA Learning | [Privacy Policy](/privacy/)