---
title: "Using AWS Data Pipelines to Move Data (Step by Step Guide)"
description: "Learn how to leverage AWS Data Pipelines and move data between DynamoDB and S3 sources in this step-by-step tutorial!"
canonical: "https://adamtheautomator.com/aws-data-pipelines/"
---

# Using AWS Data Pipelines to Move Data (Step by Step Guide)

> Learn how to leverage AWS Data Pipelines and move data between DynamoDB and S3 sources in this step-by-step tutorial!

Source: https://adamtheautomator.com/aws-data-pipelines/

---

ATA Learning

Tap to hide

[

ATA Learning

](/)

*   [Home](/)
*   [Tutorials](/tutorials/)
*   [Instructors](/author/)
*   [Advertising](/advertising/)
*   [Recommended Resources](/resources/)
*   [About Adam](/about-adam/)

Search for:  

*   [](https://twitter.com/adbertram)
*   [](https://github.com/Adam-the-Automator)
*   [](https://www.linkedin.com/company/adam-the-automator-llc)
*   [](/feed/)

![Using AWS Data Pipelines to Move Data (Step by Step Guide)](https://adamtheautomator.com/wp-content/uploads/2022/02/How-To-Move-Data-Between-Sources-With-the-AWS-Data-Pipeline-Service.jpg)

# Using AWS Data Pipelines to Move Data (Step by Step Guide)

[![](https://secure.gravatar.com/avatar/7784b2856d14deb6ab96179dba6277721730e10e6979cf84e5699ddf4bb309d7?s=192&d=mm&r=g)Adedoyin Adeyemi](https://adamtheautomator.com/author/adedoyin-adeyemi/)4 February 20228 min. read

Categories: [Cloud](/category/cloud/)

Tags:[AWS](/tag/aws/)

Table of Contents

*   [Prerequisites](#prerequisites)
*   [Creating Data in a DynamoDB Table](#creating-data-in-a-dynamodb-table)
*   [Providing Access to Resources with IAM Policy and Role](#providing-access-to-resources-with-iam-policy-and-role)
*   [Creating an S3 Bucket](#creating-an-s3-bucket)
*   [Conclusion](#conclusion)

Transfer of data occurs every second between users, devices, and the internet. And if you’re looking for a way to move data from one source to another, the AWS Data Pipelines service is what you can rely on.

In this tutorial, you’ll learn how to keep your data secure by automating data transfer between sources with the AWS Data Pipelines.

Read on and start transferring data like a pro!

## Prerequisites

This tutorial will be a hands-on demonstration but doesn’t require many tools to get started. If you’d like to follow along, be sure you have an [AWS account](https://portal.aws.amazon.com/billing/signup#/start).

## Creating Data in a DynamoDB Table

What is the AWS data pipeline anyway? AWS data pipeline is a web service that helps move data within AWS compute and storage services as well as on-premises data sources at specified intervals. And as part of preparations for moving data between sources, you’ll create an [AWS](https://adamtheautomator.com/terraform-aws/) data pipeline.

Since you’ll be using DynamoDB to transfer data to an S3 bucket, start by creating a DynamoDB table.

Related:[Getting Started with the Terraform AWS Provider](https://adamtheautomator.com/terraform-aws/)

1\. Open your favorite web browser and log in to your AWS account. Locate the **Search** input field, enter “DynamoDB”, and click the **DynamoDB** item, as shown below.

![Accessing the DynamoDB Dashboard](https://adamtheautomator.com/wp-content/uploads/2022/02/image.png)

Accessing the DynamoDB Dashboard

2\. On the **DynamoDB** dashboard, click on the **Create table** button (right-most) to initiate creating a DynamoDB table.

![Initiating Table Creation](https://adamtheautomator.com/wp-content/uploads/2022/02/image-1.png)

Initiating Table Creation

3\. Enter a **Table name**, [**Partition key**, and **Sort key**](https://aws.amazon.com/blogs/database/choosing-the-right-dynamodb-partition-key/), as shown below. Leave the **Default settings** option selected as it’s the fastest way to create your table. You can always modify these settings after you’ve created the table.

![Configuring Table Settings ](https://adamtheautomator.com/wp-content/uploads/2022/02/image-2.png)

Configuring Table Settings

Scroll down and optionally add tags. But for this example, skip adding tags and click on the **Create table** button to finalize table creation.

![Creating the Table](https://adamtheautomator.com/wp-content/uploads/2022/02/image-3.png)

Creating the Table

4\. Now, click the newly created table (**Ata-empoyee\_id**) to view its properties. Your table will look similar to the one below.

Click on the **Actions** dropdown button and select **Create item** to start creating items in the table.

![Viewing the new DynamoDB Table Information](https://adamtheautomator.com/wp-content/uploads/2022/02/image-4.png)

Viewing the new DynamoDB Table Information

5\. Click on the **Add new attribute** dropdown button, select an attribute to add, and set its value. Repeat the process and add more attributes as you want, then click **Create item** to create the item on the table.

This table would serve as your data to be transferred to an S3 bucket.

![Creating Items in the DynamoDB Table](https://adamtheautomator.com/wp-content/uploads/2022/02/image-5.png)

Creating Items in the DynamoDB Table

6\. Back to your table’s (**ATA-employee\_id**) overview page, click on the **View items** button (top-right) to see the newly created item.

![Accessing List of Items in the Table](https://adamtheautomator.com/wp-content/uploads/2022/02/image-6.png)

Accessing List of Items in the Table

7\. Put a check on the item, then click the **Action** dropdown button, and choose **Duplicate** to duplicate the item as many times as you like.

![Duplicating the newly created item in table ATA-employee\_id](https://adamtheautomator.com/wp-content/uploads/2022/02/image-7.png)

Duplicating the newly created item in table ATA-employee\_id

## Providing Access to Resources with IAM Policy and Role

You now have the DynamoDB table set up. But to move data to the S3, you’ll create an [IAM policy and a role](https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-iam-roles.html) to provide access. Both IAM policy and a role that also allows or denies certain actions on services.

Related:[Learning Identity and Access Management (IAM) AWS Through Examples](https://adamtheautomator.com/iam-aws/)

1\. Search for IAM on the dashboard, and click the **IAM** item, as shown below to access the IAM dashboard.

![Searching for IAM policy and role](https://adamtheautomator.com/wp-content/uploads/2022/02/image-8.png)

Searching for IAM policy and role

2\. On the IAM dashboard, select roles on the left panel, then click on **Create roles** access the page where you can add the data pipeline as a use case.

![Initialing Role Creation](https://adamtheautomator.com/wp-content/uploads/2022/02/image-9.png)

Initialing Role Creation

3\. Next, choose the **AWS service** option, then click on **Data Pipeline** at the bottom as the use case shown below, and click **Next**.

![Selecting Data Pipeline as the Use Case for the AWS Service](https://adamtheautomator.com/wp-content/uploads/2022/02/image-10.png)

Selecting Data Pipeline as the Use Case for the AWS Service

4\. Click on **Next: Tags** to skip adding permission since the permission for this service is automatically given.

![Skipping Adding Permissions](https://adamtheautomator.com/wp-content/uploads/2022/02/image-11.png)

Skipping Adding Permissions

5\. Enter a unique name for the role (**Role name** field) and a role description (optional), but for this example, the role is named **ata-data**. This role allows the user account access services like the clusters needed for the data pipelines to run, as shown below.

Click on **Create role** to finalize creating the role.

![Creating an IAM role for the data pipeline](https://adamtheautomator.com/wp-content/uploads/2022/02/image-12.png)

Creating an IAM role for the data pipeline

6\. Now, repeat steps two to five. But this time, scroll down and choose **EC2 Role for Data Pipeline** for the use case, as shown below. For this demo, the EC2 role is named **ATA-eC2employeerole**.

This use case lets you create another role to provide access to your S3 and DynamoDB for EC2 instances.

![Creating EC2 role for the data pipeline](https://adamtheautomator.com/wp-content/uploads/2022/02/image-13.png)

Creating EC2 role for the data pipeline

7\. Navigate back to the **IAM** dashboard to confirm the roles you’ve created exist.

![Confirming IAM role and EC2 role created for the data pipeline](https://adamtheautomator.com/wp-content/uploads/2022/02/image-14.png)

Confirming IAM role and EC2 role created for the data pipeline

8\. Click the **Policies** menu on the left panel of the IAM dashboard, then click on **Create Policy**. Your browser redirects you to a page (step nine) where you can create policies to attach to the role for the data pipeline.

Policies are needed to be attached to roles to restrict or give access to resources. For this tutorial, access to perform actions on the S3, EC2, and DynamoDB needs to be specified.

![Policy tab in the IAM dashboard](https://adamtheautomator.com/wp-content/uploads/2022/02/image-15.png)

Policy tab in the IAM dashboard

9\. On the **Create policy** page, click on the **JSON** tab to write a policy in JSON format.

![Choosing JSON as the policy format](https://adamtheautomator.com/wp-content/uploads/2022/02/image-16.png)

Choosing JSON as the policy format

10\. Copy and paste the code below to allow the user account access to the data pipeline, then click on **Next: Tags** to save the policy.

```javascript
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "datapipeline:*",
            "Resource": "*"
        }
    ]
}
```

![Creating data pipeline policy](https://adamtheautomator.com/wp-content/uploads/2022/02/image-17.png)

Creating data pipeline policy

11\. Add tags optionally for ease of identification and organization, then click on **Next: Review**.

![Adding tags to the policy](https://adamtheautomator.com/wp-content/uploads/2022/02/image-18.png)

Adding tags to the policy

12\. Enter your preferred name (**ATA-pipelinepolicy**) and description for the policy, and click **Create policy** to finalize creating the policy.

![Adding name and description to the policy ](https://adamtheautomator.com/wp-content/uploads/2022/02/image-19.png)

Adding name and description to the policy

As you see below, your browser automatically redirects to the **Policies** page in the **IAM** dashboard to view the newly created policy (**ATA-pipelinepolicy**).

![Viewing Policies](https://adamtheautomator.com/wp-content/uploads/2022/02/image-20.png)

Viewing Policies

13\. Navigate to the **Roles** page in the IAM dashboard, and click on the data pipeline role (**ata-data**) you previously created (step five). Doing so redirects your browser to the role’s **Summary** page (step 14).

![Accessing ata-data role’s summary information](https://adamtheautomator.com/wp-content/uploads/2022/02/image-21.png)

Accessing ata-data role’s summary information

14\. Now, click on **Attach policies** and you’ll see a page (step 15) where you can search and choose policies to attach to the data pipeline role (**ata-data**).

![Accessing Policies to Attach to the Role](https://adamtheautomator.com/wp-content/uploads/2022/02/image-22.png)

Accessing Policies to Attach to the Role

15\. Search for the policy (**ATA-pipelinepolicy**) you created (step 12). Once found, check the box in front of the policy and click **Attach policy** to attach the policy to the data pipeline role (**ata-data**).

![Searching for Policy to Attach to the Data Pipeline Role](https://adamtheautomator.com/wp-content/uploads/2022/02/image-23.png)

Searching for Policy to Attach to the Data Pipeline Role

16\. Navigate back to the **Roles** page and choose the EC2 data pipeline (**ATA-eC2employeerole**) you previously created (step six). Repeat steps 14 to 15 to attach the **ATA-pipelinepolicy** to the **ATA-eC2employeerole** role.

![Accessing ATA-eC2employeerole role’s summary information](https://adamtheautomator.com/wp-content/uploads/2022/02/image-24.png)

Accessing ATA-eC2employeerole role’s summary information

17\. Finally, click on any of the roles on the **Roles** page in the IAM dashboard. Confirm if you’ve successfully attached the **ATA-pipelinepolicy** policy to the roles, as shown below.

![Viewing attached policy to a role (ata-data)](https://adamtheautomator.com/wp-content/uploads/2022/02/image-25.png)

Viewing attached policy to a role (ata-data)

## Creating an S3 Bucket

You now have data you can transfer between sources. But where exactly will you transfer the data? Create an S3 bucket that will serve as storage where you can transfer data from your DynamoDB table (**ATA-employee\_id**).

1\. Login to your AWS account and enter **S3** on the search box, as shown below. Click on the **S3** item to access the Amazon S3 dashboard.

![Searching for S3 dashboard.](https://adamtheautomator.com/wp-content/uploads/2022/02/image-26.png)

Searching for S3 dashboard.

2\. Next, click on the **Buckets** menu (left panel) in the S3 dashboard, and click **Create bucket** on the right-hand side to initialize creating an S3 bucket.

![Initializing Creating Bucket](https://adamtheautomator.com/wp-content/uploads/2022/02/image-27.png)

Initializing Creating Bucket

3\. Now, enter a unique bucket name. But for this demo, the bucket name is set as **ata-amp-data**. Ensure the location is the same as your DyanmoDB table’s.

![Setting S3 Bucket name and Region](https://adamtheautomator.com/wp-content/uploads/2022/02/image-28.png)

Setting S3 Bucket name and Region

4\. Scroll down, untick the **Block _all_ public access** option, and tick the confirmation box. Doing so allows the bucket to be publicly accessible.

![Making the S3 Bucket Publicly Accessible](https://adamtheautomator.com/wp-content/uploads/2022/02/image-29.png)

Making the S3 Bucket Publicly Accessible

5\. Leave the rest of the defaults and click **Create bucket** (bottom-right) to finalize creating the S3 bucket. Your browser automatically redirects to the Amazon S3 dashboard.

![Creating S3 Bucket](https://adamtheautomator.com/wp-content/uploads/2022/02/image-30.png)

Creating S3 Bucket

6\. After creating your S3 bucket, click the newly created bucket to view the bucket’s settings.

![Accessing the S3 Bucket Settings](https://adamtheautomator.com/wp-content/uploads/2022/02/image-31.png)

Accessing the S3 Bucket Settings

7\. In the bucket’s settings page, select the **Permissions** tab where you can manage policies. Scroll down to the **Bucket policy** section, and click the **Edit** button to add a policy.

![Accessing Permissions Tab to add a Policy](https://adamtheautomator.com/wp-content/uploads/2022/02/image-32.png)

Accessing Permissions Tab to add a Policy

8\. Copy and paste the code below to the policy editor. Be sure to replace `ata-emp-data` with the name you set for the newly created bucket (step three), and save the changes. The browser automatically redirects back to the **Permissions** tab.

The code below enables the S3 bucket to be accessible only to your AWS account. Replace **444455556866** with your account ID. It can be found at the top right corner of the webpage.

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::444455556866:role/ata-data",
                    "arn:aws:iam::444455556866:role/ATA-eC2employeerole"
                ]
            },
            "Action": [
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::ata-emp-data/*"
            ]
        }
    ]
}
```

![Bucket Policy](https://adamtheautomator.com/wp-content/uploads/2022/02/image-33.png)

Bucket Policy

Below, you can see that your S3 bucket is now accessible only to your AWS account, this simply means you only have access to this bucket.

![ Verifying S3 Bucket is now accessible only to you.](https://adamtheautomator.com/wp-content/uploads/2022/02/image-34.png)

Verifying S3 Bucket is now accessible only to you.

**Moving Data Between DynamoDB Table and S3 Bucket**

Now that the IAM roles and policy are set up, you can now create and configure an [AWS Data Pipeline](https://aws.amazon.com/datapipeline/) to move data between DynamoDB and S3.

1\. Search and choose **Data Pipeline** in the AWS console search box, as shown below. This action redirects your browser to the AWS Data Pipeline home page (step two).

![Searching for Data Pipeline Service](https://adamtheautomator.com/wp-content/uploads/2022/02/image-35.png)

Searching for Data Pipeline Service

2\. Click **Get started** now to initiate creating and configuring an AWS Data Pipeline.

![Getting Started with AWS Data Pipeline](https://adamtheautomator.com/wp-content/uploads/2022/02/image-36.png)

Getting Started with AWS Data Pipeline

3\. Configure the data pipeline with the following:

*   Set a **Name** and **Description** for the pipeline. For this demo, the name is set to **ATA-employees**.
*   Click the dropdown box under **Source** and select the **Export DyanmoDB table to S3** option, as shown below. This option sets the action the data pipeline will take, which is to export the data from your DynamoDB table to your S3 bucket.
*   Under the **Parameters** section, enter the name of your DynamoDB table (**ATA-employees\_id**) to the **Source DynamoDB table** **name** field.
*   Choose an S3 output folder. This folder is where you’ll export data from the DyanmoDB table.
*   Leave the **DynamoDB read throughput ratio** at default (**0.25**) and ensure the region of the DyanmoDB table is correct. In this demo, the region is set to **us-east-1**.
*   Under the **Schedule** section, choose to run the pipeline **on pipeline activation**. This option runs the pipeline once on activation.

![Configuring Data Pipeline](https://adamtheautomator.com/wp-content/uploads/2022/02/image-37.png)

Configuring Data Pipeline

*   Scroll down and select the **Enabled** option under **Pipeline Configuration** and choose an S3 location to save the logs. Selecting the **Enabled** option lets you view the logs of data transfer from DynamoDB to S3.
*   Under **Security/Access**, select the data pipeline role (**ata-data**) and EC2 instance role (**ATA-eC2employeerole**) you created.
*   Click **Activate** to activate and run the pipeline. Your browser then redirects to a page where you can see the list of available pipelines (step four).

![Configuring and Activating Data Pipeline](https://adamtheautomator.com/wp-content/uploads/2022/02/image-38.png)

Configuring and Activating Data Pipeline

4\. Click on the newly created data pipeline when the **Health Status** column shows the pipeline is **HEALTHY** to see the logs of the data transfer.

![Accessing the Data Pipeline’s Data Transfer Logs](https://adamtheautomator.com/wp-content/uploads/2022/02/image-39.png)

Accessing the Data Pipeline’s Data Transfer Logs

5\. Finally, confirm if your S3 bucket has the data that you moved from the DynamoDB table in the logs similar to the one below.

![Viewing Data Transfer Logs](https://adamtheautomator.com/wp-content/uploads/2022/02/image-40.png)

Viewing Data Transfer Logs

## **Conclusion**

In this tutorial, you learned how to move data between sources, specifically with DynamoDB and S3 bucket using the AWS Data Pipeline service. You’ve also touched on setting up IAM roles and policies to provide access for moving data between resources.

At this point, you can now transfer various data using the AWS data pipeline. So what’s next for you? Why not [build data warehouses](https://aws.amazon.com/getting-started/hands-on/deploy-data-warehouse/) to run queries easily?

Share this article

[Share on X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fadamtheautomator.com%2Faws-data-pipelines%2F&text=Using%20AWS%20Data%20Pipelines%20to%20Move%20Data%20\(Step%20by%20Step%20Guide\))[Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fadamtheautomator.com%2Faws-data-pipelines%2F)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fadamtheautomator.com%2Faws-data-pipelines%2F)

## Related Posts

![](https://adamtheautomator.com/wp-content/uploads/2026/08/featured_image-1.webp)

### [Survive the Senior Cloud Engineer AWS Interview Loop](/senior-cloud-engineer-interview-loop/)

Prepare for a senior cloud engineer interview with AWS EKS and Control Tower system design reasoning, Terraform locking, and salary negotiation tactics.

![](https://adamtheautomator.com/wp-content/uploads/2026/02/featured_image-1-scaled.jpg)

### [Transition from SysAdmin to Cloud Engineer](/transition-sysadmin-cloud-engineer/)

System administrators already have the networking, Linux, and operational expertise that cloud engineering demands. This guide maps the philosophy shift, technical domains, certifications, and portfolio steps to make the transition.

![](https://adamtheautomator.com/wp-content/uploads/2021/09/Terraform-AWS.jpg)

### [Terraform AWS Guide: Building Robust Amazon Infrastructure](/terraform-aws/)

Master Terraform AWS with our comprehensive guide: Learn step-by-step how to manage and optimize your Amazon Infrastructure effectively and efficiently.

## Categories

*   [IT Ops](/category/it-ops/)
*   [Cloud](/category/cloud/)
*   [DevOps](/category/devops/)
*   [Home Ops](/category/home-ops/)
*   [Information Security](/category/infosec/)
*   [Software Development](/category/software-development/)

## Site

*   [Home](/)
*   [Tutorials](/tutorials/)
*   [Instructors](/author/)
*   [Advertising](/advertising/)
*   [Recommended Resources](/resources/)
*   [About Adam](/about-adam/)

Copyright 2026© ATA Learning | [Privacy Policy](/privacy/)