---
title: "Automating Tasks Using Bash Scripts and Cron Jobs with AWS"
description: "Discover how to combine bash scripts with cron jobs and leverage the power of AWS for automating tasks in this ATA Learning tutorial!"
canonical: "https://adamtheautomator.com/automating-tasks/"
---

# Automating Tasks Using Bash Scripts and Cron Jobs with AWS

> Discover how to combine bash scripts with cron jobs and leverage the power of AWS for automating tasks in this ATA Learning tutorial!

Source: https://adamtheautomator.com/automating-tasks/

---

ATA Learning

Tap to hide

[

ATA Learning

](/)

*   [Home](/)
*   [Tutorials](/tutorials/)
*   [Instructors](/author/)
*   [Advertising](/advertising/)
*   [Recommended Resources](/resources/)
*   [About Adam](/about-adam/)

Search for:  

*   [](https://twitter.com/adbertram)
*   [](https://github.com/Adam-the-Automator)
*   [](https://www.linkedin.com/company/adam-the-automator-llc)
*   [](/feed/)

![Automating Tasks Using Bash Scripts and Cron Jobs with AWS](https://adamtheautomator.com/wp-content/uploads/2023/11/automating-tasks.jpg)

# Automating Tasks Using Bash Scripts and Cron Jobs with AWS

[![](https://secure.gravatar.com/avatar/a3e6f24b6b657f0acd2615f1802513a18663dea2e04d24e1df9c1530290e9b48?s=192&d=mm&r=g)Uzma Younas](https://adamtheautomator.com/author/uzma-younas/)6 November 20236 min. read

Categories: [DevOps](/category/devops/)

Tags:[AWS](/tag/aws/)[Linux](/tag/linux/)

Table of Contents

*   [Prerequisites](#prerequisites)
*   [Creating a Bash Script for Automating Tasks](#creating-a-bash-script-for-automating-tasks)
*   [Transferring Scripts to AWS EC2 Instance](#transferring-scripts-to-aws-ec2-instance)
*   [Installing Required Packages to the AWS EC2 Instance](#installing-required-packages-to-the-aws-ec2-instance)
*   [Enabling Access to the S3 Bucket](#enabling-access-to-the-s3-bucket)
*   [Automating Tasks by Scheduling Cron Jobs](#automating-tasks-by-scheduling-cron-jobs)
*   [Conclusion](#conclusion)

Have you ever found yourself buried under a mountain of routine chores? Do not lose hope! Embrace automation as your gateway to increased productivity and streamlined efficiency. If you yearn to free up your precious time, automating tasks will be your ticket to liberation.

In this tutorial, you will dive headfirst into the world of automation using Bash scripts and cron jobs with AWS.

Wipe those tears and journey towards a more efficient and productive workflow!

## Prerequisites

Before you dive into automating tasks, ensure you are geared up and ready to make the most of this learning experience. Below are a few things you should have in place to follow along in this tutorial:

*   A system that supports AWS operations – This tutorial uses an [Ubuntu 22.04.2 LTS](https://adamtheautomator.com/install-ubuntu/).

Related:[Install Ubuntu Server 20.04: A Step-by-Step Walkthrough](https://adamtheautomator.com/install-ubuntu/)

*   An [AWS account](https://portal.aws.amazon.com/billing/signup#/start/email) – A free tier option will suffice.
*   An [EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html) running Amazon Linux 2.
*   An [S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) access for storing processed data.
*   An [IAM](https://www.ibm.com/docs/en/atcm/1.3.1?topic=cir-creating-iam-role-s3-access-in-same-aws-account) role assigned to the EC2 instance, granting access to your S3 bucket.

Related:[Helpful Guide to IAM in AWS Through Examples](https://adamtheautomator.com/iam-aws/)

*   A [Python script](https://github.com/UzmaYounas794/Daily_cron_/blob/master/script.py) (_[script.py](http://script.py)_) designed for web scraping and a _[requirements.txt](https://github.com/UzmaYounas794/Daily_cron_/blob/master/requirements.txt)_ file listing the necessary Python packages for your project.
    
    Your initial project directory structure should look like the one below:
    

![](https://adamtheautomator.com/wp-content/uploads/2023/11/image-16.png)

Illustrating the initial project directory structure

## Creating a Bash Script for Automating Tasks

Moving on from the prerequisites, you will venture into having a digital assistant to carry out routine tasks on your behalf — a Bash script. You will create a Bash script that automates extracting and uploading valuable data from a designated website to an AWS S3 bucket.

> _💡 Bash is readily available on nearly every Linux/Unix system, so scripts crafted for one system can be executed on another._

To create a Bash script for automated data extraction, follow these steps:

1\. Create a new file named _bash.sh_ with your preferred editor (i.e., Visual Studio Code) in your project directory (i.e., _/Daily\_cron_).

Related:[Master VS Code Snippets for Efficient Coding](https://adamtheautomator.com/vs-code-snippets/)

2\. Next, add the following code to your _bash.sh_ file to define the necessary variables.

```bash
#!/bin/bash

# Define a variable for your Python script file
PYTHON_SCRIPT="script.py"
# Define a variable for the output file post-data-scaping
OUTPUT_CSV="crypto.csv"
# Define a variable for your S3 bucket's name
S3_BUCKET_NAME="news--bucket"
```

3\. Include the following, which executes your Python script and saves the output to a timestamped CSV file.

```bash
# Run the Python script and save the output to a timestamped CSV file
echo "Starting the scraper script..."
python3 $PYTHON_SCRIPT > $OUTPUT_CSV || { echo "Error: Python script execution failed"; exit 1; }
```

4\. Add the following code at the bottom of the _bash.sh_ file, which uploads the saved CSV file to your AWS S3 bucket with a timestamp.

```bash
# Generate a timestamp for the new file name
TIMESTAMP=$(date +'%Y-%m-%d %H-%M-%S')
NEW_FILE="crypto-${TIMESTAMP}.csv"

# Rename the file with the timestamp
mv "$OUTPUT_CSV" "$NEW_FILE"

# Upload the CSV file to your AWS S3 bucket
echo "Uploading data to S3..."
aws s3 cp $NEW_FILE  "s3://$S3_BUCKET_NAME/$NEW_FILE" || { echo "Error: Failed to upload data to S3"; exit 1; }
```

5\. Lastly, input the following, which removes the local CSV file once uploaded to your AWS S3 bucket.

```bash
rm "$NEW_FILE" || { echo "Error: Failed to delete local CSV file"; exit 1; }
```

Once compiled in the _bash.sh_ file, the code should look like the one below.

![Illustrating the complete Bash script](https://adamtheautomator.com/wp-content/uploads/2023/11/image-17.png)

Illustrating the complete Bash script

## Transferring Scripts to AWS EC2 Instance

With a script at your disposal, you must ensure you can run automated tasks in a scalable and flexible environment — transferring scripts to your AWS EC2 instance. Imagine having the ability to execute your scripts from anywhere in the world.

To transfer scripts to your AWS EC3 instance, carry out the steps below:

Open a terminal, navigate to your project directory, and execute the following `zip` command to compress your project into a zip file.

Ensure you replace `Daily_cron.zip` with your preferred name and _`Daily_cron`_ with your project directory path.

```bash
zip -r Daily_cron.zip Daily_cron
```

![Zipping the project folder](https://adamtheautomator.com/wp-content/uploads/2023/11/image-18.png)

Zipping the project folder

Related:[Learn How to Zip and Unzip Files in Linux and be a Zip Master](https://adamtheautomator.com/unzip-files-in-linux/)

Now, run the below `scp` command to upload your zipped project to your AWS EC2 instance.

Make sure you replace the following accordingly:

*   `AWS-KEY.pem` – Your key pair file (or full path).
*   `Daily_cron.zip` – Your zipped project’s name.
*   `user@AWS-IP` – Your AWS EC2 instance’s username and the IP address (or hostname).
*   `/path/to/destination/` – Your S3 bucket’s full path.

```bash
scp -i AWS-KEY.pem Daily_cron.zip user@AWS-IP:/path/to/destination/
```

![Transferring the project to the AWS EC2 instance](https://adamtheautomator.com/wp-content/uploads/2023/11/image-19.png)

Transferring the project to the AWS EC2 instance

Related:[Securely Copy Files With the SCP Command](https://adamtheautomator.com/scp-command/)

## Installing Required Packages to the AWS EC2 Instance

Having successfully transferred your scripts to the AWS EC2 instance, what is the next step? You must ensure the environment is properly equipped to execute your scripts seamlessly. This process involves installing the necessary packages directly onto the instance.

To install the required packages to your AWS EC2 instance, proceed with the following steps:

1\. Execute the below `ssh` command to connect to your AWS EC2 instance via SSH. Make sure you replace the following:

*   _`AWS-KEY.pem`_ – Your key file’s local path.
*   `user@AWS-IP` – Your AWS EC2 instance’s username public IP address.

```bash
ssh -i AWS-KEY.pem user@AWS-IP
```

![Connecting to the AWS EC2 instance via SSH](https://adamtheautomator.com/wp-content/uploads/2023/11/image-22.png)

Connecting to the AWS EC2 instance via SSH

2\. Next, navigate to where you uploaded your project, and run the following `unzip` command to unzip the project folder (_`Daily_cron.zip`_) on your AWS EC2 instance.

```bash
unzip Daily_cron.zip
```

![Extracting the zipped project to the AWS EC2 instance](https://adamtheautomator.com/wp-content/uploads/2023/11/image-21.png)

Extracting the zipped project to the AWS EC2 instance

3\. Now, run each command below to navigate (`cd`) to the extracted project directory (i.e., `Daily_cron`) and install the required packages listed in the `requirements.txt` file.

```bash
# Navigate to the extracted project directory 
cd Daily_cron
# Install the required packages listed in the requirements.txt file
pip install -r requirements.txt
```

![Install necessary packages](https://adamtheautomator.com/wp-content/uploads/2023/11/image-20.png)

Install necessary packages

## Enabling Access to the S3 Bucket

You have successfully installed the required packages, and you are almost ready to automate tasks. But before that, you will ensure seamless interaction between your AWS EC2 instance and S3 bucket.

EC2 instances use an IAM role to manage the permissions for AWS services. Thus, you must enable access to your S3 bucket. In this example, you will grant the appropriate permissions via the AWS Command Line Interface (CLI).

Related:[Upload Files to S3 with AWS CLI: A Comprehensive Guide](https://adamtheautomator.com/upload-file-to-s3/)

To enable access to your S3 bucket, complete these steps:

1\. Execute the following command to `install` the AWS CLI (`awscli`).

```bash
sudo apt-get update
sudo apt-get install awscli
```

![Installing the AWS CLI ](https://adamtheautomator.com/wp-content/uploads/2023/11/image-25.png)

Installing the AWS CLI

Related:[Learning Ubuntu Apt Get Through Examples](https://adamtheautomator.com/ubuntu-apt-get/)

2\. Once installed, run the `aws` command below to `configure` your AWS CLI installation.

```bash
aws configure
```

When prompted, provide the necessary information, including your [AWS Access Key ID and Secret Access Key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).

![Providing credentials for the IAM role having S3 bucket access](https://adamtheautomator.com/wp-content/uploads/2023/11/image-24.png)

Providing credentials for the IAM role having S3 bucket access

3\. Now, invoke the following `bash` command to test your script (`bash.sh`) to ensure the script runs as intended and produces the desired results.

```bash
bash bash.sh
```

If all goes well, you will see an output similar to the one below stating the script execution and data upload completed successfully.

![Testing the bash script](https://adamtheautomator.com/wp-content/uploads/2023/11/image-23.png)

Testing the bash script

## Automating Tasks by Scheduling Cron Jobs

Now you have enabled access to your S3 bucket, automating tasks is within your reach. Picture this: you have routine tasks to be executed at specific times, like clockwork — scheduling cron jobs is the answer.

By scheduling cron jobs, you can automate timed tasks, ensuring that essential operations run smoothly. This way, you eliminate the need for manual intervention or constant supervision.

To schedule a cron job for task automation, execute the following steps:

```bash
crontab -e
```

Related:[How To Execute and List Cron Jobs for a Linux System via PHP](https://adamtheautomator.com/list-cron-jobs/)

> _💡 If you are on Windows, turn to Task Scheduler instead since crontab is exclusive for Unix-like systems._

Related:[Setting up a Cron Job for Windows Using the Task Scheduler](https://adamtheautomator.com/cron-for-windows/)

Below is the crontab file where you can configure cron jobs for automation.

![Accessing the crontab file](https://adamtheautomator.com/wp-content/uploads/2023/11/image-29.png)

Accessing the crontab file

2\. Next, add the following configuration, save the changes (Ctrl+O), exit the editor (Ctrl+X), and take note of the current time.

This configuration sets your script to run every minute (for demonstration), so ensure you replace `/path/to/MY-SCRIP` with the file name or full path of your script.

```bash
* * * * * /path/to/MY-SCRIPT
```

![Scheduling a cron job to run every minute ](https://adamtheautomator.com/wp-content/uploads/2023/11/image-28.png)

Scheduling a cron job to run every minute

Alternatively, you can schedule a cron job using the following general syntax.

![Illustrating the cron job syntax](https://adamtheautomator.com/wp-content/uploads/2023/11/image-27.png)

Illustrating the cron job syntax

3\. Finally, wait a minute, and run the below `aws s3` command to list (`ls`) all files in your S3 bucket (`news--bucket`).

```bash
aws s3 ls news--bucket
```

Compare the file’s timestamp with the time you saved your cron job’s configuration in step one. Assuming everything works correctly, the timestamp should be past a minute.

![Listing all files with timestamps dumped to the S3 bucket](https://adamtheautomator.com/wp-content/uploads/2023/11/image-26.png)

Listing all files with timestamps dumped to the S3 bucket

## Conclusion

Throughout this tutorial, you have accomplished something remarkable — automating tasks! By diving into Bash scripts and cron jobs, you have put your AWS workflow on autopilot. You can now channel your energy towards the more exciting parts of your projects.

Now, would you extend this work with some future ideas? Why not start generating AI news from the daily data by leveraging powerful APIs like [Op](https://openai.com/)[e](https://openai.com/)[nAI](https://openai.com/) or frameworks like [langchain](https://www.langchain.com/)? Or expand the data pipeline to extract from APIs, transform data, and load into data warehouses like [Amazon Redshift](https://aws.amazon.com/redshift/)?

Keep exploring more tasks to automate, and find clever ways to integrate other AWS services!

Related:[Getting Started with AWS Redshift](https://adamtheautomator.com/aws-redshift/)

Share this article

[Share on X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fadamtheautomator.com%2Fautomating-tasks%2F&text=Automating%20Tasks%20Using%20Bash%20Scripts%20and%20Cron%20Jobs%20with%20AWS)[Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fadamtheautomator.com%2Fautomating-tasks%2F)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fadamtheautomator.com%2Fautomating-tasks%2F)

## Related Posts

![](https://adamtheautomator.com/wp-content/uploads/2022/07/Move-Development-to-the-Cloud-with-the-AWS-Cloud9-IDE.jpg)

### [Simplify Development with AWS Cloud9: Your Cloud-Based IDE](/aws-cloud9/)

Simplify your development workflow, boost productivity, and unlock the power of collaboration with a cloud-based IDE. This AWS Cloud9 tutorial shows you how.

![](https://adamtheautomator.com/wp-content/uploads/2022/06/Effective-Job-Management-with-AWS-Batch.jpg)

### [AWS Batch: Your Answer to Efficient Job Scheduling & Execution](/aws-batch/)

Seeking a scalable, streamlined solution for batch computing? Explore how AWS Batch automates job management, saves you time, and reduces operational complexity.

![](https://adamtheautomator.com/wp-content/uploads/2023/10/aws-amplify.jpg)

### [AWS Amplify: Your Launchpad for Full Stack Apps](/aws-amplify/)

Learn how to create and deploy web applications with AWS Amplify and take your development to the next level in this ATA Learning tutorial!

## Categories

*   [IT Ops](/category/it-ops/)
*   [Cloud](/category/cloud/)
*   [DevOps](/category/devops/)
*   [Home Ops](/category/home-ops/)
*   [Information Security](/category/infosec/)
*   [Software Development](/category/software-development/)

## Site

*   [Home](/)
*   [Tutorials](/tutorials/)
*   [Instructors](/author/)
*   [Advertising](/advertising/)
*   [Recommended Resources](/resources/)
*   [About Adam](/about-adam/)

Copyright 2026© ATA Learning | [Privacy Policy](/privacy/)