---
title: "How To Use Data Cleaning Python Tools"
description: "Struggling with formatting errors, misspelled words or missing data that makes it hard to analyze data? Learn how to use data cleaning Python tools in this step-by-step tutorial!"
canonical: "https://adamtheautomator.com/data-cleaning-python/"
---

# How To Use Data Cleaning Python Tools

> Struggling with formatting errors, misspelled words or missing data that makes it hard to analyze data? Learn how to use data cleaning Python tools in this step-by-step tutorial!

Source: https://adamtheautomator.com/data-cleaning-python/

---

ATA Learning

Tap to hide

[

ATA Learning

](/)

*   [Home](/)
*   [Tutorials](/tutorials/)
*   [Instructors](/author/)
*   [Advertising](/advertising/)
*   [Recommended Resources](/resources/)
*   [About Adam](/about-adam/)

Search for:  

*   [](https://twitter.com/adbertram)
*   [](https://github.com/Adam-the-Automator)
*   [](https://www.linkedin.com/company/adam-the-automator-llc)
*   [](/feed/)

![How To Use Data Cleaning Python Tools](https://adamtheautomator.com/wp-content/uploads/2021/12/How-To-Use-Data-Cleaning-Python-Tools.jpg)

# How To Use Data Cleaning Python Tools

[![](https://secure.gravatar.com/avatar/1bc5e1d52466fc3739f550c8d78be310684747bf1466a98d698320627ea243e2?s=192&d=mm&r=g)Michael Nguyen Tu](https://adamtheautomator.com/author/michael-nguyen-tu/)17 December 20216 min. read

Categories: [IT Ops](/category/it-ops/)

Tags:[Python](/tag/python/)

Table of Contents

*   [Prerequisites](#prerequisites)
*   [Importing Data Cleaning Python Pandas Library](#importing-data-cleaning-python-pandas-library)
*   [Removing Whitespaces in Datasets](#removing-whitespaces-in-datasets)
*   [Removing Duplicate Values](#removing-duplicate-values)
*   [Filling in the Missing Values](#filling-in-the-missing-values)
*   [Fixing Formatting Errors](#fixing-formatting-errors)
*   [Correcting Misspelled Words](#correcting-misspelled-words)
*   [Conclusion](#conclusion)

Data is the lifeblood of every company, and in a machine learning setting, data is generated from several sources. Data cleaning is crucial for a machine learning setting to work correctly. But how do you perform data cleaning? Data cleaning Python tools are just what you need!

In this tutorial, you will learn what data cleaning is and how to clean data with Python tools so that you can enjoy fresh and clean data.

## Prerequisites

This tutorial will be a hands-on demonstration. If you’d like to follow along, be sure you have the following:

*   A Window or Linux machine – This tutorial uses Windows 10 21H1 Build 19043.
*   [Jupyter Lab](https://jupyterlab.readthedocs.io/en/stable/) (version 3.12.1 is used in this tutorial) and [Python](https://www.python.org/) 3 or higher.

Related:[How Do You Install Python 3.6?](https://adamtheautomator.com/install-python-36/)

*   Download the [Pokémon dataset](https://github.com/Adam-the-Automator/Scripts/blob/975a1e02d5f69cd5944c79b3565953a500160f3f/data-cleaning-python/pokemon.csv) to use for the demos.

## Importing Data Cleaning Python Pandas Library

Python has several built-in libraries to help with data cleaning. The two most popular libraries are [pandas](https://pandas.pydata.org/) and [numpy](https://numpy.org/), but you’ll be using pandas for this tutorial. Pandas library allows you to work with [pandas dataframe](https://www.geeksforgeeks.org/python-pandas-dataframe/) for data analysis and manipulation.

Before you can perform data cleansing with [Python](https://adamtheautomator.com/read-csv-in-python/) pandas, import the pandas library and your dataset (CSV file) first:

Related:[Python 101: How to Manage and Read CSV in Python](https://adamtheautomator.com/read-csv-in-python/)

[Launch your JupyterLab](https://docs.jupyter.org/en/latest/running.html), and then drag and drop the Pokémon dataset into your JupyterLab.

Now, run the below commands in sequence to read the dataset and display a preview of the data, so you can check if you have any import errors.

```python
# Import the pandas library and set pd as the standard way 
# to reference pandas.
import pandas as pd
# Read the data from the dataset into your pandas dataframe.
data = pd.read_csv("pokemon.csv")
# Display a preview of the data.
data.head()
```

![Importing pokemon.csv into JupyterLab.](https://adamtheautomator.com/wp-content/uploads/2021/12/image-140.png)

Importing pokemon.csv into JupyterLab.

## Removing Whitespaces in Datasets

Now that you’ve imported your dataset, you can start cleaning your data. There are many ways to clean your dataset, like removing whitespaces. Whitespaces unnecessarily increase the size of your dataset in your database and make finding duplicate data a challenge.

1\. Check your dataset if there are whitespaces like what you see in the **Name**, **Type**, and **Weaknesses** columns below. You’ll remove these irrelevant parts of the data systematically.

![Viewing Whitespaces in Dataset](https://adamtheautomator.com/wp-content/uploads/2021/12/image-141.png)

Viewing Whitespaces in Dataset

2\. Copy and paste the following codes to your code shell, and press **Shift**+**Enter** keys to execute the code. The code below passes the column name to the [`replace()`](https://pandas.pydata.org/docs/reference/api/pandas.Series.str.replace.html) function to remove leading and trailing whitespaces in your dataset.

Related:[Getting Started: Python Functions for Newbies](https://adamtheautomator.com/python-function-return/)

```python
# remove whitespaces from Name column
data["Name"].str.replace(' ', '')
# remove whitespaces from Weight column
data["Type"].str.replace(' ', '')
# remove whitespaces from Type column
data["Weaknesses"].str.replace(' ', '')
```

3\. Finally, check your dataset again to confirm the whitespaces are gone similar to the one below.

![Verifying Whitespaces are Removed](https://adamtheautomator.com/wp-content/uploads/2021/12/image-142.png)

Verifying Whitespaces are Removed

## Removing Duplicate Values

Whitespaces is not the only one you’ll need to look out for in a dataset. With tons of data in your dataset, you may have overlooked some duplicates. So what’s the process of detecting and removing duplicates? You’ll first look for duplicates by column name in your dataset and remove them.

Each entry in your dataset should have unique data under the **Name** column. But as you can see below, **Blastoise** has two entries, one at row 10 and another one at row 11. Since the **Height** column should only contain numbers, you’ll remove the entry at row 11, which has the excess **inches** text in its **Height** column.

![Reviewing Duplicates in Dataset](https://adamtheautomator.com/wp-content/uploads/2021/12/image-143.png)

Reviewing Duplicates in Dataset

Run the following commands to remove the first duplicate [`data.drop_duplicates`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.drop_duplicates.html) and keep the last (`keep="last"`) occurrence.

```python
data.drop_duplicates(subset=["Name"], keep="last", inplace=True)
data.head(15)
```

As you can see below, the duplicate on row 10 is now gone, so you don’t have to worry about fixing that excess “inches” string anymore.

![Verifying Duplicates are Removed](https://adamtheautomator.com/wp-content/uploads/2021/12/image-144.png)

Verifying Duplicates are Removed

## Filling in the Missing Values

So far, you’ve tackled removing excess data (whitespaces and duplicates), but what about missing data? With the [`data.info()`](https://www.w3resource.com/pandas/dataframe/dataframe-info.php) command, you can check columns with missing data in your dataset.

> _From this point, filling in the missing data is crucial, or else you’ll get an error when running commands in the following sections._

1\. Run the [`data.info()`](https://www.w3resource.com/pandas/dataframe/dataframe-info.php) command below to check for missing values in your dataset.

```python
data.info()
```

There’s a total of **151** entries in the dataset. In the output shown below, you can tell that three columns are missing data. Both the **Height** and **Weight** columns have **150** entries, and the **Type** column only has **149** entries.

![Finding missing data](https://adamtheautomator.com/wp-content/uploads/2021/12/image-145.png)

Finding missing data

2\. Next, run the following command to show all entries with at least one (`.any(axis=1)`) missing data [`data.isnull()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.isnull.html).

```python
data[data.isnull().any(axis=1)]
```

Notice below that the **Height**, **Weight**, and **Type** columns have the Not a Number (**NaN**) value. The NaN values indicate the columns have null or missing data.

In the output below, you can see Golbat is missing **Height** and **Weight** data that you’ll fill in on the next step, so be sure to note Golbat’s entry number (**42**).

![Finding entries with missing data](https://adamtheautomator.com/wp-content/uploads/2021/12/image-146.png)

Finding entries with missing data

3\. Look for [Golbat’s information on the Pokémon website](https://www.pokemon.com/us/pokedex/golbat) on your web browser. In Golbat’s data below, you can see the **Height** value is **5′ 03″** (63 inches), while the **Weight** value is **121.3 lbs**. Note the height and weight value to fill in the missing data for Golbat in your dataset.

![height and weight value to fill in the missing data for Golbat in your dataset](https://adamtheautomator.com/wp-content/uploads/2021/12/image-147.png)

height and weight value to fill in the missing data for Golbat in your dataset

Now, run the following commands to fill Golbat’s missing data in your dataset.

> _The same set of commands apply to modifying existing values in the dataset_

```python
# Pass in ID number (Golbat's entry number=42)
golbat = data.loc[42]
# Sets the Height Value
golbat["Height (in)"] = 63
# Sets the Weight
golbat["Weight (lbs)"] = 121.30
```

5\. Run the `data.loc[]` command below, where `42` is the entry’s ID number, to list the entry’s data and check any empty values`.`

```python
data.loc[42]
```

Below, you can see that Golbat’s data are all filled in completely.

![Viewing Entry Data ](https://adamtheautomator.com/wp-content/uploads/2021/12/image-148.png)

Viewing Entry Data

> _After filling in the missing data and there is still at least one element missing, you should remove the entire row that’s missing data from the dataset._

6\. Finally, repeat the same steps (three to five) to fill in the missing data for other entries.

## Fixing Formatting Errors

Instead of missing data, another typical scenario in a dataset is formatting errors. Inaccurate records can be a pain, but no worries, you can still fix them up!

Perhaps you have an entry in your data set with words separated by dashes like the one below instead of commas and spaces. If so, running the [`apply()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html) and [`replace()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.replace.html) commands will do the trick.

1\. Run the command below to see how the data looks in your dataframe. Replace the `104` with the entry number of the data with a formatting error.

```powershell
data.loc[104]
```

![Showing Data of Specific Entry](https://adamtheautomator.com/wp-content/uploads/2021/12/image-149.png)

Showing Data of Specific Entry

2 Next, run the command below to replace dashes with commas (`lambda x: x.replace(" -", ",")`) in the data entry’s Weaknesses (`data["Weaknesses"]`) column.

```python
data["Weaknesses"] = data["Weaknesses"].apply(lambda x: x.replace(" -", ","))
```

3\. Rerun the `data.loc[104]` command as you did in step one to check for any dashes in the data.

```python
data.loc[104]
```

As you can see below, the output shows commas now separate the words.

![Replacing dashes with space commas](https://adamtheautomator.com/wp-content/uploads/2021/12/image-150.png)

Replacing dashes with space commas

## Correcting Misspelled Words

Besides formatting errors, misspelled words in a dataset can also make it hard to analyze data. The good news is that you can use some ready-made [spell-checker](https://www.geeksforgeeks.org/spelling-checker-in-python/) Python libraries. But since you already have pandas installed, you don’t have to worry about installing anything else.

1\. Run the following commands to list all unique words [`unique()`](https://pandas.pydata.org/docs/reference/api/pandas.unique.html) in the `Type` column. Replace `Type` if you prefer to list unique words from other columns.

```python
# Turn output into a list of unique words from the Type column
unique_type = list(data["Type"].str.split(", ", expand=True).stack().unique())
# Print out the list
unique_type
```

As you can see below, there are two misspelled words (**Posion** and **Fie**) that should be “Poison” and “Fire.” Now you can go through the dataset, find which rows have misspelled words, and fix them.

![Finding Misspelled Words](https://adamtheautomator.com/wp-content/uploads/2021/12/image-151.png)

Finding Misspelled Words

Run the below command to show all rows that [`contains()`](https://pandas.pydata.org/docs/reference/api/pandas.Series.str.contains.html) the word `Posion` from the `Type` column. The regex argument is set to false (`regex=False`) to treat the string (`Posion`) as a literal string and not a regular expression.

```python
data[data["Type"].str.contains("Posion", regex=False)]
```

In the following output, there are four rows (**Arbok(24)**, **Nidorina(30)**, **Nidoqueen(30)** and **Nidoran(32)**), that have the misspelled word **Posion** in the **Type** column.

![Viewing Misspelled Words](https://adamtheautomator.com/wp-content/uploads/2021/12/image-152.png)

Viewing Misspelled Words

3\. Now, run the commands below to replace `Posion` for all entries in the `Type` column with the word `Poison`.

```python
# Replace Posion with the word Poison
data["Type"] = data["Type"].apply(lambda x: x.replace("Posion", "Poison"))
	# Lists data entries from 0-30
data.head(30)
```

![Replacing Misspelled Word "Posion" with "Poison" ](https://adamtheautomator.com/wp-content/uploads/2021/12/image-153.png)

Replacing Misspelled Word “Posion” with “Poison”

If the replacement is successful, you’ll see you’ve corrected the misspelled words from “Posion” to “Poison” in entry numbers 24 and 30-32.

![Verifying Misspelled Words are Corrected](https://adamtheautomator.com/wp-content/uploads/2021/12/image-154.png)

Verifying Misspelled Words are Corrected

4\. Finally, repeat the steps (two to three) to correct other misspelled words.

## Conclusion

In this tutorial, you’ve learned how to perform data cleaning with Python in many ways for different use cases. You’ve also come to realize that pandas, a popular Python library, is just right around the corner to let you save time cleaning data.

With this newfound knowledge, why not learn more about [handy pandas techniques in Python for data manipulation](https://www.analyticsvidhya.com/blog/2016/01/12-pandas-techniques-python-data-manipulation/)?

Share this article

[Share on X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fadamtheautomator.com%2Fdata-cleaning-python%2F&text=How%20To%20Use%20Data%20Cleaning%20Python%20Tools)[Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fadamtheautomator.com%2Fdata-cleaning-python%2F)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fadamtheautomator.com%2Fdata-cleaning-python%2F)

## Related Posts

![](https://adamtheautomator.com/wp-content/uploads/2026/02/featured_image-3.webp)

### [How to Ace the Modern Coding Interview as a SysAdmin](/ace-modern-coding-interview-sysadmin/)

Master coding interviews for sysadmin and DevOps roles with practical preparation strategies. Learn Python, Bash, and platform-specific techniques that translate your operational experience into interview success.

![](https://adamtheautomator.com/wp-content/uploads/2023/11/pytorch.jpg)

### [How to Install PyTorch on Window](/pytorch/)

Embark on this journey and unleash PyTorch’s potential — your gateway to machine learning and AI exploration, through this ATA Learning tutorial!

![](https://adamtheautomator.com/wp-content/uploads/2023/08/install-python-macos.jpg)

### [Get Started with Programming and Install Python on macOS](/install-python-on-macos/)

Learn how to get started with programming and automation and install Python on macOS in this ATA Learning tutorial!

## Categories

*   [IT Ops](/category/it-ops/)
*   [Cloud](/category/cloud/)
*   [DevOps](/category/devops/)
*   [Home Ops](/category/home-ops/)
*   [Information Security](/category/infosec/)
*   [Software Development](/category/software-development/)

## Site

*   [Home](/)
*   [Tutorials](/tutorials/)
*   [Instructors](/author/)
*   [Advertising](/advertising/)
*   [Recommended Resources](/resources/)
*   [About Adam](/about-adam/)

Copyright 2026© ATA Learning | [Privacy Policy](/privacy/)