---
title: "Learn Event Streaming With This Apache Kafka Tutorial"
description: "Learn how to install Apache Kafka to manage and stream event data like a pro in this step-by-step tutorial by ATA Learning!"
canonical: "https://adamtheautomator.com/apache-kafka/"
---

# Learn Event Streaming With This Apache Kafka Tutorial

> Learn how to install Apache Kafka to manage and stream event data like a pro in this step-by-step tutorial by ATA Learning!

Source: https://adamtheautomator.com/apache-kafka/

---

ATA Learning

Tap to hide

[

ATA Learning

](/)

*   [Home](/)
*   [Tutorials](/tutorials/)
*   [Instructors](/author/)
*   [Advertising](/advertising/)
*   [Recommended Resources](/resources/)
*   [About Adam](/about-adam/)

Search for:  

*   [](https://twitter.com/adbertram)
*   [](https://github.com/Adam-the-Automator)
*   [](https://www.linkedin.com/company/adam-the-automator-llc)
*   [](/feed/)

![Learn Event Streaming With This Apache Kafka Tutorial](https://adamtheautomator.com/wp-content/uploads/2022/02/Learn-Event-Streaming-With-This-Apache-Kafka-Tutorial.jpg)

# Learn Event Streaming With This Apache Kafka Tutorial

[![](https://secure.gravatar.com/avatar/2788bb1a3f735603f81eca51d68daec56a9d97e805a10268fb2c20afcc76b81b?s=192&d=mm&r=g)Nicholas Xuan Nguyen](https://adamtheautomator.com/author/nicholas-xuan-nguyen/)24 February 20227 min. read

Categories: [IT Ops](/category/it-ops/)

Tags:[Apache Kafka](/tag/apache-kafka/)[Linux](/tag/linux/)

Table of Contents

*   [Prerequisites](#prerequisites)
*   [Installing Apache Kafka](#installing-apache-kafka)
*   [Configuring the Apache Kafka Server](#configuring-the-apache-kafka-server)
*   [Restricting the Kafka User](#restricting-the-kafka-user)
*   [Conclusion](#conclusion)

Need a streaming platform to handle large amounts of data? You’ve undoubtedly heard about Apache Kafka on Linux. Apache Kafka is perfect for real-time data processing, and it’s becoming more and more popular. Installing Apache Kafka on Linux can be a bit tricky, but no worries, this tutorial has got you covered.

In this tutorial, you’ll learn to install and configure Apache Kafka, so you can start processing your data like a pro, making your business more efficient and productive.

Read on and start streaming data with Apache Kafka today!

## Prerequisites

This tutorial will be a hands-on demonstration. If you’d like to follow along, be sure you have the following.

*   A Linux machine – This demo uses Debian 10, but any Linux distribution will work.
*   A non-root user account with [sudo privileges](https://www.liquidweb.com/kb/how-to-set-up-and-manage-sudo-permissions/), necessary to run Kafka, and named `kafka` in this tutorial.
*   A dedicated sudo user for Kafka – This tutorial uses a sudo user called kafka.
*   [Java](https://opensource.com/article/19/11/install-java-linux) – Java is an integral part of Apache Kafka installation.
*   [Git](https://www.atlassian.com/git/tutorials/install-git#linux) – This tutorial uses Git for downloading the Apache Kafka Unit files.

## **Installing Apache Kafka**

Before streaming data, you’ll first have to install Apache Kafka on your machine. Since you have a dedicated account for Kafka, you can install Kafka without worrying about breaking your system.

1\. Run the `mkdir` command below to create the _/home/kafka/Downloads_ directory. You can name the directory as you prefer, but the directory is called _Downloads_ for this demo. This directory will store the Kafka binaries. This action ensures that all your files for Kafka are available to the `kafka` user.

```bash
mkdir Downloads
```

2\. Next, run the below [`apt update`](https://linuxize.com/post/how-to-use-apt-command/#updating-package-index-apt-update) command to update your system’s package index.

```bash
sudo apt update -y
```

Enter the password for your kafka user when prompted.

![Updating your apt Package Manager](https://adamtheautomator.com/wp-content/uploads/2022/02/image-311.png)

Updating your apt Package Manager

3\. Run the `curl` command below to download Kafka binaries from the Apache Foundation website to output (`-o`) to a binary file (`kafka.tgz`) in your `~/Downloads` directory. You will use this binary file to install Kafka.

Be sure to replace kafka/3.1.0/kafka\_2.13-3.1.0.tgz with the latest version of [Kafka binaries](https://kafka.apache.org/downloads). As of this writing, the current Kafka version is 3.1.0.

```bash
curl "https://dlcdn.apache.org/kafka/3.1.0/kafka_2.13-3.1.0.tgz" -o ~/Downloads/kafka.tgz
```

![Downloading Kafka binaries](https://adamtheautomator.com/wp-content/uploads/2022/02/image-312.png)

Downloading Kafka binaries

Related:[How to Download Files with Python Wget (A Curl Alternative)](https://adamtheautomator.com/python-wget/)

4\. Now, run the [`tar`](https://www.geeksforgeeks.org/tar-command-linux-examples) command below to extract (`-x`) the Kafka binaries (`~/Downloads/kafka.tgz`) into the automatically created _kafka_ directory. The options in the `tar` command perform the following:

The options in the `tar` command perform the following:

*   `-v` – Tells the `tar` command to list all files as they get extracted.
    
*   `-z` – Tells the `tar` command to gzip the archive as it’s being uncompressed. This behavior is not required in this case but is an excellent option, especially if you need a quick compressed/zipped file to move around.
    
*   `-f` – Tells the `tar` command which archive file to extract.
    
*   `-strip 1` -Instructs the `tar` command to strip the first level of directories from your file name list. As a result, automatically create a subdirectory named _kafka_ containing all of the extracted files from the `~/Downloads/kafka.tgz` file.
    

```bash
tar -xvzf ~/Downloads/kafka.tgz --strip 1
```

![Extracting the kafka.tgz Binary File](https://adamtheautomator.com/wp-content/uploads/2022/02/image-313.png)

Extracting the kafka.tgz Binary File

## Configuring the Apache Kafka Server

At this point, you have downloaded and installed the Kafka binaries to your _~/Downloads_ directory. You can’t use the Kafka server just yet since, by default, Kafka does not allow you to delete or modify any topics, a category necessary to organize log messages.

To configure your Kafka server, you will have to edit the Kafka configuration file (_/etc/kafka/server.properties)._

1\. Open the Kafka configuration file (_/etc/kafka/server.properties_) in your preferred text editor.

2\. Next, add the **delete.topic.enable = true** line at the bottom of the _/kafka/config/server.properties_ file content, save the changes and close the editor.

This configuration property gives you permission to delete or modify topics, so ensure you know what you are doing before deleting topics. Deleting a topic deletes partitions for that topic as well. Any data stored in those partitions are no longer accessible once they are gone.

> _Be sure there are no spaces at the beginning of each line, or else the file will not be recognized, and your Kafka server will not work._

![Configuring your Kafka Server](https://adamtheautomator.com/wp-content/uploads/2022/02/image-314.png)

Configuring your Kafka Server

3\. Run the `git` command below to `clone` the `ata-kafka` project to your local machine so that you can modify it for use as a unit file for your Kafka service.

```bash
sudo git clone https://github.com/Adam-the-Automator/apache-kafka.git
```

![cloning the ata-kafka project](https://adamtheautomator.com/wp-content/uploads/2022/02/image-315.png)

cloning the ata-kafka project

Now, run the below commands to move into the `apache-kafka` directory and list the files inside.

```bash
cd apache-kafka
ls
```

Now that you are in the _ata-kafka_ directory, you can see that you have two files inside: kafka.service and zookeeper.service, as shown below.

![Viewing the ata-kafka directory](https://adamtheautomator.com/wp-content/uploads/2022/02/image-316.png)

Viewing the ata-kafka directory

5\. Open the _zookeeper.service_ file in your preferred text editor. You’ll use this file as a reference to create the _kafka.service_ file.

Customize each section below in the _zookeeper.service_ file, as needed. But this demo uses this file as is, without modifications.

*   The `[Unit]` section configures the startup properties for this unit. This section tells the systemd what to use when starting the zookeeper service.
    
*   The \[Service\] section defines how, when, and where to start the Kafka service using the _kafka-server-start.sh_ script. This section also defines basic information such as name, description, and command-line arguments (what follows ExecStart=).
    
*   The `[Install]` section sets the [runlevel](https://www.landoflinux.com/linux_runlevels_systemd.html) to start the service when entering multi-user mode.
    

![Viewing the zookeeper.service file](https://adamtheautomator.com/wp-content/uploads/2022/02/image-317.png)

Viewing the _zookeeper.service_ file

6\. Open the _kafka.service_ file in your preferred text editor, and configure how your Kafka server looks when running as a systemd service.

This demo uses the default values that are in the _kafka.service_ file, but you can customize the file as needed. Note that this file is referring to the _zookeeper.service_ file, which you might modify at some point.

![Viewing the kafka.service File](https://adamtheautomator.com/wp-content/uploads/2022/02/image-318.png)

Viewing the _kafka.service_ File

7\. Run the below command to `start` the `kafka` service.

```bash
sudo systemctl start kafka
```

> _Remember to stop and start your Kafka server as a service. If you don’t, the process will remain in memory, and you can only stop the process by killing it. This behavior can lead to data loss if you have topics that are being written or updated as the process shuts down._

Since you’ve created _kafka.service_ and _zookeeper.service_ files, you can also run either of the commands below to stop or restart your systemd-based Kafka server.

```bash
sudo systemctl stop kafka
sudo systemctl restart kafka
```

Related:[Controlling Systemd services with Ubuntu systemctl](https://adamtheautomator.com/ubuntu-systemctl/)

8\. Now, run the [`journalctl`](https://man7.org/linux/man-pages/man1/journalctl.1.html) command below to verify that the service has started up successfully.

This command lists all of the logs for the kafka service.

```bash
sudo journalctl -u kafka
```

If you’ve configured everything correctly, you’ll see a message that says Started kafka.service, as shown below. Congratulations! You now have a fully-functional Kafka server that will run as systemd services.

![Verifying the Kafka Service Started up Successfully](https://adamtheautomator.com/wp-content/uploads/2022/02/image-319.png)

Verifying the Kafka Service Started up Successfully

## **Restricting the Kafka User**

At this point, the Kafka Service runs as the kafka user. The kafka user is a system-level user and should not be exposed to users who connect to Kafka.

Any client who connects to Kafka through this broker will effectively have root-level access on the broker machine, which is not recommended. To mitigate the risk, you’ll remove the kafka user from the sudoers file and disable the password for the kafka user.

1\. Run the `exit` command below to switch back to your normal user account.

```bash
exit
```

![Switching Back to your Normal User Account](https://adamtheautomator.com/wp-content/uploads/2022/02/image-320.png)

Switching Back to your Normal User Account

2\. Next, run the `sudo deluser kafka sudo` and press **Enter** to confirm that you want to remove the `kafka` user from sudoers.

```bash
sudo deluser kafka sudo
```

![Removing the Kafka user from sudoers](https://adamtheautomator.com/wp-content/uploads/2022/02/image-321.png)

Removing the Kafka user from sudoers

3\. Run the below command to disable the password for the kafka user. Doing so further improves the security of your Kafka installation.

```bash
sudo passwd kafka -l
```

![Disabling the password for the kafka user](https://adamtheautomator.com/wp-content/uploads/2022/02/image-322.png)

Disabling the password for the kafka user

4\. Now, rerun the following command to remove the kafka user from the sudoers list.

```javascript
sudo deluser kafka sudo
```

![Removing the kafka user from the sudoers list](https://adamtheautomator.com/wp-content/uploads/2022/02/image-323.png)

Removing the kafka user from the sudoers list

5\. Run the below [`su`](https://www.howtogeek.com/111479/htg-explains-whats-the-difference-between-sudo-su) command to set only authorized users like root users can run commands as the `kafka` user.

```bash
sudo su - kafka
```

![Setting root users to run commands as the kafka user](https://adamtheautomator.com/wp-content/uploads/2022/02/image-324.png)

Setting root users to run commands as the kafka user

6\. Next, run the below command to create a new [Kafka topic](https://dattell.com/data-architecture-blog/what-is-a-kafka-topic/) named `ATA` to verify that your Kafka server is running correctly.

Kafka topics are feeds of messages to/from the server, which helps eliminate the complications of having messy and unorganized data in the Kafka Servers

```bash
cd /usr/local/kafka-server && bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic ATA
```

![Creating a new Kafka Topic (ATA)](https://adamtheautomator.com/wp-content/uploads/2022/02/image-325.png)

Creating a new Kafka Topic (ATA)

7\. Run the below command to create a [Kafka producer](https://docs.confluent.io/platform/current/clients/producer.html) using the `kafka-console-producer.sh` script. Kafka producers write data to topics.

```bash
echo "Hello World, this sample provided by ATA" | bin/kafka-console-producer.sh --broker-list localhost:9092 --topic ATA > /dev/null
```

![Creating a Kafka Producer](https://adamtheautomator.com/wp-content/uploads/2022/02/image-326.png)

Creating a Kafka Producer

8\. Finally, run the below command to create a [kafka consumer](https://www.oreilly.com/library/view/kafka-the-definitive/9781491936153/ch04.html) using the `kafka-console-consumer.sh` script. This command consumes all of the messages in the kafka topic (`--topic ATA`) and then prints out the message value.

```bash
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic ATA --from-beginning
```

You’ll see the message in the output below because your messages are printed by the Kafka console consumer from the ATA Kafka topic, as shown below. The consumer script continues to run at this point, waiting for more messages.

You can open another terminal to add more messages to your topic and press Ctrl+C to stop the consumer script once you are done testing.

![Testing your kafka server](https://adamtheautomator.com/wp-content/uploads/2022/02/image-327.png)

Testing your kafka server

## Conclusion

Throughout this tutorial, you’ve learned to set up and configure Apache Kafka on your machine. You’ve also touched on consuming messages from a Kafka topic produced by the Kafka producer, resulting in effective event log management.

Now, why not build on this newfound knowledge by installing [Kafka with Flume](https://towardsdatascience.com/apache-kafka-and-flume-installation-guide-import-data-from-kafka-to-hdfs-c908b0df034c) to better distribute and manage your messages? You can also explore [Kafka’s Streams API](https://dzone.com/articles/kafka-streams-more-than-just-dumb-storage) and build applications that read and write data to Kafka. Doing so transforms data as needed before writing it out to another system like [HDFS](https://community.cloudera.com/t5/Support-Questions/Streaming-data-from-Kafka-to-HDFS-All-relevant-solutions/td-p/312063), [HBase](http://onurtokat.com/spark-streaming-from-kafka-to-hbase-use-case/), or [Elasticsearch](https://www.confluent.io/blog/kafka-elasticsearch-connector-tutorial/).

Share this article

[Share on X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fadamtheautomator.com%2Fapache-kafka%2F&text=Learn%20Event%20Streaming%20With%20This%20Apache%20Kafka%20Tutorial)[Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fadamtheautomator.com%2Fapache-kafka%2F)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fadamtheautomator.com%2Fapache-kafka%2F)

## Related Posts

![](https://adamtheautomator.com/wp-content/uploads/2022/09/Practical-Linux-Unix-Tee-Commands-for-the-Linux-Admin.jpg)

### [Master Unix tee Commands for Real-World Linux Admin Tasks](/unix-tee/)

Simplify Linux output management and streamline your workflow with the “Unix tee” command. This tutorial guides you through its practical applications.

![](https://adamtheautomator.com/wp-content/uploads/2024/03/openldap-4.jpg)

### [How to Install and Configure an OpenLDAP Ubuntu Server](/openldap/)

Unlock the power of OpenLDAP on Ubuntu for centralized user authentication, seamless access control management, and enhanced directory services!

![](https://adamtheautomator.com/wp-content/uploads/2024/03/pipe-command-in-linux-1.jpg)

### [Unleashing the Power of the Pipe Command in Linux](/pipe-command-in-linux/)

Unlock the prowess of the pipe command in Linux to streamline tasks, boost productivity, and simplify complex operations effortlessly!

## Categories

*   [IT Ops](/category/it-ops/)
*   [Cloud](/category/cloud/)
*   [DevOps](/category/devops/)
*   [Home Ops](/category/home-ops/)
*   [Information Security](/category/infosec/)
*   [Software Development](/category/software-development/)

## Site

*   [Home](/)
*   [Tutorials](/tutorials/)
*   [Instructors](/author/)
*   [Advertising](/advertising/)
*   [Recommended Resources](/resources/)
*   [About Adam](/about-adam/)

Copyright 2026© ATA Learning | [Privacy Policy](/privacy/)