---
title: "Deploying an EFK Stack with Docker"
description: "Learn how to deploy the EFK stack, one of the most popular software stacks for log analysis and monitoring, in this step-by-step tutorial!"
canonical: "https://adamtheautomator.com/efk-stack/"
---
# Deploying an EFK Stack with Docker
> Learn how to deploy the EFK stack, one of the most popular software stacks for log analysis and monitoring, in this step-by-step tutorial!
Source: https://adamtheautomator.com/efk-stack/
---
ATA Learning
Tap to hide
[
ATA Learning
](/)
* [Home](/)
* [Tutorials](/tutorials/)
* [Instructors](/author/)
* [Advertising](/advertising/)
* [Recommended Resources](/resources/)
* [About Adam](/about-adam/)
Search for:
* [](https://twitter.com/adbertram)
* [](https://github.com/Adam-the-Automator)
* [](https://www.linkedin.com/company/adam-the-automator-llc)
* [](/feed/)
# Deploying an EFK Stack with Docker
[Arvid Larson](https://adamtheautomator.com/author/arvid-larson/)12 April 20228 min. read
Categories: [IT Ops](/category/it-ops/)
Tags:[Docker](/tag/docker/)[Monitoring](/tag/monitoring/)
Table of Contents
* [Prerequisites](#prerequisites)
* [Setting up an EFK Stack Project](#setting-up-an-efk-stack-project)
* [Deploying EFK Stack with Docker](#deploying-efk-stack-with-docker)
* [Configuring Kibana Index Pattern](#configuring-kibana-index-pattern)
* [Running a Docker Container with Fluentd Log Driver](#running-a-docker-container-with-fluentd-log-driver)
* [Conclusion](#conclusion)
Log monitoring and analysis have become critical nowadays, whether for applications or server/container infrastructure. One of the most popular software stacks for log analysis and monitoring you can choose is Elasticsearch, Fluentd, and Kibana (EFK stack).
EFK stack is a distributed and scalable search engine that allows structured search and analytics. And in this tutorial, you’ll learn how to set up EFK stack log monitoring with Docker and centralize container logs to the EFK stack.
Ready? Read on and ease up your log analysis!
## Prerequisites
This tutorial comprises hands-on demonstrations. To follow along, ensure you have the following:
* A Linux host – This example uses the Debian 11 Bullseye server with a memory capacity of 6GB.
Related:[How to Install Ubuntu 20.04 \[Step-by-Step\]](https://adamtheautomator.com/install-ubuntu/)
* [A Docker CE (Community Edition)](https://computingforgeeks.com/install-docker-ce-on-linux-systems/) and [Docker Compose installed](https://adamtheautomator.com/docker-compose-tutorial/#h-installing-docker-compose) on your Linux host.
Related:[How to Install and Use Docker on Ubuntu (In the Real World)](https://adamtheautomator.com/docker-ubuntu/)
## Setting up an EFK Stack Project
EFK Stack is an enterprise-ready log aggregation and logs analysis framework for bare-metal and container infrastructure. But before deploying an EFK stack, you’ll first set up a project directory and create a Docker configuration for deploying EFK Stack on your Docker host.
For this example, you’ll use Docker images with the following specs:
* Elasticsearch 7.17.0 – Capable of storing data with fast lightning Apache Lucene-based search capabilities
* Kibana 7.17.0 – Open-source data aggregation and collector that supports JSON data, and
* Fluentd Custom image based on v1.14.1 – Data visualization software for Elasticsearch.
Related:[How to Update Your Docker Images to the Latest Version](https://adamtheautomator.com/update-docker/)
To set up your EFK stack project:
1\. Open a terminal and log in to your server.
2\. Run the below commands to verify both the Docker and Docker Compose are installed on your system.
```bash
# Checking Docker version
docker --version
# Checking docker-compose version
docker-compose version
```
As you see below, the installed versions of Docker CE (Community Edition) v20.10.12 and the Docker Compose v1.29.2.
Checking Docker and Docker Compose Version
3\. Run the following command to create a new project directory (`mkdir`) and set it as the working directory (`cd`).
You can name the directory as you prefer, but in this tutorial, the directory is named efk. This directory will store all of the EFK Stack configuration files in this tutorial.
```bash
mkdir -p ~/efk; cd ~/efk
```
4\. Now, create a new configuration file (`docker-compose.yml`) using your preferred editor and populate the following configurations.
The configuration below uses the Docker Compose script v3 and defines all EFK stack containers.
```yaml
version: "3"
# Define the Docker volume named esdata for the Elasticsearch container.
volumes:
esdata:
# Deploying three container services (fluentd, elasticsearch, and kibana)
services:
# Deploy using the custom image automatically be created during the build process.
fluentd:
build: ./fluentd
links: # Sends incoming logs to the elasticsearch container.
- elasticsearch
depends_on:
- elasticsearch
ports: # Exposes the port 24224 on both TCP and UDP protocol for log aggregation
- 24224:24224
- 24224:24224/udp
# Created using the Docker image elasticsearch:7.17.0
elasticsearch:
image: elasticsearch:7.17.0
expose: # Exposes the default port 9200
- 9200
environment:
- discovery.type=single-node # Runs as a single-node
volumes: # Stores elasticsearch data locally on the esdata Docker volume
- esdata:/usr/share/elasticsearch/data
# Created using the Docker image kibana:7.17.0
kibana:
image: kibana:7.17.0
links: # Links kibana service to the elasticsearch container
- elasticsearch
depends_on:
- elasticsearch
ports: # Runs kibana service on default port 5601
- 5601:5601
environment: # Defined host configuration
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
```
5\. Run the below command to create a new directory `fluentd` and navigate to that directory. The `fluentd` directory will store `fluentd` service configurations.
```bash
mkdir -p fluentd/; cd fluentd/
```
6\. Inside the _~/efk/fluentd_ directory, create a new _Dockerfile_ using your preferred editor and populate the following configuration.
This configuration creates the fluentd custom image containing the elasticsearch client driver and the fluentd-plugin-elasticsearch.
Ensure to use the same version between elasticsearch and elasticsearch client driver — this tutorial uses version 7.17.0.
```docker
# image based on fluentd v1.14-1
FROM fluentd:v1.14-1
# Use root account to use apk
USER root
# below RUN includes plugin as examples elasticsearch is not required# you may customize including plugins as you wish
RUN apk add --no-cache --update --virtual .build-deps \
sudo build-base ruby-dev \
&& gem uninstall -I elasticsearch \
&& gem install elasticsearch -v 7.17.0 \
&& sudo gem install fluent-plugin-elasticsearch \
&& sudo gem sources --clear-all \
&& apk del .build-deps \
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
# copy fluentd configuration from host image
COPY ./conf/fluent.conf /fluentd/etc/
# copy binary start file
COPY entrypoint.sh /bin/
RUN chmod +x /bin/entrypoint.sh
USER fluent
```
7\. Next, create another configuration file (`entrypoint.sh`) using your preferred editor and populate the following configuration. This script executes when the `fluentd` container service starts.
Below is the starter script for the fluentd container service, which executes the basic command fluentd –config /fluentd/etc/fluentd.conf –plugin /etc/fluentd/plugins.
```bash
#!/bin/sh
#source vars if file exists
DEFAULT=/etc/default/fluentd
if [ -r $DEFAULT ]; then
set -o allexport
. $DEFAULT
set +o allexport
fi
# If the user has supplied only arguments append them to `fluentd` commandif [ "${1#-}" != "$1" ]; then
set -- fluentd "$@"
fi
# If user does not supply config file or plugins, use the defaultif [ "$1" = "fluentd" ]; then
if ! echo $@ | grep -e ' \-c' -e ' \-\-config' ; then
set -- "$@" --config /fluentd/etc/${FLUENTD_CONF}
fi
if ! echo $@ | grep -e ' \-p' -e ' \-\-plugin' ; then
set -- "$@" --plugin /fluentd/plugins
fi
fi
exec "$@"
```
8\. Run the below command to create a new directory `conf` under the _~/efk/fluentd_ directory.
```bash
mkdir -p conf
```
9\. Now, create a fluentd configuration (_conf/fluent.conf_) using your preferred editor and populate the following configuration.
This configuration allows the fluentd container service to receive log messages, and forward them to the elasticsearch container service.
```powershell
# bind fluentd on IP 0.0.0.0
# port 24224
@type forward
port 24224
bind 0.0.0.0
# sendlog to the elasticsearch
# the host must match to the elasticsearch
# container service
@type copy
@type elasticsearch
host elasticsearch
port 9200
logstash_format true
logstash_prefix fluentd
logstash_dateformat %Y%m%d
include_tag_key true
type_name access_log
tag_key @log_name
flush_interval 300s
@type stdout
```
10\. Lastly, run the below commands to check the structure of the EFK Stack project directory.
> _If you don’t have the tree command, install it using the following command: apt install tree -y_
```bash
# Checking list of files and directory
ls
# Checking directory structure
tree
```
Below is the complete structure of the EFK Stack project directory.
Viewing EFK Stack Project Directory Structure
## Deploying EFK Stack with Docker
You’ve now created all configuration files for deploying EFK Stack using Docker and Docker Compose. The next step is to deploy the EFK Stack using the [`docker-compose`](https://docs.docker.com/engine/reference/commandline/compose/) command, and the deployment will happen in your project directory (_~/efk_).
1\. First, run the below command to change the working directory to the `efk` project directory.
```bash
cd ~/efk/
```
2\. Next, run the `docker-compose` command below to deploy (`up`) the EFK Stack log analysis and log monitoring system.
This command automatically downloads Docker images Elasticsearch and Kibana. And the Fluentd Docker image automatically builds using the _Dockerfile_ in the _fluentd_ directory.
Deployment may take some time, depending on the specs of the Docker host.
```bash
docker-compose up -d
```
Below is the screenshot of the build process for the Fluentd Docker image.
Deploying Process of Building Docker Image fluentd
And below is the screenshot showing the deployment is complete, and the Kibana container service is running.
EFK Stack deployment is completed
3\. Run each command below to check logs of the EFK stack build process. Always run these commands whenever you get an error in the deployment process.
```bash
# Checking logs for service fluentd
docker-compose logs fluentd
# Checking logs for service kibana
docker-compose logs kibana
```
Below is the log message when the fluentd container service is running.
Checking log messages from fluentd service
And below is the log for the kibana container.
Checking log messages from the kibana service
4\. Now, run the below command to check all container services’ status (`ps`).
```bash
docker-compose ps
```
As you see below, the EFK Stack container service is Up. Note down the elasticsearch container name (efk\_elasticsearch\_1) to further verify the container is properly running in the next step.
Checking EFK Stack Container Services
5\. Additionally, run the below command to verify the elasticsearch container service. This command prints the detailed settings of the `efk_elasticsearch_1` container.
```bash
docker inspect efk_elasticsearch_1
```
As you can see below, the efk\_elasticsearch\_1 container gets an IP address of 172.18.0.2.
Checking Network Settings of the elasticsearch service
6\. Lastly, run the below command to access and verify the elasticsearch container by IP address (`172.18.0.2`). Port `9200` is the default port for the elasticsearch container.
```bash
curl 172.18.0.2:9200
```
You’ll see an output similar to the one below if the elasticsearch container on your machine is running.
Checking elasticsearch Service Status
## Configuring Kibana Index Pattern
Now that you’ve completed deploying the EFK Stack in the Docker environment, you’ll open Kibana from your web browser. You’ll set up an index pattern for log monitoring and analysis.
1\. Open your favorite web browser and navigate to the server IP address followed by the Kibana service port 5601 (i.e., [http://172.16.1.10:5601](http://172.16.1.10:5601)).
2\. Next, click the **Explore on my own** button on the welcome page below.
Elastic Kibana Welcome Page
3\. Click the **Stack Management** option to set up the Kibana index pattern in the Management section.
Accessing the Stack Management
4\. On the **Kibana** left menu section, click menu **Index Patterns** and click the **Create Index Pattern** button to create a new index pattern.
Creating New Index Pattern
5\. Now, input the index pattern **Name** as **fluentd-**\*, set the **Timestamp field** to **@timestamp**, and click the **Create index pattern** button to confirm the index pattern settings.
On the right side, you can see available index patterns from the fluentd such as fluentd-%Y%m%d. The %Y%m%d date format is based on the fluentd configuration (_fluentd.conf_).
Creating an Index Pattern for fluentd
6\. Lastly, click on the top left menu (ellipsis), then click the **Discover** menu to show the logs monitoring.
Accessing the Logs Monitoring
Below is the screenshot of the Kibana log monitoring and analysis dashboard. All listed logs are taken from the Elasticsearch and shipped by the Fluentd log aggregation.
Showing Kibana Dashboard for Log Monitoring
## Running a Docker Container with Fluentd Log Driver
After configuring the Kibana index pattern, you’ll run a Docker container with Fluentd log drive, automatically sending logs to the EFK stack.
1\. Run the below command to download the NGINX image. The `alpine` version is smaller than normal images based on Ubuntu, CentOS, or Fedora.
```bash
docker pull nginx:alpine
```
Downloading nginx:alpine Docker Image
Related:[Troubleshooting Docker Permission Denied Problems](https://adamtheautomator.com/docker-permission-denied/)
2\. Next, run the below command to start a new NGINX container (`nginx_container`) in detached mode (`-d`).
The command also sets the log drive to Fluentd (–log-driver=fluentd) and exposes port 8080 on the Docker host machine for the container (nginx\_container).
```bash
docker run --name nginx_container -d --log-driver=fluentd -p 8080:80 nginx:alpine
```
Running a New Container (nginx\_container)
3\. After running the container, run the `docker` command below to check all running containers.
```bash
docker ps
```
You should see that the nginx\_container is Up and running on the host port 8080.
Running nginx\_container and checking list containers
4\. Now, run the below command to access the `nginx_container` and generate access logs.
```bash
curl localhost:8080
```
Accessing nginx\_container
Alternatively, open a new tab on your web browser and type the server IP address followed by port 8080 (i.e., [http://172.168.1.10:8080](http://172.168.1.10:8080/)).
If all goes well, you’ll see the default _index.html_ page from the nginx\_container.
Accessing nginx\_container via a Web Browser
5\. Lastly, switch back to the Kibana dashboard, and click the **Discover** menu on the left side.
Click the container\_name : nginx\_container query on the KQL (Kibana Query Language) field, and you’ll see logs from the nginx\_container, as shown below.
Showing logs of the nginx\_container in the Kibana dashboard
## Conclusion
You’ve learned how to deploy EFK Stack (Elasticsearch, Fluentd, and Kibana) throughout this tutorial for log monitoring and analysis using Docker. You’ve also learned how to set up logging for the Docker container using the Fluentd log driver. And at this point, you now have a fully functional log monitoring for applications and services.
For the next stage, you may be interested in using [KQL](https://www.elastic.co/guide/en/kibana/current/kuery-query.html) (Kibana Query Language) to [visuali](https://www.elastic.co/guide/en/kibana/current/dashboard.html)[ze](https://www.elastic.co/guide/en/kibana/current/dashboard.html) log monitoring and analysis.
Share this article
[Share on X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fadamtheautomator.com%2Fefk-stack%2F&text=Deploying%20an%20EFK%20Stack%20with%20Docker)[Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fadamtheautomator.com%2Fefk-stack%2F)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fadamtheautomator.com%2Fefk-stack%2F)
## Related Posts
### [Building OCI Images with Buildah](/buildah/)
Master the art of crafting OCI images with buildah for seamless containerization—flexible, efficient, and tailored to your container needs.
### [How to Install Docker on Raspberry Pi 4](/docker-on-raspberry-pi/)
Excerpt: Learn how to get started with installing Docker on Raspberry Pi 4 and offload development to a small flexible system.
### [Discover Proactive Linux Monitoring with Pandora FMS](/pandora-fms/)
Learn about how Pandora FMS is a powerful Linux monitoring tool that helps admins stay ahead of problems, identify trends, and optimize system performance.
## Categories
* [IT Ops](/category/it-ops/)
* [Cloud](/category/cloud/)
* [DevOps](/category/devops/)
* [Home Ops](/category/home-ops/)
* [Information Security](/category/infosec/)
* [Software Development](/category/software-development/)
## Site
* [Home](/)
* [Tutorials](/tutorials/)
* [Instructors](/author/)
* [Advertising](/advertising/)
* [Recommended Resources](/resources/)
* [About Adam](/about-adam/)
Copyright 2026© ATA Learning | [Privacy Policy](/privacy/)