--- title: "Utilizing Grafana & Prometheus Kubernetes Cluster Monitoring" description: "Learn how to effectively monitor your Grafana and Prometheus Kubernetes cluster in this tutorial!" canonical: "https://adamtheautomator.com/prometheus-kubernetes/" --- # Utilizing Grafana & Prometheus Kubernetes Cluster Monitoring > Learn how to effectively monitor your Grafana and Prometheus Kubernetes cluster in this tutorial! Source: https://adamtheautomator.com/prometheus-kubernetes/ --- ATA Learning Tap to hide [ ATA Learning ](/) * [Home](/) * [Tutorials](/tutorials/) * [Instructors](/author/) * [Advertising](/advertising/) * [Recommended Resources](/resources/) * [About Adam](/about-adam/) Search for: * [](https://twitter.com/adbertram) * [](https://github.com/Adam-the-Automator) * [](https://www.linkedin.com/company/adam-the-automator-llc) * [](/feed/) ![Utilizing Grafana & Prometheus Kubernetes Cluster Monitoring](https://adamtheautomator.com/wp-content/uploads/2022/04/Getting-Started-with-Grafana-Prometheus-Kubernetes-Cluster-Monitoring-1.jpg) # Utilizing Grafana & Prometheus Kubernetes Cluster Monitoring [![](https://secure.gravatar.com/avatar/ff757e22f03c8a84ce8a2b8f87c53481ba627afec8bd1c8fda5c576097ea027c?s=192&d=mm&r=g)Joseph Eshiett](https://adamtheautomator.com/author/joseph-eshiett/)19 April 20228 min. read Categories: [IT Ops](/category/it-ops/) Tags:[Grafana](/tag/grafana/)[Kubernetes](/tag/kubernetes/)[Linux](/tag/linux/)[Monitoring](/tag/monitoring/)[Ubuntu Linux](/tag/ubuntu-linux/) Table of Contents * [Prerequisites](#prerequisites) * [Deploying the Kube-Prometheus Stack Helm Chart](#deploying-the-kube-prometheus-stack-helm-chart) * [Accessing the Prometheus Instance](#accessing-the-prometheus-instance) * [Viewing Prometheus Kubernetes Cluster Internal State Metrics](#viewing-prometheus-kubernetes-cluster-internal-state-metrics) * [Visualizing a Cluster’s Internal State Metric on Prometheus](#visualizing-a-clusters-internal-state-metric-on-prometheus) * [Accessing then Grafana Dashboard](#accessing-then-grafana-dashboard) * [Interacting with Grafana](#interacting-with-grafana) * [Conclusion](#conclusion) Monitoring your Kubernetes cluster is critical for ensuring that your services are always available and running. And before you scour the internet for a monitoring system, why not try Grafana and Prometheus Kubernetes cluster monitoring? In this guide, you’ll learn how to monitor your Kubernetes cluster, viewing internal state metrics with a [Prometheus](https://prometheus.io/docs/introduction/overview/) and [Grafana](https://grafana.com/grafana/) dashboard. Read on so you can keep a close watch on your resources! ## Prerequisites * A Linux machine with Docker installed — This tutorial uses an [Ubuntu 20.04](https://releases.ubuntu.com/20.04/) LTS machine with [Docker](https://docs.docker.com/engine/install/ubuntu/) version 20.10.7. Here’s how to [install Ubuntu](https://adamtheautomator.com/install-ubuntu/). Related:[How to Install Ubuntu 20.04 \[Step-by-step\]](https://adamtheautomator.com/install-ubuntu/) * A single node Kubernetes Cluster. Related:[Step-by-Step Tutorial: Installing Kubernetes on Ubuntu](https://adamtheautomator.com/installing-kubernetes-on-ubuntu/) * Helm Package Manager installed — For deploying the Prometheus operator. Related:[Getting Started with Kubernetes Helm](https://adamtheautomator.com/kubernetes-helm/) * The [Kubectl Command-line interface](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) is installed and configured to your Cluster. ## Deploying the Kube-Prometheus Stack Helm Chart Grafana and Prometheus Kubernetes Cluster monitoring provides information on potential performance bottlenecks, cluster health, performance metrics. At the same time, visualize network usage, resource usage patterns of pods, and a high-level overview of what is going on in your cluster. But before setting up a monitoring system with Grafana and Prometheus, you’ll first deploy the [kube-prometheus stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) Helm chart. The stack contains Prometheus, Grafana, Alertmanager, Prometheus operator, and other monitoring resources. 1\. SSH into your Ubuntu 20.04 machine _(if you are running on a cloud server)_ or simply log into your locally installed Ubuntu 20.04 machine to begin. 2\. Next, run the [`kubectl create`](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#create) command below to create a `namespace` named `monitoring` for all the Prometheus and Grafana related deployments. ```bash kubectl create namespace monitoring ``` ![Creating a Namespace](https://adamtheautomator.com/wp-content/uploads/2022/04/image-144.png) Creating a Namespace 3\. Run the following [`helm repo`](https://helm.sh/docs/helm/helm_repo/) commands to add the (`prometheus-community`) Helm repo, and update your Helm repo. ```bash # Add prometheus-community repo helm repo add prometheus-community https://prometheus-community.github.io/helm-charts # Update helm repo helm repo update ``` 4\. After adding the Helm repo, run the [`helm install`](https://helm.sh/docs/helm/helm_install/) command below to deploy the kube-prometheus stack Helm chart. Replace `prometheus` with your desired release name. This Helm chart sets up a full Prometheus kubernetes monitoring stack by acting based on a set of Custom Resource Definitions (CRDs). ```bash helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring ``` Once the deployment completes, you’ll get the following output. ![Deploying the kube-prometheus Stack](https://adamtheautomator.com/wp-content/uploads/2022/04/image-145.png) Deploying the kube-prometheus Stack 5\. Finally, run the following command to confirm your kube-prometheus stack deployment. ```bash kubectl get pods -n monitoring ``` The output below shows the deployment of the kube-prometheus stack. As you can see, each component in the stack is running in your cluster. ![Listing Deployed Components from kube-prometheus Stack in monitoring Namespace](https://adamtheautomator.com/wp-content/uploads/2022/04/image-146.png) Listing Deployed Components from kube-prometheus Stack in monitoring Namespace ## Accessing the Prometheus Instance You’ve successfully deployed your Prometheus instance onto your cluster, and you’re almost ready to monitor your Kubernetes cluster. But how do you access your Prometheus instance? You’ll forward a local port `9090` to your cluster via your Prometheus service with the [`kubectl port-forward`](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#port-forward) command. 1\. Run the [`kubectl get`](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get) command below to view all services in the monitoring namespace to check for your Prometheus service. ```bash kubectl get svc -n monitoring ``` All the services deployed in the monitoring namespace are shown below, including the Prometheus service. You’ll use the Prometheus service to set up port-forwarding so your Prometheus instance can be accessible outside of your cluster. ![Listing Deployed Services in the monitoring Namespace](https://adamtheautomator.com/wp-content/uploads/2022/04/image-147.png) Listing Deployed Services in the monitoring Namespace 2\. Next, run the below `kubectl port-forward` command to forward the local port `9090` to your cluster via the Prometheus service (`svc/prometheus-kube-prometheus-prometheus`). ```bash kubectl port-forward svc/prometheus-kube-prometheus-prometheus -n monitoring 9090 ``` But if you’re running a single-node Kubernetes cluster on a cloud server, run the following command instead. ```bash kuebctl port-forward --address 0.0.0.0 svc/prometheus-kube-prometheus-prometheus -n monitoring 9090 ``` > _To run the kubectl port-forward command as a background process, freeing up your terminal for further use, append the & symbol at the end of the command. Follow up by pressing Ctrl+C keys to stop the port-forward foreground process (Doing so will not affect the port-forward background process)._ 3\. Open your favorite web browser, and navigate to either of the URLs below to access your Prometheus instance. * Navigate to [](http://localhost:9090)_[http://localhost:9090](http://localhost:9090)_ if you’re following along with a local Ubuntu machine * Navigate to your server’s IP address followed by port 9090 (i.e., _http://YOUR\_SERVER\_IP:9090)_ if you’re using a cloud server. For this tutorial, Prometheus is running on a cloud server. If your Prometheus service works, you’ll get the following page on your web browser. ![Accessing Prometheus](https://adamtheautomator.com/wp-content/uploads/2022/04/image-148.png) Accessing Prometheus 4\. Lastly, on your terminal, press the Ctrl+C keys to close the port-forwarding process. Doing so makes Prometheus inaccessible on your browser. ## Viewing Prometheus Kubernetes Cluster Internal State Metrics Viewing your Kubernetes cluster’s internal state metrics is made possible with the [Kube-state-metrics (KSM) tool](https://github.com/kubernetes/kube-state-metrics). With the KSM tool you can keep track of the health and usage of your resources, and also internal state objects. Some of the data points that can be potentially viewed via KSM are; node metrics, deployment metrics, and pod metrics. > _The KSM tool comes pre-packaged in the kube-prometheus stack and is deployed automatically with the rest of the monitoring components._ You’ll port-forward a local port to your cluster via the `kube-state-metrics` service. Doing so lets KSM scrape the internal system metrics of your cluster and output a list of queries and values. But before port-forwarding, verify your KSM Kubernetes service first. 1\. Run the below command to check for your `kube-state-metrics` Kubernetes service. ```bash kubectl get svc -n monitoring | grep kube-state-metrics ``` Below, you can see the KSM Kubernetes service name (prometheus-kube-state-metrics) along with the ClusterIP. Note down the KSM Kubernetes service name as you’ll need it to perform the port forwarding in the next step. ![Verifying the KSM Kubernetes Service](https://adamtheautomator.com/wp-content/uploads/2022/04/image-149.png) Verifying the KSM Kubernetes Service 2\. Next, run the below command to `port-forward` the `prometheus-kube-state-metrics` service to port `8080`. ```bash kubectl port-forward svc/prometheus-kube-state-metrics -n monitoring 8080 ``` > _If you are following along with this tutorial with an Ubuntu 20.04 machine hosted by a cloud provider, add the (–address 0.0.0.0) flag to the kubectl port-forward command. Doing so allows external access to the local port via your server’s public IP address._ 3\. Finally, on your web browser, navigate to either of the URLs below to view the Kube Metrics page, as shown below. * Navigate to [](http://localhost:9090)_http://localhost:8080_ if you’re on a local Ubuntu machine * Navigate to your server’s IP address followed by port 8080 (i.e., _http://YOUR\_SERVER\_IP:8080)_ if you’re using a cloud server. Click on the metrics link to access your cluster’s internal state metrics. ![Accessing Kube Metrics](https://adamtheautomator.com/wp-content/uploads/2022/04/image-150.png) Accessing Kube Metrics You can see below a cluster’s internal state metrics similar to yours. ![Listing Cluster Internal State Metrics](https://adamtheautomator.com/wp-content/uploads/2022/04/image-151.png) Listing Cluster Internal State Metrics ## Visualizing a Cluster’s Internal State Metric on Prometheus You’ve successfully performed kube-prometheus stack Helm chart deployment, kube-state-metrics scrape, and Prometheus job configurations. As a result, CoreDNS, kube-api server, Prometheus operator, and other Kubernetes components have been automatically set up as targets on Prometheus. 1\. Navigate to either of the http://localhost:9090/targets or _http:// _Remember to port-forward Prometheus as shown in the “Accessing the Prometheus Instance” section in step two, before navigating to the endpoint. You can as well run it as a background process._ As you can see below, different Kubernetes internal components and monitoring components are configured as targets on Prometheus. ![Viewing Pre-configured Monitoring Components as Prometheus Targets](https://adamtheautomator.com/wp-content/uploads/2022/04/image-152.png) Viewing Pre-configured Monitoring Components as Prometheus Targets 2\. Click on the **Graph** menu to get to a page where you’ll run a [PromQL (Prometheus Query Language)](https://prometheus.io/docs/prometheus/latest/querying/basics/) query. ![](https://adamtheautomator.com/wp-content/uploads/2022/04/image-153.png) Accessing the Graph Page 3\. Insert the sample PromQL _(Prometheus Query Language)_ query below into the expression space provided, then click on **Execute.** The query returns the total amount of unused memory in your Cluster. ```javascript sum((container_memory_usage_bytes{container!="POD",container!=""} - on (namespace,pod,container) avg by (namespace,pod,container)(kube_pod_container_resource_requests{resource="memory"})) * -1 >0 ) / (1024*1024*1024) ``` ![Executing a PromQL Query](https://adamtheautomator.com/wp-content/uploads/2022/04/image-154.png) Executing a PromQL Query 4\. To view the results of the PromQL query executed in step 3 in a graphical format, click on _Graph_. This graph will display the total amount of unused memory in your cluster per given time. With everything set up correctly, the sample cluster metric should look similar to the graph below. ![Graphical view of PromQL query of cluster metric](https://adamtheautomator.com/wp-content/uploads/2022/04/image-155.png) Graphical view of PromQL query of cluster metric ## Accessing then Grafana Dashboard You may have noticed that the visualization capabilities of Prometheus are limited, as you are stuck with only a _Graph_ option. Prometheus is great for scraping metrics from targets configured as jobs, aggregating those metrics, and storing them in a TSDB locally in the Ubuntu machine. But when it comes to standard resource monitoring, Prometheus and Grafana are a great duo. Prometheus aggregates the metrics exported by the server components such as node exporter, CoreDNS, etc. While Grafana, with visualization being its strong suit, receives these metrics from Prometheus and displays them through numerous visualization options. > _During the kube-prometheus stack helm deployment, Grafana had been automatically installed and configured, so you can configure access to Grafana on your Cluster._ To access your Grafana dashboard, you will first need to fetch your username and password stored as secretes automatically created by default in your Kubernetes cluster. 1\. Run the following `kubectl` command to view data stored as `secret` in your Kubernetes cluster (`prometheus-grafana`) in YAML format (`-o yaml`). ```bash kubectl get secret -n monitoring prometheus-grafana -o yaml ``` As you see below, the username and password for accessing your Grafana dashboard are encoded in [base64](https://developer.mozilla.org/en-US/docs/Glossary/Base64). Note down the values of the admin-password and admin-user secrets as you’ll need to decode them in the next step. ![Viewing Secrets (admin-password and admin-user)](https://adamtheautomator.com/wp-content/uploads/2022/04/image-156.png) Viewing Secrets (admin-password and admin-user) 2\. Next, run each command below to `--decode` both secrets (`admin-password` and `admin-user`). Replace `YOUR_USERNAME`, and `YOUR_PASSWORD` with the `admin-password` and `admin-user` secret values you noted in step one. > _This tutorial doesn’t have an output for each command due to security reasons._ ```bash # Decode and print the username echo YOUR_USERNAME | base64 --decode # Decode and print the password echo YOUR_PASSWORD | base64 --decode ``` 3\. Run the `kubectl` command below to `port-forward` to a local port at `3000` by binding the Grafana port `80` to port `3000`. Doing so provides you access to Grafana’s web UI on your browser. ```bash kubectl port-forward svc/prometheus-grafana -n monitoring 3000:80 ``` > _Add the –address 0.0.0.0 flag if you are following along using an Ubuntu 20.04 machine hosted by a cloud provider._ 4\. Finally, on your browser, navigate to any of the endpoints below depending on your machine setup: * [](http://localhost:3000)_http://localhost:3000_ (local) * or _http://:3000_ (cloud) Enter your decoded secret value for admin-user and admin-password in the username and password in the space provided. ![Entering Grafana Username and Password](https://adamtheautomator.com/wp-content/uploads/2022/04/image-157.png) Entering Grafana Username and Password Once you are logged in, you’ll get the Grafana dashboard, as shown below. ![Accessing Grafana Dashboard](https://adamtheautomator.com/wp-content/uploads/2022/04/image-158.png) Accessing Grafana Dashboard ## Interacting with Grafana By default, the Kube-Prometheus stack deploys Grafana with some pre-configured dashboards for each target configured in Prometheus. With these pre-configured dashboards, you will not need to manually set up a dashboard to visualize each metric aggregated by Prometheus. Related:[Get Started With Monitoring in this Grafana Tutorial](https://adamtheautomator.com/grafana-tutorial/) Click on the dashboard icon —> **Browse** and your browser redirects to a page where you’ll see a list of dashboards (step two). ![Accessing the list of Pre-configured Dashboards](https://adamtheautomator.com/wp-content/uploads/2022/04/image-159.png) Accessing the list of Pre-configured Dashboards Click on any of the pre-configured dashboards below to view its visual compute resource. But for this tutorial, click on the **Kubernetes / Compute Resources / Namespace (Pods)** dashboard. ![Viewing a Pre-configured Dashboard](https://adamtheautomator.com/wp-content/uploads/2022/04/image-160.png) Viewing a Pre-configured Dashboard Below is a sample pre-configured dashboard for visualizing compute resource usage by Pods in any of the available namespaces. For this tutorial, the **Data source** has been set to **Prometheus** and the **namespace** for visualization is set to **monitoring**. ![Visualizing Compute Resource Usage](https://adamtheautomator.com/wp-content/uploads/2022/04/image-161.png) Visualizing Compute Resource Usage ## Conclusion In this tutorial, you’ve learned how to deploy the Prometheus operator using Helm and viewed your Cluster internal state metrics to monitor your Kubernetes cluster. You’ve also configured Grafana and viewed your Cluster metrics by configuring your Grafana dashboard. At this point, you already have fully functional Kubernetes cluster monitoring. But you can personally improve on this newfound knowledge, like [configuring Alertmanager with Slack to receive real-time alerts](https://grafana.com/blog/2020/02/25/step-by-step-guide-to-setting-up-prometheus-alertmanager-with-slack-pagerduty-and-gmail/). Share this article [Share on X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fadamtheautomator.com%2Fprometheus-kubernetes%2F&text=Utilizing%20Grafana%20%26%20Prometheus%20Kubernetes%20Cluster%20Monitoring)[Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fadamtheautomator.com%2Fprometheus-kubernetes%2F)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fadamtheautomator.com%2Fprometheus-kubernetes%2F) ## Related Posts ![](https://adamtheautomator.com/wp-content/uploads/2024/03/openldap-4.jpg) ### [How to Install and Configure an OpenLDAP Ubuntu Server](/openldap/) Unlock the power of OpenLDAP on Ubuntu for centralized user authentication, seamless access control management, and enhanced directory services! ![](https://adamtheautomator.com/wp-content/uploads/2023/08/install-go-on-ubuntu-1.jpg) ### [How to Install GO on Ubuntu Linux](/install-go-on-ubuntu/) Learn the ins-and-outs of how to install GO Ubuntu Linux and see how you can take advantage of this popular programming language! ![](https://adamtheautomator.com/wp-content/uploads/2023/07/update-the-kernel-in-ubuntu.jpg) ### [Learn how to Update the Kernel In Ubuntu Linux Safely](/update-the-kernel-in-ubuntu/) In this ATA Learning tutorial learn the ins-and-outs to update the kernel in Ubuntu Linux safely through several different methods! ## Categories * [IT Ops](/category/it-ops/) * [Cloud](/category/cloud/) * [DevOps](/category/devops/) * [Home Ops](/category/home-ops/) * [Information Security](/category/infosec/) * [Software Development](/category/software-development/) ## Site * [Home](/) * [Tutorials](/tutorials/) * [Instructors](/author/) * [Advertising](/advertising/) * [Recommended Resources](/resources/) * [About Adam](/about-adam/) Copyright 2026© ATA Learning | [Privacy Policy](/privacy/)