CICD Pipeline

Grafana and Prometheus Integration on EC2 instances along with NodeExporter

ADITYA PAWAR's photo
ADITYA PAWAR
·

5 min read

In this article, we are going to have a look at how to integrate Grafana and Prometheus on AWS EC2 instances along with NodeExporter.

What is Grafana?

Grafana is a multi-platform, open-source, analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources.

What is Prometheus?

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud.

What is NodeExporter?

It is an open-source service that installs through a single static binary. Node Exporter monitors a host by exposing its hardware and OS metrics which Prometheus pulls from.

Integration of Grafana and Prometheus on NodeExporter through AWS EC2 instances.

Now to perform the task the first thing we need to do is create three EC2 instances on AWS.

For that log into your AWS Console and go into the EC2 section.

This is your EC2 instance section in AWS. On this page click on launch instances.

Give any name to the instances currently and select OS images as Amazon Linux and the no. of instances as three.

The instance type can be kept as t2.micro and we need to create a new key pair. For that select create new key-pair.

Create the key pair in PPK format.

Now in network settings click on Edit. We will be adding the port numbers in the instances for us to be able to log in the services on the specific port number.

Click on add security group rule.

Add the following port numbers:

  1. 22 for SSH (enabled by default)

  2. 9090 for Prometheus

  3. 9100 for NodeExporter

  4. 3000 for Grafana

And add the source as 0.0.0.0/0 for all port numbers which indicates that the ports can be accessed from any of the IP addresses.

And finally, select on Launch Instance

Now our instances are up and running.

We can now rename the instances below

Now we need to install Grafana, Prometheus and Node Exporter on the mentioned instances.

Firstly we will install Grafana on the Grafana instance. We now need to do SSH into the instances by Putty. If you don't have putty installed in your system you can download it from the official site of putty.

Now copy the Public IPv4 address of the Grafana instance.

Put the Public IPv4 of the instance in the Putty.

And in the Connections -> Credentials put the PPK file which we created while launching the instances

Now click on Open, and then the SSH session will be opened.

We are having OS as Amazon Linux, so we need to login as ec2-user.

Now add the following commands

sudo yum update -y

Then we add a new YUM respository for the operating system to know where to download Grafana.

sudo vim /etc/yum.repos.d/grafana.repo

Add the lines below to grafana.repo. This setting will install to the Open Source version of Grafana

[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

Install Grafana.

sudo yum install grafana

Reload the systemd to load the new settings. Start Grafana Server, then check for its status.

sudo systemctl daemon-reload

Start the Grafana Server.

sudo systemctl start grafana-server

Check the status of the Grafana Server

sudo systemctl status grafana-server

Now your Grafana server is up and Running

To access your Grafana server on the webpage type your IPv4 instance colon 3000.

Type user Id and PWD both as admin.

This screen should be visible to you in the final step of Grafana installation.

Now we will be installing Prometheus on the Prometheus instance.

Login to SSH in the Prometheus server same as we did for the Grafana server.

Add the following commands

sudo useradd --no-create-home prometheus 
sudo mkdir /etc/prometheus 
sudo mkdir /var/lib/prometheus

Now we need to install Prometheus.

wget https://github.com/prometheus/prometheus/releases/download/v2.19.0/prometheus-2.19.0.linux-amd64.tar.gz
tar xvfz prometheus-2.19.0.linux-amd64.tar.gz

sudo cp prometheus-2.19.0.linux-amd64/prometheus /usr/local/bin
sudo cp prometheus-2.19.0.linux-amd64/promtool /usr/local/bin/
sudo cp -r prometheus-2.19.0.linux-amd64/consoles /etc/prometheus
sudo cp -r prometheus-2.19.0.linux-amd64/console_libraries /etc/prometheus

sudo cp prometheus-2.19.0.linux-amd64/promtool /usr/local/bin/
rm -rf prometheus-2.19.0.linux-amd64.tar.gz prometheus-2.19.0.linux-amd64

Now we need to replace the content of the file /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  external_labels:
    monitor: 'prometheus'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['0.0.0.0:9090']

Now in /etc/systemd/system/prometheus.service file add the following content.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Now change the permission of files and directories which we have added to our system

sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
sudo chown -R prometheus:prometheus /var/lib/prometheus

Now configure the systemd

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

Now our Prometheus Server is up and running. You can access the server by typing IPv4:9090 on the web browser.

We can see the Prometheus server running on the web browser

Now we will be installing NodeExporter on the NodeExporter Instance

You need to again SSH into the NodeExporter EC2 instance in the same way as you did for Grafana and Prometheus Server.

Then put the following commands to create a user for Prometheus Node Exporter.

sudo useradd --no-create-home node_exporter

Now we are ready to install Node Exporter binaries.

wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
tar xzf node_exporter-1.0.1.linux-amd64.tar.gz
sudo cp node_exporter-1.0.1.linux-amd64/node_exporter /usr/local/bin/node_exporter
rm -rf node_exporter-1.0.1.linux-amd64.tar.gz node_exporter-1.0.1.linux-amd64

Now configure a service. Create /etc/systemd/system/node-exporter.service if it doesn’t exist.

[Unit]
Description=Prometheus Node Exporter Service
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Now Configure systemd

sudo systemctl daemon-reload
sudo systemctl enable node-exporter
sudo systemctl start node-exporter
sudo systemctl status node-exporter

Then the Node Exporter is finally ready.

Now go into the Prometheus Server and in status select Targets

You will be able to see that the Prometheus server is up and running

Now in Grafana select on adding a data source

Then select Data Source as Prometheus.

Then copy the URL of Prometheus and paste it into the source code URL

Then click on Save and Test. You'll be able to see that the data source is working.

Then click on New dashboard and add a visualization.

Then select any metric and click on apply

Our sample visualisation is ready.

We have now reached the end and we have successfully scrapped metrics from Prometheus and visualized the dashboard in Grafana.

Grafana#prometheusAWS#nodeexporter