In this blog, we will see how to monitor multiple AWS MSK Clusters in a single machine without creating a separate job or machines

Problem Outline

You can monitor multiple AWS MSK Clusters in a single machine by creating multiple jobs for each and every cluster or defining multiple brokers in a single job. In a single job, you can not identify data that belongs to which cluster. In multiple jobs each and every time you have to specify a job name while you access data which is a tedious process. Also, data will be stored in a single Prometheus so you will not be able to segregate data and performance completely.

Is it possible to achieve the same without creating separate jobs or individual machines for each and every Prometheus? Hmmm! 🤔 Yes, it is possible. 🤩

In this blog, we will connect and monitor our 3 AWS MSK clusters named as Mercury, Venus and Mars with Grafana on the same machine.

Requirements

1. Linux Machine
2. AWS MSK Cluster (cluster should be accessible from the same machine)
3. Grafana

1) Create users

We will create new user accounts for each and every cluster. These new user accounts will help us to maintain the layer between Prometheus and the system so that it does not affect our system. Also, this user account will not be able to login into the machine.

sudo useradd --no-create-home --shell /bin/false mercury
sudo useradd --no-create-home --shell /bin/false venus
sudo useradd --no-create-home --shell /bin/false mars

2) Create directories for clusters

We will create separate directories for storing Prometheus files and data using below commands,

sudo mkdir /etc/mercury
sudo mkdir /var/lib/mercury

sudo mkdir /etc/venus
sudo mkdir /var/lib/venus

sudo mkdir /etc/mars
sudo mkdir /var/lib/mars

After creating directories, we have to assign appropriate user and group ownership to created users.

sudo chown mercury:mercury /etc/mercury
sudo chown mercury:mercury /var/lib/mercury

sudo chown venus:venus /etc/venus
sudo chown venus:venus /var/lib/venus

sudo chown mars:mars /etc/mars
sudo chown mars:mars /var/lib/mars

Once the ownership is assigned, we can download Prometheus and move it to the appropriate directory to isolate each and every Prometheus.

3) Download and Configure Prometheus

sudo yum update -y

curl -LO url -LO https://github.com/prometheus/prometheus/releases/download/v2.23.0/prometheus-2.23.0.linux-amd64.tar.gz

tar -xvf prometheus-2.23.0.linux-amd64.tar.gz

Note: Here we have used Prometheus version 2.23.0 but you can check and download the latest version from the official website

Once the file is successfully downloaded, extract the file and copy Prometheus and Promtool binary to bin directory using following commands

sudo mkdir /usr/local/bin/mercury
sudo cp prometheus-2.23.0.linux-amd64/prometheus /usr/local/bin/mercury/
sudo cp prometheus-2.23.0.linux-amd64/promtool /usr/local/bin/mercury/

sudo mkdir /usr/local/bin/venus
sudo cp prometheus-2.23.0.linux-amd64/prometheus /usr/local/bin/venus/
sudo cp prometheus-2.23.0.linux-amd64/promtool /usr/local/bin/venus/

sudo mkdir /usr/local/bin/mars
sudo cp prometheus-2.23.0.linux-amd64/prometheus /usr/local/bin/mars/
sudo cp prometheus-2.23.0.linux-amd64/promtool /usr/local/bin/mars/

Once, the binary files are successfully moved, we have to change the ownership of binary files

sudo chown mercury:mercury /usr/local/bin/mercury/prometheus
sudo chown mercury:mercury /usr/local/bin/mercury/promtool

sudo chown venus:venus /usr/local/bin/venus/prometheus
sudo chown venus:venus /usr/local/bin/venus/promtool

sudo chown mars:mars /usr/local/bin/mars/prometheus
sudo chown mars:mars /usr/local/bin/mars/promtool

Now, copy the console and console_libraries to etc folder

sudo cp -r prometheus-2.23.0.linux-amd64/consoles /etc/mercury/prometheus
sudo cp -r prometheus-2.23.0.linux-amd64/console_libraries /etc/mercury/prometheus

sudo cp -r prometheus-2.23.0.linux-amd64/consoles /etc/venus/prometheus
sudo cp -r prometheus-2.23.0.linux-amd64/console_libraries /etc/venus/prometheus

sudo cp -r prometheus-2.23.0.linux-amd64/consoles /etc/mars/prometheus
sudo cp -r prometheus-2.23.0.linux-amd64/console_libraries /etc/venus/prometheus

Once, the libraries files are successfully moved, change the ownership of binary files using following commands

sudo chown -R mercury:mercury /etc/mercury/prometheus/consoles
sudo chown -R mercury:mercury /etc/mercury/prometheus/console_libraries

sudo chown -R venus:venus /etc/venus/prometheus/consoles
sudo chown -R venus:venus /etc/venus/prometheus/console_libraries

sudo chown -R mars:mars /etc/mars/prometheus/consoles
sudo chown -R mars:mars /etc/mars/prometheus/console_libraries

We have downloaded and moved Prometheus successfully.

4) Configure Prometheus

Now, we have to create a Prometheus configuration file which we will write in YAML format. We will create three service files for each and every cluster. Execute the below command to create a configuration file

sudo vi /etc/mercury/prometheus/prometheus.yml

and paste the below content

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  scrape_timeout: 15s
  # scrape_timeout is set to the global default (10s).

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'broker'
    #metrics_path: /node_exporter/metrics
    scheme: http
    file_sd_configs:
    - files:
      - './targets.json'

Now, we have to create targets.json file which will contain our broker url of AWS MSK cluster.

Note: if you don’t know how to get a broker url, refer this.

sudo vi /etc/mercury/prometheus/targets.json

Paste the below content and change targets to your broker url.

[
  {
    "labels": {
      "job": "jmx"
    },
    "targets": [
      "B-1.stage-msk.xxx.c3.kafka.ap-south-1.amazonaws.com:11001",
      "b-2.stage-msk.xxx.c3.kafka.ap-south-1.amazonaws.com:11001"
     ]
  },
  {
    "labels": {
      "job": "node"
    },
    "targets": [
      "b-1.kafka-stage-multilane.xxx.c3.kafka.ap-south-1.amazonaws.com:11002",
      "b-2.kafka-stage-multilane.xxx.c3.kafka.ap-south-1.amazonaws.com:11002"
     ]
  }
]

We have created a prometheus.yml file and targets.json file for the mercury cluster, you have to follow the same steps for venus and mars. Also, change the static_configs section in the prometheus.yml file as mentioned below.

for mercury,

static_configs:
    - targets: ['localhost:9090']

in venus,

static_configs:
    - targets: ['localhost:9091']

for mars:

static_configs:
    - targets: ['localhost:9092']

5) Create service file

sudo vi /etc/systemd/system/mercury_prometheus.service

and paste the following content

[Unit]
Description=MercuryPrometheus
Wants=network-online.target
After=network-online.target

[Service]
User=mercury
Group=mercury
Type=simple
ExecStart=/usr/local/bin/mercury/prometheus \
    --config.file /etc/mercury/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/mercury/prometheus/ \
    --web.console.templates=/etc/mercury/prometheus/consoles \
    --web.console.libraries=/etc/mercury/prometheus/console_libraries \
    --web.listen-address=:9090

[Install]
WantedBy=multi-user.target

Create the same for venus and mars.

For Venus,

sudo vi /etc/systemd/system/venus_prometheus.service

and paste the following content

[Unit]
Description=VenusPrometheus
Wants=network-online.target
After=network-online.target

[Service]
User=venus
Group=venus
Type=simple
ExecStart=/usr/local/bin/venus/prometheus \
    --config.file /etc/venus/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/venus/prometheus/ \
    --web.console.templates=/etc/venus/prometheus/consoles \
    --web.console.libraries=/etc/venus/prometheus/console_libraries \
    --web.listen-address=:9091

[Install]
WantedBy=multi-user.target

For Mars,

sudo vi /etc/systemd/system/mars_prometheus.service

and paste the following content

[Unit]
Description=MarsPrometheus
Wants=network-online.target
After=network-online.target

[Service]
User=mars
Group=mars
Type=simple
ExecStart=/usr/local/bin/mars/prometheus \
    --config.file /etc/mars/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/mars/prometheus/ \
    --web.console.templates=/etc/mars/prometheus/consoles \
    --web.console.libraries=/etc/mars/prometheus/console_libraries \
    --web.listen-address=:9092

[Install]
WantedBy=multi-user.target

Once three service files are successfully created. Now, we have to reload systemd.

sudo systemctl daemon-reload

Now, we can start our created services using the below commands,

sudo systemctl start mercury_prometheus
sudo systemctl enable mercury_prometheus

sudo systemctl start venus_prometheus
sudo systemctl enable venus_prometheus

sudo systemctl start mars_prometheus
sudo systemctl enable mars_prometheus

Once the services are started, you can check the status of the services using the below command and make sure that your services are in running state.

sudo systemctl status mercury_prometheus
sudo systemctl status venus_prometheus
sudo systemctl status mars_prometheus

You can access the mercury prometheus console on http://localhost:9090, venus prometheus console on http://localhost:9091 and mars prometheus console on http://localhost:9092.

Congratulations 🎉 Now, you can monitor multiple AWS MSK clusters in a single machine.