|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
Alias: ["Prometheus"]
|
|
|
|
|
Tag: ["Computer", "Server", "Monitoring"]
|
|
|
|
|
Date: 2022-03-19
|
|
|
|
|
DocType: "Personal"
|
|
|
|
|
Hierarchy: "NonRoot"
|
|
|
|
|
TimeStamp:
|
|
|
|
|
location: [47.3639129,8.55627491017841]
|
|
|
|
|
CollapseMetaTable: Yes
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
Parent:: [[Selfhosting]], [[Configuring Caddy|caddy]], [[Server Tools]]
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
```button
|
|
|
|
|
name Save
|
|
|
|
|
type command
|
|
|
|
|
action Save current file
|
|
|
|
|
id Save
|
|
|
|
|
```
|
|
|
|
|
^button-ConfiguringPrometheusNSave
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
# Configuring Prometheus
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
```ad-abstract
|
|
|
|
|
title: Summary
|
|
|
|
|
collapse: open
|
|
|
|
|
This not runs through the installation and use of Prometheus as a monitoring tool.
|
|
|
|
|
Prometheus interacts better with json logs rather than common log language, which is caddy's output.
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
```toc
|
|
|
|
|
style: number
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
### Introduction
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
[Prometheus](https://prometheus.io/docs/introduction/overview/) is a free and open-source monitoring and alerting tool that was initially used for monitoring metrics at SoundCloud back in 2012. It is written in Go programming language.
|
|
|
|
|
|
|
|
|
|
Prometheus monitors and records real-time events in a time-series database. Since then it has grown in leaps and bounds and had been adopted by many organizations to monitor their infrastructure metrics. Prometheus provides flexible queries and real-time alerting which helps in quick diagnosis and troubleshooting of errors.
|
|
|
|
|
|
|
|
|
|
Prometheus comprises the following major components:
|
|
|
|
|
|
|
|
|
|
- The main Prometheus server for scraping and storing time-series data.
|
|
|
|
|
- Unique exporters for services such as Graphite, HAProxy, StatsD and so much more
|
|
|
|
|
- An alert manager for handling alerts
|
|
|
|
|
- A push-gateway for supporting transient jobs
|
|
|
|
|
- Client libraries for instrumenting application code
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
### Installing Prometheus
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
#### Installing the main modules
|
|
|
|
|
|
|
|
|
|
But first, we need to create the configuration and data directories for Prometheus.
|
|
|
|
|
|
|
|
|
|
To create the configuration directory, run the command:
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
sudo mkdir -p /etc/prometheus
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
For the data directory, execute:
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
sudo mkdir -p /var/lib/prometheus
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
Once the directories are created, grab the compressed installation file:
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
wget https://github.com/prometheus/prometheus/releases/download/v2.31.0/prometheus-2.31.0.linux-amd64.tar.gz
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
Once downloaded, extract the tarball file.
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
tar -xvf prometheus-2.31.3.linux-amd64.tar.gz
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
Then navigate to the Prometheus folder.
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
cd prometheus-2.31.3.linux-amd64
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
Once in the [directory move](https://linoxide.com/mv-command-in-linux/) the `prometheus` and `promtool` binary files to `/usr/local/bin/` folder.
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
sudo mv prometheus promtool /usr/local/bin/
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
Additionally, move console files in `console` directory and library files in the `console_libraries` directory to `/etc/prometheus/` directory.
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
sudo mv consoles/ console_libraries/ /etc/prometheus/
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
Also, ensure to move the prometheus.yml template configuration file to the **`/etc/prometheus/`** directory.
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
sudo mv prometheus.yml /etc/prometheus/prometheus.yml
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
At this point, Prometheus has been successfully installed. To check the version of Prometheus installed, run the command:
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
prometheus --version
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
Output:
|
|
|
|
|
|
|
|
|
|
```ad-code
|
|
|
|
|
~~~bash
|
|
|
|
|
prometheus, version 2.31.3 (branch: HEAD, revision: f29caccc42557f6a8ec30ea9b3c8c089391bd5df)
|
|
|
|
|
build user: root@5cff4265f0e3
|
|
|
|
|
build date: 20211005-16:10:52
|
|
|
|
|
go version: go1.17.1
|
|
|
|
|
platform: linux/amd64
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
promtool --version
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
Output:
|
|
|
|
|
|
|
|
|
|
```ad-code
|
|
|
|
|
~~~bash
|
|
|
|
|
promtool, version 2.31.3 (branch: HEAD, revision: f29caccc42557f6a8ec30ea9b3c8c089391bd5df)
|
|
|
|
|
build user: root@5cff4265f0e3
|
|
|
|
|
build date: 20211005-16:10:52
|
|
|
|
|
go version: go1.17.1
|
|
|
|
|
platform: linux/amd64
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
If your output resembles what I have, then you are on the right track. In the next step, we will create a system group and user.
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
#### Permissions & User Management
|
|
|
|
|
|
|
|
|
|
It's essential that we create a Prometheus group and user before proceeding to the next step which involves creating a system file for Prometheus.
|
|
|
|
|
|
|
|
|
|
To create a `prometheus` [group](https://linoxide.com/groupadd-command/) execute the command:
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
sudo groupadd --system prometheus
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
Thereafter, Create `prometheus` user and assign it to the just-created `prometheus` group.
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
sudo useradd -s /sbin/nologin --system -g prometheus prometheus
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
Next, configure the directory ownership and permissions as follows.
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
sudo chown -R prometheus:prometheus /etc/prometheus/ /var/lib/prometheus/$ sudo chmod -R 775 /etc/prometheus/ /var/lib/prometheus/
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The only part remaining is to make Prometheus a systemd service so that we can easily manage its running status.
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
#### Configuring the service
|
|
|
|
|
|
|
|
|
|
Using your favorite text editor, create a systemd service file:
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
sudo nano /etc/systemd/system/prometheus.service
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
Paste the following lines of code.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```ad-code
|
|
|
|
|
~~~bash
|
|
|
|
|
[Unit]
|
|
|
|
|
Description=Prometheus
|
|
|
|
|
Wants=network-online.target
|
|
|
|
|
After=network-online.target
|
|
|
|
|
|
|
|
|
|
[Service]
|
|
|
|
|
User=prometheus
|
|
|
|
|
Group=prometheus
|
|
|
|
|
Restart=always
|
|
|
|
|
Type=simple
|
|
|
|
|
ExecStart=/usr/local/bin/prometheus \
|
|
|
|
|
--config.file=/etc/prometheus/prometheus.yml \
|
|
|
|
|
--storage.tsdb.path=/var/lib/prometheus/ \
|
|
|
|
|
--web.console.templates=/etc/prometheus/consoles \
|
|
|
|
|
--web.console.libraries=/etc/prometheus/console_libraries \
|
|
|
|
|
--web.listen-address=0.0.0.0:9090
|
|
|
|
|
|
|
|
|
|
[Install]
|
|
|
|
|
WantedBy=multi-user.target
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Save the changes and exit the systemd file.
|
|
|
|
|
|
|
|
|
|
Then proceed and start the Prometheus service.
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
sudo systemctl start prometheus
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
Enable the Prometheus service to run at startup. Therefore invoke the command:
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
sudo systemctl enable prometheus
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
Then confirm the status of the Prometheus service.
|
|
|
|
|
|
|
|
|
|
```ad-command
|
|
|
|
|
~~~bash
|
|
|
|
|
sudo systemctl status prometheus
|
|
|
|
|
~~~
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
![Check status of Prometheus services](https://linoxide.com/wp-content/uploads/2021/11/2021-10-1003-Check-status-of-Prometheus-services.png)![Check status of Prometheus services](https://linoxide.com/wp-content/uploads/2021/11/2021-10-1003-Check-status-of-Prometheus-services.png)
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
#### Configuration of user acccess
|
|
|
|
|
|
|
|
|
|
Finally, to access Prometheus, parameter your reverse-proxy ([[Configuring Caddy|caddy]]) to point back to the service.
|
|
|
|
|
|
|
|
|
|
It is accessible below, under internal port 9090:
|
|
|
|
|
|
|
|
|
|
```ad-address
|
|
|
|
|
https://prometheus.mfxm.fr
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
![prometheus dashboard](https://linoxide.com/wp-content/uploads/2021/11/2021-10-1003-Prometheus-dashboard-1024x440.png)![prometheus dashboard](https://linoxide.com/wp-content/uploads/2021/11/2021-10-1003-Prometheus-dashboard-1024x440.png)
|
|
|
|
|
|
|
|
|
|
 
|
|
|
|
|
 
|