Prometheus + InfluxDB + Grafana (Docker on ARM64)

Date created: Sunday, October 17, 2021 5:44:02 PM. Last modified: Sunday, May 19, 2024 7:29:18 PM

Prometheus + InfluxDB + Grafana (Docker on ARM64)

Running Prometheus (and exporters) and Grafana in Docker on Ubuntu 20.04 on ARM64 (ODOIRD-N2).

Docker Install and Test

# Install Docker
wget https://get.docker.com -O docker-install.sh && chmod a+x docker-install.sh
$ ./docker-install.sh

$ sudo systemctl enable docker

$ sudo usermod -aG docker bensley

# Run the test container (this will download the hello-world container from Docker hub)
$ sudo docker run hello-world

# With IPv6 only connectivity, if the hello-world container is missing (it will be on a fresh install),
# then docker will fail to download it, because the Docker hub is IPv4 only.
# IPv6 isn't natively supported on Docker hub yet (still in beta), so one must;
# Register for a free account at: https://hub.docker.com
# Run: $ sudo docker login registry.ipv6.docker.com
# Login with the new free account details
# Manually pull any images whilst specifying the IPv6 hub address:
# e.g. $ sudo docker pull registry.ipv6.docker.com/library/hello-world:latest
# or $ sudo docker pull registry.ipv6.docker.com/docker/foobar:latest
# Not that the /library URL is for offical docker images
# Then run the container using:
# $ sudo docker run registry.ipv6.docker.com/library/hello-world
# In any Dockerfile, instead of using: FROM ubuntu:20.04
# One must now use: FROM registry.ipv6.docker.com/library/ubuntu:20.04

$ sudo docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f6553123d598 hello-world "/hello" 12 minutes ago Exited (0) 12 minutes ago wizardly_varahamihira

$ sudo docker container rm f6553123d598
f6553123d598

$ sudo docker image rm 18e5af790473
Untagged: hello-world:latest
Untagged: hello-world@sha256:37a0b92b08d4919615c3ee023f7ddb068d12b8387475d64c622ac30f45c29c51
Deleted: sha256:18e5af7904737ba5ef7fbbd7d59de5ebe6c4437907bd7fc436bf9b3ef3149ea9
Deleted: sha256:64df1d35ad6c0c754bb2fb894dc41c8d22497dec795ee030971903774ad1c00d

# Try an ARM64 Ubuntu container
$ sudo docker image pull ubuntu:21.10

$ sudo docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
ubuntu 21.10 f99cc3098237 38 hours ago 69.9MB

$ sudo docker run -it --rm ubuntu:21.10

# Ensure docker and containerd are set to start on system boot:
$ sudo systemctl enable docker.service
$ sudo systemctl enable containerd.service

Install Prometheus Container

# Pull Prometheus container
$ sudo docker pull prom/prometheus

# Create a directory which will store the Prometheus config and create a base config.
$ sudo mkdir /opt/prometheus
$ sudo chown bensley:bensley /opt/prometheus
$ cd /opt/prometheus
$ vi prometheus.yml

# Example starter config which allows Prometheus to monitor itself:
global:
  scrape_interval: 15s # Default: 1m
scrape_configs:
  # This is the local Prometheus instance...
  - job_name: 'prometheus'
    #metrics_path: "/metrics"
    #scheme: "http"
    scrape_interval: 5s # Overide global default
    static_configs:
      - targets: ['localhost:9090']

# Start the container in interactive mode with self-deletion after exit, and bind-mount this config file to test it's working
$ docker run -it --rm -p 9090:9090 -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
# Prometheus will now have a primitive web interface running on port 9090, it exposes local stats at http://localhost:9090/metrics

# To start in daemon mode with a name use:
$ sudo docker run -d -p 9090:9090 --restart always -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml --name prometheus prom/prometheus

Updating Prometheus Config

When updates are made to prometheus.yml, Prometheus can be asked to reload it's config in one of two ways;

# 1. Send SIGHUP signal to Prometheus docker
$ docker ps | grep prometheus
71cd0226d49b prom/prometheus "/bin/prometheus --c…" 2 days ago Up 42 hours 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp prometheus

$ sudo docker kill --signal="SIGHUP" 71cd0226d49b


# 2. Post to the reload API. Note: This must have been enabled when the docker container was first started (--web.enable-lifecycle) - it is disabled by default for security reasons.
# When adding a new CLI Arg to the docker run command, all the defaults are whipped out and need to be specified too:

$ sudo docker run -d -p 9090:9090 --restart always -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml --name prometheus prom/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --web.console.libraries=/usr/share/prometheus/console_libraries --web.console.templates=/usr/share/prometheus/consoles --web.enable-lifecycle

# Then one can POST to the reload API endpoint using curl;
$ curl -X POST http://localhost:9090/-/reload

Install Grafana Container

# Pull Grafana container
$ sudo docker pull grafana/grafana
# Test it in interactive mode with self-deletion after exit
$ sudo docker run -it --rm -p 3000:3000 grafana/grafana
# The main Grafana web interface will now be available at http://localhost:3000

# To start in daemon mode with a name use:
$ sudo docker run -d -p 3000:3000 --restart always --name grafana grafana/grafana

# Then one can start/stop by name
$ sudo docker stop grafana
$ sudo docker start grafana

Install and Config InfluxDB as Container

# UPDATE: Influxdb v2.x doesn't work with Promtheus anymore due ot Influx API updates.
# The latest v1.x InfluxDB must be used instead (v 1.8 at the time of writing).

##################################################
# v1.8 SETUP
# Grab latest v1.x version
docker pull influxdb:1.8

# Create mount point for persistent database storage and to store config file on host
$ sudo mkdir /opt/influxdb
$ sudo chown bensley:bensley /opt/influxdb/
$ cd $_

# Build the default config file:
$ docker run --rm influxdb:1.8 influxd config > influxdb.conf

# Start the container bind-mounting the config file and volume-mounting the persistent storage, on the host
# Influx v1.8 listens on two TCP ports:
# 8086 is for client-server communication using the InfluxDB API.
# 8088 is for the RPC service to perform back up and restore operations.

# Run a test:
$ docker run -it --rm -p 8086:8086 -p 8088:8088 --name influxdb --volume /opt/influxdb/:/var/lib/influxdb -v /opt/influxdb/influxdb.conf:/etc/influxdb/influxdb.conf influxdb:1.8

# If all is OK, then run as a daemon:
$ docker run -d -p 8086:8086 -p 8088:8088 --restart always --name influxdb --volume /opt/influxdb/:/var/lib/influxdb -v /opt/influxdb/influxdb.conf:/etc/influxdb/influxdb.conf influxdb:1.8

# Connect to the InfluxDB instance and create a new database for Prometheus.
# To do this - start bash in the container then use the influx CLI tool (which connects to localhost:8086 by default).
$ docker exec -it influxdb bash
$ influx -precision rfc3339
# Only the default _internal database should exist initially:
> show databases;
# Create new DB called "prometheus" and check
> CREATE DATABASE prometheus;
> show databases;

# Optionally, one can create a retention policy because the default policy has an infinite retention period:
> use prometheus
> show RETENTION POLICIES
name    duration shardGroupDuration replicaN default
----    -------- ------------------ -------- -------
autogen 0s       168h0m0s           1        true
> ALTER RETENTION POLICY "autogen" on "prometheus" DURATION 26w
> show RETENTION POLICIES
name    duration  shardGroupDuration replicaN default
----    --------  ------------------ -------- -------
autogen 4368h0m0s 168h0m0s           1        true

# Add the InfluxDB API endpoints to to the Prometheus config file
$ vi /opt/prometheus/prometheus.yml

remote_write:
  - url: "http://192.168.58.6:8086/api/v1/prom/write?db=prometheus"

remote_read:
  - url: "http://192.168.58.6:8086/api/v1/prom/read?db=prometheus"

# Signal Prometheus to reload it's config
$ docker kill --signal="SIGHUP" prometheus
##################################################

##################################################
# v2.0.9 SETUP - Doesn't work with Prometheus!!!!!
# Get latest docker image (v2.0.9 at the time of writing)
# $ docker pull influxdb:latest

# Create mount point for persistent data storage and to store config file on host
$ sudo mkdir /opt/influxdb
$ sudo chown bensley:bensley /opt/influxdb/
$ cd $_

# Build the default config file:
$ docker run --rm influxdb:1.8 influxd print-config > config.yml

# Start the container bind-mounting the config file, volume-mounting persistent data storage on the host, and using the Influx CLI ARG --reporting-disabled to stop Influx from sending telemetry back to Influx Corp.

$ docker run -d -p 8086:8086 -p 8088:8088 --restart always --name influxdb --volume /opt/influxdb/:/var/lib/influxdb2 -v /opt/influxdb/config.yml:/etc/influxdb2/config.yml influxdb:1.8 --reporting-disabled

# Web GUI for initial setup is now available at http://localhost:8086
# Don't need to use Influx CLI for InfluxDB 2.x
# In the Influx GUI create API token via web GUI for Prometheus.
# Test with:
$ curl --get "http://localhost:8086/api/v2" \
  --header "Authorization: Token YOUR_API_TOKEN" \
  --header 'Content-type: application/json' \
  --data-urlencode "db=mydb" \
  --data-urlencode "q=SELECT * FROM cpu_usage"
##################################################

Install and Configure snmp_exporter as Container

# Ubuntu 20.04...
$ sudo apt-get install golang git make unzip build-essential libsnmp-dev p7zip-full

# This system only has go 1.12 in the repo - need to manually install go1.14 to meet snmp_exporter minimum requirements
$ wget https://golang.org/dl/go1.14.15.linux-arm64.tar.gz
$ tar -xvf go1.14.15.linux-arm64.tar.gz
$ sudo mv go /opt/go1.14.15
$ PATH=/opt/go1.14.15/bin/:$PATH

# Normal install resumes here..
$ git clone https://github.com/prometheus/snmp_exporter.git
$ cd snmp_exporter
$ make

# Comment out the line in the SNMP config which prevents the loading of mibs.
# Commenting this out will enabling the loading of MIBs by SNMP libraries so that we can get the OID names:
sudo vi /etc/snmp/snmp.conf

# Use the generator to compile SNMP MIBs into a config file for snmp_exporter
cd generator/
mkdir ~/.snmp/mibs/
./generator generate
./generator parse_errors
cp ./snmp.yml ../
cd ../

# Now start snmp_exporter using this generated config file
$ ./snmp_exporter

# A HTTP endpoint is now available http://localhost:9116/snmp
# When combined with the "module" and "target" arugment, snmp_exporter will
# poll the specific OIDs under "module" in the config file, against "target"
# http://localhost:9116/snmp?module=cisco_ios&target=192.0.2.1

# Set up and run snmp_exporter as a docker container
$ vi Dockerfile
# Change the Dockerfile ARCH to be "arm64"
# Docker wouldn't build until the COPY line for the snmp_exporter binary was changed to:
# COPY ./snmp_exporter  /bin/snmp_exporter
$ docker build -t snmp_exporter-0.20.0 .

$ sudo mkdir /opt/snmp_exporter
$ sudo chown bensley:bensley /opt/snmp_exporter
$ cp snmp.yml /opt/snmp_exporter/

# Quick test:
$ docker run -it --rm -p 9116:9116 -v /opt/snmp_exporter/snmp.yml:/etc/snmp_exporter/snmp.yml --name snmp_exporter snmp_exporter-0.20.0
$ curl "http://localhost:9116/snmp?module=cisco_ios&target=192.0.2.1"
# Now run a daemon
$ docker run -d -p 9116:9116 --restart always -v /opt/snmp_exporter/snmp.yml:/etc/snmp_exporter/snmp.yml --name snmp_exporter snmp_exporter-0.20.0

# Add snmp_exporter to Prometheus conconfig
sudo vi /opt/prometheus/prometheus.yml
# Add the following:
  - job_name: 'ios_cpe'
    static_configs:
      - targets:
        - 192.168.58.1 # Cisco CPE
    metrics_path: /snmp
    params:
      module: [cisco_ios]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.58.6:9116  # The SNMP exporter's real hostname:port.

# Reload Prometheus config
$ curl -X POST http://localhost:9090/-/reload

Install and Configure blackbox_exporter as Container

# Prereq:
# Pre-req: One needs to enabled IPv6 for Docker if it isn't already (assuming IPv6 probes are wanted)!
# Example (create this file if it doesn't exist then restart docker):
#$ cat /etc/docker/daemon.json
#{
#    "default-address-pools": [
#        {"base":"172.16.0.0/16","size":24}
#    ],
#    "ipv6": true,
#    "fixed-cidr-v6": "2001:db8:1::/64"
#}
# This creates a routed subnet with the ::1 address on docker0 bridge interface.
# Or bind the docker container to the host's IPv4/IPv6 address(es) using
# docker --network host


# Install latest version at the time of writing:
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.19.0/blackbox_exporter-0.19.0.linux-arm64.tar.gz
tar -xvf blackbox_exporter-0.19.0.linux-arm64.tar.gz
cd blackbox_exporter-0.19.0.linux-arm64

# All IPv4 ICMP tests will fail by default - a special capability is required to use ICMP.
# We can add it to the binary for testing:
$ sudo setcap cap_net_raw+ep blackbox_exporter

# Create a config, example below:
vi blackbox.yml

modules:
  icmpv4:
    timeout: 3s
    prober: icmp
    icmp:
      preferred_ip_protocol: "ip4"
      ip_protocol_fallback: false
      dont_fragment: true

# Check config and test blackbox_exporter:
$ ./blackbox_exporter --config.check
$ ./blackbox_exporter
# Web GUI is now available at http://localhost:9115
# Test a ping using:
$ curl http://localhost:9115/probe?target=192.0.2.1\&module=icmpv4

# Now build a docker. We can add the ICMP capability to the docker container too using:
# --cap-add CAP_NET_RAW
# This requires that the container be run as root though!
$ wget https://raw.githubusercontent.com/prometheus/blackbox_exporter/master/Dockerfile
# Edit Dockerfile ARCH to "arm64"
# Change binary COPY line to:
# COPY ./blackbox_exporter /bin/blackbox_exporter
vi Dockerfile
$ docker build -t blackbox_exporter-0.19.0 .

$ sudo mkdir /opt/blackbox_exporter/
$ sudo chown bensley:bensley /opt/blackbox_exporter
$ cp blackbox.yml /opt/blackbox_exporter/

$ sudo docker run -d --network host -p 9115:9115 --restart always --cap-add CAP_NET_RAW -v /opt/blackbox_exporter/blackbox.yml:/etc/blackbox_exporter/config.yml --name blackbox_exporter blackbox_exporter-0.19.0 --config.file=/etc/blackbox_exporter/config.yml

# Add the targets to Prometheus:
$ vi /opt/prometheus/prometheus.yml

  - job_name: 'blackbox-icmpv4'
    metrics_path: /probe
    params:
      module: [icmpv4]
    static_configs:
      - targets:
        - 192.168.0.1 # gateway
        - e.root-servers.net
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115  # The blackbox exporter's hostname:port

# Restart Prometheus to load new config
$ curl -X POST http://localhost:9090/-/reload

Install and Configure node_exporter as Container

# Download latest version at time of writing
$ wget https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-arm64.tar.gz
$ tar -xzvf node_exporter-1.2.2.linux-arm64.tar.gz
$ cd node_exporter-1.2.2.linux-arm64

# Create a Dockerfile because one isn't included with node_exporter:
vi ./Dockerfile
ARG ARCH="arm64" # "armv7" for RaspPi
ARG OS="linux"

FROM ubuntu:20.04
COPY ./node_exporter  /bin/node_exporter

EXPOSE      9100
ENTRYPOINT  [ "/bin/node_exporter" ]
CMD         [ "--no-collector.arp", "--no-collector.bcache", "--no-collector.btrfs", "--no-collector.entropy", "--no-collector.fibrechannel", "--no-collector.infiniband", "--no-collector.ipvs", "--no-collector.nfs", "--no-collector.nfsd", "--no-collector.schedstat", "--no-collector.tapestats", "--no-collector.uname", "--no-collector.xfs", "--no-collector.zfs", "--no-collector.cpufreq", "--no-collector.netstat", "--no-collector.sockstat", "--no-collector.timex", "--no-collector.vmstat" ]

$ docker build -t node_exporter-1.2.2 .
# Test the container:
$ docker run -it --rm -p 9100:9100 --name node_exporter node_exporter-1.2.2

# If OK, run as daemon (node_exporter is design to run as an unprivileged, run as root only if you need to monitor something not available to an unprivileged user):
$ docker run -d -p 9100:9100 --restart always --name node_exporter node_exporter-1.2.2

# Optionally, one can expose the host file system to the container as read-only, to expose more stats.
# Add the following to the CMD list in the Dockerfile and build the container again:
"--path.rootfs=/hostfs",
# Then create a RO bind mount between host and container using the docker command:
$ docker run -d -p 9100:9100 -v "/:/hostfs:ro,rslave" --restart always --name node_exporter node_exporter-1.2.2


# HTTP endpoint is now available at http://localhost/metrics
# Enable in Prometheus:
$ vi /opt/prometheus/prometheus.yml

  - job_name: 'node_exporter'
    metrics_path: /metrics
    static_configs:
      - targets: ['192.168.58.6:9100']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.58.6:9100  # The blackbox exporter's hostname:port

Install and Configure pihole_exporter as Container

This can run on the Docker server and locally polled by Prometheus. The PiHole needs a web API token. This is automatically created if the admin password was created when PiHole was set up. If no admin password was set, first set one using "$ pihole -a -p", then get the derived API token using "$ grep WEBPASSWORD /etc/pihole/setupVars.conf".

# Don't pull the latest version from Docker, the latest is for amd64 only, pull the most recent release which supports arm64
$ docker pull ekofr/pihole-exporter:v0.0.11

# Give it a test run.
# Parse environment variables to the container to specify the details of the PiHole server.
$ docker run -it --rm -e "PIHOLE_HOSTNAME=192.168.0.1" -e "PIHOLE_API_TOKEN=abc123" -e "INTERVAL=60s" -e "PORT=9617" --name pihole-exporter ekofr/pihole-exporter:v0.0.11

# Acces the metrics page to check it's working:
$ curl http://127.0.0.1:9617/metrics

# CTRL+C the test container and now run it permanently:
$ docker run -d -p 9617:9617 --restart always -e "PIHOLE_HOSTNAME=192.168.0.1" -e "PIHOLE_API_TOKEN=abc123" -e "INTERVAL=60s" -e "PORT=9617" --name pihole-exporter ekofr/pihole-exporter:v0.0.11

# Update Prometheus to poll pihole_exporter:
$ vi /opt/prometheus/prometheus

  - job_name: 'pihole'
    metrics_path: /metrics
    static_configs:
      - targets: ['192.168.58.6:9617'] # The pihole exporter's hostname:port

Install and Configure unbound_exporter as Container

This is run on the unbound server and remotely polled by Prometheus. This required an unbound upgrade to a newer version than that in the Raspbian repo. The repo version generates SSL certificates withou a SAN. Statistic collection won't work from unbound_exporter because Go has deprecated support for SSL certs without a SAN. After building the latest unbound and running "$ sudo unbound-control-setup", generated certs were still created without a SAN. "unbound-control-setup" is a shell script, manually added the patch details here: https://github.com/letsencrypt/unbound_exporter/issues/20 and then use "$sudo unbound-control-setup -r" to regenerate the certificates. Use "sudo openssl x509 -in /etc/unbound/unbound_server.pem -text -noout" to check if a "X509v3 Subject Alternative Name" and "X509v3 Basic Constraints: critical" are present.

Compiling the newest unbound_exporter also required the installation of a new version of Go than was available in the Raspbian repo.

$ cd /opt
$ git clone https://github.com/letsencrypt/unbound_exporter.git
$ cd unbound_exporter
$ go build
$ go install

# Enable remote statistics in the unbound config:
$ sudo vi /etc/unbound/unbound.conf

server:
    # Enable extra stats for Prometheus.
    # Requires remote control be also enabled...
    extended-statistics: yes
    statistics-cumulative: no

# enable remote-control
remote-control:
    control-enable: yes
    control-interface: 172.17.0.1 # Note that this is the docker0 interface on the unbound server
    control-port: 8953

# Restart unbound

# Create a Dockerfile because one isn't included with unbound_exporter:
vi ./Dockerfile
ARG ARCH="armv7" # for RaspPi
ARG OS="linux"

FROM ubuntu:20.04
COPY ./unbound_exporter  /bin/unbound_exporter

EXPOSE      9167
ENTRYPOINT  [ "/bin/unbound_exporter" ]
CMD ["-unbound.host", "tcp://172.17.0.1:8953"] # Note that this is the docker0 interface on the unbound server

# Build the container
$ docker build -t unbound_exporter .

# Test the container.
# unbound_exporter needs access to the SSL certificates and keys to connect to the control socket to get stats/
# Map the directory containing these certificates and keys using the docker -v option:
$ docker run -it --rm -p 9167:9167 --name unbound_exporter -v /etc/unbound/:/etc/unbound/ unbound_exporter

# If OK, run as daemon:
$ docker run -d -p 9167:9167 --restart always --name unbound_exporter -v /etc/unbound/:/etc/unbound/ unbound_exporter

# Now add to Prometheus
vi /opt/prometheus/prometheus

  - job_name: 'unbound'
    metrics_path: /metrics
    static_configs:
      - targets: ['192.168.58.2:9617'] # The unbound exporters hostname:port

^ Top

Previous page: Pound Proxy
Next page: Ptkgen

Navigation