Prometheus

Overview

Prometheus is deployed using the official image:

• Image source: https://hub.docker.com/r/prom/prometheus

• Documentation: https://prometheus.io/docs/prometheus/latest/installation/#using-docker

Container

If you want to use different version of container you can modify the following variable:

prometheus_docker_image: prom/prometheus:v2.44.0

Data persistence

Prometheus container uses named volume to persist the data. The container can be safely deleted and recreated without the loss of data.

    volumes:
       - prom-data:/prometheus

Deleting data

To delete the data and start from scratch you will need to execute the following command on the server:

docker kill prometheus 
docker container rm prometheus 
docker volume rm observability_dc_prom-data

Configuration

How Prometheus is configured is dictated by the deployment type. In a single server deployment, Prometheus uses a "static" configuration defined in prometheus.yml file. In a distributed deployment, file based Service Discovery feature of Prometheus is used. In both cases the configuration is fully parameterized and customizable. You can review the configuration templates here: https://github.com/SlingNode/slingnode-ansible-ethereum-observability/tree/master/templates/prometheus

Single server deployment

In a single server deployment, Prometheus uses a "static" configuration defined in prometheus.yml file. The config file always contains scrape targets for all agents (node-exporter, cAdvisor, ethereum-metrics-exporter). The targets for Ethereum clients are generated dynamically based on the types of clients that are deployed (as specified by the "clients" variable).

You can customize Prometheus by providing your own configuration file.

# Use the below variables to provide your own Prometheus config file:
prometheus_config_template: templates/prometheus/prometheus.yml.j2

Distributed deployment

In a distributed deployment, file based Service Discovery feature of Prometheus is used. In summary the file based Service Discover (SD) allows for listing scrape target hosts in text files. Prometheus periodically reads those files and starts scraping new hosts. There are two SD file templates included in the role:

• sd_all_targets.yml - includes all agent targets (node-exporter, cAdvisor, ethereum-metrics-exporter), all hosts run the agents

• sd_client_targets.yml - includes Ethereum client targets with config dependent on the type of the client

Both files will be automatically populated based on the group_vars or host_vars.

You can customize the configuration by providing your own configuration files.

# If you customized Prometheus config file and don't use Service Dicovery
# set this variable to an empty list in your Playbook
prometheus_service_discovery_target_templates:
  - src: templates/prometheus/{{ prometheus_service_discovery_all_targets_file }}.j2
    dest: "{{ observability_root_path }}/prometheus/{{ prometheus_service_discovery_all_targets_file }}"
  - src: templates/prometheus/{{ prometheus_service_discovery_client_targets_file }}.j2
    dest: "{{ observability_root_path }}/prometheus/{{ prometheus_service_discovery_client_targets_file }}"

prometheus_service_discovery_all_targets_file: sd_all_targets.yml
prometheus_service_discovery_client_targets_file: sd_client_targets.yml