GitLab Metrics Exporter

NOTE: This project is under development and considered experimental. It is not in use at GitLab currently.

GitLab Metrics Exporter (GME) is meant to subsume the functionality of gitlab-exporter and in-app Ruby Prometheus exporters distributed with and running in gitlab-rails.

It has the following goals:

• Provide a single, unified mechanism to export application-specific metrics to Prometheus.
• Ability to run as a standalone system or as a sidecar process to gitlab-rails.
• Ability to collect and serve a variety of metrics using a flexible approach to probing.
• Ability to collect and serve metrics in a memory- and CPU-efficient way.
• Ability to exploit concurrency when running multiple probes.

GME follows a simple design based on two primary concepts:

• Probes. A probe is responsible for sampling data. It is up to the probe how that is accomplished; it could query a connected data store, send HTTP requests, perform IPC or read from the file system. A probe returns metric data as a series of MetricGroups that can be consumed by a renderer. Probes can run in parallel. Metrics within a MetricGroup need not be ordered. However, all samples belonging to a particular metric must appear in a single MetricGroup.
• Renderers. A renderer takes a series of MetricGroups emitted from one or more probes and serializes it, so it can be ingested by Prometheus.

The exporter exposes a single endpoint, /metrics. When a client requests this endpoint, all probes will execute in parallel and their results are combined, serialized, and rendered to the client. This can be understood as a function: render(merge(probe1(), ..., probeN())).

Which probes should run in response to a request is specified via configuration switches.

For reasons of efficiency and simplicity, GME does not use the data model from the official Go Prometheus client. GME's data model is flat instead of hierarchical and only uses two concepts: Samples and MetricGroups. A Sample is a named and labeled 64 bit float value. A MetricGroup organizes lists of samples by metric type.

There is currently no MetricFamily in GME; however, a MetricGroup can be understood as a series of one or more metric families. If multiple metric families are emitted by a probe as one MetricGroup, then they will share a HELP line when rendered. If this is undesirable, one must emit a single MetricGroup per MetricFamily.

The differences between the two data models are summarized below.

The following probes are currently implemented:

• self. This probe samples Go runtime, procfs, and HTTP server metrics.
• mmap. This probe reads db files written by prometheus-client-mmap. It is the format used to emit application metrics from gitlab-rails.

The following renderers are currently implemented:

• text. This renderer implements Prometheus' default Text Exposition Format.

You can see a list of configuration options by running gitlab-metrics-exporter -h.

Settings are passed as command line flags or through the environment.

See the following examples for how to run the server.

This binds the server to all network interfaces using the default port and runs the self probe:

$ gitlab-metrics-exporter

You can change the default host and port as follows:

$ gitlab-metrics-exporter --host 127.0.0.1 --port 1234

Or, alternatively, via the GME_SERVER_HOST and GME_SERVER_PORT environment variables.

To start the server on the default port and run the self and mmap probes:

$ gitlab-metrics-exporter --probe self --probe mmap --mmapdir /path/to/metrics

Probes can be set via the environment as well:

$ GME_PROBES=self,mmap gitlab-metrics-exporter --mmapdir /path/to/metrics

As can other flags:

$ GME_MMAP_METRICS_DIR=/path/to/metrics gitlab-metrics-exporter --probe self --probe mmap 

You can specify a log file (including stderr and stdout), log format, and log level:

$ gitlab-metrics-exporter --log-file /path/to/log --log-format json --log-level info

Or via environment variables:

$ GME_LOG_FILE=/path/to/log GME_LOG_FORMAT=json GME_LOG_LEVEL=info gitlab-metrics-exporter

You can enable HTTPS via TLS as follows:

$ gitlab-metrics-exporter --cert-file /path/to/cert.pem --cert-key /path/to/key.pem

Or via environment variables:

$ GME_CERT_FILE=/path/to/cert.pem GME_CERT_KEY=/path/to/key.pem gitlab-metrics-exporter

Note:

• Certificate and key files must be encoded in PEM format.
• The cert-file must be a bundle file i.e. contain the server certifcate, the root CA and all intermediaries.
• Mutual TLS (mTLS) is not currently supported.

See CONTRIBUTING.