Alexey, an enthusiastic advocate for observability, has recently unveiled a noteworthy project: the pgpro-otel-collector. This OpenTelemetry collector, developed by PostgresPro, is specifically designed to gather metrics and logs from PostgreSQL databases.
A few words about OpenTelemetry
OpenTelemetry serves as a foundational framework for observability, providing a modular toolkit that allows users to create customized monitoring agents. One of its essential components is the OpenTelemetry Collector, which enables the aggregation of various data sources and facilitates their transmission to designated endpoints. While there is a wealth of information available on OpenTelemetry, it is worth noting that the existing PostgreSQL receiver is somewhat limited in functionality. In response to this gap, Alexey and his team developed their own receiver, which has been integrated into the OpenTelemetry builder, culminating in the creation of the pgpro-otel-collector.
What can pgpro-otel-collector do?
- Collect Postgres metrics: The primary function of the collector is to gather metrics from PostgreSQL. Built upon a dedicated internal metrics library, it simplifies the addition of new features and the resolution of issues. The collector connects to Postgres and queries system catalogs, providing access to a wealth of data.
- Collect OS-level metrics: Through the
hostmetrics, it can also gather CPU, memory, disk, and network usage statistics, which are often essential for comprehensive monitoring. - Collect Postgres logs: Utilizing the
filelogreceiver, the collector can read Postgres logs formatted in CSV and JSON, seamlessly integrating log data with metrics. - Internal processors: These processors facilitate batch processing of metrics and logs, optimizing data before it is sent to the monitoring system.
- Export to OTLP-compatible backends: Currently, the collector supports exporting to Elasticsearch via the
otlphttpexporter, with plans for additional backend support in the future. - Prometheus exporter: For those utilizing Prometheus, the Prometheus exporter allows for straightforward exposure of metrics.
How to use it
To monitor a PostgreSQL instance, one simply installs the pgpro-otel-collector on the same host, configures the Postgres connection, and specifies the destination for the collected data. This single tool efficiently manages metrics, logs, and data exportation, leaving users to focus on creating insightful Grafana dashboards. The collector is capable of gathering a broad range of PostgreSQL statistics from standard system views, including client activity, replication, and I/O metrics, among others. It even supports advanced features like CFS and pg_wait_sampling. A comprehensive list of available metrics can be found here.
Installation and startup are straightforward, requiring only about five commands. The default configuration includes basic metrics, while additional modules can be enabled for more detailed monitoring. Once operational, metrics are accessible at *:8889/metrics, with Prometheus scraping enabled by default. Depending on the monitoring setup, integration can be tailored to either scrape data directly or configure the collector for OTLP-compatible systems like Grafana and Kibana.
What’s next
- Enhancements to support additional system views and extensions.
- Integration with Shardman and BiHA.
- Collection of custom query-based metrics.
- Inclusion of a Grafana dashboard within the package.
- Various internal improvements to enhance functionality.
Retrospective
Alexey’s journey into the realm of monitoring systems began in the early 2000s with tools like The Dude and Cacti. His experience evolved through various platforms, including Zabbix, where he developed numerous scripts, many of which remain active on GitHub. His professional path led him to DataEgret, where he engaged with PostgreSQL daily, often troubleshooting production issues without adequate tools, which inspired the creation of pgcenter.
With the advent of Prometheus and Grafana, Alexey continued to refine his monitoring expertise. He embarked on a project to develop a Postgres monitoring SaaS, which, despite its innovative features, ultimately did not gain traction due to challenges in sales and marketing. However, the experience led to the development of pgSCV, a tool designed for collecting metrics from Postgres, Pgbouncer, and Patroni.
After discovering OpenTelemetry, Alexey appreciated its vendor-neutral approach, which offered greater flexibility and applicability compared to previous projects. In late 2023, with the support of his manager, he initiated the development of the OpenTelemetry Collector, starting with the metrics library and progressing to the collector itself. Despite facing bureaucratic challenges, the project culminated in a public release, making the pgpro-otel-collector available for users seeking a robust solution for PostgreSQL observability.
