Prometheus

Prometheus is known by not accepting out-of-order samples during ingestion, but recently (2 years ago) support was added behind a feature-flag. Since then, many improvements have been made to out-of-order ingestion and it has become one of the requirements for OTLP ingestion in Prometheus. We want to deliver Prometheus 3.0 in a few months, and that requires marking out-or-order ingestion as a stable feature. In this project we will clean up several smaller issues around out-of-order ingestion, hopefully marking it as stable by the end of the mentorship.

In the past few quarters, a lot of work has gone into the new Remote Write 2.0 (PROM-35) proposal, and a new spec has been successfully established https://prometheus.io/docs/specs/prw/remote_write_spec_2_0/. But there is still a lot of work that needs to be done to declare it stable, in terms of stability and performance in Prometheus, and general adoption from the wider Prometheus ecosystem. Expected Outcome: Since this is a large initiative, for this round of mentorship, we want to focus on the following tasks, 1. Add 2.0 support to compliance test; make it easy to test write and receive implementations. 2. Ensure RW new features works on agent mode (test for agent mode with metadata-wal-records and type-and-unit features). 3. Ensure Prometheus uses the official RW client

Prometheus remote write allows users to send their metrics to other time series databases. Though the Prombench tool has existed for a number of years, it has never been extended to support performance testing of Remote Write in a realistic production like environment. With the upcomming Remote Write 2.0 changes to both the underlying implementation as well as the wire format, the need for benchmarking of remote write beyond static Go bechmark tests has increased. Expected Outcome: Build additional (or extends existing) tooling, similar to Prombench’s load-generator and avalanche, to support scenarios under which remote write should be performance tested. For example; allowing gradual increases/decreases in # of active series, sudden spikes in active series, various amounts of latency in the server receiving the remote write data, etc. Time permitted, extend Prombench's test suite to include a set of Remote Write tests that can be run via a new command.

In recent discussions with the team, we decided that Prometheus won't be exporting its data with the OTLP format, however, Prometheus is still committed to have good import/export compatibility with OpenTelemetry. Last year Prometheus release the second version of its Remote-Write protocol, which translates a lot better with the OTLP format and the team started working on a PRW receiver in the collector-contrib project. This project is about getting this component into the finish line and publish it as an stable component in the collector. Expected Outcome: PrometheusRemoteWriteReceiver considered Alpha and released with OpenTelemetry-Collector-Contrib.

Through a LFX Mentorship done in March-May, we performed UX research focusing on different ways Prometheus could deal with OpenTelemetry's Resource Attributes. One of the insights that this research showed us is that Prometheus and OpenTelemetry were primarily designed for different use-cases. While Prometheus focuses on monitoring metrics that the user knows that exist, one of OpenTelemetry's goals is to enable exploration of your data while the user doesn't really know what he/she is looking for. This mentorship will take the next steps on those research findings, exploring ideas about how Prometheus can enable the "Exploration" use case. Expected Outcome: 1. Two or more sets of wireframes based on ideation workshop 2. Spoken presentation and report summarizing the findings and deliverables 3. Stretch goal: Apply to present findings at an upcoming conference