The Opstrace distribution exposes standards-compliant APIs. Because it exposes the Prometheus remote read and write APIs, any existing Prometheus instance can also send data to Opstrace. While many people know that this is possible, not as many people are aware of how to do it or have tried it. In this post, we'll walk through a demonstration so you can feel confident using Prometheus to send data to a long-term storage engine such as Opstrace.
The Prometheus format is widely exposed by metrics libraries in use today. Many popular services both expose metrics in Prometheus format and provide sets of relevant PromQL queries to monitor their service's performance.
One example of this is CockroachDB, a scalable, distributed Postgres-compatible database. It comes with pre-defined PromQL queries and Grafana dashboards. It's growing popularity makes it a good choice for this post.
Let's walk through their existing Prometheus instructions and modify them to also remote write all samples to Opstrace. We will use the CockroachDB monitoring demo provided on GitHub; this requires Docker Compose to run everything locally.
The easiest way to install Opstrace is to follow the instructions in
our quick start. It boils down to 3 simple steps: (1)
opstrace CLI, (2) create a simple
config.yaml file, and (3) run
opstrace create <cloud> <instance> -c config.yaml.
Once the installation has begun, feel free to proceed with the instructions below.
First, set up the environment:
export INSTANCE=<my_instance>export TENANT=<my_tenant_name>export TOKEN_FILE=$PATH_TO_TOKEN/tenant-api-token-$TENANT
The bearer token file can be found in the directory where
opstrace create was
initiated. (This will authenticate the collecting Prometheus against the
Check out the
cockroach repo, which contains a
Grafana dashboard files:
git clone [email protected]:cockroachdb/cockroach # fyi, it's bigcd cockroach/monitoring/demo
This compose file contains everything needed to run CockroachDB and Prometheus
locally. However, before we launch it, we will configure
remote_write to tell
Prometheus where the Opstrace instance is:
~/cockroach/monitoring/demo $ cat <<EOF >> config/prometheus.ymlremote_write:- url: https://cortex.$TENANT.$INSTANCE.opstrace.io/api/v1/pushbearer_token_file: /etc/prometheus/tenant-api-tokenEOF
Then, we configure a volume so that Prometheus can locate the tenant-api-token file, as required by Opstrace's secure-by-default API.
~/cockroach/monitoring/demo $ sed -i '' "s|prometheus.yml\"|prometheus.yml\"\, \"$TOKEN_FILE:/etc/prometheus/tenant-api-token\"|" ./docker-compose.yaml
Now we're all set to start up the demo:
~/cockroach/monitoring/demo $ docker compose up -d
To verify everything works, inspect the Prom logs using Docker; if you see a message like this that means it's working correctly:
docker logs demo_prometheus_1...ts=2021-08-12T17:04:34.358Z caller=dedupe.go:112 component=remote level=info remote_name=486e72 url=https://cortex.$TENANT.$INSTANCE.opstrace.io/api/v1/push msg="Done replaying WAL" duration=6.723334045s...
To make the demo more interesting, start the
movr synthetic workload
on the Cockroach instance (so we can observe SQL statements occurring):
If the workload is being ingested successfully, the CockroachDB UI will look something like this:
Cockroach comes with pre-defined Grafana dashboards. Let's import them in Opstrace to view the metrics we are now remote writing.
The dashboards are located in the
cockroachdb/cockroach repo, just a level up
from where we executed the demo—remember this for later:
~/cockroach/monitoring/demo $ ls ../grafana-dashboardschangefeeds.json hardware.json queues.json runtime.json sql.json
To import the dashboards:
- From the Opstrace UI, select the chosen tenant in the upper left-hand corner.
- Select Dashboards from the navigation on the left.
- Then, click on View Dashboards on the right.
- Once the Grafana UI is open, select Create > Import.
- Choose Upload JSON File, and navigate to the
- Upload three of the dashboards:
Now, observe the incoming metrics:
Hopefully, this exercise shows you how easy it is to take an existing Prometheus instance and write its collected metrics to Opstrace, a horizontally scalable, highly reliable monitoring platform with cost-effective long-term storage.