Monitoring CockroachDB with Opstrace, Part 1

The Opstrace distribution exposes standards-compliant APIs. Because it exposes the Prometheus remote read and write APIs, any existing Prometheus instance can also send data to Opstrace. While many people know that this is possible, not as many people are aware of how to do it or have tried it. In this post, we'll walk through a demonstration so you can feel confident using Prometheus to send data to a long-term storage engine such as Opstrace.

The Prometheus format is widely exposed by metrics libraries in use today. Many popular services both expose metrics in Prometheus format and provide sets of relevant PromQL queries to monitor their service's performance.

One example of this is CockroachDB, a scalable, distributed Postgres-compatible database. It comes with pre-defined PromQL queries and Grafana dashboards. It's growing popularity makes it a good choice for this post.

Let's walk through their existing Prometheus instructions and modify them to also remote write all samples to Opstrace. We will use the CockroachDB monitoring demo provided on GitHub; this requires Docker Compose to run everything locally.

Install Opstrace

The easiest way to install Opstrace is to follow the instructions in our quick start. It boils down to 3 simple steps: (1) download the opstrace CLI, (2) create a simple config.yaml file, and (3) run opstrace create <cloud> <instance> -c config.yaml.

Once the installation has begun, feel free to proceed with the instructions below.

Launch CockroachDB and Write to Opstrace

First, set up the environment:

export INSTANCE=<my_instance>
export TENANT=<my_tenant_name>
export TOKEN_FILE=$PATH_TO_TOKEN/tenant-api-token-$TENANT

The bearer token file can be found in the directory where opstrace create was initiated. (This will authenticate the collecting Prometheus against the Opstrace API.)

Check out the cockroach repo, which contains a docker-compose.yaml and Grafana dashboard files:

git clone [email protected]:cockroachdb/cockroach  # fyi, it's big
cd cockroach/monitoring/demo

This compose file contains everything needed to run CockroachDB and Prometheus locally. However, before we launch it, we will configure remote_write to tell Prometheus where the Opstrace instance is:

~/cockroach/monitoring/demo $ cat <<EOF >> config/prometheus.yml

remote_write:
  - url: https://cortex.$TENANT.$INSTANCE.opstrace.io/api/v1/push
    bearer_token_file: /etc/prometheus/tenant-api-token
EOF

Then, we configure a volume so that Prometheus can locate the tenant-api-token file, as required by Opstrace's secure-by-default API.

~/cockroach/monitoring/demo $ sed -i '' "s|prometheus.yml\"|prometheus.yml\"\, \"$TOKEN_FILE:/etc/prometheus/tenant-api-token\"|" ./docker-compose.yaml

Now we're all set to start up the demo:

~/cockroach/monitoring/demo $ docker compose up -d

To verify everything works, inspect the Prom logs using Docker; if you see a message like this that means it's working correctly:

docker logs demo_prometheus_1
...
ts=2021-08-12T17:04:34.358Z caller=dedupe.go:112 component=remote level=info remote_name=486e72 url=https://cortex.$TENANT.$INSTANCE.opstrace.io/api/v1/push msg="Done replaying WAL" duration=6.723334045s
...

Generate Load

To make the demo more interesting, start the movr synthetic workload on the Cockroach instance (so we can observe SQL statements occurring):

cockroach workload init movr \
'postgresql://[email protected]:26257?sslmode=disable'

cockroach workload run movr --duration=5m \
'postgresql://[email protected]:26257?sslmode=disable'

If the workload is being ingested successfully, the CockroachDB UI will look something like this:

http://localhost:8080/#/metrics/overview/cluster

Install and View Dashboards with Opstrace

Cockroach comes with pre-defined Grafana dashboards. Let's import them in Opstrace to view the metrics we are now remote writing.

The dashboards are located in the cockroachdb/cockroach repo, just a level up from where we executed the demo—remember this for later:

~/cockroach/monitoring/demo $ ls ../grafana-dashboards
changefeeds.json  hardware.json     queues.json       runtime.json      sql.json

To import the dashboards:

1. From the Opstrace UI, select the chosen tenant in the upper left-hand corner.
2. Select Dashboards from the navigation on the left.
3. Then, click on View Dashboards on the right.
4. Once the Grafana UI is open, select Create > Import.
5. Choose Upload JSON File, and navigate to the cockroach/monitoring/grafana-dashboards directory.
6. Upload three of the dashboards:

• runtime.json
• sql.json
• storage.json

Now, observe the incoming metrics:

Hopefully, this exercise shows you how easy it is to take an existing Prometheus instance and write its collected metrics to Opstrace, a horizontally scalable, highly reliable monitoring platform with cost-effective long-term storage.

We've posted before about Opstrace Integrations. If you are interested in a first-class integration with CockroachDB, please drop us a line in our community Slack or at [email protected].

References

• https://www.cockroachlabs.com/docs/v21.1/start-a-local-cluster
• https://www.cockroachlabs.com/docs/stable/monitor-cockroachdb-with-prometheus.html#step-5-visualize-metrics-in-grafana
• https://github.com/cockroachdb/cockroach/tree/master/monitoring