Aug 13 | 4 min read
Monitoring CockroachDB with Opstrace, Part 1
Concrete example of how to use Prometheus to remote_write metrics from CockroachDB to Opstrace and import dashboards
The Opstrace distribution exposes standards-compliant APIs. Because it exposes the Prometheus remote read and write APIs, any existing Prometheus instance can also send data to Opstrace. While many people know that this is possible, not as many people are aware of how to do it or have tried it. In this post, we'll walk through a demonstration so you can feel confident using Prometheus to send data to a long-term storage engine such as Opstrace.
The Prometheus format is widely exposed by metrics libraries in use today. Many popular services both expose metrics in Prometheus format and provide sets of relevant PromQL queries to monitor their service's performance.
One example of this is CockroachDB, a scalable, distributed Postgres-compatible database. It comes with pre-defined PromQL queries and Grafana dashboards. It's growing popularity makes it a good choice for this post.
Let's walk through their existing Prometheus instructions and modify them to also remote write all samples to Opstrace. We will use the CockroachDB monitoring demo provided on GitHub; this requires Docker Compose to run everything locally.
Install Opstrace
The easiest way to install Opstrace is to follow the instructions in
our quick start. It boils down to 3 simple steps: (1)
download the opstrace
CLI, (2) create a simple config.yaml
file, and (3) run
opstrace create <cloud> <instance> -c config.yaml
.
Once the installation has begun, feel free to proceed with the instructions below.
Launch CockroachDB and Write to Opstrace
First, set up the environment:
export INSTANCE=<my_instance>export TENANT=<my_tenant_name>export TOKEN_FILE=$PATH_TO_TOKEN/tenant-api-token-$TENANT
The bearer token file can be found in the directory where opstrace create
was
initiated. (This will authenticate the collecting Prometheus against the
Opstrace API.)
Check out the cockroach
repo, which contains a docker-compose.yaml
and
Grafana dashboard files:
cd cockroach/monitoring/demo
This compose file contains everything needed to run CockroachDB and Prometheus
locally. However, before we launch it, we will configure remote_write
to tell
Prometheus where the Opstrace instance is:
~/cockroach/monitoring/demo $ cat <<EOF >> config/prometheus.ymlremote_write:- url: https://cortex.$TENANT.$INSTANCE.opstrace.io/api/v1/pushbearer_token_file: /etc/prometheus/tenant-api-tokenEOF
Then, we configure a volume so that Prometheus can locate the tenant-api-token file, as required by Opstrace's secure-by-default API.
~/cockroach/monitoring/demo $ sed -i '' "s|prometheus.yml\"|prometheus.yml\"\, \"$TOKEN_FILE:/etc/prometheus/tenant-api-token\"|" ./docker-compose.yaml
Now we're all set to start up the demo:
~/cockroach/monitoring/demo $ docker compose up -d
To verify everything works, inspect the Prom logs using Docker; if you see a message like this that means it's working correctly:
docker logs demo_prometheus_1...ts=2021-08-12T17:04:34.358Z caller=dedupe.go:112 component=remote level=info remote_name=486e72 url=https://cortex.$TENANT.$INSTANCE.opstrace.io/api/v1/push msg="Done replaying WAL" duration=6.723334045s...
Generate Load
To make the demo more interesting, start the
movr
synthetic workload
on the Cockroach instance (so we can observe SQL statements occurring):
cockroach workload init movr \'postgresql://[email protected]:26257?sslmode=disable'cockroach workload run movr --duration=5m \'postgresql://[email protected]:26257?sslmode=disable'
If the workload is being ingested successfully, the CockroachDB UI will look something like this:
http://localhost:8080/#/metrics/overview/cluster
Install and View Dashboards with Opstrace
Cockroach comes with pre-defined Grafana dashboards. Let's import them in Opstrace to view the metrics we are now remote writing.
The dashboards are located in the cockroachdb/cockroach
repo, just a level up
from where we executed the demo—remember this for later:
~/cockroach/monitoring/demo $ ls ../grafana-dashboardschangefeeds.json hardware.json queues.json runtime.json sql.json
To import the dashboards:
- From the Opstrace UI, select the chosen tenant in the upper left-hand corner.
- Select Dashboards from the navigation on the left.
- Then, click on View Dashboards on the right.
- Once the Grafana UI is open, select Create > Import.
- Choose Upload JSON File, and navigate to the
cockroach/monitoring/grafana-dashboards
directory. - Upload three of the dashboards:
- runtime.json
- sql.json
- storage.json
Now, observe the incoming metrics:
Hopefully, this exercise shows you how easy it is to take an existing Prometheus instance and write its collected metrics to Opstrace, a horizontally scalable, highly reliable monitoring platform with cost-effective long-term storage.
We've posted before about Opstrace Integrations. If you are interested in a first-class integration with CockroachDB, please drop us a line in our community Slack or at [email protected].