Around a year ago, we began to wonder: how do companies monitor their infrastructure?
We come from the container world, which is seeing quick adoption of Prometheus and other CNCF projects. Working with these open source projects is like building LEGOS™ with half of the necessary instructions. Smart engineers can create a solution based on these building blocks. This is currently a common path, particularly because of the rising usage of Kubernetes and other cloud native technologies.
But it’s complex and expensive to build a robust and scalable observability system. It requires a tremendous engineering effort, diverting resources away from your core business activities. Instead, you could choose an external service provider like Datadog, who can get your monitoring system up and running in minutes. They’ll handle the backups, monitoring, fixing, and updates. But for this convenience you pay for every byte of data you send to them.
While this model made sense when it was challenging to manage data, it now incentivizes vendors to build tools that get you to generate more data. So, unfortunately, to control costs, you’re now motivated to actually limit what you observe or how long you keep it. To make matters worse, as you scale, your costs grow almost exponentially and unpredictably. You’re also paying a cloud provider for network egress costs.
Ultimately, it’s a decision between control and convenience. With control, you hire expensive experts to build and maintain an open source solution for you internally. With convenience, service providers charge you for the simplicity, effectively forcing you to limit the data you send and retain (unless you have deep pockets).
We created an observability platform that brings together the best of both worlds. It’s accessible for anyone, expert or not. And you’re not penalized for maximizing your monitoring coverage.
We think you’ll love the Opstrace observability platform. It runs entirely in your cloud account. Your data never leaves your network. You have full control over how much data you keep and for how long. Since it’s all in your cloud account, you benefit from any negotiated pricing you might already have with your cloud provider.
Opstrace clusters expose a secure, horizontally scalable Prometheus API backed by Cortex for aggregating metrics and a Loki API for log collection. You can point your existing Prometheus or FluentD/Promtail instances to an Opstrace cluster in minutes. We also plan to support a wide variety of APIs that will, for example, allow you to point your Datadog agent at an Opstrace cluster with minimal effort.
For creating an Opstrace Cluster you need our command-line interface. It talks directly to your cloud provider, using your credentials, to create a fully functioning, secure observability platform, ready to receive your data. After your cluster is running, our Controller inside the cluster will maintain things over time. All of your data resides safely (and inexpensively) in your own cloud storage (e.g., S3) buckets.
Teams building their own solution based on existing open source components take weeks or months to get started. In addition, they must perform ongoing maintenance. Once done, critical features such as exposing API endpoints securely (which means at the very least TLS and some form of authentication) are often not implemented.
You can now access all of the features and functionality you’ve just read about. And we’re already working to evolve this foundation into a complete solution. You’ll experience a new user interface to query, visualize, inspect, and alert. Ultimately, Opstrace will be user-friendly for the beginner (e.g., making PromQL and complex time series math optional) while being advanced enough for experts who want to code, share, and work on their dashboards. We will bring these features—and much more—in the same fully-automated, easy-of-use way we’ve released in the foundation.
We are releasing Opstrace as a proper open source project under the Apache License, Version 2.0. We do offer a commercial version with additional security features such as custom single sign-on and support. Since it's priced per user; we will never charge you based on the amount of data you send—that is between you and your cloud provider.
Seb & Mat Co-founders