Sheet 04

Observability expertise

We help your team monitor its own systems: from the first metric to an on-call rota your team runs itself.

We don’t run anything for you and we don’t do on-call. What we set up together runs on your side. You end up with dashboards and alert rules under your own control, and a team that learned to read them during the training.

Questions? Email info@databrein.com or call +31 40 236 49 75.

Who does what

Who does what does it helps

Who does what	Databrein	Your team
Reviewing your current setup	does it	helps
Choosing the stack	helps	does it
Deciding what to measure	helps	does it
Instrumenting the software	does it	does it
Setting up dashboards and alerts	does it	does it
Training the team	does it	does it
Handover: from here your team takes over
Running and maintaining the stack	–	does it
Being on call and responding to alerts	–	does it

You choose the stack; the options are compared under Tools.

What you measure

Logs: What happened, line by line, with time and context.
Metrics: Numbers over time: requests per minute, error rate, memory.
Traces: The path of one request through your systems, with the time per step. Your application needs a tracer for this. Below you can see what that looks like, measured on this page.
Service levels: Targets for what counts as good enough, so an alert means something and doesn’t go off for nothing at three in the morning.

Waterfall chart of this page, measured in your browser at …. Each row is one step of the load, with start + duration.

With JavaScript enabled, your browser draws here, step by step, how this page loads.

Total load time: …

OpenTelemetry

OpenTelemetry is the open standard for getting logs, metrics and traces out of your software and sending them on. We’re glad that standard exists and use it wherever we can: you instrument once and stay free to choose where the data goes.

Prometheus, Loki, Tempo and Datadog all accept OpenTelemetry. The project is open source and part of the Cloud Native Computing Foundation, as is Prometheus.

Your application with a tracer

OTLP

Collector OpenTelemetry

metrics Prometheus
logs Loki
traces Tempo

Your application sends everything to the collector, which passes each kind of data on to its own store. If you choose Datadog, you only change the collector, not your code.

Tracer: Traces only exist if your application creates them: for that it needs a tracer. Automatic instrumentation records the standard steps, such as HTTP requests and database queries. The steps that matter for your work, we build in together with your team.
Collector: The OpenTelemetry Collector receives logs, metrics and traces in one place and passes them on. Switching stores happens here, not in your code.
Storage: Metrics in Prometheus, logs in Loki and traces in a trace store such as Tempo. Grafana puts them side by side, so you can click through from a spike in a graph to the slow request and its log lines.

Tools

Self-hosted: Prometheus for metrics, Loki for logs, a trace store such as Tempo and Grafana for dashboards and alerts. When the number of metrics grows, you add Mimir behind Prometheus. If you’d rather not run monitoring yourself, there is Datadog. We choose with you what suits your team and your systems.

VGDB, which we develop and run, runs on Grafana and Prometheus, so we use them ourselves, in production. How VGDB runs.

Tools
Tool	What it is	Where it runs	Choose it when
Prometheus	Open source. Collects metrics from your applications and stores them.	In your own environment	you want to keep the data under your own control
Mimir	Open source. Stores Prometheus metrics for the long term and at scale.	In your own environment, or as a service from Grafana Labs	you bring together metrics from several environments, or want to keep them for months or years
Loki	Open source. Stores logs and makes them searchable in Grafana.	In your own environment, or as a service from Grafana Labs	you want logs next to your metrics, without running a heavy search engine
Tempo	Open source. Stores traces: the path of one request through your systems.	In your own environment, or as a service from Grafana Labs	you want to go from a slow measurement to the request that was slow
Grafana	Open source. Dashboards and alerts, on Prometheus, Loki, Tempo and other sources.	In your own environment, or as a service from Grafana Labs	you want one place for dashboards and alerts
Datadog	A paid service: metrics, logs and traces in one product.	At Datadog, as SaaS	you would rather not run monitoring yourself

Principles first, then tools

These tools are a starting point; we are not tied to them. Good observability rests on principles that hold for any stack: measure what your users notice, alert on symptoms rather than causes, and tie logs, metrics and traces together through the trace ID. If your team already works with other tools, or your situation calls for something else, we work with those and fit the setup to your systems and your team.

Tell us what you need.

Email us a few lines about your question, or call us. Then we’ll set up a first meeting.

Email info@databrein.com

Call +31 40 236 49 75