A complete monitoring stack for servers and Kubernetes
A complete monitoring stack is not Grafana with a pretty dashboard. A dashboard helps you look at the system, but on its own it does not answer the main operational question: what broke, where exactly, how serious it is, and who needs to be woken up.
A small product usually needs five layers: metrics, logs, traces, alerts, and a sane storage architecture. Prometheus, Grafana, Loki, Tempo, OpenTelemetry, and Alertmanager cover these layers without locking you into a single cloud provider. You can run this stack on one VPS, on a handful of servers, or in Kubernetes. But you should not deploy it as a pile of trendy containers. You have to assemble it as a system your team can actually operate and get value from.
What a complete stack has to cover
Monitoring does not start with picking a tool. It starts with the list of questions your team wants to answer in production.
First question: is the service alive. For that you need availability metrics, health checks, latency, error rate, and basic SLOs. If the API returns 200 but p95 latency has climbed to five seconds, that is already an incident, even if uptime is formally intact.
Second question: are there enough resources. CPU, RAM, disk I/O, free space, network, file descriptors, container state, database pressure — this is the boring part of monitoring, but it is the part that most often saves you from nighttime outages. A full disk or a runaway log volume takes down small products more often than complex distributed bugs.
Third question: what happened inside the application. Metrics alone are not enough here. You need structured logs and request tracing: which endpoint is slow, which external call hangs, where the error appeared, and which services the request passed through.
Fourth question: who finds out about the problem. An alert without routing is just an entry in a UI. You need rules, grouping, suppression of dependent alerts, silences during maintenance, and delivery to Telegram, Slack, email, PagerDuty, or another on-call system.
Fifth question: how much it costs to store. Retention for metrics, logs, and traces has to be a deliberate decision. Keeping every debug log for a year in self-hosted Loki is usually not a strategy but a future disk outage.
The role of each component
In this stack every tool has its own area of responsibility.
Prometheus collects and stores metrics as time series. Its strengths are the pull model, service discovery, labels, and PromQL. For servers it pulls data from node_exporter, for containers from cAdvisor or runtime metrics, and for Kubernetes from the kubelet, kube-state-metrics, the ingress controller, databases, and applications. Prometheus answers "how much", "how fast", "how often", and "when did it start" well. (Prometheus)
Grafana is the investigation interface. Grafana connects to Prometheus, Loki, Tempo, and other data sources, builds dashboards, helps you explore data through Explore, and links different signals together. Grafana should not be the only source of truth: if Prometheus or Loki disappears, pretty panels will not save you. But as a single pane of glass for operations, it has become the default choice. (Grafana)
Loki stores logs. Its key difference from the Elasticsearch approach is that Loki does not index the full text of logs; it indexes labels for log streams instead. That lowers storage cost but demands discipline: labels should help you filter by service, namespace, pod, host, and environment, not turn into an endless set of unique values. (Loki)
Tempo stores distributed traces. Traces matter where a single user request passes through an API, a queue, a worker, a database, and an external service. A metric shows the rise in latency. A log shows a single event. A trace shows the path of the request and the place where time was spent. (Tempo)
OpenTelemetry is the layer for instrumenting and routing telemetry. Its value is not in being "yet another agent" but in letting you instrument the application once and send traces, metrics, and logs to different backends through a shared protocol and the Collector. That reduces vendor lock-in and helps you avoid rewriting the application when you change the storage backend. (OpenTelemetry)
Alertmanager receives alerts from Prometheus, groups them, deduplicates them, suppresses dependent events, and sends notifications to receivers. Prometheus decides that a condition has become true. Alertmanager decides how not to turn that into spam and exactly who should get the signal. (Alertmanager)
Architecture for one or several servers
For a single VPS or a small group of servers you do not need to start with a Kubernetes-native design. A tidy self-hosted stack you can bring up with Docker Compose or systemd services is enough.
A minimal layout looks like this:
| Layer | Components | What it collects |
|---|---|---|
| Server metrics | node_exporter + Prometheus |
CPU, RAM, disk, network, load average |
| Container metrics | cAdvisor or a runtime exporter | container usage, restarts, limits |
| Application metrics | Prometheus endpoint or OpenTelemetry Collector | request rate, latency, errors, business metrics |
| Logs | Grafana Alloy or another Loki-compatible agent + Loki | container stdout/stderr, system logs, app logs |
| Traces | OpenTelemetry SDK/agent + Collector + Tempo | the request path between services |
| UI and alerts | Grafana + Alertmanager | dashboards, Explore, notifications |
On a single server you can keep Prometheus, Loki, Tempo, Grafana, and Alertmanager next to the application if the load is small. But that is a trade-off: when the server dies entirely, monitoring dies with it. A more mature setup is a separate small monitoring VPS that watches the production servers from the outside and keeps at least the basic metrics apart from the application.
A practical minimum for a VPS: node_exporter, Prometheus, Grafana, Loki, a log agent, and Alertmanager. Tempo and OpenTelemetry are worth adding once you have several services, queues, external dependencies, or regular latency investigations. For a monolith on one server traces can be useful, but they should not delay the launch of basic monitoring.
Architecture for Kubernetes
In Kubernetes monitoring becomes not just a set of processes but part of the operational control plane. Here service discovery, labels, namespaces, ownership, and scrape rules matter.
The base layer is usually built around kube-prometheus-stack. It installs the Prometheus Operator, Prometheus, Alertmanager, Grafana, node exporter, kube-state-metrics, and a set of ready-made rules. The main value of the Prometheus Operator is not that it "installs Prometheus" but its Kubernetes-native configuration model: ServiceMonitor, PodMonitor, PrometheusRule, and declarative management of scrape targets. (Prometheus Operator)
For logs in the cluster, an agent is deployed as a DaemonSet on every node. It reads container logs, adds labels such as namespace, pod, container, app, cluster, and pushes the streams to Loki. The same label discipline applies as above: request_id, user_id, and raw URLs should stay inside the log line or structured payload, not in the index.
For traces, applications are instrumented through the OpenTelemetry SDK, auto-instrumentation, or a sidecar/agent approach. The OpenTelemetry Collector can run as a DaemonSet for node-local telemetry ingestion or as a Deployment gateway for centralized processing. On a small cluster the gateway mode is usually enough. On a large one, an agent + gateway combination reduces network loss and centralizes sampling, enrichment, and export.
In Kubernetes, Tempo usually receives traces over OTLP, Jaeger, or Zipkin-compatible endpoints and stores them in object storage or a local backend if it is a small environment. For a production cluster, a local disk as long-term trace storage is a weak point. Object storage is almost always the more honest choice.
How to connect metrics, logs, and traces
The main mistake in self-hosted observability is to set up three storage backends and leave them as three separate worlds. Then an engineer sees a spike in Prometheus, manually hunts for logs in Loki, and then guesses at a trace in Tempo. That is better than nothing, but it is not yet a complete stack.
The connection starts with consistent labels: service, environment, cluster, namespace, pod, instance. If the service is called api in Prometheus, backend-api in Loki, and main-http in traces, an investigation turns into a manual translation between systems.
The second connection is trace_id and span_id in logs. When the application writes structured logs with trace context, Grafana can open a trace from a specific log line. In the other direction, Tempo can show the logs related to a span. This is especially useful for errors that are visible only on a single request and do not create a noticeable spike on a dashboard.
The third connection is exemplars. A latency metric shows an aggregate, but an exemplar can tie a point on the graph to a specific trace. The investigation path then gets shorter: spike on the graph → trace of the slow request → logs of the specific span → root cause.
This is exactly where OpenTelemetry matters more than it seems at the start. It provides a shared context for telemetry signals and keeps you from collecting metrics, logs, and traces as three independent projects.
Where a self-hosted stack is justified
A self-hosted monitoring stack makes sense when the team has a reason to own both the data and the operation.
The first reason is control. In closed networks, regulated environments, B2B products with sensitive logs, or infrastructure without stable outbound access, sending telemetry to an external SaaS may be forbidden or inconvenient.
The second reason is cost at scale. With managed observability, cost often grows with the number of metric series, log ingestion, trace volume, and retention. At a small volume SaaS can be cheaper because it does not require an engineer. At a large volume a self-hosted stack sometimes wins, if the team knows how to manage cardinality, retention, and object storage.
The third reason is independence. Prometheus, Loki, Tempo, and OpenTelemetry let you build a portable architecture: the application does not have to know that today traces go to Tempo and tomorrow part of the telemetry is duplicated to another backend.
The fourth reason is operational learning. A team that has set up its own alerts, dashboards, scrape configs, and retention usually understands its systems better. But that advantage only appears when there is an owner. An abandoned self-hosted monitoring setup is worse than a managed service, because it creates the illusion of control.
Where a self-hosted stack is dangerous
The main risk is to underestimate that monitoring is a production system too. It has to be updated, backed up, secured, scaled, and tested.
The first typical mistake is cardinality. In Prometheus you cannot thoughtlessly add labels like user_id, email, session_id, path with raw parameters, or request_id. Every unique combination of labels creates a new time series. One bad metric can eat more resources than the whole application.
The second mistake is logs without a budget. If every application writes debug level in production, Loki quickly turns into a disk-burning machine. Logs have to be normalized: levels, sampling, retention, exclusion of noisy lines, and a separate policy for audit logs.
The third mistake is noisy alerts. An alert that fires ten times a day and requires no action teaches the team to ignore monitoring. A good alert should mean: a human needs to step in, or a user or the business will suffer.
The fourth mistake is monitoring inside the same failure domain. If all observability lives in the same cluster it is supposed to diagnose, a network or storage outage can take down both the application and your investigation tools. For small projects this is an acceptable trade-off, but you have to understand it.
A minimal starting setup
For a VPS this is the same base set of metrics, logs, Grafana, and Alertmanager from the servers section. There is only one thing worth adding separately: an external uptime check. An internal Prometheus will not always show that the service is unreachable for users on the internet, because it may end up on the same side of the outage.
For Kubernetes the starting setup is different: kube-prometheus-stack, Loki with a DaemonSet agent, Grafana data sources for Prometheus and Loki, Alertmanager routes, and separate PrometheusRules for applications. After that, add the OpenTelemetry Collector and Tempo for the services where latency and inter-service calls genuinely require traces.
Do not start with Thanos, Mimir, multi-cluster federation, a complex HA design, and a year of retention if you have one cluster and a team of a few people. Those are tools for the next stage. The first stage is to see symptoms before the user does, to have working alerts, and to move quickly from a graph to logs.
A practical baseline:
| Scenario | Minimum | When to expand |
|---|---|---|
| Single VPS | Prometheus, node_exporter, Grafana, Loki, Alertmanager | When several services, queues, or frequent latency investigations appear |
| Several servers | A separate monitoring VPS, exporters on the hosts, centralized Loki | When you need HA Prometheus or long retention |
| One Kubernetes cluster | kube-prometheus-stack, Loki, Grafana, Alertmanager | When you need traces, cross-signal correlation, and object storage for long-term retention |
| Several clusters | A separate observability plane, remote write or federation, consistent labels | When teams and environments start getting in each other's way |
Short conclusion
A complete monitoring stack for servers and Kubernetes is not a list of trendy open-source projects. It is an architecture in which Prometheus handles metrics, Loki handles logs, Tempo handles traces, OpenTelemetry handles collection and telemetry portability, Grafana handles investigation, and Alertmanager handles delivering notifications to people.
The self-hosted option is good when you need control, predictable cost, and independence from a managed platform. But it demands discipline: labels, retention, alerts, updates, storage, and owners all have to be thought through from day one.
If you want a short practical piece of advice: start with metrics, logs, and decent alerts. Then add traces and OpenTelemetry where they actually shorten investigations. Monitoring should reduce the time it takes to understand a problem, not become yet another system that nobody has time to monitor.
And if your team or product needs this kind of monitoring but lacks the capacity to build and run it in-house, delegating the setup and operation is a perfectly sound engineering decision. What can be handed off is laid out in our service catalog: from a one-off monitoring and logging setup to ongoing SRE support.