Logs are the stepchild of many systems. You rarely need them — but when you do, you need them urgently. In Kubernetes clusters with dozens of services and ephemeral pods, log management quickly becomes a nightmare unless you invest in suitable tooling.
Why not Elasticsearch
We lived with an Elasticsearch-based stack for a long time. ES is powerful but has two real-world problems: it is expensive (RAM-hungry, scales linearly with data volume) and it tempts teams into building complex index strategies that nobody wants to maintain later.
For a mid-sized setup (our scale: ~50 GB of logs per day across 20 services), ES was overkill. Every search meant background full-text indexing, and the resulting indices consumed three times the space of the raw data.
How Loki works differently
Loki is fundamentally different: it indexes only metadata (labels), not log content. That makes Loki compact and cheap — a cluster that runs on 8 GB of RAM where ES needed 64 GB.
The trade-off: full-text search is slower. In practice this is acceptable, because good label hygiene (by service, namespace, severity) usually narrows the search before full-text comes into play.
Our label schema
We agreed on five standard labels that every service MUST emit:
service— name of the service (e.g.web-core-gateway)namespace— Kubernetes namespaceenv— prod/staging/devlevel— debug/info/warn/errorrequest_id— only for service logs with request context
request_id is the most important one. It enables cross-service correlation: a request hits the gateway, moves on to the orchestrator, then to three agents. All logs for that request carry the same request_id and can be combined into a timeline in Grafana.
The "LogQL moment"
LogQL is Loki's query language. It mirrors PromQL and feels familiar to Prometheus users. The decisive moment in the team was when we learned to use | json and | unwrap — JSON logs can be parsed and filtered directly in LogQL.
A typical query for us:
{service="web-core-gateway"} | json | duration > 1s
That gives us all slow requests in the gateway in seconds. No index preparation, no aggregation pipeline.
Grafana as the cockpit
We have seven dashboards we use daily:
- Cluster health: CPU/memory/disk per node
- Service health: up/down, latency p50/p95/p99 per service
- Error rate: 5xx rate per endpoint, top-10 errors
- RAG pipeline: retrieval latency, LLM latency, token usage
- Auth events: login successes/failures, token rotations
- Content pipeline: indexer throughput, index freshness
- Business KPIs: active sessions, conversion rates
The key: each dashboard answers a specific question. No dashboards filled with "anything that might be useful".
Alerting: less is more
We deliberately keep alerts few. The rule: an alert must provide a clear action signal. An alert that only says "something is weird" gets muted within two weeks and never looked at again.
Our alert list:
- Service down > 2 minutes → Slack + PagerDuty
- 5xx rate > 1% for 5 minutes → Slack
- Disk > 85% → Slack + ticket
- Cert expiry in < 7 days → Slack
That's it. Everything else lives in the dashboards.
Outlook
The next step is OpenTelemetry traces to better understand cross-service latencies. Loki + Tempo + Mimir (all Grafana stack) should provide this without adding a second observability provider.