**Loki** is Grafana Labs' horizontally scalable, multi-tenant log aggregation system designed to be cost-effective and easy to operate. Unlike Elasticsearch, Loki does not index the full content of log lines — it only indexes labels (metadata) and stores log chunks compressed in object storage. This dramatically reduces storage and indexing costs. --- ### First Principle: Logs are expensive to index. Index metadata, not content. Query the rest when you need it. Full-text indexing makes every word searchable but is expensive — storage, memory, and CPU all grow with log volume. Loki indexes only the labels you attach (pod name, namespace, service name) and compresses raw log streams. When you query, Loki filters by labels first (cheap) then grep's matching streams (fast enough for most cases). --- ### Key Considerations - **Label-Based Model**: Logs are stored as streams identified by a set of labels (`{app="nginx", namespace="prod"}`). Good label design is crucial — high cardinality labels cause performance issues. - **LogQL**: Loki's query language. Supports label filtering, line filters (`|= "error"`), and metric extraction (`rate`, `count_over_time`) — similar syntax to PromQL. - **Promtail / Alloy**: Promtail (classic log shipper) or Grafana Alloy ([[OpenTelemetry]]-compatible) scrape logs from files and Kubernetes pods and ship them to Loki. - **Object Storage Backend**: Loki stores chunks in [[MinIO]], [[Ceph]], or S3 — making log retention cost scale with storage prices, not compute. - **[[Grafana]] Integration**: Loki is a native Grafana data source. Within a single dashboard, operators can correlate [[Prometheus]] metrics with Loki log lines at the same point in time. --- ### How It Fits ``` Pods / systemd services (log output) → Promtail / Alloy (ships logs to Loki) → Loki (stores compressed chunks in [[MinIO]]) → [[Grafana]] (query via LogQL, correlate with metrics) ``` [[Grafana]] | [[Prometheus]] | [[OpenTelemetry]] | [[MinIO]] | [[Open Source Hyperscaler MoC]]