Hathcock Homelab Wiki

Loki

2 min read

Loki Log Aggregation

← Back to Monitoring Stack

IP: 192.168.1.XXX:3101

Overview

Centralized logging using Grafana Loki. Collects logs from all Docker containers across both ProxMoxBox and Pi5. System logs are available via journalctl directly on each host.

Access

http://192.168.1.XXX:3101

Components

ServiceHostPortPurpose
LokiProxMoxBox3101Log database
PromtailProxMoxBox1514 (syslog)Local log collector
PromtailPi5-Remote log collector

Logs Collected

✅ Working:

  • Docker container logs from ProxMoxBox (/var/lib/docker/containers/)
  • Docker container logs from Pi5 (via remote Promtail)
  • Synology NAS syslog - 🟡 Partial (RFC 3164 format causes parse warnings, but logs arrive and are searchable)

❌ Not Collected:

  • System logs - Use journalctl directly on each host
  • Auth logs - Use journalctl -u sshd or journalctl -g "authentication" on each host

Note: Ubuntu 24.04’s systemd-journald does not forward to rsyslog by default. Docker container logs provide the most valuable insights for homelab monitoring.

Stack Locations

# ProxMoxBox
/opt/monitoring/
├── loki/
│   └── loki-config.yml
└── promtail/
    └── promtail-config.yml

# Pi5
~/promtail/
├── docker-compose.yaml
└── promtail-config.yml

Useful LogQL Queries

{host="pi5"}                      # All Pi5 Docker logs
{host="proxmoxbox", job="docker"} # ProxMox Docker logs
{job="syslog-nas"}                # Synology NAS logs
{filename=~".*grafana.*"}         # Grafana container logs
{filename=~".*prometheus.*"}      # Prometheus container logs
{stream="stderr"}                 # All container stderr logs

Retention

  • 30 days for all logs

Docker Compose

loki:
  image: grafana/loki:latest
  container_name: loki
  restart: unless-stopped
  ports:
    - "3101:3100"
  volumes:
    - ./loki/loki-config.yml:/etc/loki/local-config.yaml:ro
    - loki_data:/loki
  command: -config.file=/etc/loki/local-config.yaml
  deploy:
    resources:
      limits:
        memory: 512M  # Current usage ~158 MB (31%)
 
promtail:
  image: grafana/promtail:latest
  container_name: promtail
  restart: unless-stopped
  ports:
    - "1514:1514"  # Syslog receiver for NAS
  volumes:
    - ./promtail/promtail-config.yml:/etc/promtail/config.yml:ro
    - /var/lib/docker/containers:/var/lib/docker/containers:ro
  command: -config.file=/etc/promtail/config.yml
  deploy:
    resources:
      limits:
        memory: 128M  # Current usage ~45 MB (35%)

Synology NAS Syslog - 🟡 PARTIAL

Status: Messages arriving but format mismatch causes parse warnings.

Issue

  • Synology sends BSD format (RFC 3164)
  • Promtail expects RFC 5424 format with version number
  • Parse warnings: expecting a version value in the range 1-999

Future Enhancement

  • Add syslog relay container for RFC 3164 → RFC 5424 conversion
  • Or use Promtail file scraping via NFS mount of /var/log

Decision: Accepted partial implementation - messages arriving and searchable in Loki, format conversion can be addressed later if needed.

Grafana Integration

Loki datasource is provisioned automatically in Grafana. Access via:

  • Loki Logs dashboard
  • Homelab Overview dashboard (logs panel)
  • Explore tab in Grafana

Graph View

Backlinks