MiLog

The monitor for the box you run yourself.

One bash file watches your logs, your host, your file integrity and your kernel, then tells you on Discord when something changes. No agent fleet, no collector, no retention tier.

Written for people who run one server, or three, and read the logs themselves. This page is also the command reference.

v0.6.0 MIT, single file bash 4+, Linux apt / dnf / yum / pacman / apk

curl -fsSL https://raw.githubusercontent.com/chud-lori/milog/main/install.sh | sudo bash

Prefer to read it first? install.sh is plain bash. It runs on Linux only and exits on any other OS before installing anything.

SURFACES

Six things it watches, one alert pipeline

Everything below fires through one alert pipeline, with the same per-rule cooldown, silences, routing and hooks. Request-log rules also share a dedup keyed on (ip, path), so a line that matches two rules alerts once. Turning on a new surface does not mean a new place to configure alerts.

Request logs nginx, journald, docker, text

nginx combined format gets the full treatment. Other sources are tailed, grepped and searched, and skip the parsing modes.

  • Per-app req/min, 2xx through 5xx, p50 / p95 / p99
  • Top paths, top source IPs, slowest endpoints
  • Anomaly against the same minute over the last 14 days

Host vitals /proc

Read straight from the kernel, so nothing extra has to be installed.

  • CPU, memory, disk, worker count against your thresholds
  • No sar, no iostat, no collector

Attack payloads heuristic

Pattern matching over request lines, tuned for what actually hits a small box.

  • LFI, RCE, SQLi and XSS shapes in URLs
  • Infra probes: .git, .env, wp-admin, phpMyAdmin
  • CMS scans and dotfile probes

Scanners user-agent catalog

Ranked by behaviour, not just by name, so a renamed agent still surfaces.

  • zgrab, masscan, nikto, sqlmap
  • Shodan, CensysInspect, bare HTTP libraries
  • AI crawlers: GPTBot, ClaudeBot, CCBot

Host integrity milog audit, any host

Baseline once while the box is known good, then diff. Needs no web server at all, so it works on a host that serves nothing.

  • FIM over passwd, shadow, sudoers, crontab, sshd_config, authorized_keys
  • Persistence: cron dirs, systemd services and timers, rc.local, ld.so.preload
  • Listening ports, passwd and sudoers, SSH keys
  • YARA over the webroot, rootkit heuristics

Kernel events eBPF sidecar, Linux

An optional Go binary. Catches the short-lived process that a log tail never sees.

  • exec: a shell under a web worker, a binary run from /tmp
  • tcp connects to destinations outside the allowlisted CIDRs, except milog's own alert sends
  • file opens of shadow, sudoers and SSH config by processes outside the allowlist
  • ptrace, kmod, bpf-load
  • Retransmits and syscall rate, Welford sigma

REFERENCE

Every command

The same list milog help prints. Any command takes --help for its own detail. Before any log source is configured, only help, config, doctor, version, update, install, completions, alert, alerts, silence, audit, probe and bench run; the rest exit and point at milog config init.

DASHBOARDS

milog monitor
Bash dashboard: requests, CPU, memory, disk, workers. q quit · p pause · r refresh · +/- rate
milog tui
Richer Charm TUI, same data. needs the milog-tui Go binary
milog rate
Requests per minute only, nothing else on screen.
milog daemon
Headless alerter. No TUI, fires your webhooks.

ANALYSIS

milog health
2xx, 3xx, 4xx and 5xx counts per app.
milog top [N]
Busiest source IPs. default 10
milog top-ip-by-app [N]
Busiest source IPs, split per app. default 5
milog top-paths [N]
Requests, 4xx, 5xx and p95 per URL. default 20
milog attacker <ip>
Everything one address did, across every app.
milog slow [N]
Slowest endpoints by p95. needs $request_time; excludes websockets
milog ws [N]
WebSocket sessions: count, duration, busiest paths.
milog stats <app>
Hourly request histogram for one app.
milog suspects [N] [W]
Heuristic bot ranking. default top 20, window 2000 lines per app
milog trend [app] [H]
Sparkline of req/min from stored history. default all apps, 24h
milog diff
Requests now against 1 day ago and 7 days ago.
milog auto-tune [D]
Suggest thresholds from your own history. default 7 days
milog replay <file>
Postmortem summary for one archived log file.
milog search <pattern>
Grep every app at once. --since --app --path --regex --archives

ALERTING

milog alert on [url]
Turn alerts on and install milog.service, which runs as the user who ran sudo. Discord webhook URL; sudo for the service
milog alert off
Turn alerts off and stop the service.
milog alert status
Webhook, service and recent-fire state.
milog alert test
Send one test message now.
milog alert stats [window]
Fires per rule, busiest first. default 7d
milog alerts [window]
What fired locally. today · Nh · Nd · Nw · all
milog silence
Mute one rule while you fix the cause. milog silence --help
milog digest [window]
Summary for the window. day · week · Nh · Nd
milog report [window]
Static markdown report of traffic, top IPs, alert fires, anomalies and audit drift. default 7d · --html for one self-contained page · -o FILE

TAILING

milog logs
Tail every source, colour prefixed by app. also what bare milog does
milog <app>
Raw tail of one source.
milog grep <app> <pattern>
Filtered tail of one source.
milog errors
4xx, 5xx and app-error patterns live. --since for a summary instead
milog exploits
LFI, RCE, SQLi, XSS and infra-probe payloads only.
milog probes
Scanner and bot traffic only.
milog patterns
App error signatures: panics, OOM, stack traces. milog patterns list prints the catalog

HOST INTEGRITY

milog audit fim
File integrity: baseline, then drift.
milog audit persistence
New cron, systemd units, rc.local: the re-entry surface.
milog audit ports
New TCP and UDP listeners against a baseline.
milog audit accounts
Line-level diff of passwd, sudoers and SSH keys.
milog audit yara
YARA over the webroot: webshells and obfuscation.
milog audit rootkit
Hidden processes, ld.so.preload, exec from tmp.
milog probe status
Is the eBPF sidecar running? Linux only
milog probe install-service
Install and start the sidecar unit. needs sudo
milog probe uninstall-service
Remove the sidecar unit. needs sudo

WEB DASHBOARD

milog web
Start the read-only dashboard. data routes need the token, /static does not; loopback only unless --bind ADDR --trust
milog web stop
Stop it, whether systemd or foreground started it.
milog web status
Running? On which port?
milog web install-service
Install and start the systemd user unit.
milog web uninstall-service
Remove that user unit.
milog web rotate-token
Regenerate the web token in place.

CONFIG AND DIAGNOSTICS

milog config
Resolved config, and the path it came from.
milog config init
Write a template config file.
milog config validate
Typos, out-of-range values, paths that do not exist.
milog config add <app>
Append a source to LOGS.
milog config rm <app>
Remove a source from LOGS.
milog config dir <path>
Set LOG_DIR.
milog config set <k> <v>
Set any variable: REFRESH, THRESH_*, and the rest.
milog config edit
Open the config in $EDITOR.
milog doctor
Checklist: tools, logs, log format, alerting, history, geoip, web, systemd. counts alert deliveries that failed in the last 24h
milog bottleneck
One-shot host-slowdown audit: names the saturated resource and the processes behind it. --json
milog version
Version stamp of milog and of each companion binary found.
milog update [--check]
Replace milog and its installed companion binaries with the latest GitHub release. checksum-verified · --check exits 10 when an update exists
milog install <feature>
Add an optional feature: geoip, web, history.
milog update-rules
Fetch newer exploit and probe rules from the latest release.
milog bench [--full]
Benchmark against synthetic fixtures.
milog completions <shell>
Install or print bash, zsh or fish completions.
milog help
Print the command list.

CONFIG

Point it at your logs

A bare name means an nginx access log in LOG_DIR. A prefix means something else, which is how MiLog watches an app that never touches nginx.

# ~/.config/milog/config.sh
LOGS=(
  api                                  # $LOG_DIR/api.access.log
  nginx:storefront                     # the same thing, said explicitly
  text:worker:/var/log/worker/out.log  # any text file
  journal:mybot.service                # streams via journalctl
  docker:postgres-prod                 # the container's stdout
)
SourceResolves toSupports
api$LOG_DIR/api.access.logeverything
nginx:apithe same file, explicitlyeverything
text:name:/paththat filetail, grep, search, errors, patterns
journal:unitjournalctl -u unit -ftail, grep, search, errors, patterns
docker:namethe container JSON logtail, grep, search, errors, patterns

The counting modes need the nginx combined format to read a status code and a latency, so they stay with nginx sources. Host integrity audits and the kernel probe read no logs at all and work on any host.

Environment overrides, every key

These environment variables win over the config file, so one run can differ without editing anything. MILOG_APPS replaces LOGS.

MILOG_CONFIG            MILOG_LOG_DIR           MILOG_REFRESH
MILOG_ALERTS_ENABLED    MILOG_ALERT_ROUTES      MILOG_ALERT_COOLDOWN
MILOG_ALERT_DEDUP_WINDOW  MILOG_ALERT_HOOK_TIMEOUT  MILOG_ALERT_LOG_MAX_BYTES
MILOG_DISCORD_WEBHOOK   MILOG_SLACK_WEBHOOK     MILOG_TELEGRAM_BOT_TOKEN
MILOG_TELEGRAM_CHAT_ID  MILOG_MATRIX_HOMESERVER MILOG_MATRIX_ROOM
MILOG_MATRIX_TOKEN      MILOG_WEBHOOK_URL       MILOG_WEBHOOK_TEMPLATE
MILOG_WEBHOOK_CONTENT_TYPE  MILOG_HOOKS_DIR
MILOG_APPS              MILOG_AUDIT_ENABLED     MILOG_AUDIT_FIM_INTERVAL
MILOG_AUDIT_PERSISTENCE_INTERVAL  MILOG_AUDIT_PORTS_INTERVAL
MILOG_AUDIT_YARA_INTERVAL  MILOG_AUDIT_YARA_PATHS  MILOG_AUDIT_YARA_RULES_DIR
MILOG_AUDIT_ACCOUNTS_INTERVAL  MILOG_AUDIT_ACCOUNTS_PATHS
MILOG_AUDIT_ROOTKIT_INTERVAL
MILOG_HISTORY_ENABLED   MILOG_HISTORY_DB        MILOG_ANOMALY_ENABLED
MILOG_ANOMALY_SIGMA     MILOG_ANOMALY_MIN_DAYS  MILOG_ANOMALY_FLOOR_REQ
MILOG_ANOMALY_FLOOR_C5XX  MILOG_ANOMALY_FLOOR_P95
MILOG_PATTERNS_ENABLED  MILOG_SLOW_EXCLUDE_PATHS
MILOG_GEOIP_ENABLED     MILOG_MMDB_PATH         MILOG_DOCKER_ROOT
MILOG_CROWDSEC_CTI_KEY
MILOG_WEB_PORT          MILOG_WEB_BIND

Each one is described in docs/configuration.md.

Getting it running

Step 1 is the whole install. The rest are the things worth turning on afterwards, in the order they pay off.

  1. Install

    curl -fsSL https://raw.githubusercontent.com/chud-lori/milog/main/install.sh | sudo bash

    Linux only. Drops milog into /usr/local/bin, pulls gawk, curl and sqlite3 through apt, dnf, yum, pacman or apk, and stops with an error on a host with none of those. The milog-web, milog-tui and milog-probe binaries come from the latest release, and each archive is checked against that release's checksums.txt (SHA-256); a mismatch aborts the install. checksums.txt is not signed, so this catches a corrupted or swapped download, not a compromised release. milog.sh itself is fetched from the main branch and is not checksummed.

    The release page also has .deb, .rpm and .apk packages for amd64 and arm64.

  2. Look at it

    milog doctor     # what is missing before anything else
    milog monitor    # the dashboard

    Run doctor first. It checks the log format, which is the one thing that quietly halves what MiLog can tell you.

  3. Get told about it

    sudo milog alert on 'https://discord.com/api/webhooks/ID/TOKEN'
    milog alert test

    alert on writes the Discord webhook to your config and installs milog.service, which runs as the user who ran sudo. alert test sends one message to every configured destination. Slack, Telegram, Matrix and a generic webhook are set with milog config set, described in docs/alerts.md; alert on still wants a Discord webhook before it installs the service.

  4. Baseline the host while it is still clean

    milog audit fim baseline
    milog audit ports baseline
    milog audit accounts baseline

    Drift detection compares against a known-good snapshot, so a baseline taken after a compromise records the compromise as normal. This is the step worth doing today rather than eventually.

  5. Bank history, then let it judge

    milog config set HISTORY_ENABLED 1
    sudo systemctl restart milog.service
    # two weeks later
    milog config set ANOMALY_ENABLED 1
    milog auto-tune 7

    The anomaly detector compares each minute against the same minute over the previous days, so it needs days before it says anything. It stays quiet until it has them.

  6. Kernel events optional, Linux

    sudo milog probe install-service
    sudo journalctl -u milog-probe.service -f

    Catches what a log tail structurally cannot: the process that started and exited between two reads. Same webhooks, same cooldown. The unit runs the probe as root, limited by a systemd capability bounding set, and the alerts it raises run as the user who ran sudo, through that user's config. Details in docs/probe.md.

GUIDES

Longer form

Each guide opens with what the thing is, then why you would turn it on, then how, with the failure modes named.

PRINCIPLES

How it is built, and why