My homelab is a laptop that refuses to die

My homelab is boring hardware running paranoid software. The main node is an M2 Max MacBook Pro with 64 GB of RAM and a 16 TB external drive, wired into the router. Laptops make underrated home servers: the battery is a built-in UPS, the machine is silent, it idles under ten watts where rack gear budgets hundreds, and this one was already paid for. Two more nodes complete the fleet, a Raspberry Pi and a free ARM VM in someone else’s datacenter, and most of what follows is about how three machines keep each other honest.

The request path has one weird hop

About twenty-five containers run the usual self-hosting roster plus my own work: Immich for photos, paperless-ngx for documents, Home Assistant, audiobooks in Audiobookshelf, PDF tooling from Stirling PDF, Glance dashboards, a local LLM front end in Open WebUI, and SnapOtter, which I develop and dogfood here. Nginx Proxy Manager terminates TLS, Pi-hole on the Raspberry Pi filters DNS, Tailscale carries everything administrative, and the router forwards exactly two ports.

An external request does not go straight to Docker, and the reason is the most macOS-specific fact in this post. Docker Desktop runs containers inside a VM and NATs every forwarded connection, so by the time a request reaches a containerized proxy, every client on earth appears to come from the same gateway address. Tooling that reads those logs to decide who to ban learns nothing. So a native nginx on the host owns ports 80 and 443, logs real client addresses, and forwards into Docker, where NPM does TLS and routing. Two proxies deep before a service sees a byte, and each layer exists for a reason the other cannot cover.

Docker’s own defaults needed correcting twice more. Its address pool overlaps the same private ranges home routers use, so the daemon is pinned to 172.16.0.0/12 before it collides with the LAN, and a Docker engine update quietly broke host.docker.internal resolution for nginx’s resolver, so the proxy targets the VM’s gateway address directly. Neither fix appears in any tutorial; both came out of debugging evenings.

What it’s actually for

All that plumbing exists to protect things that are almost boringly ordinary. Immich holds the family photo library and is used by relatives abroad, which is the actual reason it has to stay public instead of hiding behind Tailscale like everything administrative. paperless-ngx OCRs every scanned bill, form, and letter into something you can actually search; “where did I put that visa paperwork from two years ago” stopped being a real question. Home Assistant runs the house itself: four smoke and CO alarms feed into one alert path, the CCTV camera records 30-second motion clips with a 5-second lookback and prunes itself after 30 days, and the air conditioners, fans, and balcony blinds take a schedule or a voice command instead of a remote somebody has to go find. Audiobookshelf plays audiobooks and podcasts to whichever phone is nearby. Open WebUI talks to a locally running Ollama, and asking a model a question never has to leave the apartment. And SnapOtter, the file-processing tool I build for a living, runs here first, on my own files, catching most bugs before a customer does.

CrowdSec decides, pf enforces

CrowdSec tails the native nginx logs with the nginx, base-http-scenarios, and http-cve collections, merges its findings with a community blocklist about fifteen thousand addresses deep, and publishes ban decisions on a localhost-only API. Enforcement happens at the kernel: a pf anchor holds one persistent table, and a rule drops matching traffic before any application sees it.

table <crowdsec_blocklist> persist
block drop in quick from <crowdsec_blocklist> to any

CrowdSec ships no pf bouncer for macOS, so mine is forty lines of shell on a root launchd daemon: every thirty seconds, pull the decision list, write it to a file, and pfctl -T replace the table wholesale. Wholesale replacement is the reliable choice here: the table always equals the current decision list, there is no drift to reconcile, and a crashed cycle heals itself thirty seconds later. The table usually holds a few hundred addresses that earned their place.

The database that died for eight days

The reliability engineering exists because of one incident. The Immich Postgres data directory used to live on the 16 TB USB drive, the drive dropped for a moment in August, and the database corrupted. Nothing alerted, because the web app kept returning HTTP 200 while its database lay dead behind it, and the outage ran eight days before a human noticed the photos were gone.

Every fix since traces back to that lie. The database moved to the internal SSD, so a USB dropout now costs photo visibility and never data integrity. Uptime Kuma grew container-level monitors on a read-only Docker socket, because an HTTP check believes a service that answers, and a container check believes only a process that runs. And the drive itself got a watchdog with hands.

Watchdogs that fix things, and know when to stop

Seven launchd jobs patrol the machine on tight loops, and the interesting ones remediate before they page.

The drive watchdog checks the 16 TB mount every sixty seconds; on a dropout it attempts three remounts a minute apart, restarts the three stacks that depend on the volume, verifies their health, and only then reports the outcome. It also refuses to restart containers while the drive is absent, because healing into a missing mount just manufactures a second incident.

The container watchdog is the one with the subtle trick. A Docker restart: always policy revives a stopped container but reuses its cached configuration, so if a bind mount vanished, the restart fails exactly where the crash did. The watchdog instead runs docker compose up -d --force-recreate --no-deps, which rebuilds the container object and re-evaluates every mount as if fresh. And it carries a flap guard: two heals per container per six hours, then one “flapping” alert and a deliberate stop, because a watchdog without a give-up threshold is an outage amplifier with good intentions.

The bluntest job is a scheduled Sunday 04:30 restart of Docker Desktop itself. Five wedges in three weeks, every one on a VM that had been up for weeks, is an empirical argument that macOS Docker uptime is a failure mode, so uptime now caps at seven days by appointment. macOS had its own time bomb too: the OS shipped with automatic update installs enabled, meaning it could reboot itself into a FileVault unlock screen at 3 a.m. and take the whole lab down until a human typed a password. That setting is off now, FileVault stays on deliberately, and the rare unattended reboot is a risk the external watchdogs carry.

Which is where the other two machines come in. The Raspberry Pi pings the Mac on a cron loop and alerts if the whole machine goes dark, the one failure the Mac’s own watchdogs can never report, and the off-site ARM VM holds the dead man's switch pointing the other way. The Mac watches the Pi back. Every node has another node that will notice its death, which is the closest a three-machine lab gets to consensus.

Backups that have actually been restored

The nightly job at 03:00 dumps the Immich database with pg_dumpall, tars ten named Docker volumes through throwaway Alpine containers, and snapshots the SQLite databases with sqlite3 .backup, because copying a live database file is how you back up corruption. Every artifact gets verified on the spot, gzip -t and tar -tzf, and the job aborts loudly if free disk drops below its floor.

Secrets and the small databases are encrypted with age before they leave the machine and pushed to a private git vault with a trick I like: each push is an orphan commit force-pushed over the last, so the repo carries tonight’s backup and no history, and stays small forever. Oversized artifacts get slimmed to fit, including the uptime monitor’s database, which drops from hundreds of megabytes to a few once the heartbeat history tables are purged. The age key lives in the password manager, which is the recovery path if the laptop itself is gone.

The one thing the vault and the nightly dump don’t cover is the 16 TB of photos and documents that make the lab worth protecting in the first place, and that’s Backblaze’s job. Its Personal Backup client runs continuously in the background and covers both the internal SSD and the external drive, but it only sees ordinary files. Docker Desktop’s VM disk image is opaque to it, so a named Docker volume living inside that image never shows up in a Backblaze scan, which is exactly why the nightly script tars each one out to a plain file first. The Immich database dump is Backblaze’s alone to protect; at 772 MB it is too big for the git vault’s 90 MB ceiling. Default version history is 30 days. That’s longer than the eight days the corrupted database sat unnoticed, but not by a margin I trust, so it’s extended to a year for about two dollars a month. None of it is fast: restoring multiple terabytes over residential fiber takes days, and Backblaze’s restore-by-mail option exists for exactly that reason.

Two safeguards make the pipeline trustworthy. The job pushes a heartbeat to the uptime monitor on success, and silence for twenty-six hours raises an alert, so a quietly broken backup gets one day to exist. And a restore-test script loads the newest dump into a scratch container and asserts the photo count is nonzero, using the same Postgres image as production, because Immich’s vector extensions mean a generic Postgres would pass the test and fail the real restore.

The dashboard survives the thing it monitors

An old phone mounted under the monitor runs a kiosk dashboard served by a container, which raises an obvious question: what does the dashboard show when Docker itself dies? A small stats agent runs on the host, outside Docker, and the kiosk page detects the dashboard going quiet, covers the stale frame with a plain “Docker is down” banner, and keeps rendering live CPU and memory from the agent. Observability that dies with the thing it observes is decoration.

What the lab taught

1. Docker on macOS launders every attacker into the gateway IP, so the first hop must live outside Docker. The native-nginx-plus-pf sandwich exists because of that one NAT fact.
2. HTTP 200 is not health. Immich answered cheerfully for eight days while its database was dead; monitor the process and the query path, and treat a green HTTP check as the weakest witness.
3. restart: always heals crashes and preserves the cause. Recreating the container re-evaluates mounts and config; restarting it replays them.
4. A watchdog needs a give-up threshold. Two heals per six hours, then alert and stand down, or the healer becomes the incident.
5. A backup without a heartbeat fails silently, and a backup without a restore test is a rumor with a filename. Test restores against the production image, because extensions make a generic database a false witness.
6. Cap uptime when the evidence says so. Five Docker wedges in three weeks bought a standing Sunday restart, and the wedges stopped.
7. A dead server cannot email you. Every monitor needs one watcher outside its own failure domain, which is why a Raspberry Pi and a free VM each hold a piece of the alerting.