diff --git a/nixos/hosts/README.md b/nixos/hosts/README.md new file mode 100644 index 0000000..3e668b1 --- /dev/null +++ b/nixos/hosts/README.md @@ -0,0 +1,206 @@ +# Hosts + +Per-host NixOS configs for the homelab and admin Mac. Each `.nix` +declares the host's role and services; the `-hardware.nix` siblings +(where present) describe disks, kernel modules, firmware. Bootstrap + +disko configs live one level up in `../`. + +## Topology + +``` + ┌────────────────────────────────────────────┐ + │ vps-relay (Hetzner, public IP, ZT peer) │ +public traffic ──TLS:443──────│ Caddy + Let's Encrypt → reverse_proxy │ + │ over ZeroTier to a clan backend │ + └──────────────────┬─────────────────────────┘ + │ (ZeroTier mesh) + │ + ┌──────────────────────┬───────────┴───────────┬──────────────────────┐ + │ │ │ │ + sunken-ship phantom-ship distant-shore foreign-port + (LAN, wifi) (LAN, wired) (LAN, wifi) (LAN, wifi) + ZT controller NAT for blank slate blank slate + media + mulbo rusty-anchor (room to grow) (room to grow) + services hub + + ── outside the clan ────────────────────────────────────────────────────────────────────── + Daniel-Macbook-Air rusty-anchor + (admin, runs clan-cli) (downstream of phantom-ship's NAT) +``` + +ZeroTier IPv6 addresses for the four clan machines are declared in +`../../flake-modules/clan.nix` (`sunkenShipZTv6` / `phantomShipZTv6` / +`vpsRelayZTv6` / etc.). They land in every host's `/etc/hosts` as +`.clan` so data-mesher and ad-hoc SSH can resolve over the mesh. + +--- + +## Hosts + +### sunken-ship · media + ZT controller + +- **Hardware:** see `sunken-ship-hardware.nix`. WiFi-only, no LUKS (boots + unattended). +- **Network:** LAN over WiFi, on the ZT mesh as the **controller** (manages + ZT membership for the whole fleet). ZT IPv6 is the clan's "internet" + target for `clan machines update`. +- **Role:** media + the long-running personal services that don't fit on + phantom-ship. +- **Current services:** `navidrome` (subsonic API, `/srv/music`), `uxplay` + (AirPlay receiver), `mulbo-server` (+ `-pull` / `-backfill` / `-enrich` + timers), `fitness-bot` (+ `-pull` / `-shipyard` variants), + `dm-pull-deploy-push` (announces origin/main rev to the mesh every 15 m). + +### phantom-ship · services hub + LAN NAT + +- **Hardware:** see `phantom-ship-hardware.nix`. WiFi for WAN, wired + ethernet (`enp0s31f6`) serves the lab subnet (NAT + dnsmasq for + `rusty-anchor`). +- **Network:** LAN over WiFi, on the ZT mesh. Backends are exposed only on + the ZT interface (`firewall.interfaces."zt+".allowedTCPPorts = [ … ]`) + so vps-relay's Caddy can reach them. WAN side stays closed. +- **Role:** where new self-hosted apps default to going. Hosts a growing + list of mini-app backends + a couple of long-running daemons. +- **Current services:** `forgejo` (`git.dannydannydanny.me`), + `claude-channels` (Telegram bridge for `@HarakatBot`), + `hara-gmail-mcp` + `hara-heartbeat` (timer), + Mini-App backends (`shelfish`, `scuttle`, `bananasimulator`, + `komtolk`, `escape-hormuz`, `bon`), `ollama` (local LLM), `shipyard`, + `dnsmasq` (lab subnet DHCP/DNS). `openclaw-gateway` is disabled — + superseded by `claude-channels` but kept for easy rollback. + +### vps-relay · public reverse proxy + +- **Hardware:** Hetzner Cloud vServer (BIOS-boot, virtio). Disk via + `../disko-cloud.nix`. +- **Network:** public IP `89.167.39.251`. Inbound: SSH/22, HTTP/80, + HTTPS/443 only. fail2ban guards SSH. Outbound to clan backends over ZT. +- **Role:** terminates public TLS, reverse-proxies subdomains over ZT to + whichever clan host runs the backend. **No application data ever lands + here** — this box is a relay. New public app = add a `virtualHosts` + entry + a GoDaddy A record pointing at `89.167.39.251`. +- **Current vhosts:** `navidrome.`, `bbbot.`, `shelfish.`, `scuttle.`, + `bananasimulator.`, `komtolk.`, `git.`, `escapehormuz.`, etc. + +### distant-shore · ThinkPad X13 Gen 2, blank slate + +- **Hardware:** see `distant-shore-hardware.nix`. Intel i5-1145G7, 16 GB. + WiFi-only, headless, no LUKS. Secure-Boot-chained boot (shim + MOK, + see comments in `distant-shore.nix`). +- **Network:** LAN over WiFi, on the ZT mesh. +- **Role:** _to be assigned_. In the clan inventory; auto-rebuilds via + dm-pull-deploy. Drop a service in to give it a purpose. + +### foreign-port · laptop, blank slate (WIP) + +- **Hardware:** see `foreign-port-hardware.nix`. WiFi-only, headless, + no LUKS. Vendor-signed-shim boot chain. +- **Status:** still being wired up — not in the clan inventory yet. +- **Role:** _to be assigned_, same flow as `distant-shore`. + +### daniel-macbook-air · admin + +- **Hardware:** MacBook Air (the daily driver). +- **Role:** outside the clan. Runs `clan machines update` to push to + the servers + holds the SSH keys that authorize root@ on each clan + host. Also a ZT peer. + +### wsl + +- **Role:** WSL development environment (legacy / occasional). + +--- + +## Deployment + +### Automatic (the default) + +`dm-pull-deploy` (clan-community module wired in `clan.nix`): + +1. **Push announcement:** sunken-ship's `dm-pull-deploy-push` timer runs + `dm-send-deploy` every 15 m. It signs and broadcasts the current + `origin/main` rev over the data-mesher gossip protocol. +2. **Pull + rebuild:** each `roles.default` machine (currently + `sunken-ship`, `phantom-ship`) runs a `.path` watcher that fires when + the gossiped rev changes; it `git fetch`es and `nixos-rebuild switch`es. + +So **a push to `origin/main` rolls out within ~15 m** on the two +production hosts. No SSH-from-Mac required. + +`vps-relay` and `distant-shore` are **not** in `roles.default` — they +need a manual deploy (see below) until/unless their role changes. + +### Manual + +From `~/dotfiles` on the Mac: + +``` +nix run 'git+https://git.clan.lol/clan/clan-core#clan-cli' -- \ + machines update +``` + +Caveats encountered in practice: + +- The Mac's `ssh-agent` often has the wrong key loaded for clan deploys. + Prefix with `env -u SSH_AUTH_SOCK` to force `~/.ssh/config` identity + selection. +- A nixpkgs bump may register a new generation but refuse to live-switch + due to "switch inhibitors". Add `--no-check` to force. +- `vps-relay` only accepts `~/.ssh/id_ed25519_sunken_ship` (the Mac's + copy of sunken-ship's authorized key). The agent's other keys won't + open it. + +Putting both together: + +``` +env -u SSH_AUTH_SOCK nix run 'git+https://git.clan.lol/clan/clan-core#clan-cli' \ + -- machines update phantom-ship --no-check +``` + +### From sunken-ship + +`vps-relay` was originally only reachable from sunken-ship's SSH key. +That still works as a fallback — SSH to sunken-ship and run the same +`nix run … -- machines update vps-relay` command from `/etc/dotfiles` +there. The dotfiles checkout at `/etc/dotfiles` is maintained by +dm-pull-deploy. + +--- + +## Public traffic pattern + +``` +user → DNS *.dannydannydanny.me → 89.167.39.251 (vps-relay) + → Caddy (Let's Encrypt, ports 80/443) + → reverse_proxy http://[]: + → service on sunken-ship or phantom-ship +``` + +To add a new public app: + +1. Add a `virtualHosts` entry to `vps-relay.nix` pointing at the + backend's ZT IPv6 and port. +2. Add the GoDaddy A record `.dannydannydanny.me → 89.167.39.251`. +3. Run the backend on the chosen host. Either: + - bind to `127.0.0.1:` (if backend + Caddy are co-resident — not + the case here), **or** + - bind to `0.0.0.0` (or `::`) and add the port to + `networking.firewall.interfaces."zt+".allowedTCPPorts` on the + backend host so only the ZT interface accepts inbound. +4. Push dotfiles. Production hosts auto-rebuild via dm-pull-deploy. + vps-relay needs a manual `clan machines update vps-relay`. + +--- + +## SSH keys (quick reference) + +- **`~/.ssh/id_ed25519_phantom_ship`** (Mac) — authorized as `danny@` and + `root@` on phantom-ship. +- **`~/.ssh/id_ed25519_sunken_ship`** (Mac) — authorized as `danny@` (and + via root mirror) on sunken-ship; also the authorized key on `vps-relay`. +- **sunken-ship `~/.ssh/id_ed25519`** — sunken-ship's own key; used by + cluster-internal ops (mulbo-pull, dm-send-deploy, fallback path for + vps-relay deploys). +- **`~/.ssh/id_ed25519_github`** (Mac) — GitHub auth, not clan. + +Authorized-keys lists live in each host's `users.users.{danny,root}.openssh.authorizedKeys.keys`.