nixos/hosts: add README documenting host roles + deploy flow
Deferred Phase-1-completion task from the de-platform-from-GitHub roadmap (vimwiki/diary/2026-05-03.md). Documents: - per-host role + current services (sunken-ship, phantom-ship, vps-relay, distant-shore, foreign-port, daniel-macbook-air, wsl); - ZT mesh topology + ASCII overview; - the auto-rebuild path (dm-pull-deploy push from sunken-ship → pull/ rebuild on roles.default hosts within ~15 m); - the manual clan-cli flow, including the env -u SSH_AUTH_SOCK and --no-check gotchas we hit in practice; - the vps-relay reverse-proxy pattern for new public apps; - SSH key quick-reference.
This commit is contained in:
DannyDannyDanny
parent
05896f6d3b
commit
a903d76f65
1 changed files with 206 additions and 0 deletions
206
nixos/hosts/README.md
Normal file
206
nixos/hosts/README.md
Normal file
|
|
@ -0,0 +1,206 @@
|
||||||
|
# Hosts
|
||||||
|
|
||||||
|
Per-host NixOS configs for the homelab and admin Mac. Each `<host>.nix`
|
||||||
|
declares the host's role and services; the `<host>-hardware.nix` siblings
|
||||||
|
(where present) describe disks, kernel modules, firmware. Bootstrap +
|
||||||
|
disko configs live one level up in `../`.
|
||||||
|
|
||||||
|
## Topology
|
||||||
|
|
||||||
|
```
|
||||||
|
┌────────────────────────────────────────────┐
|
||||||
|
│ vps-relay (Hetzner, public IP, ZT peer) │
|
||||||
|
public traffic ──TLS:443──────│ Caddy + Let's Encrypt → reverse_proxy │
|
||||||
|
│ over ZeroTier to a clan backend │
|
||||||
|
└──────────────────┬─────────────────────────┘
|
||||||
|
│ (ZeroTier mesh)
|
||||||
|
│
|
||||||
|
┌──────────────────────┬───────────┴───────────┬──────────────────────┐
|
||||||
|
│ │ │ │
|
||||||
|
sunken-ship phantom-ship distant-shore foreign-port
|
||||||
|
(LAN, wifi) (LAN, wired) (LAN, wifi) (LAN, wifi)
|
||||||
|
ZT controller NAT for blank slate blank slate
|
||||||
|
media + mulbo rusty-anchor (room to grow) (room to grow)
|
||||||
|
services hub
|
||||||
|
|
||||||
|
── outside the clan ──────────────────────────────────────────────────────────────────────
|
||||||
|
Daniel-Macbook-Air rusty-anchor
|
||||||
|
(admin, runs clan-cli) (downstream of phantom-ship's NAT)
|
||||||
|
```
|
||||||
|
|
||||||
|
ZeroTier IPv6 addresses for the four clan machines are declared in
|
||||||
|
`../../flake-modules/clan.nix` (`sunkenShipZTv6` / `phantomShipZTv6` /
|
||||||
|
`vpsRelayZTv6` / etc.). They land in every host's `/etc/hosts` as
|
||||||
|
`<machine>.clan` so data-mesher and ad-hoc SSH can resolve over the mesh.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Hosts
|
||||||
|
|
||||||
|
### sunken-ship · media + ZT controller
|
||||||
|
|
||||||
|
- **Hardware:** see `sunken-ship-hardware.nix`. WiFi-only, no LUKS (boots
|
||||||
|
unattended).
|
||||||
|
- **Network:** LAN over WiFi, on the ZT mesh as the **controller** (manages
|
||||||
|
ZT membership for the whole fleet). ZT IPv6 is the clan's "internet"
|
||||||
|
target for `clan machines update`.
|
||||||
|
- **Role:** media + the long-running personal services that don't fit on
|
||||||
|
phantom-ship.
|
||||||
|
- **Current services:** `navidrome` (subsonic API, `/srv/music`), `uxplay`
|
||||||
|
(AirPlay receiver), `mulbo-server` (+ `-pull` / `-backfill` / `-enrich`
|
||||||
|
timers), `fitness-bot` (+ `-pull` / `-shipyard` variants),
|
||||||
|
`dm-pull-deploy-push` (announces origin/main rev to the mesh every 15 m).
|
||||||
|
|
||||||
|
### phantom-ship · services hub + LAN NAT
|
||||||
|
|
||||||
|
- **Hardware:** see `phantom-ship-hardware.nix`. WiFi for WAN, wired
|
||||||
|
ethernet (`enp0s31f6`) serves the lab subnet (NAT + dnsmasq for
|
||||||
|
`rusty-anchor`).
|
||||||
|
- **Network:** LAN over WiFi, on the ZT mesh. Backends are exposed only on
|
||||||
|
the ZT interface (`firewall.interfaces."zt+".allowedTCPPorts = [ … ]`)
|
||||||
|
so vps-relay's Caddy can reach them. WAN side stays closed.
|
||||||
|
- **Role:** where new self-hosted apps default to going. Hosts a growing
|
||||||
|
list of mini-app backends + a couple of long-running daemons.
|
||||||
|
- **Current services:** `forgejo` (`git.dannydannydanny.me`),
|
||||||
|
`claude-channels` (Telegram bridge for `@HarakatBot`),
|
||||||
|
`hara-gmail-mcp` + `hara-heartbeat` (timer),
|
||||||
|
Mini-App backends (`shelfish`, `scuttle`, `bananasimulator`,
|
||||||
|
`komtolk`, `escape-hormuz`, `bon`), `ollama` (local LLM), `shipyard`,
|
||||||
|
`dnsmasq` (lab subnet DHCP/DNS). `openclaw-gateway` is disabled —
|
||||||
|
superseded by `claude-channels` but kept for easy rollback.
|
||||||
|
|
||||||
|
### vps-relay · public reverse proxy
|
||||||
|
|
||||||
|
- **Hardware:** Hetzner Cloud vServer (BIOS-boot, virtio). Disk via
|
||||||
|
`../disko-cloud.nix`.
|
||||||
|
- **Network:** public IP `89.167.39.251`. Inbound: SSH/22, HTTP/80,
|
||||||
|
HTTPS/443 only. fail2ban guards SSH. Outbound to clan backends over ZT.
|
||||||
|
- **Role:** terminates public TLS, reverse-proxies subdomains over ZT to
|
||||||
|
whichever clan host runs the backend. **No application data ever lands
|
||||||
|
here** — this box is a relay. New public app = add a `virtualHosts`
|
||||||
|
entry + a GoDaddy A record pointing at `89.167.39.251`.
|
||||||
|
- **Current vhosts:** `navidrome.`, `bbbot.`, `shelfish.`, `scuttle.`,
|
||||||
|
`bananasimulator.`, `komtolk.`, `git.`, `escapehormuz.`, etc.
|
||||||
|
|
||||||
|
### distant-shore · ThinkPad X13 Gen 2, blank slate
|
||||||
|
|
||||||
|
- **Hardware:** see `distant-shore-hardware.nix`. Intel i5-1145G7, 16 GB.
|
||||||
|
WiFi-only, headless, no LUKS. Secure-Boot-chained boot (shim + MOK,
|
||||||
|
see comments in `distant-shore.nix`).
|
||||||
|
- **Network:** LAN over WiFi, on the ZT mesh.
|
||||||
|
- **Role:** _to be assigned_. In the clan inventory; auto-rebuilds via
|
||||||
|
dm-pull-deploy. Drop a service in to give it a purpose.
|
||||||
|
|
||||||
|
### foreign-port · laptop, blank slate (WIP)
|
||||||
|
|
||||||
|
- **Hardware:** see `foreign-port-hardware.nix`. WiFi-only, headless,
|
||||||
|
no LUKS. Vendor-signed-shim boot chain.
|
||||||
|
- **Status:** still being wired up — not in the clan inventory yet.
|
||||||
|
- **Role:** _to be assigned_, same flow as `distant-shore`.
|
||||||
|
|
||||||
|
### daniel-macbook-air · admin
|
||||||
|
|
||||||
|
- **Hardware:** MacBook Air (the daily driver).
|
||||||
|
- **Role:** outside the clan. Runs `clan machines update` to push to
|
||||||
|
the servers + holds the SSH keys that authorize root@ on each clan
|
||||||
|
host. Also a ZT peer.
|
||||||
|
|
||||||
|
### wsl
|
||||||
|
|
||||||
|
- **Role:** WSL development environment (legacy / occasional).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
### Automatic (the default)
|
||||||
|
|
||||||
|
`dm-pull-deploy` (clan-community module wired in `clan.nix`):
|
||||||
|
|
||||||
|
1. **Push announcement:** sunken-ship's `dm-pull-deploy-push` timer runs
|
||||||
|
`dm-send-deploy` every 15 m. It signs and broadcasts the current
|
||||||
|
`origin/main` rev over the data-mesher gossip protocol.
|
||||||
|
2. **Pull + rebuild:** each `roles.default` machine (currently
|
||||||
|
`sunken-ship`, `phantom-ship`) runs a `.path` watcher that fires when
|
||||||
|
the gossiped rev changes; it `git fetch`es and `nixos-rebuild switch`es.
|
||||||
|
|
||||||
|
So **a push to `origin/main` rolls out within ~15 m** on the two
|
||||||
|
production hosts. No SSH-from-Mac required.
|
||||||
|
|
||||||
|
`vps-relay` and `distant-shore` are **not** in `roles.default` — they
|
||||||
|
need a manual deploy (see below) until/unless their role changes.
|
||||||
|
|
||||||
|
### Manual
|
||||||
|
|
||||||
|
From `~/dotfiles` on the Mac:
|
||||||
|
|
||||||
|
```
|
||||||
|
nix run 'git+https://git.clan.lol/clan/clan-core#clan-cli' -- \
|
||||||
|
machines update <host>
|
||||||
|
```
|
||||||
|
|
||||||
|
Caveats encountered in practice:
|
||||||
|
|
||||||
|
- The Mac's `ssh-agent` often has the wrong key loaded for clan deploys.
|
||||||
|
Prefix with `env -u SSH_AUTH_SOCK` to force `~/.ssh/config` identity
|
||||||
|
selection.
|
||||||
|
- A nixpkgs bump may register a new generation but refuse to live-switch
|
||||||
|
due to "switch inhibitors". Add `--no-check` to force.
|
||||||
|
- `vps-relay` only accepts `~/.ssh/id_ed25519_sunken_ship` (the Mac's
|
||||||
|
copy of sunken-ship's authorized key). The agent's other keys won't
|
||||||
|
open it.
|
||||||
|
|
||||||
|
Putting both together:
|
||||||
|
|
||||||
|
```
|
||||||
|
env -u SSH_AUTH_SOCK nix run 'git+https://git.clan.lol/clan/clan-core#clan-cli' \
|
||||||
|
-- machines update phantom-ship --no-check
|
||||||
|
```
|
||||||
|
|
||||||
|
### From sunken-ship
|
||||||
|
|
||||||
|
`vps-relay` was originally only reachable from sunken-ship's SSH key.
|
||||||
|
That still works as a fallback — SSH to sunken-ship and run the same
|
||||||
|
`nix run … -- machines update vps-relay` command from `/etc/dotfiles`
|
||||||
|
there. The dotfiles checkout at `/etc/dotfiles` is maintained by
|
||||||
|
dm-pull-deploy.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Public traffic pattern
|
||||||
|
|
||||||
|
```
|
||||||
|
user → DNS *.dannydannydanny.me → 89.167.39.251 (vps-relay)
|
||||||
|
→ Caddy (Let's Encrypt, ports 80/443)
|
||||||
|
→ reverse_proxy http://[<backend ZT IPv6>]:<port>
|
||||||
|
→ service on sunken-ship or phantom-ship
|
||||||
|
```
|
||||||
|
|
||||||
|
To add a new public app:
|
||||||
|
|
||||||
|
1. Add a `virtualHosts` entry to `vps-relay.nix` pointing at the
|
||||||
|
backend's ZT IPv6 and port.
|
||||||
|
2. Add the GoDaddy A record `<sub>.dannydannydanny.me → 89.167.39.251`.
|
||||||
|
3. Run the backend on the chosen host. Either:
|
||||||
|
- bind to `127.0.0.1:<port>` (if backend + Caddy are co-resident — not
|
||||||
|
the case here), **or**
|
||||||
|
- bind to `0.0.0.0` (or `::`) and add the port to
|
||||||
|
`networking.firewall.interfaces."zt+".allowedTCPPorts` on the
|
||||||
|
backend host so only the ZT interface accepts inbound.
|
||||||
|
4. Push dotfiles. Production hosts auto-rebuild via dm-pull-deploy.
|
||||||
|
vps-relay needs a manual `clan machines update vps-relay`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SSH keys (quick reference)
|
||||||
|
|
||||||
|
- **`~/.ssh/id_ed25519_phantom_ship`** (Mac) — authorized as `danny@` and
|
||||||
|
`root@` on phantom-ship.
|
||||||
|
- **`~/.ssh/id_ed25519_sunken_ship`** (Mac) — authorized as `danny@` (and
|
||||||
|
via root mirror) on sunken-ship; also the authorized key on `vps-relay`.
|
||||||
|
- **sunken-ship `~/.ssh/id_ed25519`** — sunken-ship's own key; used by
|
||||||
|
cluster-internal ops (mulbo-pull, dm-send-deploy, fallback path for
|
||||||
|
vps-relay deploys).
|
||||||
|
- **`~/.ssh/id_ed25519_github`** (Mac) — GitHub auth, not clan.
|
||||||
|
|
||||||
|
Authorized-keys lists live in each host's `users.users.{danny,root}.openssh.authorizedKeys.keys`.
|
||||||
Loading…
Add table
Add a link
Reference in a new issue