Alertmanager binds [::1]:9093 but Prometheus was dialing
127.0.0.1:9093 — connection refused, so alerts fired internally
but never reached Alertmanager. Switch the target to [::1]:9093
to match the bind.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The NixOS module concatenates listenAddress and port as `${a}:${p}`,
so "::" became ":::9100" and node_exporter rejected it ("too many
colons in address"). Use "[::]" so the result is "[::]:9100".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
node_exporter on all three hosts (port 9100, ZT-only). Prometheus
server scrapes via the clan ZT IPv6s. Alertmanager routes alerts to
@HarakatBot (chat 66070351); critical repeats every 1h, others 4h.
Starter rule: HostDown when up==0 for 5m. Grafana on :3000 over ZT,
provisioned with the local Prometheus as default datasource.
Manual secrets on sunken-ship: /etc/alertmanager/telegram-token and
/etc/grafana/secret-key.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop the cloudflared Quick Tunnel (URL changed on every restart →
unworkable for shipyard's apps.json). Move to the same pattern
every other tenant uses:
- vps-relay Caddy: new virtualHost b3.dannydannydanny.me →
reverse_proxy to sunken-ship's ZT IPv6 :8081.
- sunken-ship: open port 8081 on the zt+ firewall interface
(was 8080 + 8091, now 8080 + 8081 + 8091).
- fitness-bot-shipyard service: set WEBAPP_URL=https://b3...
so start.py skips its own tunnel attempt; drop pkgs.cloudflared
from path now that nothing in the unit needs it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
shipyard_poc_bot is the shared "POC slot" Telegram bot that hosts
whatever experiment is currently being staged; B3Bot staging is
just the current tenant. Repoint EnvironmentFile and
ConditionPathExists at /home/danny/.secrets/shipyard_poc_bot.env.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion oneshot for mulbo-server. python312 env adds mutagen
(tag writeback); pkgs.yt-dlp on PATH for SoundCloud lookups.
Same User/SupplementaryGroups/EnvironmentFile/StateDirectory as
mulbo-server-backfill. TimeoutSec=8h covers a full library pass.
Trigger: sudo systemctl start mulbo-server-enrich
Follow: journalctl -fu mulbo-server-enrich
Add MULBO_ACOUSTID_KEY to /home/danny/.secrets/mulbo-server-navidrome
to enable the AcoustID source; the yt-dlp + filename sources need
no keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors the prod fitness-bot setup but watches origin/staging,
runs in /home/danny/tg_fitness_bot_shipyard, listens on port 8081,
and loads its bot token from
/home/danny/.secrets/bigbiggerbiggestbot-shipyard.env via
EnvironmentFile (separate from prod's secrets file).
ConditionPathExists keeps the service from start-looping until the
secrets file is written. No WEBAPP_URL set, so start.py boots an
ephemeral cloudflared Quick Tunnel; the bot updates its Telegram
menu button to that URL on every start (same as prod was originally).
Pull-timer fires every 15 min on the :13/28/43/58 offset to spread
load against the existing fitness-bot-pull (:07/15) and
mulbo-server-pull (:11/15) timers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
/srv/music is RO bind-mount; deletes/quarantines have to go through
the underlying /home/danny/music. New env var separates the read-side
(MUSIC_ROOT, used for hashing) from the write-side (MUSIC_WRITE_ROOT,
used for unlink + move-to-quarantine).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A/B-tested 7B vs 3B on a real NETTO receipt. 7B took 3.6 min/receipt
vs ~30s for 3B. Accuracy gain was minimal — 7B still picked a line
item ('ARLA SEOMELK 1.') as merchant when the OCR header was missing,
just a different one than 3B picked ('REJESALAT'). The merchant
problem isn't a model-size problem; it's an OCR problem (Tesseract
missed the NETTO logo entirely on this receipt).
Keeping both models in loadModels so we can flip back via env var
without a fresh pull.
3B was making column-parsing mistakes on real receipts (conflating
qty/price, nominating line items as merchant). 7B Q4_K_M is ~3x slower
on phantom-ship CPU (~5min vs ~1.5min per receipt) but materially
better at structured extraction. Background task — speed isn't critical.
Keep 3B in loadModels as a fallback knob (BON_OLLAMA_MODEL env).
notes serves both notes.dannydannydanny.me (blog) and
dannydannydanny.me (apex landing) from the same FastAPI process,
switching on Host header. Source rsync'd from ~/python-projects/26_notes/
to /home/danny/notes/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion oneshot for mulbo-server: populates the dedup index
(tracks_index) from Navidrome's existing 15k tracks. Without it,
GET /tracks/by-hash misses for every existing offshore track and
the upload path duplicates content.
Inherits same User/SupplementaryGroups as the running service.
chromaprint added to PATH for fpcalc. TimeoutSec=8h covers full
274 GB hashing run with headroom.
Triggered manually — not auto-scheduled:
sudo systemctl start mulbo-server-backfill
journalctl -fu mulbo-server-backfill
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mulbo-server's /folders endpoint reads navidrome.db directly because
the Subsonic API's path field is tag-virtual (not real fs paths).
Three pieces:
- services.navidrome UMask = 0027 (force) so future DB writes are
group-readable; default was 0077.
- tmpfiles z-rules to chmod 0640 the existing navidrome.db, -wal, -shm
(created under the old umask).
- mulbo-server gets SupplementaryGroups=[navidrome] so the unit's
process can read those files.
Trade-off: couples mulbo-server to Navidrome's schema (specifically
media_file.id + media_file.path). Acceptable given Navidrome 0.61.1
has been stable on these columns; we'll catch breakage at the /health
navidrome_db_readable probe.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds:
- MULBO_MUSIC_ROOT=/srv/music (for the /folders fs walk)
- EnvironmentFile=/home/danny/.secrets/mulbo-server-navidrome (creds
for Subsonic API calls — file is mode 600, owned by danny, not in
source control)
Required for the new /folders endpoint and the upcoming POST /tracks
which needs to call search3.view + startScan.view.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 5 of the mulbo Navidrome-pivot — companion HTTP service co-
located with Navidrome that owns uploads + the dedup index + the
real on-disk folder layout (which Navidrome's tag-virtual API can't
expose). Wire spec lives in the mulbo repo at 20_mulbo/SERVER_API.md.
Runs as `danny` so writes pass through to /home/danny/music/mulbo-
uploads via the existing /srv/music ro bind-mount — no mount changes
needed. Bound to [::]:8091 (8090 was taken by escape-hormuz on
phantom-ship); firewall scopes it to the ZT mesh, same trick bbbot
uses on 8080.
Pulls the python-projects repo via SSH using sunken-ship's id_ed25519
(registered as a read-only deploy key on the repo). Auto-pull timer
runs every 15 min, offset from fitness-bot-pull and dotfiles-rebuild.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hara (openclaw) shipped escape_hormuz imperatively — service runs but
firewall + Caddy vhost weren't declared, so the public URL didn't
resolve and the firewall rule would've been wiped on next
dotfiles-rebuild. Bring it under nix:
phantom-ship.nix
- systemd.services.escape-hormuz on port 8090, binds :: for ZT
- 8090 added to zt+ allowedTCPPorts
- tmpfiles entry for /home/danny/.local/share/escape_hormuz
vps-relay.nix
- Caddy vhost escapehormuz.dannydannydanny.me → ZT [::]:8090
Phase 1 of the de-platform-from-GitHub roadmap (vimwiki/diary/2026-05-03.md).
- phantom-ship: services.forgejo bound to 0.0.0.0:3000, sqlite, lfs on,
registration disabled, sign-in required.
- phantom-ship: add :3000 to the existing zt+ allowedTCPPorts list
(joins shelfish/scuttle — never exposed on WAN/Wi-Fi).
- vps-relay: Caddy vhost git.dannydannydanny.me reverse-proxies over
ZT to phantom-ship:3000.
Manual steps before reachable:
1. GoDaddy A record git.dannydannydanny.me -> 89.167.39.251
2. clan machines update phantom-ship && clan machines update vps-relay
3. On phantom-ship: bootstrap admin (registration is disabled)
KomTolk is the rebranded translate-platform — same Copenhagen
translation gigs Mini App, new name. Service on port 8080, mirrors
shelfish/scuttle/banana setup. New tmpfiles dir + zt+ firewall
opening + caddy vhost.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bananasimulator.service mirrors shelfish/scuttle (fastapi + uvicorn
+ httpx + python-telegram-bot). Port 8083. ENV BS_RIPE_MIN_PER_STAGE=2
in prod (30 min total banana lifetime); preview uses 0.5 for fast
testing.
vps-relay gets a fifth vhost (bananasimulator.dannydannydanny.me)
reverse-proxying to phantom-ship over ZeroTier. The shipyard manifest
has been pointing at this URL as a placeholder since day one — now
it's actually live.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scuttle.service mirrors shelfish — fastapi/uvicorn/httpx/python-telegram-bot
plus websockets, runs uvicorn --host :: --port 8082, DB at
~/.local/share/scuttle/scuttle.db (tmpfiles rule + zt+ firewall port
added alongside shelfish's).
vps-relay gets a fourth virtualHost (scuttle.dannydannydanny.me)
reverse-proxying to phantom-ship over ZeroTier. WebSocket upgrade is
transparent under Caddy's reverse_proxy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds two write-capable IMAP tools:
- gmail_mark_read: sets \Seen flag on a message
- gmail_archive: copies to [Gmail]/All Mail and removes from INBOX
The IMAP connection already used SELECT (read-write mode); this just
exposes the mutation surface through MCP.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
shelfish was only listening on 127.0.0.1 — vps-relay's Caddy
couldn't reach it over the ZT mesh. Bind 0.0.0.0 and allow 8081
inbound on \`zt+\` interfaces (not the global firewall — same
pattern sunken-ship uses for bbbot).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Original commit added Caddy directly on phantom-ship and opened
ports 80/443 — that would have exposed the home connection's
public IP via DNS. Reverting that and using the existing relay
pattern instead: vps-relay (Hetzner) terminates public TLS and
reverse-proxies over ZeroTier to phantom-ship's ZT IPv6 on 8081.
phantom-ship now just runs shelfish.service bound to 127.0.0.1:8081;
it accepts connections only from the ZT mesh interface (since
caddy/firewall changes are gone, the only listeners are the
existing trusted-interface ones plus this loopback).
vps-relay gets a third virtualHost alongside navidrome and bbbot.
DNS: shelfish.dannydannydanny.me → 89.167.39.251 (vps-relay public IP),
NOT phantom-ship's home IP.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Caddy fronts 80/443 with auto-Let's-Encrypt; reverse-proxies
shelfish.dannydannydanny.me to the local shelfish service on
127.0.0.1:8081. ACME issues the cert once the subdomain A-records
to this host's static IP.
Shelfish service mirrors shipyard's pattern: nix-built python env,
SHIPYARD_BOT_TOKEN_FILE pointed at the existing secret, DB stored
outside the rsynced code dir at ~/.local/share/shelfish/ so deploys
don't clobber state.
Code itself is rsync'd from ~/python-projects/27_shelfish/ to
/home/danny/shelfish/ (same convention as shipyard).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an MCP server exposing read tools (list_inbox, search, read_email)
across three personal Gmail accounts using existing app passwords in
/etc/openclaw/. Wired into claude-channels via --mcp-config. Slated for
replacement by an OAuth2 Gmail+Calendar server in path 2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
VPS public SSH: enable fail2ban with bantime-increment so brute-force
probers get evicted with exponential backoff (1h → 4h → 16h → 2.7d →
10.7d, capped at 30d). Default jail covers sshd; maxretry=5 in 10m.
server-debug-tools: htop, tcpdump, dnsutils, jq, curl. Imported by
sunken-ship + phantom-ship via flake.nixosModules.server-debug-tools.
These are the practical bits we'd otherwise pick up by enabling
clan.core.enableRecommendedDefaults — but the full clan defaults flip
systemd-networkd/resolved on, which broke dnsmasq + navidrome's resolv
.conf bind-mount on the homelab servers, so we cherry-pick instead.
Move the imperative SSH-key-related scars accumulated during the
clan/VPS rollout into nix config so future installs and rebuilds
reproduce the same state:
- danny@sunken-ship + danny@phantom-ship: trust the mac admin key
(id_ed25519_<host> on Daniel-Macbook-Air) and the host's own
self-loopback key (used by clan ssh-ng:// nix-copy-closure back
to the same host during `clan machines update`).
- root@sunken-ship + root@phantom-ship: trust the mac admin key so
`clan machines update` can run its SOPS-key upload step that
SSHes to root@<host> to write /var/lib/sops-nix/key.txt.
Existing key files (~/.ssh/id_ed25519 on each host) stay where they
are; the keypair was generated once during initial bootstrap and the
public side is now declared above. Reinstalls would regenerate and
need the pubkey re-pinned here.
