Drop the cloudflared Quick Tunnel (URL changed on every restart →
unworkable for shipyard's apps.json). Move to the same pattern
every other tenant uses:
- vps-relay Caddy: new virtualHost b3.dannydannydanny.me →
reverse_proxy to sunken-ship's ZT IPv6 :8081.
- sunken-ship: open port 8081 on the zt+ firewall interface
(was 8080 + 8091, now 8080 + 8081 + 8091).
- fitness-bot-shipyard service: set WEBAPP_URL=https://b3...
so start.py skips its own tunnel attempt; drop pkgs.cloudflared
from path now that nothing in the unit needs it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
shipyard_poc_bot is the shared "POC slot" Telegram bot that hosts
whatever experiment is currently being staged; B3Bot staging is
just the current tenant. Repoint EnvironmentFile and
ConditionPathExists at /home/danny/.secrets/shipyard_poc_bot.env.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion oneshot for mulbo-server. python312 env adds mutagen
(tag writeback); pkgs.yt-dlp on PATH for SoundCloud lookups.
Same User/SupplementaryGroups/EnvironmentFile/StateDirectory as
mulbo-server-backfill. TimeoutSec=8h covers a full library pass.
Trigger: sudo systemctl start mulbo-server-enrich
Follow: journalctl -fu mulbo-server-enrich
Add MULBO_ACOUSTID_KEY to /home/danny/.secrets/mulbo-server-navidrome
to enable the AcoustID source; the yt-dlp + filename sources need
no keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors the prod fitness-bot setup but watches origin/staging,
runs in /home/danny/tg_fitness_bot_shipyard, listens on port 8081,
and loads its bot token from
/home/danny/.secrets/bigbiggerbiggestbot-shipyard.env via
EnvironmentFile (separate from prod's secrets file).
ConditionPathExists keeps the service from start-looping until the
secrets file is written. No WEBAPP_URL set, so start.py boots an
ephemeral cloudflared Quick Tunnel; the bot updates its Telegram
menu button to that URL on every start (same as prod was originally).
Pull-timer fires every 15 min on the :13/28/43/58 offset to spread
load against the existing fitness-bot-pull (:07/15) and
mulbo-server-pull (:11/15) timers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
/srv/music is RO bind-mount; deletes/quarantines have to go through
the underlying /home/danny/music. New env var separates the read-side
(MUSIC_ROOT, used for hashing) from the write-side (MUSIC_WRITE_ROOT,
used for unlink + move-to-quarantine).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion oneshot for mulbo-server: populates the dedup index
(tracks_index) from Navidrome's existing 15k tracks. Without it,
GET /tracks/by-hash misses for every existing offshore track and
the upload path duplicates content.
Inherits same User/SupplementaryGroups as the running service.
chromaprint added to PATH for fpcalc. TimeoutSec=8h covers full
274 GB hashing run with headroom.
Triggered manually — not auto-scheduled:
sudo systemctl start mulbo-server-backfill
journalctl -fu mulbo-server-backfill
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mulbo-server's /folders endpoint reads navidrome.db directly because
the Subsonic API's path field is tag-virtual (not real fs paths).
Three pieces:
- services.navidrome UMask = 0027 (force) so future DB writes are
group-readable; default was 0077.
- tmpfiles z-rules to chmod 0640 the existing navidrome.db, -wal, -shm
(created under the old umask).
- mulbo-server gets SupplementaryGroups=[navidrome] so the unit's
process can read those files.
Trade-off: couples mulbo-server to Navidrome's schema (specifically
media_file.id + media_file.path). Acceptable given Navidrome 0.61.1
has been stable on these columns; we'll catch breakage at the /health
navidrome_db_readable probe.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds:
- MULBO_MUSIC_ROOT=/srv/music (for the /folders fs walk)
- EnvironmentFile=/home/danny/.secrets/mulbo-server-navidrome (creds
for Subsonic API calls — file is mode 600, owned by danny, not in
source control)
Required for the new /folders endpoint and the upcoming POST /tracks
which needs to call search3.view + startScan.view.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 5 of the mulbo Navidrome-pivot — companion HTTP service co-
located with Navidrome that owns uploads + the dedup index + the
real on-disk folder layout (which Navidrome's tag-virtual API can't
expose). Wire spec lives in the mulbo repo at 20_mulbo/SERVER_API.md.
Runs as `danny` so writes pass through to /home/danny/music/mulbo-
uploads via the existing /srv/music ro bind-mount — no mount changes
needed. Bound to [::]:8091 (8090 was taken by escape-hormuz on
phantom-ship); firewall scopes it to the ZT mesh, same trick bbbot
uses on 8080.
Pulls the python-projects repo via SSH using sunken-ship's id_ed25519
(registered as a read-only deploy key on the repo). Auto-pull timer
runs every 15 min, offset from fitness-bot-pull and dotfiles-rebuild.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move the imperative SSH-key-related scars accumulated during the
clan/VPS rollout into nix config so future installs and rebuilds
reproduce the same state:
- danny@sunken-ship + danny@phantom-ship: trust the mac admin key
(id_ed25519_<host> on Daniel-Macbook-Air) and the host's own
self-loopback key (used by clan ssh-ng:// nix-copy-closure back
to the same host during `clan machines update`).
- root@sunken-ship + root@phantom-ship: trust the mac admin key so
`clan machines update` can run its SOPS-key upload step that
SSHes to root@<host> to write /var/lib/sops-nix/key.txt.
Existing key files (~/.ssh/id_ed25519 on each host) stay where they
are; the keypair was generated once during initial bootstrap and the
public side is now declared above. Reinstalls would regenerate and
need the pubkey re-pinned here.
Stage 4.5: declare a Hetzner-hosted reverse-proxy VPS as a clan machine.
- nixos/hosts/vps-relay.nix: Debian→NixOS cx23 in hel1. Caddy at public
80/443 reverse-proxies navidrome.dannydannydanny.me and
bbbot.dannydannydanny.me over ZT to sunken-ship.
- nixos/disko-cloud.nix: simple GPT + ext4 root, no LUKS — cloud provider
has physical disk anyway and there's no operator at boot.
- flake-modules/clan.nix: register vps-relay as an inventory machine,
zerotier peer, internet networking target at its clan-generated ZT
IPv6, and add vps-relay.clan to clanHostsModule /etc/hosts.
- sunken-ship fitness-bot: drop pkgs.cloudflared from PATH + set
WEBAPP_URL=https://bbbot.dannydannydanny.me. Paired with the bbbot
upstream patch (start.py honors env WEBAPP_URL and skips cloudflared
when set) — once the 15-min fitness-bot-pull timer pulls that change,
bbbot will stop churning trycloudflare.com URLs.
Vars (zerotier identity/ip + sops machine key) generated on sunken-ship
because clan's hermetic sandbox on macOS fails to run the zerotier
identity generator (same workaround as for data-mesher earlier).
VPS install flow: Hetzner-created Debian box, then `clan machines
install vps-relay --target-host root@<public-ipv4>` reinstalls to
NixOS; subsequent updates go over ZT.
Stage 4d of the clan migration. Navidrome is now reachable only over
the ZeroTier mesh (port 4533 on sunken-ship's ZT IPv6 address, or via
the sunken-ship-zt SSH alias). Dropped:
- systemd.services.cloudflare-tunnel
- clan.core.vars.generators.cloudflare-tunnel
- cloudflared from environment.systemPackages
- vars/per-machine/sunken-ship/cloudflare-tunnel/
Manual follow-ups still needed on sunken-ship:
- rm /home/danny/.secrets/cloudflare-tunnel-token (old unmanaged token)
- delete the tunnel itself in the Cloudflare Zero Trust dashboard
- unlink the DNS record music.dannydannydanny.me if it was separate
Declare a clan.core.vars.generators.cloudflare-tunnel generator that
prompts for the tunnel token on first run and stores it SOPS-encrypted
under vars/per-machine/sunken-ship/cloudflare-tunnel/tunnel-token.
systemd.services.cloudflare-tunnel ExecStart now reads the decrypted
secret at runtime from \${config.clan.core.vars...path} (lives at
/run/secrets/vars/...) instead of the unmanaged
/home/danny/.secrets/cloudflare-tunnel-token file.
Stage 4c of the clan migration. The tunnel itself is slated for
retirement in 4d — ZeroTier-only access after that. Cloudflare token
was rotated during this migration; old value no longer valid.
clan-cli silently ignores the `?dir=` URL parameter when resolving a
flake source, so with the flake at nixos/flake.nix `clan machines
update` fails with "flake.nix does not exist". Move the flake tree up
so the repo root contains flake.nix, flake.lock, flake-modules/, lib/,
modules/, sops/, and vars/. Host-specific NixOS modules stay in
nixos/{hosts,home,fish.nix,neovim.nix,…}; flake-module paths updated
accordingly.
- dotfiles-rebuild flakeRef is now "${dotfilesDir}#<host>" (was
"${dotfilesDir}/nixos#<host>").
- CLAUDE.md build commands + clan section updated. nixupdate fish alias
updated. sunken-ship hostsfile comment updated.
- Existing /etc/dotfiles checkouts on the servers will pick up the new
layout on the next `dotfiles-rebuild` timer tick; the rebuild service
was pre-updated via rsync so its flakeRef matches before the pull.
Also includes 4b follow-through: zerotier identities are now live on
both servers (sunken-ship=d553a2de33 controller, phantom-ship=6c048abbdc
peer) and IPv6 ping across the ZT mesh works.
- Add import-tree input; flake.nix now auto-loads every file under
./flake-modules so new hosts/features drop in without editing flake.nix.
- Extract the duplicated dotfiles-rebuild service, timer, and
safe.directory wiring into nixos/modules/dotfiles-rebuild.nix, exposed
via flake.nixosModules.dotfiles-rebuild.
- sunken-ship and phantom-ship now pull it in from their flake-modules;
hostname-specific flakeRef is derived from config.networking.hostName.
The dotfiles-rebuild service runs as root, but /etc/dotfiles is owned
by `danny`. The GIT_CONFIG_* env vars in the service unit only affect
the git CLI — nix/libgit2 reads safe.directory from /etc/gitconfig.
After a recent nixpkgs bump libgit2 now enforces this strictly, so the
service was failing to evaluate the flake.
Enable programs.git and set programs.git.config.safe.directory =
[ dotfilesDir ] on both sunken-ship and phantom-ship so the trust is
persistent and Nix-managed.
nixos-rebuild was failing with "fsType accessed but has no value
defined" on newer nixpkgs. Bind mounts need fsType=none explicitly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New fitness-bot-pull service + timer, modeled on dotfiles-rebuild.
Checks origin/main for new commits, pulls + restarts the service if
the HEAD moved. Offset by 7 min from dotfiles-rebuild to avoid
overlap.
Code now lives at github.com/DannyDannyDanny/bigbiggerbiggestbot
(cloned to /home/danny/tg_fitness_bot). workouts.db is gitignored
so it's preserved across pulls.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Exposes navidrome via music.dannydannydanny.me.
Bypasses CGNAT — no port forwarding needed.
Token stored outside repo at ~/.secrets/cloudflare-tunnel-token.
Code deployed separately via rsync (private repo, not referenced here).
Expects code at /home/danny/tg_fitness_bot/ and token at
~/.secrets/bigbiggerbiggestbot.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Drop PipeWire (WirePlumber fails to detect ALSA cards without a
graphical session). Use GStreamer alsasink with plughw:USB,0 to
output directly to the Scarlett Solo.
Enable PipeWire with ALSA/PulseAudio compat so GStreamer can output
audio. Move UxPlay to a user service with linger so it can reach
PipeWire. Add danny to audio group, add alsa-utils for debugging.
Restore flake checks by removing deprecated `programs.light` from sunken-ship and switching to brightnessctl guidance. Also clean up flake formatting and make the Raycast char-count script safer for empty input.
Made-with: Cursor
