The /enrich/revert endpoint shipped in 20_mulbo commit 5d4e9466 calls
enrich.write_tags, which imports mutagen. The main mulbo-server's
pythonEnv only had fastapi/uvicorn/python-multipart — first revert
attempt 500'd with "ModuleNotFoundError: No module named 'mutagen'".
(The enrich oneshot has its own env with mutagen; that's why batch
enrichment worked.)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Twice in the last few sessions a Love2D force-quit cascaded into
killing the tmux server and losing every window. Resurrect snapshots
windows / panes / cwd / pane contents (with capture-pane-contents on)
to ~/.local/share/tmux/resurrect/last. Continuum auto-saves every 15
min and auto-restores on tmux server start — so the next force-quit
just costs up to 15 min of recent activity, not the whole workspace.
Manual save: prefix + Ctrl-s. Manual restore: prefix + Ctrl-r.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stops `missing=1` rows accumulating in media_file. After Phase 7
dedupe, Navidrome's watcher minted ~4k track IDs for files briefly
present in /home/danny/music/.mulbo-quarantine; after rm -rf'ing
the quarantine, those rows stayed flagged-missing forever — and
Substreamer's cached queue then hit 500s on every play attempt
("Internal Server Error: open /srv/music/.mulbo-quarantine/...: no
such file or directory").
Cleaned the 4135 quarantine rows manually via SQL; this config
prevents recurrence. Trade-off: missing rows used to preserve
play-history across "file disappeared, came back" cycles. We prefer
client-cache hygiene.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-publishes the bot's menu button + description on every restart
so @BBBot's chat experience stays in sync with $WEBAPP_URL. Errors
are non-fatal.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bot.py was deleted upstream — neither prod nor shipyard launches a
polling bot anymore. server.py only needs python-dotenv + aiohttp.
Also refresh the prod section's comment + service description to
reflect the Mini-App-only architecture.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR clan/clan-community#25 (machine.name hyphen sanitization) merged
upstream, so swap clan-community.url from the fork branch back to
clan/clan-community/archive/main.tar.gz and update flake.lock to
upstream rev 81e4c9c. Eval confirms byte-identical host closures.
Also finishes the dotfiles-rebuild retirement: phantom-ship.nix still
referenced the now-deleted modules/dotfiles-rebuild.nix in comments.
The catppuccin nix module only generates the static flavor+accent
combinations and sets DEFAULT_THEME to e.g. catppuccin-mocha-mauve.
The auto-switching CSS files (catppuccin-<accent>-auto) ship in the
gitea-theme assets but aren't wired into THEMES.
Override DEFAULT_THEME to catppuccin-mauve-auto so the browser's
prefers-color-scheme decides — latte (light) in light mode, mocha
(dark) in dark mode. Append all auto variants + the four mauve
flavor variants to THEMES so users can still pick from the
appearance settings.
- Drop modules/dotfiles-rebuild.nix and its imports in clan.nix;
sunken-ship + phantom-ship no longer ship the legacy 15-min
rebuild-from-git timer.
- Add dm-pull-deploy-push systemd timer on sunken-ship: every 15min
runs dm-send-deploy to announce origin/main rev via data-mesher
gossip (sunken is the dm-pull-deploy push node).
- Fix mulbo-pull service path: add openssh so 'git fetch' over an
SSH remote stops failing with 'cannot run ssh'.
- vps-relay authorized_keys: rename Mac key comment to mac-admin,
add sunken-ship's actual ed25519 key for ZT mesh debugging.
- home.nix: add cinny-desktop (Matrix client).
- neovim: enable cursorline.
Adds catppuccin flake input and wires its NixOS module into phantom-ship's
imports via clan.nix. Enables catppuccin.forgejo with mocha flavor + mauve
accent on the running Forgejo instance.
Module ref: https://nix.catppuccin.com/options/main/nixos/catppuccin.forgejo/
Routes the new subdomain to the existing notes service on
phantom-ship :8092 (Host-header routed). Serves Kyranna Fardi's
architecture portfolio.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Alertmanager binds [::1]:9093 but Prometheus was dialing
127.0.0.1:9093 — connection refused, so alerts fired internally
but never reached Alertmanager. Switch the target to [::1]:9093
to match the bind.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The NixOS module concatenates listenAddress and port as `${a}:${p}`,
so "::" became ":::9100" and node_exporter rejected it ("too many
colons in address"). Use "[::]" so the result is "[::]:9100".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
node_exporter on all three hosts (port 9100, ZT-only). Prometheus
server scrapes via the clan ZT IPv6s. Alertmanager routes alerts to
@HarakatBot (chat 66070351); critical repeats every 1h, others 4h.
Starter rule: HostDown when up==0 for 5m. Grafana on :3000 over ZT,
provisioned with the local Prometheus as default datasource.
Manual secrets on sunken-ship: /etc/alertmanager/telegram-token and
/etc/grafana/secret-key.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop the cloudflared Quick Tunnel (URL changed on every restart →
unworkable for shipyard's apps.json). Move to the same pattern
every other tenant uses:
- vps-relay Caddy: new virtualHost b3.dannydannydanny.me →
reverse_proxy to sunken-ship's ZT IPv6 :8081.
- sunken-ship: open port 8081 on the zt+ firewall interface
(was 8080 + 8091, now 8080 + 8081 + 8091).
- fitness-bot-shipyard service: set WEBAPP_URL=https://b3...
so start.py skips its own tunnel attempt; drop pkgs.cloudflared
from path now that nothing in the unit needs it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
shipyard_poc_bot is the shared "POC slot" Telegram bot that hosts
whatever experiment is currently being staged; B3Bot staging is
just the current tenant. Repoint EnvironmentFile and
ConditionPathExists at /home/danny/.secrets/shipyard_poc_bot.env.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion oneshot for mulbo-server. python312 env adds mutagen
(tag writeback); pkgs.yt-dlp on PATH for SoundCloud lookups.
Same User/SupplementaryGroups/EnvironmentFile/StateDirectory as
mulbo-server-backfill. TimeoutSec=8h covers a full library pass.
Trigger: sudo systemctl start mulbo-server-enrich
Follow: journalctl -fu mulbo-server-enrich
Add MULBO_ACOUSTID_KEY to /home/danny/.secrets/mulbo-server-navidrome
to enable the AcoustID source; the yt-dlp + filename sources need
no keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors the prod fitness-bot setup but watches origin/staging,
runs in /home/danny/tg_fitness_bot_shipyard, listens on port 8081,
and loads its bot token from
/home/danny/.secrets/bigbiggerbiggestbot-shipyard.env via
EnvironmentFile (separate from prod's secrets file).
ConditionPathExists keeps the service from start-looping until the
secrets file is written. No WEBAPP_URL set, so start.py boots an
ephemeral cloudflared Quick Tunnel; the bot updates its Telegram
menu button to that URL on every start (same as prod was originally).
Pull-timer fires every 15 min on the :13/28/43/58 offset to spread
load against the existing fitness-bot-pull (:07/15) and
mulbo-server-pull (:11/15) timers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
/srv/music is RO bind-mount; deletes/quarantines have to go through
the underlying /home/danny/music. New env var separates the read-side
(MUSIC_ROOT, used for hashing) from the write-side (MUSIC_WRITE_ROOT,
used for unlink + move-to-quarantine).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A/B-tested 7B vs 3B on a real NETTO receipt. 7B took 3.6 min/receipt
vs ~30s for 3B. Accuracy gain was minimal — 7B still picked a line
item ('ARLA SEOMELK 1.') as merchant when the OCR header was missing,
just a different one than 3B picked ('REJESALAT'). The merchant
problem isn't a model-size problem; it's an OCR problem (Tesseract
missed the NETTO logo entirely on this receipt).
Keeping both models in loadModels so we can flip back via env var
without a fresh pull.
3B was making column-parsing mistakes on real receipts (conflating
qty/price, nominating line items as merchant). 7B Q4_K_M is ~3x slower
on phantom-ship CPU (~5min vs ~1.5min per receipt) but materially
better at structured extraction. Background task — speed isn't critical.
Keep 3B in loadModels as a fallback knob (BON_OLLAMA_MODEL env).
notes serves both notes.dannydannydanny.me (blog) and
dannydannydanny.me (apex landing) from the same FastAPI process,
switching on Host header. Source rsync'd from ~/python-projects/26_notes/
to /home/danny/notes/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion oneshot for mulbo-server: populates the dedup index
(tracks_index) from Navidrome's existing 15k tracks. Without it,
GET /tracks/by-hash misses for every existing offshore track and
the upload path duplicates content.
Inherits same User/SupplementaryGroups as the running service.
chromaprint added to PATH for fpcalc. TimeoutSec=8h covers full
274 GB hashing run with headroom.
Triggered manually — not auto-scheduled:
sudo systemctl start mulbo-server-backfill
journalctl -fu mulbo-server-backfill
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mulbo-server's /folders endpoint reads navidrome.db directly because
the Subsonic API's path field is tag-virtual (not real fs paths).
Three pieces:
- services.navidrome UMask = 0027 (force) so future DB writes are
group-readable; default was 0077.
- tmpfiles z-rules to chmod 0640 the existing navidrome.db, -wal, -shm
(created under the old umask).
- mulbo-server gets SupplementaryGroups=[navidrome] so the unit's
process can read those files.
Trade-off: couples mulbo-server to Navidrome's schema (specifically
media_file.id + media_file.path). Acceptable given Navidrome 0.61.1
has been stable on these columns; we'll catch breakage at the /health
navidrome_db_readable probe.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds:
- MULBO_MUSIC_ROOT=/srv/music (for the /folders fs walk)
- EnvironmentFile=/home/danny/.secrets/mulbo-server-navidrome (creds
for Subsonic API calls — file is mode 600, owned by danny, not in
source control)
Required for the new /folders endpoint and the upcoming POST /tracks
which needs to call search3.view + startScan.view.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 5 of the mulbo Navidrome-pivot — companion HTTP service co-
located with Navidrome that owns uploads + the dedup index + the
real on-disk folder layout (which Navidrome's tag-virtual API can't
expose). Wire spec lives in the mulbo repo at 20_mulbo/SERVER_API.md.
Runs as `danny` so writes pass through to /home/danny/music/mulbo-
uploads via the existing /srv/music ro bind-mount — no mount changes
needed. Bound to [::]:8091 (8090 was taken by escape-hormuz on
phantom-ship); firewall scopes it to the ZT mesh, same trick bbbot
uses on 8080.
Pulls the python-projects repo via SSH using sunken-ship's id_ed25519
(registered as a read-only deploy key on the repo). Auto-pull timer
runs every 15 min, offset from fitness-bot-pull and dotfiles-rebuild.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hara (openclaw) shipped escape_hormuz imperatively — service runs but
firewall + Caddy vhost weren't declared, so the public URL didn't
resolve and the firewall rule would've been wiped on next
dotfiles-rebuild. Bring it under nix:
phantom-ship.nix
- systemd.services.escape-hormuz on port 8090, binds :: for ZT
- 8090 added to zt+ allowedTCPPorts
- tmpfiles entry for /home/danny/.local/share/escape_hormuz
vps-relay.nix
- Caddy vhost escapehormuz.dannydannydanny.me → ZT [::]:8090
Phase 1 of the de-platform-from-GitHub roadmap (vimwiki/diary/2026-05-03.md).
- phantom-ship: services.forgejo bound to 0.0.0.0:3000, sqlite, lfs on,
registration disabled, sign-in required.
- phantom-ship: add :3000 to the existing zt+ allowedTCPPorts list
(joins shelfish/scuttle — never exposed on WAN/Wi-Fi).
- vps-relay: Caddy vhost git.dannydannydanny.me reverse-proxies over
ZT to phantom-ship:3000.
Manual steps before reachable:
1. GoDaddy A record git.dannydannydanny.me -> 89.167.39.251
2. clan machines update phantom-ship && clan machines update vps-relay
3. On phantom-ship: bootstrap admin (registration is disabled)
KomTolk is the rebranded translate-platform — same Copenhagen
translation gigs Mini App, new name. Service on port 8080, mirrors
shelfish/scuttle/banana setup. New tmpfiles dir + zt+ firewall
opening + caddy vhost.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bananasimulator.service mirrors shelfish/scuttle (fastapi + uvicorn
+ httpx + python-telegram-bot). Port 8083. ENV BS_RIPE_MIN_PER_STAGE=2
in prod (30 min total banana lifetime); preview uses 0.5 for fast
testing.
vps-relay gets a fifth vhost (bananasimulator.dannydannydanny.me)
reverse-proxying to phantom-ship over ZeroTier. The shipyard manifest
has been pointing at this URL as a placeholder since day one — now
it's actually live.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
