diff --git a/AGENTS.md b/AGENTS.md index 781bd79..9067f99 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -55,8 +55,10 @@ Ansible-driven personal infrastructure repo for Fedora and Void desktops, Fedora - Server compose render: `podman-compose -f /opt/docker/server/docker-compose.yml config` and `systemctl status podman-compose-server` - Atlas media stack: `ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff` - - Prometheus media mount: - `ansible-playbook ansible/site.yml --limit prometheus --tags rclone,navidrome --check --diff` + - Atlas phase-one rootless services: + `ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 --check --diff -e backend_phase1_enabled=true` + - Prometheus/Atlas WireGuard overlay: + `ansible-playbook ansible/site.yml --limit prometheus,atlas --tags wireguard --check --diff` - DuckDNS config only: `ansible-playbook ansible/site.yml --limit prometheus --tags duckdns --check --diff` ## Conventions @@ -108,16 +110,13 @@ The dotfile vars follow the same split: `desktop_common_dotfiles` carries mode-i - `rocky_server` is a child of both `platform_rocky` and `server`; `prometheus` is its active target. - The target must already provide `server_username` with local sudo access before the profile runs. - The Rocky profile installs Podman and podman-compose, uses firewalld, preserves SELinux enforcement, and renders the - Nginx Proxy Manager/Gitea/Navidrome-PostgreSQL Compose stack with a `podman-compose-server` systemd unit. It does not - start or enable that Compose stack, transfer data, update DNS, or cut over traffic; activating it remains manual. -- Prometheus has a gated system `rclone-music.service` and rootless Navidrome Quadlet. They remain disabled until the - Atlas WireGuard address, pinned SSH host key and Vault-backed SFTP private key are configured. The rclone mount is - read-only at `/mnt/music_atlas`; Navidrome must not start against the underlying empty mountpoint or while the legacy - rootful Navidrome container is still running. The role never removes that legacy container or its data. + existing Nginx Proxy Manager/Gitea Compose stack with a `podman-compose-server` systemd unit. PostgreSQL and + Navidrome are no longer part of the desired Prometheus configuration. The role does not stop or remove legacy + containers, delete `/opt/postgres/data`, start the Compose stack, update DNS, or cut over traffic. - Firewalld enables SSH, Cockpit (`9090/tcp`), HTTP and HTTPS. Nginx Proxy Manager publishes `80/tcp` and `443/tcp`; bind its administration interface only to `127.0.0.1:81` and use `npm-tunnel` from Ikaros or Nymph. Nextcloud remains disabled; do not provision `/srv/nextcloud` directories. -- `scripts/migrate_prometheus_data.sh` is the separate, source-host-run migration path. It dry-runs by +- `scripts/migrate_prometheus_data.sh` is the separate, source-host-run NPM/Gitea migration path. It dry-runs by default and requires explicit source-stack quiescing before copying persistent Docker data with rsync. - Atlas-only OpenZFS, NFS, Samba, and Syncthing stay selected through Atlas host variables and must not leak into `rocky_server`. Cockpit plus its Navigator and Podman extensions are selected explicitly for @@ -133,18 +132,33 @@ The dotfile vars follow the same split: `desktop_common_dotfiles` carries mode-i and Vault inputs are replaced; only then may the profile manage datasets, shares, LAN-restricted firewall rules, and rootful media Quadlets. - Atlas requires `vault_atlas_authorized_ssh_keys`, `vault_atlas_admin_password_hash` for Cockpit - and, when the relevant gates are enabled, `vault_atlas_samba_password` and `vault_atlas_immich_db_password`. Never - print these values. -- Atlas creates `archive`, `media/music`, `media/icloud_photos`, and `backups/services` only under the verified - pre-existing pool; `backups/services` has a `500G` refreservation. Existing Work, Syncthing, and - Prometheus-backup datasets remain managed and separate. + and, when the relevant gates are enabled, `vault_atlas_samba_password` and + `vault_atlas_immich_db_password`. Never print these values. +- Atlas creates the complete declared hierarchy only under the verified pre-existing pool: `work`, `archive`, + `archive/app_data`, `archive/app_data/navidrome`, `archive/app_data/syncthing`, `media`, `media/music`, + `media/photobook`, `backups`, `backups/services`, and `backup_prometheus`. `backups/services` has a `500G` + refreservation. There is no separate legacy `zpool/syncthing` dataset. - The `immich` system account is fixed to UID/GID `1100`, has no login shell or `wheel` membership, and receives only the `video` and `render` supplementary groups. Immich's rootful Quadlets run as `1100:1100`; Server and ML receive - `/dev/dri`, while the iCloud Photos external library is read-only. -- Atlas exports iCloud Photos only to the configured Aegis IP with all access squashed to UID/GID `1100`. SMB3 exposes + `/dev/dri`, while the Photobook external library is read-only at `/external/photobook`. +- Atlas exports Photobook only to the configured Aegis IP with all access squashed to UID/GID `1100`. SMB3 exposes `Archive` to Vault-backed authorized accounts and admits the configured LAN without host-specific exclusions. - Atlas NPM and Immich share a rootful Podman network. NPM publishes HTTP/HTTPS, but its administration port remains bound to `127.0.0.1:81`; do not expose it directly to the LAN or Internet. +- `profile_backend_phase1` is limited to rootless Navidrome and Syncthing user Quadlets on Atlas. Official Navidrome + `0.63.2` uses SQLite below `/data` and does not support `ND_DATABASE_URL` or an external PostgreSQL backend; do not + recreate the obsolete Prometheus `navidromedb` service. The role requires the storage role's `zpool/media/music`, + `zpool/archive/app_data`, `zpool/archive/app_data/navidrome`, and `zpool/archive/app_data/syncthing` datasets at + their exact paths. It never creates the pool. +- Keep `backend_phase1_start_services` false until the stopped Prometheus Navidrome data directory has been copied to + Atlas and its SQLite database verified. The playbook renders the target but never migrates or deletes application + data; after cutover, set the flag true to enable and start Navidrome and Syncthing. +- Phase 1 must not change Prometheus' existing NPM deployment. NPM continues to be managed exactly by `profile_server`; + use `10.0.0.2:4533` for Navidrome and `10.0.0.2:8384` for the Syncthing GUI. Native Syncthing transfer/discovery does + not use the HTTP proxy. +- `wireguard_overlay` manages the required `wg0` path between Prometheus and Atlas, persists private keys only on their + respective hosts, and exchanges only derived public keys. The first gated run must include both hosts. Prometheus + opens `51820/udp`; the Atlas backend role admits service ports only in the WireGuard firewalld zone. ## Atlas NAS TODO - Replace every Atlas `CHANGEME` value, provide the required Vault variables and validate the first @@ -159,7 +173,7 @@ The dotfile vars follow the same split: `desktop_common_dotfiles` carries mode-i or manual operations, not as the only source of configuration, and never automate snapshot rollback. - Manage the Syncthing star topology, device IDs, folders, folder modes, ignore rules and protected GUI or API access for the selected clients. -- Validate the existing WireGuard path and add its LAN/VPN-only firewalld rules before enabling remote services; +- Validate the managed WireGuard path and its LAN/VPN-only firewalld rules before enabling remote services; never expose SSH, Cockpit, NFS, SMB or Syncthing through public port forwarding. - Add the least-privilege Prometheus backup flow: remote dump generation, dedicated SSH identity, pinned host key, atomic pull, verification, retention and an Atlas systemd service/timer. diff --git a/README.it.md b/README.it.md index 9385bc5..c4aaad3 100644 --- a/README.it.md +++ b/README.it.md @@ -180,28 +180,24 @@ Lo stato attuale del profilo server include: - installazione pacchetti Rocky via DNF, EPEL e CRB - installazione di Podman e podman-compose - abilitazione dei servizi systemd dichiarati in inventory/group vars -- copia dei dotfiles server e rendering del `docker-compose.yml` per Nginx Proxy Manager, Gitea e il - database PostgreSQL esistente di Navidrome, piu l'unita `podman-compose-server` (attivazione manuale) -- mount di `/pool/media/music` da Atlas su `/mnt/music_atlas` tramite il servizio di sistema - `rclone-music.service`, e Navidrome tramite Quadlet utente rootless +- copia dei dotfiles server e rendering del `docker-compose.yml` per Nginx Proxy Manager e Gitea, + piu l'unita `podman-compose-server` (attivazione manuale) - attivazione di firewalld con SSH, Cockpit (`9090/tcp`), HTTP e HTTPS abilitati - Syncthing escluso dal profilo server Rocky +Il Compose desiderato su Prometheus non include piu Navidrome ne il database PostgreSQL obsoleto. +Navidrome e Syncthing appartengono ad Atlas; Navidrome ufficiale usa invece SQLite. Il profilo non +arresta o rimuove automaticamente eventuali container legacy e non elimina `/opt/postgres/data`. + Nginx Proxy Manager pubblica solo `80/tcp` e `443/tcp`; la sua interfaccia di amministrazione e associata a `127.0.0.1:81` ed e raggiungibile da Ikaros o Nymph con l'alias Bash `npm-tunnel`. Nextcloud resta disabilitato e il profilo non crea directory `/srv/nextcloud`. -Il mount musicale e protetto da `server_atlas_music_enabled`. Prima di abilitarlo, sostituire -l'indirizzo WireGuard e la chiave host SSH fissata in `host_vars/prometheus.yml`, quindi fornire -`vault_prometheus_atlas_sftp_private_key` tramite Vault cifrato o variabili locali non tracciate. La -chiave pubblica corrispondente deve essere gia presente nelle chiavi autorizzate gestite su Atlas. -Rclone usa il percorso remoto esatto `/pool/media/music` in sola lettura e una cache VFS completa da -`15G`; systemd lingering mantiene disponibile il manager utente per il Quadlet rootless. -Configurare il proxy host NPM di Prometheus per Navidrome come `host.containers.internal:4533`; la -porta Navidrome non viene aperta in firewalld. -Prima della prima attivazione, arrestare il vecchio container rootful `navidrome`. Il ruolo rifiuta -di avviare il sostituto rootless mentre il container precedente e in esecuzione e non rimuove mai -automaticamente il container o i dati esistenti. +La fase 1 su Atlas non modifica questo deployment NPM ne i suoi dati persistenti. Dopo aver attivato +WireGuard e i servizi Atlas, configurare i proxy host NPM correnti con upstream Navidrome +`http://10.0.0.2:4533` e upstream per la GUI Syncthing `http://10.0.0.2:8384`. Solo la GUI web di +Syncthing usa NPM; il traffico di sincronizzazione resta sulle porte native limitate a WireGuard. +Configurare l'autenticazione Syncthing e una policy di accesso NPM adeguata prima di pubblicare la GUI. ### DuckDNS @@ -226,8 +222,9 @@ salvare separatamente eventuali modifiche non committate senza copiare segreti. Dopo il provisioning Rocky, eseguire `scripts/migrate_prometheus_data.sh` **sul server Ubuntu sorgente**. Lo script usa rsync, e in dry-run di default; richiede `--quiesce-source --execute` per -fermare lo stack sorgente e copiare in modo consistente i dati PostgreSQL. Non avvia container, non -cancella dati e non esegue il cutover. +fermare lo stack sorgente e copiare in modo consistente soltanto i dati di Nginx Proxy Manager e +Gitea. Non sposta Navidrome o Syncthing, non avvia container, non cancella dati e non esegue il +cutover. Utente del profilo server: @@ -271,19 +268,68 @@ password Cockpit in chiaro. Le esecuzioni successive usano `atlas_admin_username `atlas_manage_media_stack` per ultimo, dopo aver verificato `/dev/dri`, i percorsi dei container e il segreto del database Immich. -Con la gestione storage attiva, Atlas crea `archive` (`zstd`), `media/music` (`lz4`), -`media/icloud_photos` (`lz4`) e `backups/services` (`lz4`, `refreservation=500G`) sotto il pool -preesistente. I dataset esistenti Work, Syncthing e backup Prometheus restano gestiti e separati. +Con la gestione storage attiva, Atlas crea l'intera gerarchia sotto il pool `zpool` preesistente: +`work`, `archive`, `archive/app_data`, i dataset applicativi separati +`archive/app_data/navidrome` e `archive/app_data/syncthing`, `media`, `media/music`, +`media/photobook`, `backups`, `backups/services` e `backup_prometheus`. I dataset applicativi e +di archivio usano `zstd`; media, Syncthing e backup dei servizi usano `lz4`; +`backups/services` mantiene inoltre una `refreservation` di `500G`. SMB3 pubblica `Archive` solo agli account Samba configurati con password in Vault e ammette la LAN configurata senza esclusioni specifiche per host. NFSv4 esporta soltanto -`media/icloud_photos` all'IP configurato di Aegis con `all_squash` verso UID/GID anonimi `1100`. +`media/photobook` all'IP configurato di Aegis con `all_squash` verso UID/GID anonimi `1100`. L'account di sistema `immich` usa UID/GID `1100`, shell senza login, nessuna appartenenza a `wheel` e i gruppi supplementari `video` e `render`. I Quadlet rootful di Immich Server, ML, cache compatibile Redis, PostgreSQL e NPM condividono una rete Podman. Immich viene eseguito come `1100:1100`; Server e -ML ricevono `/dev/dri` e la libreria iCloud Photos e montata in sola lettura. NPM pubblica `80` e +ML ricevono `/dev/dri` e Photobook e montato in sola lettura su `/external/photobook`. NPM pubblica `80` e `443`, mentre l'amministrazione resta vincolata a `127.0.0.1:81` per l'accesso tramite tunnel SSH. +La fase 1 e limitata ai Quadlet utente rootless di Navidrome e Syncthing su Atlas ed e protetta dal +gate `backend_phase1_enabled`. Navidrome ufficiale `0.63.2` usa il database SQLite sotto `/data` e +non supporta `ND_DATABASE_URL` ne un backend PostgreSQL esterno. Il servizio obsoleto `navidromedb` +e quindi rimosso da Prometheus invece di essere replicato su Atlas. Il ruolo deriva i percorsi dal +pool esistente `zpool`, montato in `/zpool`: musica in sola lettura da `/zpool/media/music`, stato +applicativo Navidrome e `navidrome.db` in `/zpool/archive/app_data/navidrome` e dati Syncthing in +`/zpool/archive/app_data/syncthing`. `profile_atlas` crea questi dataset quando +`atlas_manage_storage` e attivo; il ruolo backend verifica i mountpoint esatti prima di avviare i +container. Nessun ruolo crea il pool. Il ruolo separato `wireguard_overlay` +gestisce `wg0` tra Prometheus (`10.0.0.1`) e Atlas (`10.0.0.2`), genera una sola volta le chiavi +private sui rispettivi host e scambia tramite Ansible soltanto quelle pubbliche. Solo Prometheus apre +pubblicamente `51820/udp`. Le porte backend sono ammesse esclusivamente nella zona firewalld WireGuard. + +`backend_phase1_start_services` resta falso durante il trasferimento dello stato applicativo, quindi +la prima esecuzione reale del backend genera i Quadlet senza creare un database Atlas vuoto. Dopo aver +arrestato Navidrome su Prometheus, copiare l'intera directory `/opt/navidrome/data/` in +`/zpool/archive/app_data/navidrome/`, preservando `navidrome.db` e gli eventuali file SQLite laterali. +Impostare quindi questa variabile a vero e rieseguire il ruolo per abilitare e avviare Navidrome e +Syncthing. Il playbook non copia e non elimina mai i dati applicativi. + +Validare e generare i servizi Atlas con: + +```bash +ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \ +ansible-playbook ansible/site.yml --limit atlas --tags storage \ + -e atlas_manage_storage=true + +ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \ +ansible-playbook ansible/site.yml --limit prometheus,atlas --tags wireguard \ + -e wireguard_overlay_enabled=true + +ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \ +ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 --check --diff \ + -e backend_phase1_enabled=true + +ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \ +ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 \ + -e backend_phase1_enabled=true +``` + +Per il cutover, arrestare il vecchio Navidrome prima di copiare la sua directory dati, verificare +l'ownership dell'account `admin` su Atlas e confermare la presenza del database SQLite copiato prima +di impostare `backend_phase1_start_services: true` in `host_vars/atlas.yml`. Conservare i dati sorgente +e il container legacy `navidromedb` fermo finche Navidrome su Atlas e una prova di restore non sono +stati validati. + Restano da completare retention delle snapshot, topologia Syncthing, validazione WireGuard/firewall, pull di backup da Prometheus, backup cifrati con Borg su una Hetzner Storage Box, backup USB, monitoraggio e test di disaster recovery. Il backlog operativo dettagliato e in `AGENTS.md`. @@ -348,6 +394,8 @@ I principali ruoli attualmente presenti sono: | profile_workstation_dev_wsl | configurazione WSL condivisa per sviluppo | | profile_server | configurazione server | | profile_atlas | configurazione NAS Rocky Linux 9 | +| profile_backend_phase1 | Navidrome e Syncthing rootless su Atlas | +| wireguard_overlay | overlay WireGuard Prometheus/Atlas | | dotfiles_common | distribuzione dotfiles comuni | | dotfiles | distribuzione configurazioni utente | @@ -363,7 +411,9 @@ platform_void -> packages_void + services_runit platform_void & graphical_desktop -> profile_desktop_common + profile_desktop_sway + profile_desktop_niri + profile_desktop_host platform_fedora -> packages_fedora + services_systemd platform_rocky -> packages_rocky + services_systemd +wireguard_overlay -> wireguard_overlay (dopo platform_rocky) atlas -> profile_atlas +role_backend_phase1 -> profile_backend_phase1 (dopo atlas) platform_fedora & role_personal_workstation -> profile_personal_workstation platform_fedora & desktop_gnome -> profile_desktop_gnome workstation_dev_fedora -> profile_workstation_dev_common @@ -379,8 +429,8 @@ Questo significa che, allo stato attuale: - `deadalus` riceve il profilo Fedora WSL tramite play dev dedicati - il server Rocky (`prometheus`) e gestito con pacchetti, servizi, dotfiles server e firewalld - il NAS Rocky (`atlas`) usa un pool ZFS gia esistente, condivisioni NFSv4/SMB limitate alla LAN e Cockpit/45Drives -- lo stack Compose server include `gitea`, `nginx-proxy-manager` e il database PostgreSQL di - Navidrome; Navidrome usa un Quadlet rootless separato e legge il mount rclone di Atlas +- lo stack Compose server include soltanto `gitea` e `nginx-proxy-manager`; Navidrome e Syncthing + della fase 1 sono Quadlet rootless su Atlas # Dotfiles @@ -488,7 +538,7 @@ ansible-lint ansible/roles/ yamllint ansible/path/to/file.yml podman-compose -f /opt/docker/server/docker-compose.yml config ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff -ansible-playbook ansible/site.yml --limit prometheus --tags rclone,navidrome --check --diff +ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 --check --diff -e backend_phase1_enabled=true ``` ## Tag supportati dal playbook @@ -506,6 +556,7 @@ Allo stato attuale `ansible/site.yml` espone questi tag: | `always` | pre-task sempre eseguiti, inclusi caricamento vault e validazioni preliminari | common | | `ai_agents` | installazione agenti AI condivisi | Fedora, WSL | | `atlas` | account, storage, condivisioni e container Atlas | NAS Atlas | +| `backend_phase1` | Quadlet rootless Navidrome e Syncthing | NAS Atlas | | `containers` | Quadlet rootful Atlas | NAS Atlas | | `dotfiles` | distribuzione/configurazione dotfiles | tutti i profili | | `dotfiles:common` | dotfiles comuni condivisi | common, workstation, server | @@ -521,14 +572,12 @@ Allo stato attuale `ansible/site.yml` espone questi tag: | `git` | configurazione Git e GPG desktop | Fedora/GNOME, desktop Void | | `gnome` | configurazione host GNOME | Fedora/GNOME desktop | | `immich` | account e Quadlet Immich | NAS Atlas | -| `navidrome` | mount rclone e Quadlet Navidrome rootless | Prometheus | | `sway` | sessione/configurazione sway / SwayFX (Wayland) | desktop Void | | `niri` | sessione/configurazione Niri (Wayland) | desktop Void | | `npm` | installazione pacchetti npm globali | Fedora/GNOME, desktop Void, WSL | | `nvidia` | componenti NVIDIA desktop | desktop Void | | `packages` | installazione e aggiornamento pacchetti | tutti i profili | | `podman` | integrazione Podman Compose e Quadlet rootless | server | -| `rclone` | mount musica Atlas | Prometheus | | `portal` | configurazione xdg-desktop-portal | desktop Void | | `services` | gestione servizi runit/systemd | tutti i profili | | `sharing` | condivisioni NFSv4 e SMB3 | NAS Atlas | @@ -536,6 +585,7 @@ Allo stato attuale `ansible/site.yml` espone questi tag: | `theme` | configurazione del tema GTK/Qt | desktop Void | | `tmux` | configurazione e plugin tmux | desktop Fedora/Void, WSL | | `vim` | configurazione Vim | dotfiles comuni | +| `wireguard` | overlay WireGuard Prometheus/Atlas | Prometheus, NAS Atlas | | `wsl` | bootstrap e configurazione WSL | WSL | Esempi pratici: diff --git a/README.md b/README.md index 12d93e1..ccf4ca1 100644 --- a/README.md +++ b/README.md @@ -105,29 +105,23 @@ dotfiles and templates. The profile provisions configuration only: it does not t the Compose stack, update DNS, or perform a cutover. The server profile installs platform-specific packages, Podman and podman-compose, declared systemd -services, and firewalld. The manually activated `podman-compose-server` unit now contains Nginx Proxy -Manager, Gitea, and the existing Navidrome PostgreSQL database. Navidrome itself runs as a rootless -user Quadlet and reads the Atlas music dataset from the system `rclone-music.service` mount at -`/mnt/music_atlas`. The Rocky server excludes Syncthing. -Rocky bind mounts use private SELinux relabeling where supported; the read-only FUSE music mount is -passed to Navidrome without relabeling. - -The Atlas music path is gated by `server_atlas_music_enabled`. Before enabling it, replace the -WireGuard address and pinned SSH host-key placeholders in `host_vars/prometheus.yml`, and provide -`vault_prometheus_atlas_sftp_private_key` through encrypted Vault or untracked local vars. The SFTP -key's public half must already be present in Atlas' managed authorized keys. Rclone mounts the exact -remote path `/pool/media/music` read-only and uses a `15G` full VFS cache; the rootless user manager is -kept alive through systemd lingering. -Configure the Prometheus NPM proxy host for Navidrome as `host.containers.internal:4533`; the -Navidrome port is not opened through firewalld. -Before the first enablement, stop the legacy rootful `navidrome` container. The role refuses to start -the rootless replacement while that container is running and never removes the old container or data. +services, and firewalld. The manually activated `podman-compose-server` unit contains the existing +Nginx Proxy Manager and Gitea services. The desired Compose file no longer includes Navidrome, +Syncthing, or the obsolete Navidrome PostgreSQL database. Navidrome and Syncthing belong to Atlas; +official Navidrome uses SQLite instead. Applying the profile does not stop or remove legacy +containers and does not delete `/opt/postgres/data`. Firewalld enables SSH, Cockpit (`9090/tcp`), HTTP and HTTPS. Nginx Proxy Manager publishes only `80/tcp` and `443/tcp`; its administration interface is bound to `127.0.0.1:81` and can be reached from Ikaros or Nymph with the `npm-tunnel` Bash alias. Nextcloud remains disabled and the profile does not provision any `/srv/nextcloud` directories. +The Atlas phase-one work does not change this NPM deployment or its persistent data. Once WireGuard +and the Atlas services are active, configure the current NPM proxy hosts with Navidrome upstream +`http://10.0.0.2:4533` and Syncthing GUI upstream `http://10.0.0.2:8384`. Only the Syncthing web GUI +uses NPM; synchronization traffic remains on its native WireGuard-restricted ports. Configure both +Syncthing authentication and an appropriate NPM access policy before publishing its GUI. + Server identity comes from `server_username`, `server_user_group`, and `server_user_home` in `ansible/inventory/group_vars/server.yml`. `server_username` defaults to `username`, but it can be overridden, for example: ```bash @@ -160,7 +154,7 @@ back in; preserve any uncommitted work separately without copying secrets. ### Data migration Provision Rocky first, then run the migration script **on the retired Ubuntu source host**. It is -dry-run by default and requires an explicit source-stack stop before it can copy PostgreSQL data: +dry-run by default and requires an explicit source-stack stop before it can copy application data: ```bash sudo ./scripts/migrate_prometheus_data.sh \ @@ -173,11 +167,11 @@ sudo ./scripts/migrate_prometheus_data.sh \ --quiesce-source --execute ``` -The script copies Navidrome, music, Nginx Proxy Manager, PostgreSQL and Gitea data. It does not -delete data, move Syncthing, copy `/home/git/.ssh`, start containers, update DNS, or perform a -cutover. The destination SSH host key must already be trusted and the destination account needs -passwordless sudo for `rsync`. It preserves ACLs but not extended attributes, so source SELinux labels -are not transferred; the Rocky Compose bind mounts apply their own `:Z` labels when containers start. +The script copies only Nginx Proxy Manager and Gitea data. It does not delete data, move +Navidrome/Syncthing, copy `/home/git/.ssh`, start containers, update DNS, or perform a cutover. The +destination SSH host key must already be trusted and the destination account needs passwordless sudo +for `rsync`. It preserves ACLs but not extended attributes, so source SELinux labels are not +transferred; the Rocky Compose bind mounts apply their own `:Z` labels when containers start. ## DNS Filter @@ -231,19 +225,67 @@ checking the existing pool and mountpoints; enable `atlas_manage_firewall` only subnet and active firewalld zone. Enable `atlas_manage_media_stack` last, after validating `/dev/dri`, the container paths and the Immich database secret. -With storage management enabled, Atlas creates `archive` (`zstd`), `media/music` (`lz4`), -`media/icloud_photos` (`lz4`), and `backups/services` (`lz4`, `refreservation=500G`) beneath the -pre-existing pool. The existing Work, Syncthing, and Prometheus-backup datasets remain managed and -separate. SMB3 exposes `Archive` only to the configured Vault-backed Samba accounts and admits the -configured LAN without host-specific exclusions. NFSv4 exports only -`media/icloud_photos` to the configured Aegis IP, using `all_squash` with anonymous UID/GID `1100`. +With storage management enabled, Atlas creates the complete dataset hierarchy below the pre-existing +`zpool`: `work`, `archive`, `archive/app_data`, the separate `archive/app_data/navidrome` and +`archive/app_data/syncthing` application datasets, `media`, `media/music`, `media/photobook`, +`backups`, `backups/services`, and `backup_prometheus`. Application/archive datasets use `zstd`, +while media, Syncthing and service-backup datasets use `lz4`; `backups/services` also has a `500G` +refreservation. SMB3 exposes `Archive` only to the configured Vault-backed Samba accounts and admits +the configured LAN without host-specific exclusions. NFSv4 exports only +`media/photobook` to the configured Aegis IP, using `all_squash` with anonymous UID/GID `1100`. The `immich` system account is fixed to UID/GID `1100`, has no login shell or `wheel` membership, and receives `video` and `render` access. The rootful Immich Server, ML, Redis-compatible cache, PostgreSQL, and NPM Quadlets share one Podman network. Immich runs as `1100:1100`; Server and ML receive `/dev/dri`, -and iCloud Photos is mounted read-only as an external library. NPM publishes ports `80` and `443`; its +and Photobook is mounted read-only at `/external/photobook`. NPM publishes ports `80` and `443`; its administration interface remains restricted to `127.0.0.1:81` for SSH-tunnel access. +Phase 1 is limited to rootless Navidrome and Syncthing user Quadlets on Atlas and is gated by +`backend_phase1_enabled`. Official Navidrome `0.63.2` uses its SQLite database below `/data`; it does +not support `ND_DATABASE_URL` or an external PostgreSQL backend. The obsolete `navidromedb` service +was therefore removed from Prometheus instead of being reproduced on Atlas. The role derives all +storage paths from the existing `zpool` mounted at `/zpool`: music is read-only at +`/zpool/media/music`, Navidrome application state and `navidrome.db` are stored at +`/zpool/archive/app_data/navidrome`, and Syncthing persists at +`/zpool/archive/app_data/syncthing`. `profile_atlas` creates these datasets when +`atlas_manage_storage` is enabled; the backend role verifies their exact mountpoints before starting +containers. Neither role creates the pool. The separate `wireguard_overlay` role manages `wg0` +between Prometheus (`10.0.0.1`) and Atlas (`10.0.0.2`), generating private keys once +on their respective hosts and exchanging only public keys through Ansible. Prometheus alone opens +`51820/udp` publicly. Backend ports are admitted only in the WireGuard firewalld zone. + +`backend_phase1_start_services` stays false during the application-state transfer, so the first real +backend run renders the Quadlets without creating an empty Atlas database. After stopping Navidrome +on Prometheus, copy the complete `/opt/navidrome/data/` directory into +`/zpool/archive/app_data/navidrome/`, preserving `navidrome.db` and any SQLite sidecar files. Then set +this variable to true and rerun the role to enable and start Navidrome and Syncthing. The playbook +never copies or deletes application data. + +Validate and render the Atlas services with: + +```bash +ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \ +ansible-playbook ansible/site.yml --limit atlas --tags storage \ + -e atlas_manage_storage=true + +ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \ +ansible-playbook ansible/site.yml --limit prometheus,atlas --tags wireguard \ + -e wireguard_overlay_enabled=true + +ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \ +ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 --check --diff \ + -e backend_phase1_enabled=true + +ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \ +ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 \ + -e backend_phase1_enabled=true +``` + +For the cutover, stop the old Navidrome writer before copying its data directory, verify ownership by +the Atlas `admin` account and confirm that the copied SQLite database is present before changing +`backend_phase1_start_services: true` in `host_vars/atlas.yml`. Keep the source data and the stopped +legacy `navidromedb` container until Navidrome on Atlas and a restore test have been validated. + Snapshot retention, Syncthing topology, WireGuard/firewall validation, Prometheus backup pulls, encrypted Borg backups to a Hetzner Storage Box, USB backup, monitoring, and disaster-recovery tests remain follow-up work. The detailed operational backlog is kept in `AGENTS.md`. @@ -332,6 +374,8 @@ ansible-playbook ansible/site.yml --limit deadalus --tags ai_agents --check --di | `profile_workstation_dev_wsl` | WSL development setup. | | `profile_server` | Server setup. | | `profile_atlas` | Rocky Linux 9 NAS setup. | +| `profile_backend_phase1` | Rootless Navidrome and Syncthing on Atlas. | +| `wireguard_overlay` | Prometheus/Atlas WireGuard overlay. | | `profile_aegis` | Fedora IoT always-on LAN node. | | `dotfiles_common` | Shared user dotfiles. | @@ -343,8 +387,10 @@ platform_void -> packages_void + services_runit platform_void & graphical_desktop -> profile_desktop_common + profile_desktop_sway + profile_desktop_niri + profile_desktop_host platform_fedora -> packages_fedora + services_systemd platform_rocky -> packages_rocky + services_systemd +wireguard_overlay -> wireguard_overlay (after platform_rocky) role_aegis -> profile_aegis atlas -> profile_atlas +role_backend_phase1 -> profile_backend_phase1 (after atlas) rocky_server -> dotfiles_common + profile_server (after platform_rocky) platform_fedora & role_personal_workstation -> profile_personal_workstation platform_fedora & desktop_gnome -> profile_desktop_gnome @@ -414,7 +460,6 @@ ansible-lint ansible/roles/ yamllint ansible/path/to/file.yml podman-compose -f /opt/docker/server/docker-compose.yml config ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff -ansible-playbook ansible/site.yml --limit prometheus --tags rclone,navidrome --check --diff ``` ## Tags @@ -430,6 +475,7 @@ ansible-playbook ansible/site.yml --list-tags | `always` | Common pre-tasks, including optional vault loading. | | `ai_agents` | AI coding-agent install, configuration deployment, and managed-binary removal. | | `atlas` | Atlas NAS account, storage, sharing, and container configuration. | +| `backend_phase1` | Rootless Atlas Navidrome and Syncthing Quadlets. | | `containers` | Rootful Atlas Quadlets. | | `dotfiles` | User configuration across all profiles. | | `dotfiles:common` | Shared dotfiles. | @@ -440,15 +486,14 @@ ansible-playbook ansible/site.yml --list-tags | `emacs` | Shared Emacs setup and authoring dependencies. | | `gnome` | Fedora/GNOME desktop configuration. | | `immich` | Atlas Immich account and Quadlets. | -| `navidrome` | Prometheus rclone mount and rootless Navidrome Quadlet. | | `npm` | Global npm packages. | | `packages` | Package installation and updates. | | `podman` | Podman Compose and rootless Quadlet integration. | -| `rclone` | Prometheus Atlas music mount. | | `services` | runit and systemd services. | | `sharing` | Atlas NFSv4 and SMB3 configuration. | | `storage` | Atlas child ZFS datasets. | | `tmux` | tmux configuration and plugins. | +| `wireguard` | Prometheus/Atlas WireGuard overlay. | | `wsl` | WSL bootstrap and configuration. | ## Bootstrapping a new machine diff --git a/ansible/inventory/group_vars/server.yml b/ansible/inventory/group_vars/server.yml index 102942c..1157516 100644 --- a/ansible/inventory/group_vars/server.yml +++ b/ansible/inventory/group_vars/server.yml @@ -6,29 +6,6 @@ effective_username: "{{ server_username }}" effective_user_group: "{{ server_user_group }}" effective_user_home: "{{ server_user_home }}" server_container_stack_dir: /opt/docker/server -server_atlas_music_enabled: false -server_atlas_sftp_remote_name: atlas -server_atlas_sftp_host: CHANGEME_ATLAS_WIREGUARD_IP -server_atlas_sftp_port: 22 -server_atlas_sftp_username: admin -server_atlas_sftp_remote_path: /pool/media/music -server_atlas_sftp_private_key: "{{ vault_prometheus_atlas_sftp_private_key | default('') }}" -server_atlas_sftp_private_key_file: /etc/rclone/atlas_ed25519 -server_atlas_sftp_known_host: "" -server_atlas_sftp_known_hosts_file: /etc/rclone/known_hosts -server_atlas_wireguard_unit: "" -server_rclone_config_dir: /etc/rclone -server_rclone_config_file: /etc/rclone/rclone.conf -server_rclone_music_service: rclone-music.service -server_rclone_music_mountpoint: /mnt/music_atlas -server_rclone_music_cache_dir: /var/cache/rclone-music -server_navidrome_data_dir: /opt/navidrome/data -server_navidrome_quadlet_dir: "{{ server_user_home }}/.config/containers/systemd" -server_navidrome_env_dir: "{{ server_user_home }}/.config/navidrome" -server_navidrome_env_file: "{{ server_navidrome_env_dir }}/navidrome.env" -server_navidrome_image: docker.io/deluan/navidrome:latest -server_navidrome_port: 4533 -server_navidrome_db_password: "{{ vault_navidrome_db_password | default('') }}" ai_agents: {} vim_plugins_enabled: false @@ -85,10 +62,6 @@ server_directories: owner: root group: root mode: "0755" - - path: "{{ server_navidrome_data_dir }}" - owner: "{{ server_username }}" - group: "{{ server_user_group }}" - mode: "0755" - path: /opt/npm/data owner: root group: root @@ -97,10 +70,6 @@ server_directories: owner: root group: root mode: "0755" - - path: /opt/postgres/data - owner: root - group: root - mode: "0755" - path: /opt/gitea/data owner: root group: root diff --git a/ansible/inventory/host_vars/atlas.yml b/ansible/inventory/host_vars/atlas.yml index e1237ee..c9a1a6d 100644 --- a/ansible/inventory/host_vars/atlas.yml +++ b/ansible/inventory/host_vars/atlas.yml @@ -22,17 +22,35 @@ atlas_firewalld_zone: public atlas_zfs_pool: zpool atlas_zfs_dataset_work: work atlas_zfs_dataset_archive: archive +atlas_zfs_dataset_app_data: archive/app_data +atlas_zfs_dataset_navidrome: archive/app_data/navidrome +atlas_zfs_dataset_syncthing: archive/app_data/syncthing +atlas_zfs_dataset_media: media atlas_zfs_dataset_music: media/music -atlas_zfs_dataset_syncthing: syncthing atlas_zfs_dataset_backup_prometheus: backup_prometheus -atlas_zfs_dataset_icloud_photos: media/icloud_photos +atlas_zfs_dataset_photobook: media/photobook +atlas_zfs_dataset_backups: backups atlas_zfs_dataset_service_backups: backups/services atlas_mount_root: /zpool atlas_manage_storage: false atlas_manage_media_stack: false +wireguard_overlay_enabled: false +wireguard_address: 10.0.0.2/24 +wireguard_peers: + - name: prometheus + host: prometheus + endpoint: "{{ hostvars['prometheus']['ansible_host'] }}:{{ hostvars['prometheus']['wireguard_listen_port'] }}" + allowed_ips: + - 10.0.0.0/24 + persistent_keepalive: 25 + +backend_phase1_enabled: false +backend_phase1_start_services: false +backend_phase1_wireguard_address: 10.0.0.2 + rocky_manage_openzfs_repo: true -rocky_manage_syncthing_binary: true +rocky_manage_syncthing_binary: false rocky_manage_podman: true rocky_podman_packages: - podman @@ -47,7 +65,7 @@ host_packages: - zfs atlas_nfs_exports: - - path: "{{ atlas_icloud_photos_mountpoint }}" + - path: "{{ atlas_photobook_mountpoint }}" client: "{{ atlas_aegis_ip }}" options: - rw diff --git a/ansible/inventory/host_vars/prometheus.yml b/ansible/inventory/host_vars/prometheus.yml index b0b7c49..d45de73 100644 --- a/ansible/inventory/host_vars/prometheus.yml +++ b/ansible/inventory/host_vars/prometheus.yml @@ -7,19 +7,21 @@ ansible_ssh_private_key_file: /home/fscotto/.ssh/id_ed25519 server_username: rocky server_duckdns_domain: fscotto -server_atlas_music_enabled: false -server_atlas_sftp_host: CHANGEME_ATLAS_WIREGUARD_IP -server_atlas_sftp_username: admin -server_atlas_sftp_known_host: CHANGEME_ATLAS_SSH_HOST_KEY -server_atlas_wireguard_unit: wg-quick@wg0.service -server_atlas_sftp_private_key: "{{ vault_prometheus_atlas_sftp_private_key | default('') }}" + +wireguard_overlay_enabled: false +wireguard_address: 10.0.0.1/24 +wireguard_listen_port: 51820 +wireguard_enable_ipv4_forwarding: true +wireguard_peers: + - name: atlas + host: atlas + allowed_ips: + - 10.0.0.2/32 host_packages: - cockpit - cockpit-navigator - cockpit-podman - - fuse3 - - rclone host_enabled_services: - cockpit.socket diff --git a/ansible/inventory/hosts.yml b/ansible/inventory/hosts.yml index 92c65a0..e6f664c 100644 --- a/ansible/inventory/hosts.yml +++ b/ansible/inventory/hosts.yml @@ -95,3 +95,12 @@ all: rocky_server: hosts: prometheus: + + wireguard_overlay: + hosts: + prometheus: + atlas: + + role_backend_phase1: + hosts: + atlas: diff --git a/ansible/roles/profile_atlas/defaults/main.yml b/ansible/roles/profile_atlas/defaults/main.yml index 2887e39..61ebf63 100644 --- a/ansible/roles/profile_atlas/defaults/main.yml +++ b/ansible/roles/profile_atlas/defaults/main.yml @@ -23,23 +23,29 @@ atlas_firewalld_zone: public atlas_zfs_pool: CHANGEME_ZFS_POOL atlas_zfs_dataset_work: work atlas_zfs_dataset_archive: archive +atlas_zfs_dataset_app_data: archive/app_data +atlas_zfs_dataset_navidrome: archive/app_data/navidrome +atlas_zfs_dataset_syncthing: archive/app_data/syncthing +atlas_zfs_dataset_media: media atlas_zfs_dataset_music: media/music -atlas_zfs_dataset_syncthing: syncthing atlas_zfs_dataset_backup_prometheus: backup_prometheus -atlas_zfs_dataset_icloud_photos: media/icloud_photos +atlas_zfs_dataset_photobook: media/photobook +atlas_zfs_dataset_backups: backups atlas_zfs_dataset_service_backups: backups/services atlas_zfs_service_backups_refreservation: 500G atlas_mount_root: /CHANGEME_ATLAS_MOUNT_ROOT atlas_archive_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_archive }}" +atlas_app_data_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_app_data }}" +atlas_navidrome_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_navidrome }}" +atlas_syncthing_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_syncthing }}" +atlas_media_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_media }}" atlas_music_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_music }}" atlas_work_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_work }}" -atlas_syncthing_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_syncthing }}" atlas_backup_prometheus_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_backup_prometheus }}" -atlas_icloud_photos_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_icloud_photos }}" +atlas_photobook_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_photobook }}" +atlas_backups_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_backups }}" atlas_service_backups_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_service_backups }}" -atlas_syncthing_config_dir: "{{ atlas_admin_home }}/.local/state/syncthing" -atlas_syncthing_default_dir: "{{ atlas_syncthing_mountpoint }}/Sync" atlas_45drives_repo_url: https://repo.45drives.com/repofiles/rocky/45drives-enterprise.repo atlas_45drives_repo_file: /etc/yum.repos.d/45drives-enterprise.repo @@ -51,7 +57,7 @@ atlas_45drives_packages: - cockpit-scheduler atlas_nfs_exports: - - path: "{{ atlas_icloud_photos_mountpoint }}" + - path: "{{ atlas_photobook_mountpoint }}" client: "{{ atlas_aegis_ip }}" options: - rw diff --git a/ansible/roles/profile_atlas/handlers/main.yml b/ansible/roles/profile_atlas/handlers/main.yml index 3e67ded..63b201c 100644 --- a/ansible/roles/profile_atlas/handlers/main.yml +++ b/ansible/roles/profile_atlas/handlers/main.yml @@ -28,11 +28,6 @@ name: smb state: restarted -- name: Restart Atlas Syncthing service - ansible.builtin.systemd: - name: atlas-syncthing - state: restarted - - name: Restart Atlas media Quadlets ansible.builtin.systemd: name: "{{ item }}" diff --git a/ansible/roles/profile_atlas/tasks/main.yml b/ansible/roles/profile_atlas/tasks/main.yml index 1199b57..8389b6e 100644 --- a/ansible/roles/profile_atlas/tasks/main.yml +++ b/ansible/roles/profile_atlas/tasks/main.yml @@ -13,6 +13,3 @@ - name: Import Atlas media-container tasks ansible.builtin.import_tasks: containers.yml - -- name: Import Atlas Syncthing tasks - ansible.builtin.import_tasks: syncthing.yml diff --git a/ansible/roles/profile_atlas/tasks/storage.yml b/ansible/roles/profile_atlas/tasks/storage.yml index 51b0404..760e23a 100644 --- a/ansible/roles/profile_atlas/tasks/storage.yml +++ b/ansible/roles/profile_atlas/tasks/storage.yml @@ -16,13 +16,20 @@ properties: compression: zstd mountpoint: "{{ atlas_archive_mountpoint }}" - - name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_music }}" - mountpoint: "{{ atlas_music_mountpoint }}" + - name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_app_data }}" + mountpoint: "{{ atlas_app_data_mountpoint }}" owner: "{{ atlas_admin_username }}" group: "{{ atlas_admin_group }}" properties: - compression: lz4 - mountpoint: "{{ atlas_music_mountpoint }}" + compression: zstd + mountpoint: "{{ atlas_app_data_mountpoint }}" + - name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_navidrome }}" + mountpoint: "{{ atlas_navidrome_mountpoint }}" + owner: "{{ atlas_admin_username }}" + group: "{{ atlas_admin_group }}" + properties: + compression: zstd + mountpoint: "{{ atlas_navidrome_mountpoint }}" - name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_syncthing }}" mountpoint: "{{ atlas_syncthing_mountpoint }}" owner: "{{ atlas_admin_username }}" @@ -30,19 +37,40 @@ properties: compression: lz4 mountpoint: "{{ atlas_syncthing_mountpoint }}" + - name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_media }}" + mountpoint: "{{ atlas_media_mountpoint }}" + owner: "{{ atlas_admin_username }}" + group: "{{ atlas_admin_group }}" + properties: + compression: lz4 + mountpoint: "{{ atlas_media_mountpoint }}" + - name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_music }}" + mountpoint: "{{ atlas_music_mountpoint }}" + owner: "{{ atlas_admin_username }}" + group: "{{ atlas_admin_group }}" + properties: + compression: lz4 + mountpoint: "{{ atlas_music_mountpoint }}" - name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_backup_prometheus }}" mountpoint: "{{ atlas_backup_prometheus_mountpoint }}" owner: "{{ atlas_admin_username }}" group: "{{ atlas_admin_group }}" properties: mountpoint: "{{ atlas_backup_prometheus_mountpoint }}" - - name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_icloud_photos }}" - mountpoint: "{{ atlas_icloud_photos_mountpoint }}" + - name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_photobook }}" + mountpoint: "{{ atlas_photobook_mountpoint }}" owner: "{{ atlas_immich_username }}" group: "{{ atlas_immich_group }}" properties: compression: lz4 - mountpoint: "{{ atlas_icloud_photos_mountpoint }}" + mountpoint: "{{ atlas_photobook_mountpoint }}" + - name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_backups }}" + mountpoint: "{{ atlas_backups_mountpoint }}" + owner: "{{ atlas_admin_username }}" + group: "{{ atlas_admin_group }}" + properties: + compression: lz4 + mountpoint: "{{ atlas_backups_mountpoint }}" - name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_service_backups }}" mountpoint: "{{ atlas_service_backups_mountpoint }}" owner: "{{ atlas_admin_username }}" diff --git a/ansible/roles/profile_atlas/tasks/syncthing.yml b/ansible/roles/profile_atlas/tasks/syncthing.yml deleted file mode 100644 index f2f4c4e..0000000 --- a/ansible/roles/profile_atlas/tasks/syncthing.yml +++ /dev/null @@ -1,40 +0,0 @@ ---- -- name: Create Atlas Syncthing configuration directory - tags: [atlas, syncthing] - ansible.builtin.file: - path: "{{ atlas_syncthing_config_dir }}" - state: directory - owner: "{{ atlas_admin_username }}" - group: "{{ atlas_admin_group }}" - mode: "0700" - when: atlas_manage_storage | bool - -- name: Create Atlas Syncthing default data directory - tags: [atlas, syncthing] - ansible.builtin.file: - path: "{{ atlas_syncthing_default_dir }}" - state: directory - owner: "{{ atlas_admin_username }}" - group: "{{ atlas_admin_group }}" - mode: "0770" - when: atlas_manage_storage | bool - -- name: Render Atlas Syncthing systemd service - tags: [atlas, syncthing] - ansible.builtin.template: - src: atlas-syncthing.service.j2 - dest: /etc/systemd/system/atlas-syncthing.service - owner: root - group: root - mode: "0644" - notify: Restart Atlas Syncthing service - when: atlas_manage_storage | bool - -- name: Enable Atlas Syncthing service - tags: [atlas, syncthing] - ansible.builtin.systemd: - name: atlas-syncthing - enabled: true - state: started - daemon_reload: true - when: atlas_manage_storage | bool diff --git a/ansible/roles/profile_atlas/templates/atlas-immich-server.container.j2 b/ansible/roles/profile_atlas/templates/atlas-immich-server.container.j2 index 9111a07..51c2cd3 100644 --- a/ansible/roles/profile_atlas/templates/atlas-immich-server.container.j2 +++ b/ansible/roles/profile_atlas/templates/atlas-immich-server.container.j2 @@ -16,7 +16,7 @@ Network=atlas-media.network NetworkAlias=atlas-immich-server EnvironmentFile=/etc/immich/immich.env Volume={{ atlas_immich_upload_dir }}:/data:Z -Volume={{ atlas_icloud_photos_mountpoint }}:/external/icloud_photos:ro,Z +Volume={{ atlas_photobook_mountpoint }}:/external/photobook:ro,Z Volume=/etc/localtime:/etc/localtime:ro NoNewPrivileges=true DropCapability=NET_RAW diff --git a/ansible/roles/profile_atlas/templates/atlas-syncthing.service.j2 b/ansible/roles/profile_atlas/templates/atlas-syncthing.service.j2 deleted file mode 100644 index 5506d26..0000000 --- a/ansible/roles/profile_atlas/templates/atlas-syncthing.service.j2 +++ /dev/null @@ -1,16 +0,0 @@ -[Unit] -Description=Atlas Syncthing service -After=network-online.target -Wants=network-online.target - -[Service] -User={{ atlas_admin_username }} -Group={{ atlas_admin_group }} -WorkingDirectory={{ atlas_syncthing_mountpoint }} -Environment=HOME={{ atlas_syncthing_mountpoint }} -ExecStart=/usr/local/bin/syncthing serve --no-browser --no-restart --no-upgrade --home={{ atlas_syncthing_config_dir }} -Restart=on-failure -RestartSec=5 - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/profile_backend_phase1/defaults/main.yml b/ansible/roles/profile_backend_phase1/defaults/main.yml new file mode 100644 index 0000000..e50774b --- /dev/null +++ b/ansible/roles/profile_backend_phase1/defaults/main.yml @@ -0,0 +1,28 @@ +--- +backend_phase1_enabled: false +backend_phase1_start_services: false +backend_phase1_username: "{{ atlas_admin_username }}" +backend_phase1_user_group: "{{ atlas_admin_group }}" +backend_phase1_user_home: "{{ atlas_admin_home }}" +backend_phase1_quadlet_dir: "{{ backend_phase1_user_home }}/.config/containers/systemd" +backend_phase1_wireguard_interface: wg0 +backend_phase1_wireguard_address: CHANGEME_ATLAS_WIREGUARD_ADDRESS +backend_phase1_wireguard_firewalld_zone: wireguard +backend_phase1_music_dir: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_music }}" +backend_phase1_app_data_root: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_app_data }}" +backend_phase1_navidrome_data_dir: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_navidrome }}" +backend_phase1_syncthing_root: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_syncthing }}" +backend_phase1_music_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_music }}" +backend_phase1_app_data_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_app_data }}" +backend_phase1_navidrome_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_navidrome }}" +backend_phase1_syncthing_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_syncthing }}" +backend_phase1_navidrome_image: docker.io/deluan/navidrome:0.63.2 +backend_phase1_syncthing_image: docker.io/syncthing/syncthing:2.1.3 +backend_phase1_navidrome_port: 4533 +backend_phase1_syncthing_gui_port: 8384 +backend_phase1_syncthing_transfer_port: 22000 +backend_phase1_syncthing_discovery_port: 21027 +backend_phase1_timezone: Europe/Rome +backend_phase1_services: + - atlas-navidrome.service + - atlas-syncthing.service diff --git a/ansible/roles/profile_backend_phase1/handlers/main.yml b/ansible/roles/profile_backend_phase1/handlers/main.yml new file mode 100644 index 0000000..910a0aa --- /dev/null +++ b/ansible/roles/profile_backend_phase1/handlers/main.yml @@ -0,0 +1,32 @@ +--- +- name: Restart Atlas Navidrome Quadlet + tags: [backend_phase1, navidrome, podman, services] + become_user: "{{ backend_phase1_username }}" + ansible.builtin.systemd: + name: atlas-navidrome.service + scope: user + state: restarted + daemon_reload: true + environment: + XDG_RUNTIME_DIR: "/run/user/{{ backend_phase1_uid }}" + DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ backend_phase1_uid }}/bus" + when: + - backend_phase1_enabled | bool + - backend_phase1_start_services | bool + - not ansible_check_mode + +- name: Restart Atlas Syncthing Quadlet + tags: [backend_phase1, syncthing, podman, services] + become_user: "{{ backend_phase1_username }}" + ansible.builtin.systemd: + name: atlas-syncthing.service + scope: user + state: restarted + daemon_reload: true + environment: + XDG_RUNTIME_DIR: "/run/user/{{ backend_phase1_uid }}" + DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ backend_phase1_uid }}/bus" + when: + - backend_phase1_enabled | bool + - backend_phase1_start_services | bool + - not ansible_check_mode diff --git a/ansible/roles/profile_backend_phase1/tasks/main.yml b/ansible/roles/profile_backend_phase1/tasks/main.yml new file mode 100644 index 0000000..d5c4703 --- /dev/null +++ b/ansible/roles/profile_backend_phase1/tasks/main.yml @@ -0,0 +1,163 @@ +--- +- name: Configure Atlas phase-one backend services + tags: [backend_phase1, podman] + when: backend_phase1_enabled | bool + block: + - name: Validate phase-one backend inputs + ansible.builtin.assert: + that: + - not (atlas_manage_media_stack | bool) + - backend_phase1_wireguard_interface in ansible_facts.interfaces + - backend_phase1_wireguard_address != 'CHANGEME_ATLAS_WIREGUARD_ADDRESS' + - backend_phase1_music_dir.startswith('/') + - backend_phase1_app_data_root.startswith('/') + - backend_phase1_navidrome_data_dir.startswith(backend_phase1_app_data_root + '/') + - backend_phase1_syncthing_root.startswith(backend_phase1_app_data_root + '/') + fail_msg: >- + Disable the rootful media-stack gate and provide the active + WireGuard interface/address and absolute ZFS-backed paths before + enabling phase one. This role does not manage Prometheus or migrate + Navidrome application data. + + - name: Read the rootless service account + ansible.builtin.getent: + database: passwd + key: "{{ backend_phase1_username }}" + + - name: Record rootless service account IDs + ansible.builtin.set_fact: + backend_phase1_uid: "{{ ansible_facts['getent_passwd'][backend_phase1_username][1] }}" + backend_phase1_gid: "{{ ansible_facts['getent_passwd'][backend_phase1_username][2] }}" + + - name: Read system service state before starting rootless Syncthing + ansible.builtin.service_facts: + + - name: Refuse to overlap a system-level Atlas Syncthing service + ansible.builtin.assert: + that: + - >- + ansible_facts.services['atlas-syncthing.service'] is not defined + or ansible_facts.services['atlas-syncthing.service'].state != 'running' + fail_msg: >- + Remove or stop the system-level atlas-syncthing.service before + enabling the rootless Syncthing Quadlet. + + - name: Inspect required ZFS datasets + community.general.zfs_facts: + name: "{{ item.dataset }}" + properties: name,mounted,mountpoint + loop: + - dataset: "{{ backend_phase1_music_dataset }}" + mountpoint: "{{ backend_phase1_music_dir }}" + - dataset: "{{ backend_phase1_app_data_dataset }}" + mountpoint: "{{ backend_phase1_app_data_root }}" + - dataset: "{{ backend_phase1_navidrome_dataset }}" + mountpoint: "{{ backend_phase1_navidrome_data_dir }}" + - dataset: "{{ backend_phase1_syncthing_dataset }}" + mountpoint: "{{ backend_phase1_syncthing_root }}" + loop_control: + label: "{{ item.dataset }}" + register: backend_phase1_zfs_facts + + - name: Require mounted datasets at the declared paths + ansible.builtin.assert: + that: + - item.ansible_facts.ansible_zfs_datasets | length == 1 + - item.ansible_facts.ansible_zfs_datasets[0].mounted == 'yes' + - item.ansible_facts.ansible_zfs_datasets[0].mountpoint == item.item.mountpoint + fail_msg: >- + {{ item.item.dataset }} must already be mounted at + {{ item.item.mountpoint }}. The phase-one role never creates or + imports the pool. + loop: "{{ backend_phase1_zfs_facts.results }}" + loop_control: + label: "{{ item.item.dataset }}" + + - name: Enable lingering for the rootless service account + ansible.builtin.command: + argv: + - loginctl + - enable-linger + - "{{ backend_phase1_username }}" + creates: "/var/lib/systemd/linger/{{ backend_phase1_username }}" + + - name: Start the rootless user systemd manager + ansible.builtin.systemd: + name: "user@{{ backend_phase1_uid }}.service" + state: started + when: not ansible_check_mode + + - name: Create rootless Quadlet and application directories + ansible.builtin.file: + path: "{{ item.path }}" + state: directory + owner: "{{ backend_phase1_username }}" + group: "{{ backend_phase1_user_group }}" + mode: "{{ item.mode }}" + loop: + - path: "{{ backend_phase1_quadlet_dir }}" + mode: "0700" + - path: "{{ backend_phase1_navidrome_data_dir }}" + mode: "0750" + - path: "{{ backend_phase1_syncthing_root }}" + mode: "0750" + loop_control: + label: "{{ item.path }}" + + - name: Render the rootless Navidrome Quadlet + ansible.builtin.template: + src: atlas-navidrome.container.j2 + dest: "{{ backend_phase1_quadlet_dir }}/atlas-navidrome.container" + owner: "{{ backend_phase1_username }}" + group: "{{ backend_phase1_user_group }}" + mode: "0644" + notify: Restart Atlas Navidrome Quadlet + + - name: Render the rootless Syncthing Quadlet + ansible.builtin.template: + src: atlas-syncthing.container.j2 + dest: "{{ backend_phase1_quadlet_dir }}/atlas-syncthing.container" + owner: "{{ backend_phase1_username }}" + group: "{{ backend_phase1_user_group }}" + mode: "0644" + notify: Restart Atlas Syncthing Quadlet + + - name: Reload the rootless user systemd manager + become_user: "{{ backend_phase1_username }}" + ansible.builtin.systemd: + scope: user + daemon_reload: true + environment: + XDG_RUNTIME_DIR: "/run/user/{{ backend_phase1_uid }}" + DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ backend_phase1_uid }}/bus" + when: not ansible_check_mode + + - name: Permit phase-one services only through the WireGuard zone + ansible.posix.firewalld: + port: "{{ item }}" + zone: "{{ backend_phase1_wireguard_firewalld_zone }}" + state: enabled + permanent: true + immediate: true + loop: + - "{{ backend_phase1_navidrome_port }}/tcp" + - "{{ backend_phase1_syncthing_gui_port }}/tcp" + - "{{ backend_phase1_syncthing_transfer_port }}/tcp" + - "{{ backend_phase1_syncthing_transfer_port }}/udp" + - "{{ backend_phase1_syncthing_discovery_port }}/udp" + + - name: Start rootless phase-one Quadlets + become_user: "{{ backend_phase1_username }}" + ansible.builtin.systemd: + name: "{{ item }}" + scope: user + state: started + enabled: true + daemon_reload: true + environment: + XDG_RUNTIME_DIR: "/run/user/{{ backend_phase1_uid }}" + DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ backend_phase1_uid }}/bus" + loop: "{{ backend_phase1_services }}" + when: + - backend_phase1_start_services | bool + - not ansible_check_mode diff --git a/ansible/roles/profile_backend_phase1/templates/atlas-navidrome.container.j2 b/ansible/roles/profile_backend_phase1/templates/atlas-navidrome.container.j2 new file mode 100644 index 0000000..eace113 --- /dev/null +++ b/ansible/roles/profile_backend_phase1/templates/atlas-navidrome.container.j2 @@ -0,0 +1,28 @@ +# Managed by Ansible. Do not edit manually. +[Unit] +Description=Atlas rootless Navidrome backend + +[Container] +ContainerName=atlas-navidrome +Image={{ backend_phase1_navidrome_image }} +UserNS=keep-id +User={{ backend_phase1_uid }} +Group={{ backend_phase1_gid }} +PublishPort={{ backend_phase1_wireguard_address }}:{{ backend_phase1_navidrome_port }}:4533 +Environment=ND_LOGLEVEL=info +Environment=ND_SCANSCHEDULE=1h +Environment=ND_SESSIONTIMEOUT=24h +Environment=ND_ENABLETRANSCODING=true +Environment=TZ={{ backend_phase1_timezone }} +Volume={{ backend_phase1_navidrome_data_dir }}:/data:Z +Volume={{ backend_phase1_music_dir }}:/music:ro,z +NoNewPrivileges=true +DropCapability=all + +[Service] +Restart=always +RestartSec=10 +TimeoutStartSec=900 + +[Install] +WantedBy=default.target diff --git a/ansible/roles/profile_backend_phase1/templates/atlas-syncthing.container.j2 b/ansible/roles/profile_backend_phase1/templates/atlas-syncthing.container.j2 new file mode 100644 index 0000000..e14eb97 --- /dev/null +++ b/ansible/roles/profile_backend_phase1/templates/atlas-syncthing.container.j2 @@ -0,0 +1,27 @@ +# Managed by Ansible. Do not edit manually. +[Unit] +Description=Atlas rootless Syncthing backend + +[Container] +ContainerName=atlas-syncthing +Image={{ backend_phase1_syncthing_image }} +HostName=atlas-syncthing +Network=host +UserNS=keep-id +User={{ backend_phase1_uid }} +Group={{ backend_phase1_gid }} +Environment=HOME=/var/syncthing +Environment=STHOMEDIR=/var/syncthing/config +Environment=STGUIADDRESS={{ backend_phase1_wireguard_address }}:{{ backend_phase1_syncthing_gui_port }} +Environment=TZ={{ backend_phase1_timezone }} +Volume={{ backend_phase1_syncthing_root }}:/var/syncthing:Z +NoNewPrivileges=true +DropCapability=all + +[Service] +Restart=always +RestartSec=10 +TimeoutStartSec=900 + +[Install] +WantedBy=default.target diff --git a/ansible/roles/profile_server/handlers/main.yml b/ansible/roles/profile_server/handlers/main.yml index d5852e1..fd614d2 100644 --- a/ansible/roles/profile_server/handlers/main.yml +++ b/ansible/roles/profile_server/handlers/main.yml @@ -9,29 +9,3 @@ tags: [services] ansible.builtin.systemd: daemon_reload: true - -- name: Restart rclone music mount - tags: [services, rclone, navidrome] - ansible.builtin.systemd: - name: "{{ server_rclone_music_service }}" - state: restarted - daemon_reload: true - when: - - server_atlas_music_enabled | bool - - not ansible_check_mode - -- name: Restart rootless Navidrome - tags: [services, podman, rclone, navidrome] - become_user: "{{ server_username }}" - ansible.builtin.systemd: - name: navidrome.service - scope: user - state: restarted - daemon_reload: true - environment: - XDG_RUNTIME_DIR: "/run/user/{{ ansible_facts['getent_passwd'][server_username][1] }}" - DBUS_SESSION_BUS_ADDRESS: >- - unix:path=/run/user/{{ ansible_facts['getent_passwd'][server_username][1] }}/bus - when: - - server_atlas_music_enabled | bool - - not ansible_check_mode diff --git a/ansible/roles/profile_server/tasks/main.yml b/ansible/roles/profile_server/tasks/main.yml index 2627faf..f1d395d 100644 --- a/ansible/roles/profile_server/tasks/main.yml +++ b/ansible/roles/profile_server/tasks/main.yml @@ -8,17 +8,6 @@ fail_msg: >- server_firewall_backend must be firewalld for the Rocky server profile. -- name: Require server database secret variables - tags: [dotfiles, dotfiles:server, services, navidrome] - ansible.builtin.assert: - that: - - server_navidrome_db_password | length > 0 - - (vault_postgres_root_password | default('')) | length > 0 - fail_msg: >- - Define vault_navidrome_db_password and vault_postgres_root_password in Vault - before rendering the Navidrome database configuration. - no_log: true - - name: Configure DuckDNS updater tags: [dotfiles, dotfiles:server, duckdns] ansible.builtin.import_tasks: duckdns.yml @@ -72,12 +61,6 @@ tags: [services, podman] ansible.builtin.include_tasks: podman-compose.yml -- name: Configure Atlas music mount - ansible.builtin.import_tasks: rclone-music.yml - -- name: Configure rootless Navidrome - ansible.builtin.import_tasks: navidrome.yml - - name: Disable SSH root login on server tags: [services] ansible.builtin.lineinfile: diff --git a/ansible/roles/profile_server/tasks/navidrome.yml b/ansible/roles/profile_server/tasks/navidrome.yml deleted file mode 100644 index 5009007..0000000 --- a/ansible/roles/profile_server/tasks/navidrome.yml +++ /dev/null @@ -1,114 +0,0 @@ ---- -- name: Read server account details for rootless Navidrome - tags: [services, podman, rclone, navidrome] - ansible.builtin.getent: - database: passwd - key: "{{ server_username }}" - when: server_atlas_music_enabled | bool - -- name: Check for a running legacy rootful Navidrome container - tags: [services, podman, navidrome] - ansible.builtin.command: - argv: - - podman - - container - - inspect - - --format - - "{{ '{{.State.Running}}' }}" - - navidrome - register: server_legacy_navidrome_container - changed_when: false - failed_when: false - check_mode: false - when: server_atlas_music_enabled | bool - -- name: Refuse concurrent legacy and rootless Navidrome - tags: [services, podman, navidrome] - ansible.builtin.assert: - that: - - >- - server_legacy_navidrome_container.rc != 0 - or server_legacy_navidrome_container.stdout | trim != 'true' - fail_msg: >- - Stop the legacy rootful Navidrome container before enabling the rootless Quadlet. - The playbook does not remove the old container or its data automatically. - when: server_atlas_music_enabled | bool - -- name: Enable lingering for rootless Navidrome - tags: [services, podman, navidrome] - ansible.builtin.command: - argv: - - loginctl - - enable-linger - - "{{ server_username }}" - creates: "/var/lib/systemd/linger/{{ server_username }}" - when: server_atlas_music_enabled | bool - -- name: Start server user systemd manager - tags: [services, podman, navidrome] - ansible.builtin.systemd: - name: "user@{{ ansible_facts['getent_passwd'][server_username][1] }}.service" - state: started - when: - - server_atlas_music_enabled | bool - - not ansible_check_mode - -- name: Create rootless Navidrome Quadlet directory - tags: [services, podman, navidrome] - ansible.builtin.file: - path: "{{ server_navidrome_quadlet_dir }}" - state: directory - owner: "{{ server_username }}" - group: "{{ server_user_group }}" - mode: "0700" - when: server_atlas_music_enabled | bool - -- name: Create private rootless Navidrome configuration directory - tags: [services, podman, navidrome] - ansible.builtin.file: - path: "{{ server_navidrome_env_dir }}" - state: directory - owner: "{{ server_username }}" - group: "{{ server_user_group }}" - mode: "0700" - when: server_atlas_music_enabled | bool - -- name: Render Vault-backed rootless Navidrome environment - tags: [services, podman, navidrome] - ansible.builtin.template: - src: navidrome.env.j2 - dest: "{{ server_navidrome_env_file }}" - owner: "{{ server_username }}" - group: "{{ server_user_group }}" - mode: "0600" - no_log: true - diff: false - notify: Restart rootless Navidrome - when: server_atlas_music_enabled | bool - -- name: Render rootless Navidrome Quadlet - tags: [services, podman, navidrome] - ansible.builtin.template: - src: navidrome.container.j2 - dest: "{{ server_navidrome_quadlet_dir }}/navidrome.container" - owner: "{{ server_username }}" - group: "{{ server_user_group }}" - mode: "0644" - notify: Restart rootless Navidrome - when: server_atlas_music_enabled | bool - -- name: Start rootless Navidrome Quadlet - tags: [services, podman, navidrome] - become_user: "{{ server_username }}" - ansible.builtin.systemd: - name: navidrome.service - scope: user - state: started - daemon_reload: true - environment: - XDG_RUNTIME_DIR: "/run/user/{{ ansible_facts['getent_passwd'][server_username][1] }}" - DBUS_SESSION_BUS_ADDRESS: >- - unix:path=/run/user/{{ ansible_facts['getent_passwd'][server_username][1] }}/bus - when: - - server_atlas_music_enabled | bool - - not ansible_check_mode diff --git a/ansible/roles/profile_server/tasks/rclone-music.yml b/ansible/roles/profile_server/tasks/rclone-music.yml deleted file mode 100644 index ef6e555..0000000 --- a/ansible/roles/profile_server/tasks/rclone-music.yml +++ /dev/null @@ -1,105 +0,0 @@ ---- -- name: Require completed Atlas SFTP mount configuration - tags: [services, rclone, navidrome] - ansible.builtin.assert: - that: - - server_atlas_sftp_host != 'CHANGEME_ATLAS_WIREGUARD_IP' - - server_atlas_sftp_known_host != 'CHANGEME_ATLAS_SSH_HOST_KEY' - - server_atlas_sftp_known_host | length > 0 - - server_atlas_sftp_private_key | length > 0 - - server_atlas_sftp_remote_path == '/pool/media/music' - fail_msg: >- - Define the Atlas WireGuard address, pinned SSH host key and Vault-backed - SFTP private key before enabling the music mount. - no_log: true - when: server_atlas_music_enabled | bool - -- name: Create private rclone configuration directory - tags: [services, rclone, navidrome] - ansible.builtin.file: - path: "{{ server_rclone_config_dir }}" - state: directory - owner: root - group: "{{ server_user_group }}" - mode: "0750" - when: server_atlas_music_enabled | bool - -- name: Install Vault-backed Atlas SFTP private key - tags: [services, rclone, navidrome] - ansible.builtin.copy: - content: "{{ server_atlas_sftp_private_key | trim }}\n" - dest: "{{ server_atlas_sftp_private_key_file }}" - owner: "{{ server_username }}" - group: "{{ server_user_group }}" - mode: "0600" - no_log: true - diff: false - notify: - - Restart rclone music mount - - Restart rootless Navidrome - when: server_atlas_music_enabled | bool - -- name: Install pinned Atlas SSH host key - tags: [services, rclone, navidrome] - ansible.builtin.copy: - content: "{{ server_atlas_sftp_known_host | trim }}\n" - dest: "{{ server_atlas_sftp_known_hosts_file }}" - owner: "{{ server_username }}" - group: "{{ server_user_group }}" - mode: "0640" - notify: - - Restart rclone music mount - - Restart rootless Navidrome - when: server_atlas_music_enabled | bool - -- name: Render Atlas SFTP rclone configuration - tags: [services, rclone, navidrome] - ansible.builtin.template: - src: rclone.conf.j2 - dest: "{{ server_rclone_config_file }}" - owner: "{{ server_username }}" - group: "{{ server_user_group }}" - mode: "0600" - notify: - - Restart rclone music mount - - Restart rootless Navidrome - when: server_atlas_music_enabled | bool - -- name: Create rclone music directories - tags: [services, rclone, navidrome] - ansible.builtin.file: - path: "{{ item }}" - state: directory - owner: "{{ server_username }}" - group: "{{ server_user_group }}" - mode: "0750" - loop: - - "{{ server_rclone_music_mountpoint }}" - - "{{ server_rclone_music_cache_dir }}" - loop_control: - label: "{{ item }}" - when: server_atlas_music_enabled | bool - -- name: Render rclone music system service - tags: [services, rclone, navidrome] - ansible.builtin.template: - src: rclone-music.service.j2 - dest: "/etc/systemd/system/{{ server_rclone_music_service }}" - owner: root - group: root - mode: "0644" - notify: - - Restart rclone music mount - - Restart rootless Navidrome - when: server_atlas_music_enabled | bool - -- name: Enable and start rclone music mount - tags: [services, rclone, navidrome] - ansible.builtin.systemd: - name: "{{ server_rclone_music_service }}" - enabled: true - state: started - daemon_reload: true - when: - - server_atlas_music_enabled | bool - - not ansible_check_mode diff --git a/ansible/roles/profile_server/templates/navidrome.container.j2 b/ansible/roles/profile_server/templates/navidrome.container.j2 deleted file mode 100644 index 0c2e754..0000000 --- a/ansible/roles/profile_server/templates/navidrome.container.j2 +++ /dev/null @@ -1,21 +0,0 @@ -# Managed by Ansible. Do not edit manually. -[Unit] -Description=Rootless Navidrome music server - -[Container] -ContainerName=navidrome -Image={{ server_navidrome_image }} -Network=host -EnvironmentFile={{ server_navidrome_env_file }} -Volume={{ server_navidrome_data_dir }}:/data -Volume={{ server_rclone_music_mountpoint }}:/music:ro -SecurityLabelDisable=true - -[Service] -ExecStartPre=/usr/bin/mountpoint -q {{ server_rclone_music_mountpoint }} -Restart=always -RestartSec=10s -TimeoutStartSec=900 - -[Install] -WantedBy=default.target diff --git a/ansible/roles/profile_server/templates/navidrome.env.j2 b/ansible/roles/profile_server/templates/navidrome.env.j2 deleted file mode 100644 index 73c2596..0000000 --- a/ansible/roles/profile_server/templates/navidrome.env.j2 +++ /dev/null @@ -1,7 +0,0 @@ -# Managed by Ansible. Contains a Vault secret; do not edit or copy. -ND_DATABASE_URL={{ ('postgres://navidrome:' ~ server_navidrome_db_password ~ '@127.0.0.1:5432/navidrome_db?sslmode=disable') | to_json }} -ND_PORT={{ server_navidrome_port }} -ND_SCANSCHEDULE="1h" -ND_SESSIONTIMEOUT="24h" -ND_ENABLETRANSCODING="true" -ND_LOGLEVEL="info" diff --git a/ansible/roles/profile_server/templates/rclone-music.service.j2 b/ansible/roles/profile_server/templates/rclone-music.service.j2 deleted file mode 100644 index 6c9a3e0..0000000 --- a/ansible/roles/profile_server/templates/rclone-music.service.j2 +++ /dev/null @@ -1,29 +0,0 @@ -[Unit] -Description=Read-only Atlas music mount via rclone SFTP -Wants=network-online.target -After=network-online.target -{% if server_atlas_wireguard_unit | length > 0 %} -Wants={{ server_atlas_wireguard_unit }} -After={{ server_atlas_wireguard_unit }} -{% endif %} - -[Service] -Type=notify -User={{ server_username }} -Group={{ server_user_group }} -ExecStart=/usr/bin/rclone mount \ - {{ server_atlas_sftp_remote_name }}:{{ server_atlas_sftp_remote_path }} \ - {{ server_rclone_music_mountpoint }} \ - --config {{ server_rclone_config_file }} \ - --cache-dir {{ server_rclone_music_cache_dir }} \ - --read-only \ - --vfs-cache-mode full \ - --vfs-cache-max-size 15G \ - --vfs-read-chunk-size 5M \ - --dir-cache-time 24h -ExecStop=-/usr/bin/fusermount3 -uz {{ server_rclone_music_mountpoint }} -Restart=on-failure -RestartSec=10s - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/profile_server/templates/rclone.conf.j2 b/ansible/roles/profile_server/templates/rclone.conf.j2 deleted file mode 100644 index 863ebd0..0000000 --- a/ansible/roles/profile_server/templates/rclone.conf.j2 +++ /dev/null @@ -1,9 +0,0 @@ -# Managed by Ansible. Authentication uses the separately deployed Vault-backed key. -[{{ server_atlas_sftp_remote_name }}] -type = sftp -host = {{ server_atlas_sftp_host }} -user = {{ server_atlas_sftp_username }} -port = {{ server_atlas_sftp_port }} -key_file = {{ server_atlas_sftp_private_key_file }} -known_hosts_file = {{ server_atlas_sftp_known_hosts_file }} -shell_type = unix diff --git a/ansible/roles/wireguard_overlay/defaults/main.yml b/ansible/roles/wireguard_overlay/defaults/main.yml new file mode 100644 index 0000000..2021f1c --- /dev/null +++ b/ansible/roles/wireguard_overlay/defaults/main.yml @@ -0,0 +1,15 @@ +--- +wireguard_overlay_enabled: false +wireguard_interface: wg0 +wireguard_config_dir: /etc/wireguard +wireguard_private_key_path: "{{ wireguard_config_dir }}/{{ wireguard_interface }}.key" +wireguard_config_path: "{{ wireguard_config_dir }}/{{ wireguard_interface }}.conf" +wireguard_address: CHANGEME_WIREGUARD_ADDRESS +wireguard_listen_port: 0 +wireguard_mtu: 1420 +wireguard_firewalld_zone: wireguard +wireguard_public_firewalld_zone: public +wireguard_enable_ipv4_forwarding: false +wireguard_peers: [] +wireguard_packages: + - wireguard-tools diff --git a/ansible/roles/wireguard_overlay/handlers/main.yml b/ansible/roles/wireguard_overlay/handlers/main.yml new file mode 100644 index 0000000..2473499 --- /dev/null +++ b/ansible/roles/wireguard_overlay/handlers/main.yml @@ -0,0 +1,10 @@ +--- +- name: Restart WireGuard interface + tags: [wireguard, services] + ansible.builtin.systemd: + name: "wg-quick@{{ wireguard_interface }}.service" + state: restarted + daemon_reload: true + when: + - wireguard_overlay_enabled | bool + - not ansible_check_mode diff --git a/ansible/roles/wireguard_overlay/tasks/main.yml b/ansible/roles/wireguard_overlay/tasks/main.yml new file mode 100644 index 0000000..3295c35 --- /dev/null +++ b/ansible/roles/wireguard_overlay/tasks/main.yml @@ -0,0 +1,158 @@ +--- +- name: Configure WireGuard overlay + tags: [wireguard, services] + when: wireguard_overlay_enabled | bool + block: + - name: Validate WireGuard host configuration + ansible.builtin.assert: + that: + - wireguard_address != 'CHANGEME_WIREGUARD_ADDRESS' + - wireguard_address is match('^[0-9]{1,3}(\.[0-9]{1,3}){3}/[0-9]{1,2}$') + - wireguard_peers | length > 0 + - wireguard_peers | map(attribute='host') | difference(ansible_play_hosts_all) | length == 0 + fail_msg: >- + Configure this host's WireGuard address and peers, and run the first + key bootstrap against every peer in the same play. + + - name: Install WireGuard userspace tools + ansible.builtin.dnf: + name: "{{ wireguard_packages }}" + state: present + + - name: Create private WireGuard configuration directory + ansible.builtin.file: + path: "{{ wireguard_config_dir }}" + state: directory + owner: root + group: root + mode: "0700" + + - name: Check for an existing WireGuard private key + ansible.builtin.stat: + path: "{{ wireguard_private_key_path }}" + register: wireguard_private_key_stat + + - name: Generate a missing WireGuard private key + ansible.builtin.command: + argv: + - wg + - genkey + register: wireguard_generated_private_key + changed_when: true + no_log: true + when: + - not wireguard_private_key_stat.stat.exists + - not ansible_check_mode + + - name: Persist the generated WireGuard private key + ansible.builtin.copy: + content: "{{ wireguard_generated_private_key.stdout }}\n" + dest: "{{ wireguard_private_key_path }}" + owner: root + group: root + mode: "0600" + no_log: true + when: + - not wireguard_private_key_stat.stat.exists + - not ansible_check_mode + + - name: Require a private key during check mode + ansible.builtin.assert: + that: + - wireguard_private_key_stat.stat.exists + fail_msg: >- + The initial WireGuard key generation cannot be simulated safely in + check mode. Run the gated WireGuard play once without --check. + when: ansible_check_mode + + - name: Read the persisted WireGuard private key + ansible.builtin.slurp: + src: "{{ wireguard_private_key_path }}" + register: wireguard_private_key_material + no_log: true + + - name: Derive this host's WireGuard public key + ansible.builtin.command: + argv: + - wg + - pubkey + stdin: "{{ wireguard_private_key_material.content | b64decode | trim }}" + register: wireguard_derived_public_key + changed_when: false + no_log: true + + - name: Publish this host's WireGuard public key + ansible.builtin.set_fact: + wireguard_public_key: "{{ wireguard_derived_public_key.stdout }}" + + - name: Require every peer's generated public key + ansible.builtin.assert: + that: + - hostvars[item.host].wireguard_public_key is defined + - hostvars[item.host].wireguard_public_key | length > 0 + fail_msg: >- + The public key for {{ item.host }} is unavailable. The first + WireGuard run must include every overlay host. + loop: "{{ wireguard_peers }}" + loop_control: + label: "{{ item.name }}" + + - name: Render the private WireGuard interface configuration + ansible.builtin.template: + src: wg.conf.j2 + dest: "{{ wireguard_config_path }}" + owner: root + group: root + mode: "0600" + diff: false + no_log: true + notify: Restart WireGuard interface + + - name: Enable IPv4 forwarding for the ingress host + ansible.posix.sysctl: + name: net.ipv4.ip_forward + value: "1" + state: present + sysctl_set: true + reload: true + when: wireguard_enable_ipv4_forwarding | bool + + - name: Create the WireGuard firewalld zone + ansible.posix.firewalld: + zone: "{{ wireguard_firewalld_zone }}" + state: present + permanent: true + register: wireguard_firewalld_zone_result + + - name: Reload firewalld after creating the WireGuard zone + ansible.builtin.systemd: + name: firewalld.service + state: reloaded + when: + - wireguard_firewalld_zone_result is changed + - not ansible_check_mode + + - name: Assign the WireGuard interface to its firewalld zone + ansible.posix.firewalld: + interface: "{{ wireguard_interface }}" + zone: "{{ wireguard_firewalld_zone }}" + state: enabled + permanent: true + immediate: true + + - name: Permit this host's public WireGuard listener + ansible.posix.firewalld: + port: "{{ wireguard_listen_port }}/udp" + zone: "{{ wireguard_public_firewalld_zone }}" + state: enabled + permanent: true + immediate: true + when: wireguard_listen_port | int > 0 + + - name: Enable the WireGuard interface + ansible.builtin.systemd: + name: "wg-quick@{{ wireguard_interface }}.service" + enabled: true + state: started + daemon_reload: true + when: not ansible_check_mode diff --git a/ansible/roles/wireguard_overlay/templates/wg.conf.j2 b/ansible/roles/wireguard_overlay/templates/wg.conf.j2 new file mode 100644 index 0000000..fa98245 --- /dev/null +++ b/ansible/roles/wireguard_overlay/templates/wg.conf.j2 @@ -0,0 +1,25 @@ +# Managed by Ansible. Do not edit manually. +[Interface] +Address = {{ wireguard_address }} +PrivateKey = {{ wireguard_private_key_material.content | b64decode | trim }} +MTU = {{ wireguard_mtu }} +SaveConfig = false +{% if wireguard_listen_port | int > 0 %} +ListenPort = {{ wireguard_listen_port }} +{% endif %} + +{% for peer in wireguard_peers %} +[Peer] +# {{ peer.name }} +PublicKey = {{ hostvars[peer.host].wireguard_public_key }} +AllowedIPs = {{ peer.allowed_ips | join(', ') }} +{% if peer.endpoint is defined %} +Endpoint = {{ peer.endpoint }} +{% endif %} +{% if peer.persistent_keepalive | default(0) | int > 0 %} +PersistentKeepalive = {{ peer.persistent_keepalive }} +{% endif %} +{% if not loop.last %} + +{% endif %} +{% endfor %} diff --git a/ansible/site.yml b/ansible/site.yml index 6c57bd2..f45e490 100644 --- a/ansible/site.yml +++ b/ansible/site.yml @@ -73,6 +73,13 @@ - packages_rocky - services_systemd +- name: Configure WireGuard overlay + hosts: wireguard_overlay + become: true + + roles: + - wireguard_overlay + - name: Configure Aegis Fedora IoT profile hosts: role_aegis become: true @@ -87,6 +94,13 @@ roles: - profile_atlas +- name: Configure Atlas phase-one backend services + hosts: role_backend_phase1 + become: true + + roles: + - profile_backend_phase1 + - name: Configure Rocky Linux server hosts: rocky_server become: true diff --git a/ansible/templates/server/docker-compose.yml.j2 b/ansible/templates/server/docker-compose.yml.j2 index 0ca56c1..1a47af9 100644 --- a/ansible/templates/server/docker-compose.yml.j2 +++ b/ansible/templates/server/docker-compose.yml.j2 @@ -38,22 +38,6 @@ services: # networks: # - web - navidromedb: - image: docker.io/library/postgres:13 - container_name: navidromedb - restart: unless-stopped - mem_limit: 2048m - environment: - POSTGRES_DB: "navidrome_db" - POSTGRES_USER: "navidrome" - POSTGRES_PASSWORD: "{{ vault_postgres_root_password }}" - volumes: - - "/opt/postgres/data:/var/lib/postgresql/data{{ ':' ~ selinux_volume_option if selinux_volume_option else '' }}" - ports: - - "127.0.0.1:5432:5432" - networks: - - web - gitea: image: docker.gitea.com/gitea:1.25.2 container_name: gitea diff --git a/scripts/migrate_prometheus_data.sh b/scripts/migrate_prometheus_data.sh index 7339829..11dca6f 100644 --- a/scripts/migrate_prometheus_data.sh +++ b/scripts/migrate_prometheus_data.sh @@ -1,6 +1,6 @@ #!/usr/bin/env sh -# Copy the persistent Docker data from the retired Ubuntu server to the Rocky +# Copy the persistent NPM and Gitea data from the retired Ubuntu server to the Rocky # replacement. Run this script on the Ubuntu source as root. It is a dry run # unless --execute and --quiesce-source are both supplied. Extended attributes # are deliberately not copied: Rocky must assign its own SELinux labels. @@ -14,11 +14,8 @@ EXECUTE=false QUIESCE_SOURCE=false DATA_PATHS=' -/opt/navidrome/data -/opt/music /opt/npm/data /opt/npm/letsencrypt -/opt/postgres/data /opt/gitea/data ' @@ -26,8 +23,8 @@ usage() { cat <<'EOF' Usage: sudo ./scripts/migrate_prometheus_data.sh --destination USER@HOST [options] -Copies persistent Navidrome, Nginx Proxy Manager, PostgreSQL and Gitea data to -the Rocky server with rsync. The destination Docker containers must be stopped. +Copies persistent Nginx Proxy Manager and Gitea data to the Rocky server with +rsync. The destination Docker containers must be stopped. Options: --destination USER@HOST Rocky SSH destination (required). @@ -97,7 +94,7 @@ if [ -n "$IDENTITY_FILE" ]; then fi if [ "$EXECUTE" = true ] && [ "$QUIESCE_SOURCE" != true ]; then - fail '--execute requires --quiesce-source to keep PostgreSQL data consistent' + fail '--execute requires --quiesce-source to keep application data consistent' fi require_command rsync diff --git a/secrets/vault.yml.example b/secrets/vault.yml.example index e91c5a9..e3930e4 100644 --- a/secrets/vault.yml.example +++ b/secrets/vault.yml.example @@ -9,8 +9,6 @@ vault_icloud_mail_password: "REPLACE_ME" vault_git_work_email: "REPLACE_ME" vault_git_work_gpg: "REPLACE_ME" vault_openai_api_key: "REPLACE_ME" -vault_navidrome_db_password: "REPLACE_ME" -vault_postgres_root_password: "REPLACE_ME" vault_ikaros_authorized_ssh_keys: - "ssh-ed25519 REPLACE_ME" vault_atlas_authorized_ssh_keys: @@ -18,5 +16,3 @@ vault_atlas_authorized_ssh_keys: vault_atlas_admin_password_hash: "REPLACE_WITH_A_SHADOW_COMPATIBLE_HASH" vault_atlas_samba_password: "REPLACE_ME" vault_atlas_immich_db_password: "REPLACE_ME" -vault_prometheus_atlas_sftp_private_key: | - REPLACE_WITH_A_DEDICATED_ATLAS_SFTP_PRIVATE_KEY