Compare commits

18 Commits

Author SHA1 Message Date
Fabio Scotto di Santolo
fa1c8c0b82 Document Atlas NAS backups and monitoring 2026-09-25 14:06:28 +02:00
Fabio Scotto di Santolo
0b6efc9ad8 Harden Atlas USB backup cleanup and document restorecon workflow 2026-09-25 08:02:44 +02:00
Fabio Scotto di Santolo
361ee77d72 Add manual Atlas USB backup and reminder services 2026-09-24 08:55:52 +02:00
Fabio Scotto di Santolo
d4e40d423a Document deferred Cerberus management node plan 2026-09-23 09:01:18 +02:00
Fabio Scotto di Santolo
0a5c2ac1a4 Add progress logging to Atlas Borg backups 2026-09-22 21:21:21 +02:00
Fabio Scotto di Santolo
48a7f57f7e Add Cockpit storage management to Atlas and Prometheus 2026-09-22 20:49:18 +02:00
Fabio Scotto di Santolo
defa98c968 Install Cockpit Podman plugin on Atlas 2026-09-22 15:00:10 +02:00
Fabio Scotto di Santolo
21e41f4fc1 Prepare Atlas encryption dependencies and update Vault placeholders 2026-09-22 14:37:25 +02:00
Fabio Scotto di Santolo
e10c6694f8 Document populated Atlas archive and pending Borg evaluation 2026-09-21 22:15:42 +02:00
Fabio Scotto di Santolo
4c10af3187 Add openssl to Fedora base packages 2026-09-21 12:07:50 +02:00
Fabio Scotto di Santolo
3ac732751c Add pinned Java 25 to Mise on WSL workstation 2026-09-21 11:57:51 +02:00
Fabio Scotto di Santolo
e837b0059b Document validated Atlas Borg backup activation 2026-09-18 10:09:12 +02:00
Fabio Scotto di Santolo
7e498514dd Add encrypted Borg backups for Atlas 2026-09-18 10:00:45 +02:00
Fabio Scotto di Santolo
e7836ea25f Manage Atlas ZFS snapshots and scrubs 2026-09-17 23:10:53 +02:00
Fabio Scotto di Santolo
de2c24d15c Mount the shared Archive in Syncthing 2026-09-17 19:41:23 +02:00
Fabio Scotto di Santolo
bd13cb65cc Move temporary Navidrome and Syncthing services to Atlas 2026-09-17 10:18:23 +02:00
Fabio Scotto di Santolo
77afdda0a3 Move WireGuard gateway to Aegis 2026-09-17 09:08:25 +02:00
Fabio Scotto di Santolo
add75d74e9 refactor(atlas): separate service and host backup datasets 2026-09-17 00:06:46 +02:00
63 changed files with 3715 additions and 410 deletions

203
AGENTS.md
View File

@@ -59,10 +59,20 @@ Ansible-driven personal infrastructure repo for Fedora and Void desktops, Fedora
`ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff` `ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff`
- Atlas network/share hardening: - Atlas network/share hardening:
`ansible-playbook ansible/site.yml --limit atlas --tags hardening,sharing --check --diff` `ansible-playbook ansible/site.yml --limit atlas --tags hardening,sharing --check --diff`
- Atlas phase-one rootless services: - Atlas ZFS snapshot retention and scrub timers:
`ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 --check --diff` `ansible-playbook ansible/site.yml --limit atlas --tags snapshots,scrub --check --diff`
- Prometheus/Atlas WireGuard overlay: - Atlas encrypted Borg backup:
`ansible-playbook ansible/site.yml --limit prometheus,atlas --tags wireguard --check --diff` `ansible-playbook ansible/site.yml --limit atlas --tags packages,borg --check --diff`
- Atlas Borg progress logging only:
`ansible-playbook ansible/site.yml --limit atlas --tags borg_logging --check --diff`
- Atlas manual offline USB backup and 45Drives Alerts reminder:
`ansible-playbook ansible/site.yml --limit atlas --tags usb_backup,usb_reminder --check --diff`
- Atlas pool, disk, capacity, temperature, and job monitoring:
`ansible-playbook ansible/site.yml --limit atlas --tags monitoring --check --diff`
- Atlas explicit post-restore SELinux relabeling:
`ansible-playbook ansible/site.yml --limit atlas --tags restorecon --check -e '{"atlas_restorecon_paths":["/zpool/archive"]}'`
- Prometheus/Aegis WireGuard gateway:
`ansible-playbook ansible/site.yml --limit prometheus,aegis --tags wireguard --check --diff`
- DuckDNS config only: `ansible-playbook ansible/site.yml --limit prometheus --tags duckdns --check --diff` - DuckDNS config only: `ansible-playbook ansible/site.yml --limit prometheus --tags duckdns --check --diff`
## Conventions ## Conventions
@@ -139,10 +149,10 @@ The dotfile vars follow the same split: `desktop_common_dotfiles` carries mode-i
- Atlas requires `vault_atlas_admin_password_hash` for Cockpit and, while sharing is enabled, - Atlas requires `vault_atlas_admin_password_hash` for Cockpit and, while sharing is enabled,
`vault_atlas_samba_password`. The future rootful media stack also requires `vault_atlas_samba_password`. The future rootful media stack also requires
`vault_atlas_immich_db_password`. Never print these values. `vault_atlas_immich_db_password`. Never print these values.
- Atlas creates the complete declared hierarchy only under the verified existing or explicitly bootstrapped pool: `work`, `archive`, - Atlas creates the complete declared hierarchy only under the verified existing or explicitly bootstrapped pool: `archive`,
`archive/app_data`, `archive/app_data/navidrome`, `archive/app_data/syncthing`, `media`, `media/music`, `services`, `services/data`, `services/data/navidrome`, `services/data/syncthing`, `media`, `media/music`,
`media/photobook`, `backups`, `backups/services`, and `backup_prometheus`. `backups/services` has a `500G` `media/photobook`, `backup`, `backup/hosts`, and `backup/hosts/prometheus`. `backup` has a `500G`
refreservation. There is no separate legacy `zpool/syncthing` dataset. reservation covering its descendants. `archive` is the SMB-shared raw-data namespace; container state is never beneath it.
- The `immich` system account is fixed to UID/GID `1100`, has no login shell or `wheel` membership, and receives only - The `immich` system account is fixed to UID/GID `1100`, has no login shell or `wheel` membership, and receives only
the `video` and `render` supplementary groups. Immich's rootful Quadlets run as `1100:1100`; Server and ML receive the `video` and `render` supplementary groups. Immich's rootful Quadlets run as `1100:1100`; Server and ML receive
`/dev/dri`, while the Photobook external library is read-only at `/external/photobook`. `/dev/dri`, while the Photobook external library is read-only at `/external/photobook`.
@@ -151,58 +161,136 @@ The dotfile vars follow the same split: `desktop_common_dotfiles` carries mode-i
`1100`. Targeted SELinux is enforced persistently; a required reboot is reported but never initiated automatically. The primary LAN interface is assigned explicitly to the managed firewalld zone, and firewall rules are applied before NFS or SMB are started; their service state and TCP listeners are then verified. SMB3 exposes `Archive` to Vault-backed authorized accounts on mandatory encrypted, signed SMB3 over TCP/445 only and admits the configured LAN without host-specific exclusions. `1100`. Targeted SELinux is enforced persistently; a required reboot is reported but never initiated automatically. The primary LAN interface is assigned explicitly to the managed firewalld zone, and firewall rules are applied before NFS or SMB are started; their service state and TCP listeners are then verified. SMB3 exposes `Archive` to Vault-backed authorized accounts on mandatory encrypted, signed SMB3 over TCP/445 only and admits the configured LAN without host-specific exclusions.
- Atlas NPM and Immich share a rootful Podman network. NPM publishes HTTP/HTTPS, but its administration port remains - Atlas NPM and Immich share a rootful Podman network. NPM publishes HTTP/HTTPS, but its administration port remains
bound to `127.0.0.1:81`; do not expose it directly to the LAN or Internet. bound to `127.0.0.1:81`; do not expose it directly to the LAN or Internet.
- `profile_backend_phase1` is limited to rootless Navidrome and Syncthing user Quadlets on Atlas. Official Navidrome - `profile_backend_phase1` temporarily runs rootless Navidrome and Syncthing on Atlas until Uranus replaces
`0.63.2` uses SQLite below `/data` and does not support `ND_DATABASE_URL` or an external PostgreSQL backend; do not them. It binds only to Atlas' LAN IP, never `wg0`; Navidrome and the Syncthing GUI admit only Aegis as
recreate the obsolete Prometheus `navidromedb` service. The role requires the storage role's `zpool/media/music`, the source-NAT gateway, while native Syncthing ports admit the configured LAN. It initializes fresh
`zpool/archive/app_data`, `zpool/archive/app_data/navidrome`, and `zpool/archive/app_data/syncthing` datasets at state only and never migrates or deletes source application data.
their exact paths. It never creates the pool. - `wireguard_overlay` manages `wg0` between Prometheus (`10.0.0.1`) and Aegis (`10.0.0.2`). It persists private
- Keep `backend_phase1_start_services` false until the stopped Prometheus Navidrome data directory has been copied to keys only on their respective hosts, exchanges only derived public keys through Ansible, and verifies a real peer
Atlas and its SQLite database verified. The playbook renders the target but never migrates or deletes application handshake. Prometheus opens `51820/udp`; Aegis is the LAN gateway. Its persistent IPv4 forwarding, narrowly scoped
data; after cutover, set the flag true to enable and start Navidrome and Syncthing. WireGuard-to-LAN firewalld policy, and source masquerading permit Prometheus to reach LAN services without a static
- Phase 1 must not change Prometheus' existing NPM deployment. NPM continues to be managed exactly by `profile_server`; route on the router. Prometheus includes `192.168.178.0/24` in Aegis' peer `AllowedIPs`; add the Uranus VIP there
use `10.0.0.2:4533` for Navidrome and `10.0.0.2:8384` for the Syncthing GUI. Syncthing does not use host networking: when it is assigned. After a firewalld reload, restore Prometheus' rootful Podman networking with
its GUI, transfer, QUIC and discovery ports are explicitly published only on `10.0.0.2`; native transfer/discovery does not use the HTTP proxy. `podman network reload --all` so the existing proxy stack retains container DNS.
- `wireguard_overlay` manages the required `wg0` path between Prometheus and Atlas, persists private keys only on their
respective hosts, exchanges only derived public keys, and verifies a real peer handshake. The initial run must
include both hosts. Prometheus
opens `51820/udp`; after creating the WireGuard firewalld zone, restore Prometheus' rootful Podman networking with
`podman network reload --all` so the existing proxy stack retains container DNS. The Atlas backend role admits
service ports only in the WireGuard firewalld zone.
## Atlas NAS TODO ## Atlas NAS TODO
Completed validation: the existing RAIDZ2 pool and datasets, SELinux, LAN firewall, SSH, Cockpit with Completed validation: the existing RAIDZ2 pool and datasets, SELinux, LAN firewall, SSH, Cockpit with
the selected 45Drives plugins, encrypted SMB3 `Archive`, the Aegis-only NFSv4 `photobook` export, and the selected 45Drives plugins, encrypted SMB3 `Archive`, the Aegis-only NFSv4 `photobook` export, and the
the Prometheus--Atlas WireGuard path are operational. Aegis has validated NFSv4.2 read, write, delete, Prometheus--Aegis WireGuard gateway are operational. The gateway handshake, forwarding, source masquerading,
and `all_squash` mapping to UID/GID `1100` end-to-end. and TCP reachability to Atlas were verified. Temporary Navidrome and Syncthing are available through their
- Complete the Phase 1 Navidrome cutover: stop the Prometheus writer, copy and verify its complete manual NPM Proxy Hosts; Syncthing uses `/data/Org` backed by the SMB-shared Archive dataset. Aegis has also
`/opt/navidrome/data/` directory (including SQLite sidecars) under validated NFSv4.2 read, write, delete, and `all_squash` mapping to UID/GID `1100` end-to-end. The ZFS
`/zpool/archive/app_data/navidrome/`, then set `backend_phase1_start_services: true` and validate snapshot timers are active and the first recursive hourly snapshot completed successfully; the first
Navidrome on Atlas through WireGuard. Do not delete the source until a restore test succeeds. scheduled retention prune and monthly scrub remain runtime checks.
- Start and validate the rendered Syncthing Quadlet only after its device IDs, star topology, folders,
folder modes, ignore rules, and GUI/API protection are declared. Validate its GUI and native transfer ### Priority 1 - Data protection
ports through WireGuard only. - [x] Deploy Ansible-managed recursive ZFS snapshots with 24 hourly, 30 daily, 8 weekly, and 12 monthly
- Decide whether a common SMB/NFS namespace is required. `Archive` (SMB) and `photobook` (NFS) are generations, plus a monthly scrub on the first Sunday at 03:00. The timers and first hourly snapshot were
intentionally distinct today; only if a shared namespace is selected, finalize its UID/GID, group, verified on Atlas. Still observe the first scheduled retention prune and scrub; Cockpit Scheduler is for
and POSIX ACL model and test the same files through both protocols. visibility or manual operations only, and snapshot rollback is never automated.
- Keep `atlas_manage_media_stack` disabled until the future Immich deployment has validated `/dev/dri`, - [x] Activate and validate the encrypted offsite Borg backup to the Hetzner Storage Box. Atlas uses the
container paths, and the required Vault database secret. dedicated SSH identity, pinned ED25519 host key, Vault-backed `repokey` encryption, and a locked
- Add Ansible-managed ZFS snapshot retention and scrub timers. Use Cockpit Scheduler for visibility non-login `borg` account with no sudo or supplementary groups. The initial snapshot-consistent backup,
or manual operations, not as the only source of configuration, and never automate snapshot rollback. Borg repository check, and temporary-directory restore completed successfully; the restored `Archive`
- Add the Atlas-initiated least-privilege Prometheus backup pull: Prometheus exposes only prepared tree matched the live data, and temporary snapshots and mounts were removed. The exported recovery key
was copied offline. Daily backup retries and logging, 30 daily, 8 weekly and 12 monthly archives,
compaction, and monthly repository checks are enabled. Future runs report a ZFS-based estimated
percentage, and a post-exit helper handles host-namespace temporary snapshot cleanup. The active
run predates the new progress logging and still requires an observed final cleanup result.
- [x] Populate `/zpool/archive` with the currently available data so offsite and offline backup tests run
against a representative load.
- [ ] Run and evaluate Borg against the populated pool: duration, repository capacity, deduplication, and
a subsequent incremental archive must be observed before considering the offsite path fully validated.
- [x] Add the UUID-bound offline USB backup with versioned rsync, locking, capacity checks, verification,
safe unmounting and a tested restore procedure; never trigger it for an arbitrary USB disk. The
LUKS/ext4 identities were read-only verified; the manual service and 45Drives Alerts reminder timer were
deployed on Atlas. Interactive LUKS unlock is part of the manual service; only the reminder is
scheduled for the first Saturday of each month at 10:00 Europe/Rome via the existing 45Drives
notifier. A manual test produced an Alerts notification, not an email. The first USB attempt failed
on a `security.selinux` xattr and was interrupted; the xattr filter is deployed and the temporary
recursive snapshot and open LUKS mapper were cleaned up. A later run reported checksum verification
and published the USB version, but failed while removing host-namespace ZFS snapshot mounts. Those
exact mounts and snapshots were cleaned up. An `ExecStopPost` helper now removes only the named
temporary snapshot after the backup process exits. A new full run checksum-verified and published a
USB version; the service ended successfully, the mapper closed, no temporary USB snapshot remained,
and the pool was healthy. On 2026-09-25 an independent, read-only USB restore test copied one file from
the published `atlas/latest` version into `/var/tmp` and matched contents, owner, mode, size, mtime and
POSIX ACL. The temporary copy and mount were removed, the mapper closed, and the pool remained healthy.
- [x] Test restores independently from a ZFS snapshot, Borg, and the offline USB backup before relying on
any backup path. The earlier Borg temporary-directory restore passed. On 2026-09-25 a separate,
read-only ZFS snapshot test restored one file to `/var/tmp`, confirmed matching contents, ownership,
mode, mtime and ACL, then removed its temporary copy and on-demand mount. This is a file-level smoke
test, not full dataset recovery. An independent USB file restore passed on 2026-09-25 with matching
content and metadata; full disaster recovery remains a separate Priority 2 task.
- [x] Add monitoring and alerting for pool health, scrub/resilver, SMART data, temperatures, free space,
snapshot/local-backup growth, Hetzner Storage Box quota, and failed maintenance or backup timers.
The half-hourly Atlas health monitor and systemd final-failure hooks are deployed. A live probe
found no issues; the service and timer succeeded, and a labelled 45Drives Alerts test notification
was submitted. Alerts are deduplicated; email delivery is not claimed. The Storage Box quota probe
runs `df -m` over the dedicated pinned-key SSH identity and does not open the Borg repository.
Detailed archive size and deduplication remain part of the pending Borg evaluation.
### Priority 2 - NAS operability and recovery
- [ ] Document and test disaster recovery: rebuild Atlas with Ansible, import the existing pool, restore
from snapshot/USB/Hetzner, preserve Vault and Borg recovery material offline, and define RPO/RTO.
- [ ] Define a controlled Rocky kernel/OpenZFS update and reboot procedure.
- [ ] Add the Atlas-initiated least-privilege Prometheus backup pull: Prometheus exposes only prepared
read-only dumps through a dedicated account and Atlas retains the private SSH key, pinned host key, read-only dumps through a dedicated account and Atlas retains the private SSH key, pinned host key,
atomic pull, verification, retention and systemd service/timer. atomic pull, verification, retention and systemd service/timer.
- Add the encrypted offsite backup with Borg to a Hetzner Storage Box: use a dedicated SSH identity, - [ ] Decide whether a common SMB/NFS namespace is required. `Archive` (SMB) and `photobook` (NFS) are
pin the host key, keep Borg repository credentials and encryption material in Vault, use intentionally distinct today; only if a shared namespace is selected, finalize its UID/GID, group,
snapshot-consistent sources, and manage retries, logging, pruning, repository checks and restores. and POSIX ACL model and test the same files through both protocols.
- Add the UUID-bound offline USB backup with versioned rsync, locking, capacity checks, verification,
safe unmounting and a tested restore procedure; never trigger it for an arbitrary USB disk. ### Priority 3 - Service expansion
- Add monitoring and alerting for pool health, scrub/resilver, SMART data, temperatures, free space and - [ ] After data protection and recovery are validated, populate `/zpool/media/music` and validate Navidrome.
failed backup timers, plus a controlled Rocky kernel/OpenZFS update and reboot procedure. - [ ] Design and deploy Nextcloud as another explicitly temporary Atlas service before Uranus. Give it
- Document and test disaster recovery: rebuild Atlas with Ansible, import the existing pool, restore separate persistent application, database, and cache storage; keep credentials in Vault; publish it only
from snapshot/USB/Hetzner, preserve Vault and Borg recovery material offline, and define RPO/RTO. through NPM over the Prometheus--Aegis gateway; and define backup, upgrade, and eventual Uranus-migration
- Optionally design iCloud photo ingestion and an Aegis persistent NFS mount as a separate workflow procedures before exposing user data. Do not deploy Nextcloud before the data-protection checklist is complete.
after the storage and backup layers are validated; do not make either a dependency of the Atlas - [ ] Keep `atlas_manage_media_stack` disabled until the future Immich deployment has validated `/dev/dri`,
baseline. container paths, and the required Vault database secret.
### Priority 4 - Optional workflows
- [ ] After data protection is validated, move iCloudPD photo ingestion from Aegis to Atlas as a
temporary service until Uranus is ready. Plan to store photos in `/zpool/archive/Pictures` and
persistent application/MFA state outside `Archive`; validate permissions, SELinux, backups and
recovery before cutover. Keep the current Aegis service and Photobook NFS export unchanged until
the Atlas workflow is tested, then retire them explicitly if no longer needed.
## Cerberus Management Node (Deferred)
`cerberus` is postponed until the office in the new house is physically set up. It is not an inventory
host and this section is a design and implementation backlog, not authorization to provision it early.
The planned node is a Lenovo ThinkCentre M700 Tiny with an Intel Core i3-6100T, 8 GB RAM, a 256 GB SSD,
and native 1 Gbps Ethernet. It will connect to a multi-input KVM switch using a passive DisplayPort-to-HDMI
cable, sharing the monitor and peripherals with Ikaros. Fedora Sericea (immutable Fedora with the Sway
Wayland compositor) is the intended OS. Cerberus is an isolated management plane: a dedicated Toolbox
environment will run Ansible for future `uranus` cluster provisioning. Rootless Podman will host Grafana,
Prometheus, and Loki. The 256 GB local SSD is the hot tier retaining metrics and logs for 30 days; scheduled,
validated exports of older historical data will use a dedicated Atlas NFS dataset as cold storage.
### Implementation plan
- [ ] Confirm the office, KVM switch, passive DisplayPort-to-HDMI path, shared monitor/peripherals, and native
1 Gbps Ethernet are physically operational before adding Cerberus to inventory.
- [ ] Install and update Fedora Sericea with Sway; document the immutable-host lifecycle and keep host changes
declarative rather than treating the base OS as a mutable workstation.
- [ ] Model Cerberus as its own host with independent platform, role, desktop, network, and storage inputs;
do not repurpose Ikaros variables or make it a Uranus cluster member.
- [ ] Provision an isolated Toolbox-based Ansible controller with the required collections and a reproducible
project checkout; define its least-privilege SSH access, known-host handling, and Vault workflow without
storing secrets in the image or repository.
- [ ] Define the explicit Uranus provisioning workflow from Cerberus, including inventory boundaries,
validation-only runs, and separate approval for any destructive cluster operation.
- [ ] Design rootless Podman/Quadlet services for Grafana, Prometheus, and Loki, including persistent local
state, service ownership, LAN exposure/authentication, resource limits, updates, and backups.
- [ ] Size and enforce a 30-day local hot-retention policy for metrics and logs on the 256 GB SSD; validate
actual disk growth and alert before capacity exhaustion.
- [ ] Create and validate a dedicated Atlas NFS cold-storage dataset and least-privilege export for Cerberus;
do not use a broad existing share or couple it to unrelated Atlas application state.
- [ ] Implement scheduled, idempotent exports of data older than 30 days to the Atlas NFS cold tier, with
locking, capacity checks, integrity verification, retention rules, failure monitoring, and a tested restore.
- [ ] Validate management-plane recovery: rebuild Cerberus, restore observability history from Atlas, and
confirm that Uranus provisioning can resume without depending on unreproducible local state.
## Coding Agent Notes ## Coding Agent Notes
- Shared agent definitions and lifecycle flags live in `ai_agents` in `ansible/inventory/group_vars/all.yml`. - Shared agent definitions and lifecycle flags live in `ai_agents` in `ansible/inventory/group_vars/all.yml`.
@@ -229,8 +317,9 @@ and `all_squash` mapping to UID/GID `1100` end-to-end.
- `aegis` is a remote Fedora IoT Raspberry Pi 4 node. Bootstrap it once with - `aegis` is a remote Fedora IoT Raspberry Pi 4 node. Bootstrap it once with
`ansible/bootstrap/aegis.bu`; the remaining configuration is applied by `profile_aegis` over SSH. `ansible/bootstrap/aegis.bu`; the remaining configuration is applied by `profile_aegis` over SSH.
- Fedora IoT is immutable. Do not add it to mutable Fedora package or shared dotfile roles. - Fedora IoT is immutable. Do not add it to mutable Fedora package or shared dotfile roles.
- `profile_aegis` owns the `nfs-utils` rpm-ostree layer used as the Atlas NFS client and reports the - `profile_aegis` owns the `nfs-utils` and `wireguard-tools` rpm-ostree layers and reports the required reboot
required reboot without initiating it. It also owns rootful Podman Quadlets, persistent container without initiating it. `wireguard_overlay` then configures Aegis as the WireGuard LAN gateway with persistent IPv4
forwarding, a scoped inter-zone policy, and source masquerading. It also owns rootful Podman Quadlets, persistent container
state under `/var/lib`, the Podman auto-update timer, LAN-restricted firewalld rules, and SSH hardening. Keep state under `/var/lib`, the Podman auto-update timer, LAN-restricted firewalld rules, and SSH hardening. Keep
`aegis_lan_subnet`, `aegis_adguard_web_port`, and `aegis_network_connection_uuid` host-specific; `aegis_lan_subnet`, `aegis_adguard_web_port`, and `aegis_network_connection_uuid` host-specific;
SSH permits only the declared SSH permits only the declared

View File

@@ -95,6 +95,27 @@ Nota sullo stato attuale del playbook principale:
- `ansible/site.yml` applica il profilo server Rocky a `prometheus` con DNF, systemd, dotfiles server e firewalld - `ansible/site.yml` applica il profilo server Rocky a `prometheus` con DNF, systemd, dotfiles server e firewalld
- `ansible/site.yml` applica il profilo NAS Rocky su `atlas` tramite SSH remoto - `ansible/site.yml` applica il profilo NAS Rocky su `atlas` tramite SSH remoto
## Nodo pianificato e posticipato: Cerberus
`cerberus` e un nodo di management **posticipato**, in attesa dell'allestimento
fisico dell'ufficio nella nuova casa. Non e ancora presente nell'inventory e non
esistono ruoli o playbook che lo prendano come target.
L'hardware previsto e un Lenovo ThinkCentre M700 Tiny (Intel Core i3-6100T,
8 GB di RAM e SSD da 256 GB) con Ethernet nativa a 1 Gbps. Condividera monitor
e periferiche di Ikaros tramite uno switch KVM a ingressi multipli, usando un
cavo passivo DisplayPort-HDMI per il collegamento video. Il sistema operativo
previsto e Fedora Sericea, la variante Fedora immutabile con compositor Wayland
Sway.
Cerberus sara un management plane isolato: Ansible verra eseguito in un ambiente
Toolbox dedicato per il provisioning del futuro cluster `uranus`, anziche da
Ikaros o da un host non gestito. Lo stack di osservabilita rootless Podman
eseguira Grafana, Prometheus e Loki. L'SSD locale sara l'hot storage, con
metriche e log conservati per 30 giorni; esportazioni programmate trasferiranno
i dati storici piu vecchi su un dataset Atlas montato via NFS come cold storage.
Il piano di implementazione, con prerequisiti espliciti, e in `AGENTS.md`.
## Desktop ## Desktop
Target operativi: Target operativi:
@@ -246,90 +267,252 @@ ansible-playbook ansible/site.yml --limit prometheus -e server_username=myuser -
## NAS ## NAS
`atlas` e un NAS Rocky Linux 9 raggiunto tramite SSH. Normalmente il pool ZFS esiste gia e il profilo `atlas` è un NAS Rocky Linux 9 raggiunto via SSH. Normalmente il pool esiste già e il profilo gestisce
gestisce solo i dataset figli. Un bootstrap RAIDZ2 una tantum e disponibile solo con conferma esplicita solo i dataset figli. La creazione iniziale del RAIDZ2 richiede esplicitamente `atlas_create_pool=true`
(`atlas_create_pool=true`) e quattro percorsi reali e verificati `/dev/disk/by-id/...` in e quattro percorsi `/dev/disk/by-id/...` verificati in `atlas_zpool_disks`. Il ruolo non partiziona,
`atlas_zpool_disks`. Non partiziona, forza, distrugge, esegue rollback o modifica il layout vdev di un forza, distrugge, ripristina né modifica il layout vdev di un pool esistente. I client Linux usano NFSv4,
pool esistente. I client Linux usano NFSv4, quelli Windows/WSL SMB; entrambi restano limitati alla LAN quelli Windows/WSL SMB; l'accesso è limitato alla LAN configurata.
configurata.
Per il primo avvio fornire `vault_atlas_authorized_ssh_keys`, `vault_atlas_admin_password_hash`, Per il primo avvio servono `vault_atlas_admin_password_hash`, `vault_atlas_samba_password` e
`vault_atlas_samba_password` e `vault_atlas_immich_db_password`. Eseguire il bootstrap tramite `vault_atlas_immich_db_password`; il primo è un hash compatibile con `/etc/shadow`, non una password
l'amministratore esistente: Cockpit in chiaro. Il bootstrap usa l'amministratore preesistente:
```bash ```bash
ansible-playbook ansible/site.yml --limit atlas \ ansible-playbook ansible/site.yml --limit atlas \
-e atlas_connection_username=<existing-admin> -e atlas_connection_username=<existing-admin>
``` ```
`vault_atlas_admin_password_hash` deve essere un hash compatibile con `/etc/shadow`, non una Le esecuzioni successive usano `atlas_admin_username`. Storage, condivisioni e firewall LAN sono
password Cockpit in chiaro. Le esecuzioni successive usano `atlas_admin_username`. Atlas dichiara abilitati; prima dell'applicazione verificare pool, mountpoint, subnet e zona firewalld. La creazione
abilitati storage, condivisioni e regole firewall LAN. Prima della prima applicazione verificare pool e del pool è protetta da un gate esplicito e avviene solo se è assente. Atlas non fa più parte della VPN
mountpoint esistenti, subnet LAN e zona firewalld attiva. `atlas_manage_media_stack` resta disabilitato WireGuard: la vecchia interfaccia è stata ritirata manualmente dopo la verifica del collegamento tra
finche non saranno validati `/dev/dri`, i percorsi dei container e il segreto del database Immich. Prometheus e Aegis. Le chiavi SSH autorizzate sono in file separati sotto
`~/.ssh/authorized_keys.d/`. `atlas_manage_media_stack` resta disabilitato finché `/dev/dri`, percorsi
dei container e segreto del database Immich non sono validati.
Con la gestione storage attiva, Atlas crea l'intera gerarchia sotto il pool `zpool` esistente o creato esplicitamente: Sotto `zpool` Atlas crea `archive` (SMB), `services/data` con i dataset applicativi
`work`, `archive`, `archive/app_data`, i dataset applicativi separati `services/data/navidrome` e `services/data/syncthing`, `media`, `media/music`, `media/photobook` e
`archive/app_data/navidrome` e `archive/app_data/syncthing`, `media`, `media/music`, `backup/hosts/prometheus`. Archivio e applicazioni usano `zstd`; media, Syncthing e backup host usano
`media/photobook`, `backups`, `backups/services` e `backup_prometheus`. I dataset applicativi e `lz4`. `backup` ha una riserva di `500G` che copre i discendenti. SELinux targeted è persistente;
di archivio usano `zstd`; media, Syncthing e backup dei servizi usano `lz4`; l'eventuale riavvio necessario viene segnalato, non eseguito. Atlas assegna l'interfaccia primaria
`backups/services` mantiene inoltre una `refreservation` di `500G`. alla zona firewalld gestita, rifiuta redirect e source route, registra i martian, mantiene il reverse-path
Atlas impone SELinux targeted in modo persistente e segnala, senza avviarlo, l’eventuale reboot necessario per attivarlo. Assegna esplicitamente l’interfaccia LAN primaria alla zona firewalld gestita e applica hardening persistente del kernel di rete: rifiuta redirect e source-route, registra i martian, usa reverse-path filtering loose per WireGuard e disabilita il forwarding IPv4. SSH consente solo l’amministratore dichiarato tramite chiave pubblica; root, password, agent e forwarding filter loose e disabilita il forwarding IPv4. SSH consente soltanto l'amministratore dichiarato con
remoto sono disabilitati, mentre il forwarding locale resta disponibile per tunnel amministrativi privati. SMB3 pubblica `Archive` solo agli account Samba configurati con password in Vault e chiave pubblica: root, password, agent forwarding e remote forwarding sono disabilitati, mentre il
ammette la LAN configurata su SMB3 cifrato e firmato, esclusivamente su TCP/445. NFSv4 esporta soltanto forwarding locale resta disponibile per i tunnel amministrativi. SMB3 espone `Archive` agli account
`media/photobook` all'IP configurato di Aegis su TCP/2049, con `all_squash` verso UID/GID anonimi `1100`. autorizzati da Vault sulla LAN, solo su TCP/445 con cifratura e firma obbligatorie. NFSv4 espone
soltanto `media/photobook` all'IP di Aegis su TCP/2049, con `all_squash` verso UID/GID `1100`.
L'account di sistema `immich` usa UID/GID `1100`, shell senza login, nessuna appartenenza a `wheel` e L'account di sistema `immich` usa UID/GID `1100`, non ha shell di login né gruppo `wheel` e riceve i
i gruppi supplementari `video` e `render`. I Quadlet rootful di Immich Server, ML, cache compatibile gruppi `video` e `render`. Lo stack Immich futuro prevede Quadlet rootful per Server, ML, cache,
Redis, PostgreSQL e NPM condividono una rete Podman. Immich viene eseguito come `1100:1100`; Server e PostgreSQL e NPM su una rete Podman comune. Immich gira come `1100:1100`, Server e ML ricevono
ML ricevono `/dev/dri` e Photobook e montato in sola lettura su `/external/photobook`. NPM pubblica `80` e `/dev/dri` e Photobook è montato in sola lettura su `/external/photobook`. NPM pubblica `80` e `443`;
`443`, mentre l'amministrazione resta vincolata a `127.0.0.1:81` per l'accesso tramite tunnel SSH. l'interfaccia amministrativa resta su `127.0.0.1:81`, raggiungibile via tunnel SSH.
La fase 1 e limitata ai Quadlet utente rootless di Navidrome e Syncthing su Atlas. E abilitata nella Atlas ospita temporaneamente Navidrome e Syncthing rootless fino alla sostituzione con Uranus. I
configurazione host di Atlas e puo essere impostata a `false` solo per una sospensione intenzionale. Navidrome ufficiale `0.63.2` usa il database SQLite sotto `/data` e servizi sono inizializzati **ex novo**, senza migrare lo stato precedente, rispettivamente sotto
non supporta `ND_DATABASE_URL` ne un backend PostgreSQL esterno. Il servizio obsoleto `navidromedb` `/zpool/services/data/navidrome` e `/zpool/services/data/syncthing`; la musica in
e quindi rimosso da Prometheus invece di essere replicato su Atlas. Il ruolo deriva i percorsi dal `/zpool/media/music` viene popolata separatamente. Sono vincolati all'indirizzo LAN di Atlas
pool `zpool`, montato in `/zpool`: musica in sola lettura da `/zpool/media/music`, stato (`192.168.178.55`), mai a WireGuard. `wireguard_overlay` collega invece Prometheus (`10.0.0.1`)
applicativo Navidrome e `navidrome.db` in `/zpool/archive/app_data/navidrome` e dati Syncthing in e Aegis (`10.0.0.2`): le chiavi private restano sui rispettivi host e Ansible scambia solo le pubbliche.
`/zpool/archive/app_data/syncthing`. `profile_atlas` crea questi dataset quando Prometheus apre `51820/udp`; Aegis inoltra soltanto il traffico overlay→LAN dichiarato e applica
`atlas_manage_storage` e attivo; il ruolo backend verifica i mountpoint esatti prima di avviare i source NAT, evitando interfacce VPN su Atlas/Uranus e route statiche sul router. Navidrome (`4533/tcp`)
container. Il ruolo backend non crea mai il pool. Il ruolo separato `wireguard_overlay` e la GUI Syncthing (`8384/tcp`) ammettono solo Aegis, mentre le porte native Syncthing sono limitate
gestisce `wg0` tra Prometheus (`10.0.0.1`) e Atlas (`10.0.0.2`), genera una sola volta le chiavi alla LAN. Dopo la verifica dei servizi, configurare manualmente i Proxy Host NPM verso
private sui rispettivi host e scambia tramite Ansible soltanto quelle pubbliche. Solo Prometheus apre `http://192.168.178.55:4533` e `http://192.168.178.55:8384`. Il peer Prometheus include la LAN
pubblicamente `51820/udp`. Le porte backend sono ammesse esclusivamente nella zona firewalld WireGuard. negli `AllowedIPs`; aggiungere la VIP Uranus quando esisterà. Dopo il reload di firewalld, Ansible
ricarica le reti Podman rootful di Prometheus per conservare DNS e connettività del proxy.
`backend_phase1_start_services` resta falso durante il trasferimento dello stato applicativo, quindi Validare il gateway con:
la prima esecuzione reale del backend genera i Quadlet senza creare un database Atlas vuoto. Dopo aver
arrestato Navidrome su Prometheus, copiare l'intera directory `/opt/navidrome/data/` in
`/zpool/archive/app_data/navidrome/`, preservando `navidrome.db` e gli eventuali file SQLite laterali.
Impostare quindi questa variabile a vero e rieseguire il ruolo per abilitare e avviare Navidrome e
Syncthing. Il playbook non copia e non elimina mai i dati applicativi.
Validare e generare i servizi Atlas con:
```bash ```bash
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \ ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit atlas --tags storage ansible-playbook ansible/site.yml --limit prometheus,aegis --tags wireguard --check --diff
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit prometheus,atlas --tags wireguard
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 --check --diff
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1
``` ```
Per il cutover, arrestare il vecchio Navidrome prima di copiare la sua directory dati, verificare La prima esecuzione reale WireGuard deve includere entrambi i peer. Se Aegis ha appena installato il
l'ownership dell'account `admin` su Atlas e confermare la presenza del database SQLite copiato prima layer `wireguard-tools`, riavviarlo manualmente e rieseguire senza `--check`: il ruolo attende un
di impostare `backend_phase1_start_services: true` in `host_vars/atlas.yml`. Conservare i dati sorgente handshake effettivo.
e il container legacy `navidromedb` fermo finche Navidrome su Atlas e una prova di restore non sono
stati validati.
Restano da completare retention delle snapshot, topologia Syncthing, validazione WireGuard/firewall, Gli snapshot ZFS ricorsivi coprono l'intero pool: 24 orari al minuto 05, 30 giornalieri alle 00:15,
pull di backup da Prometheus, backup cifrati con Borg su una Hetzner Storage Box, backup USB, 8 settimanali la domenica alle 01:00 e 12 mensili il primo giorno alle 02:00. La retention elimina
monitoraggio e test di disaster recovery. Il backlog operativo dettagliato e in `AGENTS.md`. solo gli snapshot con prefisso gestito `atlas-auto` e non esegue rollback. Lo scrub OpenZFS mensile è
previsto la prima domenica alle 03:00; il timer settimanale incompatibile è disabilitato. Il primo
snapshot orario ricorsivo è riuscito; la prima pulizia pianificata e il primo scrub schedulato
richiedono ancora una verifica a runtime.
```bash
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit atlas --tags snapshots,scrub --check --diff
```
Il backup Borg cifrato usa il sub-account Hetzner `u660064-sub1`, il repository relativo `./borg-data`
e Borg remoto 1.4 su SSH porta 23. La chiave ED25519 del server è fissata; una chiave client dedicata
appartiene all'account `borg`, bloccato e senza login, sudo o gruppi supplementari. La chiave privata
resta in `/etc/atlas-borg`; la passphrase proviene da `vault_atlas_borg_passphrase` ed è resa in un
file `0600`. Solo il wrapper root crea snapshot e mount; avvia il client come `borg` con il minimo
accesso temporaneo in lettura, senza concedergli gestione ZFS o sudo.
Il backup giornaliero parte alle 04:30 con un ritardo casuale fino a 30 minuti. Crea uno snapshot ZFS
ricorsivo temporaneo e ricostruisce tutti i dataset sotto `/zpool` in un albero di bind mount in sola
lettura, per inserirli in un unico archivio coerente. Il wrapper smonta ricorsivamente l'albero privato;
un helper `ExecStopPost` mirato rimuove eventuali mount dello snapshot nel namespace host e lo snapshot
temporaneo dopo l'uscita del processo. Borg conserva 30 archivi giornalieri, 8 settimanali e 12
mensili, poi compatta il repository. Il controllo completo di metadati e repository si svolge il 15
di ogni mese alle 06:00. Le operazioni usano un lock comune, journal e retry systemd limitati. Le
nuove esecuzioni riportano al massimo una riga di avanzamento al minuto: percentuale **stimata**,
dataset, file elaborati e byte originali/compressi/deduplicati. Il denominatore è la somma dei
`logicalreferenced` ZFS dello snapshot, non un totale Borg: può superare il 100% e non comprende
retention, compattazione o controlli. Le righe di progresso non riportano i nomi dei file; eventuali
warning possono farlo. Seguire il job con `sudo journalctl -fu atlas-borg-backup.service`; modifiche
all'helper non cambiano un'esecuzione già avviata.
Attivazione iniziale esplicita:
1. Inserire una passphrase unica in `secrets/vault.yml` con `ansible-vault edit`.
2. Generare e mostrare solo la chiave pubblica con
`ansible-playbook ansible/site.yml --limit atlas --tags borg_key`.
3. Installarla nel sub-account Hetzner, poi applicare con
`ansible-playbook ansible/site.yml --limit atlas --tags packages,borg`.
4. Copiare `secrets/recovery/atlas-borg-repokey.export` su un supporto davvero offline: la copia
locale ignorata da Git non è di per sé un backup offline.
Il ruolo inizializza solo un repository `repokey` assente, non accetta password SSH né host key non
fissate e non avvia manualmente il primo backup. Validazione:
```bash
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit atlas --tags packages,borg --check --diff
```
L'attivazione iniziale è riuscita: backup e controllo del repository, restore completo in una
directory temporanea confrontato con l'albero `Archive`, esportazione offline della chiave di recupero
e pulizia di snapshot/mount temporanei. Il 2026-09-25 un test separato da snapshot ZFS giornaliero ha
copiato un file di `/zpool/archive` in `/var/tmp`, verificando contenuto, proprietario, modalità,
mtime e ACL POSIX; copia e mount temporanei sono stati rimossi senza interrompere Borg. Non è un test
di ripristino dell'intero dataset.
Il backup USB offline è distribuito come **servizio solo manuale** (`atlas_manage_usb_backup: true`):
Ansible non formatta, sblocca, monta né avvia automaticamente il disco. Il disco esistente è stato
verificato in sola lettura il 2026-09-23: UUID LUKS `577b3c43-ea37-4611-81a9-39d555cdfbd4`,
UUID ext4 interno `758e2d2e-a427-4797-aad9-39c3a9f17c7e`, mapper `zpool-backup`. All'ispezione
era montato in `/mnt/zpool-backup`; il servizio richiede invece che il mapper **non sia montato** prima
dell'avvio. Se serve, `systemd-ask-password` chiede interattivamente la passphrase LUKS tramite
l'agente di `systemctl start` e la passa direttamente a `cryptsetup`, senza salvarla, esporla negli
argomenti o memorizzarla nella cache. Lo script monta il disco privatamente, crea uno snapshot ZFS
ricorsivo, copia tutti i dataset in `atlas/snapshots/<timestamp>/` con `rsync --link-dest`, verifica
con un dry-run basato sui checksum, aggiorna atomicamente `atlas/latest`, smonta e chiude LUKS. Un
errore non sostituisce `latest` né cancella versioni complete precedenti. Borg e USB possono operare
contemporaneamente su snapshot distinti, ma la lettura concorrente può ridurre il throughput.
La copia USB conserva le ACL ma non gli attributi estesi generici, compreso `security.selinux`: la
policy della destinazione deve ricreare le etichette dopo un restore. Per un percorso esplicito:
```bash
ansible-playbook ansible/site.yml --limit atlas --tags restorecon \
-e '{"atlas_restorecon_paths":["/zpool/archive"]}'
```
Il task accetta solo percorsi sotto la radice del pool Atlas, esegue `restorecon -RFv` solo su quelli
indicati ed è altrimenti inattivo; non va lanciato sull'intero pool durante i run ordinari. Le vecchie
versioni USB non vengono eliminate automaticamente senza una retention deliberata. Il controllo di
capacità include il trasferimento stimato e una riserva libera di 10 GiB. Dopo un backup riuscito,
scollegare fisicamente il disco per renderlo davvero offline.
Validare la configurazione senza avviare il backup e, separatamente, un eventuale relabel pianificato:
```bash
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit atlas --tags usb_backup,usb_reminder --check --diff
ansible-playbook ansible/site.yml --limit atlas --tags restorecon --check \
-e '{"atlas_restorecon_paths":["/zpool/archive"]}'
```
Prima dell'avvio manuale smontare in sicurezza `/mnt/zpool-backup`, se ancora montato. Con il mapper
chiuso, `sudo systemctl start atlas-usb-backup.service` chiede la passphrase e avvia il backup; né la
password LUKS né un keyfile vanno in Ansible. Seguire con
`sudo journalctl -fu atlas-usb-backup.service`. **Non esiste un timer di backup USB.** Soltanto
`atlas-usb-reminder.timer` è schedulato il primo sabato del mese alle 10:00 `Europe/Rome`: invia un
promemoria al notifier 45Drives Houston, senza avviare il backup. Un test manuale ha prodotto una
notifica in 45Drives Alerts, **non un'email**; il log conferma l'invio della notifica, non la consegna
di posta. Il primo evento pianificato era il 2026-10-03 alle 10:00 CEST. Controllare timer e risultato
con `systemctl list-timers atlas-usb-reminder.timer` e in 45Drives Alerts.
Il primo tentativo USB del 2026-09-23 fallì su `security.selinux` e, dopo l'interruzione, lasciò
snapshot e mapper aperti. Applicato il filtro rsync, furono rimossi lo snapshot fallito, il mapper
smontato e lo stato failed; non rimase una copia valida di quel tentativo. Un run del 2026-09-24
pubblicò una versione verificata ma fallì nella distruzione dello snapshot a causa di mount
`.zfs/snapshot` aperti nel namespace host. Dopo la pulizia non forzata, è stato aggiunto un helper
`ExecStopPost` mirato e testato con uno snapshot usa-e-getta. Un run successivo del 2026-09-24 ha
verificato i checksum, pubblicato la versione ed è terminato con successo: mapper chiuso, nessuno
snapshot USB temporaneo e pool sano. Il 2026-09-25 un test di restore indipendente ha aperto il disco
in sola lettura, montato ext4 con `ro,noload`, copiato un file di 5.707.945 byte da `atlas/latest` in
una directory vuota sotto `/var/tmp` e confrontato contenuto, proprietario, modalità, dimensione,
mtime e ACL POSIX. Il test ha rimosso copia e mount temporanei, chiuso LUKS e lasciato il pool sano
mentre Borg continuava. È un test su file, non un esercizio completo di disaster recovery.
Il monitoraggio Atlas è eseguito ogni 30 minuti da `atlas-health-monitor.timer`. Sonde in sola
lettura controllano stato/errori del pool e dei vdev, scrub/resilver, SMART dei quattro dischi del
pool e dell'NVMe di sistema, temperature dei dischi e CPU, spazio di sistema/pool/snapshot, crescita
di `zpool/backup` e quota Hetzner tramite `df -m` via SSH con l'account `borg` e la chiave fissata.
La query remota non apre il repository Borg né il suo lock. Gli alert di crescita richiedono una
baseline di circa 24 ore. Sono controllati anche attivazione e freschezza dei timer; hook systemd
`OnFailure` segnalano errori di snapshot, scrub, Borg, USB, promemoria e monitoraggio. Il monitor non
riavvia Borg; avvisa solo se un run supera 14 giorni. Soglie e percorsi stabili dei dischi sono nelle
variabili host. Gli avvisi usano 45Drives Houston con deduplicazione; **la consegna email non è stata
verificata**. Il controllo live del 2026-09-25 non ha trovato problemi; la notifica di prova è stata
inviata e lo Storage Box risultava occupato al 22%. Dimensione dell'archivio Borg e deduplicazione
dettagliata richiedono ancora la fine del backup in corso.
```bash
ansible-playbook ansible/site.yml --limit atlas --tags monitoring --check --diff
sudo /usr/local/libexec/atlas-health-monitor --dry-run
sudo journalctl -u atlas-health-monitor.service -n 100 --no-pager
systemctl list-timers atlas-health-monitor.timer
```
`--dry-run` non invia alert e non modifica lo stato del monitor. Un controllo reale si avvia con
`sudo systemctl start atlas-health-monitor.service`, senza avviare servizi di backup. Per una prova
etichettata di 45Drives Alerts usare
`sudo /usr/local/libexec/atlas-health-monitor --test-notification`.
### Timer systemd di Atlas
Tutti i nove timer gestiti sono abilitati. Gli orari sono locali ad Atlas (`Europe/Rome`); Borg e
monitoraggio aggiungono il ritardo casuale indicato. Tutti hanno `Persistent=true`: un evento perso
viene recuperato quando il timer torna attivo.
| Timer | Pianificazione (`OnCalendar`) | Azione |
| --- | --- | --- |
| `atlas-zfs-snapshot-hourly.timer` | `*-*-* *:05:00` — ogni ora al minuto 05 | Snapshot ricorsivo orario e retention |
| `atlas-zfs-snapshot-daily.timer` | `*-*-* 00:15:00` — ogni giorno alle 00:15 | Snapshot ricorsivo giornaliero e retention |
| `atlas-zfs-snapshot-weekly.timer` | `Sun *-*-* 01:00:00` — domenica alle 01:00 | Snapshot ricorsivo settimanale e retention |
| `atlas-zfs-snapshot-monthly.timer` | `*-*-01 02:00:00` — primo giorno del mese alle 02:00 | Snapshot ricorsivo mensile e retention |
| `zfs-scrub-monthly@zpool.timer` | `Sun *-*-01..07 03:00:00` — prima domenica alle 03:00 | Scrub ZFS |
| `atlas-borg-backup.timer` | `*-*-* 04:30:00` — ogni giorno alle 04:30, più 0–30 min casuali | Backup cifrato offsite |
| `atlas-borg-check.timer` | `*-*-15 06:00:00` — giorno 15 alle 06:00, più 0–30 min casuali | Controllo repository Borg |
| `atlas-usb-reminder.timer` | `Sat *-*-01..07 10:00:00 Europe/Rome` — primo sabato alle 10:00 | Solo promemoria 45Drives Alerts |
| `atlas-health-monitor.timer` | `*:0/30` — ogni mezz'ora, più 0–5 min casuali | Controlli di salute in sola lettura |
`atlas-usb-backup.service` **non ha timer** e va avviato manualmente. Il timer del fornitore
`zfs-scrub-weekly@zpool.timer` è disabilitato a favore dello scrub mensile. Il futuro pull del backup
Prometheus non ha ancora un timer, perché non è implementato. Durante un backup Borg attivo,
`systemctl list-timers` può mostrare `-` per il prossimo evento senza che il timer sia disabilitato.
Per vedere la pianificazione corrente: `systemctl list-timers --all` su Atlas.
Nextcloud è previsto come servizio temporaneo su Atlas prima di Uranus, ma solo dopo la validazione
della protezione dei dati: richiede storage applicativo, database e cache separati, segreti Vault,
pubblicazione solo tramite NPM e Aegis, procedure di backup, aggiornamento e migrazione. Non
distribuirlo prima di completare la checklist di protezione dei dati.
La destinazione futura per l'importazione foto iCloud è Atlas, non Aegis. Dopo la validazione dei
backup, pianificare una migrazione esplicita di iCloudPD con foto sotto `/zpool/archive/Pictures` e
stato applicativo/MFA fuori da `Archive`; testare permessi, SELinux, backup e restore prima del
cutover. L'attuale iCloudPD su Aegis e l'export NFS Photobook restano configurati fino
all'approvazione e alla verifica di questa migrazione separata. Anche il servizio Atlas sarà
temporaneo in attesa di Uranus.
Il pull dei backup di Prometheus, la valutazione delle dimensioni degli archivi Borg e i test completi
di disaster recovery restano da fare. Il backlog prioritizzato è in `AGENTS.md`.
--- ---

345
README.md
View File

@@ -67,6 +67,27 @@ The official ChatGPT desktop RPM is enabled only on `ikaros` and `nymph`. The
playbook configures OpenAI's signed RPM repository and imports its pinned RPM playbook configures OpenAI's signed RPM repository and imports its pinned RPM
signing key before installation; subsequent updates are handled by DNF. signing key before installation; subsequent updates are handled by DNF.
## Deferred planned node: Cerberus
`cerberus` is a **postponed** management-plane node, pending the physical setup
of the office in the new house. It is not yet an inventory host and no role or
playbook targets it.
The planned hardware is a Lenovo ThinkCentre M700 Tiny (Intel Core i3-6100T,
8 GB RAM, and a 256 GB SSD) with native 1 Gbps Ethernet. It will share Ikaros'
monitor and peripherals through a multi-input KVM switch, using a passive
DisplayPort-to-HDMI cable for its video connection. Fedora Sericea, the
immutable Fedora variant with the Sway Wayland compositor, is the intended
operating system.
Cerberus will be an isolated management plane: Ansible will run from a
dedicated Toolbox environment to provision the future `uranus` cluster, rather
than from Ikaros or an unmanaged host. Its rootless Podman observability stack
will run Grafana, Prometheus, and Loki. The local SSD is the hot tier and
retains metrics and logs for 30 days; scheduled exports will place older
historical data on an NFS-mounted Atlas dataset as the cold tier. The detailed,
implementation-gated plan is maintained in `AGENTS.md`.
## Desktop profiles ## Desktop profiles
- `ikaros`: stable Fedora Workstation + GNOME desktop. - `ikaros`: stable Fedora Workstation + GNOME desktop.
@@ -111,21 +132,19 @@ the Compose stack, update DNS, or perform a cutover.
The server profile installs platform-specific packages, Podman and podman-compose, declared systemd The server profile installs platform-specific packages, Podman and podman-compose, declared systemd
services, and firewalld. The manually activated `podman-compose-server` unit contains the existing services, and firewalld. The manually activated `podman-compose-server` unit contains the existing
Nginx Proxy Manager and Gitea services. The desired Compose file no longer includes Navidrome, Nginx Proxy Manager and Gitea services. The desired Compose file no longer includes Navidrome,
Syncthing, or the obsolete Navidrome PostgreSQL database. Navidrome and Syncthing belong to Atlas; Syncthing, or the obsolete Navidrome PostgreSQL database; their temporary Atlas deployment is managed
official Navidrome uses SQLite instead. Applying the profile does not stop or remove legacy by `profile_backend_phase1`. Applying the profile does not stop or remove legacy containers and does
containers and does not delete `/opt/postgres/data`. not delete `/opt/postgres/data`.
Firewalld enables SSH, Cockpit (`9090/tcp`), HTTP and HTTPS. Nginx Proxy Manager publishes only Firewalld enables SSH, Cockpit (`9090/tcp`), HTTP and HTTPS. Nginx Proxy Manager publishes only
`80/tcp` and `443/tcp`; its administration interface is bound to `127.0.0.1:81` and can be reached `80/tcp` and `443/tcp`; its administration interface is bound to `127.0.0.1:81` and can be reached
from Ikaros or Nymph with the `npm-tunnel` Bash alias. Nextcloud remains disabled and the profile from Ikaros or Nymph with the `npm-tunnel` Bash alias. Nextcloud remains disabled and the profile
does not provision any `/srv/nextcloud` directories. does not provision any `/srv/nextcloud` directories.
The Atlas phase-one work does not change this NPM deployment or its persistent data. Once WireGuard NPM remains managed only by `profile_server`. Its WireGuard peer is Aegis (`10.0.0.2`), which forwards
and the Atlas services are active, configure the current NPM proxy hosts with Navidrome upstream selected requests to LAN addresses and source-NATs them so no static route is required on the router.
`http://10.0.0.2:4533` and Syncthing GUI upstream `http://10.0.0.2:8384`. Only the Syncthing web GUI Use an Atlas LAN address for any current NAS-backed upstream; when Uranus receives its VIP, add that VIP
uses NPM; synchronization traffic remains on explicitly published native ports bound only to the Atlas to Prometheus' Aegis peer `AllowedIPs` and declare the corresponding proxy target separately.
WireGuard address. Configure both Syncthing authentication and an appropriate NPM access policy before
publishing its GUI.
Server identity comes from `server_username`, `server_user_group`, and `server_user_home` in `ansible/inventory/group_vars/server.yml`. `server_username` defaults to `username`, but it can be overridden, for example: Server identity comes from `server_username`, `server_user_group`, and `server_user_home` in `ansible/inventory/group_vars/server.yml`. `server_username` defaults to `username`, but it can be overridden, for example:
@@ -193,8 +212,10 @@ ansible/bootstrap/generate-aegis-ign.sh --write IMAGE DEVICE
The controller manages it remotely as `pi@aegis`; unlike local desktop profiles, Aegis is The controller manages it remotely as `pi@aegis`; unlike local desktop profiles, Aegis is
intentionally an SSH inventory target. `profile_aegis` manages rootful Podman Quadlets for AdGuard intentionally an SSH inventory target. `profile_aegis` manages rootful Podman Quadlets for AdGuard
Home and iCloudPD, persistent data under `/var/lib`, the Podman auto-update timer, LAN-restricted Home and iCloudPD, persistent data under `/var/lib`, the Podman auto-update timer, LAN-restricted
firewalld rules, SSH key-only access for `pi`, the `nfs-utils` rpm-ostree layer required by the firewalld rules, SSH key-only access for `pi`, the `nfs-utils` and `wireguard-tools` rpm-ostree layers,
Atlas NFS client, and `wake-ikaros`. A new layered package deployment requires a manual reboot; the and `wake-ikaros`. `wireguard_overlay` makes Aegis the internal endpoint and LAN gateway for Prometheus:
it enables persistent IPv4 forwarding, installs a scoped WireGuard-to-LAN firewalld policy, and source-NATs
forwarded tunnel traffic so the router needs no static route. A new layered package deployment requires a manual reboot; the
role reports this condition but never reboots Aegis automatically. Set the host-local role reports this condition but never reboots Aegis automatically. Set the host-local
`aegis_lan_subnet`, `aegis_adguard_web_port`, and `aegis_network_connection_uuid` values before `aegis_lan_subnet`, `aegis_adguard_web_port`, and `aegis_network_connection_uuid` values before
applying it. The playbook permits applying it. The playbook permits
@@ -225,7 +246,7 @@ ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit aegis --tags dns --ask-become-pass ansible-playbook ansible/site.yml --limit aegis --tags dns --ask-become-pass
``` ```
Layer the Atlas NFS client package independently, then reboot Aegis manually when the role reports Layer the Aegis NFS and WireGuard client tools independently, then reboot Aegis manually when the role reports
that the new deployment is ready: that the new deployment is ready:
```bash ```bash
@@ -243,17 +264,9 @@ clients use NFSv4 and Windows/WSL clients use SMB; both are restricted to the co
For the first run, provide `vault_atlas_admin_password_hash`, `vault_atlas_samba_password`, and For the first run, provide `vault_atlas_admin_password_hash`, `vault_atlas_samba_password`, and
`vault_atlas_immich_db_password`. Bootstrap the host through its `vault_atlas_immich_db_password`. Bootstrap the host through its
existing administrator. Open `51820/udp` towards Prometheus in the provider firewall first, then existing administrator. The explicit pool gate is safe to repeat: the role creates the RAIDZ2 pool only when
include both WireGuard peers in the same idempotent playbook run: it is absent. Atlas no longer participates in the WireGuard overlay; its old interface is retired manually only after
Prometheus and Aegis have completed the replacement handshake.
```bash
ansible-playbook ansible/site.yml --limit prometheus,atlas \
-e atlas_connection_username=<existing-admin> \
-e atlas_create_pool=true
```
The explicit pool gate is safe to repeat: the role creates the RAIDZ2 pool only when it is absent.
WireGuard waits for a real peer handshake before the play continues.
`vault_atlas_admin_password_hash` must be an `/etc/shadow`-compatible hash, not a clear-text `vault_atlas_admin_password_hash` must be an `/etc/shadow`-compatible hash, not a clear-text
Cockpit password. Subsequent runs use `atlas_admin_username`. Atlas declares storage, sharing, and its Cockpit password. Subsequent runs use `atlas_admin_username`. Atlas declares storage, sharing, and its
@@ -263,11 +276,11 @@ paths, and the Immich database secret are validated. Atlas reads its declared SS
separate files below `~/.ssh/authorized_keys.d/`. separate files below `~/.ssh/authorized_keys.d/`.
With storage management enabled, Atlas creates the complete dataset hierarchy below the existing or With storage management enabled, Atlas creates the complete dataset hierarchy below the existing or
explicitly bootstrapped `zpool`: `work`, `archive`, `archive/app_data`, the separate `archive/app_data/navidrome` and explicitly bootstrapped `zpool`: SMB-shared `archive`, private `services/data` with separate
`archive/app_data/syncthing` application datasets, `media`, `media/music`, `media/photobook`, `services/data/navidrome` and `services/data/syncthing` application datasets, `media`, `media/music`,
`backups`, `backups/services`, and `backup_prometheus`. Application/archive datasets use `zstd`, `media/photobook`, and `backup/hosts/prometheus`. Application/archive datasets use `zstd`, while media,
while media, Syncthing and service-backup datasets use `lz4`; `backups/services` also has a `500G` Syncthing, and host-backup datasets use `lz4`; `backup` has a `500G` reservation covering its descendants.
refreservation. Atlas enforces targeted SELinux persistently and reports, without initiating, any reboot required to activate it. It assigns its primary LAN interface explicitly to the managed firewalld zone and applies persistent kernel network hardening: redirects and source routes are rejected, martians logged, reverse-path filtering remains loose for WireGuard, and IPv4 forwarding is disabled. SSH permits only the declared administrator using public-key authentication; root login, passwords, Atlas enforces targeted SELinux persistently and reports, without initiating, any reboot required to activate it. It assigns its primary LAN interface explicitly to the managed firewalld zone and applies persistent kernel network hardening: redirects and source routes are rejected, martians logged, reverse-path filtering remains loose for WireGuard, and IPv4 forwarding is disabled. SSH permits only the declared administrator using public-key authentication; root login, passwords,
agent and remote forwarding are disabled, while local forwarding remains available for private administrative tunnels. SMB3 exposes `Archive` only to the configured Vault-backed agent and remote forwarding are disabled, while local forwarding remains available for private administrative tunnels. SMB3 exposes `Archive` only to the configured Vault-backed
Samba accounts on encrypted, signed SMB3 over TCP/445 only and admits the configured LAN without host-specific Samba accounts on encrypted, signed SMB3 over TCP/445 only and admits the configured LAN without host-specific
exclusions. NFSv4 exports only `media/photobook` to the configured Aegis IP over TCP/2049, using exclusions. NFSv4 exports only `media/photobook` to the configured Aegis IP over TCP/2049, using
@@ -279,53 +292,243 @@ and NPM Quadlets share one Podman network. Immich runs as `1100:1100`; Server an
and Photobook is mounted read-only at `/external/photobook`. NPM publishes ports `80` and `443`; its and Photobook is mounted read-only at `/external/photobook`. NPM publishes ports `80` and `443`; its
administration interface remains restricted to `127.0.0.1:81` for SSH-tunnel access. administration interface remains restricted to `127.0.0.1:81` for SSH-tunnel access.
Phase 1 is limited to rootless Navidrome and Syncthing user Quadlets on Atlas. It is enabled in the Atlas temporarily hosts rootless Navidrome and Syncthing until Uranus replaces them. They bind only to
Atlas host configuration and can be set to `false` only for a deliberate suspension. Official Navidrome `0.63.2` uses its SQLite database below `/data`; it does Atlas' LAN address (`192.168.178.55`); WireGuard remains exclusively between Prometheus (`10.0.0.1`)
not support `ND_DATABASE_URL` or an external PostgreSQL backend. The obsolete `navidromedb` service and Aegis (`10.0.0.2`). Their state is initialized ex novo in `/zpool/services/data/navidrome` and
was therefore removed from Prometheus instead of being reproduced on Atlas. The role derives all `/zpool/services/data/syncthing`; no source application state is migrated. The music library at
storage paths from the `zpool` mounted at `/zpool`: music is read-only at `/zpool/media/music` is populated separately.
`/zpool/media/music`, Navidrome application state and `navidrome.db` are stored at
`/zpool/archive/app_data/navidrome`, and Syncthing persists at
`/zpool/archive/app_data/syncthing`. `profile_atlas` creates these datasets when
`atlas_manage_storage` is enabled; the backend role verifies their exact mountpoints before starting
containers. The backend role never creates the pool. The separate `wireguard_overlay` role manages `wg0`
between Prometheus (`10.0.0.1`) and Atlas (`10.0.0.2`), generating private keys once
on their respective hosts and exchanging only public keys through Ansible. Prometheus alone opens
`51820/udp` publicly. When the WireGuard zone is created, Ansible reloads firewalld and immediately
reloads Prometheus' rootful Podman networks so the existing proxy stack retains container DNS and
connectivity. Backend ports are admitted only in the WireGuard firewalld zone.
`backend_phase1_start_services` stays false during the application-state transfer, so the first real The separate `wireguard_overlay` role manages `wg0` between Prometheus (`10.0.0.1`) and Aegis
backend run renders the Quadlets without creating an empty Atlas database. After stopping Navidrome (`10.0.0.2`), generating private keys once on their respective hosts and exchanging only public keys
on Prometheus, copy the complete `/opt/navidrome/data/` directory into through Ansible. Prometheus alone opens `51820/udp`. Aegis forwards only the declared overlay-to-LAN
`/zpool/archive/app_data/navidrome/`, preserving `navidrome.db` and any SQLite sidecar files. Then set traffic and source-NATs it, so Atlas and future Uranus nodes require neither a VPN interface nor a router
this variable to true and rerun the role to enable and start Navidrome and Syncthing. The playbook static route. Atlas permits Navidrome (`4533/tcp`) and the Syncthing GUI (`8384/tcp`) only from Aegis;
never copies or deletes application data. Syncthing native ports are limited to the LAN. Configure NPM manually with
`http://192.168.178.55:4533` and `http://192.168.178.55:8384` after the services are healthy.
Prometheus' peer includes the LAN subnet in `AllowedIPs`; add the Uranus VIP there when it exists.
When the WireGuard zone is created, Ansible reloads firewalld and immediately reloads Prometheus'
rootful Podman networks so the existing proxy stack retains container DNS and connectivity.
Validate and render the Atlas services with: Validate the gateway with:
```bash ```bash
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \ ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit atlas --tags storage ansible-playbook ansible/site.yml --limit prometheus,aegis --tags wireguard --check --diff
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit prometheus,atlas --tags wireguard
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 --check --diff
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1
``` ```
For the cutover, stop the old Navidrome writer before copying its data directory, verify ownership by The first real WireGuard run must include both peers. If Fedora IoT has just layered `wireguard-tools`,
the Atlas `admin` account and confirm that the copied SQLite database is present before changing reboot Aegis manually and rerun the command without `--check`; the role then waits for a real peer
`backend_phase1_start_services: true` in `host_vars/atlas.yml`. Keep the source data and the stopped handshake.
legacy `navidromedb` container until Navidrome on Atlas and a restore test have been validated.
Snapshot retention, Syncthing topology, WireGuard/firewall validation, Prometheus backup pulls, Atlas declares recursive, systemd-timed ZFS snapshots for the complete pool hierarchy: 24 hourly
encrypted Borg backups to a Hetzner Storage Box, USB backup, monitoring, and disaster-recovery tests snapshots at minute 05, 30 daily snapshots at 00:15, 8 weekly snapshots on Sunday at 01:00, and 12
remain follow-up work. The detailed operational backlog is kept in `AGENTS.md`. monthly snapshots on the first day at 02:00. The retention helper prunes only snapshots carrying its
managed `atlas-auto` prefix and never rolls back a dataset. The OpenZFS monthly scrub timer is scheduled
for the first Sunday at 03:00; the conflicting weekly scrub timer is disabled explicitly. The first recursive
hourly snapshot completed successfully on Atlas; retention pruning and the first scheduled scrub still await
live runtime evidence. Validate this layer independently with:
```bash
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit atlas --tags snapshots,scrub --check --diff
```
Atlas also declares an encrypted Borg backup to the dedicated Hetzner Storage Box sub-account
`u660064-sub1`. The repository is the sub-account-relative `./borg-data` path and uses the explicitly
selected remote Borg 1.4 binary over SSH port 23. The ED25519 server key is pinned; a dedicated client
key is generated for the locked, non-login `borg` system account, and its private half never leaves
`/etc/atlas-borg`. The account has no sudo or supplementary groups and owns only its SSH identity,
passphrase, cache, and Borg state. Borg receives its passphrase through a mode `0600` file rendered from
`vault_atlas_borg_passphrase`.
The daily backup starts at 04:30 with up to 30 minutes of randomized delay. It creates a temporary,
recursive ZFS snapshot and reconstructs every dataset below `/zpool` as a read-only bind-mounted tree,
so parent and child datasets enter one consistent Borg archive. The wrapper recursively unmounts its
private source tree; a narrowly scoped `ExecStopPost` helper removes any remaining host-namespace ZFS
snapshot mounts and the named temporary snapshot after the backup process exits. Only the root wrapper
performs snapshot and mount operations; it launches the Borg client as `borg` with temporary read-search
capability and no ZFS, sudo, or pool-management privileges. Borg retains 30 daily, 8 weekly, and 12
monthly archives, then compacts the standard
read-write repository. A full metadata and repository check runs as `borg` on the fifteenth day of each
month at 06:00. Both operations use a common lock, journal logging, and bounded systemd retries.
New backup runs also log the create phase and a compact progress line at most once per minute: an
**estimated** percentage, dataset, files processed, and original/compressed/deduplicated bytes. The
denominator is the summed ZFS `logicalreferenced` size of the backup's own recursive snapshot, not a
Borg-reported total: the estimate can exceed 100% and does not cover retention, compaction, or checks.
Progress lines omit individual filenames; warnings may still name affected files.
Follow the current run with
`sudo journalctl -fu atlas-borg-backup.service` on Atlas; changes to the helper do not alter a run
already in progress.
Initial activation remains explicit:
1. Add a strong unique `vault_atlas_borg_passphrase` with `ansible-vault edit secrets/vault.yml`.
2. Generate and display only the dedicated public key with
`ansible-playbook ansible/site.yml --limit atlas --tags borg_key`.
3. Install that public key in the Hetzner sub-account, then apply with
`ansible-playbook ansible/site.yml --limit atlas --tags packages,borg`.
4. Copy the ignored `secrets/recovery/atlas-borg-repokey.export` file to genuinely offline storage.
The controller-side copy is not an offline backup by itself.
The role initializes only the missing `repokey` repository and never accepts an unpinned host key or
password authentication. It does not start the first backup manually. Validate the rendered state with:
```bash
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit atlas --tags packages,borg --check --diff
```
Atlas runtime activation is complete: the initial backup and repository check succeeded, a full restore
to a temporary directory was validated against the live `Archive` tree, the recovery-key export was copied
to offline storage, and the temporary snapshot and bind mounts were cleaned up.
On 2026-09-25 a separate ZFS restore smoke test copied a small file from an automatic daily
`zpool/archive` snapshot to `/var/tmp`, then confirmed matching contents, ownership, mode, mtime and
POSIX ACL. The temporary copy and on-demand snapshot mount were removed; Borg kept running. This
does not validate a full dataset recovery.
The offline USB backup is deployed as a manual-only service (`atlas_manage_usb_backup: true`):
Ansible never formats, unlocks, mounts, backs up to, or schedules the disk. Atlas' existing USB disk was verified
read-only on 2026-09-23 as LUKS UUID `577b3c43-ea37-4611-81a9-39d555cdfbd4`, containing ext4 UUID
`758e2d2e-a427-4797-aad9-39c3a9f17c7e` through mapper `zpool-backup`. It was mounted at
`/mnt/zpool-backup` at inspection time. The service deliberately requires the verified mapper to be
**not mounted** before starting. When necessary, `systemd-ask-password` requests the LUKS passphrase
through the `systemctl start` password agent; it is piped directly to `cryptsetup` without saving it,
passing it as a command argument, or caching it. The service then mounts the disk privately, takes a recursive ZFS snapshot,
copies every dataset to a versioned `atlas/snapshots/<timestamp>/` directory using `rsync --link-dest`,
verifies the result with a checksum-based dry run, atomically updates `atlas/latest`, unmounts and closes
LUKS. A failed run never replaces `latest` or removes an earlier complete version. Borg and the USB
backup may run concurrently from separate snapshots; both reading the same pool can reduce throughput.
The USB copy preserves ACLs but not generic extended attributes; `security.selinux` is also intentionally
excluded because the target SELinux policy must recreate labels during a restore. Do not restore data into
service paths without relabeling. After restoring an explicit dataset path, apply its destination policy with:
```bash
ansible-playbook ansible/site.yml --limit atlas --tags restorecon \
-e '{"atlas_restorecon_paths":["/zpool/archive"]}'
```
The task accepts only paths below the Atlas pool mount root, runs `restorecon -RFv` only for the paths
provided at invocation, and is otherwise a no-op. It must not be used on the whole pool during routine runs.
Old USB versions are not pruned automatically, to avoid deleting the only offline
copy without an explicitly chosen retention policy; capacity checks include an estimated transfer size
and a 10 GiB free-space reserve. The disk must be physically disconnected after a successful backup
to make the copy offline.
To check the USB backup and reminder configuration without starting a backup, run:
```bash
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit atlas --tags usb_backup,usb_reminder --check --diff
```
To validate a planned, explicit post-restore relabel operation without changing labels, run:
```bash
ansible-playbook ansible/site.yml --limit atlas --tags restorecon --check \
-e '{"atlas_restorecon_paths":["/zpool/archive"]}'
```
Before the first **manual** service start, safely unmount the currently mounted
`/mnt/zpool-backup`; never run it on an arbitrary mounted disk. Future starts
can begin with the mapper closed: `sudo systemctl start atlas-usb-backup.service` prompts for the
passphrase interactively and then performs the backup. Neither the LUKS password nor a key file belongs
in Ansible. Inspect the run with
`sudo journalctl -fu atlas-usb-backup.service`. There is intentionally no timer. Independently test a
read-only mount and restore from `atlas/latest` into an empty temporary directory before marking the
USB recovery path complete. Only `atlas-usb-reminder.timer` is enabled, for the first Saturday of each
month at 10:00 Europe/Rome. Its warning notification uses the existing 45Drives Houston notifier.
A manual test confirmed a notification in 45Drives Alerts, **not** an email. The reminder service log
reports notification submission, not email delivery; the role does not depend on SMTP/OAuth settings.
The reminder never starts the backup. Check its schedule with
`systemctl list-timers atlas-usb-reminder.timer` and the result in 45Drives Alerts.
The timer was verified active with its first scheduled run at 2026-10-03 10:00 CEST. No email
delivery is claimed.
The first manual USB attempt on 2026-09-23 did not complete: rsync was denied while removing
`security.selinux` on the USB filesystem, then the interrupted service left its recursive
`atlas-usb-20260923T185748Z-2469168` snapshot and the `zpool-backup` LUKS mapper open. The
rsync xattr filter was deployed afterward. The incomplete USB directory was absent on inspection;
the exact failed snapshot was removed, the verified and unmounted mapper closed, and the service
failed state cleared. A final check found no remnant snapshot, mount, mapper, or staging directory.
The failed attempt was not a valid backup, and no USB restore had been tested at that point.
On 2026-09-24 a later run reported a checksum-verified, published USB version and closed the LUKS
mapper, but the service failed while destroying its temporary ZFS snapshot: OpenZFS still had
on-demand `.zfs/snapshot` mounts open in the host namespace. Those exact temporary snapshots were
unmounted normally and removed; no force or rollback was used. The backup service now records its
snapshot name and runs a narrowly scoped `ExecStopPost` cleanup after the private backup process
exits. The cleanup helper was tested with a disposable recursive snapshot and an active snapshot
mount. A complete run on 2026-09-24 later checksum-verified and published a new USB version; the
service ended successfully, the LUKS mapper closed, no temporary USB snapshot remained, and the pool
was healthy. On 2026-09-25 an independent restore test opened the configured USB disk read-only, mounted
ext4 with `ro,noload`, restored a 5,707,945-byte file from the published `atlas/latest` version to an
empty `/var/tmp` directory, and matched its content, owner, mode, size, mtime, and POSIX ACL against
the USB source. The test removed its temporary copy and mount, closed the LUKS mapper, and left the
pool healthy while Borg continued running. This is a file-level recovery smoke test, not a full dataset
or disaster-recovery exercise.
Atlas health monitoring runs every 30 minutes through `atlas-health-monitor.timer`. Its read-only probes
check pool/vdev state and errors, scrub/resilver status, four pool disks and the system NVMe via SMART,
disk and CPU temperatures, system/pool/snapshot space, local `zpool/backup` growth, and the Hetzner
Storage Box quota via `df -m` over the dedicated `borg` account's pinned-key SSH connection. The remote
query never opens the Borg repository or its lock. Growth alerts compare against a roughly 24-hour
baseline and therefore begin only after enough samples exist. The monitor also checks
maintenance/backup timer activation and freshness; systemd `OnFailure` hooks report snapshot,
scrub, Borg, USB, reminder, and monitoring services when they enter the failed state. The ongoing
initial Borg run is never restarted by the monitor; only a run exceeding 14 days raises a warning.
Thresholds and stable disk paths are declared in Atlas host variables. Alerts use the existing 45Drives
Houston notifier and repeated issues are deduplicated; **email delivery is not verified**. The
2026-09-25 live probe found no issues and a labelled test notification was submitted. The Storage Box
reported 22% used. Detailed Borg archive size and deduplication still require the active run to finish.
```bash
ansible-playbook ansible/site.yml --limit atlas --tags monitoring --check --diff
sudo /usr/local/libexec/atlas-health-monitor --dry-run
sudo journalctl -u atlas-health-monitor.service -n 100 --no-pager
systemctl list-timers atlas-health-monitor.timer
```
`--dry-run` sends no alerts and does not change monitor state. A real check is
`sudo systemctl start atlas-health-monitor.service`; do not start the backup services merely to test
monitoring. For a labelled 45Drives Alerts delivery test, use
`sudo /usr/local/libexec/atlas-health-monitor --test-notification`.
### Atlas systemd timers
All nine managed timers below are enabled. Times are local to Atlas (`Europe/Rome`); Borg and monitoring
add the indicated randomized delay. Every timer has `Persistent=true`, so a missed calendar run is
scheduled after the timer becomes active again.
| Timer | Schedule (`OnCalendar`) | Action |
| --- | --- | --- |
| `atlas-zfs-snapshot-hourly.timer` | `*-*-* *:05:00` — every hour at :05 | Recursive hourly snapshot and retention |
| `atlas-zfs-snapshot-daily.timer` | `*-*-* 00:15:00` — daily at 00:15 | Recursive daily snapshot and retention |
| `atlas-zfs-snapshot-weekly.timer` | `Sun *-*-* 01:00:00` — Sunday at 01:00 | Recursive weekly snapshot and retention |
| `atlas-zfs-snapshot-monthly.timer` | `*-*-01 02:00:00` — first day of the month at 02:00 | Recursive monthly snapshot and retention |
| `zfs-scrub-monthly@zpool.timer` | `Sun *-*-01..07 03:00:00` — first Sunday at 03:00 | ZFS scrub |
| `atlas-borg-backup.timer` | `*-*-* 04:30:00` — daily at 04:30, plus 0–30 min random delay | Encrypted offsite backup |
| `atlas-borg-check.timer` | `*-*-15 06:00:00` — 15th of the month at 06:00, plus 0–30 min random delay | Borg repository check |
| `atlas-usb-reminder.timer` | `Sat *-*-01..07 10:00:00 Europe/Rome` — first Saturday at 10:00 | 45Drives Alerts reminder only |
| `atlas-health-monitor.timer` | `*:0/30` — every half-hour, plus 0–5 min random delay | Read-only health checks |
`atlas-usb-backup.service` has **no timer**: the encrypted USB backup must be started manually.
The vendor's `zfs-scrub-weekly@zpool.timer` is intentionally disabled in favor of the monthly scrub.
The future Prometheus backup pull has no timer yet because that workflow is not implemented. While a
Borg backup is still running, `systemctl list-timers` may show `-` for its next trigger; this does not
mean the timer has been disabled. Inspect the current schedule on Atlas with
`systemctl list-timers --all`.
A temporary Nextcloud deployment on Atlas is also planned before Uranus: it requires separately
declared persistent application, database, and cache storage, Vault-backed credentials, NPM-only
publishing through Aegis, and defined backup, upgrade, and eventual migration procedures. Do not deploy
it before the data-protection checklist is complete.
The desired future iCloud photo-ingestion host is Atlas, not Aegis. After data-protection validation,
plan an explicit iCloudPD migration with photos under `/zpool/archive/Pictures` and application/MFA
state outside `Archive`, then test permissions, SELinux, backups and recovery before cutting over.
The current Aegis iCloudPD service and Atlas Photobook NFS export remain configured until that
separate migration is approved and validated; the eventual Atlas service is temporary until Uranus.
Prometheus backup pulls, Borg archive-size evaluation, and full disaster-recovery tests remain follow-up
work. The prioritized operational backlog is kept in `AGENTS.md`.
## How layering works ## How layering works
@@ -412,8 +615,8 @@ ansible-playbook ansible/site.yml --limit deadalus --tags ai_agents --check --di
| `profile_workstation_dev_wsl` | WSL development setup. | | `profile_workstation_dev_wsl` | WSL development setup. |
| `profile_server` | Server setup. | | `profile_server` | Server setup. |
| `profile_atlas` | Rocky Linux 9 NAS setup. | | `profile_atlas` | Rocky Linux 9 NAS setup. |
| `profile_backend_phase1` | Rootless Navidrome and Syncthing on Atlas. | | `profile_backend_phase1` | Temporary rootless Atlas Navidrome and Syncthing services. |
| `wireguard_overlay` | Prometheus/Atlas WireGuard overlay. | | `wireguard_overlay` | Prometheus/Aegis WireGuard LAN gateway. |
| `profile_aegis` | Fedora IoT always-on LAN node. | | `profile_aegis` | Fedora IoT always-on LAN node. |
| `dotfiles_common` | Shared user dotfiles. | | `dotfiles_common` | Shared user dotfiles. |
@@ -425,8 +628,8 @@ platform_void -> packages_void + services_runit
platform_void & graphical_desktop -> profile_desktop_common + profile_desktop_sway + profile_desktop_niri + profile_desktop_host platform_void & graphical_desktop -> profile_desktop_common + profile_desktop_sway + profile_desktop_niri + profile_desktop_host
platform_fedora -> packages_fedora + services_systemd platform_fedora -> packages_fedora + services_systemd
platform_rocky -> packages_rocky + services_systemd platform_rocky -> packages_rocky + services_systemd
wireguard_overlay -> wireguard_overlay (after platform_rocky)
role_aegis -> profile_aegis role_aegis -> profile_aegis
wireguard_overlay -> wireguard_overlay (after Aegis profile and platform_rocky)
atlas -> profile_atlas atlas -> profile_atlas
role_backend_phase1 -> profile_backend_phase1 (after atlas) role_backend_phase1 -> profile_backend_phase1 (after atlas)
rocky_server -> dotfiles_common + profile_server (after platform_rocky) rocky_server -> dotfiles_common + profile_server (after platform_rocky)
@@ -531,7 +734,7 @@ ansible-playbook ansible/site.yml --list-tags
| `sharing` | Atlas NFSv4 and SMB3 configuration. | | `sharing` | Atlas NFSv4 and SMB3 configuration. |
| `storage` | Atlas child ZFS datasets. | | `storage` | Atlas child ZFS datasets. |
| `tmux` | tmux configuration and plugins. | | `tmux` | tmux configuration and plugins. |
| `wireguard` | Prometheus/Atlas WireGuard overlay. | | `wireguard` | Prometheus/Aegis WireGuard LAN gateway. |
| `wsl` | WSL bootstrap and configuration. | | `wsl` | WSL bootstrap and configuration. |
## Bootstrapping a new machine ## Bootstrapping a new machine

View File

@@ -9,6 +9,7 @@ fedora_packages_base:
- jq - jq
- make - make
- nodejs - nodejs
- openssl
- ripgrep - ripgrep
fedora_manage_docker_repo: true fedora_manage_docker_repo: true

View File

@@ -40,4 +40,3 @@ server_firewalld_services:
server_firewalld_ports: [] server_firewalld_ports: []
server_sshd_service_name: sshd server_sshd_service_name: sshd
server_compose_selinux_mount_option: Z server_compose_selinux_mount_option: Z
server_syncthing_enabled: false

View File

@@ -75,21 +75,6 @@ server_directories:
group: root group: root
mode: "0755" mode: "0755"
server_syncthing_enabled: true
server_syncthing_directories:
- path: /opt/syncthing/config
owner: root
group: root
mode: "0755"
- path: /srv/syncthing
owner: root
group: root
mode: "0755"
- path: /srv/syncthing/data
owner: "1000"
group: "1000"
mode: "0755"
server_sshd_settings: server_sshd_settings:
PermitRootLogin: "no" PermitRootLogin: "no"

View File

@@ -12,9 +12,10 @@ workstation_dev_wsl_packages:
- python3-pip - python3-pip
- tmux - tmux
# Java 11 and Maven are managed by Mise on this Fedora WSL profile. Keep their # Java 11, Java 25 and Maven are managed by Mise on this Fedora WSL profile.
# versions pinned; update them deliberately. # Keep their versions pinned; update them deliberately.
workstation_mise_java_version: temurin-11.0.31+11 workstation_mise_java_version: temurin-11.0.31+11
workstation_mise_java_25_version: 25.0.2
workstation_mise_maven_version: 3.9.16 workstation_mise_maven_version: 3.9.16
workstation_is_wsl: true workstation_is_wsl: true

View File

@@ -15,6 +15,26 @@ aegis_host_dns_servers:
- 192.168.178.1 - 192.168.178.1
aegis_host_dns_search_domains: aegis_host_dns_search_domains:
- fritz.box - fritz.box
# Aegis is the internal WireGuard endpoint and NAT gateway for the LAN.
wireguard_overlay_enabled: true
wireguard_address: 10.0.0.2/24
wireguard_enable_ipv4_forwarding: true
wireguard_reload_rootful_podman_networks: true
wireguard_forwarding_policies:
- name: aegis-wireguard-to-lan
ingress_zone: wireguard
egress_zone: public
source: 10.0.0.0/24
destination: "{{ aegis_lan_subnet }}"
masquerade: true
wireguard_peers:
- name: prometheus
host: prometheus
endpoint: "{{ hostvars['prometheus']['ansible_host'] }}:{{ hostvars['prometheus']['wireguard_listen_port'] }}"
allowed_ips:
- 10.0.0.0/24
persistent_keepalive: 25
aegis_ssh_authorized_keys: aegis_ssh_authorized_keys:
- name: ikaros - name: ikaros
key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINrIxXjA3ffPwziKGR5gzc4gAoBehQPlnEMcXF4Wl0ZS ikaros" key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINrIxXjA3ffPwziKGR5gzc4gAoBehQPlnEMcXF4Wl0ZS ikaros"

View File

@@ -35,35 +35,112 @@ atlas_zpool_disks:
- /dev/disk/by-id/ata-ST4000VN006-3CW104_WW6A6VJK - /dev/disk/by-id/ata-ST4000VN006-3CW104_WW6A6VJK
- /dev/disk/by-id/ata-ST4000VN006-3CW104_WW6AC1LM - /dev/disk/by-id/ata-ST4000VN006-3CW104_WW6AC1LM
atlas_create_pool: false atlas_create_pool: false
atlas_zfs_dataset_work: work
atlas_zfs_dataset_archive: archive atlas_zfs_dataset_archive: archive
atlas_zfs_dataset_app_data: archive/app_data atlas_zfs_dataset_services: services
atlas_zfs_dataset_navidrome: archive/app_data/navidrome atlas_zfs_dataset_app_data: services/data
atlas_zfs_dataset_syncthing: archive/app_data/syncthing atlas_zfs_dataset_navidrome: services/data/navidrome
atlas_zfs_dataset_syncthing: services/data/syncthing
atlas_zfs_dataset_media: media atlas_zfs_dataset_media: media
atlas_zfs_dataset_music: media/music atlas_zfs_dataset_music: media/music
atlas_zfs_dataset_backup_prometheus: backup_prometheus atlas_zfs_dataset_backup: backup
atlas_zfs_dataset_host_backups: backup/hosts
atlas_zfs_dataset_backup_prometheus: backup/hosts/prometheus
atlas_zfs_backup_reservation: 500G
atlas_zfs_dataset_photobook: media/photobook atlas_zfs_dataset_photobook: media/photobook
atlas_zfs_dataset_backups: backups
atlas_zfs_dataset_service_backups: backups/services
atlas_mount_root: /zpool atlas_mount_root: /zpool
atlas_manage_storage: true atlas_manage_storage: true
atlas_manage_zfs_snapshots: true
atlas_zfs_snapshot_prefix: atlas-auto
atlas_zfs_snapshot_policies:
- name: hourly
calendar: "*-*-* *:05:00"
keep: 24
- name: daily
calendar: "*-*-* 00:15:00"
keep: 30
- name: weekly
calendar: "Sun *-*-* 01:00:00"
keep: 8
- name: monthly
calendar: "*-*-01 02:00:00"
keep: 12
atlas_manage_zfs_scrub: true
atlas_zfs_scrub_calendar: "Sun *-*-01..07 03:00:00"
atlas_manage_borg_backup: true
atlas_borg_repository_host: u660064-sub1.your-storagebox.de
atlas_borg_repository_user: u660064-sub1
atlas_borg_repository_port: 23
atlas_borg_repository_path: ./borg-data
atlas_borg_remote_path: borg-1.4
# Verified against Hetzner's published ED25519 fingerprint on 2026-09-17:
# SHA256:XqONwb1S0zuj5A1CDxpOSuD2hnAArV1A3wKY7Z3sdgM
atlas_borg_host_key: >-
[u660064-sub1.your-storagebox.de]:23 ssh-ed25519
AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs
atlas_borg_backup_calendar: "*-*-* 04:30:00"
atlas_borg_check_calendar: "*-*-15 06:00:00"
atlas_borg_randomized_delay: 30m
atlas_borg_keep_daily: 30
atlas_borg_keep_weekly: 8
atlas_borg_keep_monthly: 12
atlas_manage_usb_backup: true
# Read-only lsblk verification on Atlas, 2026-09-23. Never store the LUKS password here.
atlas_usb_backup_luks_uuid: 577b3c43-ea37-4611-81a9-39d555cdfbd4
atlas_usb_backup_fs_uuid: 758e2d2e-a427-4797-aad9-39c3a9f17c7e
atlas_usb_backup_mapper_name: zpool-backup
atlas_manage_usb_reminder: true
atlas_usb_reminder_calendar: "Sat *-*-01..07 10:00:00 Europe/Rome"
atlas_manage_monitoring: true
# Physical pool disks and the system NVMe; the disconnected USB disk is intentionally excluded.
atlas_monitor_smart_devices:
- { name: pool-1, path: "{{ atlas_zpool_disks[0] }}", warning_c: 50, critical_c: 55 }
- { name: pool-2, path: "{{ atlas_zpool_disks[1] }}", warning_c: 50, critical_c: 55 }
- { name: pool-3, path: "{{ atlas_zpool_disks[2] }}", warning_c: 50, critical_c: 55 }
- { name: pool-4, path: "{{ atlas_zpool_disks[3] }}", warning_c: 50, critical_c: 55 }
- name: system-nvme
path: /dev/disk/by-id/nvme-Patriot_M.2_P320_256GB_P320ADB26011606111
warning_c: 70
critical_c: 85
atlas_monitor_timers:
- { name: atlas-zfs-snapshot-hourly.timer, max_age_hours: 3 }
- { name: atlas-zfs-snapshot-daily.timer, max_age_hours: 36 }
- { name: atlas-zfs-snapshot-weekly.timer, max_age_hours: 216 }
- { name: atlas-zfs-snapshot-monthly.timer, max_age_hours: 960 }
- { name: zfs-scrub-monthly@zpool.timer, max_age_hours: 960 }
- { name: atlas-borg-backup.timer, max_age_hours: 48 }
- { name: atlas-borg-check.timer, max_age_hours: 960 }
# The first manual USB reminder is not due until October; activation is checked, not age.
- { name: atlas-usb-reminder.timer, max_age_hours: 0 }
atlas_monitor_failure_units:
- atlas-zfs-snapshot@.service
- zfs-scrub@zpool.service
- atlas-borg-backup.service
- atlas-borg-check.service
- atlas-usb-backup.service
- atlas-usb-reminder.service
- atlas-health-monitor.service
atlas_monitor_remote_capacity:
user: "{{ atlas_borg_repository_user }}"
host: "{{ atlas_borg_repository_host }}"
run_as: "{{ atlas_borg_username }}"
ssh_wrapper: "{{ atlas_borg_ssh_wrapper_path }}"
warning_percent: 80
critical_percent: 90
growth_warning_gib_day: 500
atlas_manage_sharing: true atlas_manage_sharing: true
atlas_manage_media_stack: false atlas_manage_media_stack: false
# Planned after data-protection validation: move iCloudPD photo ingestion from
# Aegis to Atlas, with photos under /zpool/archive/Pictures and persistent
# application/MFA state outside Archive. Do not deploy or cut over yet.
wireguard_overlay_enabled: true # WireGuard is retired on Atlas. These rootless services are a temporary home
wireguard_address: 10.0.0.2/24 # until Uranus replaces them.
wireguard_peers:
- name: prometheus
host: prometheus
endpoint: "{{ hostvars['prometheus']['ansible_host'] }}:{{ hostvars['prometheus']['wireguard_listen_port'] }}"
allowed_ips:
- 10.0.0.0/24
persistent_keepalive: 25
backend_phase1_enabled: true backend_phase1_enabled: true
backend_phase1_start_services: false backend_phase1_start_services: true
backend_phase1_wireguard_address: 10.0.0.2 backend_phase1_bind_address: "{{ ansible_host }}"
backend_phase1_firewalld_zone: "{{ atlas_firewalld_zone }}"
backend_phase1_npm_source_ip: "{{ atlas_aegis_ip }}"
backend_phase1_syncthing_native_subnet: "{{ atlas_lan_subnet }}"
rocky_manage_openzfs_repo: true rocky_manage_openzfs_repo: true
rocky_manage_syncthing_binary: false rocky_manage_syncthing_binary: false
@@ -73,13 +150,21 @@ rocky_podman_packages:
host_packages: host_packages:
- cockpit - cockpit
- cockpit-podman
- cockpit-storaged
- realmd
- pcp
- python3-pcp
- cryptsetup
- nfs-utils - nfs-utils
- policycoreutils - policycoreutils
- policycoreutils-python-utils - policycoreutils-python-utils
- python3-libselinux - python3-libselinux
- setroubleshoot-server
- samba - samba
- samba-client - samba-client
- samba-common-tools - samba-common-tools
- borgbackup
- zfs - zfs
atlas_nfs_exports: atlas_nfs_exports:
@@ -113,4 +198,5 @@ atlas_firewalld_rich_rules:
host_enabled_services: host_enabled_services:
- sshd - sshd
- cockpit.socket - cockpit.socket
- pmlogger.service
- zfs.target - zfs.target

View File

@@ -21,15 +21,23 @@ wireguard_listen_port: 51820
wireguard_enable_ipv4_forwarding: true wireguard_enable_ipv4_forwarding: true
wireguard_reload_rootful_podman_networks: true wireguard_reload_rootful_podman_networks: true
wireguard_peers: wireguard_peers:
- name: atlas - name: aegis
host: atlas host: aegis
# TODO: Aggiungere VIP di Uranus in futuro
allowed_ips: allowed_ips:
- 10.0.0.2/32 - 10.0.0.2/32
- 192.168.178.0/24
host_packages: host_packages:
- cockpit - cockpit
- cockpit-navigator - cockpit-navigator
- cockpit-podman - cockpit-podman
- cockpit-storaged
- realmd
- pcp
- python3-pcp
- setroubleshoot-server
host_enabled_services: host_enabled_services:
- cockpit.socket - cockpit.socket
- pmlogger.service

View File

@@ -99,7 +99,7 @@ all:
wireguard_overlay: wireguard_overlay:
hosts: hosts:
prometheus: prometheus:
atlas: aegis:
role_backend_phase1: role_backend_phase1:
hosts: hosts:

View File

@@ -2,6 +2,7 @@
aegis_hostname: aegis aegis_hostname: aegis
aegis_layered_packages: aegis_layered_packages:
- nfs-utils - nfs-utils
- wireguard-tools
aegis_network_connection_name: "" aegis_network_connection_name: ""
aegis_network_connection_uuid: "" aegis_network_connection_uuid: ""
aegis_host_dns_servers: [] aegis_host_dns_servers: []

View File

@@ -1,6 +1,6 @@
--- ---
- name: Manage Aegis layered packages - name: Manage Aegis layered packages
tags: [aegis, packages, nfs] tags: [aegis, packages, nfs, wireguard]
community.general.rpm_ostree_pkg: community.general.rpm_ostree_pkg:
name: "{{ aegis_layered_packages }}" name: "{{ aegis_layered_packages }}"
state: present state: present
@@ -8,7 +8,7 @@
when: aegis_layered_packages | length > 0 when: aegis_layered_packages | length > 0
- name: Report reboot required for Aegis layered packages - name: Report reboot required for Aegis layered packages
tags: [aegis, packages, nfs] tags: [aegis, packages, nfs, wireguard]
ansible.builtin.debug: ansible.builtin.debug:
msg: Reboot Aegis to activate the newly layered packages, then rerun the playbook. msg: Reboot Aegis to activate the newly layered packages, then rerun the playbook.
when: aegis_layered_packages_result.needs_reboot | default(false) when: aegis_layered_packages_result.needs_reboot | default(false)

View File

@@ -47,31 +47,101 @@ atlas_hardening_sysctls:
net.ipv6.conf.all.accept_source_route: "0" net.ipv6.conf.all.accept_source_route: "0"
net.ipv6.conf.default.accept_source_route: "0" net.ipv6.conf.default.accept_source_route: "0"
atlas_zfs_pool: CHANGEME_ZFS_POOL atlas_zfs_pool: CHANGEME_ZFS_POOL
atlas_zfs_dataset_work: work
atlas_zfs_dataset_archive: archive atlas_zfs_dataset_archive: archive
atlas_zfs_dataset_app_data: archive/app_data atlas_zfs_dataset_services: services
atlas_zfs_dataset_navidrome: archive/app_data/navidrome atlas_zfs_dataset_app_data: services/data
atlas_zfs_dataset_syncthing: archive/app_data/syncthing atlas_zfs_dataset_navidrome: services/data/navidrome
atlas_zfs_dataset_syncthing: services/data/syncthing
atlas_zfs_dataset_media: media atlas_zfs_dataset_media: media
atlas_zfs_dataset_music: media/music atlas_zfs_dataset_music: media/music
atlas_zfs_dataset_backup_prometheus: backup_prometheus atlas_zfs_dataset_backup: backup
atlas_zfs_dataset_host_backups: backup/hosts
atlas_zfs_dataset_backup_prometheus: backup/hosts/prometheus
atlas_zfs_backup_reservation: 500G
atlas_zfs_dataset_photobook: media/photobook atlas_zfs_dataset_photobook: media/photobook
atlas_zfs_dataset_backups: backups
atlas_zfs_dataset_service_backups: backups/services
atlas_zfs_service_backups_refreservation: 500G
atlas_mount_root: /CHANGEME_ATLAS_MOUNT_ROOT atlas_mount_root: /CHANGEME_ATLAS_MOUNT_ROOT
atlas_manage_zfs_snapshots: false
atlas_zfs_snapshot_prefix: atlas-auto
atlas_zfs_snapshot_policies: []
atlas_manage_zfs_scrub: false
atlas_zfs_scrub_calendar: ""
atlas_manage_borg_backup: false
atlas_borg_username: borg
atlas_borg_group: borg
atlas_borg_home: /var/lib/atlas-borg
atlas_borg_repository_host: CHANGEME_BORG_HOST
atlas_borg_repository_user: CHANGEME_BORG_USER
atlas_borg_repository_port: 23
atlas_borg_repository_path: ./borg-data
atlas_borg_remote_path: borg-1.4
atlas_borg_host_key: ""
atlas_borg_ssh_private_key_path: /etc/atlas-borg/id_ed25519
atlas_borg_known_hosts_path: /etc/atlas-borg/known_hosts
atlas_borg_passphrase_path: /etc/atlas-borg/passphrase
atlas_borg_ssh_wrapper_path: /usr/local/libexec/atlas-borg-ssh
atlas_borg_passphrase: "{{ vault_atlas_borg_passphrase | default('') }}"
atlas_borg_encryption_mode: repokey
atlas_borg_archive_prefix: atlas
atlas_borg_snapshot_prefix: atlas-borg
atlas_borg_compression: auto,zstd,3
atlas_borg_backup_calendar: ""
atlas_borg_check_calendar: ""
atlas_borg_randomized_delay: 30m
atlas_borg_keep_daily: 30
atlas_borg_keep_weekly: 8
atlas_borg_keep_monthly: 12
atlas_borg_config_dir: /var/lib/atlas-borg
atlas_borg_cache_dir: /var/cache/atlas-borg
atlas_borg_lock_path: /var/lib/atlas-borg/backup.lock
atlas_borg_recovery_export_path: "{{ playbook_dir }}/../secrets/recovery/atlas-borg-repokey.export"
# Manual-only offline backup. No USB device is formatted or mounted by Ansible.
atlas_manage_usb_backup: false
atlas_usb_backup_luks_uuid: ""
atlas_usb_backup_fs_uuid: ""
atlas_usb_backup_mapper_name: atlas-usb-backup
atlas_usb_backup_min_free_bytes: 10737418240
atlas_usb_backup_snapshot_prefix: atlas-usb
atlas_manage_usb_reminder: false
atlas_usb_reminder_calendar: ""
atlas_usb_reminder_notifier: /opt/45drives/houston/houston-notify
# Read-only health probes and 45Drives Alerts; disabled outside Atlas host vars.
atlas_manage_monitoring: false
atlas_monitor_calendar: "*:0/30"
atlas_monitor_notifier: "{{ atlas_usb_reminder_notifier }}"
atlas_monitor_smart_devices: []
atlas_monitor_timers: []
atlas_monitor_failure_units: []
atlas_monitor_remote_capacity: {}
atlas_monitor_pool_warning_percent: 80
atlas_monitor_pool_critical_percent: 90
atlas_monitor_root_warning_percent: 80
atlas_monitor_root_critical_percent: 90
atlas_monitor_snapshot_warning_percent: 10
atlas_monitor_snapshot_critical_percent: 20
atlas_monitor_snapshot_growth_warning_gib_day: 100
atlas_monitor_backup_growth_warning_gib_day: 100
atlas_monitor_cpu_warning_c: 85
atlas_monitor_cpu_critical_c: 95
atlas_monitor_borg_max_runtime_days: 14
# Explicit post-restore relabeling only; never relabel datasets during ordinary runs.
atlas_restorecon_paths: []
atlas_archive_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_archive }}" atlas_archive_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_archive }}"
atlas_services_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_services }}"
atlas_app_data_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_app_data }}" atlas_app_data_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_app_data }}"
atlas_navidrome_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_navidrome }}" atlas_navidrome_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_navidrome }}"
atlas_syncthing_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_syncthing }}" atlas_syncthing_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_syncthing }}"
atlas_media_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_media }}" atlas_media_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_media }}"
atlas_music_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_music }}" atlas_music_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_music }}"
atlas_work_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_work }}" atlas_backup_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_backup }}"
atlas_host_backups_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_host_backups }}"
atlas_backup_prometheus_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_backup_prometheus }}" atlas_backup_prometheus_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_backup_prometheus }}"
atlas_photobook_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_photobook }}" atlas_photobook_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_photobook }}"
atlas_backups_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_backups }}"
atlas_service_backups_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_service_backups }}"
atlas_45drives_repo_url: https://repo.45drives.com/repofiles/rocky/45drives-enterprise.repo atlas_45drives_repo_url: https://repo.45drives.com/repofiles/rocky/45drives-enterprise.repo
atlas_45drives_repo_file: /etc/yum.repos.d/45drives-enterprise.repo atlas_45drives_repo_file: /etc/yum.repos.d/45drives-enterprise.repo

View File

@@ -0,0 +1,59 @@
#!/usr/bin/env python3
"""Turn Borg's JSON progress stream into bounded, readable journal entries."""
import argparse
import json
import sys
import time
def size(value):
if not isinstance(value, (int, float)):
return "unknown"
return f"{value / (1024 ** 3):.2f} GiB"
parser = argparse.ArgumentParser()
parser.add_argument("--estimated-total-bytes", type=int, required=True)
args = parser.parse_args()
if args.estimated_total_bytes <= 0:
parser.error("estimated total must be positive")
last_progress = 0.0
for line in sys.stdin:
try:
event = json.loads(line)
except json.JSONDecodeError:
print(line.rstrip(), flush=True)
continue
kind = event.get("type")
if kind == "archive_progress":
now = time.monotonic()
if now - last_progress < 60 and not event.get("finished"):
continue
path = event.get("path") or ""
parts = path.split("/")
dataset = parts[1] if len(parts) > 1 and parts[0] == "source" else "unknown"
original_size = event.get("original_size")
if isinstance(original_size, (int, float)) and original_size >= 0:
percent = original_size / args.estimated_total_bytes * 100
estimated_progress = (
f"{percent:.1f}%" if percent < 100 else ">=100% (ZFS estimate exceeded)"
)
else:
estimated_progress = "unknown"
print(
"Borg create progress: "
f"estimated={estimated_progress} dataset={dataset} "
f"files={event.get('nfiles', 'unknown')} "
f"original={size(original_size)} "
f"compressed={size(event.get('compressed_size'))} "
f"deduplicated={size(event.get('deduplicated_size'))}",
flush=True,
)
last_progress = now
elif kind == "log_message":
print(f"Borg {event.get('levelname', 'INFO')}: {event.get('message', '')}", flush=True)
elif kind == "progress_message" and event.get("message"):
print(f"Borg: {event['message']}", flush=True)

View File

@@ -0,0 +1,396 @@
#!/usr/bin/python3
"""Read-only Atlas health probes with deduplicated 45Drives Alerts."""
import argparse
import fcntl
import json
import os
import re
import subprocess
import sys
import tempfile
import time
from datetime import datetime, timezone
from pathlib import Path
CONFIG_PATH = Path("/etc/atlas-health-monitor.json")
STATE_DIR = Path("/var/lib/atlas-health-monitor")
STATE_PATH = STATE_DIR / "state.json"
GIB = 1024**3
def run(*argv, timeout=40):
return subprocess.run(argv, capture_output=True, text=True, timeout=timeout, check=False)
def issue(issues, key, severity, message):
issues[key] = {"severity": severity, "message": message}
def notify(config, event, severity, subject, message):
now = datetime.now(timezone.utc)
payload = {
"timestamp": now.isoformat(timespec="seconds"),
"unixtime": int(now.timestamp()),
"event": event,
"severity": severity,
"subject": subject,
"email_message": message,
}
result = run(config["notifier"], json.dumps(payload, ensure_ascii=False), timeout=30)
if result.returncode:
raise RuntimeError(f"45Drives notifier exited {result.returncode}: {result.stderr.strip()}")
def parse_fields(text):
return dict(line.split("=", 1) for line in text.splitlines() if "=" in line)
def systemd_fields(unit, *properties):
result = run("systemctl", "show", unit, *(f"-p{item}" for item in properties))
if result.returncode:
raise RuntimeError(f"systemctl show {unit} exited {result.returncode}")
return parse_fields(result.stdout)
def unix_time(text):
if not text or text == "n/a":
return None
result = run("date", "-d", text, "+%s")
if result.returncode:
raise ValueError(f"Cannot parse systemd timestamp: {text}")
return int(result.stdout.strip())
def check_pool(config, issues, measurements):
pool = config["pool"]
listing = run("zpool", "list", "-H", "-p", "-o", "size,alloc,capacity,health", pool)
if listing.returncode:
issue(issues, "pool.probe", "critical", f"Cannot query ZFS pool {pool}")
return
try:
size, alloc, capacity, health = listing.stdout.strip().split("\t")
size, alloc, capacity = int(size), int(alloc), int(capacity)
except (ValueError, TypeError):
issue(issues, "pool.probe", "critical", "Invalid ZFS pool capacity response")
return
measurements.update(pool_size_bytes=size, pool_alloc_bytes=alloc, pool_capacity_percent=capacity)
if health != "ONLINE":
issue(issues, "pool.health", "critical", f"ZFS pool {pool} state is {health}")
if capacity >= config["pool_critical_percent"]:
issue(issues, "pool.capacity", "critical", f"ZFS pool {pool} is {capacity}% full")
elif capacity >= config["pool_warning_percent"]:
issue(issues, "pool.capacity", "warning", f"ZFS pool {pool} is {capacity}% full")
status = run("zpool", "status", "-P", pool)
if status.returncode:
issue(issues, "pool.status", "critical", f"Cannot query detailed ZFS status for {pool}")
return
bad_vdevs = []
for line in status.stdout.splitlines():
match = re.match(r"^\s*(\S+)\s+(ONLINE|DEGRADED|FAULTED|OFFLINE|UNAVAIL|REMOVED)\s+(\d+)\s+(\d+)\s+(\d+)", line)
if match:
name, state, reads, writes, checksums = match.groups()
if state != "ONLINE" or any(int(value) for value in (reads, writes, checksums)):
bad_vdevs.append(f"{name}: {state}, READ={reads}, WRITE={writes}, CKSUM={checksums}")
if bad_vdevs:
issue(issues, "pool.vdevs", "critical", "ZFS vdev errors: " + "; ".join(bad_vdevs))
errors = re.search(r"^errors:\s*(.*)$", status.stdout, re.MULTILINE)
if not errors or errors.group(1).strip() != "No known data errors":
issue(issues, "pool.data_errors", "critical", "ZFS status reports data errors; inspect zpool status -v")
if re.search(r"^\s*scan:\s*resilver in progress", status.stdout, re.MULTILINE | re.IGNORECASE):
issue(issues, "pool.resilver", "warning", "ZFS resilver is in progress; inspect zpool status")
scan = re.search(r"^\s*scan:\s*(.*)$", status.stdout, re.MULTILINE)
if scan and re.search(r"\bwith [1-9][0-9]* errors\b", scan.group(1)):
issue(issues, "pool.scan_errors", "critical", f"ZFS scan reported errors: {scan.group(1)}")
def check_capacity(config, issues, measurements):
pool = config["pool"]
listing = run("zfs", "list", "-H", "-p", "-o", "name,usedbysnapshots", "-r", pool)
if listing.returncode:
issue(issues, "snapshot.probe", "warning", "Cannot query ZFS snapshot space")
else:
try:
snapshots = sum(int(line.split("\t")[1]) for line in listing.stdout.splitlines())
measurements["snapshots_bytes"] = snapshots
size = measurements.get("pool_size_bytes")
if size:
percent = snapshots * 100 // size
measurements["snapshots_percent"] = percent
if percent >= config["snapshot_critical_percent"]:
issue(issues, "snapshot.capacity", "critical", f"Snapshots use {percent}% of pool size")
elif percent >= config["snapshot_warning_percent"]:
issue(issues, "snapshot.capacity", "warning", f"Snapshots use {percent}% of pool size")
except (ValueError, IndexError):
issue(issues, "snapshot.probe", "warning", "Invalid ZFS snapshot-space response")
backup = run("zfs", "list", "-H", "-p", "-o", "used", config["backup_dataset"])
if backup.returncode:
issue(issues, "backup.capacity_probe", "warning", "Cannot query local backup dataset space")
else:
try:
measurements["backup_bytes"] = int(backup.stdout.strip())
except ValueError:
issue(issues, "backup.capacity_probe", "warning", "Invalid local backup space response")
try:
filesystem = os.statvfs("/")
total = filesystem.f_blocks * filesystem.f_frsize
available = filesystem.f_bavail * filesystem.f_frsize
used_percent = (total - available) * 100 // total
measurements["root_capacity_percent"] = used_percent
if used_percent >= config["root_critical_percent"]:
issue(issues, "root.capacity", "critical", f"Atlas system filesystem is {used_percent}% full")
elif used_percent >= config["root_warning_percent"]:
issue(issues, "root.capacity", "warning", f"Atlas system filesystem is {used_percent}% full")
except (OSError, ZeroDivisionError):
issue(issues, "root.capacity_probe", "warning", "Cannot query Atlas system filesystem space")
def check_remote_capacity(config, issues, measurements):
"""Query only the Storage Box quota; do not open or inspect the Borg repository."""
remote = config["remote_capacity"]
try:
result = run("runuser", "-u", remote["run_as"], "--", remote["ssh_wrapper"],
f"{remote['user']}@{remote['host']}", "df", "-m", timeout=65)
if result.returncode:
raise ValueError(f"SSH df exited {result.returncode}")
lines = result.stdout.strip().splitlines()
if len(lines) != 2:
raise ValueError("Unexpected Storage Box df output")
fields = lines[1].split()
if len(fields) < 5:
raise ValueError("Incomplete Storage Box df output")
total_mib, used_mib, available_mib = (int(value) for value in fields[1:4])
percent = int(fields[4].rstrip("%"))
if total_mib <= 0 or not 0 <= percent <= 100 or available_mib < 0:
raise ValueError("Invalid Storage Box quota values")
except (OSError, ValueError, subprocess.TimeoutExpired):
issue(issues, "remote.capacity_probe", "warning", "Cannot query Hetzner Storage Box quota via pinned-key SSH")
return
measurements.update(remote_capacity_percent=percent, remote_bytes=used_mib * 1024**2,
remote_available_bytes=available_mib * 1024**2)
if percent >= remote["critical_percent"]:
issue(issues, "remote.capacity", "critical", f"Hetzner Storage Box quota is {percent}% full")
elif percent >= remote["warning_percent"]:
issue(issues, "remote.capacity", "warning", f"Hetzner Storage Box quota is {percent}% full")
def check_smart(config, issues, measurements):
for device in config["smart_devices"]:
name, path = device["name"], device["path"]
try:
result = run("smartctl", "-j", "-a", path, timeout=60)
data = json.loads(result.stdout)
status = int(data.get("smartctl", {}).get("exit_status", result.returncode))
except (subprocess.TimeoutExpired, json.JSONDecodeError, ValueError) as exc:
issue(issues, f"smart.{name}.probe", "critical", f"SMART probe failed for {name}: {type(exc).__name__}")
continue
if status:
severity = "critical" if status & 0b00001111 else "warning"
issue(issues, f"smart.{name}.status", severity, f"SMART reported exit status {status} for {name}")
passed = data.get("smart_status", {}).get("passed")
if passed is False:
issue(issues, f"smart.{name}.health", "critical", f"SMART self-assessment failed for {name}")
elif passed is None:
issue(issues, f"smart.{name}.health", "warning", f"SMART self-assessment unavailable for {name}")
temperature = data.get("temperature", {}).get("current")
if isinstance(temperature, (int, float)):
measurements[f"smart_{name}_c"] = temperature
if temperature >= device["critical_c"]:
issue(issues, f"smart.{name}.temperature", "critical", f"{name} temperature is {temperature} C")
elif temperature >= device["warning_c"]:
issue(issues, f"smart.{name}.temperature", "warning", f"{name} temperature is {temperature} C")
else:
issue(issues, f"smart.{name}.temperature", "warning", f"Temperature unavailable for {name}")
for attribute in data.get("ata_smart_attributes", {}).get("table", []):
attribute_id = attribute.get("id")
if attribute_id in (5, 187, 197, 198):
raw = attribute.get("raw", {}).get("value", 0)
if isinstance(raw, int) and raw > 0:
severity = "critical" if attribute_id in (197, 198) else "warning"
issue(issues, f"smart.{name}.ata_{attribute_id}", severity,
f"{name} SMART attribute {attribute_id} raw count is {raw}")
nvme = data.get("nvme_smart_health_information_log", {})
if isinstance(nvme, dict):
if int(nvme.get("critical_warning", 0)):
issue(issues, f"smart.{name}.nvme_warning", "critical", f"{name} NVMe critical warning is nonzero")
if int(nvme.get("media_errors", 0)):
issue(issues, f"smart.{name}.nvme_media", "critical", f"{name} NVMe media errors are nonzero")
def check_cpu(config, issues, measurements):
sensors = []
for hwmon in Path("/sys/class/hwmon").glob("hwmon*"):
try:
if (hwmon / "name").read_text().strip() != "coretemp":
continue
sensors.extend(int(path.read_text().strip()) / 1000 for path in hwmon.glob("temp*_input"))
except (OSError, ValueError):
continue
if not sensors:
issue(issues, "cpu.temperature_probe", "warning", "CPU temperature sensors are unavailable")
return
hottest = max(sensors)
measurements["cpu_max_c"] = hottest
if hottest >= config["cpu_critical_c"]:
issue(issues, "cpu.temperature", "critical", f"CPU temperature is {hottest:g} C")
elif hottest >= config["cpu_warning_c"]:
issue(issues, "cpu.temperature", "warning", f"CPU temperature is {hottest:g} C")
def check_jobs(config, issues, measurements, now):
for timer in config["timers"]:
name = timer["name"]
try:
fields = systemd_fields(name, "ActiveState", "UnitFileState", "LastTriggerUSec", "ActiveEnterTimestamp")
if fields.get("ActiveState") != "active" or fields.get("UnitFileState") != "enabled":
issue(issues, f"timer.{name}", "critical", f"Timer {name} is not active and enabled")
max_age = int(timer["max_age_hours"]) * 3600
if max_age:
last = unix_time(fields.get("LastTriggerUSec"))
if last is None:
last = unix_time(fields.get("ActiveEnterTimestamp"))
if last is not None and now - last > max_age:
issue(issues, f"timer.{name}.stale", "warning",
f"Timer {name} has not fired in {int((now-last)/3600)} hours")
except (RuntimeError, ValueError, subprocess.TimeoutExpired):
issue(issues, f"timer.{name}.probe", "warning", f"Cannot query timer {name}")
for unit in config["failure_units"]:
if unit.endswith("@.service"):
continue
try:
fields = systemd_fields(unit, "ActiveState", "Result", "ExecMainStartTimestamp")
state = fields.get("ActiveState")
if state == "failed" or (state == "inactive" and fields.get("Result") not in (None, "", "success")):
issue(issues, f"service.{unit}", "critical", f"Service {unit} failed: {fields.get('Result')}")
if unit == "atlas-borg-backup.service" and fields.get("ActiveState") == "activating":
started = unix_time(fields.get("ExecMainStartTimestamp"))
if started is not None and now - started > config["borg_max_runtime_days"] * 86400:
issue(issues, "backup.borg_long_running", "warning",
"Borg has run longer than its configured limit")
except (RuntimeError, ValueError, subprocess.TimeoutExpired):
issue(issues, f"service.{unit}.probe", "warning", f"Cannot query service {unit}")
def check_growth(config, issues, measurements, samples, now):
previous = [sample for sample in samples if 20 * 3600 <= now - sample.get("time", now) <= 48 * 3600]
if previous:
baseline = min(previous, key=lambda sample: abs(now - sample["time"] - 86400))
days = (now - baseline["time"]) / 86400
for name, threshold in (("snapshots", config["snapshot_growth_warning_gib_day"]),
("backup", config["backup_growth_warning_gib_day"]),
("remote", config["remote_capacity"]["growth_warning_gib_day"])):
current, old = measurements.get(f"{name}_bytes"), baseline.get(f"{name}_bytes")
if isinstance(current, int) and isinstance(old, int) and days > 0:
growth_gib_day = (current - old) / GIB / days
measurements[f"{name}_growth_gib_day"] = round(growth_gib_day, 1)
if growth_gib_day >= threshold:
issue(issues, f"{name}.growth", "warning",
f"Local {name} usage grew {growth_gib_day:.1f} GiB/day over {days:.1f} days")
def allowed_failure_unit(config, unit):
for allowed in config["failure_units"]:
if allowed == unit:
return True
if allowed.endswith("@.service") and unit.startswith(allowed[:-9] + "@") and unit.endswith(".service"):
return True
return False
def load_state():
if not STATE_PATH.exists():
return {"active": {}, "samples": []}
with STATE_PATH.open(encoding="utf-8") as stream:
state = json.load(stream)
if not isinstance(state.get("active"), dict) or not isinstance(state.get("samples"), list):
raise ValueError("Invalid Atlas monitor state; refusing to overwrite it")
return state
def save_state(state):
with tempfile.NamedTemporaryFile("w", dir=STATE_DIR, prefix=".state-", delete=False,
encoding="utf-8") as stream:
path = Path(stream.name)
os.chmod(path, 0o600)
json.dump(state, stream, sort_keys=True)
stream.write("\n")
stream.flush()
os.fsync(stream.fileno())
os.replace(path, STATE_PATH)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--dry-run", action="store_true", help="probe without notifications or state changes")
parser.add_argument("--test-notification", action="store_true", help="submit a labelled test alert")
parser.add_argument("--job-failed", metavar="UNIT", help="notify about a failed configured service")
args = parser.parse_args()
with CONFIG_PATH.open(encoding="utf-8") as stream:
config = json.load(stream)
if args.test_notification:
notify(config, "atlas_monitor_test", "warning", "Test monitoraggio Atlas",
"Notifica di prova: il monitoraggio Atlas raggiunge 45Drives Alerts. Non conferma l'invio email.")
print("Atlas monitor test submitted to 45Drives Alerts; email delivery is not verified.")
return 0
if args.job_failed:
if not allowed_failure_unit(config, args.job_failed):
raise ValueError("Unconfigured Atlas failure unit")
notify(config, "atlas_job_failed", "critical", f"Job Atlas fallito: {args.job_failed}",
f"Il servizio {args.job_failed} e' fallito. Controlla: "
f"sudo journalctl -u {args.job_failed} -n 100 --no-pager")
print(f"Atlas job failure submitted to 45Drives Alerts: {args.job_failed}")
return 0
now = int(time.time())
issues, measurements = {}, {}
check_pool(config, issues, measurements)
check_capacity(config, issues, measurements)
check_remote_capacity(config, issues, measurements)
check_smart(config, issues, measurements)
check_cpu(config, issues, measurements)
check_jobs(config, issues, measurements, now)
if args.dry_run:
print(json.dumps({"issues": issues, "measurements": measurements}, sort_keys=True))
return 0
STATE_DIR.mkdir(mode=0o700, exist_ok=True)
with (STATE_DIR / "monitor.lock").open("w") as lock:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
state = load_state()
check_growth(config, issues, measurements, state["samples"], now)
active, failed_notifications = state["active"], []
for key, details in issues.items():
old = active.get(key)
if old is None or old.get("severity") != details["severity"]:
try:
notify(config, "atlas_health_issue", details["severity"],
f"Atlas: {key}", details["message"])
active[key] = details
print(f"ALERT {details['severity']} {key}: {details['message']}", flush=True)
except (RuntimeError, subprocess.TimeoutExpired) as exc:
failed_notifications.append(key)
print(f"NOTIFICATION FAILED {key}: {exc}", file=sys.stderr, flush=True)
for key in set(active) - set(issues):
print(f"RECOVERED {key}", flush=True)
del active[key]
state["samples"] = [sample for sample in state["samples"] if now - sample.get("time", 0) < 48 * 3600]
state["samples"].append({"time": now, **{key: value for key, value in measurements.items()
if key in ("snapshots_bytes", "backup_bytes", "remote_bytes")}})
save_state(state)
print(f"Atlas health: issues={len(issues)} notifications_failed={len(failed_notifications)} "
f"pool={measurements.get('pool_capacity_percent', 'unknown')}% "
f"remote={measurements.get('remote_capacity_percent', 'unknown')}% "
f"snapshots={measurements.get('snapshots_bytes', 'unknown')} bytes "
f"backup={measurements.get('backup_bytes', 'unknown')} bytes", flush=True)
return 1 if failed_notifications else 0
if __name__ == "__main__":
try:
sys.exit(main())
except (OSError, RuntimeError, ValueError, subprocess.TimeoutExpired) as error:
print(f"Atlas health monitor failed: {error}", file=sys.stderr)
sys.exit(1)

View File

@@ -28,6 +28,18 @@
name: smb name: smb
state: restarted state: restarted
- name: Restart Atlas Borg timers
ansible.builtin.systemd:
name: "{{ item }}"
state: restarted
daemon_reload: true
loop:
- atlas-borg-backup.timer
- atlas-borg-check.timer
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Restart Atlas media Quadlets - name: Restart Atlas media Quadlets
ansible.builtin.systemd: ansible.builtin.systemd:
name: "{{ item }}" name: "{{ item }}"

View File

@@ -0,0 +1,538 @@
---
- name: Validate Atlas Borg backup configuration
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.assert:
that:
- atlas_manage_storage | bool
- atlas_zfs_pool != 'CHANGEME_ZFS_POOL'
- atlas_mount_root.startswith('/')
- atlas_borg_username is match('^[a-z_][a-z0-9_-]*$')
- atlas_borg_group is match('^[a-z_][a-z0-9_-]*$')
- atlas_borg_username not in ['root', atlas_admin_username]
- atlas_borg_group != 'wheel'
- atlas_borg_home.startswith('/var/lib/')
- atlas_borg_repository_host is match('^[A-Za-z0-9.-]+$')
- atlas_borg_repository_user is match('^[A-Za-z0-9_-]+$')
- atlas_borg_repository_port | int > 0
- atlas_borg_repository_port | int < 65536
- atlas_borg_repository_path is match('^\./[A-Za-z0-9][A-Za-z0-9._/-]*$')
- "'/../' not in ('/' ~ atlas_borg_repository_path ~ '/')"
- atlas_borg_remote_path is match('^borg-[0-9]+\.[0-9]+$')
- atlas_borg_host_key.startswith(
'[' ~ atlas_borg_repository_host ~ ']:' ~ (atlas_borg_repository_port | string) ~ ' ssh-ed25519 '
)
- atlas_borg_ssh_private_key_path.startswith('/etc/atlas-borg/')
- atlas_borg_known_hosts_path.startswith('/etc/atlas-borg/')
- atlas_borg_passphrase_path.startswith('/etc/atlas-borg/')
- atlas_borg_ssh_wrapper_path.startswith('/usr/local/libexec/')
- atlas_borg_encryption_mode == 'repokey'
- atlas_borg_archive_prefix is match('^[a-z0-9][a-z0-9_-]*$')
- atlas_borg_snapshot_prefix is match('^[a-z0-9][a-z0-9_-]*$')
- atlas_borg_keep_daily | int > 0
- atlas_borg_keep_weekly | int > 0
- atlas_borg_keep_monthly | int > 0
fail_msg: >-
Atlas Borg needs a safe relative repository path, a pinned ED25519 host
key, positive retention counts, and valid dedicated SSH settings.
when: atlas_manage_borg_backup | bool
- name: Create the Atlas Borg system group
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.group:
name: "{{ atlas_borg_group }}"
system: true
state: present
when: atlas_manage_borg_backup | bool
- name: Create the least-privilege Atlas Borg account
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.user:
name: "{{ atlas_borg_username }}"
group: "{{ atlas_borg_group }}"
groups: []
append: false
comment: Atlas Borg backup service
home: "{{ atlas_borg_home }}"
create_home: false
shell: /sbin/nologin
password_lock: true
system: true
state: present
when: atlas_manage_borg_backup | bool
- name: Read Atlas Borg account group membership
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.command:
argv:
- id
- -nG
- "{{ atlas_borg_username }}"
register: atlas_borg_account_groups
changed_when: false
check_mode: false
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Require the Atlas Borg account to have no supplementary groups
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.assert:
that:
- atlas_borg_account_groups.stdout.split() == [atlas_borg_group]
fail_msg: >-
The Atlas Borg service account must belong only to its private primary
group and must never receive wheel or other supplementary membership.
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Validate Atlas Borg systemd calendars
tags: [atlas, storage, backup, borg]
ansible.builtin.command:
argv:
- systemd-analyze
- calendar
- "{{ item }}"
loop:
- "{{ atlas_borg_backup_calendar }}"
- "{{ atlas_borg_check_calendar }}"
changed_when: false
check_mode: false
when: atlas_manage_borg_backup | bool
- name: Create Atlas Borg configuration directory
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.file:
path: /etc/atlas-borg
state: directory
owner: root
group: "{{ atlas_borg_group }}"
mode: "0750"
when: atlas_manage_borg_backup | bool
- name: Generate the dedicated Atlas Borg SSH identity
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.command:
argv:
- ssh-keygen
- -q
- -t
- ed25519
- -N
- ""
- -C
- atlas-borg@atlas
- -f
- "{{ atlas_borg_ssh_private_key_path }}"
creates: "{{ atlas_borg_ssh_private_key_path }}"
when: atlas_manage_borg_backup | bool
- name: Protect the Atlas Borg private SSH identity
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.file:
path: "{{ atlas_borg_ssh_private_key_path }}"
owner: "{{ atlas_borg_username }}"
group: "{{ atlas_borg_group }}"
mode: "0600"
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Set permissions on the Atlas Borg public SSH identity
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.file:
path: "{{ atlas_borg_ssh_private_key_path }}.pub"
owner: "{{ atlas_borg_username }}"
group: "{{ atlas_borg_group }}"
mode: "0644"
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Read the dedicated Atlas Borg public SSH identity
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.slurp:
src: "{{ atlas_borg_ssh_private_key_path }}.pub"
register: atlas_borg_public_key
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Report the public SSH identity to install in the Hetzner sub-account
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.debug:
msg: "{{ atlas_borg_public_key.content | b64decode | trim }}"
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Pin the Hetzner Storage Box SSH host key
tags: [atlas, storage, backup, borg]
ansible.builtin.copy:
content: "{{ atlas_borg_host_key }}\n"
dest: "{{ atlas_borg_known_hosts_path }}"
owner: "{{ atlas_borg_username }}"
group: "{{ atlas_borg_group }}"
mode: "0600"
when: atlas_manage_borg_backup | bool
- name: Create Atlas Borg state directories
tags: [atlas, storage, backup, borg]
ansible.builtin.file:
path: "{{ item }}"
state: directory
owner: "{{ atlas_borg_username }}"
group: "{{ atlas_borg_group }}"
mode: "0700"
loop:
- "{{ atlas_borg_config_dir }}"
- "{{ atlas_borg_cache_dir }}"
when: atlas_manage_borg_backup | bool
- name: Create the shared Atlas Borg operation lock
tags: [atlas, storage, backup, borg]
ansible.builtin.copy:
content: ""
dest: "{{ atlas_borg_lock_path }}"
owner: "{{ atlas_borg_username }}"
group: "{{ atlas_borg_group }}"
mode: "0600"
force: false
when: atlas_manage_borg_backup | bool
- name: Require the Atlas Borg encryption passphrase from Vault
tags: [atlas, storage, backup, borg]
ansible.builtin.assert:
that:
- atlas_borg_passphrase | length >= 20
fail_msg: >-
Define vault_atlas_borg_passphrase with a strong unique value in the
encrypted Vault before activating the Borg repository.
no_log: true
when: atlas_manage_borg_backup | bool
- name: Install the Atlas Borg passphrase
tags: [atlas, storage, backup, borg]
ansible.builtin.copy:
content: "{{ atlas_borg_passphrase }}\n"
dest: "{{ atlas_borg_passphrase_path }}"
owner: "{{ atlas_borg_username }}"
group: "{{ atlas_borg_group }}"
mode: "0600"
diff: false
no_log: true
when: atlas_manage_borg_backup | bool
- name: Install the Atlas Borg backup helper
tags: [atlas, storage, backup, borg, borg_logging]
ansible.builtin.template:
src: atlas-borg-backup.sh.j2
dest: /usr/local/sbin/atlas-borg-backup
owner: root
group: root
mode: "0750"
when: atlas_manage_borg_backup | bool
- name: Install the Atlas Borg snapshot cleanup helper
tags: [atlas, storage, backup, borg, borg_logging]
ansible.builtin.template:
src: atlas-borg-snapshot-cleanup.sh.j2
dest: /usr/local/sbin/atlas-borg-snapshot-cleanup
owner: root
group: root
mode: "0750"
when: atlas_manage_borg_backup | bool
- name: Install the Atlas Borg check helper
tags: [atlas, storage, backup, borg]
ansible.builtin.template:
src: atlas-borg-check.sh.j2
dest: /usr/local/sbin/atlas-borg-check
owner: root
group: "{{ atlas_borg_group }}"
mode: "0750"
when: atlas_manage_borg_backup | bool
- name: Create the local libexec directory for the Atlas Borg SSH wrapper
tags: [atlas, storage, backup, borg, borg_logging]
ansible.builtin.file:
path: "{{ atlas_borg_ssh_wrapper_path | dirname }}"
state: directory
owner: root
group: root
mode: "0755"
when: atlas_manage_borg_backup | bool
- name: Install the Atlas Borg progress formatter
tags: [atlas, storage, backup, borg, borg_logging]
ansible.builtin.copy:
src: atlas-borg-progress.py
dest: /usr/local/libexec/atlas-borg-progress
owner: root
group: root
mode: "0755"
when: atlas_manage_borg_backup | bool
- name: Install the capability-dropping Atlas Borg SSH wrapper
tags: [atlas, storage, backup, borg]
ansible.builtin.template:
src: atlas-borg-ssh.sh.j2
dest: "{{ atlas_borg_ssh_wrapper_path }}"
owner: root
group: root
mode: "0755"
when: atlas_manage_borg_backup | bool
- name: Install Atlas Borg systemd units
tags: [atlas, storage, backup, borg, borg_logging]
ansible.builtin.template:
src: "{{ item }}.j2"
dest: "/etc/systemd/system/{{ item }}"
owner: root
group: root
mode: "0644"
loop:
- atlas-borg-backup.service
- atlas-borg-backup.timer
- atlas-borg-check.service
- atlas-borg-check.timer
notify: Restart Atlas Borg timers
when: atlas_manage_borg_backup | bool
- name: Verify dedicated SSH access to the Hetzner Storage Box
tags: [atlas, storage, backup, borg]
ansible.builtin.command:
argv:
- ssh
- -T
- -i
- "{{ atlas_borg_ssh_private_key_path }}"
- -p
- "{{ atlas_borg_repository_port | string }}"
- -o
- BatchMode=yes
- -o
- IdentitiesOnly=yes
- -o
- StrictHostKeyChecking=yes
- -o
- "UserKnownHostsFile={{ atlas_borg_known_hosts_path }}"
- "{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}"
- pwd
register: atlas_borg_ssh_probe
become: true
become_user: "{{ atlas_borg_username }}"
changed_when: false
failed_when: false
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Require the dedicated public key on the Hetzner sub-account
tags: [atlas, storage, backup, borg]
ansible.builtin.assert:
that:
- atlas_borg_ssh_probe.rc == 0
fail_msg: >-
Install the reported Atlas Borg public key in the Hetzner sub-account
before rerunning the Borg tasks. Password authentication is never used.
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Probe the remote Atlas Borg repository path
tags: [atlas, storage, backup, borg]
ansible.builtin.command:
argv:
- ssh
- -T
- -i
- "{{ atlas_borg_ssh_private_key_path }}"
- -p
- "{{ atlas_borg_repository_port | string }}"
- -o
- BatchMode=yes
- -o
- IdentitiesOnly=yes
- -o
- StrictHostKeyChecking=yes
- -o
- "UserKnownHostsFile={{ atlas_borg_known_hosts_path }}"
- "{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}"
- stat
- "{{ atlas_borg_repository_path }}"
register: atlas_borg_repository_path_probe
become: true
become_user: "{{ atlas_borg_username }}"
changed_when: false
failed_when: false
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Probe the Atlas Borg repository
tags: [atlas, storage, backup, borg]
ansible.builtin.command:
argv:
- /usr/bin/borg
- --remote-path
- "{{ atlas_borg_remote_path }}"
- info
- >-
ssh://{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}:
{{- atlas_borg_repository_port }}/{{ atlas_borg_repository_path }}
environment:
BORG_CACHE_DIR: "{{ atlas_borg_cache_dir }}"
BORG_CONFIG_DIR: "{{ atlas_borg_config_dir }}"
BORG_PASSCOMMAND: "cat {{ atlas_borg_passphrase_path }}"
BORG_RSH: "{{ atlas_borg_ssh_wrapper_path }}"
register: atlas_borg_repository_probe
become: true
become_user: "{{ atlas_borg_username }}"
changed_when: false
failed_when: false
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- atlas_borg_repository_path_probe.rc == 0
- name: Reject an existing path that is not the configured Borg repository
tags: [atlas, storage, backup, borg]
ansible.builtin.assert:
that:
- atlas_borg_repository_probe.rc == 0
fail_msg: >-
The remote repository path already exists but Borg could not open it.
Refusing to initialize over existing data; verify the path, passphrase,
and repository state manually.
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- atlas_borg_repository_path_probe.rc == 0
- name: Initialize the encrypted Atlas Borg repository
tags: [atlas, storage, backup, borg]
ansible.builtin.command:
argv:
- /usr/bin/borg
- --remote-path
- "{{ atlas_borg_remote_path }}"
- init
- --encryption
- "{{ atlas_borg_encryption_mode }}"
- >-
ssh://{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}:
{{- atlas_borg_repository_port }}/{{ atlas_borg_repository_path }}
environment:
BORG_CACHE_DIR: "{{ atlas_borg_cache_dir }}"
BORG_CONFIG_DIR: "{{ atlas_borg_config_dir }}"
BORG_PASSCOMMAND: "cat {{ atlas_borg_passphrase_path }}"
BORG_RSH: "{{ atlas_borg_ssh_wrapper_path }}"
become: true
become_user: "{{ atlas_borg_username }}"
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- atlas_borg_repository_path_probe.rc != 0
- name: Verify the encrypted Atlas Borg repository
tags: [atlas, storage, backup, borg]
ansible.builtin.command:
argv:
- /usr/bin/borg
- --remote-path
- "{{ atlas_borg_remote_path }}"
- info
- >-
ssh://{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}:
{{- atlas_borg_repository_port }}/{{ atlas_borg_repository_path }}
environment:
BORG_CACHE_DIR: "{{ atlas_borg_cache_dir }}"
BORG_CONFIG_DIR: "{{ atlas_borg_config_dir }}"
BORG_PASSCOMMAND: "cat {{ atlas_borg_passphrase_path }}"
BORG_RSH: "{{ atlas_borg_ssh_wrapper_path }}"
become: true
become_user: "{{ atlas_borg_username }}"
changed_when: false
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Check for the local Atlas Borg recovery-key export
tags: [atlas, storage, backup, borg]
ansible.builtin.stat:
path: "{{ atlas_borg_recovery_export_path }}"
register: atlas_borg_recovery_export
delegate_to: localhost
become: false
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Export the Atlas Borg recovery key for offline preservation
tags: [atlas, storage, backup, borg]
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- not atlas_borg_recovery_export.stat.exists
no_log: true
block:
- name: Create the local recovery-material directory
ansible.builtin.file:
path: "{{ atlas_borg_recovery_export_path | dirname }}"
state: directory
mode: "0700"
delegate_to: localhost
become: false
- name: Export the encrypted Borg repository key on Atlas
ansible.builtin.command:
argv:
- /usr/bin/borg
- --remote-path
- "{{ atlas_borg_remote_path }}"
- key
- export
- >-
ssh://{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}:
{{- atlas_borg_repository_port }}/{{ atlas_borg_repository_path }}
- "{{ atlas_borg_config_dir }}/atlas-borg-repokey.export"
environment:
BORG_CACHE_DIR: "{{ atlas_borg_cache_dir }}"
BORG_CONFIG_DIR: "{{ atlas_borg_config_dir }}"
BORG_PASSCOMMAND: "cat {{ atlas_borg_passphrase_path }}"
BORG_RSH: "{{ atlas_borg_ssh_wrapper_path }}"
become: true
become_user: "{{ atlas_borg_username }}"
- name: Fetch the encrypted Borg recovery key from Atlas
ansible.builtin.fetch:
src: "{{ atlas_borg_config_dir }}/atlas-borg-repokey.export"
dest: "{{ atlas_borg_recovery_export_path }}"
flat: true
- name: Protect the local Borg recovery-key export
ansible.builtin.file:
path: "{{ atlas_borg_recovery_export_path }}"
mode: "0600"
delegate_to: localhost
become: false
always:
- name: Remove the temporary recovery-key export from Atlas
ansible.builtin.file:
path: "{{ atlas_borg_config_dir }}/atlas-borg-repokey.export"
state: absent
- name: Enable Atlas Borg backup and check timers
tags: [atlas, storage, backup, borg]
ansible.builtin.systemd:
name: "{{ item }}"
enabled: true
state: started
daemon_reload: true
loop:
- atlas-borg-backup.timer
- atlas-borg-check.timer
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode

View File

@@ -14,6 +14,21 @@
- name: Import Atlas storage tasks - name: Import Atlas storage tasks
ansible.builtin.import_tasks: storage.yml ansible.builtin.import_tasks: storage.yml
- name: Import Atlas ZFS maintenance tasks
ansible.builtin.import_tasks: zfs_maintenance.yml
- name: Import Atlas Borg backup tasks
ansible.builtin.import_tasks: borg_backup.yml
- name: Import Atlas offline USB backup tasks
ansible.builtin.import_tasks: usb_backup.yml
- name: Import Atlas health monitoring tasks
ansible.builtin.import_tasks: monitoring.yml
- name: Import Atlas post-restore SELinux relabeling tasks
ansible.builtin.import_tasks: restorecon.yml
- name: Import Atlas file sharing tasks - name: Import Atlas file sharing tasks
ansible.builtin.import_tasks: sharing.yml ansible.builtin.import_tasks: sharing.yml

View File

@@ -0,0 +1,201 @@
---
- name: Validate Atlas health monitoring policy
tags: [atlas, monitoring]
ansible.builtin.assert:
that:
- atlas_manage_storage | bool
- atlas_zfs_pool != 'CHANGEME_ZFS_POOL'
- atlas_monitor_calendar | length > 0
- atlas_monitor_smart_devices | length > 0
- atlas_monitor_timers | length > 0
- atlas_monitor_failure_units | length > 0
- atlas_monitor_remote_capacity.user == atlas_borg_repository_user
- atlas_monitor_remote_capacity.host == atlas_borg_repository_host
- atlas_monitor_remote_capacity.run_as == atlas_borg_username
- atlas_monitor_remote_capacity.ssh_wrapper == atlas_borg_ssh_wrapper_path
- >-
0 < atlas_monitor_remote_capacity.warning_percent | int
< atlas_monitor_remote_capacity.critical_percent | int < 100
- atlas_monitor_remote_capacity.growth_warning_gib_day | int > 0
- atlas_monitor_notifier.startswith('/opt/45drives/houston/')
- 0 < atlas_monitor_pool_warning_percent | int < atlas_monitor_pool_critical_percent | int < 100
- 0 < atlas_monitor_root_warning_percent | int < atlas_monitor_root_critical_percent | int < 100
- 0 < atlas_monitor_snapshot_warning_percent | int < atlas_monitor_snapshot_critical_percent | int < 100
- atlas_monitor_snapshot_growth_warning_gib_day | int > 0
- atlas_monitor_backup_growth_warning_gib_day | int > 0
- 0 < atlas_monitor_cpu_warning_c | int < atlas_monitor_cpu_critical_c | int
- atlas_monitor_borg_max_runtime_days | int > 0
fail_msg: >-
Atlas health monitoring needs real devices, job units, a valid calendar,
positive ordered thresholds, and the existing Houston notifier.
when: atlas_manage_monitoring | bool
- name: Validate monitored Atlas SMART devices
tags: [atlas, monitoring]
ansible.builtin.assert:
that:
- item.name is match('^[a-z0-9][a-z0-9_-]*$')
- item.path.startswith('/dev/disk/by-id/')
- 0 < item.warning_c | int < item.critical_c | int
fail_msg: "Every monitored disk needs a stable by-id path and ordered temperature thresholds."
loop: "{{ atlas_monitor_smart_devices }}"
loop_control:
label: "{{ item.name }}"
when: atlas_manage_monitoring | bool
- name: Validate monitored Atlas timer names and age thresholds
tags: [atlas, monitoring]
ansible.builtin.assert:
that:
- item.name is match('^[a-zA-Z0-9@_.-]+\\.timer$')
- item.max_age_hours | int >= 0
loop: "{{ atlas_monitor_timers }}"
loop_control:
label: "{{ item.name }}"
when: atlas_manage_monitoring | bool
- name: Validate monitored Atlas failure unit names
tags: [atlas, monitoring]
ansible.builtin.assert:
that:
- item is match('^[a-zA-Z0-9@_.-]+\\.service$')
loop: "{{ atlas_monitor_failure_units }}"
when: atlas_manage_monitoring | bool
- name: Validate Atlas health monitor calendar
tags: [atlas, monitoring]
ansible.builtin.command:
argv: [systemd-analyze, calendar, "{{ atlas_monitor_calendar }}"]
changed_when: false
check_mode: false
when: atlas_manage_monitoring | bool
- name: Install SMART tooling for Atlas health checks
tags: [atlas, monitoring, packages]
ansible.builtin.dnf:
name: smartmontools
state: present
when: atlas_manage_monitoring | bool
- name: Inspect the existing 45Drives notifier for monitoring
tags: [atlas, monitoring]
ansible.builtin.stat:
path: "{{ atlas_monitor_notifier }}"
register: atlas_monitor_notifier_file
when: atlas_manage_monitoring | bool
- name: Require the existing 45Drives notifier for monitoring
tags: [atlas, monitoring]
ansible.builtin.assert:
that:
- atlas_monitor_notifier_file.stat.executable | default(false)
fail_msg: "The existing 45Drives Houston notifier must be executable."
when: atlas_manage_monitoring | bool
- name: Create private Atlas health monitor state directory
tags: [atlas, monitoring]
ansible.builtin.file:
path: /var/lib/atlas-health-monitor
state: directory
owner: root
group: root
mode: "0700"
when: atlas_manage_monitoring | bool
- name: Install Atlas health monitor configuration
tags: [atlas, monitoring]
ansible.builtin.template:
src: atlas-health-monitor.json.j2
dest: /etc/atlas-health-monitor.json
owner: root
group: root
mode: "0600"
when: atlas_manage_monitoring | bool
- name: Install Atlas health monitor helper
tags: [atlas, monitoring]
ansible.builtin.copy:
src: atlas-health-monitor.py
dest: /usr/local/libexec/atlas-health-monitor
owner: root
group: root
mode: "0750"
when: atlas_manage_monitoring | bool
- name: Install Atlas health monitoring units
tags: [atlas, monitoring]
ansible.builtin.template:
src: "{{ item }}.j2"
dest: "/etc/systemd/system/{{ item }}"
owner: root
group: root
mode: "0644"
loop:
- atlas-health-monitor.service
- atlas-health-monitor.timer
- atlas-monitor-failure@.service
when: atlas_manage_monitoring | bool
- name: Create failure hook directories for monitored Atlas jobs
tags: [atlas, monitoring]
ansible.builtin.file:
path: "/etc/systemd/system/{{ item }}.d"
state: directory
owner: root
group: root
mode: "0755"
loop: "{{ atlas_monitor_failure_units }}"
when: atlas_manage_monitoring | bool
- name: Notify 45Drives Alerts when an Atlas job fails
tags: [atlas, monitoring]
ansible.builtin.template:
src: atlas-monitor-failure.conf.j2
dest: "/etc/systemd/system/{{ item }}.d/atlas-monitor.conf"
owner: root
group: root
mode: "0644"
loop: "{{ atlas_monitor_failure_units }}"
when: atlas_manage_monitoring | bool
- name: Reload systemd after installing Atlas monitoring
tags: [atlas, monitoring]
ansible.builtin.systemd:
daemon_reload: true
when:
- atlas_manage_monitoring | bool
- not ansible_check_mode
- name: Enable the Atlas health monitoring timer
tags: [atlas, monitoring]
ansible.builtin.systemd:
name: atlas-health-monitor.timer
enabled: true
state: started
when:
- atlas_manage_monitoring | bool
- not ansible_check_mode
- name: Validate the deployed Atlas health monitoring units
tags: [atlas, monitoring]
ansible.builtin.command:
argv:
- systemd-analyze
- verify
- atlas-health-monitor.service
- atlas-health-monitor.timer
- atlas-monitor-failure@.service
changed_when: false
when:
- atlas_manage_monitoring | bool
- not ansible_check_mode
- name: Probe Atlas health without sending notifications
tags: [atlas, monitoring]
ansible.builtin.command:
argv: [/usr/local/libexec/atlas-health-monitor, --dry-run]
register: atlas_monitor_dry_run
changed_when: false
when:
- atlas_manage_monitoring | bool
- not ansible_check_mode

View File

@@ -0,0 +1,27 @@
---
- name: Validate requested Atlas post-restore relabel paths
tags: [atlas, restorecon, recovery]
ansible.builtin.assert:
that:
- item is string
- item.startswith(atlas_mount_root ~ '/')
- item != atlas_mount_root
fail_msg: >-
Post-restore relabeling accepts only explicit paths below the Atlas pool
mount root. Do not relabel the whole pool during routine provisioning.
loop: "{{ atlas_restorecon_paths }}"
when: atlas_restorecon_paths | length > 0
- name: Restore SELinux labels on explicitly restored Atlas paths
tags: [atlas, restorecon, recovery]
ansible.builtin.command:
argv:
- restorecon
- -RFv
- "{{ item }}"
register: atlas_restorecon_result
changed_when: atlas_restorecon_result.stdout | length > 0
loop: "{{ atlas_restorecon_paths }}"
when:
- atlas_restorecon_paths | length > 0
- not ansible_check_mode

View File

@@ -3,12 +3,6 @@
tags: [atlas, storage] tags: [atlas, storage]
ansible.builtin.set_fact: ansible.builtin.set_fact:
atlas_zfs_datasets: atlas_zfs_datasets:
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_work }}"
mountpoint: "{{ atlas_work_mountpoint }}"
owner: "{{ atlas_admin_username }}"
group: "{{ atlas_admin_group }}"
properties:
mountpoint: "{{ atlas_work_mountpoint }}"
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_archive }}" - name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_archive }}"
mountpoint: "{{ atlas_archive_mountpoint }}" mountpoint: "{{ atlas_archive_mountpoint }}"
owner: "{{ atlas_admin_username }}" owner: "{{ atlas_admin_username }}"
@@ -16,6 +10,13 @@
properties: properties:
compression: zstd compression: zstd
mountpoint: "{{ atlas_archive_mountpoint }}" mountpoint: "{{ atlas_archive_mountpoint }}"
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_services }}"
mountpoint: "{{ atlas_services_mountpoint }}"
owner: "{{ atlas_admin_username }}"
group: "{{ atlas_admin_group }}"
properties:
compression: zstd
mountpoint: "{{ atlas_services_mountpoint }}"
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_app_data }}" - name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_app_data }}"
mountpoint: "{{ atlas_app_data_mountpoint }}" mountpoint: "{{ atlas_app_data_mountpoint }}"
owner: "{{ atlas_admin_username }}" owner: "{{ atlas_admin_username }}"
@@ -51,11 +52,27 @@
properties: properties:
compression: lz4 compression: lz4
mountpoint: "{{ atlas_music_mountpoint }}" mountpoint: "{{ atlas_music_mountpoint }}"
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_backup }}"
mountpoint: "{{ atlas_backup_mountpoint }}"
owner: "{{ atlas_admin_username }}"
group: "{{ atlas_admin_group }}"
properties:
compression: lz4
mountpoint: "{{ atlas_backup_mountpoint }}"
reservation: "{{ atlas_zfs_backup_reservation }}"
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_host_backups }}"
mountpoint: "{{ atlas_host_backups_mountpoint }}"
owner: "{{ atlas_admin_username }}"
group: "{{ atlas_admin_group }}"
properties:
compression: lz4
mountpoint: "{{ atlas_host_backups_mountpoint }}"
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_backup_prometheus }}" - name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_backup_prometheus }}"
mountpoint: "{{ atlas_backup_prometheus_mountpoint }}" mountpoint: "{{ atlas_backup_prometheus_mountpoint }}"
owner: "{{ atlas_admin_username }}" owner: "{{ atlas_admin_username }}"
group: "{{ atlas_admin_group }}" group: "{{ atlas_admin_group }}"
properties: properties:
compression: lz4
mountpoint: "{{ atlas_backup_prometheus_mountpoint }}" mountpoint: "{{ atlas_backup_prometheus_mountpoint }}"
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_photobook }}" - name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_photobook }}"
mountpoint: "{{ atlas_photobook_mountpoint }}" mountpoint: "{{ atlas_photobook_mountpoint }}"
@@ -64,21 +81,6 @@
properties: properties:
compression: lz4 compression: lz4
mountpoint: "{{ atlas_photobook_mountpoint }}" mountpoint: "{{ atlas_photobook_mountpoint }}"
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_backups }}"
mountpoint: "{{ atlas_backups_mountpoint }}"
owner: "{{ atlas_admin_username }}"
group: "{{ atlas_admin_group }}"
properties:
compression: lz4
mountpoint: "{{ atlas_backups_mountpoint }}"
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_service_backups }}"
mountpoint: "{{ atlas_service_backups_mountpoint }}"
owner: "{{ atlas_admin_username }}"
group: "{{ atlas_admin_group }}"
properties:
compression: lz4
mountpoint: "{{ atlas_service_backups_mountpoint }}"
refreservation: "{{ atlas_zfs_service_backups_refreservation }}"
- name: Require completed Atlas storage placeholders - name: Require completed Atlas storage placeholders
tags: [atlas, storage] tags: [atlas, storage]

View File

@@ -0,0 +1,145 @@
---
- name: Validate Atlas offline USB backup configuration
tags: [atlas, storage, backup, usb_backup]
ansible.builtin.assert:
that:
- atlas_manage_storage | bool
- atlas_zfs_pool != 'CHANGEME_ZFS_POOL'
- atlas_mount_root.startswith('/')
- atlas_usb_backup_luks_uuid is match('^[0-9a-fA-F]{8}(-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12}$')
- atlas_usb_backup_fs_uuid is match('^[0-9a-fA-F]{8}(-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12}$')
- atlas_usb_backup_luks_uuid != atlas_usb_backup_fs_uuid
- atlas_usb_backup_mapper_name is match('^[a-z][a-z0-9_-]*$')
- atlas_usb_backup_min_free_bytes | int > 0
- atlas_usb_backup_snapshot_prefix is match('^[a-z0-9][a-z0-9_-]*$')
- atlas_usb_backup_snapshot_prefix != atlas_borg_snapshot_prefix
- atlas_usb_backup_snapshot_prefix != atlas_zfs_snapshot_prefix
fail_msg: >-
The manual Atlas USB backup needs verified LUKS and ext4 UUIDs, a safe
mapper name, positive free-space reserve, and a unique snapshot prefix.
when: atlas_manage_usb_backup | bool
- name: Install rsync for the Atlas offline USB backup
tags: [atlas, storage, backup, usb_backup]
ansible.builtin.dnf:
name: rsync
state: present
when: atlas_manage_usb_backup | bool
- name: Install the manual Atlas offline USB backup helper
tags: [atlas, storage, backup, usb_backup]
ansible.builtin.template:
src: atlas-usb-backup.sh.j2
dest: /usr/local/sbin/atlas-usb-backup
owner: root
group: root
mode: "0750"
when: atlas_manage_usb_backup | bool
- name: Install the Atlas USB snapshot cleanup helper
tags: [atlas, storage, backup, usb_backup]
ansible.builtin.template:
src: atlas-usb-snapshot-cleanup.sh.j2
dest: /usr/local/sbin/atlas-usb-snapshot-cleanup
owner: root
group: root
mode: "0750"
when: atlas_manage_usb_backup | bool
- name: Install the manual Atlas offline USB backup service
tags: [atlas, storage, backup, usb_backup]
ansible.builtin.template:
src: atlas-usb-backup.service.j2
dest: /etc/systemd/system/atlas-usb-backup.service
owner: root
group: root
mode: "0644"
when: atlas_manage_usb_backup | bool
- name: Reload systemd for the Atlas offline USB backup service
tags: [atlas, storage, backup, usb_backup]
ansible.builtin.systemd:
daemon_reload: true
when:
- atlas_manage_usb_backup | bool
- not ansible_check_mode
- name: Validate the 45Drives Atlas USB reminder configuration
tags: [atlas, backup, usb_reminder]
ansible.builtin.assert:
that:
- atlas_manage_usb_backup | bool
- atlas_usb_reminder_calendar | length > 0
- atlas_usb_reminder_notifier.startswith('/opt/45drives/houston/')
fail_msg: >-
Enable the manual USB backup and declare a systemd calendar before
enabling its 45Drives Alerts reminder.
when: atlas_manage_usb_reminder | bool
- name: Validate the Atlas USB reminder calendar
tags: [atlas, backup, usb_reminder]
ansible.builtin.command:
argv:
- systemd-analyze
- calendar
- "{{ atlas_usb_reminder_calendar }}"
changed_when: false
check_mode: false
when: atlas_manage_usb_reminder | bool
- name: Inspect the existing 45Drives notifier
tags: [atlas, backup, usb_reminder]
ansible.builtin.stat:
path: "{{ atlas_usb_reminder_notifier }}"
register: atlas_usb_reminder_notifier_file
when: atlas_manage_usb_reminder | bool
- name: Require the configured 45Drives notifier for USB reminders
tags: [atlas, backup, usb_reminder]
ansible.builtin.assert:
that:
- atlas_usb_reminder_notifier_file.stat.executable | default(false)
fail_msg: >-
The existing 45Drives Houston notifier must be executable.
when: atlas_manage_usb_reminder | bool
- name: Install the 45Drives Atlas USB reminder helper
tags: [atlas, backup, usb_reminder]
ansible.builtin.template:
src: atlas-usb-reminder.py.j2
dest: /usr/local/libexec/atlas-usb-reminder
owner: root
group: root
mode: "0750"
when: atlas_manage_usb_reminder | bool
- name: Install the 45Drives Atlas USB reminder service
tags: [atlas, backup, usb_reminder]
ansible.builtin.template:
src: atlas-usb-reminder.service.j2
dest: /etc/systemd/system/atlas-usb-reminder.service
owner: root
group: root
mode: "0644"
when: atlas_manage_usb_reminder | bool
- name: Install the 45Drives Atlas USB reminder timer
tags: [atlas, backup, usb_reminder]
ansible.builtin.template:
src: atlas-usb-reminder.timer.j2
dest: /etc/systemd/system/atlas-usb-reminder.timer
owner: root
group: root
mode: "0644"
when: atlas_manage_usb_reminder | bool
- name: Enable only the Atlas USB notification reminder timer
tags: [atlas, backup, usb_reminder]
ansible.builtin.systemd:
name: atlas-usb-reminder.timer
enabled: true
state: started
daemon_reload: true
when:
- atlas_manage_usb_reminder | bool
- not ansible_check_mode

View File

@@ -0,0 +1,175 @@
---
- name: Validate Atlas ZFS snapshot policy
tags: [atlas, storage, snapshots]
ansible.builtin.assert:
that:
- atlas_manage_storage | bool
- atlas_zfs_pool != 'CHANGEME_ZFS_POOL'
- atlas_zfs_snapshot_prefix is match('^[a-z0-9][a-z0-9_-]*$')
- atlas_zfs_snapshot_policies | length > 0
- >-
(atlas_zfs_snapshot_policies | map(attribute='name') | unique | list | length)
== (atlas_zfs_snapshot_policies | length)
fail_msg: >-
Enable Atlas storage and declare a non-empty snapshot policy with a safe
prefix and unique policy names before managing automatic snapshots.
when: atlas_manage_zfs_snapshots | bool
- name: Validate Atlas ZFS snapshot policy entries
tags: [atlas, storage, snapshots]
ansible.builtin.assert:
that:
- item.name is match('^[a-z][a-z0-9_-]*$')
- item.keep | int > 0
- item.calendar | length > 0
fail_msg: >-
Every Atlas snapshot policy needs a safe name, a positive retention
count, and a systemd calendar expression.
loop: "{{ atlas_zfs_snapshot_policies }}"
loop_control:
label: "{{ item.name | default('unnamed') }}"
when: atlas_manage_zfs_snapshots | bool
- name: Validate Atlas ZFS snapshot calendars
tags: [atlas, storage, snapshots]
ansible.builtin.command:
argv:
- systemd-analyze
- calendar
- "{{ item.calendar }}"
loop: "{{ atlas_zfs_snapshot_policies }}"
loop_control:
label: "{{ item.name }}: {{ item.calendar }}"
changed_when: false
check_mode: false
when: atlas_manage_zfs_snapshots | bool
- name: Install Atlas ZFS snapshot and retention helper
tags: [atlas, storage, snapshots]
ansible.builtin.template:
src: atlas-zfs-snapshot.sh.j2
dest: /usr/local/sbin/atlas-zfs-snapshot
owner: root
group: root
mode: "0750"
when: atlas_manage_zfs_snapshots | bool
- name: Install Atlas ZFS snapshot systemd service
tags: [atlas, storage, snapshots]
ansible.builtin.template:
src: atlas-zfs-snapshot@.service.j2
dest: /etc/systemd/system/atlas-zfs-snapshot@.service
owner: root
group: root
mode: "0644"
when: atlas_manage_zfs_snapshots | bool
- name: Install Atlas ZFS snapshot systemd timers
tags: [atlas, storage, snapshots]
ansible.builtin.template:
src: atlas-zfs-snapshot.timer.j2
dest: "/etc/systemd/system/atlas-zfs-snapshot-{{ item.name }}.timer"
owner: root
group: root
mode: "0644"
loop: "{{ atlas_zfs_snapshot_policies }}"
loop_control:
label: "{{ item.name }}"
when: atlas_manage_zfs_snapshots | bool
- name: Enable Atlas ZFS snapshot systemd timers
tags: [atlas, storage, snapshots]
ansible.builtin.systemd:
name: "atlas-zfs-snapshot-{{ item.name }}.timer"
enabled: true
state: started
daemon_reload: true
loop: "{{ atlas_zfs_snapshot_policies }}"
loop_control:
label: "{{ item.name }}"
when:
- atlas_manage_zfs_snapshots | bool
- not ansible_check_mode
- name: Validate Atlas ZFS scrub policy
tags: [atlas, storage, scrub]
ansible.builtin.assert:
that:
- atlas_manage_storage | bool
- atlas_zfs_pool != 'CHANGEME_ZFS_POOL'
- atlas_zfs_scrub_calendar | length > 0
fail_msg: >-
Enable Atlas storage and declare a systemd calendar expression before
managing periodic ZFS scrubs.
when: atlas_manage_zfs_scrub | bool
- name: Validate Atlas ZFS scrub calendar
tags: [atlas, storage, scrub]
ansible.builtin.command:
argv:
- systemd-analyze
- calendar
- "{{ atlas_zfs_scrub_calendar }}"
changed_when: false
check_mode: false
when: atlas_manage_zfs_scrub | bool
- name: Require OpenZFS scrub systemd units
tags: [atlas, storage, scrub]
ansible.builtin.command:
argv:
- systemctl
- cat
- "{{ item }}"
loop:
- "zfs-scrub@{{ atlas_zfs_pool }}.service"
- "zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer"
- "zfs-scrub-weekly@{{ atlas_zfs_pool }}.timer"
loop_control:
label: "{{ item }}"
changed_when: false
check_mode: false
when: atlas_manage_zfs_scrub | bool
- name: Create Atlas ZFS scrub timer override directory
tags: [atlas, storage, scrub]
ansible.builtin.file:
path: "/etc/systemd/system/zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer.d"
state: directory
owner: root
group: root
mode: "0755"
when: atlas_manage_zfs_scrub | bool
- name: Configure Atlas ZFS monthly scrub schedule
tags: [atlas, storage, scrub]
ansible.builtin.template:
src: atlas-zfs-scrub-timer.conf.j2
dest: >-
/etc/systemd/system/zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer.d/override.conf
owner: root
group: root
mode: "0644"
when: atlas_manage_zfs_scrub | bool
- name: Disable the conflicting weekly OpenZFS scrub timer
tags: [atlas, storage, scrub]
ansible.builtin.systemd:
name: "zfs-scrub-weekly@{{ atlas_zfs_pool }}.timer"
enabled: false
state: stopped
daemon_reload: true
when:
- atlas_manage_zfs_scrub | bool
- not ansible_check_mode
- name: Enable the Atlas monthly OpenZFS scrub timer
tags: [atlas, storage, scrub]
ansible.builtin.systemd:
name: "zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer"
enabled: true
state: started
daemon_reload: true
when:
- atlas_manage_zfs_scrub | bool
- not ansible_check_mode

View File

@@ -0,0 +1,41 @@
[Unit]
Description=Back up Atlas ZFS datasets to the encrypted Borg repository
Documentation=man:borg-create(1) man:borg-prune(1) man:borg-compact(1)
Requires=zfs.target
Wants=network-online.target
After=zfs.target network-online.target
StartLimitIntervalSec=6h
StartLimitBurst=3
ConditionFileIsExecutable=/usr/local/sbin/atlas-borg-backup
ConditionPathExists={{ atlas_borg_passphrase_path }}
ConditionPathExists={{ atlas_borg_ssh_private_key_path }}
ConditionPathExists={{ atlas_borg_known_hosts_path }}
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/atlas-borg-backup
ExecStopPost=+/usr/local/sbin/atlas-borg-snapshot-cleanup
User=root
Group=root
UMask=0077
SuccessExitStatus=1
Restart=on-failure
RestartSec=30m
TimeoutStartSec=infinity
RuntimeDirectory=atlas-borg
RuntimeDirectoryMode=0750
Nice=15
IOSchedulingClass=best-effort
IOSchedulingPriority=7
NoNewPrivileges=true
PrivateMounts=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths={{ atlas_borg_cache_dir }} {{ atlas_borg_config_dir }} /run/atlas-borg /run/lock
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictRealtime=true
LockPersonality=true

View File

@@ -0,0 +1,212 @@
#!/usr/bin/env bash
set -Eeuo pipefail
export LC_ALL=C.utf8
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
export BORG_CACHE_DIR={{ atlas_borg_cache_dir | quote }}
export BORG_CONFIG_DIR={{ atlas_borg_config_dir | quote }}
export BORG_PASSCOMMAND={{ ('cat ' ~ atlas_borg_passphrase_path) | quote }}
export BORG_RSH={{ atlas_borg_ssh_wrapper_path | quote }}
readonly pool={{ atlas_zfs_pool | quote }}
readonly mount_root={{ atlas_mount_root | quote }}
readonly repository={{ ('ssh://' ~ atlas_borg_repository_user ~ '@' ~ atlas_borg_repository_host
~ ':' ~ (atlas_borg_repository_port | string) ~ '/' ~ atlas_borg_repository_path) | quote }}
readonly remote_path={{ atlas_borg_remote_path | quote }}
readonly archive_prefix={{ atlas_borg_archive_prefix | quote }}
readonly snapshot_prefix={{ atlas_borg_snapshot_prefix | quote }}
readonly compression={{ atlas_borg_compression | quote }}
readonly stage=/run/atlas-borg/source
readonly snapshot_marker=/run/atlas-borg/snapshot-name
readonly borg_user={{ atlas_borg_username | quote }}
readonly borg_group={{ atlas_borg_group | quote }}
readonly borg_home={{ atlas_borg_home | quote }}
readonly borg_lock={{ atlas_borg_lock_path | quote }}
readonly progress_filter=/usr/local/libexec/atlas-borg-progress
snapshot_name=""
mounted_targets=()
# Invoked through the EXIT trap below.
# shellcheck disable=SC2329
cleanup() {
local status=$?
local cleanup_status=0
local index
local source_mount_failed=false
trap - EXIT HUP INT TERM
set +e
{% raw %}
for ((index = ${#mounted_targets[@]} - 1; index >= 0; index--)); do
{% endraw %}
if mountpoint -q "${mounted_targets[$index]}" && ! umount -R "${mounted_targets[$index]}"; then
printf 'Source snapshot mount cleanup failed: %s\n' "${mounted_targets[$index]}" >&2
source_mount_failed=true
fi
if mountpoint -q "${mounted_targets[$index]}"; then
printf 'Source snapshot mount is still active: %s\n' "${mounted_targets[$index]}" >&2
source_mount_failed=true
else
rmdir -- "${mounted_targets[$index]}" 2>/dev/null || true
fi
done
if [[ "$source_mount_failed" == false ]]; then
if [[ -d "$stage" ]]; then
rmdir -- "$stage" 2>/dev/null || cleanup_status=2
fi
else
cleanup_status=2
printf 'Source bind mount cleanup failed; keeping the snapshot for recovery\n' >&2
fi
if ((status == 0 && cleanup_status != 0)); then
status=$cleanup_status
fi
exit "$status"
}
trap cleanup EXIT
trap 'exit 143' HUP INT TERM
run_as_borg() {
setpriv \
--reuid "$borg_user" \
--regid "$borg_group" \
--clear-groups \
--inh-caps=-all,+dac_read_search \
--ambient-caps=-all,+dac_read_search \
--bounding-set=-all,+dac_read_search \
-- env HOME="$borg_home" USER="$borg_user" LOGNAME="$borg_user" "$@"
}
exec 8>"$borg_lock"
flock 8
exec 9>/run/lock/atlas-zfs-snapshot.lock
zpool list -H -o name "$pool" >/dev/null
rm -rf "$stage"
mkdir -p "$stage"
chown root:"$borg_group" /run/atlas-borg "$stage"
chmod 0750 /run/atlas-borg "$stage"
flock 9
while IFS= read -r stale_snapshot; do
stale_suffix="${stale_snapshot#"${pool}@${snapshot_prefix}-"}"
if [[ "$stale_suffix" =~ ^[0-9]{8}T[0-9]{6}Z$ ]]; then
zfs destroy -r "$stale_snapshot"
printf 'Removed stale Borg source snapshot %s\n' "$stale_snapshot"
fi
done < <(
zfs list -H -t snapshot -o name -r "$pool" |
grep -E "^${pool}@${snapshot_prefix}-[0-9]{8}T[0-9]{6}Z$" || true
)
timestamp="$(date -u +%Y%m%dT%H%M%SZ)"
readonly timestamp
snapshot_name="${snapshot_prefix}-${timestamp}"
readonly snapshot_name
printf '%s\n' "$snapshot_name" >"$snapshot_marker"
zfs snapshot -r "${pool}@${snapshot_name}"
flock -u 9
printf 'Created recursive Borg source snapshot %s@%s\n' "$pool" "$snapshot_name"
while IFS=$'\t' read -r dataset dataset_mountpoint mounted; do
if [[ "$mounted" != yes ]]; then
printf 'Dataset %s is not mounted; refusing an incomplete backup\n' "$dataset" >&2
exit 65
fi
if [[ "$dataset_mountpoint" != "$mount_root" && "$dataset_mountpoint" != "$mount_root/"* ]]; then
printf 'Dataset %s has unexpected mountpoint %s\n' "$dataset" "$dataset_mountpoint" >&2
exit 65
fi
dataset_suffix="${dataset#"$pool"}"
source_path="${dataset_mountpoint}/.zfs/snapshot/${snapshot_name}"
target_path="${stage}${dataset_suffix}"
mkdir -p "$target_path"
mount --bind "$source_path" "$target_path"
mounted_targets+=("$target_path")
mount -o remount,bind,ro "$target_path"
done < <(zfs list -H -o name,mountpoint,mounted -s name -r "$pool")
estimated_source_bytes=0
while IFS=$'\t' read -r source_snapshot logical_bytes; do
if [[ "$source_snapshot" == *"@${snapshot_name}" ]]; then
[[ "$logical_bytes" =~ ^[0-9]+$ ]] || {
printf 'Invalid logical size for Borg source snapshot %s\n' "$source_snapshot" >&2
exit 74
}
estimated_source_bytes=$((estimated_source_bytes + logical_bytes))
fi
done < <(zfs list -H -p -t snapshot -o name,logicalreferenced -r "$pool")
((estimated_source_bytes > 0)) || {
printf 'Could not estimate the Borg source snapshot size\n' >&2
exit 74
}
printf 'Estimated Borg source logical size: %s bytes (ZFS; progress percentage is approximate)\n' \
"$estimated_source_bytes"
archive="${archive_prefix}-${timestamp}"
readonly archive
borg_status=0
printf 'Starting Borg archive %s from snapshot %s@%s\n' "$archive" "$pool" "$snapshot_name"
set +e
(
cd /run/atlas-borg
run_as_borg borg --remote-path "$remote_path" --lock-wait 600 --log-json --progress create \
--show-rc \
--stats \
--checkpoint-interval 900 \
--compression "$compression" \
"${repository}::${archive}" \
source 2>&1
) | /usr/bin/python3 -u "$progress_filter" --estimated-total-bytes "$estimated_source_bytes"
create_pipeline_status=("${PIPESTATUS[@]}")
set -e
create_status=${create_pipeline_status[0]}
if ((create_pipeline_status[1] != 0)); then
printf 'Borg progress logging failed with status %s\n' "${create_pipeline_status[1]}" >&2
exit 2
fi
if ((create_status >= 2)); then
exit "$create_status"
fi
borg_status=$create_status
printf 'Borg archive %s created; applying retention\n' "$archive"
set +e
run_as_borg borg --remote-path "$remote_path" --lock-wait 600 prune \
--show-rc \
--list \
--glob-archives "${archive_prefix}-*" \
--keep-daily {{ atlas_borg_keep_daily | int }} \
--keep-weekly {{ atlas_borg_keep_weekly | int }} \
--keep-monthly {{ atlas_borg_keep_monthly | int }} \
"$repository"
prune_status=$?
set -e
if ((prune_status >= 2)); then
exit "$prune_status"
fi
if ((prune_status > borg_status)); then
borg_status=$prune_status
fi
printf 'Borg retention complete; compacting repository\n'
set +e
run_as_borg borg --remote-path "$remote_path" --lock-wait 600 compact \
--show-rc \
"$repository"
compact_status=$?
set -e
if ((compact_status >= 2)); then
exit "$compact_status"
fi
if ((compact_status > borg_status)); then
borg_status=$compact_status
fi
printf 'Borg backup %s completed with status %s\n' "$archive" "$borg_status"
exit "$borg_status"

View File

@@ -0,0 +1,12 @@
[Unit]
Description=Schedule the encrypted Atlas Borg backup
[Timer]
OnCalendar={{ atlas_borg_backup_calendar }}
Persistent=true
RandomizedDelaySec={{ atlas_borg_randomized_delay }}
AccuracySec=1min
Unit=atlas-borg-backup.service
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,32 @@
[Unit]
Description=Check the encrypted Atlas Borg repository
Documentation=man:borg-check(1)
Wants=network-online.target
After=network-online.target atlas-borg-backup.service
ConditionFileIsExecutable=/usr/local/sbin/atlas-borg-check
ConditionPathExists={{ atlas_borg_passphrase_path }}
ConditionPathExists={{ atlas_borg_ssh_private_key_path }}
ConditionPathExists={{ atlas_borg_known_hosts_path }}
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/atlas-borg-check
User={{ atlas_borg_username }}
Group={{ atlas_borg_group }}
UMask=0077
SuccessExitStatus=1
TimeoutStartSec=infinity
Nice=15
IOSchedulingClass=best-effort
IOSchedulingPriority=7
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths={{ atlas_borg_cache_dir }} {{ atlas_borg_config_dir }}
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictRealtime=true
LockPersonality=true

View File

@@ -0,0 +1,23 @@
#!/usr/bin/env bash
set -Eeuo pipefail
export LC_ALL=C
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
export BORG_CACHE_DIR={{ atlas_borg_cache_dir | quote }}
export BORG_CONFIG_DIR={{ atlas_borg_config_dir | quote }}
export BORG_PASSCOMMAND={{ ('cat ' ~ atlas_borg_passphrase_path) | quote }}
export BORG_RSH={{ atlas_borg_ssh_wrapper_path | quote }}
readonly repository={{ ('ssh://' ~ atlas_borg_repository_user ~ '@' ~ atlas_borg_repository_host
~ ':' ~ (atlas_borg_repository_port | string) ~ '/' ~ atlas_borg_repository_path) | quote }}
readonly remote_path={{ atlas_borg_remote_path | quote }}
readonly archive_prefix={{ atlas_borg_archive_prefix | quote }}
readonly borg_lock={{ atlas_borg_lock_path | quote }}
exec 8>"$borg_lock"
flock 8
exec borg --remote-path "$remote_path" --lock-wait 600 check \
--show-rc \
--glob-archives "${archive_prefix}-*" \
"$repository"

View File

@@ -0,0 +1,12 @@
[Unit]
Description=Schedule checks of the encrypted Atlas Borg repository
[Timer]
OnCalendar={{ atlas_borg_check_calendar }}
Persistent=true
RandomizedDelaySec={{ atlas_borg_randomized_delay }}
AccuracySec=1min
Unit=atlas-borg-check.service
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,51 @@
#!/usr/bin/env bash
set -Eeuo pipefail
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
readonly pool={{ atlas_zfs_pool | quote }}
readonly mount_root={{ atlas_mount_root | quote }}
readonly snapshot_prefix={{ atlas_borg_snapshot_prefix | quote }}
readonly marker=/run/atlas-borg/snapshot-name
[[ -e "$marker" ]] || exit 0
[[ -f "$marker" && ! -L "$marker" ]] || {
printf 'Unsafe Atlas Borg snapshot marker; leaving snapshots unchanged\n' >&2
exit 2
}
IFS= read -r snapshot_name <"$marker"
[[ "$snapshot_name" =~ ^${snapshot_prefix}-[0-9]{8}T[0-9]{6}Z$ ]] || {
printf 'Invalid Atlas Borg snapshot marker; leaving snapshots unchanged\n' >&2
exit 2
}
exec 9>/run/lock/atlas-zfs-snapshot.lock
flock 9
if zfs list -H -t snapshot -o name "${pool}@${snapshot_name}" >/dev/null 2>&1; then
# The private bind mounts are gone, but ZFS may leave its on-demand
# .zfs/snapshot mounts in the host namespace until explicitly unmounted.
snapshot_mounts=()
snapshot_sources=()
while IFS=$'\t' read -r dataset dataset_mountpoint; do
[[ "$dataset_mountpoint" == "$mount_root" || "$dataset_mountpoint" == "$mount_root/"* ]] || continue
snapshot_mounts+=("${dataset_mountpoint}/.zfs/snapshot/${snapshot_name}")
snapshot_sources+=("${dataset}@${snapshot_name}")
done < <(zfs list -H -o name,mountpoint -s name -r "$pool")
{% raw %}
for ((index = ${#snapshot_mounts[@]} - 1; index >= 0; index--)); do
{% endraw %}
mounted_source="$(findmnt -rn -M "${snapshot_mounts[$index]}" -o SOURCE || true)"
[[ -n "$mounted_source" ]] || continue
[[ "$mounted_source" == "${snapshot_sources[$index]}" ]] || {
printf 'Unexpected source on Atlas Borg snapshot mount: %s\n' \
"${snapshot_mounts[$index]}" >&2
exit 2
}
umount "${snapshot_mounts[$index]}"
done
zfs destroy -r "${pool}@${snapshot_name}"
printf 'Removed recursive Atlas Borg source snapshot %s@%s after backup exit\n' \
"$pool" "$snapshot_name"
fi

View File

@@ -0,0 +1,19 @@
#!/usr/bin/env bash
set -euo pipefail
# Borg receives CAP_DAC_READ_SEARCH only for local snapshot traversal. Drop it
# before starting the network transport so SSH runs as the plain service user.
exec setpriv \
--inh-caps=-all \
--ambient-caps=-all \
-- /usr/bin/ssh \
-i {{ atlas_borg_ssh_private_key_path | quote }} \
-p {{ atlas_borg_repository_port | int }} \
-o BatchMode=yes \
-o IdentitiesOnly=yes \
-o StrictHostKeyChecking=yes \
-o UserKnownHostsFile={{ atlas_borg_known_hosts_path | quote }} \
-o ConnectTimeout=30 \
-o ServerAliveInterval=60 \
-o ServerAliveCountMax=3 \
"$@"

View File

@@ -0,0 +1,20 @@
{
"pool": {{ atlas_zfs_pool | to_json }},
"backup_dataset": {{ (atlas_zfs_pool ~ '/' ~ atlas_zfs_dataset_backup) | to_json }},
"notifier": {{ atlas_monitor_notifier | to_json }},
"smart_devices": {{ atlas_monitor_smart_devices | to_json }},
"timers": {{ atlas_monitor_timers | to_json }},
"failure_units": {{ atlas_monitor_failure_units | to_json }},
"remote_capacity": {{ atlas_monitor_remote_capacity | to_json }},
"pool_warning_percent": {{ atlas_monitor_pool_warning_percent | int }},
"pool_critical_percent": {{ atlas_monitor_pool_critical_percent | int }},
"root_warning_percent": {{ atlas_monitor_root_warning_percent | int }},
"root_critical_percent": {{ atlas_monitor_root_critical_percent | int }},
"snapshot_warning_percent": {{ atlas_monitor_snapshot_warning_percent | int }},
"snapshot_critical_percent": {{ atlas_monitor_snapshot_critical_percent | int }},
"snapshot_growth_warning_gib_day": {{ atlas_monitor_snapshot_growth_warning_gib_day | int }},
"backup_growth_warning_gib_day": {{ atlas_monitor_backup_growth_warning_gib_day | int }},
"cpu_warning_c": {{ atlas_monitor_cpu_warning_c | int }},
"cpu_critical_c": {{ atlas_monitor_cpu_critical_c | int }},
"borg_max_runtime_days": {{ atlas_monitor_borg_max_runtime_days | int }}
}

View File

@@ -0,0 +1,25 @@
[Unit]
Description=Check Atlas pool, disks, capacity, temperatures and maintenance jobs
Wants=houston-dbus.service network-online.target
After=zfs.target houston-dbus.service network-online.target
ConditionFileIsExecutable=/usr/local/libexec/atlas-health-monitor
[Service]
Type=oneshot
ExecStart=/usr/local/libexec/atlas-health-monitor
User=root
Group=root
UMask=0077
StateDirectory=atlas-health-monitor
StateDirectoryMode=0700
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/atlas-health-monitor
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictRealtime=true
LockPersonality=true

View File

@@ -0,0 +1,11 @@
[Unit]
Description=Schedule Atlas health checks
[Timer]
OnCalendar={{ atlas_monitor_calendar }}
Persistent=true
RandomizedDelaySec=5min
Unit=atlas-health-monitor.service
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,2 @@
[Unit]
OnFailure=atlas-monitor-failure@%n.service

View File

@@ -0,0 +1,19 @@
[Unit]
Description=Submit a 45Drives Alert for failed Atlas job %I
Requires=houston-dbus.service
After=houston-dbus.service
ConditionFileIsExecutable=/usr/local/libexec/atlas-health-monitor
[Service]
Type=oneshot
ExecStart=/usr/local/libexec/atlas-health-monitor --job-failed %I
User=root
Group=root
UMask=0077
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
RestrictAddressFamilies=AF_UNIX
RestrictRealtime=true
LockPersonality=true

View File

@@ -0,0 +1,30 @@
[Unit]
Description=Run a manual, UUID-bound offline USB backup of Atlas ZFS datasets
Requires=zfs.target
After=zfs.target
ConditionFileIsExecutable=/usr/local/sbin/atlas-usb-backup
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/atlas-usb-backup
ExecStopPost=+/usr/local/sbin/atlas-usb-snapshot-cleanup
User=root
Group=root
UMask=0077
TimeoutStartSec=infinity
RuntimeDirectory=atlas-usb-backup
RuntimeDirectoryMode=0700
Nice=15
IOSchedulingClass=best-effort
IOSchedulingPriority=7
PrivateMounts=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/run/atlas-usb-backup /run/lock
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_UNIX
RestrictRealtime=true
LockPersonality=true

View File

@@ -0,0 +1,242 @@
#!/usr/bin/env bash
set -Eeuo pipefail
export LC_ALL=C.utf8
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
readonly pool={{ atlas_zfs_pool | quote }}
readonly mount_root={{ atlas_mount_root | quote }}
readonly luks_uuid={{ atlas_usb_backup_luks_uuid | quote }}
readonly fs_uuid={{ atlas_usb_backup_fs_uuid | quote }}
readonly mapper_name={{ atlas_usb_backup_mapper_name | quote }}
readonly snapshot_prefix={{ atlas_usb_backup_snapshot_prefix | quote }}
readonly min_free_bytes={{ atlas_usb_backup_min_free_bytes | int }}
readonly mapper="/dev/mapper/${mapper_name}"
readonly outer="/dev/disk/by-uuid/${luks_uuid}"
readonly runtime_dir=/run/atlas-usb-backup
readonly snapshot_marker="${runtime_dir}/snapshot-name"
readonly source_dir="${runtime_dir}/source"
readonly usb_mount="${runtime_dir}/target"
readonly backup_root="${usb_mount}/atlas"
snapshot_name=""
mapper_opened_by_script=false
usb_mounted=false
published=false
partial=""
mounted_targets=()
# shellcheck disable=SC2329
cleanup() {
local status=$?
local cleanup_status=0
local index
local source_mount_failed=false
trap - EXIT HUP INT TERM
set +e
if [[ -n "$partial" && "$published" == false && "$usb_mounted" == true ]]; then
rm -rf -- "$partial" || cleanup_status=2
fi
if [[ "$usb_mounted" == true ]]; then
umount "$usb_mount" || cleanup_status=2
fi
{% raw %}
for ((index = ${#mounted_targets[@]} - 1; index >= 0; index--)); do
{% endraw %}
if mountpoint -q "${mounted_targets[$index]}" && ! umount -R "${mounted_targets[$index]}"; then
printf 'Source snapshot mount cleanup failed: %s\n' "${mounted_targets[$index]}" >&2
source_mount_failed=true
fi
if mountpoint -q "${mounted_targets[$index]}"; then
printf 'Source snapshot mount is still active: %s\n' "${mounted_targets[$index]}" >&2
source_mount_failed=true
else
rmdir -- "${mounted_targets[$index]}" 2>/dev/null || true
fi
done
if [[ "$source_mount_failed" == false ]]; then
rmdir -- "$source_dir" 2>/dev/null || true
else
cleanup_status=2
printf 'Source bind mount cleanup failed; keeping the snapshot for recovery\n' >&2
fi
if [[ "$usb_mounted" == true || "$mapper_opened_by_script" == true ]] &&
! mountpoint -q "$usb_mount" &&
! findmnt -rn -S "$mapper" >/dev/null; then
cryptsetup close "$mapper_name" || cleanup_status=2
fi
rmdir -- "$usb_mount" 2>/dev/null || true
if ((status == 0 && cleanup_status != 0)); then
status=$cleanup_status
fi
exit "$status"
}
trap cleanup EXIT
trap 'exit 143' HUP INT TERM
exec 8>/run/lock/atlas-usb-backup.lock
flock -n 8 || { printf 'Atlas USB backup is already running\n' >&2; exit 75; }
exec 9>/run/lock/atlas-zfs-snapshot.lock
zpool list -H -o name "$pool" >/dev/null
[[ -b "$outer" ]] || { printf 'Configured LUKS UUID is not connected\n' >&2; exit 66; }
[[ "$(blkid -s TYPE -o value "$outer")" == crypto_LUKS ]] || {
printf 'Configured outer UUID is not a LUKS container\n' >&2
exit 65
}
[[ "$(blkid -s UUID -o value "$outer")" == "$luks_uuid" ]] || exit 65
if ! cryptsetup status "$mapper_name" >/dev/null; then
printf 'Requesting the LUKS passphrase for the configured USB disk\n'
systemd-ask-password -n --no-tty --timeout=300 \
--id="atlas-usb-backup:${luks_uuid}" \
'Atlas offline USB backup LUKS passphrase:' |
cryptsetup open --type luks2 --key-file - "$outer" "$mapper_name"
mapper_opened_by_script=true
fi
backing_device="$(cryptsetup status "$mapper_name" | awk '$1 == "device:" { print $2 }')"
[[ -n "$backing_device" && "$(readlink -f "$backing_device")" == "$(readlink -f "$outer")" ]] || {
printf 'The unlocked mapper does not belong to the configured LUKS UUID\n' >&2
exit 65
}
[[ "$(blkid -s TYPE -o value "$mapper")" == ext4 ]] || {
printf 'The unlocked USB filesystem is not ext4\n' >&2
exit 65
}
[[ "$(blkid -s UUID -o value "$mapper")" == "$fs_uuid" ]] || {
printf 'The unlocked USB filesystem UUID does not match\n' >&2
exit 65
}
if findmnt -rn -S "$mapper" >/dev/null; then
printf 'The USB filesystem is already mounted elsewhere\n' >&2
exit 65
fi
[[ ! -e "$source_dir" && ! -e "$usb_mount" ]] || {
printf 'USB backup staging directories already exist; inspect them manually\n' >&2
exit 65
}
mkdir -m 0700 "$usb_mount"
mount -t ext4 -o nodev,nosuid,noexec "$mapper" "$usb_mount"
usb_mounted=true
[[ "$(readlink -f "$(findmnt -nro SOURCE --target "$usb_mount")")" == "$(readlink -f "$mapper")" ]] || {
printf 'Mounted USB source does not match the verified mapper\n' >&2
exit 65
}
for path in "$backup_root" "$backup_root/snapshots"; do
[[ ! -L "$path" ]] || { printf 'Unsafe symlink in USB backup destination\n' >&2; exit 65; }
mkdir -p -- "$path"
[[ -d "$path" ]] || exit 65
chown root:root -- "$path"
chmod 0700 -- "$path"
done
free_bytes="$(df -B1 --output=avail "$usb_mount" | tail -n 1 | tr -d ' ')"
if ((free_bytes < min_free_bytes)); then
printf 'USB free space (%s bytes) is below the required reserve (%s bytes)\n' \
"$free_bytes" "$min_free_bytes" >&2
exit 73
fi
mkdir -m 0700 "$source_dir"
flock 9
timestamp="$(date -u +%Y%m%dT%H%M%SZ)"
snapshot_name="${snapshot_prefix}-${timestamp}-$$"
printf '%s\n' "$snapshot_name" >"$snapshot_marker"
zfs snapshot -r "${pool}@${snapshot_name}"
flock -u 9
printf 'Created recursive USB source snapshot %s@%s\n' "$pool" "$snapshot_name"
while IFS=$'\t' read -r dataset dataset_mountpoint mounted; do
if [[ "$mounted" != yes ]]; then
printf 'Dataset %s is not mounted; refusing an incomplete backup\n' "$dataset" >&2
exit 65
fi
if [[ "$dataset_mountpoint" != "$mount_root" && "$dataset_mountpoint" != "$mount_root/"* ]]; then
printf 'Dataset %s has unexpected mountpoint %s\n' "$dataset" "$dataset_mountpoint" >&2
exit 65
fi
dataset_suffix="${dataset#"$pool"}"
source_path="${dataset_mountpoint}/.zfs/snapshot/${snapshot_name}"
target_path="${source_dir}${dataset_suffix}"
mkdir -p "$target_path"
mount --bind "$source_path" "$target_path"
mounted_targets+=("$target_path")
mount -o remount,bind,ro "$target_path"
done < <(zfs list -H -o name,mountpoint,mounted -s name -r "$pool")
previous=""
if [[ -e "$backup_root/latest" || -L "$backup_root/latest" ]]; then
[[ -L "$backup_root/latest" ]] || { printf 'latest is not a symlink\n' >&2; exit 65; }
previous="$(readlink -e "$backup_root/latest")"
[[ -n "$previous" && "$previous" == "$backup_root/snapshots/"* && -d "$previous" ]] || {
printf 'latest does not point to a complete snapshot on the USB disk\n' >&2
exit 65
}
fi
backup_name="${timestamp}-$$"
candidate_partial="${backup_root}/snapshots/.incomplete-${backup_name}"
complete="${backup_root}/snapshots/${backup_name}"
[[ ! -e "$candidate_partial" && ! -L "$candidate_partial" && ! -e "$complete" && ! -L "$complete" ]] || exit 65
mkdir -m 0700 "$candidate_partial"
partial="$candidate_partial"
printf 'Copying the consistent pool tree to USB backup %s\n' "$backup_name"
# Preserve POSIX ACLs, ownership, modes, timestamps, hard links, and sparse
# files. Do not preserve generic xattrs: Rocky 9's rsync 3.2.7 fails when
# combining xattrs with --link-dest, while SELinux labels were intentionally
# excluded because restores must relabel for their destination host.
rsync_args=(-aHAS --numeric-ids "--info=progress2,stats2")
estimate_args=(-aHAS --numeric-ids --dry-run --stats)
if [[ -n "$previous" ]]; then
rsync_args+=("--link-dest=$previous")
estimate_args+=("--link-dest=$previous")
fi
# The rsync dry run estimates changed file bytes after link-dest deduplication.
# Metadata and filesystem allocation still require the separate free-space reserve.
estimate="$(rsync "${estimate_args[@]}" "${source_dir}/" "${partial}/")"
transfer_bytes="$(printf '%s\n' "$estimate" | awk -F: \
'/^Total transferred file size:/ { gsub(/[^0-9]/, "", $2); print $2 }')"
[[ "$transfer_bytes" =~ ^[0-9]+$ ]] || {
printf 'Could not determine the USB transfer size\n' >&2
exit 74
}
if ((free_bytes - transfer_bytes < min_free_bytes)); then
printf 'Insufficient USB space: %s bytes free, %s estimated transfer, %s reserved\n' \
"$free_bytes" "$transfer_bytes" "$min_free_bytes" >&2
exit 73
fi
rsync "${rsync_args[@]}" "${source_dir}/" "${partial}/"
printf 'Verifying USB backup %s with a checksum-based dry run\n' "$backup_name"
verification="${runtime_dir}/verification.out"
rsync -aHAS --numeric-ids \
--checksum --dry-run --delete --itemize-changes \
"${source_dir}/" "${partial}/" >"$verification"
if [[ -s "$verification" ]]; then
printf 'USB verification found mismatches; refusing to publish the backup\n' >&2
exit 74
fi
free_bytes="$(df -B1 --output=avail "$usb_mount" | tail -n 1 | tr -d ' ')"
if ((free_bytes < min_free_bytes)); then
printf 'USB backup completed below the free-space reserve; refusing to publish it\n' >&2
exit 73
fi
mv -- "$partial" "$complete"
partial=""
ln -s "snapshots/${backup_name}" "${backup_root}/.latest-${backup_name}"
mv -Tf -- "${backup_root}/.latest-${backup_name}" "${backup_root}/latest"
published=true
sync -f "$complete"
sync -f "$backup_root"
printf 'USB backup %s verified and published; unmounting and closing LUKS\n' "$backup_name"

View File

@@ -0,0 +1,31 @@
#!/usr/bin/python3
"""Submit a manual-backup reminder through Atlas' existing Houston notifier."""
import json
import subprocess
from datetime import datetime, timezone
now = datetime.now(timezone.utc)
message = {
"timestamp": now.isoformat(timespec="seconds"),
"unixtime": int(now.timestamp()),
"event": "atlas_usb_backup_reminder",
"severity": "warning",
"subject": "Promemoria backup USB offline Atlas",
"email_message": (
"Collega il disco USB di backup ad Atlas ed esegui manualmente il backup offline.\n"
"Il promemoria non avvia il backup. Controlla che il disco non sia\n"
"montato; poi esegui:\n\n"
" sudo systemctl start atlas-usb-backup.service\n\n"
"Verifica l'esito con:\n"
" sudo journalctl -u atlas-usb-backup.service -n 100 --no-pager\n\n"
"Dopo la riuscita, scollega fisicamente il disco."
),
}
subprocess.run(
[{{ atlas_usb_reminder_notifier | to_json }}, json.dumps(message)],
check=True,
)
print("Atlas USB backup reminder submitted to 45Drives Alerts; email delivery is not verified.", flush=True)

View File

@@ -0,0 +1,22 @@
[Unit]
Description=45Drives Alerts reminder to run the manual Atlas offline USB backup
Requires=houston-dbus.service
After=houston-dbus.service
ConditionFileIsExecutable=/usr/local/libexec/atlas-usb-reminder
[Service]
Type=oneshot
ExecStart=/usr/local/libexec/atlas-usb-reminder
User=root
Group=root
UMask=0077
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_UNIX
RestrictRealtime=true
LockPersonality=true

View File

@@ -0,0 +1,10 @@
[Unit]
Description=Remind the administrator to run the manual Atlas offline USB backup
[Timer]
OnCalendar={{ atlas_usb_reminder_calendar }}
Persistent=true
Unit=atlas-usb-reminder.service
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,51 @@
#!/usr/bin/env bash
set -Eeuo pipefail
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
readonly pool={{ atlas_zfs_pool | quote }}
readonly mount_root={{ atlas_mount_root | quote }}
readonly snapshot_prefix={{ atlas_usb_backup_snapshot_prefix | quote }}
readonly marker=/run/atlas-usb-backup/snapshot-name
[[ -e "$marker" ]] || exit 0
[[ -f "$marker" && ! -L "$marker" ]] || {
printf 'Unsafe Atlas USB snapshot marker; leaving snapshots unchanged\n' >&2
exit 2
}
IFS= read -r snapshot_name <"$marker"
[[ "$snapshot_name" =~ ^${snapshot_prefix}-[0-9]{8}T[0-9]{6}Z-[0-9]+$ ]] || {
printf 'Invalid Atlas USB snapshot marker; leaving snapshots unchanged\n' >&2
exit 2
}
exec 9>/run/lock/atlas-zfs-snapshot.lock
flock 9
if zfs list -H -t snapshot -o name "${pool}@${snapshot_name}" >/dev/null 2>&1; then
# ZFS can leave its on-demand .zfs/snapshot mounts in the host namespace
# even after the backup's private bind mounts and process have exited.
snapshot_mounts=()
snapshot_sources=()
while IFS=$'\t' read -r dataset dataset_mountpoint; do
[[ "$dataset_mountpoint" == "$mount_root" || "$dataset_mountpoint" == "$mount_root/"* ]] || continue
snapshot_mounts+=("${dataset_mountpoint}/.zfs/snapshot/${snapshot_name}")
snapshot_sources+=("${dataset}@${snapshot_name}")
done < <(zfs list -H -o name,mountpoint -s name -r "$pool")
{% raw %}
for ((index = ${#snapshot_mounts[@]} - 1; index >= 0; index--)); do
{% endraw %}
mounted_source="$(findmnt -rn -M "${snapshot_mounts[$index]}" -o SOURCE || true)"
[[ -n "$mounted_source" ]] || continue
[[ "$mounted_source" == "${snapshot_sources[$index]}" ]] || {
printf 'Unexpected source on Atlas USB snapshot mount: %s\n' \
"${snapshot_mounts[$index]}" >&2
exit 2
}
umount "${snapshot_mounts[$index]}"
done
zfs destroy -r "${pool}@${snapshot_name}"
printf 'Removed recursive Atlas USB source snapshot %s@%s after backup exit\n' \
"$pool" "$snapshot_name"
fi

View File

@@ -0,0 +1,6 @@
[Timer]
OnCalendar=
OnCalendar={{ atlas_zfs_scrub_calendar }}
Persistent=true
RandomizedDelaySec=0
AccuracySec=1min

View File

@@ -0,0 +1,72 @@
#!/usr/bin/env bash
set -euo pipefail
export LC_ALL=C
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
if [[ $# -ne 1 ]]; then
printf 'Usage: %s <policy>\n' "$0" >&2
exit 64
fi
readonly pool={{ atlas_zfs_pool | quote }}
readonly prefix={{ atlas_zfs_snapshot_prefix | quote }}
readonly period="$1"
case "$period" in
{% for policy in atlas_zfs_snapshot_policies %}
{{ policy.name | quote }})
keep={{ policy.keep | int }}
;;
{% endfor %}
*)
printf 'Unknown Atlas ZFS snapshot policy: %s\n' "$period" >&2
exit 64
;;
esac
readonly keep
zpool list -H -o name "$pool" >/dev/null
exec 9>/run/lock/atlas-zfs-snapshot.lock
flock 9
timestamp="$(date -u +%Y%m%dT%H%M%SZ)"
readonly timestamp
readonly snapshot_prefix="${pool}@${prefix}-${period}-"
readonly snapshot="${snapshot_prefix}${timestamp}"
zfs snapshot -r "$snapshot"
printf 'Created recursive ZFS snapshot %s\n' "$snapshot"
snapshot_listing="$(zfs list -H -t snapshot -o name -s creation -r "$pool")"
managed_snapshots=()
while IFS= read -r snapshot_name; do
if [[ "$snapshot_name" == "$snapshot_prefix"* ]]; then
snapshot_suffix="${snapshot_name#"$snapshot_prefix"}"
if [[ "$snapshot_suffix" =~ ^[0-9]{8}T[0-9]{6}Z$ ]]; then
managed_snapshots+=("$snapshot_name")
fi
fi
done <<< "$snapshot_listing"
{% raw %}
managed_snapshot_count="${#managed_snapshots[@]}"
{% endraw %}
prune_count=$((managed_snapshot_count - keep))
if ((prune_count <= 0)); then
printf 'Retaining %d of %d managed %s snapshots\n' \
"$managed_snapshot_count" "$keep" "$period"
exit 0
fi
for ((index = 0; index < prune_count; index++)); do
candidate="${managed_snapshots[$index]}"
if [[ "$candidate" != "$snapshot_prefix"* ]]; then
printf 'Refusing to destroy unexpected snapshot: %s\n' "$candidate" >&2
exit 65
fi
zfs destroy -r "$candidate"
printf 'Pruned recursive ZFS snapshot %s\n' "$candidate"
done

View File

@@ -0,0 +1,11 @@
[Unit]
Description=Schedule {{ item.name }} ZFS snapshots for {{ atlas_zfs_pool }}
[Timer]
OnCalendar={{ item.calendar }}
Persistent=true
AccuracySec=1min
Unit=atlas-zfs-snapshot@{{ item.name }}.service
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,25 @@
[Unit]
Description=Create and retain %i ZFS snapshots for {{ atlas_zfs_pool }}
Documentation=man:zfs-snapshot(8) man:zfs-destroy(8)
Requires=zfs.target
After=zfs.target
ConditionFileIsExecutable=/usr/local/sbin/atlas-zfs-snapshot
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/atlas-zfs-snapshot %i
User=root
Group=root
UMask=0077
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictRealtime=true
LockPersonality=true

View File

@@ -5,14 +5,18 @@ backend_phase1_username: "{{ atlas_admin_username }}"
backend_phase1_user_group: "{{ atlas_admin_group }}" backend_phase1_user_group: "{{ atlas_admin_group }}"
backend_phase1_user_home: "{{ atlas_admin_home }}" backend_phase1_user_home: "{{ atlas_admin_home }}"
backend_phase1_quadlet_dir: "{{ backend_phase1_user_home }}/.config/containers/systemd" backend_phase1_quadlet_dir: "{{ backend_phase1_user_home }}/.config/containers/systemd"
backend_phase1_wireguard_interface: wg0 backend_phase1_bind_address: CHANGEME_ATLAS_BIND_ADDRESS
backend_phase1_wireguard_address: CHANGEME_ATLAS_WIREGUARD_ADDRESS backend_phase1_firewalld_zone: public
backend_phase1_wireguard_firewalld_zone: wireguard backend_phase1_npm_source_ip: CHANGEME_AEGIS_IP
backend_phase1_syncthing_native_subnet: CHANGEME_LAN_SUBNET
backend_phase1_music_dir: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_music }}" backend_phase1_music_dir: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_music }}"
backend_phase1_archive_dir: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_archive }}"
backend_phase1_syncthing_folder_path: "{{ backend_phase1_archive_dir }}/Org"
backend_phase1_app_data_root: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_app_data }}" backend_phase1_app_data_root: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_app_data }}"
backend_phase1_navidrome_data_dir: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_navidrome }}" backend_phase1_navidrome_data_dir: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_navidrome }}"
backend_phase1_syncthing_root: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_syncthing }}" backend_phase1_syncthing_root: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_syncthing }}"
backend_phase1_music_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_music }}" backend_phase1_music_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_music }}"
backend_phase1_archive_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_archive }}"
backend_phase1_app_data_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_app_data }}" backend_phase1_app_data_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_app_data }}"
backend_phase1_navidrome_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_navidrome }}" backend_phase1_navidrome_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_navidrome }}"
backend_phase1_syncthing_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_syncthing }}" backend_phase1_syncthing_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_syncthing }}"

View File

@@ -7,17 +7,22 @@
ansible.builtin.assert: ansible.builtin.assert:
that: that:
- not (atlas_manage_media_stack | bool) - not (atlas_manage_media_stack | bool)
- backend_phase1_wireguard_interface in ansible_facts.interfaces - backend_phase1_bind_address != 'CHANGEME_ATLAS_BIND_ADDRESS'
- backend_phase1_wireguard_address != 'CHANGEME_ATLAS_WIREGUARD_ADDRESS' - backend_phase1_bind_address is match('^[0-9]{1,3}(\.[0-9]{1,3}){3}$')
- backend_phase1_firewalld_zone | length > 0
- backend_phase1_npm_source_ip != 'CHANGEME_AEGIS_IP'
- backend_phase1_syncthing_native_subnet != 'CHANGEME_LAN_SUBNET'
- backend_phase1_music_dir.startswith('/') - backend_phase1_music_dir.startswith('/')
- backend_phase1_archive_dir.startswith('/')
- backend_phase1_syncthing_folder_path.startswith(backend_phase1_archive_dir + '/')
- backend_phase1_app_data_root.startswith('/') - backend_phase1_app_data_root.startswith('/')
- backend_phase1_navidrome_data_dir.startswith(backend_phase1_app_data_root + '/') - backend_phase1_navidrome_data_dir.startswith(backend_phase1_app_data_root + '/')
- backend_phase1_syncthing_root.startswith(backend_phase1_app_data_root + '/') - backend_phase1_syncthing_root.startswith(backend_phase1_app_data_root + '/')
fail_msg: >- fail_msg: >-
Disable the rootful media-stack gate and provide the active Disable the rootful media-stack gate and provide the Atlas LAN bind
WireGuard interface/address and absolute ZFS-backed paths before address, firewall sources, and absolute ZFS-backed paths before
enabling phase one. This role does not manage Prometheus or migrate enabling phase one. This role does not manage Prometheus or migrate
Navidrome application data. application data.
- name: Read the rootless service account - name: Read the rootless service account
ansible.builtin.getent: ansible.builtin.getent:
@@ -49,6 +54,8 @@
loop: loop:
- dataset: "{{ backend_phase1_music_dataset }}" - dataset: "{{ backend_phase1_music_dataset }}"
mountpoint: "{{ backend_phase1_music_dir }}" mountpoint: "{{ backend_phase1_music_dir }}"
- dataset: "{{ backend_phase1_archive_dataset }}"
mountpoint: "{{ backend_phase1_archive_dir }}"
- dataset: "{{ backend_phase1_app_data_dataset }}" - dataset: "{{ backend_phase1_app_data_dataset }}"
mountpoint: "{{ backend_phase1_app_data_root }}" mountpoint: "{{ backend_phase1_app_data_root }}"
- dataset: "{{ backend_phase1_navidrome_dataset }}" - dataset: "{{ backend_phase1_navidrome_dataset }}"
@@ -101,6 +108,8 @@
mode: "0750" mode: "0750"
- path: "{{ backend_phase1_syncthing_root }}" - path: "{{ backend_phase1_syncthing_root }}"
mode: "0750" mode: "0750"
- path: "{{ backend_phase1_syncthing_folder_path }}"
mode: "0770"
loop_control: loop_control:
label: "{{ item.path }}" label: "{{ item.path }}"
@@ -132,19 +141,35 @@
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ backend_phase1_uid }}/bus" DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ backend_phase1_uid }}/bus"
when: not ansible_check_mode when: not ansible_check_mode
- name: Permit phase-one services only through the WireGuard zone - name: Permit NPM access to phase-one web interfaces through Aegis
ansible.posix.firewalld: ansible.posix.firewalld:
port: "{{ item }}" rich_rule: >-
zone: "{{ backend_phase1_wireguard_firewalld_zone }}" rule family="ipv4" source address="{{ backend_phase1_npm_source_ip }}"
port port="{{ item }}" protocol="tcp" accept
zone: "{{ backend_phase1_firewalld_zone }}"
state: enabled state: enabled
permanent: true permanent: true
immediate: true immediate: true
loop: loop:
- "{{ backend_phase1_navidrome_port }}/tcp" - "{{ backend_phase1_navidrome_port }}"
- "{{ backend_phase1_syncthing_gui_port }}/tcp" - "{{ backend_phase1_syncthing_gui_port }}"
- "{{ backend_phase1_syncthing_transfer_port }}/tcp"
- "{{ backend_phase1_syncthing_transfer_port }}/udp" - name: Permit native Syncthing traffic from the LAN
- "{{ backend_phase1_syncthing_discovery_port }}/udp" ansible.posix.firewalld:
rich_rule: >-
rule family="ipv4" source address="{{ backend_phase1_syncthing_native_subnet }}"
port port="{{ item.port }}" protocol="{{ item.protocol }}" accept
zone: "{{ backend_phase1_firewalld_zone }}"
state: enabled
permanent: true
immediate: true
loop:
- port: "{{ backend_phase1_syncthing_transfer_port }}"
protocol: tcp
- port: "{{ backend_phase1_syncthing_transfer_port }}"
protocol: udp
- port: "{{ backend_phase1_syncthing_discovery_port }}"
protocol: udp
- name: Start rootless phase-one Quadlets - name: Start rootless phase-one Quadlets
become_user: "{{ backend_phase1_username }}" become_user: "{{ backend_phase1_username }}"

View File

@@ -8,7 +8,7 @@ Image={{ backend_phase1_navidrome_image }}
UserNS=keep-id UserNS=keep-id
User={{ backend_phase1_uid }} User={{ backend_phase1_uid }}
Group={{ backend_phase1_gid }} Group={{ backend_phase1_gid }}
PublishPort={{ backend_phase1_wireguard_address }}:{{ backend_phase1_navidrome_port }}:4533 PublishPort={{ backend_phase1_bind_address }}:{{ backend_phase1_navidrome_port }}:4533
Environment=ND_LOGLEVEL=info Environment=ND_LOGLEVEL=info
Environment=ND_SCANSCHEDULE=1h Environment=ND_SCANSCHEDULE=1h
Environment=ND_SESSIONTIMEOUT=24h Environment=ND_SESSIONTIMEOUT=24h

View File

@@ -9,15 +9,17 @@ HostName=atlas-syncthing
UserNS=keep-id UserNS=keep-id
User={{ backend_phase1_uid }} User={{ backend_phase1_uid }}
Group={{ backend_phase1_gid }} Group={{ backend_phase1_gid }}
PublishPort={{ backend_phase1_wireguard_address }}:{{ backend_phase1_syncthing_gui_port }}:{{ backend_phase1_syncthing_gui_port }} PublishPort={{ backend_phase1_bind_address }}:{{ backend_phase1_syncthing_gui_port }}:{{ backend_phase1_syncthing_gui_port }}
PublishPort={{ backend_phase1_wireguard_address }}:{{ backend_phase1_syncthing_transfer_port }}:{{ backend_phase1_syncthing_transfer_port }} PublishPort={{ backend_phase1_bind_address }}:{{ backend_phase1_syncthing_transfer_port }}:{{ backend_phase1_syncthing_transfer_port }}
PublishPort={{ backend_phase1_wireguard_address }}:{{ backend_phase1_syncthing_transfer_port }}:{{ backend_phase1_syncthing_transfer_port }}/udp PublishPort={{ backend_phase1_bind_address }}:{{ backend_phase1_syncthing_transfer_port }}:{{ backend_phase1_syncthing_transfer_port }}/udp
PublishPort={{ backend_phase1_wireguard_address }}:{{ backend_phase1_syncthing_discovery_port }}:{{ backend_phase1_syncthing_discovery_port }}/udp PublishPort={{ backend_phase1_bind_address }}:{{ backend_phase1_syncthing_discovery_port }}:{{ backend_phase1_syncthing_discovery_port }}/udp
Environment=HOME=/var/syncthing Environment=HOME=/var/syncthing
Environment=STHOMEDIR=/var/syncthing/config Environment=STHOMEDIR=/var/syncthing/config
Environment=STGUIADDRESS=0.0.0.0:{{ backend_phase1_syncthing_gui_port }} Environment=STGUIADDRESS=0.0.0.0:{{ backend_phase1_syncthing_gui_port }}
Environment=TZ={{ backend_phase1_timezone }} Environment=TZ={{ backend_phase1_timezone }}
Volume={{ backend_phase1_syncthing_root }}:/var/syncthing:Z Volume={{ backend_phase1_syncthing_root }}:/var/syncthing:Z
# Archive is intentionally shared with Samba; use a shared SELinux container label.
Volume={{ backend_phase1_archive_dir }}:/data:z
NoNewPrivileges=true NoNewPrivileges=true
DropCapability=all DropCapability=all

View File

@@ -20,15 +20,7 @@
owner: "{{ item.owner }}" owner: "{{ item.owner }}"
group: "{{ item.group }}" group: "{{ item.group }}"
mode: "{{ item.mode }}" mode: "{{ item.mode }}"
loop: >- loop: "{{ server_directories | default([]) }}"
{{
(server_directories | default([]))
+ (
server_syncthing_directories | default([])
if server_syncthing_enabled | default(true) | bool
else []
)
}}
loop_control: loop_control:
label: "{{ item.path }}" label: "{{ item.path }}"

View File

@@ -66,6 +66,17 @@
changed_when: false changed_when: false
failed_when: false failed_when: false
- name: Check whether the pinned Java 25 version is installed with Mise
tags: [packages, mise, java, wsl]
ansible.builtin.command:
cmd: "mise where java@{{ workstation_mise_java_25_version }}"
become_user: "{{ username }}"
environment:
HOME: "{{ user_home }}"
register: workstation_mise_java_25_where
changed_when: false
failed_when: false
- name: Check whether the pinned Maven version is installed with Mise - name: Check whether the pinned Maven version is installed with Mise
tags: [packages, mise, maven, wsl] tags: [packages, mise, maven, wsl]
ansible.builtin.command: ansible.builtin.command:
@@ -86,6 +97,7 @@
HOME: "{{ user_home }}" HOME: "{{ user_home }}"
when: >- when: >-
workstation_mise_java_where.rc != 0 or workstation_mise_java_where.rc != 0 or
workstation_mise_java_25_where.rc != 0 or
workstation_mise_maven_where.rc != 0 workstation_mise_maven_where.rc != 0
- name: Ensure WSL boot configuration file exists - name: Ensure WSL boot configuration file exists

View File

@@ -10,6 +10,12 @@ wireguard_mtu: 1420
wireguard_firewalld_zone: wireguard wireguard_firewalld_zone: wireguard
wireguard_public_firewalld_zone: public wireguard_public_firewalld_zone: public
wireguard_enable_ipv4_forwarding: false wireguard_enable_ipv4_forwarding: false
# Legacy zone-level masquerading; policy-level masquerading is required for inter-zone forwarding.
wireguard_enable_masquerade: false
# A list of narrowly scoped inter-zone forwarding rules, rendered as persistent
# firewalld policies. Each item requires name, ingress_zone, egress_zone,
# source, and destination.
wireguard_forwarding_policies: []
wireguard_reload_rootful_podman_networks: false wireguard_reload_rootful_podman_networks: false
wireguard_handshake_retries: 12 wireguard_handshake_retries: 12
wireguard_handshake_delay: 5 wireguard_handshake_delay: 5

View File

@@ -14,10 +14,44 @@
Configure this host's WireGuard address and peers, and run the first Configure this host's WireGuard address and peers, and run the first
key bootstrap against every peer in the same play. key bootstrap against every peer in the same play.
- name: Install WireGuard userspace tools - name: Validate WireGuard forwarding policies
ansible.builtin.assert:
that:
- item.name is defined
- item.ingress_zone is defined
- item.egress_zone is defined
- item.source is defined
- item.destination is defined
fail_msg: >-
Every WireGuard forwarding policy requires name, ingress_zone,
egress_zone, source, and destination.
loop: "{{ wireguard_forwarding_policies }}"
loop_control:
label: "{{ item.name | default('unnamed policy') }}"
- name: Install WireGuard userspace tools on mutable hosts
ansible.builtin.dnf: ansible.builtin.dnf:
name: "{{ wireguard_packages }}" name: "{{ wireguard_packages }}"
state: present state: present
when: "'platform_fedora_iot' not in group_names"
- name: Require WireGuard userspace tools in the booted deployment
ansible.builtin.command:
argv:
- wg
- --version
changed_when: false
failed_when: false
register: wireguard_userspace_tools
- name: Require active WireGuard userspace tools
ansible.builtin.assert:
that:
- wireguard_userspace_tools.rc == 0
fail_msg: >-
WireGuard userspace tools are not present in the booted deployment.
On Fedora IoT, reboot after rpm-ostree layers wireguard-tools, then
rerun the WireGuard play.
- name: Create private WireGuard configuration directory - name: Create private WireGuard configuration directory
ansible.builtin.file: ansible.builtin.file:
@@ -124,12 +158,33 @@
permanent: true permanent: true
register: wireguard_firewalld_zone_result register: wireguard_firewalld_zone_result
- name: Reload firewalld after creating the WireGuard zone - name: Create the firewalld policy directory
ansible.builtin.file:
path: /etc/firewalld/policies
state: directory
owner: root
group: root
mode: "0755"
when: wireguard_forwarding_policies | length > 0
- name: Render WireGuard forwarding policies
ansible.builtin.template:
src: wireguard-forwarding-policy.xml.j2
dest: "/etc/firewalld/policies/{{ item.name }}.xml"
owner: root
group: root
mode: "0644"
loop: "{{ wireguard_forwarding_policies }}"
loop_control:
label: "{{ item.name }}"
register: wireguard_forwarding_policy_result
- name: Reload firewalld after WireGuard firewall changes
ansible.builtin.systemd: ansible.builtin.systemd:
name: firewalld.service name: firewalld.service
state: reloaded state: reloaded
when: when:
- wireguard_firewalld_zone_result is changed - wireguard_firewalld_zone_result is changed or (wireguard_forwarding_policy_result | default({})) is changed
- not ansible_check_mode - not ansible_check_mode
- name: Restore rootful Podman networking after firewalld reload - name: Restore rootful Podman networking after firewalld reload
@@ -142,7 +197,7 @@
register: wireguard_podman_network_reload register: wireguard_podman_network_reload
changed_when: wireguard_podman_network_reload.stdout_lines | length > 0 changed_when: wireguard_podman_network_reload.stdout_lines | length > 0
when: when:
- wireguard_firewalld_zone_result is changed - wireguard_firewalld_zone_result is changed or (wireguard_forwarding_policy_result | default({})) is changed
- wireguard_reload_rootful_podman_networks | bool - wireguard_reload_rootful_podman_networks | bool
- not ansible_check_mode - not ansible_check_mode
@@ -154,6 +209,14 @@
permanent: true permanent: true
immediate: true immediate: true
- name: Manage legacy WireGuard zone masquerading
ansible.posix.firewalld:
zone: "{{ wireguard_firewalld_zone }}"
masquerade: true
state: "{{ 'enabled' if wireguard_enable_masquerade | bool else 'disabled' }}"
permanent: true
immediate: true
- name: Permit this host's public WireGuard listener - name: Permit this host's public WireGuard listener
ansible.posix.firewalld: ansible.posix.firewalld:
port: "{{ wireguard_listen_port }}/udp" port: "{{ wireguard_listen_port }}/udp"

View File

@@ -11,7 +11,8 @@ ListenPort = {{ wireguard_listen_port }}
{% for peer in wireguard_peers %} {% for peer in wireguard_peers %}
[Peer] [Peer]
# {{ peer.name }} # {{ peer.name }}
PublicKey = {{ hostvars[peer.host].wireguard_public_key }} {% if peer.comment is defined %}# {{ peer.comment }}
{% endif %}PublicKey = {{ hostvars[peer.host].wireguard_public_key }}
AllowedIPs = {{ peer.allowed_ips | join(', ') }} AllowedIPs = {{ peer.allowed_ips | join(', ') }}
{% if peer.endpoint is defined %} {% if peer.endpoint is defined %}
Endpoint = {{ peer.endpoint }} Endpoint = {{ peer.endpoint }}

View File

@@ -0,0 +1,15 @@
<?xml version="1.0" encoding="utf-8"?>
<policy target="CONTINUE">
<short>WireGuard forwarding: {{ item.name }}</short>
<description>Managed WireGuard-to-LAN forwarding policy.</description>
{% if item.masquerade | default(false) %}
<masquerade/>
{% endif %}
<ingress-zone name="{{ item.ingress_zone }}"/>
<egress-zone name="{{ item.egress_zone }}"/>
<rule family="ipv4">
<source address="{{ item.source }}"/>
<destination address="{{ item.destination }}"/>
<accept/>
</rule>
</policy>

View File

@@ -73,13 +73,6 @@
- packages_rocky - packages_rocky
- services_systemd - services_systemd
- name: Configure WireGuard overlay
hosts: wireguard_overlay
become: true
roles:
- wireguard_overlay
- name: Configure Aegis Fedora IoT profile - name: Configure Aegis Fedora IoT profile
hosts: role_aegis hosts: role_aegis
become: true become: true
@@ -87,6 +80,13 @@
roles: roles:
- profile_aegis - profile_aegis
- name: Configure WireGuard overlay
hosts: wireguard_overlay
become: true
roles:
- wireguard_overlay
- name: Configure Atlas NAS profile - name: Configure Atlas NAS profile
hosts: atlas hosts: atlas
become: true become: true

View File

@@ -56,24 +56,6 @@ services:
- "3000:3000" - "3000:3000"
- "127.0.0.1:222:22" - "127.0.0.1:222:22"
{% if server_syncthing_enabled | default(true) | bool %}
syncthing:
image: docker.io/syncthing/syncthing:2
container_name: syncthing
hostname: syncthing
restart: unless-stopped
expose:
- "8384"
volumes:
- "/opt/syncthing/config:/var/syncthing{{ ':' ~ selinux_volume_option if selinux_volume_option else '' }}"
- "/srv/syncthing/data:/data{{ ':' ~ selinux_volume_option if selinux_volume_option else '' }}"
ports:
- "22000:22000/tcp"
- "22000:22000/udp"
- "21027:21027/udp"
networks:
- web
{% endif %}
networks: networks:
web: web:

View File

@@ -1,3 +1,6 @@
[tools] [tools]
java = "temurin-11.0.31+11" java = ["25.0.2", "temurin-11.0.31+11"]
maven = "3.9.16" maven = "3.9.16"
[env]
JAVA_HOME = "{{ env.HOME }}/.local/share/mise/installs/java/25.0.2"

1
secrets/.gitignore vendored
View File

@@ -2,3 +2,4 @@
.vault_pass* .vault_pass*
vault.local.yml vault.local.yml
vault.local.yml* vault.local.yml*
/recovery/

View File

@@ -1,78 +1,83 @@
$ANSIBLE_VAULT;1.1;AES256 $ANSIBLE_VAULT;1.1;AES256
31386434333363613930316363393564373332303236306233643365386639346631336232373361 61353065386233646137323235306631353635663530363237636231316265643562353465323430
3732633931376438313835373537666438383664306266380a633531316432343166323361303465 6165646466623962313835313537633137633766373930380a316335323962616265643136346666
39306136386664356163346266303963373839373763343136346135633236333333326331313063 63336133336131346336383534356637623831363138323165633262386333363535393365383233
6264306265373865310a373735393632373861333433326632303931633732636535326264346537 6234393835653439370a313963313365373633323464343263383661383336363662633133643232
39633766326361666432396432623132616666363462373237623664363664373362346366383230 34366634383862363635653034313531623330396639616462343630326162316535643465653532
65326665303762356233356531306331333231376163353638363936626562623937323166623065 36326534333637376462353561343964633636366331363833313263353133383636623537303663
62633165383033303362336238343037353930396430336537363565333132666532323662653262 35393032316439336666343161653439643638376134363535656262343963393365623432336433
65373531313264353938666161373064393239653165666462336665323962336134343432316134 35383934313762313037326430316666363731666231336534326661353034333063643364343230
32373361613539356262313333663964623034313230356439626139383539326261663765373034 65333739303566366263333565333465613136646237623937393733623438613832393634663463
36633933366662656464306561393130373238313730336638663932633935313037363133636266 39376131313234333039633735613233373931613232653036663665316636303961653834366339
36663935303735623132663464376633306431396662306166393831313566323238363865303162 36353730316132316233303964303839363161346564396163336137663134353062363733656430
66303732376530653435613966373832333161333137366230613166303061333433336461386234 37643339326661653031376265646132623162373562393437373437313732396537383939333666
63393030616662373666643164663862653037383336323766646330653131623930353265336231 62353036316633306666313461663033303830393765396131643035353730383931646239663935
39646335353635616465663763306262353931316562616437313362616136623735646535616431 32626461316364386135303761383837613063336466363162323332663764616464373565383231
33363439623361343231363663363535663265366166346439623935323632366336363135373462 61346463336566346533326535376439643133613762383633396131323632356533636139336365
62646662316538323734333334393566333463653833326162663065646634363336373865623932 62393838316634623932643034376631333539343965383436613364643962363834346337353334
33646533636638386534366561663930613536663935663638313137646537626431393035653466 32656439366439313734353963343133333533653839613632323338336131373566613835393536
36623435613966323166363630383531656436373366343364346439343034333934623836336636 31663433616334373432376531346435336530303936356461303163646463613661643161313661
38653534343563393434646430306662343435653465363439326261373537666233353731306436 66663866343565616631616338353737356164353562366164383736346131666662623132333466
35323739336361343665633239366633393530346335396635316238323435663466316235376536 39383865653631373232393433663430643961646265386166333137643966303834363262373636
38303839313433643038353236616632626363653339346334303136306138336461383831626536 62396434373363353636376133666133663162653265313139313732353639336232333862643036
39303966393034333739363061363865326165303236656438316537396463383935393764346135 64386231336561396537326139346566306434633934343038663165396665363032383466633662
66323632323833353833643434316566326366633562666262386232653730303038613336663264 62336163633964363435386630343966333162333730336138333239646631633132663931376462
62636332383834393861373665393364653362636339346632303463386565633830336363393065 33663139356261313065376636613930353735396131306538306664646135636336643032623131
31383530613161366436323163393366386635316562633436383134623061353937326363396439 38346264333331353633326535326431626563323036313665643337353563333339646430386564
38636438396637616362373937666361303536376164636533653536306338356263313965623336 31613435383036313430316366323636663735326336393338353835323861333564363832656462
32633461623861643138613734313164633562613932386163643062353636376266633166373838 35336435623261326363633033316130393062616339353263643062633331646137376135656365
39613438653531316333663736366161313832613830353566356461393435656234383037353335 35636139336564346164616235616431326531333433646330386134323932373339646536356464
32343661376266353538643531313239306432646335383734623233373063316430383362643531 66343533326534326165323564663533653666633035343163633832393361336462343937623165
35393263343534363936373361316265653934383735663933663365363564663966646335353337 62383931326630363036396333313931393836366439653433623165666166356338653364336534
63366365383466393765336533363130643236313331633537346237353631623334633330376164 35333936653833386163633738326164386166613561333530633937343230363366333662666539
32616530623032663761643437336536323332343130393339313232623364656334613233323833 39333361633933663735303438663239303536363433313962643137386533633539326365383765
64346637343738386335343631333035636337633732333662663763326362636432646136353064 37636538386339333935386132353265353031643662616330316463623661663738353433313830
30393061326435323837633632333765623931663265326137373135303035623464333366653566 36373963633166333464653338343830373063323536383364393033393235326639613662343737
66613133396339376264313964353932396136613538383264363865633536373839663465333437 38663362636331343061646465313237313431373433353361353265333766633463353632646536
37613765626131343330653063663764346364316264363635653438396331363263623562333735 31323231306138323031396630656538363930373439336234343963616334363632653738316465
30646563303439663639376430336336383761363365353838613036333032306434623661636138 63653938373830336362313238656266613362636634616537653863336132343931616262396130
63616439636437663732386164346365643834376631303263376563323662373734653631613730 66393239303866656232653832343132366537333537343635666563343639323433383163613335
31373039656238303462643930303531613931373065316435303661343862623034653364653736 39613533376634316133633430303535306266656333626264343733666335393661666561396633
37613835616630316133333130633631653639336266313438633166316539373433333661613839 39346265316137326465326635396362333565393133623637633132616232326263663662343137
32356139366232313336343062323265366563633735383661393335386163653133313664613264 33363733306135363361643031306265363733656362386666306334333035393839636533343363
63303739323863333439316461366137656434666366306466356633306663623730653939663430 35396638616636633639343930373136376339346162393061393765363837646365383866636131
39313764663534316562326435316264633236373834323665356266346532323565323532333438 33653465666239393133616232636231333332396138376332393664343364643835306530393238
33306237623430613463393164383332356533303433343465343930636563363862373330613632 34663237303530303837663535646263393931373531393039356336316561653130356262636562
38643937613263313435626166313464316133373338373261666331653436373063393162636339 38336362326639653237626634376334666565653036353236313634376364626338646538386536
66356434663239646334383433626566336433653265336332323866633666646132663836313537 38626636386466373566646166393963643164343536373236396138303532393161363335386638
66663766306438396131623533613834626331653731323137303539303835306632663132616363 32633032393737626363613463323366366637616361313537356136626661626633613739323338
64303438353836613136623562626664326364656133383865383730373762666539383036396337 35383963666431343566356562333234663936376562616638636261303466633539376334303331
66393636343561333435313032353939316138306336316337353861383238326136373265663433 39303834663234663063356233313962326664383839393832303462643636393034383434303465
62366566343866626336646466386561353061343735306565383437333931613635393034363430 64333635376135326333356435373734643430623736373234643335343130383066326436356664
37633765663239643435623066313331353862393966306563393838386334393162306562363062 63346663326364343634303930343338336139313864316165366232643537366635653764353763
61633363386564373834383432323861323364623365306439353631323463326461383039356466 31363863633261643263303433373330366161323166366462336332313135366338393334653764
34396464653233626435656463326439303665633532656639376633353932666630626564616564 66353733653137663835663731373364613030373334663061313433373861613665363236633130
34326638383634353033326232646339393638663637313136653763336265616635326666336530 65613965366636343465336533613438373466383737373366653965633437323562643966396431
33663261376262626136346265656130653831636662306132393837306135643831353534626636 39303033643438633762633263326132663466643438656366363431616237633031333936313831
34383762386665666363313932336632326230646439663366663037323562633630373137333232 30323930383233313032323638356333626230333764363662313662646536643839353032353462
37393164656137303131313738396131613561306332356436303436636338623233343637363332 30326166653937353130623133303533343934633565393831623033303234316330353432313266
36613137316337666461333237373266326238303531396432383461616239316630346230303735 30636536633933376365623665616262663236383731633633346232613366333137396139306363
63636535383766353338643932366339333130663632386337643932636630316432313465393766 35633336643266326335303261666666653536666630613639376336373237646134306462616537
39613065623631316165313134386231616165366266323634643632626230303861376461323433 33343561373162666332613634643837343566646161373065366637653135613632353334636363
35346137326563646332303334313530383464373838373633363635373766333362386466323836 63363232303963646530333366663862323264326536643337323266396566316233613630303637
32613236646561343365656239346239353866663336393930356238653463336361333033643331 66646366376466373931613734363931316230323063373666653062373364396433633762633762
61323430663031383239353363616666336637666230663633616464616638303966343631386632 38613933323733653238383935623230383562646563363833653838636165626365646537383639
32333735643936353638666336303133356435396338653465323234316234626536666635333238 33666535656363393562316336633439636138373365623431393965653765306138646234663938
38313661303466333464666538363938656631396666643566343763396638336663376430646532 65653133663663393731646337386535333261643932336132396237323930306136643534353930
38666234653330646262633062356238343536336637356432313137313561363937363936383364 65636438396432623034626561613137336138623265393064383034623863303166356138393564
30643937383833663339643862363234643765386164316138636565643434373734383339363138 37373164626634653662326234333539663735323464613334616130643937373730363263633366
36323163633837326132393365333236633264386664373234313061373835346634663137383837 31393937326432386165343338313031376565313866363731643534313233303064373935303538
62666561353532303663346365343131316233633163323938623066656332383030393864363536 31343832336230393636653432653162336361383963633766343461653466316337353931333363
38383939383935613432613837333863313239653831333438383133343763633838353964353161 63313137303564336630343937356564643763383764613362366634373362666465626334336539
61323462343835613937653465633563306462613631323762656437626133336638396663646362 64366533376165306532343461613265366266383862323032333465336161663161376630316465
30323661383134653336366234663333336261353162373030626266656336356233316265636661 30306562666163646235656664653635366461366435663961623635383437663564356563346462
34303865313433633138363936373561636537353831373033303163646436303932626138356633 31636234663765623838333237393239373564366262613637363938653463396530613963643837
63656364353163313037613262396338636230646330666331616534313466306361363433656132 38636634376637366332623035313465393762653865623130336263343663303066366135616639
66633231626665303165346339373764666264313838313063323732653837383736633235363064 63333964356466613038303263366462346261353030646532366361393965306435613131316463
37353632336238623366313432376163653535656134633634313065356533343933666135396633 65366266376637323764643239323730366565633335666638666334663635373961303637383861
30613134646132613637656461303431613064393438363231383464663765316638 35313431646434656562333937663837393038386361616630626532636339306432353434656165
33663261383166386432383465666136376237346565303164363461666663346130346162316338
62373061353034316234303835663439396434343738303764376665336239626238386436386234
61306166383637366266393730323732386163366261393630336431633862353761343763363665
61323039396234393835303633363339373633653334343766653032313230343464326664356566
3462623830666664626633373966363866333337383730313066

View File

@@ -3,14 +3,13 @@ vault_duckdns_token: "CHANGEME"
vault_personal_full_name: "REPLACE_ME" vault_personal_full_name: "REPLACE_ME"
vault_git_email: "REPLACE_ME" vault_git_email: "REPLACE_ME"
vault_git_signing_key: "REPLACE_ME" vault_git_signing_key: "REPLACE_ME"
vault_icloud_email: "REPLACE_ME"
vault_protonmail_email: "REPLACE_ME"
vault_icloud_mail_password: "REPLACE_ME" vault_icloud_mail_password: "REPLACE_ME"
vault_git_work_email: "REPLACE_ME" vault_git_work_email: "REPLACE_ME"
vault_git_work_gpg: "REPLACE_ME" vault_git_work_gpg: "REPLACE_ME"
vault_openai_api_key: "REPLACE_ME"
vault_ikaros_authorized_ssh_keys: vault_ikaros_authorized_ssh_keys:
- "ssh-ed25519 REPLACE_ME" - "ssh-ed25519 REPLACE_ME"
vault_aegis_icloudpd_apple_id: "REPLACE_ME"
vault_atlas_admin_password_hash: "REPLACE_WITH_A_SHADOW_COMPATIBLE_HASH" vault_atlas_admin_password_hash: "REPLACE_WITH_A_SHADOW_COMPATIBLE_HASH"
vault_atlas_samba_password: "REPLACE_ME" vault_atlas_samba_password: "REPLACE_ME"
vault_atlas_immich_db_password: "REPLACE_ME" vault_atlas_immich_db_password: "REPLACE_ME"
vault_atlas_borg_passphrase: "REPLACE_WITH_A_STRONG_UNIQUE_PASSPHRASE"