diff --git a/AGENTS.md b/AGENTS.md index 059cbe6..ace08c3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -59,6 +59,8 @@ Ansible-driven personal infrastructure repo for Fedora and Void desktops, Fedora `ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff` - Atlas network/share hardening: `ansible-playbook ansible/site.yml --limit atlas --tags hardening,sharing --check --diff` + - Atlas ZFS snapshot retention and scrub timers: + `ansible-playbook ansible/site.yml --limit atlas --tags snapshots,scrub --check --diff` - Prometheus/Aegis WireGuard gateway: `ansible-playbook ansible/site.yml --limit prometheus,aegis --tags wireguard --check --diff` - DuckDNS config only: `ansible-playbook ansible/site.yml --limit prometheus --tags duckdns --check --diff` @@ -163,35 +165,51 @@ The dotfile vars follow the same split: `desktop_common_dotfiles` carries mode-i ## Atlas NAS TODO Completed validation: the existing RAIDZ2 pool and datasets, SELinux, LAN firewall, SSH, Cockpit with -the selected 45Drives plugins, encrypted SMB3 `Archive`, the Aegis-only NFSv4 `photobook` export, and -the former Prometheus--Atlas WireGuard path were operational. Aegis has validated NFSv4.2 read, write, delete, -and `all_squash` mapping to UID/GID `1100` end-to-end. -- Validate the Prometheus--Aegis WireGuard gateway after migration: peer handshake and counters, Aegis IPv4 - forwarding and masquerading, and an NPM request from Prometheus to an Atlas LAN address. Add the Uranus VIP to - Prometheus' Aegis peer when the cluster control plane is assigned. -- Validate temporary Atlas Navidrome and Syncthing through Aegis before creating their NPM Proxy Hosts. - Keep NPM host configuration manual; plan their eventual Uranus migration with storage and routing declared - separately from the NAS baseline. -- Decide whether a common SMB/NFS namespace is required. `Archive` (SMB) and `photobook` (NFS) are - intentionally distinct today; only if a shared namespace is selected, finalize its UID/GID, group, - and POSIX ACL model and test the same files through both protocols. -- Keep `atlas_manage_media_stack` disabled until the future Immich deployment has validated `/dev/dri`, - container paths, and the required Vault database secret. -- Add Ansible-managed ZFS snapshot retention and scrub timers. Use Cockpit Scheduler for visibility - or manual operations, not as the only source of configuration, and never automate snapshot rollback. -- Add the Atlas-initiated least-privilege Prometheus backup pull: Prometheus exposes only prepared - read-only dumps through a dedicated account and Atlas retains the private SSH key, pinned host key, - atomic pull, verification, retention and systemd service/timer. -- Add the encrypted offsite backup with Borg to a Hetzner Storage Box: use a dedicated SSH identity, +the selected 45Drives plugins, encrypted SMB3 `Archive`, the Aegis-only NFSv4 `photobook` export, and the +Prometheus--Aegis WireGuard gateway are operational. The gateway handshake, forwarding, source masquerading, +and TCP reachability to Atlas were verified. Temporary Navidrome and Syncthing are available through their +manual NPM Proxy Hosts; Syncthing uses `/data/Org` backed by the SMB-shared Archive dataset. Aegis has also +validated NFSv4.2 read, write, delete, and `all_squash` mapping to UID/GID `1100` end-to-end. The ZFS +snapshot timers are active and the first recursive hourly snapshot completed successfully; the first +scheduled retention prune and monthly scrub remain runtime checks. + +### Priority 1 - Data protection +- [x] Deploy Ansible-managed recursive ZFS snapshots with 24 hourly, 30 daily, 8 weekly, and 12 monthly + generations, plus a monthly scrub on the first Sunday at 03:00. The timers and first hourly snapshot were + verified on Atlas. Still observe the first scheduled retention prune and scrub; Cockpit Scheduler is for + visibility or manual operations only, and snapshot rollback is never automated. +- [ ] Add the encrypted offsite backup with Borg to a Hetzner Storage Box: use a dedicated SSH identity, pin the host key, keep Borg repository credentials and encryption material in Vault, use snapshot-consistent sources, and manage retries, logging, pruning, repository checks and restores. -- Add the UUID-bound offline USB backup with versioned rsync, locking, capacity checks, verification, +- [ ] Add the UUID-bound offline USB backup with versioned rsync, locking, capacity checks, verification, safe unmounting and a tested restore procedure; never trigger it for an arbitrary USB disk. -- Add monitoring and alerting for pool health, scrub/resilver, SMART data, temperatures, free space and - failed backup timers, plus a controlled Rocky kernel/OpenZFS update and reboot procedure. -- Document and test disaster recovery: rebuild Atlas with Ansible, import the existing pool, restore +- [ ] Test restores independently from a ZFS snapshot, Borg, and the offline USB backup before relying on + any backup path. +- [ ] Add monitoring and alerting for pool health, scrub/resilver, SMART data, temperatures, free space, + snapshot/backup capacity growth, and failed maintenance or backup timers. + +### Priority 2 - NAS operability and recovery +- [ ] Document and test disaster recovery: rebuild Atlas with Ansible, import the existing pool, restore from snapshot/USB/Hetzner, preserve Vault and Borg recovery material offline, and define RPO/RTO. -- Optionally design iCloud photo ingestion and an Aegis persistent NFS mount as a separate workflow +- [ ] Define a controlled Rocky kernel/OpenZFS update and reboot procedure. +- [ ] Add the Atlas-initiated least-privilege Prometheus backup pull: Prometheus exposes only prepared + read-only dumps through a dedicated account and Atlas retains the private SSH key, pinned host key, + atomic pull, verification, retention and systemd service/timer. +- [ ] Decide whether a common SMB/NFS namespace is required. `Archive` (SMB) and `photobook` (NFS) are + intentionally distinct today; only if a shared namespace is selected, finalize its UID/GID, group, + and POSIX ACL model and test the same files through both protocols. + +### Priority 3 - Service expansion +- [ ] After data protection and recovery are validated, populate `/zpool/media/music` and validate Navidrome. +- [ ] Design and deploy Nextcloud as another explicitly temporary Atlas service before Uranus. Give it + separate persistent application, database, and cache storage; keep credentials in Vault; publish it only + through NPM over the Prometheus--Aegis gateway; and define backup, upgrade, and eventual Uranus-migration + procedures before exposing user data. Do not deploy Nextcloud before the data-protection checklist is complete. +- [ ] Keep `atlas_manage_media_stack` disabled until the future Immich deployment has validated `/dev/dri`, + container paths, and the required Vault database secret. + +### Priority 4 - Optional workflows +- [ ] Optionally design iCloud photo ingestion and an Aegis persistent NFS mount as a separate workflow after the storage and backup layers are validated; do not make either a dependency of the Atlas baseline. diff --git a/README.md b/README.md index 639cfbb..fff76b1 100644 --- a/README.md +++ b/README.md @@ -299,9 +299,27 @@ The first real WireGuard run must include both peers. If Fedora IoT has just lay reboot Aegis manually and rerun the command without `--check`; the role then waits for a real peer handshake. -Snapshot retention, Syncthing topology, WireGuard/firewall validation, Prometheus backup pulls, -encrypted Borg backups to a Hetzner Storage Box, USB backup, monitoring, and disaster-recovery tests -remain follow-up work. The detailed operational backlog is kept in `AGENTS.md`. +Atlas declares recursive, systemd-timed ZFS snapshots for the complete pool hierarchy: 24 hourly +snapshots at minute 05, 30 daily snapshots at 00:15, 8 weekly snapshots on Sunday at 01:00, and 12 +monthly snapshots on the first day at 02:00. The retention helper prunes only snapshots carrying its +managed `atlas-auto` prefix and never rolls back a dataset. The OpenZFS monthly scrub timer is scheduled +for the first Sunday at 03:00; the conflicting weekly scrub timer is disabled explicitly. The first recursive +hourly snapshot completed successfully on Atlas; retention pruning and the first scheduled scrub still await +live runtime evidence. Validate this layer independently with: + +```bash +ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \ +ansible-playbook ansible/site.yml --limit atlas --tags snapshots,scrub --check --diff +``` + +A temporary Nextcloud deployment on Atlas is also planned before Uranus: it requires separately +declared persistent application, database, and cache storage, Vault-backed credentials, NPM-only +publishing through Aegis, and defined backup, upgrade, and eventual migration procedures. Do not deploy +it before the data-protection checklist is complete. + +Prometheus backup pulls, encrypted Borg backups to a Hetzner Storage Box, USB backup, restore testing, +monitoring, and disaster-recovery tests remain follow-up work. The prioritized operational backlog is kept +in `AGENTS.md`. ## How layering works diff --git a/ansible/inventory/host_vars/atlas.yml b/ansible/inventory/host_vars/atlas.yml index 2525d44..a654998 100644 --- a/ansible/inventory/host_vars/atlas.yml +++ b/ansible/inventory/host_vars/atlas.yml @@ -49,6 +49,23 @@ atlas_zfs_backup_reservation: 500G atlas_zfs_dataset_photobook: media/photobook atlas_mount_root: /zpool atlas_manage_storage: true +atlas_manage_zfs_snapshots: true +atlas_zfs_snapshot_prefix: atlas-auto +atlas_zfs_snapshot_policies: + - name: hourly + calendar: "*-*-* *:05:00" + keep: 24 + - name: daily + calendar: "*-*-* 00:15:00" + keep: 30 + - name: weekly + calendar: "Sun *-*-* 01:00:00" + keep: 8 + - name: monthly + calendar: "*-*-01 02:00:00" + keep: 12 +atlas_manage_zfs_scrub: true +atlas_zfs_scrub_calendar: "Sun *-*-01..07 03:00:00" atlas_manage_sharing: true atlas_manage_media_stack: false diff --git a/ansible/roles/profile_atlas/defaults/main.yml b/ansible/roles/profile_atlas/defaults/main.yml index 2f1f4c7..dd48564 100644 --- a/ansible/roles/profile_atlas/defaults/main.yml +++ b/ansible/roles/profile_atlas/defaults/main.yml @@ -61,6 +61,12 @@ atlas_zfs_backup_reservation: 500G atlas_zfs_dataset_photobook: media/photobook atlas_mount_root: /CHANGEME_ATLAS_MOUNT_ROOT +atlas_manage_zfs_snapshots: false +atlas_zfs_snapshot_prefix: atlas-auto +atlas_zfs_snapshot_policies: [] +atlas_manage_zfs_scrub: false +atlas_zfs_scrub_calendar: "" + atlas_archive_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_archive }}" atlas_services_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_services }}" atlas_app_data_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_app_data }}" diff --git a/ansible/roles/profile_atlas/tasks/main.yml b/ansible/roles/profile_atlas/tasks/main.yml index a422a9b..b0f31c2 100644 --- a/ansible/roles/profile_atlas/tasks/main.yml +++ b/ansible/roles/profile_atlas/tasks/main.yml @@ -14,6 +14,9 @@ - name: Import Atlas storage tasks ansible.builtin.import_tasks: storage.yml +- name: Import Atlas ZFS maintenance tasks + ansible.builtin.import_tasks: zfs_maintenance.yml + - name: Import Atlas file sharing tasks ansible.builtin.import_tasks: sharing.yml diff --git a/ansible/roles/profile_atlas/tasks/zfs_maintenance.yml b/ansible/roles/profile_atlas/tasks/zfs_maintenance.yml new file mode 100644 index 0000000..6b3cbb5 --- /dev/null +++ b/ansible/roles/profile_atlas/tasks/zfs_maintenance.yml @@ -0,0 +1,175 @@ +--- +- name: Validate Atlas ZFS snapshot policy + tags: [atlas, storage, snapshots] + ansible.builtin.assert: + that: + - atlas_manage_storage | bool + - atlas_zfs_pool != 'CHANGEME_ZFS_POOL' + - atlas_zfs_snapshot_prefix is match('^[a-z0-9][a-z0-9_-]*$') + - atlas_zfs_snapshot_policies | length > 0 + - >- + (atlas_zfs_snapshot_policies | map(attribute='name') | unique | list | length) + == (atlas_zfs_snapshot_policies | length) + fail_msg: >- + Enable Atlas storage and declare a non-empty snapshot policy with a safe + prefix and unique policy names before managing automatic snapshots. + when: atlas_manage_zfs_snapshots | bool + +- name: Validate Atlas ZFS snapshot policy entries + tags: [atlas, storage, snapshots] + ansible.builtin.assert: + that: + - item.name is match('^[a-z][a-z0-9_-]*$') + - item.keep | int > 0 + - item.calendar | length > 0 + fail_msg: >- + Every Atlas snapshot policy needs a safe name, a positive retention + count, and a systemd calendar expression. + loop: "{{ atlas_zfs_snapshot_policies }}" + loop_control: + label: "{{ item.name | default('unnamed') }}" + when: atlas_manage_zfs_snapshots | bool + +- name: Validate Atlas ZFS snapshot calendars + tags: [atlas, storage, snapshots] + ansible.builtin.command: + argv: + - systemd-analyze + - calendar + - "{{ item.calendar }}" + loop: "{{ atlas_zfs_snapshot_policies }}" + loop_control: + label: "{{ item.name }}: {{ item.calendar }}" + changed_when: false + check_mode: false + when: atlas_manage_zfs_snapshots | bool + +- name: Install Atlas ZFS snapshot and retention helper + tags: [atlas, storage, snapshots] + ansible.builtin.template: + src: atlas-zfs-snapshot.sh.j2 + dest: /usr/local/sbin/atlas-zfs-snapshot + owner: root + group: root + mode: "0750" + when: atlas_manage_zfs_snapshots | bool + +- name: Install Atlas ZFS snapshot systemd service + tags: [atlas, storage, snapshots] + ansible.builtin.template: + src: atlas-zfs-snapshot@.service.j2 + dest: /etc/systemd/system/atlas-zfs-snapshot@.service + owner: root + group: root + mode: "0644" + when: atlas_manage_zfs_snapshots | bool + +- name: Install Atlas ZFS snapshot systemd timers + tags: [atlas, storage, snapshots] + ansible.builtin.template: + src: atlas-zfs-snapshot.timer.j2 + dest: "/etc/systemd/system/atlas-zfs-snapshot-{{ item.name }}.timer" + owner: root + group: root + mode: "0644" + loop: "{{ atlas_zfs_snapshot_policies }}" + loop_control: + label: "{{ item.name }}" + when: atlas_manage_zfs_snapshots | bool + +- name: Enable Atlas ZFS snapshot systemd timers + tags: [atlas, storage, snapshots] + ansible.builtin.systemd: + name: "atlas-zfs-snapshot-{{ item.name }}.timer" + enabled: true + state: started + daemon_reload: true + loop: "{{ atlas_zfs_snapshot_policies }}" + loop_control: + label: "{{ item.name }}" + when: + - atlas_manage_zfs_snapshots | bool + - not ansible_check_mode + +- name: Validate Atlas ZFS scrub policy + tags: [atlas, storage, scrub] + ansible.builtin.assert: + that: + - atlas_manage_storage | bool + - atlas_zfs_pool != 'CHANGEME_ZFS_POOL' + - atlas_zfs_scrub_calendar | length > 0 + fail_msg: >- + Enable Atlas storage and declare a systemd calendar expression before + managing periodic ZFS scrubs. + when: atlas_manage_zfs_scrub | bool + +- name: Validate Atlas ZFS scrub calendar + tags: [atlas, storage, scrub] + ansible.builtin.command: + argv: + - systemd-analyze + - calendar + - "{{ atlas_zfs_scrub_calendar }}" + changed_when: false + check_mode: false + when: atlas_manage_zfs_scrub | bool + +- name: Require OpenZFS scrub systemd units + tags: [atlas, storage, scrub] + ansible.builtin.command: + argv: + - systemctl + - cat + - "{{ item }}" + loop: + - "zfs-scrub@{{ atlas_zfs_pool }}.service" + - "zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer" + - "zfs-scrub-weekly@{{ atlas_zfs_pool }}.timer" + loop_control: + label: "{{ item }}" + changed_when: false + check_mode: false + when: atlas_manage_zfs_scrub | bool + +- name: Create Atlas ZFS scrub timer override directory + tags: [atlas, storage, scrub] + ansible.builtin.file: + path: "/etc/systemd/system/zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer.d" + state: directory + owner: root + group: root + mode: "0755" + when: atlas_manage_zfs_scrub | bool + +- name: Configure Atlas ZFS monthly scrub schedule + tags: [atlas, storage, scrub] + ansible.builtin.template: + src: atlas-zfs-scrub-timer.conf.j2 + dest: >- + /etc/systemd/system/zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer.d/override.conf + owner: root + group: root + mode: "0644" + when: atlas_manage_zfs_scrub | bool + +- name: Disable the conflicting weekly OpenZFS scrub timer + tags: [atlas, storage, scrub] + ansible.builtin.systemd: + name: "zfs-scrub-weekly@{{ atlas_zfs_pool }}.timer" + enabled: false + state: stopped + daemon_reload: true + when: + - atlas_manage_zfs_scrub | bool + - not ansible_check_mode + +- name: Enable the Atlas monthly OpenZFS scrub timer + tags: [atlas, storage, scrub] + ansible.builtin.systemd: + name: "zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer" + enabled: true + state: started + daemon_reload: true + when: + - atlas_manage_zfs_scrub | bool + - not ansible_check_mode diff --git a/ansible/roles/profile_atlas/templates/atlas-zfs-scrub-timer.conf.j2 b/ansible/roles/profile_atlas/templates/atlas-zfs-scrub-timer.conf.j2 new file mode 100644 index 0000000..f55984c --- /dev/null +++ b/ansible/roles/profile_atlas/templates/atlas-zfs-scrub-timer.conf.j2 @@ -0,0 +1,6 @@ +[Timer] +OnCalendar= +OnCalendar={{ atlas_zfs_scrub_calendar }} +Persistent=true +RandomizedDelaySec=0 +AccuracySec=1min diff --git a/ansible/roles/profile_atlas/templates/atlas-zfs-snapshot.sh.j2 b/ansible/roles/profile_atlas/templates/atlas-zfs-snapshot.sh.j2 new file mode 100644 index 0000000..bd69d70 --- /dev/null +++ b/ansible/roles/profile_atlas/templates/atlas-zfs-snapshot.sh.j2 @@ -0,0 +1,72 @@ +#!/usr/bin/env bash +set -euo pipefail + +export LC_ALL=C +export PATH=/usr/sbin:/usr/bin:/sbin:/bin + +if [[ $# -ne 1 ]]; then + printf 'Usage: %s \n' "$0" >&2 + exit 64 +fi + +readonly pool={{ atlas_zfs_pool | quote }} +readonly prefix={{ atlas_zfs_snapshot_prefix | quote }} +readonly period="$1" + +case "$period" in +{% for policy in atlas_zfs_snapshot_policies %} + {{ policy.name | quote }}) + keep={{ policy.keep | int }} + ;; +{% endfor %} + *) + printf 'Unknown Atlas ZFS snapshot policy: %s\n' "$period" >&2 + exit 64 + ;; +esac +readonly keep + +zpool list -H -o name "$pool" >/dev/null + +exec 9>/run/lock/atlas-zfs-snapshot.lock +flock 9 + +timestamp="$(date -u +%Y%m%dT%H%M%SZ)" +readonly timestamp +readonly snapshot_prefix="${pool}@${prefix}-${period}-" +readonly snapshot="${snapshot_prefix}${timestamp}" + +zfs snapshot -r "$snapshot" +printf 'Created recursive ZFS snapshot %s\n' "$snapshot" + +snapshot_listing="$(zfs list -H -t snapshot -o name -s creation -r "$pool")" +managed_snapshots=() +while IFS= read -r snapshot_name; do + if [[ "$snapshot_name" == "$snapshot_prefix"* ]]; then + snapshot_suffix="${snapshot_name#"$snapshot_prefix"}" + if [[ "$snapshot_suffix" =~ ^[0-9]{8}T[0-9]{6}Z$ ]]; then + managed_snapshots+=("$snapshot_name") + fi + fi +done <<< "$snapshot_listing" + +{% raw %} +managed_snapshot_count="${#managed_snapshots[@]}" +{% endraw %} +prune_count=$((managed_snapshot_count - keep)) +if ((prune_count <= 0)); then + printf 'Retaining %d of %d managed %s snapshots\n' \ + "$managed_snapshot_count" "$keep" "$period" + exit 0 +fi + +for ((index = 0; index < prune_count; index++)); do + candidate="${managed_snapshots[$index]}" + if [[ "$candidate" != "$snapshot_prefix"* ]]; then + printf 'Refusing to destroy unexpected snapshot: %s\n' "$candidate" >&2 + exit 65 + fi + + zfs destroy -r "$candidate" + printf 'Pruned recursive ZFS snapshot %s\n' "$candidate" +done diff --git a/ansible/roles/profile_atlas/templates/atlas-zfs-snapshot.timer.j2 b/ansible/roles/profile_atlas/templates/atlas-zfs-snapshot.timer.j2 new file mode 100644 index 0000000..398186b --- /dev/null +++ b/ansible/roles/profile_atlas/templates/atlas-zfs-snapshot.timer.j2 @@ -0,0 +1,11 @@ +[Unit] +Description=Schedule {{ item.name }} ZFS snapshots for {{ atlas_zfs_pool }} + +[Timer] +OnCalendar={{ item.calendar }} +Persistent=true +AccuracySec=1min +Unit=atlas-zfs-snapshot@{{ item.name }}.service + +[Install] +WantedBy=timers.target diff --git a/ansible/roles/profile_atlas/templates/atlas-zfs-snapshot@.service.j2 b/ansible/roles/profile_atlas/templates/atlas-zfs-snapshot@.service.j2 new file mode 100644 index 0000000..332be95 --- /dev/null +++ b/ansible/roles/profile_atlas/templates/atlas-zfs-snapshot@.service.j2 @@ -0,0 +1,25 @@ +[Unit] +Description=Create and retain %i ZFS snapshots for {{ atlas_zfs_pool }} +Documentation=man:zfs-snapshot(8) man:zfs-destroy(8) +Requires=zfs.target +After=zfs.target +ConditionFileIsExecutable=/usr/local/sbin/atlas-zfs-snapshot + +[Service] +Type=oneshot +ExecStart=/usr/local/sbin/atlas-zfs-snapshot %i +User=root +Group=root +UMask=0077 +Nice=10 +IOSchedulingClass=best-effort +IOSchedulingPriority=7 +NoNewPrivileges=true +PrivateTmp=true +ProtectHome=true +ProtectSystem=strict +ProtectKernelTunables=true +ProtectKernelModules=true +ProtectControlGroups=true +RestrictRealtime=true +LockPersonality=true