Manage Atlas ZFS snapshots and scrubs

AGENTS.md

@@ -59,6 +59,8 @@ Ansible-driven personal infrastructure repo for Fedora and Void desktops, Fedora
     `ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff`
   - Atlas network/share hardening:
     `ansible-playbook ansible/site.yml --limit atlas --tags hardening,sharing --check --diff`
+  - Atlas ZFS snapshot retention and scrub timers:
+    `ansible-playbook ansible/site.yml --limit atlas --tags snapshots,scrub --check --diff`
   - Prometheus/Aegis WireGuard gateway:
     `ansible-playbook ansible/site.yml --limit prometheus,aegis --tags wireguard --check --diff`
   - DuckDNS config only: `ansible-playbook ansible/site.yml --limit prometheus --tags duckdns --check --diff`
@@ -163,35 +165,51 @@ The dotfile vars follow the same split: `desktop_common_dotfiles` carries mode-i
 
 ## Atlas NAS TODO
 Completed validation: the existing RAIDZ2 pool and datasets, SELinux, LAN firewall, SSH, Cockpit with
-the selected 45Drives plugins, encrypted SMB3 `Archive`, the Aegis-only NFSv4 `photobook` export, and
+the selected 45Drives plugins, encrypted SMB3 `Archive`, the Aegis-only NFSv4 `photobook` export, and the
-the former Prometheus--Atlas WireGuard path were operational. Aegis has validated NFSv4.2 read, write, delete,
+Prometheus--Aegis WireGuard gateway are operational. The gateway handshake, forwarding, source masquerading,
-and `all_squash` mapping to UID/GID `1100` end-to-end.
+and TCP reachability to Atlas were verified. Temporary Navidrome and Syncthing are available through their
-- Validate the Prometheus--Aegis WireGuard gateway after migration: peer handshake and counters, Aegis IPv4
+manual NPM Proxy Hosts; Syncthing uses `/data/Org` backed by the SMB-shared Archive dataset. Aegis has also
-  forwarding and masquerading, and an NPM request from Prometheus to an Atlas LAN address. Add the Uranus VIP to
+validated NFSv4.2 read, write, delete, and `all_squash` mapping to UID/GID `1100` end-to-end. The ZFS
-  Prometheus' Aegis peer when the cluster control plane is assigned.
+snapshot timers are active and the first recursive hourly snapshot completed successfully; the first
-- Validate temporary Atlas Navidrome and Syncthing through Aegis before creating their NPM Proxy Hosts.
+scheduled retention prune and monthly scrub remain runtime checks.
-  Keep NPM host configuration manual; plan their eventual Uranus migration with storage and routing declared
+
-  separately from the NAS baseline.
+### Priority 1 - Data protection
-- Decide whether a common SMB/NFS namespace is required. `Archive` (SMB) and `photobook` (NFS) are
+- [x] Deploy Ansible-managed recursive ZFS snapshots with 24 hourly, 30 daily, 8 weekly, and 12 monthly
-  intentionally distinct today; only if a shared namespace is selected, finalize its UID/GID, group,
+  generations, plus a monthly scrub on the first Sunday at 03:00. The timers and first hourly snapshot were
-  and POSIX ACL model and test the same files through both protocols.
+  verified on Atlas. Still observe the first scheduled retention prune and scrub; Cockpit Scheduler is for
-- Keep `atlas_manage_media_stack` disabled until the future Immich deployment has validated `/dev/dri`,
+  visibility or manual operations only, and snapshot rollback is never automated.
-  container paths, and the required Vault database secret.
+- [ ] Add the encrypted offsite backup with Borg to a Hetzner Storage Box: use a dedicated SSH identity,
-- Add Ansible-managed ZFS snapshot retention and scrub timers. Use Cockpit Scheduler for visibility
-  or manual operations, not as the only source of configuration, and never automate snapshot rollback.
-- Add the Atlas-initiated least-privilege Prometheus backup pull: Prometheus exposes only prepared
-  read-only dumps through a dedicated account and Atlas retains the private SSH key, pinned host key,
-  atomic pull, verification, retention and systemd service/timer.
-- Add the encrypted offsite backup with Borg to a Hetzner Storage Box: use a dedicated SSH identity,
   pin the host key, keep Borg repository credentials and encryption material in Vault, use
   snapshot-consistent sources, and manage retries, logging, pruning, repository checks and restores.
-- Add the UUID-bound offline USB backup with versioned rsync, locking, capacity checks, verification,
+- [ ] Add the UUID-bound offline USB backup with versioned rsync, locking, capacity checks, verification,
   safe unmounting and a tested restore procedure; never trigger it for an arbitrary USB disk.
-- Add monitoring and alerting for pool health, scrub/resilver, SMART data, temperatures, free space and
+- [ ] Test restores independently from a ZFS snapshot, Borg, and the offline USB backup before relying on
-  failed backup timers, plus a controlled Rocky kernel/OpenZFS update and reboot procedure.
+  any backup path.
-- Document and test disaster recovery: rebuild Atlas with Ansible, import the existing pool, restore
+- [ ] Add monitoring and alerting for pool health, scrub/resilver, SMART data, temperatures, free space,
+  snapshot/backup capacity growth, and failed maintenance or backup timers.
+
+### Priority 2 - NAS operability and recovery
+- [ ] Document and test disaster recovery: rebuild Atlas with Ansible, import the existing pool, restore
   from snapshot/USB/Hetzner, preserve Vault and Borg recovery material offline, and define RPO/RTO.
-- Optionally design iCloud photo ingestion and an Aegis persistent NFS mount as a separate workflow
+- [ ] Define a controlled Rocky kernel/OpenZFS update and reboot procedure.
+- [ ] Add the Atlas-initiated least-privilege Prometheus backup pull: Prometheus exposes only prepared
+  read-only dumps through a dedicated account and Atlas retains the private SSH key, pinned host key,
+  atomic pull, verification, retention and systemd service/timer.
+- [ ] Decide whether a common SMB/NFS namespace is required. `Archive` (SMB) and `photobook` (NFS) are
+  intentionally distinct today; only if a shared namespace is selected, finalize its UID/GID, group,
+  and POSIX ACL model and test the same files through both protocols.
+
+### Priority 3 - Service expansion
+- [ ] After data protection and recovery are validated, populate `/zpool/media/music` and validate Navidrome.
+- [ ] Design and deploy Nextcloud as another explicitly temporary Atlas service before Uranus. Give it
+  separate persistent application, database, and cache storage; keep credentials in Vault; publish it only
+  through NPM over the Prometheus--Aegis gateway; and define backup, upgrade, and eventual Uranus-migration
+  procedures before exposing user data. Do not deploy Nextcloud before the data-protection checklist is complete.
+- [ ] Keep `atlas_manage_media_stack` disabled until the future Immich deployment has validated `/dev/dri`,
+  container paths, and the required Vault database secret.
+
+### Priority 4 - Optional workflows
+- [ ] Optionally design iCloud photo ingestion and an Aegis persistent NFS mount as a separate workflow
   after the storage and backup layers are validated; do not make either a dependency of the Atlas
   baseline.
 

README.md

@@ -299,9 +299,27 @@ The first real WireGuard run must include both peers. If Fedora IoT has just lay
 reboot Aegis manually and rerun the command without `--check`; the role then waits for a real peer
 handshake.
 
-Snapshot retention, Syncthing topology, WireGuard/firewall validation, Prometheus backup pulls,
+Atlas declares recursive, systemd-timed ZFS snapshots for the complete pool hierarchy: 24 hourly
-encrypted Borg backups to a Hetzner Storage Box, USB backup, monitoring, and disaster-recovery tests
+snapshots at minute 05, 30 daily snapshots at 00:15, 8 weekly snapshots on Sunday at 01:00, and 12
-remain follow-up work. The detailed operational backlog is kept in `AGENTS.md`.
+monthly snapshots on the first day at 02:00. The retention helper prunes only snapshots carrying its
+managed `atlas-auto` prefix and never rolls back a dataset. The OpenZFS monthly scrub timer is scheduled
+for the first Sunday at 03:00; the conflicting weekly scrub timer is disabled explicitly. The first recursive
+hourly snapshot completed successfully on Atlas; retention pruning and the first scheduled scrub still await
+live runtime evidence. Validate this layer independently with:
+
+```bash
+ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
+ansible-playbook ansible/site.yml --limit atlas --tags snapshots,scrub --check --diff
+```
+
+A temporary Nextcloud deployment on Atlas is also planned before Uranus: it requires separately
+declared persistent application, database, and cache storage, Vault-backed credentials, NPM-only
+publishing through Aegis, and defined backup, upgrade, and eventual migration procedures. Do not deploy
+it before the data-protection checklist is complete.
+
+Prometheus backup pulls, encrypted Borg backups to a Hetzner Storage Box, USB backup, restore testing,
+monitoring, and disaster-recovery tests remain follow-up work. The prioritized operational backlog is kept
+in `AGENTS.md`.
 
 ## How layering works
 

ansible/inventory/host_vars/atlas.yml

@@ -49,6 +49,23 @@ atlas_zfs_backup_reservation: 500G
 atlas_zfs_dataset_photobook: media/photobook
 atlas_mount_root: /zpool
 atlas_manage_storage: true
+atlas_manage_zfs_snapshots: true
+atlas_zfs_snapshot_prefix: atlas-auto
+atlas_zfs_snapshot_policies:
+  - name: hourly
+    calendar: "*-*-* *:05:00"
+    keep: 24
+  - name: daily
+    calendar: "*-*-* 00:15:00"
+    keep: 30
+  - name: weekly
+    calendar: "Sun *-*-* 01:00:00"
+    keep: 8
+  - name: monthly
+    calendar: "*-*-01 02:00:00"
+    keep: 12
+atlas_manage_zfs_scrub: true
+atlas_zfs_scrub_calendar: "Sun *-*-01..07 03:00:00"
 atlas_manage_sharing: true
 atlas_manage_media_stack: false
 

ansible/roles/profile_atlas/defaults/main.yml

@@ -61,6 +61,12 @@ atlas_zfs_backup_reservation: 500G
 atlas_zfs_dataset_photobook: media/photobook
 atlas_mount_root: /CHANGEME_ATLAS_MOUNT_ROOT
 
+atlas_manage_zfs_snapshots: false
+atlas_zfs_snapshot_prefix: atlas-auto
+atlas_zfs_snapshot_policies: []
+atlas_manage_zfs_scrub: false
+atlas_zfs_scrub_calendar: ""
+
 atlas_archive_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_archive }}"
 atlas_services_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_services }}"
 atlas_app_data_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_app_data }}"

ansible/roles/profile_atlas/tasks/main.yml

@@ -14,6 +14,9 @@
 - name: Import Atlas storage tasks
   ansible.builtin.import_tasks: storage.yml
 
+- name: Import Atlas ZFS maintenance tasks
+  ansible.builtin.import_tasks: zfs_maintenance.yml
+
 - name: Import Atlas file sharing tasks
   ansible.builtin.import_tasks: sharing.yml
 

ansible/roles/profile_atlas/tasks/zfs_maintenance.yml

@@ -0,0 +1,175 @@
+---
+- name: Validate Atlas ZFS snapshot policy
+  tags: [atlas, storage, snapshots]
+  ansible.builtin.assert:
+    that:
+      - atlas_manage_storage | bool
+      - atlas_zfs_pool != 'CHANGEME_ZFS_POOL'
+      - atlas_zfs_snapshot_prefix is match('^[a-z0-9][a-z0-9_-]*$')
+      - atlas_zfs_snapshot_policies | length > 0
+      - >-
+        (atlas_zfs_snapshot_policies | map(attribute='name') | unique | list | length)
+        == (atlas_zfs_snapshot_policies | length)
+    fail_msg: >-
+      Enable Atlas storage and declare a non-empty snapshot policy with a safe
+      prefix and unique policy names before managing automatic snapshots.
+  when: atlas_manage_zfs_snapshots | bool
+
+- name: Validate Atlas ZFS snapshot policy entries
+  tags: [atlas, storage, snapshots]
+  ansible.builtin.assert:
+    that:
+      - item.name is match('^[a-z][a-z0-9_-]*$')
+      - item.keep | int > 0
+      - item.calendar | length > 0
+    fail_msg: >-
+      Every Atlas snapshot policy needs a safe name, a positive retention
+      count, and a systemd calendar expression.
+  loop: "{{ atlas_zfs_snapshot_policies }}"
+  loop_control:
+    label: "{{ item.name | default('unnamed') }}"
+  when: atlas_manage_zfs_snapshots | bool
+
+- name: Validate Atlas ZFS snapshot calendars
+  tags: [atlas, storage, snapshots]
+  ansible.builtin.command:
+    argv:
+      - systemd-analyze
+      - calendar
+      - "{{ item.calendar }}"
+  loop: "{{ atlas_zfs_snapshot_policies }}"
+  loop_control:
+    label: "{{ item.name }}: {{ item.calendar }}"
+  changed_when: false
+  check_mode: false
+  when: atlas_manage_zfs_snapshots | bool
+
+- name: Install Atlas ZFS snapshot and retention helper
+  tags: [atlas, storage, snapshots]
+  ansible.builtin.template:
+    src: atlas-zfs-snapshot.sh.j2
+    dest: /usr/local/sbin/atlas-zfs-snapshot
+    owner: root
+    group: root
+    mode: "0750"
+  when: atlas_manage_zfs_snapshots | bool
+
+- name: Install Atlas ZFS snapshot systemd service
+  tags: [atlas, storage, snapshots]
+  ansible.builtin.template:
+    src: atlas-zfs-snapshot@.service.j2
+    dest: /etc/systemd/system/atlas-zfs-snapshot@.service
+    owner: root
+    group: root
+    mode: "0644"
+  when: atlas_manage_zfs_snapshots | bool
+
+- name: Install Atlas ZFS snapshot systemd timers
+  tags: [atlas, storage, snapshots]
+  ansible.builtin.template:
+    src: atlas-zfs-snapshot.timer.j2
+    dest: "/etc/systemd/system/atlas-zfs-snapshot-{{ item.name }}.timer"
+    owner: root
+    group: root
+    mode: "0644"
+  loop: "{{ atlas_zfs_snapshot_policies }}"
+  loop_control:
+    label: "{{ item.name }}"
+  when: atlas_manage_zfs_snapshots | bool
+
+- name: Enable Atlas ZFS snapshot systemd timers
+  tags: [atlas, storage, snapshots]
+  ansible.builtin.systemd:
+    name: "atlas-zfs-snapshot-{{ item.name }}.timer"
+    enabled: true
+    state: started
+    daemon_reload: true
+  loop: "{{ atlas_zfs_snapshot_policies }}"
+  loop_control:
+    label: "{{ item.name }}"
+  when:
+    - atlas_manage_zfs_snapshots | bool
+    - not ansible_check_mode
+
+- name: Validate Atlas ZFS scrub policy
+  tags: [atlas, storage, scrub]
+  ansible.builtin.assert:
+    that:
+      - atlas_manage_storage | bool
+      - atlas_zfs_pool != 'CHANGEME_ZFS_POOL'
+      - atlas_zfs_scrub_calendar | length > 0
+    fail_msg: >-
+      Enable Atlas storage and declare a systemd calendar expression before
+      managing periodic ZFS scrubs.
+  when: atlas_manage_zfs_scrub | bool
+
+- name: Validate Atlas ZFS scrub calendar
+  tags: [atlas, storage, scrub]
+  ansible.builtin.command:
+    argv:
+      - systemd-analyze
+      - calendar
+      - "{{ atlas_zfs_scrub_calendar }}"
+  changed_when: false
+  check_mode: false
+  when: atlas_manage_zfs_scrub | bool
+
+- name: Require OpenZFS scrub systemd units
+  tags: [atlas, storage, scrub]
+  ansible.builtin.command:
+    argv:
+      - systemctl
+      - cat
+      - "{{ item }}"
+  loop:
+    - "zfs-scrub@{{ atlas_zfs_pool }}.service"
+    - "zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer"
+    - "zfs-scrub-weekly@{{ atlas_zfs_pool }}.timer"
+  loop_control:
+    label: "{{ item }}"
+  changed_when: false
+  check_mode: false
+  when: atlas_manage_zfs_scrub | bool
+
+- name: Create Atlas ZFS scrub timer override directory
+  tags: [atlas, storage, scrub]
+  ansible.builtin.file:
+    path: "/etc/systemd/system/zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer.d"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+  when: atlas_manage_zfs_scrub | bool
+
+- name: Configure Atlas ZFS monthly scrub schedule
+  tags: [atlas, storage, scrub]
+  ansible.builtin.template:
+    src: atlas-zfs-scrub-timer.conf.j2
+    dest: >-
+      /etc/systemd/system/zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer.d/override.conf
+    owner: root
+    group: root
+    mode: "0644"
+  when: atlas_manage_zfs_scrub | bool
+
+- name: Disable the conflicting weekly OpenZFS scrub timer
+  tags: [atlas, storage, scrub]
+  ansible.builtin.systemd:
+    name: "zfs-scrub-weekly@{{ atlas_zfs_pool }}.timer"
+    enabled: false
+    state: stopped
+    daemon_reload: true
+  when:
+    - atlas_manage_zfs_scrub | bool
+    - not ansible_check_mode
+
+- name: Enable the Atlas monthly OpenZFS scrub timer
+  tags: [atlas, storage, scrub]
+  ansible.builtin.systemd:
+    name: "zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer"
+    enabled: true
+    state: started
+    daemon_reload: true
+  when:
+    - atlas_manage_zfs_scrub | bool
+    - not ansible_check_mode

ansible/roles/profile_atlas/templates/atlas-zfs-scrub-timer.conf.j2

@@ -0,0 +1,6 @@
+[Timer]
+OnCalendar=
+OnCalendar={{ atlas_zfs_scrub_calendar }}
+Persistent=true
+RandomizedDelaySec=0
+AccuracySec=1min

ansible/roles/profile_atlas/templates/atlas-zfs-snapshot.sh.j2

@@ -0,0 +1,72 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+export LC_ALL=C
+export PATH=/usr/sbin:/usr/bin:/sbin:/bin
+
+if [[ $# -ne 1 ]]; then
+  printf 'Usage: %s <policy>\n' "$0" >&2
+  exit 64
+fi
+
+readonly pool={{ atlas_zfs_pool | quote }}
+readonly prefix={{ atlas_zfs_snapshot_prefix | quote }}
+readonly period="$1"
+
+case "$period" in
+{% for policy in atlas_zfs_snapshot_policies %}
+  {{ policy.name | quote }})
+    keep={{ policy.keep | int }}
+    ;;
+{% endfor %}
+  *)
+    printf 'Unknown Atlas ZFS snapshot policy: %s\n' "$period" >&2
+    exit 64
+    ;;
+esac
+readonly keep
+
+zpool list -H -o name "$pool" >/dev/null
+
+exec 9>/run/lock/atlas-zfs-snapshot.lock
+flock 9
+
+timestamp="$(date -u +%Y%m%dT%H%M%SZ)"
+readonly timestamp
+readonly snapshot_prefix="${pool}@${prefix}-${period}-"
+readonly snapshot="${snapshot_prefix}${timestamp}"
+
+zfs snapshot -r "$snapshot"
+printf 'Created recursive ZFS snapshot %s\n' "$snapshot"
+
+snapshot_listing="$(zfs list -H -t snapshot -o name -s creation -r "$pool")"
+managed_snapshots=()
+while IFS= read -r snapshot_name; do
+  if [[ "$snapshot_name" == "$snapshot_prefix"* ]]; then
+    snapshot_suffix="${snapshot_name#"$snapshot_prefix"}"
+    if [[ "$snapshot_suffix" =~ ^[0-9]{8}T[0-9]{6}Z$ ]]; then
+      managed_snapshots+=("$snapshot_name")
+    fi
+  fi
+done <<< "$snapshot_listing"
+
+{% raw %}
+managed_snapshot_count="${#managed_snapshots[@]}"
+{% endraw %}
+prune_count=$((managed_snapshot_count - keep))
+if ((prune_count <= 0)); then
+  printf 'Retaining %d of %d managed %s snapshots\n' \
+    "$managed_snapshot_count" "$keep" "$period"
+  exit 0
+fi
+
+for ((index = 0; index < prune_count; index++)); do
+  candidate="${managed_snapshots[$index]}"
+  if [[ "$candidate" != "$snapshot_prefix"* ]]; then
+    printf 'Refusing to destroy unexpected snapshot: %s\n' "$candidate" >&2
+    exit 65
+  fi
+
+  zfs destroy -r "$candidate"
+  printf 'Pruned recursive ZFS snapshot %s\n' "$candidate"
+done

ansible/roles/profile_atlas/templates/atlas-zfs-snapshot.timer.j2

@@ -0,0 +1,11 @@
+[Unit]
+Description=Schedule {{ item.name }} ZFS snapshots for {{ atlas_zfs_pool }}
+
+[Timer]
+OnCalendar={{ item.calendar }}
+Persistent=true
+AccuracySec=1min
+Unit=atlas-zfs-snapshot@{{ item.name }}.service
+
+[Install]
+WantedBy=timers.target

ansible/roles/profile_atlas/templates/atlas-zfs-snapshot@.service.j2

@@ -0,0 +1,25 @@
+[Unit]
+Description=Create and retain %i ZFS snapshots for {{ atlas_zfs_pool }}
+Documentation=man:zfs-snapshot(8) man:zfs-destroy(8)
+Requires=zfs.target
+After=zfs.target
+ConditionFileIsExecutable=/usr/local/sbin/atlas-zfs-snapshot
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/atlas-zfs-snapshot %i
+User=root
+Group=root
+UMask=0077
+Nice=10
+IOSchedulingClass=best-effort
+IOSchedulingPriority=7
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectHome=true
+ProtectSystem=strict
+ProtectKernelTunables=true
+ProtectKernelModules=true
+ProtectControlGroups=true
+RestrictRealtime=true
+LockPersonality=true