Compare commits

...

3 Commits

Author SHA1 Message Date
Fabio Scotto di Santolo
e837b0059b Document validated Atlas Borg backup activation 2026-09-18 10:09:12 +02:00
Fabio Scotto di Santolo
7e498514dd Add encrypted Borg backups for Atlas 2026-09-18 10:00:45 +02:00
Fabio Scotto di Santolo
e7836ea25f Manage Atlas ZFS snapshots and scrubs 2026-09-17 23:10:53 +02:00
22 changed files with 1404 additions and 107 deletions

View File

@@ -59,6 +59,10 @@ Ansible-driven personal infrastructure repo for Fedora and Void desktops, Fedora
`ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff`
- Atlas network/share hardening:
`ansible-playbook ansible/site.yml --limit atlas --tags hardening,sharing --check --diff`
- Atlas ZFS snapshot retention and scrub timers:
`ansible-playbook ansible/site.yml --limit atlas --tags snapshots,scrub --check --diff`
- Atlas encrypted Borg backup:
`ansible-playbook ansible/site.yml --limit atlas --tags packages,borg --check --diff`
- Prometheus/Aegis WireGuard gateway:
`ansible-playbook ansible/site.yml --limit prometheus,aegis --tags wireguard --check --diff`
- DuckDNS config only: `ansible-playbook ansible/site.yml --limit prometheus --tags duckdns --check --diff`
@@ -163,35 +167,55 @@ The dotfile vars follow the same split: `desktop_common_dotfiles` carries mode-i
## Atlas NAS TODO
Completed validation: the existing RAIDZ2 pool and datasets, SELinux, LAN firewall, SSH, Cockpit with
the selected 45Drives plugins, encrypted SMB3 `Archive`, the Aegis-only NFSv4 `photobook` export, and
the former Prometheus--Atlas WireGuard path were operational. Aegis has validated NFSv4.2 read, write, delete,
and `all_squash` mapping to UID/GID `1100` end-to-end.
- Validate the Prometheus--Aegis WireGuard gateway after migration: peer handshake and counters, Aegis IPv4
forwarding and masquerading, and an NPM request from Prometheus to an Atlas LAN address. Add the Uranus VIP to
Prometheus' Aegis peer when the cluster control plane is assigned.
- Validate temporary Atlas Navidrome and Syncthing through Aegis before creating their NPM Proxy Hosts.
Keep NPM host configuration manual; plan their eventual Uranus migration with storage and routing declared
separately from the NAS baseline.
- Decide whether a common SMB/NFS namespace is required. `Archive` (SMB) and `photobook` (NFS) are
intentionally distinct today; only if a shared namespace is selected, finalize its UID/GID, group,
and POSIX ACL model and test the same files through both protocols.
- Keep `atlas_manage_media_stack` disabled until the future Immich deployment has validated `/dev/dri`,
container paths, and the required Vault database secret.
- Add Ansible-managed ZFS snapshot retention and scrub timers. Use Cockpit Scheduler for visibility
or manual operations, not as the only source of configuration, and never automate snapshot rollback.
- Add the Atlas-initiated least-privilege Prometheus backup pull: Prometheus exposes only prepared
the selected 45Drives plugins, encrypted SMB3 `Archive`, the Aegis-only NFSv4 `photobook` export, and the
Prometheus--Aegis WireGuard gateway are operational. The gateway handshake, forwarding, source masquerading,
and TCP reachability to Atlas were verified. Temporary Navidrome and Syncthing are available through their
manual NPM Proxy Hosts; Syncthing uses `/data/Org` backed by the SMB-shared Archive dataset. Aegis has also
validated NFSv4.2 read, write, delete, and `all_squash` mapping to UID/GID `1100` end-to-end. The ZFS
snapshot timers are active and the first recursive hourly snapshot completed successfully; the first
scheduled retention prune and monthly scrub remain runtime checks.
### Priority 1 - Data protection
- [x] Deploy Ansible-managed recursive ZFS snapshots with 24 hourly, 30 daily, 8 weekly, and 12 monthly
generations, plus a monthly scrub on the first Sunday at 03:00. The timers and first hourly snapshot were
verified on Atlas. Still observe the first scheduled retention prune and scrub; Cockpit Scheduler is for
visibility or manual operations only, and snapshot rollback is never automated.
- [x] Activate and validate the encrypted offsite Borg backup to the Hetzner Storage Box. Atlas uses the
dedicated SSH identity, pinned ED25519 host key, Vault-backed `repokey` encryption, and a locked
non-login `borg` account with no sudo or supplementary groups. The initial snapshot-consistent backup,
Borg repository check, and temporary-directory restore completed successfully; the restored `Archive`
tree matched the live data, and temporary snapshots and mounts were removed. The exported recovery key
was copied offline. Daily backup retries and logging, 30 daily, 8 weekly and 12 monthly archives,
compaction, and monthly repository checks are enabled.
- [ ] Add the UUID-bound offline USB backup with versioned rsync, locking, capacity checks, verification,
safe unmounting and a tested restore procedure; never trigger it for an arbitrary USB disk.
- [ ] Test restores independently from a ZFS snapshot, Borg, and the offline USB backup before relying on
any backup path.
- [ ] Add monitoring and alerting for pool health, scrub/resilver, SMART data, temperatures, free space,
snapshot/backup capacity growth, and failed maintenance or backup timers.
### Priority 2 - NAS operability and recovery
- [ ] Document and test disaster recovery: rebuild Atlas with Ansible, import the existing pool, restore
from snapshot/USB/Hetzner, preserve Vault and Borg recovery material offline, and define RPO/RTO.
- [ ] Define a controlled Rocky kernel/OpenZFS update and reboot procedure.
- [ ] Add the Atlas-initiated least-privilege Prometheus backup pull: Prometheus exposes only prepared
read-only dumps through a dedicated account and Atlas retains the private SSH key, pinned host key,
atomic pull, verification, retention and systemd service/timer.
- Add the encrypted offsite backup with Borg to a Hetzner Storage Box: use a dedicated SSH identity,
pin the host key, keep Borg repository credentials and encryption material in Vault, use
snapshot-consistent sources, and manage retries, logging, pruning, repository checks and restores.
- Add the UUID-bound offline USB backup with versioned rsync, locking, capacity checks, verification,
safe unmounting and a tested restore procedure; never trigger it for an arbitrary USB disk.
- Add monitoring and alerting for pool health, scrub/resilver, SMART data, temperatures, free space and
failed backup timers, plus a controlled Rocky kernel/OpenZFS update and reboot procedure.
- Document and test disaster recovery: rebuild Atlas with Ansible, import the existing pool, restore
from snapshot/USB/Hetzner, preserve Vault and Borg recovery material offline, and define RPO/RTO.
- Optionally design iCloud photo ingestion and an Aegis persistent NFS mount as a separate workflow
- [ ] Decide whether a common SMB/NFS namespace is required. `Archive` (SMB) and `photobook` (NFS) are
intentionally distinct today; only if a shared namespace is selected, finalize its UID/GID, group,
and POSIX ACL model and test the same files through both protocols.
### Priority 3 - Service expansion
- [ ] After data protection and recovery are validated, populate `/zpool/media/music` and validate Navidrome.
- [ ] Design and deploy Nextcloud as another explicitly temporary Atlas service before Uranus. Give it
separate persistent application, database, and cache storage; keep credentials in Vault; publish it only
through NPM over the Prometheus--Aegis gateway; and define backup, upgrade, and eventual Uranus-migration
procedures before exposing user data. Do not deploy Nextcloud before the data-protection checklist is complete.
- [ ] Keep `atlas_manage_media_stack` disabled until the future Immich deployment has validated `/dev/dri`,
container paths, and the required Vault database secret.
### Priority 4 - Optional workflows
- [ ] Optionally design iCloud photo ingestion and an Aegis persistent NFS mount as a separate workflow
after the storage and backup layers are validated; do not make either a dependency of the Atlas
baseline.

View File

@@ -299,9 +299,65 @@ The first real WireGuard run must include both peers. If Fedora IoT has just lay
reboot Aegis manually and rerun the command without `--check`; the role then waits for a real peer
handshake.
Snapshot retention, Syncthing topology, WireGuard/firewall validation, Prometheus backup pulls,
encrypted Borg backups to a Hetzner Storage Box, USB backup, monitoring, and disaster-recovery tests
remain follow-up work. The detailed operational backlog is kept in `AGENTS.md`.
Atlas declares recursive, systemd-timed ZFS snapshots for the complete pool hierarchy: 24 hourly
snapshots at minute 05, 30 daily snapshots at 00:15, 8 weekly snapshots on Sunday at 01:00, and 12
monthly snapshots on the first day at 02:00. The retention helper prunes only snapshots carrying its
managed `atlas-auto` prefix and never rolls back a dataset. The OpenZFS monthly scrub timer is scheduled
for the first Sunday at 03:00; the conflicting weekly scrub timer is disabled explicitly. The first recursive
hourly snapshot completed successfully on Atlas; retention pruning and the first scheduled scrub still await
live runtime evidence. Validate this layer independently with:
```bash
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit atlas --tags snapshots,scrub --check --diff
```
Atlas also declares an encrypted Borg backup to the dedicated Hetzner Storage Box sub-account
`u660064-sub1`. The repository is the sub-account-relative `./borg-data` path and uses the explicitly
selected remote Borg 1.4 binary over SSH port 23. The ED25519 server key is pinned; a dedicated client
key is generated for the locked, non-login `borg` system account, and its private half never leaves
`/etc/atlas-borg`. The account has no sudo or supplementary groups and owns only its SSH identity,
passphrase, cache, and Borg state. Borg receives its passphrase through a mode `0600` file rendered from
`vault_atlas_borg_passphrase`.
The daily backup starts at 04:30 with up to 30 minutes of randomized delay. It creates a temporary,
recursive ZFS snapshot and reconstructs every dataset below `/zpool` as a read-only bind-mounted tree,
so parent and child datasets enter one consistent Borg archive. Cleanup always removes the temporary
mounts and managed snapshot. Only the root wrapper performs snapshot and mount operations; it launches
the Borg client as `borg` with temporary read-search capability and no ZFS, sudo, or pool-management
privileges. Borg retains 30 daily, 8 weekly, and 12 monthly archives, then compacts the standard
read-write repository. A full metadata and repository check runs as `borg` on the fifteenth day of each
month at 06:00. Both operations use a common lock, journal logging, and bounded systemd retries.
Initial activation remains explicit:
1. Add a strong unique `vault_atlas_borg_passphrase` with `ansible-vault edit secrets/vault.yml`.
2. Generate and display only the dedicated public key with
`ansible-playbook ansible/site.yml --limit atlas --tags borg_key`.
3. Install that public key in the Hetzner sub-account, then apply with
`ansible-playbook ansible/site.yml --limit atlas --tags packages,borg`.
4. Copy the ignored `secrets/recovery/atlas-borg-repokey.export` file to genuinely offline storage.
The controller-side copy is not an offline backup by itself.
The role initializes only the missing `repokey` repository and never accepts an unpinned host key or
password authentication. It does not start the first backup manually. Validate the rendered state with:
```bash
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit atlas --tags packages,borg --check --diff
```
Atlas runtime activation is complete: the initial backup and repository check succeeded, a full restore
to a temporary directory was validated against the live `Archive` tree, the recovery-key export was copied
to offline storage, and the temporary snapshot and bind mounts were cleaned up.
A temporary Nextcloud deployment on Atlas is also planned before Uranus: it requires separately
declared persistent application, database, and cache storage, Vault-backed credentials, NPM-only
publishing through Aegis, and defined backup, upgrade, and eventual migration procedures. Do not deploy
it before the data-protection checklist is complete.
Prometheus backup pulls, USB backup, monitoring, and disaster-recovery tests remain follow-up work. The
prioritized operational backlog is kept in `AGENTS.md`.
## How layering works

View File

@@ -49,6 +49,40 @@ atlas_zfs_backup_reservation: 500G
atlas_zfs_dataset_photobook: media/photobook
atlas_mount_root: /zpool
atlas_manage_storage: true
atlas_manage_zfs_snapshots: true
atlas_zfs_snapshot_prefix: atlas-auto
atlas_zfs_snapshot_policies:
- name: hourly
calendar: "*-*-* *:05:00"
keep: 24
- name: daily
calendar: "*-*-* 00:15:00"
keep: 30
- name: weekly
calendar: "Sun *-*-* 01:00:00"
keep: 8
- name: monthly
calendar: "*-*-01 02:00:00"
keep: 12
atlas_manage_zfs_scrub: true
atlas_zfs_scrub_calendar: "Sun *-*-01..07 03:00:00"
atlas_manage_borg_backup: true
atlas_borg_repository_host: u660064-sub1.your-storagebox.de
atlas_borg_repository_user: u660064-sub1
atlas_borg_repository_port: 23
atlas_borg_repository_path: ./borg-data
atlas_borg_remote_path: borg-1.4
# Verified against Hetzner's published ED25519 fingerprint on 2026-09-17:
# SHA256:XqONwb1S0zuj5A1CDxpOSuD2hnAArV1A3wKY7Z3sdgM
atlas_borg_host_key: >-
[u660064-sub1.your-storagebox.de]:23 ssh-ed25519
AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs
atlas_borg_backup_calendar: "*-*-* 04:30:00"
atlas_borg_check_calendar: "*-*-15 06:00:00"
atlas_borg_randomized_delay: 30m
atlas_borg_keep_daily: 30
atlas_borg_keep_weekly: 8
atlas_borg_keep_monthly: 12
atlas_manage_sharing: true
atlas_manage_media_stack: false
@@ -76,6 +110,7 @@ host_packages:
- samba
- samba-client
- samba-common-tools
- borgbackup
- zfs
atlas_nfs_exports:

View File

@@ -61,6 +61,42 @@ atlas_zfs_backup_reservation: 500G
atlas_zfs_dataset_photobook: media/photobook
atlas_mount_root: /CHANGEME_ATLAS_MOUNT_ROOT
atlas_manage_zfs_snapshots: false
atlas_zfs_snapshot_prefix: atlas-auto
atlas_zfs_snapshot_policies: []
atlas_manage_zfs_scrub: false
atlas_zfs_scrub_calendar: ""
atlas_manage_borg_backup: false
atlas_borg_username: borg
atlas_borg_group: borg
atlas_borg_home: /var/lib/atlas-borg
atlas_borg_repository_host: CHANGEME_BORG_HOST
atlas_borg_repository_user: CHANGEME_BORG_USER
atlas_borg_repository_port: 23
atlas_borg_repository_path: ./borg-data
atlas_borg_remote_path: borg-1.4
atlas_borg_host_key: ""
atlas_borg_ssh_private_key_path: /etc/atlas-borg/id_ed25519
atlas_borg_known_hosts_path: /etc/atlas-borg/known_hosts
atlas_borg_passphrase_path: /etc/atlas-borg/passphrase
atlas_borg_ssh_wrapper_path: /usr/local/libexec/atlas-borg-ssh
atlas_borg_passphrase: "{{ vault_atlas_borg_passphrase | default('') }}"
atlas_borg_encryption_mode: repokey
atlas_borg_archive_prefix: atlas
atlas_borg_snapshot_prefix: atlas-borg
atlas_borg_compression: auto,zstd,3
atlas_borg_backup_calendar: ""
atlas_borg_check_calendar: ""
atlas_borg_randomized_delay: 30m
atlas_borg_keep_daily: 30
atlas_borg_keep_weekly: 8
atlas_borg_keep_monthly: 12
atlas_borg_config_dir: /var/lib/atlas-borg
atlas_borg_cache_dir: /var/cache/atlas-borg
atlas_borg_lock_path: /var/lib/atlas-borg/backup.lock
atlas_borg_recovery_export_path: "{{ playbook_dir }}/../secrets/recovery/atlas-borg-repokey.export"
atlas_archive_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_archive }}"
atlas_services_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_services }}"
atlas_app_data_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_app_data }}"

View File

@@ -28,6 +28,18 @@
name: smb
state: restarted
- name: Restart Atlas Borg timers
ansible.builtin.systemd:
name: "{{ item }}"
state: restarted
daemon_reload: true
loop:
- atlas-borg-backup.timer
- atlas-borg-check.timer
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Restart Atlas media Quadlets
ansible.builtin.systemd:
name: "{{ item }}"

View File

@@ -0,0 +1,518 @@
---
- name: Validate Atlas Borg backup configuration
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.assert:
that:
- atlas_manage_storage | bool
- atlas_zfs_pool != 'CHANGEME_ZFS_POOL'
- atlas_mount_root.startswith('/')
- atlas_borg_username is match('^[a-z_][a-z0-9_-]*$')
- atlas_borg_group is match('^[a-z_][a-z0-9_-]*$')
- atlas_borg_username not in ['root', atlas_admin_username]
- atlas_borg_group != 'wheel'
- atlas_borg_home.startswith('/var/lib/')
- atlas_borg_repository_host is match('^[A-Za-z0-9.-]+$')
- atlas_borg_repository_user is match('^[A-Za-z0-9_-]+$')
- atlas_borg_repository_port | int > 0
- atlas_borg_repository_port | int < 65536
- atlas_borg_repository_path is match('^\./[A-Za-z0-9][A-Za-z0-9._/-]*$')
- "'/../' not in ('/' ~ atlas_borg_repository_path ~ '/')"
- atlas_borg_remote_path is match('^borg-[0-9]+\.[0-9]+$')
- atlas_borg_host_key.startswith(
'[' ~ atlas_borg_repository_host ~ ']:' ~ (atlas_borg_repository_port | string) ~ ' ssh-ed25519 '
)
- atlas_borg_ssh_private_key_path.startswith('/etc/atlas-borg/')
- atlas_borg_known_hosts_path.startswith('/etc/atlas-borg/')
- atlas_borg_passphrase_path.startswith('/etc/atlas-borg/')
- atlas_borg_ssh_wrapper_path.startswith('/usr/local/libexec/')
- atlas_borg_encryption_mode == 'repokey'
- atlas_borg_archive_prefix is match('^[a-z0-9][a-z0-9_-]*$')
- atlas_borg_snapshot_prefix is match('^[a-z0-9][a-z0-9_-]*$')
- atlas_borg_keep_daily | int > 0
- atlas_borg_keep_weekly | int > 0
- atlas_borg_keep_monthly | int > 0
fail_msg: >-
Atlas Borg needs a safe relative repository path, a pinned ED25519 host
key, positive retention counts, and valid dedicated SSH settings.
when: atlas_manage_borg_backup | bool
- name: Create the Atlas Borg system group
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.group:
name: "{{ atlas_borg_group }}"
system: true
state: present
when: atlas_manage_borg_backup | bool
- name: Create the least-privilege Atlas Borg account
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.user:
name: "{{ atlas_borg_username }}"
group: "{{ atlas_borg_group }}"
groups: []
append: false
comment: Atlas Borg backup service
home: "{{ atlas_borg_home }}"
create_home: false
shell: /sbin/nologin
password_lock: true
system: true
state: present
when: atlas_manage_borg_backup | bool
- name: Read Atlas Borg account group membership
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.command:
argv:
- id
- -nG
- "{{ atlas_borg_username }}"
register: atlas_borg_account_groups
changed_when: false
check_mode: false
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Require the Atlas Borg account to have no supplementary groups
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.assert:
that:
- atlas_borg_account_groups.stdout.split() == [atlas_borg_group]
fail_msg: >-
The Atlas Borg service account must belong only to its private primary
group and must never receive wheel or other supplementary membership.
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Validate Atlas Borg systemd calendars
tags: [atlas, storage, backup, borg]
ansible.builtin.command:
argv:
- systemd-analyze
- calendar
- "{{ item }}"
loop:
- "{{ atlas_borg_backup_calendar }}"
- "{{ atlas_borg_check_calendar }}"
changed_when: false
check_mode: false
when: atlas_manage_borg_backup | bool
- name: Create Atlas Borg configuration directory
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.file:
path: /etc/atlas-borg
state: directory
owner: root
group: "{{ atlas_borg_group }}"
mode: "0750"
when: atlas_manage_borg_backup | bool
- name: Generate the dedicated Atlas Borg SSH identity
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.command:
argv:
- ssh-keygen
- -q
- -t
- ed25519
- -N
- ""
- -C
- atlas-borg@atlas
- -f
- "{{ atlas_borg_ssh_private_key_path }}"
creates: "{{ atlas_borg_ssh_private_key_path }}"
when: atlas_manage_borg_backup | bool
- name: Protect the Atlas Borg private SSH identity
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.file:
path: "{{ atlas_borg_ssh_private_key_path }}"
owner: "{{ atlas_borg_username }}"
group: "{{ atlas_borg_group }}"
mode: "0600"
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Set permissions on the Atlas Borg public SSH identity
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.file:
path: "{{ atlas_borg_ssh_private_key_path }}.pub"
owner: "{{ atlas_borg_username }}"
group: "{{ atlas_borg_group }}"
mode: "0644"
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Read the dedicated Atlas Borg public SSH identity
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.slurp:
src: "{{ atlas_borg_ssh_private_key_path }}.pub"
register: atlas_borg_public_key
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Report the public SSH identity to install in the Hetzner sub-account
tags: [atlas, storage, backup, borg, borg_key]
ansible.builtin.debug:
msg: "{{ atlas_borg_public_key.content | b64decode | trim }}"
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Pin the Hetzner Storage Box SSH host key
tags: [atlas, storage, backup, borg]
ansible.builtin.copy:
content: "{{ atlas_borg_host_key }}\n"
dest: "{{ atlas_borg_known_hosts_path }}"
owner: "{{ atlas_borg_username }}"
group: "{{ atlas_borg_group }}"
mode: "0600"
when: atlas_manage_borg_backup | bool
- name: Create Atlas Borg state directories
tags: [atlas, storage, backup, borg]
ansible.builtin.file:
path: "{{ item }}"
state: directory
owner: "{{ atlas_borg_username }}"
group: "{{ atlas_borg_group }}"
mode: "0700"
loop:
- "{{ atlas_borg_config_dir }}"
- "{{ atlas_borg_cache_dir }}"
when: atlas_manage_borg_backup | bool
- name: Create the shared Atlas Borg operation lock
tags: [atlas, storage, backup, borg]
ansible.builtin.copy:
content: ""
dest: "{{ atlas_borg_lock_path }}"
owner: "{{ atlas_borg_username }}"
group: "{{ atlas_borg_group }}"
mode: "0600"
force: false
when: atlas_manage_borg_backup | bool
- name: Require the Atlas Borg encryption passphrase from Vault
tags: [atlas, storage, backup, borg]
ansible.builtin.assert:
that:
- atlas_borg_passphrase | length >= 20
fail_msg: >-
Define vault_atlas_borg_passphrase with a strong unique value in the
encrypted Vault before activating the Borg repository.
no_log: true
when: atlas_manage_borg_backup | bool
- name: Install the Atlas Borg passphrase
tags: [atlas, storage, backup, borg]
ansible.builtin.copy:
content: "{{ atlas_borg_passphrase }}\n"
dest: "{{ atlas_borg_passphrase_path }}"
owner: "{{ atlas_borg_username }}"
group: "{{ atlas_borg_group }}"
mode: "0600"
diff: false
no_log: true
when: atlas_manage_borg_backup | bool
- name: Install the Atlas Borg backup helper
tags: [atlas, storage, backup, borg]
ansible.builtin.template:
src: atlas-borg-backup.sh.j2
dest: /usr/local/sbin/atlas-borg-backup
owner: root
group: root
mode: "0750"
when: atlas_manage_borg_backup | bool
- name: Install the Atlas Borg check helper
tags: [atlas, storage, backup, borg]
ansible.builtin.template:
src: atlas-borg-check.sh.j2
dest: /usr/local/sbin/atlas-borg-check
owner: root
group: "{{ atlas_borg_group }}"
mode: "0750"
when: atlas_manage_borg_backup | bool
- name: Create the local libexec directory for the Atlas Borg SSH wrapper
tags: [atlas, storage, backup, borg]
ansible.builtin.file:
path: "{{ atlas_borg_ssh_wrapper_path | dirname }}"
state: directory
owner: root
group: root
mode: "0755"
when: atlas_manage_borg_backup | bool
- name: Install the capability-dropping Atlas Borg SSH wrapper
tags: [atlas, storage, backup, borg]
ansible.builtin.template:
src: atlas-borg-ssh.sh.j2
dest: "{{ atlas_borg_ssh_wrapper_path }}"
owner: root
group: root
mode: "0755"
when: atlas_manage_borg_backup | bool
- name: Install Atlas Borg systemd units
tags: [atlas, storage, backup, borg]
ansible.builtin.template:
src: "{{ item }}.j2"
dest: "/etc/systemd/system/{{ item }}"
owner: root
group: root
mode: "0644"
loop:
- atlas-borg-backup.service
- atlas-borg-backup.timer
- atlas-borg-check.service
- atlas-borg-check.timer
notify: Restart Atlas Borg timers
when: atlas_manage_borg_backup | bool
- name: Verify dedicated SSH access to the Hetzner Storage Box
tags: [atlas, storage, backup, borg]
ansible.builtin.command:
argv:
- ssh
- -T
- -i
- "{{ atlas_borg_ssh_private_key_path }}"
- -p
- "{{ atlas_borg_repository_port | string }}"
- -o
- BatchMode=yes
- -o
- IdentitiesOnly=yes
- -o
- StrictHostKeyChecking=yes
- -o
- "UserKnownHostsFile={{ atlas_borg_known_hosts_path }}"
- "{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}"
- pwd
register: atlas_borg_ssh_probe
become: true
become_user: "{{ atlas_borg_username }}"
changed_when: false
failed_when: false
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Require the dedicated public key on the Hetzner sub-account
tags: [atlas, storage, backup, borg]
ansible.builtin.assert:
that:
- atlas_borg_ssh_probe.rc == 0
fail_msg: >-
Install the reported Atlas Borg public key in the Hetzner sub-account
before rerunning the Borg tasks. Password authentication is never used.
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Probe the remote Atlas Borg repository path
tags: [atlas, storage, backup, borg]
ansible.builtin.command:
argv:
- ssh
- -T
- -i
- "{{ atlas_borg_ssh_private_key_path }}"
- -p
- "{{ atlas_borg_repository_port | string }}"
- -o
- BatchMode=yes
- -o
- IdentitiesOnly=yes
- -o
- StrictHostKeyChecking=yes
- -o
- "UserKnownHostsFile={{ atlas_borg_known_hosts_path }}"
- "{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}"
- stat
- "{{ atlas_borg_repository_path }}"
register: atlas_borg_repository_path_probe
become: true
become_user: "{{ atlas_borg_username }}"
changed_when: false
failed_when: false
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Probe the Atlas Borg repository
tags: [atlas, storage, backup, borg]
ansible.builtin.command:
argv:
- /usr/bin/borg
- --remote-path
- "{{ atlas_borg_remote_path }}"
- info
- >-
ssh://{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}:
{{- atlas_borg_repository_port }}/{{ atlas_borg_repository_path }}
environment:
BORG_CACHE_DIR: "{{ atlas_borg_cache_dir }}"
BORG_CONFIG_DIR: "{{ atlas_borg_config_dir }}"
BORG_PASSCOMMAND: "cat {{ atlas_borg_passphrase_path }}"
BORG_RSH: "{{ atlas_borg_ssh_wrapper_path }}"
register: atlas_borg_repository_probe
become: true
become_user: "{{ atlas_borg_username }}"
changed_when: false
failed_when: false
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- atlas_borg_repository_path_probe.rc == 0
- name: Reject an existing path that is not the configured Borg repository
tags: [atlas, storage, backup, borg]
ansible.builtin.assert:
that:
- atlas_borg_repository_probe.rc == 0
fail_msg: >-
The remote repository path already exists but Borg could not open it.
Refusing to initialize over existing data; verify the path, passphrase,
and repository state manually.
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- atlas_borg_repository_path_probe.rc == 0
- name: Initialize the encrypted Atlas Borg repository
tags: [atlas, storage, backup, borg]
ansible.builtin.command:
argv:
- /usr/bin/borg
- --remote-path
- "{{ atlas_borg_remote_path }}"
- init
- --encryption
- "{{ atlas_borg_encryption_mode }}"
- >-
ssh://{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}:
{{- atlas_borg_repository_port }}/{{ atlas_borg_repository_path }}
environment:
BORG_CACHE_DIR: "{{ atlas_borg_cache_dir }}"
BORG_CONFIG_DIR: "{{ atlas_borg_config_dir }}"
BORG_PASSCOMMAND: "cat {{ atlas_borg_passphrase_path }}"
BORG_RSH: "{{ atlas_borg_ssh_wrapper_path }}"
become: true
become_user: "{{ atlas_borg_username }}"
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- atlas_borg_repository_path_probe.rc != 0
- name: Verify the encrypted Atlas Borg repository
tags: [atlas, storage, backup, borg]
ansible.builtin.command:
argv:
- /usr/bin/borg
- --remote-path
- "{{ atlas_borg_remote_path }}"
- info
- >-
ssh://{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}:
{{- atlas_borg_repository_port }}/{{ atlas_borg_repository_path }}
environment:
BORG_CACHE_DIR: "{{ atlas_borg_cache_dir }}"
BORG_CONFIG_DIR: "{{ atlas_borg_config_dir }}"
BORG_PASSCOMMAND: "cat {{ atlas_borg_passphrase_path }}"
BORG_RSH: "{{ atlas_borg_ssh_wrapper_path }}"
become: true
become_user: "{{ atlas_borg_username }}"
changed_when: false
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Check for the local Atlas Borg recovery-key export
tags: [atlas, storage, backup, borg]
ansible.builtin.stat:
path: "{{ atlas_borg_recovery_export_path }}"
register: atlas_borg_recovery_export
delegate_to: localhost
become: false
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- name: Export the Atlas Borg recovery key for offline preservation
tags: [atlas, storage, backup, borg]
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode
- not atlas_borg_recovery_export.stat.exists
no_log: true
block:
- name: Create the local recovery-material directory
ansible.builtin.file:
path: "{{ atlas_borg_recovery_export_path | dirname }}"
state: directory
mode: "0700"
delegate_to: localhost
become: false
- name: Export the encrypted Borg repository key on Atlas
ansible.builtin.command:
argv:
- /usr/bin/borg
- --remote-path
- "{{ atlas_borg_remote_path }}"
- key
- export
- >-
ssh://{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}:
{{- atlas_borg_repository_port }}/{{ atlas_borg_repository_path }}
- "{{ atlas_borg_config_dir }}/atlas-borg-repokey.export"
environment:
BORG_CACHE_DIR: "{{ atlas_borg_cache_dir }}"
BORG_CONFIG_DIR: "{{ atlas_borg_config_dir }}"
BORG_PASSCOMMAND: "cat {{ atlas_borg_passphrase_path }}"
BORG_RSH: "{{ atlas_borg_ssh_wrapper_path }}"
become: true
become_user: "{{ atlas_borg_username }}"
- name: Fetch the encrypted Borg recovery key from Atlas
ansible.builtin.fetch:
src: "{{ atlas_borg_config_dir }}/atlas-borg-repokey.export"
dest: "{{ atlas_borg_recovery_export_path }}"
flat: true
- name: Protect the local Borg recovery-key export
ansible.builtin.file:
path: "{{ atlas_borg_recovery_export_path }}"
mode: "0600"
delegate_to: localhost
become: false
always:
- name: Remove the temporary recovery-key export from Atlas
ansible.builtin.file:
path: "{{ atlas_borg_config_dir }}/atlas-borg-repokey.export"
state: absent
- name: Enable Atlas Borg backup and check timers
tags: [atlas, storage, backup, borg]
ansible.builtin.systemd:
name: "{{ item }}"
enabled: true
state: started
daemon_reload: true
loop:
- atlas-borg-backup.timer
- atlas-borg-check.timer
when:
- atlas_manage_borg_backup | bool
- not ansible_check_mode

View File

@@ -14,6 +14,12 @@
- name: Import Atlas storage tasks
ansible.builtin.import_tasks: storage.yml
- name: Import Atlas ZFS maintenance tasks
ansible.builtin.import_tasks: zfs_maintenance.yml
- name: Import Atlas Borg backup tasks
ansible.builtin.import_tasks: borg_backup.yml
- name: Import Atlas file sharing tasks
ansible.builtin.import_tasks: sharing.yml

View File

@@ -0,0 +1,175 @@
---
- name: Validate Atlas ZFS snapshot policy
tags: [atlas, storage, snapshots]
ansible.builtin.assert:
that:
- atlas_manage_storage | bool
- atlas_zfs_pool != 'CHANGEME_ZFS_POOL'
- atlas_zfs_snapshot_prefix is match('^[a-z0-9][a-z0-9_-]*$')
- atlas_zfs_snapshot_policies | length > 0
- >-
(atlas_zfs_snapshot_policies | map(attribute='name') | unique | list | length)
== (atlas_zfs_snapshot_policies | length)
fail_msg: >-
Enable Atlas storage and declare a non-empty snapshot policy with a safe
prefix and unique policy names before managing automatic snapshots.
when: atlas_manage_zfs_snapshots | bool
- name: Validate Atlas ZFS snapshot policy entries
tags: [atlas, storage, snapshots]
ansible.builtin.assert:
that:
- item.name is match('^[a-z][a-z0-9_-]*$')
- item.keep | int > 0
- item.calendar | length > 0
fail_msg: >-
Every Atlas snapshot policy needs a safe name, a positive retention
count, and a systemd calendar expression.
loop: "{{ atlas_zfs_snapshot_policies }}"
loop_control:
label: "{{ item.name | default('unnamed') }}"
when: atlas_manage_zfs_snapshots | bool
- name: Validate Atlas ZFS snapshot calendars
tags: [atlas, storage, snapshots]
ansible.builtin.command:
argv:
- systemd-analyze
- calendar
- "{{ item.calendar }}"
loop: "{{ atlas_zfs_snapshot_policies }}"
loop_control:
label: "{{ item.name }}: {{ item.calendar }}"
changed_when: false
check_mode: false
when: atlas_manage_zfs_snapshots | bool
- name: Install Atlas ZFS snapshot and retention helper
tags: [atlas, storage, snapshots]
ansible.builtin.template:
src: atlas-zfs-snapshot.sh.j2
dest: /usr/local/sbin/atlas-zfs-snapshot
owner: root
group: root
mode: "0750"
when: atlas_manage_zfs_snapshots | bool
- name: Install Atlas ZFS snapshot systemd service
tags: [atlas, storage, snapshots]
ansible.builtin.template:
src: atlas-zfs-snapshot@.service.j2
dest: /etc/systemd/system/atlas-zfs-snapshot@.service
owner: root
group: root
mode: "0644"
when: atlas_manage_zfs_snapshots | bool
- name: Install Atlas ZFS snapshot systemd timers
tags: [atlas, storage, snapshots]
ansible.builtin.template:
src: atlas-zfs-snapshot.timer.j2
dest: "/etc/systemd/system/atlas-zfs-snapshot-{{ item.name }}.timer"
owner: root
group: root
mode: "0644"
loop: "{{ atlas_zfs_snapshot_policies }}"
loop_control:
label: "{{ item.name }}"
when: atlas_manage_zfs_snapshots | bool
- name: Enable Atlas ZFS snapshot systemd timers
tags: [atlas, storage, snapshots]
ansible.builtin.systemd:
name: "atlas-zfs-snapshot-{{ item.name }}.timer"
enabled: true
state: started
daemon_reload: true
loop: "{{ atlas_zfs_snapshot_policies }}"
loop_control:
label: "{{ item.name }}"
when:
- atlas_manage_zfs_snapshots | bool
- not ansible_check_mode
- name: Validate Atlas ZFS scrub policy
tags: [atlas, storage, scrub]
ansible.builtin.assert:
that:
- atlas_manage_storage | bool
- atlas_zfs_pool != 'CHANGEME_ZFS_POOL'
- atlas_zfs_scrub_calendar | length > 0
fail_msg: >-
Enable Atlas storage and declare a systemd calendar expression before
managing periodic ZFS scrubs.
when: atlas_manage_zfs_scrub | bool
- name: Validate Atlas ZFS scrub calendar
tags: [atlas, storage, scrub]
ansible.builtin.command:
argv:
- systemd-analyze
- calendar
- "{{ atlas_zfs_scrub_calendar }}"
changed_when: false
check_mode: false
when: atlas_manage_zfs_scrub | bool
- name: Require OpenZFS scrub systemd units
tags: [atlas, storage, scrub]
ansible.builtin.command:
argv:
- systemctl
- cat
- "{{ item }}"
loop:
- "zfs-scrub@{{ atlas_zfs_pool }}.service"
- "zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer"
- "zfs-scrub-weekly@{{ atlas_zfs_pool }}.timer"
loop_control:
label: "{{ item }}"
changed_when: false
check_mode: false
when: atlas_manage_zfs_scrub | bool
- name: Create Atlas ZFS scrub timer override directory
tags: [atlas, storage, scrub]
ansible.builtin.file:
path: "/etc/systemd/system/zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer.d"
state: directory
owner: root
group: root
mode: "0755"
when: atlas_manage_zfs_scrub | bool
- name: Configure Atlas ZFS monthly scrub schedule
tags: [atlas, storage, scrub]
ansible.builtin.template:
src: atlas-zfs-scrub-timer.conf.j2
dest: >-
/etc/systemd/system/zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer.d/override.conf
owner: root
group: root
mode: "0644"
when: atlas_manage_zfs_scrub | bool
- name: Disable the conflicting weekly OpenZFS scrub timer
tags: [atlas, storage, scrub]
ansible.builtin.systemd:
name: "zfs-scrub-weekly@{{ atlas_zfs_pool }}.timer"
enabled: false
state: stopped
daemon_reload: true
when:
- atlas_manage_zfs_scrub | bool
- not ansible_check_mode
- name: Enable the Atlas monthly OpenZFS scrub timer
tags: [atlas, storage, scrub]
ansible.builtin.systemd:
name: "zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer"
enabled: true
state: started
daemon_reload: true
when:
- atlas_manage_zfs_scrub | bool
- not ansible_check_mode

View File

@@ -0,0 +1,40 @@
[Unit]
Description=Back up Atlas ZFS datasets to the encrypted Borg repository
Documentation=man:borg-create(1) man:borg-prune(1) man:borg-compact(1)
Requires=zfs.target
Wants=network-online.target
After=zfs.target network-online.target
StartLimitIntervalSec=6h
StartLimitBurst=3
ConditionFileIsExecutable=/usr/local/sbin/atlas-borg-backup
ConditionPathExists={{ atlas_borg_passphrase_path }}
ConditionPathExists={{ atlas_borg_ssh_private_key_path }}
ConditionPathExists={{ atlas_borg_known_hosts_path }}
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/atlas-borg-backup
User=root
Group=root
UMask=0077
SuccessExitStatus=1
Restart=on-failure
RestartSec=30m
TimeoutStartSec=infinity
RuntimeDirectory=atlas-borg
RuntimeDirectoryMode=0750
Nice=15
IOSchedulingClass=best-effort
IOSchedulingPriority=7
NoNewPrivileges=true
PrivateMounts=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths={{ atlas_borg_cache_dir }} {{ atlas_borg_config_dir }} /run/atlas-borg /run/lock
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictRealtime=true
LockPersonality=true

View File

@@ -0,0 +1,176 @@
#!/usr/bin/env bash
set -Eeuo pipefail
export LC_ALL=C
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
export BORG_CACHE_DIR={{ atlas_borg_cache_dir | quote }}
export BORG_CONFIG_DIR={{ atlas_borg_config_dir | quote }}
export BORG_PASSCOMMAND={{ ('cat ' ~ atlas_borg_passphrase_path) | quote }}
export BORG_RSH={{ atlas_borg_ssh_wrapper_path | quote }}
readonly pool={{ atlas_zfs_pool | quote }}
readonly mount_root={{ atlas_mount_root | quote }}
readonly repository={{ ('ssh://' ~ atlas_borg_repository_user ~ '@' ~ atlas_borg_repository_host
~ ':' ~ (atlas_borg_repository_port | string) ~ '/' ~ atlas_borg_repository_path) | quote }}
readonly remote_path={{ atlas_borg_remote_path | quote }}
readonly archive_prefix={{ atlas_borg_archive_prefix | quote }}
readonly snapshot_prefix={{ atlas_borg_snapshot_prefix | quote }}
readonly compression={{ atlas_borg_compression | quote }}
readonly stage=/run/atlas-borg/source
readonly borg_user={{ atlas_borg_username | quote }}
readonly borg_group={{ atlas_borg_group | quote }}
readonly borg_home={{ atlas_borg_home | quote }}
readonly borg_lock={{ atlas_borg_lock_path | quote }}
snapshot_name=""
snapshot_created=false
mounted_targets=()
# Invoked through the EXIT trap below.
# shellcheck disable=SC2329
cleanup() {
local status=$?
local cleanup_status=0
local index
trap - EXIT HUP INT TERM
set +e
{% raw %}
for ((index = ${#mounted_targets[@]} - 1; index >= 0; index--)); do
{% endraw %}
if mountpoint -q "${mounted_targets[$index]}"; then
umount "${mounted_targets[$index]}" || cleanup_status=2
fi
done
rm -rf "$stage" || cleanup_status=2
if [[ "$snapshot_created" == true ]]; then
flock 9
zfs destroy -r "${pool}@${snapshot_name}" || cleanup_status=2
flock -u 9
fi
if ((status == 0 && cleanup_status != 0)); then
status=$cleanup_status
fi
exit "$status"
}
trap cleanup EXIT
trap 'exit 143' HUP INT TERM
run_as_borg() {
setpriv \
--reuid "$borg_user" \
--regid "$borg_group" \
--clear-groups \
--inh-caps=-all,+dac_read_search \
--ambient-caps=-all,+dac_read_search \
--bounding-set=-all,+dac_read_search \
-- env HOME="$borg_home" USER="$borg_user" LOGNAME="$borg_user" "$@"
}
exec 8>"$borg_lock"
flock 8
exec 9>/run/lock/atlas-zfs-snapshot.lock
zpool list -H -o name "$pool" >/dev/null
rm -rf "$stage"
mkdir -p "$stage"
chown root:"$borg_group" /run/atlas-borg "$stage"
chmod 0750 /run/atlas-borg "$stage"
flock 9
while IFS= read -r stale_snapshot; do
stale_suffix="${stale_snapshot#"${pool}@${snapshot_prefix}-"}"
if [[ "$stale_suffix" =~ ^[0-9]{8}T[0-9]{6}Z$ ]]; then
zfs destroy -r "$stale_snapshot"
printf 'Removed stale Borg source snapshot %s\n' "$stale_snapshot"
fi
done < <(
zfs list -H -t snapshot -o name -r "$pool" |
grep -E "^${pool}@${snapshot_prefix}-[0-9]{8}T[0-9]{6}Z$" || true
)
timestamp="$(date -u +%Y%m%dT%H%M%SZ)"
readonly timestamp
snapshot_name="${snapshot_prefix}-${timestamp}"
readonly snapshot_name
zfs snapshot -r "${pool}@${snapshot_name}"
snapshot_created=true
flock -u 9
printf 'Created recursive Borg source snapshot %s@%s\n' "$pool" "$snapshot_name"
while IFS=$'\t' read -r dataset dataset_mountpoint mounted; do
if [[ "$mounted" != yes ]]; then
printf 'Dataset %s is not mounted; refusing an incomplete backup\n' "$dataset" >&2
exit 65
fi
if [[ "$dataset_mountpoint" != "$mount_root" && "$dataset_mountpoint" != "$mount_root/"* ]]; then
printf 'Dataset %s has unexpected mountpoint %s\n' "$dataset" "$dataset_mountpoint" >&2
exit 65
fi
dataset_suffix="${dataset#"$pool"}"
source_path="${dataset_mountpoint}/.zfs/snapshot/${snapshot_name}"
target_path="${stage}${dataset_suffix}"
mkdir -p "$target_path"
mount --bind "$source_path" "$target_path"
mount -o remount,bind,ro "$target_path"
mounted_targets+=("$target_path")
done < <(zfs list -H -o name,mountpoint,mounted -s name -r "$pool")
archive="${archive_prefix}-${timestamp}"
readonly archive
borg_status=0
set +e
(
cd /run/atlas-borg
run_as_borg borg --remote-path "$remote_path" --lock-wait 600 create \
--show-rc \
--stats \
--checkpoint-interval 900 \
--compression "$compression" \
"${repository}::${archive}" \
source
)
create_status=$?
set -e
if ((create_status >= 2)); then
exit "$create_status"
fi
borg_status=$create_status
set +e
run_as_borg borg --remote-path "$remote_path" --lock-wait 600 prune \
--show-rc \
--list \
--glob-archives "${archive_prefix}-*" \
--keep-daily {{ atlas_borg_keep_daily | int }} \
--keep-weekly {{ atlas_borg_keep_weekly | int }} \
--keep-monthly {{ atlas_borg_keep_monthly | int }} \
"$repository"
prune_status=$?
set -e
if ((prune_status >= 2)); then
exit "$prune_status"
fi
if ((prune_status > borg_status)); then
borg_status=$prune_status
fi
set +e
run_as_borg borg --remote-path "$remote_path" --lock-wait 600 compact \
--show-rc \
"$repository"
compact_status=$?
set -e
if ((compact_status >= 2)); then
exit "$compact_status"
fi
if ((compact_status > borg_status)); then
borg_status=$compact_status
fi
exit "$borg_status"

View File

@@ -0,0 +1,12 @@
[Unit]
Description=Schedule the encrypted Atlas Borg backup
[Timer]
OnCalendar={{ atlas_borg_backup_calendar }}
Persistent=true
RandomizedDelaySec={{ atlas_borg_randomized_delay }}
AccuracySec=1min
Unit=atlas-borg-backup.service
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,32 @@
[Unit]
Description=Check the encrypted Atlas Borg repository
Documentation=man:borg-check(1)
Wants=network-online.target
After=network-online.target atlas-borg-backup.service
ConditionFileIsExecutable=/usr/local/sbin/atlas-borg-check
ConditionPathExists={{ atlas_borg_passphrase_path }}
ConditionPathExists={{ atlas_borg_ssh_private_key_path }}
ConditionPathExists={{ atlas_borg_known_hosts_path }}
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/atlas-borg-check
User={{ atlas_borg_username }}
Group={{ atlas_borg_group }}
UMask=0077
SuccessExitStatus=1
TimeoutStartSec=infinity
Nice=15
IOSchedulingClass=best-effort
IOSchedulingPriority=7
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths={{ atlas_borg_cache_dir }} {{ atlas_borg_config_dir }}
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictRealtime=true
LockPersonality=true

View File

@@ -0,0 +1,23 @@
#!/usr/bin/env bash
set -Eeuo pipefail
export LC_ALL=C
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
export BORG_CACHE_DIR={{ atlas_borg_cache_dir | quote }}
export BORG_CONFIG_DIR={{ atlas_borg_config_dir | quote }}
export BORG_PASSCOMMAND={{ ('cat ' ~ atlas_borg_passphrase_path) | quote }}
export BORG_RSH={{ atlas_borg_ssh_wrapper_path | quote }}
readonly repository={{ ('ssh://' ~ atlas_borg_repository_user ~ '@' ~ atlas_borg_repository_host
~ ':' ~ (atlas_borg_repository_port | string) ~ '/' ~ atlas_borg_repository_path) | quote }}
readonly remote_path={{ atlas_borg_remote_path | quote }}
readonly archive_prefix={{ atlas_borg_archive_prefix | quote }}
readonly borg_lock={{ atlas_borg_lock_path | quote }}
exec 8>"$borg_lock"
flock 8
exec borg --remote-path "$remote_path" --lock-wait 600 check \
--show-rc \
--glob-archives "${archive_prefix}-*" \
"$repository"

View File

@@ -0,0 +1,12 @@
[Unit]
Description=Schedule checks of the encrypted Atlas Borg repository
[Timer]
OnCalendar={{ atlas_borg_check_calendar }}
Persistent=true
RandomizedDelaySec={{ atlas_borg_randomized_delay }}
AccuracySec=1min
Unit=atlas-borg-check.service
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,19 @@
#!/usr/bin/env bash
set -euo pipefail
# Borg receives CAP_DAC_READ_SEARCH only for local snapshot traversal. Drop it
# before starting the network transport so SSH runs as the plain service user.
exec setpriv \
--inh-caps=-all \
--ambient-caps=-all \
-- /usr/bin/ssh \
-i {{ atlas_borg_ssh_private_key_path | quote }} \
-p {{ atlas_borg_repository_port | int }} \
-o BatchMode=yes \
-o IdentitiesOnly=yes \
-o StrictHostKeyChecking=yes \
-o UserKnownHostsFile={{ atlas_borg_known_hosts_path | quote }} \
-o ConnectTimeout=30 \
-o ServerAliveInterval=60 \
-o ServerAliveCountMax=3 \
"$@"

View File

@@ -0,0 +1,6 @@
[Timer]
OnCalendar=
OnCalendar={{ atlas_zfs_scrub_calendar }}
Persistent=true
RandomizedDelaySec=0
AccuracySec=1min

View File

@@ -0,0 +1,72 @@
#!/usr/bin/env bash
set -euo pipefail
export LC_ALL=C
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
if [[ $# -ne 1 ]]; then
printf 'Usage: %s <policy>\n' "$0" >&2
exit 64
fi
readonly pool={{ atlas_zfs_pool | quote }}
readonly prefix={{ atlas_zfs_snapshot_prefix | quote }}
readonly period="$1"
case "$period" in
{% for policy in atlas_zfs_snapshot_policies %}
{{ policy.name | quote }})
keep={{ policy.keep | int }}
;;
{% endfor %}
*)
printf 'Unknown Atlas ZFS snapshot policy: %s\n' "$period" >&2
exit 64
;;
esac
readonly keep
zpool list -H -o name "$pool" >/dev/null
exec 9>/run/lock/atlas-zfs-snapshot.lock
flock 9
timestamp="$(date -u +%Y%m%dT%H%M%SZ)"
readonly timestamp
readonly snapshot_prefix="${pool}@${prefix}-${period}-"
readonly snapshot="${snapshot_prefix}${timestamp}"
zfs snapshot -r "$snapshot"
printf 'Created recursive ZFS snapshot %s\n' "$snapshot"
snapshot_listing="$(zfs list -H -t snapshot -o name -s creation -r "$pool")"
managed_snapshots=()
while IFS= read -r snapshot_name; do
if [[ "$snapshot_name" == "$snapshot_prefix"* ]]; then
snapshot_suffix="${snapshot_name#"$snapshot_prefix"}"
if [[ "$snapshot_suffix" =~ ^[0-9]{8}T[0-9]{6}Z$ ]]; then
managed_snapshots+=("$snapshot_name")
fi
fi
done <<< "$snapshot_listing"
{% raw %}
managed_snapshot_count="${#managed_snapshots[@]}"
{% endraw %}
prune_count=$((managed_snapshot_count - keep))
if ((prune_count <= 0)); then
printf 'Retaining %d of %d managed %s snapshots\n' \
"$managed_snapshot_count" "$keep" "$period"
exit 0
fi
for ((index = 0; index < prune_count; index++)); do
candidate="${managed_snapshots[$index]}"
if [[ "$candidate" != "$snapshot_prefix"* ]]; then
printf 'Refusing to destroy unexpected snapshot: %s\n' "$candidate" >&2
exit 65
fi
zfs destroy -r "$candidate"
printf 'Pruned recursive ZFS snapshot %s\n' "$candidate"
done

View File

@@ -0,0 +1,11 @@
[Unit]
Description=Schedule {{ item.name }} ZFS snapshots for {{ atlas_zfs_pool }}
[Timer]
OnCalendar={{ item.calendar }}
Persistent=true
AccuracySec=1min
Unit=atlas-zfs-snapshot@{{ item.name }}.service
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,25 @@
[Unit]
Description=Create and retain %i ZFS snapshots for {{ atlas_zfs_pool }}
Documentation=man:zfs-snapshot(8) man:zfs-destroy(8)
Requires=zfs.target
After=zfs.target
ConditionFileIsExecutable=/usr/local/sbin/atlas-zfs-snapshot
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/atlas-zfs-snapshot %i
User=root
Group=root
UMask=0077
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictRealtime=true
LockPersonality=true

1
secrets/.gitignore vendored
View File

@@ -2,3 +2,4 @@
.vault_pass*
vault.local.yml
vault.local.yml*
/recovery/

View File

@@ -1,78 +1,83 @@
$ANSIBLE_VAULT;1.1;AES256
31386434333363613930316363393564373332303236306233643365386639346631336232373361
3732633931376438313835373537666438383664306266380a633531316432343166323361303465
39306136386664356163346266303963373839373763343136346135633236333333326331313063
6264306265373865310a373735393632373861333433326632303931633732636535326264346537
39633766326361666432396432623132616666363462373237623664363664373362346366383230
65326665303762356233356531306331333231376163353638363936626562623937323166623065
62633165383033303362336238343037353930396430336537363565333132666532323662653262
65373531313264353938666161373064393239653165666462336665323962336134343432316134
32373361613539356262313333663964623034313230356439626139383539326261663765373034
36633933366662656464306561393130373238313730336638663932633935313037363133636266
36663935303735623132663464376633306431396662306166393831313566323238363865303162
66303732376530653435613966373832333161333137366230613166303061333433336461386234
63393030616662373666643164663862653037383336323766646330653131623930353265336231
39646335353635616465663763306262353931316562616437313362616136623735646535616431
33363439623361343231363663363535663265366166346439623935323632366336363135373462
62646662316538323734333334393566333463653833326162663065646634363336373865623932
33646533636638386534366561663930613536663935663638313137646537626431393035653466
36623435613966323166363630383531656436373366343364346439343034333934623836336636
38653534343563393434646430306662343435653465363439326261373537666233353731306436
35323739336361343665633239366633393530346335396635316238323435663466316235376536
38303839313433643038353236616632626363653339346334303136306138336461383831626536
39303966393034333739363061363865326165303236656438316537396463383935393764346135
66323632323833353833643434316566326366633562666262386232653730303038613336663264
62636332383834393861373665393364653362636339346632303463386565633830336363393065
31383530613161366436323163393366386635316562633436383134623061353937326363396439
38636438396637616362373937666361303536376164636533653536306338356263313965623336
32633461623861643138613734313164633562613932386163643062353636376266633166373838
39613438653531316333663736366161313832613830353566356461393435656234383037353335
32343661376266353538643531313239306432646335383734623233373063316430383362643531
35393263343534363936373361316265653934383735663933663365363564663966646335353337
63366365383466393765336533363130643236313331633537346237353631623334633330376164
32616530623032663761643437336536323332343130393339313232623364656334613233323833
64346637343738386335343631333035636337633732333662663763326362636432646136353064
30393061326435323837633632333765623931663265326137373135303035623464333366653566
66613133396339376264313964353932396136613538383264363865633536373839663465333437
37613765626131343330653063663764346364316264363635653438396331363263623562333735
30646563303439663639376430336336383761363365353838613036333032306434623661636138
63616439636437663732386164346365643834376631303263376563323662373734653631613730
31373039656238303462643930303531613931373065316435303661343862623034653364653736
37613835616630316133333130633631653639336266313438633166316539373433333661613839
32356139366232313336343062323265366563633735383661393335386163653133313664613264
63303739323863333439316461366137656434666366306466356633306663623730653939663430
39313764663534316562326435316264633236373834323665356266346532323565323532333438
33306237623430613463393164383332356533303433343465343930636563363862373330613632
38643937613263313435626166313464316133373338373261666331653436373063393162636339
66356434663239646334383433626566336433653265336332323866633666646132663836313537
66663766306438396131623533613834626331653731323137303539303835306632663132616363
64303438353836613136623562626664326364656133383865383730373762666539383036396337
66393636343561333435313032353939316138306336316337353861383238326136373265663433
62366566343866626336646466386561353061343735306565383437333931613635393034363430
37633765663239643435623066313331353862393966306563393838386334393162306562363062
61633363386564373834383432323861323364623365306439353631323463326461383039356466
34396464653233626435656463326439303665633532656639376633353932666630626564616564
34326638383634353033326232646339393638663637313136653763336265616635326666336530
33663261376262626136346265656130653831636662306132393837306135643831353534626636
34383762386665666363313932336632326230646439663366663037323562633630373137333232
37393164656137303131313738396131613561306332356436303436636338623233343637363332
36613137316337666461333237373266326238303531396432383461616239316630346230303735
63636535383766353338643932366339333130663632386337643932636630316432313465393766
39613065623631316165313134386231616165366266323634643632626230303861376461323433
35346137326563646332303334313530383464373838373633363635373766333362386466323836
32613236646561343365656239346239353866663336393930356238653463336361333033643331
61323430663031383239353363616666336637666230663633616464616638303966343631386632
32333735643936353638666336303133356435396338653465323234316234626536666635333238
38313661303466333464666538363938656631396666643566343763396638336663376430646532
38666234653330646262633062356238343536336637356432313137313561363937363936383364
30643937383833663339643862363234643765386164316138636565643434373734383339363138
36323163633837326132393365333236633264386664373234313061373835346634663137383837
62666561353532303663346365343131316233633163323938623066656332383030393864363536
38383939383935613432613837333863313239653831333438383133343763633838353964353161
61323462343835613937653465633563306462613631323762656437626133336638396663646362
30323661383134653336366234663333336261353162373030626266656336356233316265636661
34303865313433633138363936373561636537353831373033303163646436303932626138356633
63656364353163313037613262396338636230646330666331616534313466306361363433656132
66633231626665303165346339373764666264313838313063323732653837383736633235363064
37353632336238623366313432376163653535656134633634313065356533343933666135396633
30613134646132613637656461303431613064393438363231383464663765316638
61353065386233646137323235306631353635663530363237636231316265643562353465323430
6165646466623962313835313537633137633766373930380a316335323962616265643136346666
63336133336131346336383534356637623831363138323165633262386333363535393365383233
6234393835653439370a313963313365373633323464343263383661383336363662633133643232
34366634383862363635653034313531623330396639616462343630326162316535643465653532
36326534333637376462353561343964633636366331363833313263353133383636623537303663
35393032316439336666343161653439643638376134363535656262343963393365623432336433
35383934313762313037326430316666363731666231336534326661353034333063643364343230
65333739303566366263333565333465613136646237623937393733623438613832393634663463
39376131313234333039633735613233373931613232653036663665316636303961653834366339
36353730316132316233303964303839363161346564396163336137663134353062363733656430
37643339326661653031376265646132623162373562393437373437313732396537383939333666
62353036316633306666313461663033303830393765396131643035353730383931646239663935
32626461316364386135303761383837613063336466363162323332663764616464373565383231
61346463336566346533326535376439643133613762383633396131323632356533636139336365
62393838316634623932643034376631333539343965383436613364643962363834346337353334
32656439366439313734353963343133333533653839613632323338336131373566613835393536
31663433616334373432376531346435336530303936356461303163646463613661643161313661
66663866343565616631616338353737356164353562366164383736346131666662623132333466
39383865653631373232393433663430643961646265386166333137643966303834363262373636
62396434373363353636376133666133663162653265313139313732353639336232333862643036
64386231336561396537326139346566306434633934343038663165396665363032383466633662
62336163633964363435386630343966333162333730336138333239646631633132663931376462
33663139356261313065376636613930353735396131306538306664646135636336643032623131
38346264333331353633326535326431626563323036313665643337353563333339646430386564
31613435383036313430316366323636663735326336393338353835323861333564363832656462
35336435623261326363633033316130393062616339353263643062633331646137376135656365
35636139336564346164616235616431326531333433646330386134323932373339646536356464
66343533326534326165323564663533653666633035343163633832393361336462343937623165
62383931326630363036396333313931393836366439653433623165666166356338653364336534
35333936653833386163633738326164386166613561333530633937343230363366333662666539
39333361633933663735303438663239303536363433313962643137386533633539326365383765
37636538386339333935386132353265353031643662616330316463623661663738353433313830
36373963633166333464653338343830373063323536383364393033393235326639613662343737
38663362636331343061646465313237313431373433353361353265333766633463353632646536
31323231306138323031396630656538363930373439336234343963616334363632653738316465
63653938373830336362313238656266613362636634616537653863336132343931616262396130
66393239303866656232653832343132366537333537343635666563343639323433383163613335
39613533376634316133633430303535306266656333626264343733666335393661666561396633
39346265316137326465326635396362333565393133623637633132616232326263663662343137
33363733306135363361643031306265363733656362386666306334333035393839636533343363
35396638616636633639343930373136376339346162393061393765363837646365383866636131
33653465666239393133616232636231333332396138376332393664343364643835306530393238
34663237303530303837663535646263393931373531393039356336316561653130356262636562
38336362326639653237626634376334666565653036353236313634376364626338646538386536
38626636386466373566646166393963643164343536373236396138303532393161363335386638
32633032393737626363613463323366366637616361313537356136626661626633613739323338
35383963666431343566356562333234663936376562616638636261303466633539376334303331
39303834663234663063356233313962326664383839393832303462643636393034383434303465
64333635376135326333356435373734643430623736373234643335343130383066326436356664
63346663326364343634303930343338336139313864316165366232643537366635653764353763
31363863633261643263303433373330366161323166366462336332313135366338393334653764
66353733653137663835663731373364613030373334663061313433373861613665363236633130
65613965366636343465336533613438373466383737373366653965633437323562643966396431
39303033643438633762633263326132663466643438656366363431616237633031333936313831
30323930383233313032323638356333626230333764363662313662646536643839353032353462
30326166653937353130623133303533343934633565393831623033303234316330353432313266
30636536633933376365623665616262663236383731633633346232613366333137396139306363
35633336643266326335303261666666653536666630613639376336373237646134306462616537
33343561373162666332613634643837343566646161373065366637653135613632353334636363
63363232303963646530333366663862323264326536643337323266396566316233613630303637
66646366376466373931613734363931316230323063373666653062373364396433633762633762
38613933323733653238383935623230383562646563363833653838636165626365646537383639
33666535656363393562316336633439636138373365623431393965653765306138646234663938
65653133663663393731646337386535333261643932336132396237323930306136643534353930
65636438396432623034626561613137336138623265393064383034623863303166356138393564
37373164626634653662326234333539663735323464613334616130643937373730363263633366
31393937326432386165343338313031376565313866363731643534313233303064373935303538
31343832336230393636653432653162336361383963633766343461653466316337353931333363
63313137303564336630343937356564643763383764613362366634373362666465626334336539
64366533376165306532343461613265366266383862323032333465336161663161376630316465
30306562666163646235656664653635366461366435663961623635383437663564356563346462
31636234663765623838333237393239373564366262613637363938653463396530613963643837
38636634376637366332623035313465393762653865623130336263343663303066366135616639
63333964356466613038303263366462346261353030646532366361393965306435613131316463
65366266376637323764643239323730366565633335666638666334663635373961303637383861
35313431646434656562333937663837393038386361616630626532636339306432353434656165
33663261383166386432383465666136376237346565303164363461666663346130346162316338
62373061353034316234303835663439396434343738303764376665336239626238386436386234
61306166383637366266393730323732386163366261393630336431633862353761343763363665
61323039396234393835303633363339373633653334343766653032313230343464326664356566
3462623830666664626633373966363866333337383730313066

View File

@@ -14,3 +14,4 @@ vault_ikaros_authorized_ssh_keys:
vault_atlas_admin_password_hash: "REPLACE_WITH_A_SHADOW_COMPATIBLE_HASH"
vault_atlas_samba_password: "REPLACE_ME"
vault_atlas_immich_db_password: "REPLACE_ME"
vault_atlas_borg_passphrase: "REPLACE_WITH_A_STRONG_UNIQUE_PASSPHRASE"