Home server backups with restic

nightly-restic

The home server holds documents, photos and the configs for everything else. Until April, its backup was me remembering to plug in a USB disk.

Project
nightly-restic
Stack
Bash, restic, systemd timers
Time spent
About 3 evenings
Status
runs at 03:15

didWhat I did

Two restic repositories: one on a second internal disk, one offsite. A short Bash script backs up to both, applies a retention policy and exits non-zero if anything fails. A systemd timer runs it every night at 03:15.

#!/usr/bin/env bash
set -euo pipefail

export RESTIC_PASSWORD_FILE=/etc/restic/password   # chmod 600, root only
SOURCES=(/srv/docs /srv/photos /etc /home/me)

for repo in /mnt/backup/restic "$OFFSITE_REPO"; do
  restic -r "$repo" backup "${SOURCES[@]}" \
    --exclude-file=/etc/restic/excludes --retry-lock 30m
  restic -r "$repo" forget --keep-daily 7 --keep-weekly 5 --keep-monthly 12
done

date -Is > /var/lib/backup/last-success

Once a week, a second unit runs restic check --read-data-subset 5%, which reads back a random slice of the actual data instead of only checking the index. Over twenty weeks that covers the whole repository.

The last line writes a timestamp. It looked unimportant when I added it; it became the hook for the status page two months later.

restic snapshots/mnt/backup/restic
$ restic snapshots --compact
ID        Time                 Host
3c9e41d0  2026-04-21 03:15:06  den
b17f0a92  2026-04-22 03:15:04  den
e40d6c3b  2026-04-23 03:15:09  den
91a2fe57  2026-04-24 03:15:05  den
-- 25 Apr: locked by prune, skipped --
5d08b7e1  2026-04-26 03:15:07  den
6 snapshots
The missing night is the prune that ran four hours and held the lock.

brokeWhat broke

Prune and backup collided. I first ran forget --prune every night. The first prune on the offsite repository took just over four hours, still held the lock when the next night's backup started, and the backup failed with "repository is already locked". Pruning now happens only on Sunday afternoons, and the backup waits up to 30 minutes for a lock instead of giving up.

A database copied mid-write. One service keeps a Postgres database in a Docker volume. Copying those files while the database is running gives you a backup that may not start. The script now runs pg_dump to a file first, backs up the dump, and excludes the live volume.

The test restore. I restored my home folder to /tmp/restore-test to see if it worked. It did, mostly. File permissions and timestamps were right, but ~/.ssh was missing. The exclude file had a line .* meant for cache folders, and it matched every dotfile in the tree.

# before: too broad
.*
# after: name the caches
/home/me/.cache
/home/me/.local/share/Trash
**/node_modules

Without that restore I would have found out the day I needed the keys.

nextWhat's next

  • An alert when last-success is more than 48 hours old. A backup that stops running is worse than one that fails loudly.
  • A restore drill every quarter, on the calendar: pick a random folder, restore it, open a few files.
  • Write down, on paper, where the repository password is kept. If the server dies, the password file goes with it.