Data Backup Strategy for ZFS-Pool on TrueNas
In theory there is one solution to backup a TrueNas Data-Share to an external hard disc: Fiddle with send/receive (see below)
In practice, there are different options:
1) use syncoid
2) use TrueNas “local replication”
3) use a script with send/receive
4) use rsync
Options 1-3 expect ZFS on the target volume, option 4 works with any filesystem.
Should i use ZFS on a single (backup)-disk? see: this thread in the truenas forum –> I’m not sure yet, so let’s have a closer look….
based on: https://esc-now.de/_/zfs-offsite-backup-auf-eine-externe-festplatte/?lang=en
explained in: https://jrs-s.net/2016/09/15/zfs-snapshots-and-cold-storage/
1) Create a pool on the disc
geom disk list
–> get USB-Device (on FreeBSD)
mount -t ext2fs /dev/da3s1 /media/usb-stick
–> Mount USB-Stick (not needed here):
Create Pool on Linux:
zpool create -f -o ashift=12 -O compression=lz4 -O mountpoint=none \
zpool export backup
on FreeBSD: same as on Linux, see https://forums.freebsd.org/threads/zfs-on-an-usb-stick.22379/
zpool create = Creates a new “simple, non redundant storage pool on a single device” (freebsd.org/doc/). There should be no partition present on the disk. Parameters are set as follows – see:
2) Use replication to transfer files to Backup-Pool
Definition: “A snapshot is a read-only copy of a file system or volume. Snapshots can be created almost instantly, and they initially consume no additional disk space within the pool. However, as data within the active dataset changes, the snapshot consumes disk space by continuing to reference the old data, thus preventing the disk space from being freed.” (https://docs.oracle.com/).
A snapshot is used to create a consistent state of the data. This snapshot is then sent to the Backup-Pool. This has to be done on dataset-level, as it is not possible to send an ecnrypted pool as a whole. So the tasks are:
a) Create a snapshot of a dataset
zfs snapshot pool/dataset@snapsh-initial
b) Send the snapshot as a datastream
and pipe the output to the receiving dataset in the backup-pool. Instead of creating an archive-file, the output is written directly to the backup-pool. A dataset can only be replicated to an empty target-dataset.
zfs send -v pool/dataset@snapshot-initial | zfs receive backup-pool/dataset
c) Subsequent incremental backups are sent via Flag -i.
szfs send this way only transfers the differences between two snapshots. This only works, if the initial snapshot is already present at the receiving side. So to work two snapshots are needed (one, which represents the initial state, and a secnd one, chich represents the current state of the data) (see https://www.truenas.com/community/threads/how-to-use-zfs-send-zfs-receive-to-back-up-my-current-pools-to-a-10tb-transfer-disk.59852/).
zfs snapshot pool/dataset@snapshot-backup
zfs send -v -i pool/dataset@snapshot-backup pool/dataset@snapshot-2 | zfs receive backup-pool/dataset
Using the Option -P copies all settings (including mount points,..) of the dataset. The Option -R copies all child-dataset as well. The Option -F to “zfs receive” will “force a rollback of the file system to the most recent snapshot before the receive is initiated. Using this option might be necessary when the file system is modified after a rollback occurs but before the receive is initiated.” (https://docs.oracle.com/cd/E19253-01/819-5461/gdsup/index.html).
d) Send/Receive encrypted Pool/Dataset
You cannot send the whole encrypted pool (to be more precicse the root-Dataset), instead you need to send each encrypted dataset separately.
3) Run script automatically when HD is attached
see: devd-freebsd Howto
on FreeBSD using devd:
Option 1) Sanoid/Syncoid
Sanoid/Syncoid is a wrapper for ZFS send/receive to make the handling easier.
see: Rsync vs ZFS
–> i decided against this, as:
– TrueNas is an “appliance” it’s not recommended and quiet difficult to install software outside of jails.
Option 2) TrueNAS: Local replication
As TrueNas is an “appliance” it’s not recommended and quiet difficult to install software outside of jails. So for Backups another option is “local replication”:
see: TrueNAS Doku
– Don’t use encryption, as it is only for the “transport” (“encrypting the data before transfer and decrypting it on the destination system”).
– Initial replication (2TB) about 20h
1) Create a Backup-Pool (with encryption) and an unencrypted dataset (Data) below it, which will be the target for the backup.
2) Create a “periodic snapshot task” for the whole dataset/pool (weekly)
3) Create a replication task, based on this snapshot (set read only):
4) Change encryption options for the root-dataset (Pool) and provide the same key as for the “original” dataset.
5) To unkock a replicated, encrypted dataset, uncheck the option “Unlock with key file” and provide the “dataset key” (dataset_Data_key.txt) instead (both for parent and child dataset, use the same keyfile, when properties had been inherited).
6) Create a NFS-Mountpoint (read only) for the Backup-Dataset.
–> i decided against this, as:
– can’t trigger this automatically, when disc is connected-
– so go for solution 3).
Option 3) Trigger local replication through script
Prerequisite is to create a Pool in TrueNas. I used the following options:
– not encrypted
– in the setting of the root DataSet set “Enable Atime” to off.
– Create a SMB-Share for the root dataset, readonly.
1) Trigger a script, when externalHDD is attached
see chapter “theory”
2) Steps in the script
– Mount Pool
– Do sanity checks on the pool
– Unlock Pool/Dataset (if needed)
– Get child-datasets
– Create Snapshot
– Replicate Snapshot using “send/receive”
– Unmount Pool
– Scrub & Housekeeping
– Send Email
All in one script, based on:
https://esc-now.de/ Kudos to the creator of this great script.
Notes and modifications:
– atime (accessing files in the backup) also changes target. This results in the error message “cannot receive incremental stream: destination […] has been modified.” (https://www.kernel-error.de/)
– Housekeeping, Sanity-Checks, Logfile and Email-Notification
– Support encrypted and nested datasets
Option 4) Use rsync to “copy” data to External HDD (et4)
– target is ext4, can be easily read on every Linux-Box
– no (additional) snapshots for replication are needed
– HDD not encrypted (need to use LUKS)
– rsync is dead-slow
So i did not opt for this.
I opted for the Option 3) – triggering a send/receive-script using devd-rules.
General Information on ZFS:
servethehome.com – An introduction to zfs – a place to start/
lawrencesystems.com – raidz3 – capacity integrity and performance
Reddit – Primer on freenas and zfs
hixsystems.com – ZFS Primer
freebsd.org – handbook/zfs
storage-insider.de – 3-2-1-backup-regel
[…] described in the previous post, i modified the awesome script from Jörg Binnewald @ https://esc-now.de/ to suit my […]