Encrypted incremental backups in QubesOS with BorgBackup

The official backup tool of QubesOS does not support incremental backups. Just to backup my around 180GB of data to an external hard drive, it takes over 7 hours. Incremental backups are much faster. Borg is my preferred backup tool, supporting encrypted incremental backups to a disk or to an SSH server. I make encrypted backups to borgbase.com and created a qrexec service for this purpose. It enables remote, encrypted, incremental backups even of network-isolated qubes. This blog post describes my approach.

+----------------+
| sys-backup-mnt |
+----------------+
      ^
      | qrexec
      v
+------------+  split SSH  +------------+
| sys-backup |<----------->| ssh-backup |
+------------+             +------------+
      ^
      | SSH
      v
+--------------+
| borgbase.com |
+--------------+

I don't run Borg directly from my app qubes, because that would mean having to enter the encryption password into a potentially untrusted qube. Instead, I use a disposable qube called sys-backup-mnt, which is network isolated. I mount the app qube's data in sys-backup-mnt using qvm-block attach. Then, I can run Borg from sys-backup-mnt. Since sys-backup-mnt is network isolated, I need an additional qube called sys-backup, which has SSH access to borgbase.com, using split SSH for authentication. The next section describes the qrexec service used for communication between sys-backup-mnt and sys-backup. Note that sys-backup only receives data from sys-backup-mnt which has already been encrypted by Borg and simply sends it on to borgbase.com over SSH. This is an important feature, since sys-backup is connected to the internet and thus should not be able to see the sensitive data of network isolated qubes.

The qrexec service can be created in the template qube of sys-backup, by creating the executable file /etc/qubes-rpc/qubes.Ssh with the following contents:

#!/bin/bash

read args
socat - "EXEC:ssh -o 'StrictHostKeyChecking=no' $args"

The client script can be created in the template qube of sys-backup-mnt, by creating the executable file /usr/bin/qubes-ssh-client containing:

#!/bin/bash
{
    echo "$@";
    cat
} | socat - 'EXEC:qrexec-client-vm sys-backup qubes.Ssh'

Backups can be created with the qrexec service described above by following these steps:

1. Start sys-backup-mnt
2. Mount the LVM image of your app qube to sys-backup-mnt (see QubesOS documentation)
3. Run borg as usual from sys-backup-mnt, with the environment variable BORG_RSH=/usr/bin/qubes-ssh-client
4. Restart sys-backup-mnt before backing up another app qube. Since sys-backup-mnt is disposable, this ensures you start from a trusted environment.

Of course this process can be automated by writing a script in dom0 that does the above steps for all the app qubes you want to back up. This is left as an exercise to the reader :-)