How to recover from a corrupt Deja Dup backup, and set up an alternative

I recently started using the automated backup system Deja Dup which makes it very easy to set up automated, encrypted, incremental backups using the Duplicity back-end.
This package has been installed by default on Ubuntu since version 11.10, so I expected it to be thoroughly tested and reliable by now. Unfortunately my backups became corrupted which meant that some files were lost, and it was quite difficult to restore others. This article will explain how I recovered my files, then set up a faster, more reliable alternative that's still encrypted.

I used Deja Dup for a few days and it appeared to be working normally and claimed to have made several backups for me successfully;
I formatted my hard-drive to do a fresh install of my OS, I went to restore my data with Deja Dup, but it then died near the start of the restoration process, giving this error:


invalid data - SHA1 hash mismatch for file:
duplicity-full.20140508T105537Z.vol2.dia href="#duplicity-sources">fftar.gz
Calculated hash: 8ae69af39a566823309fae86142ae3a2af16358d
Manifest hash: 6a332f406b0842f229e2122921c0e4c97c4f76bd


I tried running the process again several times, and tried the three different backup snapshots that were available, but the same problem kept occurring.
I was starting to panic by this stage so I looked at the documentation to see if I could get more information about what was going wrong, and use the command line to get more control over what Duplicity was doing. I recommend reading the documentation listed at the bottom of the page if you have any doubts about what I'm doing here.

I used this command to get a list of backups: duplicity collection-status

Next, you can attempt to restore a backup hopefully showing where the corrupted files are...
To do this I chose which backup I wanted to list by specifying the backup time.
duplicity collection-status (above) lists time-stamps for each backup like this Sat Apr 26 02:57:20 2014 but the documentation states that the commands require me to convert the timestamp into this format: 2014-5-26T02:57:20 (yyyy-m-ddThh:mm:ss)

To attempt to restore the whole backup to test_restore/ on my desktop I ran:
duplicity restore --time "2014-5-26T02:57:20+01:00" file:///home/ben/Desktop/backup/ /home/ben/Desktop/test_restore

I then needed to list files in this backup so we can attempt to avoid the corrupted ones while restoring.
This command will output a list of all files in your backup which can be very long, so had it output to a text file instead:
duplicity list-current-files --time "2014-5-26T02:57:20+01:00" > backup_files.txt

I made a basic shell script to restore each folder in turn, but avoid touching the corrupt files. The corrupt files in my backup happened to be in /home/ben/Downloads/torrents/ so I had to specify each non-corrupt file/folder within torrents/ in turn. All of the other folders had no corrupt files so I could just specify the top-level folders and Duplicity restored all of their contents recursively without problems.


#!/bin/sh

unset PASSPHRASE


With any luck all of the specified files will be recovered now.

Setting up an alternative backup solution

I now use an external hard drive with an encrypted Ext4 partition, then use a simple shell script that runs rsync to update my backups. I haven't got this automated as I don't have the external drive plugged in much, but now it's set up all I have to do is plug in the drive and run ./scripts/rsync-backup.sh in the terminal.

If you have Ubuntu, search for and run the Disks app. If you have Xubuntu or a different distro like me that doesn't come with it, install the gnome-disk-utility package.

Be very careful with this tool as you could easily wipe all of your data if you select the wrong disk/partition.

Select the drive/partition that you want to re-format and use as an encrypted backup drive. Click the More actions button, then Format...

Select the option shown - 'Encrypted, compatible with Linux systems (LUKS + Ext4)'

Choose a label for the partition and set an encryption key. Make a note of this password or you won't be able to mount the disk!

When formatting is complete, mount the drive and enter the password you just set. Without this password no-one can access your encrypted drive. Ubuntu will probably offer to remember the password for you in future.

Next, make a simple shell script to run the rsync command for you. Make a plaintext file and make it executeable in file properties.

Then enter the contents of the script, which for me looks like this:


#! /bin/sh
rsync -axh --progress --delete /home/ben/ /media/ben/ben_backup_drive/


Now you can run the script in the terminal by doing something like this: ./scripts/rsync-backup.sh. It will take a long time to copy all of your files the first time, but in future it will be faster as it will only be syncing changes.

Documentation
https://help.ubuntu.com/community/DuplicityBackupHowto
http://duplicity.nongnu.org/duplicity.1.html