Today I had to recover a Debian server after the
/var partition was accidentally deleted. All the user generated data (primarily website content and databases) was backed up, and I also had access to a backup copy of the installed package list as generated by
dpkg --get-selections. The instructions below don’t go into full detail, but hopefully they will set you on the right track. They should also work for Ubuntu installs.
If this happens to you, don’t panic, and whatever you do, don’t shut down or reboot the server!
To recover the server, I took the following steps:
- Create a bootstrapped Debian environment on a separate server
- Use the installed package list backup to install the same packages into the bootstrap environment
- Transfer the contents of
/varacross from the bootstrap onto the broken server
- Restore user data from backup
- Fix various file permissions
- Restart the services
- Check the logs and systemd status for any remaining problems
- Reboot the server to check that everything comes back up as normal
All commands below run as root. Be careful not to make any mistakes!
To create a bootstrapped Debian environment, run the following:
apt-get install debootstrap
debootstrap stretch /bootstrap/
cp dpkg-selections-backup.txt /bootstrap/
mount -t proc none /proc
Now you can install the packages using the steps below. This took a bit of fiddling around but there are a number of helpful posts about the likely problems if you get stuck. IMPORTANT NOTE: if you are prompted, DO NOT install grub to the MBR! This might break your existing install!
dpkg --set-selections < dpkg-selections-backup.txt
/etc/apt/sources.list(bootstrap only includes
mainby default, so you may need to add in
apt-get install dselect
You can now rsync the data across from
/var on the broken server. Your life will be much easier if you use the
--archive option. You may need to temporarily enable root SSH access in order to do this, e.g. with
Once I’d done that, I had to reinstall the mariadb-server packages on the broken server using
apt-get install --reinstall mariadb-server mariadb-server-10.1 mariadb-server-core-10.1.
The next step is to restore your user data. This will be specific to the structure that you use, so I can’t help with instructions for that. Once you’ve done that, check the permissions of the files in
/var/log against a working server (or perhaps your bootstrap) to check that they are correct. It’s quite likely that the user IDs of the system accounts will be different between the server and the bootstrap environment.
Now you should check the various services that are installed, using
service --status-all. Not all of the services should necessarily be enabled, again I recommend comparing against a working server. Try to restart any services that aren't currently running but should be. If all the permissions are correct and the relevant user data (if any) has been restored then the service should start successfully. If it doesn’t, check the systemd status with
systemctl status <service-name>, the systemd logs with
journalctl -xe and the service’s log files in
/var/log. Those should give you enough information to track down any remaining problems.
At this point, all of your services should be back up and running normally. The final step is to reboot your repaired server and verify that everything comes back up as expected after a restart.
Last but not least, if you’re trying this but getting stuck, we’re happy to help! Drop us a line at our contact page.