It's not about having a backup, but being sure the backup works
Since I discovered BorgBackup (short: Borg) I have felt way more confident about my backups. Borg gives you a space efficient storage of backups, data is saved encrypted, and you get good versioning of backups. I solved all my previous inconveniences when just using rsync. I lacked the way to keep track and order different versions. Compression and encryption as a great plus even when I just keep my backups locally.
I have felt a huge safety net concerning my personal devices backups and my my cloud services backups. With such confidence, today I was crazy and decided to dump my VPS and reinstall it from scratch for an upgrade. After a full day of dealing with installation it just didn’t work, server was not online. I could not get redis server to run on my openvz instance, because my service provider just had an old kernel on my host machine and it wouldn’t work with the latest version of Debian stretch(I upgraded from Debian Jessie, which provided old enough packages for everything to work).
Having suffered a lot from my openvz VPS, because it had an old kernel and I cannot put swap memory into it, I had had enough. Cheap does not compensate for the frustration. I thus moved to a KVM, and everything went smoothly regarding installation, I could use my Ansible playbook from the sovereign project with some of my tweaks. Almost everything worked with little extra effort. Or maybe I was just in the mindset of fixing stuff and dealing with the problems of the openvz server, that setting the new server felt easy.
Not everything is wonderful when doing a migration, I did hit some setbacks. I changed my infrastructure, service provider, server location. I had to update my DNS configuration in my nameserver, which took a long while. It is also a manual process, you must change the DNS config via a web interface from the service provider. I would be awesome if I could have Ansible change that configuration as well. Ansible is amazing, it configured my VPS faster than DNS would update and that was an issue with getting my Let’s encrypt certificates.
This is where having a backup saved my first, I could not get new SSL certificates from Let’s encrypt because my new server did not responded to the queries. The DNS still pointed to old server. But, why get new certificates? I can use the old signed ones that are in the backup. Just a simple manual copy and done.
I did experience some other few obstacles regarding the configuration. Specially since I also upgraded to Debian 9 which is not yet fully supported by the sovereign project. Some pull requests might end there too.
When I was restoring nextcloud. I had done it before for a private server from a customer. But when going through my own backups I could not find the database backups. All files where there, except the database. The backup script was indeed doing the database dump to a file, but Borg was never instructed to back it up.
What is lost? not much really. I’m still the single user of my nextcloud instance and I have multiple copies of the files, locally and in the backup. Thus I only uploaded the files and had nextcloud index all of them again. I did loose the history and changes of this files. Yes, the backup does have the multiple versions as nextcloud saves them, but I just feel is to complicated to try to recover this functionality without the information from the database and for things I care about versions I use a proper version control system. Honestly, I have never tried to recover a earlier versions of my files using nextcloud or its previous iteration owncloud.
- Test your backups. Is not enough to have them, you have to test they work. A new task for me would be to test: how fast can I restore from total disaster to a new working server? Now it looks pretty bad, about 20 hours. That is keeping in mind I had hope on the openvz working.
- Don’t dump a service before the replacement is working. This is kind of obvious but since I’m the single user and I could afford downtime on this and dedicate time until everything is fixed I took my risks on it.