Backing up a Headless Ubuntu Web Server (to Cloud & QNAP NAS)
October 12, 2020
Backup Software | Ubuntu | NAS | Linode | IT Services | MSP | Self-Hosted

Backing up a Headless Ubuntu Web Server (to Cloud & QNAP NAS)

Read my dedicated post on PC backup software if that's what you're after!

I use several Ubuntu-based VM-s hosted at Linode, which fulfill various roles (web server, JIRA, knowledgebase software). Of course I already pay for Linode's in-house backup solution, which is reliable & has saved me in the past. It is great if you have a serious issue with your whole node & need to restore it to a day ago. However, it has some limitations:

  • If Linode goes bust or accidentally deletes my account (I've read about such cases, although I'm not sure if they were resolved) or some other unforseen event occurs then having backups with Linode is not much use
  • Restoring such backups is a pain -  you have to spin up a new node & restore there, even if all you want is a file from a few days ago
  • And in any case it's good to have several indepentent backup strategies, & setting up file-level backup provides for a good opportunity to take inventory of what is worth backing up!
  • It would be nice to have a central console where I can monitor all of my backups! In any case, email notifications are a must.

There is a whole discussion on the drawbacks of Linode backups Is it just me, or is the Linode backup service about useless.

So, I've spent the last few days trying to identify a reliable, modern, & inexpensive solution for creating backups of these servers. I'm documenting my findings here mostly for myself, but I figure there must be many other devs out there who might benefit from my experiences.

Note that none of the links here are affiliate links and I haven't received any compensation for any of this. If you find it useful, please leave a comment below! I'm also open to any advice.

My primary goals were:

  • Must be able to back up from headless Ubuntu servers to cloud or my QNAP NAS (which is not directly accessible from the internet)
  • A reliable solution, so it must be fairly popular, and it must also be efficient (compression) and fast
  • I have no intention of paying a subscription fee for the software itself, but I don't mind paying a reasonable price for perpetually licensed software
  • Set-up must not be too complicated!
  • I don't want to buy through a reseller

In the end I settled on a three-pronged approach:

(1) Linode's existing backup service (which I already had)

(2) Duplicati backing up specific directories to Backblaze B2

Read my dedicated post on Off-site Backup of your Ubuntu Linode with Duplicati (with MySQL, Jira, and GitLab) + Healthchecks.io if you go with Duplicati!

(3) Nakivo Backup running on my home QNAP NAS & backing up the whole node there (NOTE! There are 2 security advisories out at the moment with regards to Nakivo Backup!)

Read my dedicated post on how to Backup your Ubuntu Linode to your QNAP NAS with Nakivo if you go with Nakivo!

Looking at pricing on Nakivo's website, you might think that you can get a perpetual license for $50/server. What isn't obvious is that you have to place a minimum order of 10 licenses! So keep that in mind before going with Nakivo. And shame on them for not making this more obvious - or even having this silly condition in the first place.

Some "runner-ups" that I actually installed and tried:

  • Syncing certain directories to my NAS using QNAP's built-in "Hybrid Backup": well this only works with plain FTP or certain cloud providers, so it was not an option for extracting data from my nodes.
  • URBackup: I was not able to get the client on the node to communicate with the server because the communication is initiated from the client to the server and not the other way around.
  • MSP 360: centrally managed from MSP360's website. Worked, but there's no way (that I found) to browse what was backed up, or to restore individual files. Also, subscription pricing starts at $360/year.
  • CloudBerry Backup (now also owned by MSP 360, but kept as a separate product) for backing up from the node to B2 cloud. It worked but quite slowly, I suspect because it seemed to upload each file individually instead of in a zipped backage.
  • Syncovery for syncing from the node to the NAS, managed and initiated from the NAS through a web-based UI, via SFTP over SSH+certificate. Worked well although it ran very slowly.

And a list of "must-reads":

  1. For an extensive list of backup software that works on Linux, see restic/others
  2. For a similar blog post that focuses on a CMD-only solution ,see "I found the Holy Grail of backups" by Stavros Korokithakis
  3. For some interesting technical reading on how backups work under the hood, see Duplicacy's github page, especially the section "Comparison with Other Backup Software"
  4. The 3-2-1 Backup Strategy

"Planning"

At first I was really at a loss as to how to approach this problem. There are so many backup solutions out there, literally hundreds, and the proliferation of marketing BS these days makes it difficult to get answers to even simple questions like does this backup solution run on headless Linux? There are also dozens of open-source solutions, many of which are free, and many which are no longer maintained. So I wasn't sure what I want. Do I want to create incremental backups on a schedule? Or do I want to just sync my files somewhere? Do I want to back up to a cloud provider? How much would that cost? Or should I just back up to my NAS? Do I want to manage the backup from the source server or centrally? Centrally from a SaaS website or from my NAS? Really, the task seemed more intimidating the more I researched!

After a bit of thinking some things became clear to me:

  • If I want to back up to my NAS, the communication has to be initiated from the NAS (because I don't want to open the NAS up to the WWW, I've had a QNAP hacked before! and it's on a dynamic IP anyway which I could work around but I like to KISS)
  • Running software in an Ubuntu VM on my NAS isn't a good idea because of performance issues
Sequential throughput is always reduced through VMs regardless what else the VM is doing (like random/IOPs type work) so would affect something like backup...Lots of Qnap people running VS starting to see this.

Cloud Backup Storage Providers

Well, there are many and you can do your own research. For me, the top two are

  1. Backblaze B2 because...it is simple to use, with straightforward pricing, and it's fast and super cheap
  2. rsync.net - also cheap, and I'd use it if I used something like borg/rsync/restic

BaaS

First, I started looking at backup as a service providers. I only found a few that actually support headless Linux hosts:

IDrive 👎

IDrive: it seemed like a great solution at first. Very attractive pricing & the software has a good balance of features and ease of use. Right now it's $52 for 5TB for the first year for unlimited computers! There are lots of good reviews out there, but I suspect many of these are paid. I also read many negative reviews where customers complain about being charged extra for using "extra" space. So, basically, they offer great deals on storage, and then when your backup fills up, instead of deleting old backups (or doing some version of incrementals forever), they start charging you extra. Also, the offers are for the first year only! Now I'm sure IDrive works great for many customers but tactics like this really put me off so I continued my search.

SpiderOak One 👍

SpiderOak actually looks like a good solution. Their pricing is decent, and they have packages that can be installed easily on APT-based distributions as well as many other operating systems. I did not end up trying them because eventually I found Duplicati, which, with B2 as a storage destination, comes out cheaper. SpiderOak also doesn't seem to have a central management console, and their website is quite lean on providing any technicals. Their support is also according to this post by JGM not very good. Also see:

CrashPlan 👍

CrashPlan came out on the top of my list after reading various reviews as well as many forum posts. They provide unlimited storage for $10/device/month. I think they provide an excellent product and I might use them one day - but my nodes cost $10-$20/month, so paying almost the same for backup seems excessive. Also see:

Duplicacy 👎  ♢   3,4k ★ on GitHub

I'm listing this in the SaaS section because they charge a low subcription fee, even though duplicacy does not itself provide cloud storage. I am very much against subscription software that doesn't actually provide an ongoing service. Yes, it's great for the developer but not so much for the user. There is actually a whole discussion about this on their forums where they defend why they don't provide a perpetual license. Yeah...no thanks.

I actually did consider it because it seems to be quite perfect. A good, web-based UI, cross-platform support, and technically an excellent solution.

Duplicacy is the only cloud backup tool that allows multiple computers to back up to the same cloud storage, taking advantage of cross-computer deduplication whenever possible, without direct communication among them. This feature turns any cloud storage server supporting only a basic set of file operations into a sophisticated deduplication-aware server.

They have an excellent technical discussion of why they are better than some of the alternatives which is worth reading. Here's an excerpt:

Other SaaS backup solutions to consider:

Commercial Solutions (perpetual license)

Nakivo Backup 👍👍👍 ← TOP PICK!

I found Nakivo because they have a package on the QNAP "store". It is surprisingly good! One drawback is that it cannot do folder/file-level backup, it backs up the whole machine. But I think that's fine, as I also back specific folders up to the cloud. Pricing is perpetual and quite reasonable. Note: there are 2 security advisories out with regards to Nakivo! Also see:

Backing up 50GB took an hour and nine minutes:

Restore is easy peasy as well. You can select to restore only certain files/folders, or the whole machine. You can download the files or restore to various types of destinations. It supports email notifications, and you can run a pre- or post-execution script (but only on the server, not on the client that you are backing up!). You can even re-brand the app if you like (good for MSP-s)! It's fast and it works.

CloudBerry Backup for Ubuntu 👍👍

I installed CloudBerry and set it up with BackBlaze. By default it runs a web server on port 43210, which I tunneled to localhost with Putty. It worked. The interface is pretty slick as well:

However, note that uploading 360 MB to BackBlaze took 26 minutes! I backed up the same directory with Duplicati in one minute and ten secods half hour before!

I tried turning off compression to see if that would help, but no. Perhaps its a misconfiguration on my end but...that is still 25x slower than Duplicati, which is free!

Also see:

Enterprise & MSP Solutions

There are also a number of enterprise solutions that are able to back up Linux machines. These are usually quite expensive (hundreds of $ / server / year), but there are exceptions. For example,

VEEAM

Veeam offers an agent for Linus for free. Although I did not try it, VEEAM certainly looks like a great solution. Lots of people mention it on various forums. If I were a larger MSP, I'd try them out for sure. The Linux agent can be controlled from the command line, or from a console UI.

It can also work well with their centrallized Veeam Backup & Replication solution, which is available in various licensing schemes, one of which is geared towards small MSP-s at 348 Eur/year for 5 licenses ("Veeam Backup Essentials"). What put me off most was that you can only purchase VEEAM through a reseller, and in places like Hungary the reseller is inevitably also an MSP, so if I bring them into the picture then they will try to just take over my clients :-(

But perhaps one day I will give VEEAM a try. They have a backup solution for almost any kind of device you'd like to create backups from.

MSP360 👍👍

I gave MSP360 a try. You get a central control panel (hosted by MSP360) where you can manage all of your backups. They also support various datasources, including Linux. It does work, albeit not too quickly. There is also no way to recover specific files, or even any kind of file browser where you can dig into your backups. Really their aim is to provide a whitelabel solution for MSP-s that offers backups as part just one service of several. As such, it's more of a swiss army knife for MSP-s - good for many things, but not great for anything specific. If I want to remote into a machine, I'll use SplashTop or AnyDesk thank you.

Also see:

I ended up uninstalling...but this is a comprehensive solution I'd like to keep my eye on! I couldn't find any screen shots on their website so here you go:

Other enterprise-geared solutions:

  • Altaro: very expensive
  • Datto: no pricing, only available through resellers

Open-Source Solutions (with a Web UI)

Duplicati 👍👍👍  ♢   5.5k ★ on GitHub ← TOP PICK!

I found out about Duplicati from a post by Jim Mendenhall and I kept coming back to it. I eventually ended up using it with BackBlaze B2.

Duplicati is a free, open source, backup client that securely stores encrypted, incremental, compressed backups on cloud storage services and remote file servers. It works with: Amazon S3, Backblaze (B2), Box, Dropbox, FTP, Google Cloud and Drive, HubiC, MEGA, Microsoft Azure and OneDrive, Rackspace Cloud Files, OpenStack Storage (Swift), Sia, Tardigrade, SSH (SFTP), WebDAV, Tencent Cloud Object Storage (COS), and more!

One drawback that put me off was that it cannot be managed centrally - you install Duplicati on the machine you wish to back up. However, I ended up trying it out and I was very impressed. The web-based UI is excellent, and it was extremely fast backing up to BackBlaze. 500 MB uploaded in a little over a minute!

Restore also worked perfectly, propery restoring file permissions as well!

Also see:

UrBackup👍👍  ♢   0,8k ★ on GitHub

UrBackup is an easy to setup Open Source client/server backup system, that through a combination of image and file backups accomplishes both data safety and a fast restoration time.

Like BackupPC, UrBackup is a centralized solution. There is also a custom package for QNAP, so it's quite easy to install. I installed it both on my NAS as well as on one of my Ubuntu nodes, but I wasn't able to get them to connect - I believe because with UrBackup, the client must reach out to the server (and not the other way around). The docs confirm this suspicion:

UrBackup is able to backup clients over the internet, enabling mixed LAN and Internet backups. This can be useful e.g. for mobile devices which are not used in the LAN all the time, but are connected to the Internet. As it causes additional strain on the backup file system this feature is disabled by default. You need to enable and configure it in the settings and restart your server to use it. The minimum you have to configure is the server name or IP on which the backup server will be available on the Internet.

Also see:

BackupPC 👍👍  ♢   0,8k ★ on GitHub

BackupPC was also at the top of my list because it is well-maintained and it is a server-client solution, where you install BackupPC onto a server which then pulls backups from the various client machines via rsync or SMB. The server is configured through a web UI. Unfortunately, it is not so simple to install on a QNAP NAS, and if you install it on a VM it runs quite slowly, and getting it to run on one of my Ubuntu nodes proved cumbersome. However, it is an excellent solution that I might try out some day!

BackupPC is a high-performance, enterprise-grade system for backing up Linux, Windows and macOS PCs and laptops to a server's disk. BackupPC is highly configurable and easy to install and maintain.
Features:
- Web Interface
- Data deduplication
- Compression
- Open-source
- No client software needed

Also see:

Bacula

Build Your Free Enterprise Backup Software System with Bacula Tools

Bacula is an interesting solution that I would love to try out one day. However, at the moment it seems overkill. It is really geared towards enterprise environments, with various components that must be set up with care and maintained over time. Too much work to back up a few virtual machines!

Open-Source Solutions (CMD only)

I also stumbled upon several highly touted open-source solutions which do not have any kind of GUI. These are excellent solutions I'm sure but I'm a big fan of having a UI since it makes configuration & monitoring so much easier. So I will list these for reference only, I did not try any of them, since I like a GUI and also most of these only run on flavors of Linux:

Borg backup 👍   ♢   6.6k ★ on GitHub

Loved by many! Large and active community. Latest update was 22 hours ago (as of 10/12/2020). Check out the docs for more info.

Restic 👍   ♢   11k ★ on GitHub

Another great CMD-only backup solution with an active community. Restic supports a wide variety of backends.

Duplicity

Duplicity is actually a command line backup program that allows to backup directories by providing an encrypted and digitally signed version tar-format of the volumes that can be uploaded to a remote or local file server. (from: How to Backup Files with Déjà Dup and Duplicity on Ubuntu 18.04 LTS)

There are many, many more - I recommend searching on github to see what is maintained.

Clients with Desktop GUI

There are also various notable solutions out there that are dedicated to Linux and come with some kind of GUI, some of these are multi-platform as well.

qBackup

A backup solution straight from Japan. Minimalist but multi-platform, and it supports cloud destinations as well.

BackInTime

Back In Time is a simple backup tool for Linux, inspired by "flyback project".

Kopia   ♢   0.34k ★ GitHub

Kopia is a simple, cross-platform tool for managing encrypted backups in the cloud. It provides fast, incremental backups, secure, client-side end-to-end encryption, compression and data deduplication.

Timeshift   ♢   2.4k ★ GitHub

Timeshift for Linux is an application that provides functionality similar to the System Restore feature in Windows and the Time Machine tool in Mac OS. Timeshift protects your system by taking incremental snapshots of the file system at regular intervals. These snapshots can be restored at a later date to undo all changes to the system...
Timeshift is similar to applications like rsnapshot, BackInTime and TimeVault but with different goals. It is designed to protect only system files and settings. User files such as documents, pictures and music are excluded. This ensures that your files remains unchanged when you restore your system to an earlier date. If you need a tool to backup your documents and files please take a look at the excellent BackInTime application which is more configurable and provides options for saving user files.

Déjà Dup

Simple backup tool for GNOME. A trivial-to-use personal backup tool to protect against accidental data loss.

Deja dup comes preinstalled with many Ubuntu deployments. It is basically a GUI for Duplicity.

Underlying Technologies / Scripts

I also wanted to list some of the underlying libs that many of the above backup solutions make use of. It's also quite possible to "just write a script" which makes use of these yourself...but as fulfilling as doing so might be, this was not the solution I was looking for.

rsnapshot   ♢   1.8k ★ GitHub

rsnapshot is a filesystem snapshot utility based on rsync. rsnapshot makes it easy to make periodic snapshots of local machines, and remote machines over ssh.

Scripts!

Synchronization Software

I also considered using some kind of sync software to simply regulary copy over the most important files form my server to my NAS. Some I considered were:

Syncthing (continuous sync)

I've used Syncthing for a while and prefer it to Resilio. It seems to be better maintained and the UI is more intuitive for me to use. I'm assuming it wouldn't work in my situation because for the NAS and the Ubuntu machine to communicate, the communication has to be initiated from the NAS (since it's behind a firewall & I prefer to keep it this way).

Resilio Sync (continuous sync)

A good sync program I use to sync my phone data to my NAS. They also have a package for Linux, see:

GoodSync (manual sync, for Windows)

I also use GoodSync at home. Unfortunately, they recently got on board the subscription-software-gravy-train, and I'd prefer to skip the ride. In any case it is Windows-only software.

Syncovery

Syncovery has a package for QNAP which provides a web interface. I managed to configure it to log into my Ubuntu node via SSH using a certificate. Very impressive! Their web ui could really do with some styling though, and transfer speeds were quite slow (~40kb/s). Perhaps in the future I will use Syncovery again!

FreeFileSync

Also see: How to Install FreeFileSync 10.25 in Ubuntu 20.04 LTS

Alternative Solutions:

Here are some more names I came across during my research. I skipped over these since they didn't look like a good match:

QNAP NAS Packages

Here are some other backup/sync solutions specifically for a QNAP NAS, as well as some that have dedicated packages for QNAP:

QSync

Qsync has an Ubuntu client, although I think for this to work the two machines have to be on the same network.

Veritas Backup Exec

Seems like somewhat outdated & I couldn't find any pricing info on their website. They are compatible with many OS-s. There are also some negative reviews shared on the net:

Very Expensive compared to other backup solutions available now in the market like Veeam, Altero which also have some backdrops like unable to backup physical machines but good comparable solutions. Veritas license for each module is very expensive. Need to buy license for each agent and a server core license and if at all if we need to use more than 1 drive to run the backups , we need to buy extra Library expansion option licenses along with core license.
In present days we have many issues with backing up several servers and with sql server instances. Support asking us to create separate jobs for drives on the servers and Sql server or My sql instances, which doesn not make any sense and after doing that too, the jobs fail. The jobs success has become very inconsistent now a days. We are on Backup exec 16 and they say that they cannot support SQL server 2017.
Several bugs found but Veritas seem incapable or unwilling to acknowledge them, seem to have no way of reporting them either, I was directed to a page for suggesting Functionality and Ideas.

The following I've written about above, but I'll include links to the QNAP-specific packages here:

Syncovery

Duplicati (889k downloads)

QUrBackup (201k downloads)

Duplicacy (62k downloads)

Duplicity (26k downloads)

Bacula (10k downloads)

BorgBackup (9k downloads)

Sources/further reading:

From Linode forums:

Author
László L. Lieszkovszky

Read more posts by this author.
—
Backing up a Headless Ubuntu Web Server (to Cloud & QNAP NAS)
Share this