Every backup works until the day it is needed. The snapshot sits on the platform that just went down. The nightly tarball has been failing silently since a disk filled up in March. The database dump was a file copy taken mid-write and restores to nothing. Backups are the one part of a server you cannot judge by looking at them — only by restoring them. This is how to build one that survives hardware failure, your own hands and someone else holding your credentials, without giving a readable copy of the machine to whoever stores it.
Four things destroy a server, and each needs something different
Design the backup around the failure, not around the tool. Four scenarios account for essentially every real loss, and each demands something different from the copy you keep. A setup that handles one of them feels completely adequate until it meets another.
Hardware or host failure
The disk, the node or the datacenter goes. Any copy held anywhere else recovers you. This is the only failure a same-platform snapshot reliably handles, which is exactly why so many people believe snapshots are enough.
Your own hands
A wrong flag, a migration script pointed at production, a cleanup that removed more than intended. The damage replicates instantly to anything that mirrors, so what saves you is history — a version from before the mistake, not a current copy of after it.
Compromise and ransomware
An attacker with root has exactly the reach your backup job has: the same credentials, the same destination, the same schedule. If the server can delete its own backups, they will be deleted. Survival depends on the copy being append-only or out of reach entirely.
Losing the account, not the server
A billing lapse, a suspended account, a takedown in the wrong jurisdiction. The hardware is fine and you simply cannot reach it. Only a copy under a different provider, with a different payment path, helps here.
Write down which of the four you are actually defending against before you choose anything. A nightly copy into a second directory on the same disk covers exactly one of them — the least likely one — while feeling entirely like a backup. Most setups that fail in production fail because nobody ever named the scenario.
Snapshots are not backups
The words get used interchangeably and they describe different objects with different failure domains. Knowing which one you have decides whether you have anything at all.
Use snapshots for what they are genuinely excellent at: a five-second undo before you touch the kernel, the bootloader or a database schema. Take one, do the risky thing, delete it when it works. What you must never do is let "there is a snapshot" become the reason there is no backup — a snapshot shares its fate with the platform holding it, and a suspended account takes both at once.
3-2-1, and the two numbers everyone drops
The old rule survives because it describes independence rather than a filing system: three copies of the data, on two different kinds of storage, one of them somewhere else. Two additions handle failures that were rare when the rule was written and are routine now.
- Three copies. The live data plus two backups. Two copies means you are one bad restore away from none.
- Two media, or two providers. On rented infrastructure, "media" really means administrative independence — a second copy under the same account, on the same platform, paid from the same balance, is one copy with extra steps.
- One offsite. Different building, different network, different jurisdiction if that matters to you. /locations is the practical version of this: pick a region that fails independently of the one you are protecting.
- One immutable or offline. A copy the server itself cannot delete, because whoever owns the server owns its credentials too.
- Zero unverified restores. A backup you have never restored is a hypothesis. The number that counts is how many restores you have completed, not how many jobs reported success.
Back up the data, rebuild the rest
The instinct is to image everything. It is expensive, slow to restore, and it faithfully preserves the compromised binary, the broken package state and the config drift you no longer remember creating. A server holds three kinds of content, and only one of them belongs in the backup.
- Include /etc, /home, /root, /srv, your web root, your application's state directory, named container volumes, and a directory of fresh database dumps.
- Include the deployment recipe itself — compose files, Ansible or shell scripts, the notes you wrote at 2am. Keep it in version control too, but put a copy in the backup so recovery never depends on a second service being reachable.
- Exclude /proc, /sys, /dev, /run and the temporary directories. They are kernel interfaces and scratch space; copying them wastes time or hangs the job outright.
- Exclude package caches, build directories, dependency trees and virtualenvs — anything a build step regenerates. On a typical application server this is most of the disk.
- Exclude the live database files if you are dumping the database properly. Backing up both means the larger, useless copy is the one someone restores at three in the morning.
- Do not exclude hidden files. Half of what matters on a Linux box starts with a dot.
Databases will not survive a file copy
This is the single most common reason a restore fails. A running database keeps state in memory and writes out of order; copying its files while it runs captures a torn, half-written state that may restore, may restore silently corrupted, or may not restore at all. Take a proper dump, then back up the dump.
- PostgreSQL: pg_dump per database, or pg_dumpall for the whole cluster including roles. Where losing a day is unacceptable, add WAL archiving so you can recover to a point in time instead of to last night.
- MySQL or MariaDB: mysqldump with --single-transaction gives a consistent snapshot on InnoDB without blocking writes. It does not on MyISAM — one more reason not to run MyISAM. For large datasets, a physical tool such as mariabackup restores far faster than replaying a dump.
- SQLite: never copy the file. Use the .backup command or VACUUM INTO, which take the lock correctly. Copying a database with an active write-ahead log is a coin flip you will lose eventually.
- Redis: trigger a background save and back up the resulting snapshot file, or run with append-only mode and back up the log. Copying a live snapshot mid-write gets you a truncated file that loads as an empty dataset.
- Containers: the volume is the data. Stop the stack for the seconds a copy takes, or run the dump tool inside the container — but do not tar a running database's volume and call it a backup.
- Anything else with a persistent process — search indexes, message queues, ledger daemons — has its own consistent-export command. Find it now, not during the outage.
Filesystem snapshots solve this from the other direction: LVM, ZFS and btrfs freeze a consistent view of the volume in milliseconds, and you back up the frozen view while the database keeps serving. That is the right answer for datasets where a dump takes hours. It is not a reason to skip the dump for a 200 MB database, where the dump is simpler, portable across engine versions, and readable by a human when something has gone strange.
Choosing a tool
Four tools cover almost every case. The property that matters most is where encryption happens. If data is encrypted only in transit, and then at rest by the storage provider, the storage provider can read it — and the backup has quietly become the weakest point in a system you hardened everywhere else.
Whatever you choose, the passphrase or key must exist somewhere other than the server being backed up. This is the trap that catches careful people: the repository key stored in /root, faithfully backed up inside the repository it unlocks. Print it, or keep it in a password manager you can open from a machine that is still alive. A repository you cannot decrypt is indistinguishable from no backup at all.
Where the second copy should live
The destination is a jurisdiction and a billing relationship as much as it is disk space. Four options, in rough order of how independent they are from the server you are protecting.
A second server in another region
The straightforward answer: a cheap instance in a different country, reachable over SSH, running nothing but sshd and a repository. A 1 GB /vps plan holds a small fleet's backups, and the account can be locked to a single forced command so a stolen key opens no shell.
Storage-class disk
Once the dataset runs to hundreds of gigabytes, HDD capacity on an unmetered uplink costs far less per terabyte than NVMe, and backup writes have no need of NVMe latency. /storage is built for this: terabyte-class RAID-protected disk, full root, no content inspection.
S3-compatible object storage
Convenient, priced per gigabyte, and often with object-lock support that gives real immutability. Read the egress pricing before relying on it — pulling a terabyte back in an emergency is a poor moment to discover what retrieval costs.
Hardware you own
An external disk or a machine at home, pulled rather than pushed. Slower and manual, and the only copy on this list that no provider, court order or billing dispute can touch. Worth keeping for data you genuinely cannot recreate, even if it lags a week behind.
Match the backup's privacy properties to the server's. Encrypting a disk with LUKS and then shipping nightly plaintext backups to a bucket registered to your card undoes the entire exercise: the data is now readable, and it is filed under your name. If the server was worth paying for anonymously, so is the copy — encrypt client-side, and buy the destination the same way you bought the origin.
Building it, step by step
- 1
Decide the two numbers first
How much data you can afford to lose — the gap between runs — and how long you can afford to be down. Everything else follows from those two answers. Hourly backups of a static site are theatre; nightly backups of an order database are a decision to make deliberately rather than inherit from a tutorial.
- 2
Create the destination
A second instance in another region with its own SSH key and a dedicated unprivileged user whose home is the repository. Nothing else runs there. Restrict the key to a forced command so a stolen credential can only append backups, never open a shell.
- 3
Generate the key and store it elsewhere
A long random passphrase, recorded somewhere that survives the loss of the server. Initialise the repository, then prove you can list its contents from a third machine using only what you wrote down. If you cannot, fix that before writing a single backup.
- 4
Dump the databases first
A short script that writes consistent dumps into a staging directory and exits non-zero if any of them fails — run before the file backup, not alongside it. A job that carries on after a failed dump is how people end up with thirty days of zero-byte files.
- 5
Back up the paths that matter
Point the tool at the include list, apply the excludes, and sanity-check the first run. If a fresh repository of a 40 GB server comes to 300 MB, something is being silently skipped; if it comes to 38 GB, your excludes are not working.
- 6
Schedule it and make failure loud
A systemd timer with a randomised delay, or cron if that is the habit. Then make the job report: a heartbeat to a monitor on success, and an alert when the heartbeat stops arriving. Silent failure is the normal failure mode of backups, because nothing visibly breaks when they stop — until everything does.
- 7
Set retention and actually prune
Hourly for a day, daily for a fortnight, weekly for a couple of months, monthly for a year. Then run the prune and confirm the repository stops growing. Retention that is configured but never executed fills the destination and takes the backups down with it.
- 8
Restore something today
Not a test job — a real file, to a scratch directory, opened and checked. Then put a date in the calendar for the whole machine. The first full restore always surfaces something: a missing path, a permission, a certificate, a database user that only ever existed on the old box.
Push the backup from the server outward and you have accepted that anyone with root on it can destroy every copy. Run it the other way — the backup host connects in, pulls, and disconnects — and a compromised server cannot reach the repository at all, because it holds no credentials for it. Pull is more work to set up and it is the single largest improvement most setups can make.
Retention, and why longer is cheaper than it sounds
Retention usually gets set by whatever fits the disk, then forgotten. It deserves one deliberate thought, because it decides which mistakes are recoverable. A seven-day window catches a file deleted on Tuesday. It does not catch corruption that started six weeks ago and surfaced when a report came out wrong, or an intruder who sat quietly on the machine for a month before acting.
Deduplication makes this far cheaper than the table suggests: the second run against a mostly-unchanged server stores only what changed, so a year of monthly restore points on a 40 GB machine typically costs a few gigabytes rather than half a terabyte. Set the policy on what you need to recover from, then check the bill. You will usually find you can afford the generous version.
Immutability: the part that stops ransomware
Everything above assumes the adversary is entropy. If the adversary is a person with root, an ordinary backup job is an instruction manual — the credentials are on the box, the destination is in the config, and the repository gets wiped before the encryption starts. Four mechanisms break that chain, and any one of them changes the outcome.
- Append-only repositories. Borg's server-side append-only mode, or restic's REST server in append-only mode, accept new data and refuse deletions. The server can write; only you, from somewhere else, can prune.
- Pull-based backups. The backup host initiates the connection and holds the only credentials. The production server has no key, no destination address, and no route to the repository at all.
- Object lock. S3-compatible storage with a retention period enforced at the bucket level, where deletion is refused by the storage layer regardless of what the credentials would otherwise permit.
- Separate credentials per host. One compromised machine should not expose every other machine's history. Distinct keys, distinct paths, distinct restrictions.
- One copy that is genuinely offline. A disk that is unplugged is immune to every remote attack ever written. Unfashionable, and unbeaten.
The restore drill
A restore is a procedure, and a procedure nobody has performed is fiction. Run this once now and once every six months, and write down what you learn — the notes end up as valuable as the data.
- 1
Deploy a blank instance
Same operating system version, nothing else installed. A short-lived VPS is enough, and the whole exercise costs less than lunch.
- 2
Restore using only what you wrote down
The repository address, the passphrase, the commands. If you need something that exists only on the server you are pretending is dead, you have just found the flaw — on a day when finding it costs nothing.
- 3
Bring the data back before the application
Load the database dump, put the files where they belong, then fix ownership and modes. Ownership is the usual surprise: numeric user IDs from the old box rarely line up on the new one.
- 4
Start the service and actually use it
Not a status command — log in, load a page, run a query, send a message. A service that starts is not the same thing as a service that works.
- 5
Time it, and write the number down
How long did the whole thing take? That is your real recovery time, and it is almost always several times the estimate. Keep the notes beside the passphrase and update both whenever the stack changes.
What it costs, honestly
Backup is the cheapest insurance in infrastructure, and it is routinely skipped on price. Some concrete numbers for a small server, assuming the data compresses and deduplicates the way ordinary data does.
Compare any of those to the cost of the outage they prevent. Naming the number is the point — the argument against backups is never really about money once the money has been written down.
Where the host fits in
A backup plan has the same two dependencies as the server it protects: somewhere independent to put the copy, and a way to pay for it that does not create a fresh record of who you are. Both matter more here than at the origin, because the backup is a complete copy of everything the origin was protecting.
Every /vps and /storage plan here deploys from a prepaid crypto balance with no KYC, across fifteen regions listed on /locations — so the second copy can sit under a different jurisdiction from the first without a second identity check anywhere in the chain. Instant snapshots are included on every instance for the five-second undo, unmetered bandwidth means neither the first upload nor the emergency restore is a metered event, and /storage adds terabyte-class RAID-protected disk from $8.99/mo with no content inspection. /pay-with covers the accepted coins, /offshore-hosting sets out what jurisdiction genuinely changes, and /guides has the companion pieces on disk encryption and on what a host can and cannot see about a machine.
The checklist
- Name the failure you are defending against before choosing a tool.
- Treat snapshots as an undo button, never as the backup.
- Back up data and configuration; rebuild the operating system from a script.
- Dump every database with its own consistent-export command, and back up the dump.
- Encrypt client-side, before anything leaves the machine.
- Keep the repository passphrase somewhere that survives the loss of the server.
- Put at least one copy under a different provider, region and payment path.
- Make one copy append-only, pull-based or offline so a compromised root cannot erase it.
- Alert on the absence of a successful run — failures are silent, and silence is the symptom.
- Restore a file today and the whole machine twice a year. Time it and write it down.
Are provider snapshots enough on their own?
No, and this is the most common gap in otherwise careful setups. A snapshot lives on the same platform as the volume it copies, so it does not survive a platform-level failure, an account suspension or a billing lapse — three of the scenarios where you most need it. It also usually keeps only a short history, so it will not recover a file deleted last month or a corruption that began six weeks ago. Snapshots are excellent as a five-second undo before a risky change: keep them, use them daily, and keep a real backup somewhere else.
restic or Borg — which should I use?
restic if the destination is object storage or a plain SSH account, because it needs nothing installed on the far end and speaks S3 natively. Borg if the destination is a Linux box you control and you want its append-only server mode and slightly tighter compression. Both deduplicate, both encrypt and authenticate client-side, both are mature and widely deployed, and either is a defensible choice. The wrong answer is spending a month comparing them while the server has no backup at all — pick one this afternoon and change your mind later if it ever matters.
How often should I back up a VPS?
The interval is simply the maximum amount of work you are willing to redo. For a static site, weekly is honest. For anything users write to, nightly is the floor and hourly is cheap once deduplication is doing the work — the second run of the day usually stores a few megabytes. Weigh the value of one day's data against the cost of storing twenty-four restore points instead of one, and the answer is normally obvious.
Should I back up the whole disk or just my data?
Data and configuration, in almost every case. A full disk image restores the machine exactly as it was, including the compromise you are recovering from and the package state you no longer understand, and it is slower to both create and restore. A file-level backup of /etc, your application data and your database dumps, paired with a script that rebuilds the operating system, restores faster and cleaner. Image the disk when you need bit-exact forensic preservation, or when the machine is a black box somebody else built.
My provider says the disks are encrypted — is my backup encrypted?
Not in any sense that helps you. Provider-side encryption protects against a drive being carried out of a rack; the provider holds the key, so the data remains readable by the provider and by anyone who can compel the provider. Client-side encryption — restic, Borg, rclone's crypt layer, or an encrypted archive — means what arrives at the destination is meaningless without a passphrase that never left your machine. For a privacy-critical stack this distinction is the whole point, because the backup is a complete copy of everything the server was protecting.
How do I stop ransomware from encrypting my backups too?
Assume that anything the server can reach, an attacker with root can destroy — the credentials, the destination and the schedule are all sitting on the box. Break that reach in one of three ways: an append-only repository that accepts writes but refuses deletions, a pull model where the backup host connects inward and the server holds no credentials at all, or object storage with a lock the storage layer itself enforces. Then add long retention, so a slow and quiet compromise does not simply age out of the window before anyone notices, and keep one copy offline where no remote attack can reach it.
