Backups That Survive Attackers, Admins, and Bad Luck

Backups die in three ways: the attacker deletes them, the automation silently stopped months ago, or the restore turns out to be impossible at the worst moment. A strategy is only real if it addresses all three.

Against attackers: immutability and distance

Modern intruders hunt backups first. The defense is a copy the compromised environment cannot reach: object storage with retention locks (write-once), a separate account with separate credentials, or genuinely offline media. The 3-2-1 rule still stands — three copies, two media, one offsite — with a modern amendment: at least one copy immutable.

Against silent failure: monitor the absence

Backup jobs fail quietly; the cron dies, the disk fills, the credential expires — and nobody notices until restore day. Do not alert on failure alone; alert on missing success. A dead man's switch that pages when no fresh backup has appeared within the window converts silent rot into a Tuesday ticket.

Against restore fiction: rehearse

An untested backup is a hope, not a plan. Schedule restore drills: pick a random backup, restore to isolated infrastructure, verify the application actually starts and the data is coherent. Time it — the drill also tells you your real recovery time, which is what every continuity promise is secretly built on.

Scope honestly

Databases need consistent snapshots or dump procedures, not file copies of live data directories. Configuration, secrets (encrypted), and the runbook to rebuild belong in scope too — a perfect data backup of a system nobody remembers how to assemble is a puzzle, not a recovery.

Start from recovery objectives

A backup strategy is easier to design when you begin with two numbers for each system. The recovery point objective is how much data you can afford to lose, measured in time: if the answer is "an hour of orders", backups must run at least hourly. The recovery time objective is how long the system can be unavailable: if the business stops after four hours without it, a restore that takes a day is not good enough. Different systems deserve different answers. A customer database may need frequent backups and fast restores, while an internal wiki can tolerate a day of loss and a slower recovery.

Writing these objectives down, and agreeing them with the people who run the business, turns backup from a technical chore into a commitment that can be tested.

What to back up beyond the data

Data is only part of what you need to recover. Include application configuration, infrastructure definitions, container images or the instructions to rebuild them, certificates and the encrypted secrets the system needs to start. Document the order in which systems must come back and the dependencies between them. Keep copies of software licences and installation media that may be hard to obtain quickly. Many restore drills fail not because the data is missing but because nobody can reconstruct the environment the data belongs to.

Databases need special care

Copying the files of a running database rarely produces a usable backup, because the files change while they are being copied. Use the database's own tools: logical dumps for smaller databases, physical backups with write-ahead or transaction log archiving for larger ones, or storage snapshots coordinated with the database so that the copy is consistent. Log archiving allows point-in-time recovery, which is invaluable when the problem is not a lost server but a bad change: you can restore to the minute before someone dropped the wrong table.

Immutability in practice

Immutable storage means that once a backup is written it cannot be modified or deleted until a retention period ends, even by an administrator. Many object-storage services offer retention locks for this purpose. Configure the lock in compliance mode where available, so that it cannot be shortened, and choose a retention period long enough to cover the time it might take to discover an intrusion. Keep the credentials that write backups separate from those that manage the storage account, and protect the account with multi-factor authentication, so that stolen production credentials cannot reach the backups at all.

Offline and offsite copies

Distance protects against disasters that affect a whole site or a whole cloud account. An offsite copy may live in a different region, with a different provider or on physical media stored elsewhere. Offline copies — disks or tapes disconnected from any network — remain the ultimate protection against an attacker with full control of your environment. For small organizations, a rotating set of encrypted external drives, taken offsite on a schedule, is still a practical and effective option.

Encryption and key management

Backups contain everything an attacker would want, so encrypt them in transit and at rest. The encryption keys are as important as the backups themselves: store them separately, protect them with strong access controls and make sure that more than one trusted person can retrieve them. A backup encrypted with a key that was stored only on the server that failed is effectively lost. Test key recovery as part of every restore drill.

Monitoring backups properly

Beyond alerting on missing success, monitor the characteristics of each backup. A sudden change in size can reveal a problem: a backup that shrinks dramatically may mean data was deleted or encrypted, and one that grows unexpectedly may include files that should not be there. Track how long each job takes, how much storage is used against the budget and how far back the oldest restorable copy reaches. Present these figures on a simple dashboard that someone reviews every week.

Ransomware-aware backup design

Ransomware operators often spend days inside a network before encrypting it, and they target backup systems deliberately. Assume that the most recent backups may contain the attacker's tools or already-encrypted files. Keep enough history to restore from before the intrusion began, protect backup management interfaces with strong authentication and separate credentials, and make sure the backup server is not joined to the same administrative domain as the systems it protects. When you restore after an attack, scan restored systems before reconnecting them.

Cloud services are not backups by themselves

Software-as-a-service platforms keep your data available, but availability is not the same as backup. Deleted items may be recoverable only for a limited time, a compromised account can delete data legitimately, and a synchronization error can propagate damage everywhere. For critical cloud services — email, file storage, source code, customer relationship systems — review what the provider actually guarantees and consider a separate backup into storage you control.

Documentation that survives the disaster

Restore procedures must be available when the systems they describe are not. Keep the recovery runbook in more than one place, including at least one copy outside your primary environment, and include the contact details of the people and suppliers you will need. Write it so that someone other than the usual administrator can follow it, because the person who knows the system best may be unavailable on the day it matters.

A practical checklist

Define recovery objectives per system. Back up data, configuration and the means to rebuild. Use database-aware methods. Keep at least one immutable copy and one offsite or offline copy. Encrypt everything and protect the keys separately. Alert on missing and abnormal backups. Rehearse restores on a schedule and record how long they take. Review the strategy whenever systems change. Backups built this way survive attackers, administrators and bad luck alike.

Versioning, snapshots and their limits

Storage snapshots and file versioning are valuable tools, but they are not complete backups on their own. Snapshots usually live on the same storage system as the original data, so a failure of that system, or an attacker with administrative access to it, can destroy both. Versioning protects against accidental overwrites and some ransomware, but versions can be deleted by anyone with sufficient permissions. Use snapshots for fast, convenient recovery from everyday mistakes, and rely on separate, immutable and offsite copies for serious incidents.

Who owns the backups

Every backup system needs a named owner who checks the dashboard, investigates failures, schedules restore drills and updates the plan when systems change. Without an owner, backups degrade quietly until the day they are needed. In a small organization the owner may also run other systems, which is fine as long as the responsibility is explicit and the weekly check is on someone's calendar.

The bottom line

A backup is worth only what you can restore from it, when you need it, after the worst plausible event. Design for attackers, silent failures and untested procedures, and prove the design regularly with drills. Organizations that do this treat even a serious incident as a recoverable problem rather than an existential threat.

Start this week with one step: pick your most important system, write down its recovery objectives and run a restore of its latest backup into an isolated environment. The result will tell you more about your real resilience than any policy document, and it will show you exactly what to fix next.