When I take over an existing infrastructure, I always ask the same question before looking at anything else: when did you last restore?
The answer is almost always the same. Never, or a long time ago, or only a single isolated file. Meanwhile the dashboard has shown green for months, and everyone sleeps soundly.
Green means nothing
A backup job that finishes without error proves exactly one thing: some bytes got written somewhere. It doesn’t prove those bytes are readable, that they form a bootable system, or that you know in what order to bring the machines back up on a Tuesday morning with fifteen people waiting.
The three failures I see most often:
- The backup runs, but silently excludes a volume added two years ago. Nobody updated the list.
- The remote repository has been full for six weeks. The alert went to an address that belongs to someone who no longer works there.
- Everything is intact, but the encryption key exists in exactly one place: on the machine that just burned down.
None of these situations produces a failed job.
The quarterly routine
What I put in place, systematically, fits on one page:
- Pick a machine at random from the list, not always the same one.
- Restore it on an isolated network, with no access to production.
- Boot it, open the business application, check one real piece of data.
- Time it. The time matters as much as the result.
- Write down what went wrong. There’s always something.
Step 5 matters most. The first restore always reveals a missing step in the procedure: a password that existed nowhere, a boot order that turns out to matter, a forgotten network dependency. Exactly what you don’t want to discover on the day of an actual disaster.
The number that matters
It’s not backup frequency. It’s the time between the disaster and getting back to work, measured once for real. Until that number has been measured, it’s not a recovery plan: it’s an intention.