A backup nobody has restored is an assumption

Ask your IT provider for the date and result of the last restore test, not just the latest successful backup job. A green tick shows that a job ran. It does not show that the application can be recovered.

Backups are easy to configure and easy to believe in. Opening one is work, and while nothing appears to be wrong there is little to prompt it.

A backup nobody has ever restored is not a backup. It is an assumption about a file you have never read.

So this week we tested our own.

What we actually did

We keep an internal knowledge base: procedures, decisions, the things a growing team keeps asking each other. It runs on our own hardware. Its backups are encrypted and written to a second machine.

The test was deliberately unkind to ourselves. We took the recovery kit, the key and password needed to open the repository, from a copy held away from the machine being backed up. We used a different computer, running a different operating system to the one the system normally lives on. Then we restored.

The repository opened and the snapshot verified. We restored the backed-up application state on another computer and checked sign-in, scoped access and a test edit: a person who should not see a document still could not.

It worked. That is the part we expected to be the story.

What broke

The day before, we had built alerting for those backups, so that a failure would reach a phone rather than sitting in a status file nobody opens.

Configuring it failed four times in a row.

The first failure was a mail server rejecting a password. The provider returned a precise, useful error message. Our system caught that message, reduced it to a single word, failed, and displayed nothing else.

The second was a form that discarded a typed credential when it rejected the entry, so every correction resubmitted a blank field.

The third was a queued test message sitting behind a fifteen minute cooldown, indistinguishable on screen from a broken feature.

The fourth was a chat integration returning a clear explanation of exactly what was wrong, which our system also discarded.

The providers were right to reject invalid credentials or incomplete configuration. Our application was wrong to hide the useful explanation, discard a correction, or obscure a queued test. Security checks alone had not exercised the operator's recovery path.

The part worth generalising

We had reviewed that alerting code carefully before anyone touched it. It passed its tests. It had been examined line by line for the kinds of faults that matter: could it be bypassed, could it leak a credential, would it fail safely. The answers were good.

That review checked credential handling and safe failure. It had not tested whether an operator could understand a failure and correct it.

That distinction matters more than it sounds. A system that works but cannot explain itself is fine while everything is fine. The moment something goes wrong, it becomes a system nobody can operate, and it goes wrong precisely when the person using it is already under pressure.

This is why the alerting failures are worth more to us than the successful restore. The restore confirmed something we hoped was true. The failures taught us something we did not know.

What we changed

The error a provider returns is now kept and shown, as a category and a code rather than raw text, because raw provider errors can carry credentials in them. A rejected form keeps what you typed. A queued test says it is queued and when it will send, rather than looking like a failure. And a pending test can no longer block a real alert, which was a defect we found in our own work afterwards: a stuck test could have silently suppressed a genuine backup failure warning.

We also wrote down what the test did not prove. Both machines are in the same building. We did not test the loss of that building, and there is no independent off-site data copy yet. Saying that plainly is part of the exercise, because a recovery claim with unstated limits is just a different kind of assumption.

If you run your own infrastructure

You do not need our tools to do this. The exercise is the same everywhere:

Take the credentials you would actually have in an emergency, not the ones on the machine that just died. Restore onto hardware you do not normally use. Start the thing up. Then check the boring parts: that permissions survived, that the work included in the chosen backup is present, that the people who should not see something still cannot.

Write down how long it took, and write down what the test did not cover.

Do it before you need it, because the day you need it you will also be dealing with whatever caused the need.

Why we published the failures

We could have written the first half of this and stopped. The restore worked, which is a good thing to be able to say.

But an infrastructure company that only publishes its successes is asking you to take its word for the rest. We would rather show the part where our own new feature failed four times in a row in front of us, and what we did about it, because that is a better description of how this work actually goes.

If you want to test your own recovery and are not sure where to start, or you would rather someone did it with you, that is the kind of work we do.

Visit us at www.asoltd.info or get in touch.