A Backup Is Not Real Until the Restore Has a Clock

A dashboard saying “backup completed” proves that a job wrote something. It does not prove that the backup contains the right data, that the encryption key is available, that the database version is compatible, or that a team can restore service before customers lose patience.

A backup becomes an operational capability only after a restore succeeds against a clock and produces evidence that the application data is usable.

Define the loss and time targets first

Use two business measures:

• Recovery point objective, or RPO: the maximum acceptable amount of data loss, expressed as time. An RPO of 15 minutes means the recovery design must be able to return close to a state no more than 15 minutes before the incident.
• Recovery time objective, or RTO: the target time to restore the service or a defined minimum capability.

Do not assign one pair to the entire company. Authentication, payments, customer documents, analytics, and marketing content may need different targets. A four-hour RPO could be acceptable for derived analytics and disastrous for completed orders.

Targets must match architecture. A daily database dump cannot honestly support a 15-minute RPO. Point-in-time recovery can narrow data loss, but only if write-ahead logs are retained, restorable, and tested. PostgreSQL’s current backup and restore documentation distinguishes SQL dumps, file-system backups, and continuous archiving for point-in-time recovery; each has different constraints.

Inventory all state, not only the primary database

List everything required to recreate a useful customer workflow:

• relational and document databases;
• object storage and uploaded files;
• search indexes and whether they can be rebuilt;
• queue state and scheduled jobs;
• identity-provider configuration;
• encryption keys and secret versions;
• DNS, certificates, and domain access;
• infrastructure definitions and application artifacts;
• provider records required for reconciliation;
• audit logs and compliance evidence.

Label each item as authoritative, reconstructable, or disposable. Document the reconstruction source and expected duration. “Search is derived” is only useful if the source data and index-building code both survive the incident.

Beware circular dependencies. A backup encrypted with a key accessible only through the failed cloud account is not independent. A runbook stored only inside the unavailable identity provider cannot guide recovery. Keep recovery credentials and instructions protected, but reachable through a separate, tested path.

Restore into a disposable environment automatically

The routine test should create an isolated environment, retrieve a selected recovery point, restore it, start a compatible application version, validate data, and destroy the environment after evidence is retained.

Automate the repeatable mechanics, but keep destructive safeguards explicit:

1. Generate a unique test environment and verify it cannot send customer email, webhooks, or payments.
2. Select a backup by timestamp, not simply “latest.”
3. Record backup size, checksum or provider identifier, database engine version, and encryption-key version.
4. Restore schemas, data, extensions, roles, and required objects.
5. Run application migrations only according to a documented compatibility plan.
6. Start a restricted application against the restored state.
7. Validate business invariants and record elapsed time.

Production data in a test environment creates privacy risk. Limit access, use an isolated account or network, block outbound side effects, protect logs, and delete the restored copy on schedule. Where possible, use protected synthetic or masked data while still running periodic controlled tests of the real recovery mechanism.

Validate meaning, not only row counts

A database can accept connections and still be unusable. Use layered checks:

• database opens without corruption errors;
• expected schemas, extensions, and migration versions exist;
• counts for critical entities are plausible relative to the backup time;
• foreign-key and uniqueness checks pass;
• a sample of tenants has users, subscriptions, and expected records;
• object references resolve to files with valid checksums;
• encrypted fields can be decrypted with the intended key;
• the application can perform a read-only critical workflow;
• payment and invoice totals reconcile for a closed period.

Pick invariants that express the product. For example, every paid invoice must belong to a customer, every active subscription must map to a plan, and every stored document reference must point to an object or a known deletion record.

Keep the restore report: source recovery point, start and finish time, target RPO and RTO, actual data freshness, validation results, exceptions, and owner. An alert that the job finished is less valuable than evidence showing what was restored and verified.

Match application and database versions

Restore procedures often assume “current production” can read any backup. That fails after incompatible migrations, extension changes, or application upgrades. GitLab’s restore documentation requires restoring to the exact same version and edition before proceeding, a concrete example of why version metadata belongs with the backup.

Retain deployment artifacts or immutable image references long enough to run the application version associated with the recovery point. Test both directions that matter: restoring an older backup into its compatible application, and applying the documented upgrade or migrations toward the current release.

For destructive migrations, preserve a rollback or forward-recovery plan before deployment. A backup may meet its storage policy yet fail the RTO if restoring requires rebuilding an old toolchain from scratch.

Separate backup protection from normal administration

If the same compromised administrator can delete production and all backups, retention is not a recovery boundary. Use provider capabilities such as separate backup accounts or projects, restricted deletion roles, immutability or retention locks where justified, MFA, and alerts on policy changes and deletions.

Protect the recovery path without making it dependent on one founder. At least two authorized people should know how to initiate it, with high-risk steps requiring controlled approval. Exercise the credentials. A sealed emergency token that expired a year ago is documentation, not access.

Routine restores and disaster exercises are different

A weekly or monthly automated restore tests media, permissions, compatibility, and data integrity. A less frequent disaster exercise tests people and systems together: detection, incident command, customer communication, DNS changes, vendor escalation, data reconciliation, and return to normal operation.

Use realistic scenarios:

• an operator deletes a production database;
• credentials are compromised and the primary account is untrusted;
• a region is unavailable;
• application writes continued after the last clean recovery point;
• object storage is intact but the database mapping is damaged.

GitLab’s public 2017 database incident postmortem remains instructive because several backup and replication assumptions failed under pressure. The lesson is not that one tool is unreliable. It is that redundant-looking mechanisms can share untested failure modes.

Next steps

1. Assign RPO and RTO targets to the three most important data domains.
2. Inventory every authoritative dependency needed to restore one customer workflow.
3. Run a manual isolated restore this week and time it.
4. Convert the steps and validation checks into a recurring job.
5. Schedule a cross-team disaster exercise and record the gaps as owned work.

Backup age, size, and job status are useful signals. The decisive metric is restore evidence: the team recovered a known point, within a measured time, and proved that the product’s important data still made sense.