Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Offline Backup and Restore

RiffDB’s POC backup and restore operations are offline, exclusive maintenance operations. They are exposed through the public gRPC API, Rust client, and riffdb CLI. They are not MCP tools, and neither the CLI nor a client receives or resolves a server filesystem path.

Configuration

For an already-active changelog V3 database, the offline incarnation stamp validates retained receipts and source fences before atomically installing a new lineage anchor and clearing replication-local progress. A destructive restore cannot continue the old replication tail; a receiver needs a new bootstrap. The public maintenance operation ID and retry protocol below are unchanged.

The offline retention-watermark stamp also validates retained V3 history before returning from an equal-value retry. A changed watermark and its exact V3 receipt commit together in the existing hardened transaction. This adds no commit or flush and changes neither watermark normalization nor the recorded tombstone chain-root binding.

Retention also runs online inside riffdbd when its configuration names a retained-sequence target; see Online retention. The online lane uses the same fencing set, tombstone chain, and watermark as the offline operations below, which remain available on a stopped database.

Offline and online retention both respect each registered follower’s durable acknowledged application frontier. A follower that has acknowledged no application commit pins the watermark at zero; a faster follower cannot release a slower follower’s fence. Retention status reports FollowerLowWater when this is the binding minimum. Unreadable or inconsistent V3 hold evidence refuses collection and prune. Budget exhaustion degrades typed Health and never releases a hold by itself. The internal coordinator supports audited retirement and configured sequence expiry, with durable degradation before expiry release and bounded background scheduling. Follower registration and retirement are available through the primary operator interfaces. Fencing and promotion remain POC-scoped.

riffdbd requires an absolute --backup-root path. The path must be lexically disjoint from the database file, capability digest keys, and idempotency digest keys. Only the server joins a checked backup name beneath this root.

The service account needs permission to:

  • read and write the backup root;
  • create and synchronize <backup-root>/.maintenance;
  • create and retire immutable named backup directories; and
  • close, reopen, and, for restore, replace the configured database target.

Do not place the database or secret key files beneath the backup root.

Names and Operation IDs

A backup name contains 1 through 64 ASCII bytes. Its first byte is a lowercase letter or digit. Remaining bytes may also contain - and _. Dots, path separators, drive prefixes, whitespace, and Unicode are rejected.

Each public API start request carries a UUIDv7 maintenance operation ID. SDK callers generate it; the CLI generates it for the operator and returns it in accepted or uncertain output. Keep that ID until the operation reaches a terminal state. An API/SDK retry uses:

  • a fresh transport request ID;
  • the same maintenance operation ID; and
  • the exact same operation kind, backup name, and restore confirmation.

Reusing an operation ID with different semantic input fails with a stable input mismatch and starts no work.

Create a Backup

riffdb --config "$HOME/.config/riffdb/client.toml" \
  backup create before-upgrade

The CLI prints the maintenance operation ID and the current bounded status. Poll an accepted operation with:

riffdb --config "$HOME/.config/riffdb/client.toml" \
  backup operation 01900000-0000-7000-8000-000000000000

The server durably creates or resolves the external maintenance receipt before reporting acceptance. It then stops ordinary admission, joins the local outbox, online retention, and background scrub owners before draining the coordinator, drains accepted work under a fixed deadline, closes the database, publishes the complete immutable backup, and performs the full startup validation again before returning to readiness.

Remote container disaster drill

The release Compose topology can exercise total database-volume loss through the same public maintenance API used by operators:

./scripts/remote-compose-acceptance --backup-restore

The drill deploys and seeds the release-owned four-subject workload corpus, creates and polls a backup from the separate operator container, creates a post-backup authority record, stops the database container, empties only its freshly allocated test data root, starts an empty replacement, and restores the backup through the public TLS endpoint. It then reruns the generated OpenFGA, MLflow, Better Auth, and Woodpecker observations, including a typed Better Auth user/session graph whose secret token is never placed in the observation or receipt. The retained Payload operational-query shape runs in the same corpus as a post-alpha regression and is labeled separately. The drill proves that the backed-up operator and application authorities work and that the post-backup authority disappeared. The operator container receives no database or backup mount and never supplies a server filesystem path.

The complete adapter gate also validates each domain’s immutable conformance manifest and installation/evolution plans before running that destructive corpus:

./scripts/adapter-disaster-recovery-acceptance --all-domains --remote

The drill writes a secret-free JSON receipt beneath target/adapter-disaster-recovery/. The receipt binds both maintenance operation identities and input hashes, the verified backup-manifest checksum, the destroyed post-backup suffix, the exact contract bundle, startup validation, the accepted four-subject inventory, the separately classified Payload regression, and the reconciled adapter-observation digest. The backup manifest is the frontier-bearing durable artifact; the receipt binds its checksum rather than decoding storage-format bytes in application or operator code. Recovery proves the declared lifecycle observation and does not promote any subject’s evidence classification. In particular, the MLflow and Woodpecker matrix results do not become upstream- framework conformance through backup and restore.

An existing named backup is never silently replaced.

Retire a Backup

Published immutable backups must not be deleted directly. Retire one exact checked name through the authorized public surface:

riffdb --config "$HOME/.config/riffdb/client.toml" \
  backup retire before-upgrade

Retirement requires the same global AdministerCapabilities authority as create and restore. The server resolves the exact succeeded create receipt and manifest; the client never submits a path. It then renames the complete four-file inventory into an operation-private maintenance directory, syncs both parents, deletes only those four known regular files, and records terminal success. Poll the returned operation ID with backup operation exactly as for create and restore.

A terminal retirement receipt is the only reason startup accepts a succeeded create receipt whose published artifact is absent. Missing artifacts without that exact receipt pair, symlinks, unknown files, manifest mismatches, and ambiguous staged state fail readiness closed. Retired names remain permanently consumed: neither create nor restore can silently reuse them.

Do not remove a backup directory or any file beneath .maintenance with filesystem tools. Routine retention should invoke backup retire after a newer backup has been fully validated. An interrupted retirement is recovered from its durable phase and exact create/manifest evidence on restart.

Restore a Backup

Restore to a nonempty target requires the exact destructive confirmation:

riffdb --config "$HOME/.config/riffdb/client.toml" \
  backup restore before-upgrade \
  --confirm-replace-current-database

The flag confirms destructive replacement only. It does not authenticate the caller, grant authority, satisfy an approval, or prove the backup is safe.

For a healthy current database, RiffDB authenticates and authorizes the caller before draining. It then validates the selected backup in private staging and freshly authenticates and authorizes the same bearer against the staged database. The current authorization result cannot satisfy staged authorization.

For an empty, unreadable, or corrupt target, the restricted recovery path validates the staged backup and authorizes only against that staged database. It does not claim a trustworthy current authorization state.

Ordinary self-contained restore verifies every immutable artifact checksum, refuses private engine repair, checks physical engine integrity, recovers the complete journal and required history, and validates bounded metadata roots, active catalog, bootstrap lifecycle and allocators before staged authorization. It then stamps and publishes the restored incarnation and performs a fresh, receipt-bound bounded startup before readiness. Success proves authorized materialization and bounded recoverability; it does not certify a complete semantic scrub of every entity, audit, outcome or other population row. First access still validates each record and fails closed on corruption. The ordinary background scrub runs after readiness and degrades Health on failure. Operators requiring complete semantic assurance can request the existing offline scrub. Supported legacy formats that need migration retain complete validation. Archive replay, follower bootstrap and promotion retain their complete proofs. On eligible native empty-target restores, temporary history reads borrow the original backup’s verified read-only sealed-extent descriptors. The backup’s complete manifest and artifact inventory remain pinned and checked; each read has an independent position and refuses changed identities, writes or links. The primary, active journal and marker are still copied and synchronized. Backups requiring an active-extent capacity conversion use complete copying and history hydration before validation. An interrupted private conversion cannot make the replacement ready or publish it; the original backup is unchanged. Complete retained-history validation and the receipt-authorized RestoreAnchor must remove every surviving source reference before temporary readers release. Original backup files remain unchanged, and publication contains exactly the three self-contained database artifacts. Publication rechecks every original backup artifact and independently reads back all three transferred artifacts. A Prepared or Published restart reacquires and fully validates the original backup; a StartupConsumed restart recovers the actual target and its later acknowledged writes. These checks do not grant readiness or replace startup validation. Complete startup and private offline validation inspect audit rows through the same bounded dispatcher: at most four workers, 64 rows and 64 MiB of copied input per batch, all on the original immutable validation snapshot. Every worker joins before the page returns; findings retain physical row order. There is no operator setting to suppress checks. Copying, hashing, physical integrity and journal recovery still scale with retained data; the sustainable TicketDesk recovery-time gate remains unqualified.

Closed backup inventory

New backups contain four fixed regular files:

database.redb
journal.riffextent
format.riffdb
manifest.riffdb

Retained journal history adds one regular file per sealed extent, named journal.sealed.<20-digit-generation>.riffextent. The manifest bounds this inventory and authenticates every sealed file in generation order. The four fixed files remain required even when there are no sealed extents.

The manifest authenticates the redb checkpoint, the complete fixed-size durability-journal extent, and the exact pre-open format marker as one unit. It also declares the inclusive binary format range allowed to restore the physical artifacts. Restore checks that range before discovering, creating, locking, staging, or replacing the target. It then checks every artifact checksum, the journal database identity and selected generation, an empty suffix at the manifest checkpoint, and the marker identity/fixture binding. Each retained sealed extent must also pass complete frame and footer validation. An oversized footer is corruption: restore rejects it before allocating the footer contents or publishing the target. The marker is published last, so startup cannot mistake a partially published restore for current data.

Unknown files, an absent current artifact, a legacy manifest without a physical range, stale or mixed identities, and checksum damage fail closed. Historical two- and three-file backups remain decodable for inspection and recovery with their source release; the current binary does not infer physical compatibility from successful decoding. Restore a legacy artifact with its source release before following docs/compatibility.md’s declared upgrade edge.

Extent creation reserves and physically zero-fills its fixed capacity before the database becomes ready. Insufficient space therefore fails during startup, backup staging, or restore staging—not after a command has been accepted.

The journal format is local recovery media. Replication and changelog records continue to derive from published durable frontiers and do not copy journal bytes.

Interruption and Recovery

Receipts live beneath:

<backup-root>/.maintenance/

This subtree is reserved and cannot be selected by a backup name. A receipt is versioned, checksummed, bounded, atomically replaced, and parent-directory synchronized. It is durable operational state, not database metadata or application history, and it is not included in backup artifacts.

The external ledger also recognizes archive-only receipt V3. It preserves the verified archive selection across retries, shares the existing maintenance ownership and inventory limit, and retains an admitted private replay stage for archive recovery. Private replay can stop at the receipt’s exact selected manifest after the archive advances; an explicitly empty selected suffix remains empty. Ordinary create/restore V1 and retire V2 receipt bytes are unchanged. Restart routes an unfinished V3 receipt through its archive-specific recovery checks. Recognizing a receipt does not validate or publish its replay stage. For an interior command stop, storage can rebuild a separate private artifact from the verified backup and checked command-prefix evidence after full original replay. It retains the original archive selection separately from its actual application/audit frontier. Ordinary source and follower opens refuse this artifact. A separate validation owner now checks its exact construction bytes, full structural/catalog evidence, reciprocal command graph and actual stopped frontier. It can release one pinned snapshot for staged authentication and current policy without local writes. Once every authorization reader closes, storage can seal the validated cut and publish it through the matching durable Offline V3 receipt. Its recorded incarnation authorizes a fresh RestoreAnchor at the actual stopped frontier; only then is the source journal created and the normal format marker restored. Interrupted private stages rebuild from the receipt’s original selection. One storage preparation owner now covers an empty suffix, the exact backup fence, earlier complete receipts and interior command cuts. It first validates the entire selected original suffix, then rebuilds an earlier stop from the same backup and selection. An explicit sequence stops at its first exact boundary; last-archived retains the selected terminal administration frontier. Both paths provide a validated immutable snapshot for staged authorization and use the same receipt-authorized publication owner. The internal driver now drains an admitted V3 receipt, resolves its existing owned archive from operator configuration, and persists its verified selection before replay, authorizes the validated immutable stage, and records the actual frontier and incarnation before publication. Success requires fresh source startup validation and complete authoritative state equality with the retained stage. Before publication is durably recorded, credential-free resumption requires exact staged/target bytes and the receipt-bound RestoreAnchor, marker and empty journal. After publication is recorded, recovery permits engine bookkeeping changes from interrupted startup, but requires the same lineage, RestoreAnchor and dual frontier, permits only empty DirtyActivation receipts from those startups, and compares every authoritative row against the retained published stage before success. It cannot rebuild an unpublished stage without credentials. The storage owner can publish a separately validated and sealed replay only against matching durable Offline V3 evidence and its recorded incarnation. It creates an empty source journal at the actual restored application/administration frontier, validates the new source completely, and then uses the existing replacement and parent-sync boundaries. Pre-receipt cleanup cannot delete an admitted V3 stage. The shared staged-authorization check uses one immutable restored-state snapshot; it can check replayed grants, expiry, approvals and revocations without local follower authority writes.

Archive restore integration status

A ready source server now accepts the distinct RestoreArchivedBackup RPC and Rust-client RestoreArchivedBackup submission. The CLI uses the same service:

riffdb --config "$HOME/.config/riffdb/client.toml" \
  storage restore before-upgrade --archive daily \
  --stop-at-sequence 42 --confirm-replace-current-database

The archive name resolves through server configuration. Omitting --stop-at-sequence selects the last archived frontier when the server first verifies the archive. An explicit sequence must be positive and fall within the verified backup and selected suffix. The operation freezes that selection before replay, and a retry retains the operation ID, archive name, stop choice and replacement confirmation. The client never falls back to an ordinary restore when the archive RPC is unavailable.

Accepted and uncertain output contains the maintenance operation ID. Poll it using backup operation; rerunning storage restore generates a new ID. Archive status includes archive_restore.archive_name, the requested stop, and, once resolved, backup_application_frontier and restored_frontier. The latter has independent application and administration positions. An absent frontier means unresolved; before_first means a known empty history. Ordinary maintenance output is unchanged.

For an interrupted archive restore admitted against a readable current database, restart accepts only the same archive restore operation ID and exact input. That retry authenticates against the current database and rechecks current restore policy before resuming. Ordinary commands, new maintenance operations, and receipt observation remain unavailable on this restricted retry host. The retained bearer must also pass staged authentication and policy before any unpublished target replacement. A different archive, stop, confirmation, or ordinary restore request cannot resume the receipt.

After complete publication, restart can recover without a bearer or archive access. It must verify the receipt-bound stage, exact restored target and full restored authority before readiness. A partially published target does not gain this credential-free authority.

When the current database is unavailable, the restricted recovery host accepts archive restore through the same distinct RPC. It verifies the full backup and selected archive prefix, reconstructs and validates the private candidate, then authenticates the bearer and checks restore policy against that restored state. A new recovery attempt persists its exact selection and admission in V3 only after those checks pass. Denial before admission leaves no receipt or target replacement. Recovery offers no ordinary application operations, backup creation, or receipt polling.

An interrupted restore with an unreadable target uses the same staged checks and its original receipt selection. It cannot choose a newer archive head or change a recorded incarnation. Complete publication evidence permits validation without a bearer or archive access; incomplete publication requires the retained backup and selected archive prefix plus fresh staged authorization. Receipt editing is unsupported.

Archive replay can advance projection controls beyond the local projection rows copied from the full backup. Before validating a replayed candidate, restore uses the same catalog-validated projection rebuild as follower startup. It reconstructs only derived rows and apply markers, keeps source controls and history unchanged, and then runs the complete structural and catalog scrub. An earlier stop repeats this process at its selected boundary; an interior command cut rebuilds its physical predecessor before reconstructing the private command prefix.

Automatic collection is enabled by the optional backup name in a configured archive entry; see Named archive directories. The collector starts from the verified backup fence or the last confirmed archive frame, validates retained history, and writes outside command acknowledgement. Collection failures stop the collector and leave primary writes available; monitor the redacted riffdb-archive-v1 diagnostics. A stopped collector needs a database restart after the cause is corrected. It cannot bridge a history gap already pruned from the source. The live sink-loss acceptance case withdraws the archive directory after confirmed collection, checks that primary writes still succeed after the collector stops, then restores only the confirmed prefix after the directory is returned. This proves write availability and prefix recovery for that failure; acknowledgement latency is measured separately.

Filesystem archive files are immutable. The collector may publish CURRENT as a checked hard link to the durable manifest, retaining the same selector bytes and atomic rename; existing independent selector files remain readable. Do not edit archive artifacts in place.

The Linux acceptance runner ./scripts/check-archive-cli builds the production CLI and runs both stop modes against a real daemon with verified TLS and automatic collection. It checks the generated operation ID, terminal JSON frontiers, restored application data and exact retry through the Rust client. These two process tests are explicitly ignored by ordinary Cargo test runs because they require the separate CLI binary; ci-all runs the dedicated runner.

The storage recovery matrix covers an archive sink refusing a frame and losing confirmation after persisting it. In both Standard and Hardened profiles, the source commits further commands before a process exit; repeated source recovery retains them, while validated archive replay includes only the durable archive prefix. The collector never reports its uncertain frame as confirmed.

Full CI at 71ae34a7 and the later sink-failure matrix checks passed; the repository bundle at release/evidence/wp-749/verification-71ae34a7/README.md records their exact scope. The later combined implementation also passed full integration CI. The same-binary archive comparison completed twenty valid cells and found measurable archive overhead: write-only median throughput fell 8.49% and request P95 rose 4.35%. The later isolated diagnostic measured acknowledgement P99 at 88.35 ms off and 114.93 ms on. Checkpoint-coupled journal durability is the named dominant mechanism, present without archive; roughly 11.7 ms remained unattributed. The checkpoint-coupled journal durability mitigation is complete, with its remaining shared tail accepted. After bounding cursor tail walks, single-pass frame digest computation, and deferring archive disk writes during background spare-extent zero-filling, the post-mitigation direct pair passes host, correctness, and acknowledgement-latency qualification (+0.00% report P50/P95, raw P50 +1.46%, raw P95 +0.94%, raw P99 +0.35%, throughput -1.17%, and zero terminal archive lag). The full replication qualification campaign remains separate, so multi-node replication paths still require their own campaign before production disaster-recovery deployment.

An SDK or direct API caller supplies the maintenance operation ID. After a lost response or process interruption, that caller retries the exact same start request with the same operation ID. It must not choose a new ID merely because the prior response was uncertain.

The CLI start commands generate an operation ID internally and cannot reissue a start request with a caller-selected ID. An accepted or uncertain CLI result includes the generated ID. Resolve it by polling:

riffdb --config "$HOME/.config/riffdb/client.toml" \
  backup operation 01900000-0000-7000-8000-000000000000

Do not rerun backup create, backup restore, or backup retire to resolve CLI uncertainty; that would generate a different operation identity. Startup validates all receipts and reconciles incomplete work only from exact receipt, manifest, inventory, and checksum evidence. It does not infer success from a partial directory or an ambiguous filesystem result.

Unknown receipt versions, invalid checksums, impossible phase transitions, or receipt/artifact mismatches fail readiness closed. Operator repair of receipt bytes is unsupported.

Restore Rewinds History

The POC preserves the DatabaseId stored in the backup. Destructive restore still rewinds sequence allocators and can reuse destroyed sequence suffixes.

A durable history_incarnation fence now makes that rewind detectable. The incarnation is a retained-metadata value bootstrapped to 1 and bumped only on destructive restore. Backups carry the value when present; responses that expose commit-sequence-derived positions include the current incarnation. Sequence-anchored requests may send optional observed_history_incarnation; when present and different from the current value, the server rejects with RDB-HISTORY-0101 before any storage work.

Detection residual risk: clients that omit observed_history_incarnation keep working unvalidated and can still silently bind stale observations. Adopt the field for any long-lived cursor, subscription, or sequence-derived assumption.

A backup may physically contain the source database’s private clean-close lifecycle record, but restore never treats that record as permission to skip validation. Destructive restore changes the history incarnation; the copied lifecycle is therefore stale and is consumed or replaced as dirty before the restored database can activate writers. A later graceful shutdown may produce a new clean record bound to the restored incarnation. Copying backup artifacts or a database file cannot forge clean eligibility.

Ordinary backup restore and staged-only backup recovery verify the copied artifacts before the first private engine open and retain its native exclusion through history hydration, complete physical verification and bounded readiness, the existing storage/catalog/retained-metadata/bootstrap/allocator joins, and fresh staged bearer authentication. Required native compatibility preparation uses the existing transformations; formats requiring migration retain complete validation afterward. Bounded readiness creates no complete semantic-scrub proof, checkpoint, registry-completion witness or clean certificate. No operational source writer or readiness capability is activated for that authorization read. Every immutable reader must close before receipt-authorized stamping. The ordinary maintenance controller retains that unique private engine through the stamp and then closes it once. Its live preparation state cannot publish. Direct closed-stage and archive workflows retain their existing close protocols.

The receipt-authorized incarnation stamp verifies the complete primary, journal and marker content seals under the same native custody and uses the existing immediate two-phase metadata transaction. Its complete retained-history evidence is bound to the same private engine and native mutation generation; stamping checks transaction-current roots before consuming that evidence. Equal-watermark retries reuse the same complete evidence while preserving every restored fencing and tombstone check. A real watermark change discards it and requires fresh physical and complete history validation. This grants no complete semantic-scrub assurance or journal replay permission.

For current ordinary receipt-authorized restore, command values that survive publication are hydrated first. Predecessor HISTORY receipts can remain in their original verified copied extents through complete validation and authorization. The canonical RestoreAnchor removes those references atomically. After checking that no command or receipt stub survives and observing engine close, the owner retires the original extents and verifies the renewed empty journal above the configured target’s extent generation. Original descriptors, content seals and file/inventory continuity remain checked until retirement. Registry completion alone cannot admit deferred history: all three native command locator tables must already exist. Images requiring table preparation retain the complete path; unsupported images refuse before authorization or publication. The owner then binds a fresh full seal of the three publication artifacts. Publication retains immutable-backup revalidation, target exclusion, full artifact checks, marker-last rename and fresh post-publication startup. On Linux, an eligible self-contained ordinary restore into an empty target on the same filesystem transfers the three original sealed files instead of copying them again. A bounded private publication intent records their original identities and seals before any move. Full readback of all three transferred files precedes final publication. Other restore states retain verified copying; an interrupted transfer cannot switch to copying or silently rebuild the target from the backup.

During eligible ordinary materialization, the retained original-backup owner performs the complete content check before first use. At native publication, that same owner revalidates all original artifacts, including sealed history. Bounded manifest-binding reads and continuity checks do not replace either full boundary check. Without the actual retained owner, the complete original-backup fallback check remains mandatory. Private artifact seals and independent post-transfer readback are unchanged.

For an eligible fresh publication in the same process, the actual verified immutable-backup and three-artifact owners now remain alive through startup consumption. Original descriptors, full source seals and completed independent transfer readback remain bound; exact file continuity and closed receipt/intent inventories are checked again before consumption. The one bounded handoff admits one consumer. A failed or abandoned retained owner cannot be replaced by a cold owner within that handoff. An actual process restart reconstructs custody and performs the complete cold checks.

Startup durably consumes that intent before opening the published native engine. If startup is interrupted after consumption, recovery uses the actual published database, performs ordinary native engine crash recovery when needed, and completes current structural, journal and catalog validation before success. It cannot use an obsolete pre-startup primary hash or a clean/checkpoint shortcut to admit changed content. Private stages and a freshly transferred target still refuse engine repair. The server keeps application admission fenced until the ordinary Succeeded receipt is durable and matching intent removal and directory synchronization finish. A crash after success can finish cleanup and normal startup, preserving subsequent writes. A failed ownership or validation check preserves the immutable backup and actual failure evidence. Existing timing gates and resource limits remain unchanged.

Private archive conversion, command-prefix reconstruction, archive validation, promoted-backup validation and archive incarnation stamping also abort engine repair requests. Private archive replay startup and projection preparation use the same refusal policy; published follower crash recovery remains available. A repaired private stage cannot gain publication authority from a subsequent successful integrity check. Before incarnation stamping, the sealed owner also rechecks its primary, journal, and format marker checksums; intended metadata writes cannot replace the seal of a changed artifact. Stamping independently refuses engine repair. Refusal leaves the immutable backup intact, and a retry rebuilds the unpublished stage from that backup.

The private stage retains the native identities of its primary, journal, and format marker through authorization, sealing, stamping, and publication. Replacing any of these files with an identical copy still refuses the operation; matching checksums alone cannot establish continuity. Archive recovery tracks its intentional removal of the empty source journal, receipt-authorized creation of the new RestoreAnchor journal, and rename of the retained marker explicitly. Ordinary and archived backup restore use their owned private stages directly for complete verification. Archive conversion, replay, projection rebuilding and an interior command-prefix reconstruction retain the original native owner. Their final immutable proof includes the complete physical, structural and catalog checks; a private cut additionally proves its exact reconstruction binding and reciprocal command graph. The server runs the ordinary startup joins over that same proof before fresh bearer authentication. All authorization readers must close before sealing. Journal absence and the original marker are sealed alongside the primary; only the receipt-authorized journal creation and marker restoration retire those bindings. The original incarnation transaction is followed by a fresh complete source proof under continuous custody before publication. A witnessed failure permanently invalidates that private candidate. The complete rehearsal remains unqualified. Its saved-population primary-loss restore gate passes; the scheduled-population 45-second release, final-source CI and both measured campaigns remain outstanding. Backup preparation precedes deliberate primary loss and remains separately visible, including its service interruption and affected offered-work windows. Recovery retains one 60-second deadline from primary loss through the first acknowledged write, exact replay and full state reconciliation. Old combined-clock failures cannot be reclassified. After the backup becomes durable, the campaign can continue reads while keeping writes fenced at its included acknowledgement prefix. It pauses and drains those readers after primary loss, charging that work to the same recovery deadline. Original offers, queue and in-flight bounds, preparation interruption and read progress remain part of the evidence; this adds no post-backup write-loss allowance.

The immutable backup artifact preserves the source bytes, including a matching certificate. Private restore staging nevertheless drives the complete structural and catalog-semantic streams through exact end before staged authentication; it never uses those bytes to skip restore validation. Ordinary backup restore consumes the complete native proof for authorization without an operational startup open; copied clean or checkpoint state cannot replace that proof. Eligibility is a property of the complete stopped database, journal, format marker, maintenance state, and retained roots, not of certificate bytes alone. A complete stopped lifecycle-unit copy preserves eligibility; a bare database-file copy does not and takes complete validation. Registry migration, an incompatible internal format, maintenance replacement, retention hold or prune, and history-incarnation rotation likewise invalidate or fail closed. A backup-bound compatible marker-only format upgrade preserves an otherwise unchanged lifecycle unit. An offline integrity scrub deliberately preserves the bytes but does not consume, refresh, or replace them.

A destructive restore removes every observation created after the restored frontier: the full backup’s included frontier for ordinary restore, or the verified selected frontier for archive-suffix restore. The removed application and administration sequence suffixes may later be reused for different records. An idempotency key that existed only in the removed suffix may be accepted again.

After restore, discard all pre-restore:

  • outcome, commit, provenance, event, outbox, projection, and audit locators beyond the restored frontier;
  • idempotency assumptions about the removed suffix;
  • pagination cursors, subscriptions, sessions, and process generations;
  • read-after-sequence expectations beyond the restored frontier; and
  • capability and authorization facts learned from the removed suffix.

Preserving DatabaseId does not make those observations valid. Archive-suffix restore supports recovery at an exact commit sequence within its verified coverage, and RedbVerifiedArchiveBackup::resolve_archive_stop_at_timestamp resolves a wall-clock target timestamp (Timestamp) against a verified baseline backup and bound RedbArchiveRepository into a deterministic ArchivePitrResolutionV1 (stop_at_sequence when an archive prefix applies, or None when the target falls on the baseline backup fence and no archived commit is <= target). Timestamp resolution refuses pre-fence targets, pre-first-commit targets, and non-monotonic commit timestamps across the target boundary fail closed (ArchivePitrResolutionError). Each confirmed archive frame also exposes an ArchiveFrameSummaryV1 (RedbArchiveRepository::frame_summaries and RedbArchiveRepository::set_confirmed_frame_observer) with its monotonic sequence and commit-timestamp range and SHA-256 digest for external object-store replication hooks. Globally stable pre-restore locators, online backup, built-in backup encryption, and a built-in remote object-store sink remain outside the POC. The archive collector currently uses an operator-configured filesystem directory.

Promotion recovery evidence

Keep the private <backup-root>/.maintenance/replication_promotion ledger with maintenance receipts and the configured database. Terminal failure retains any frozen selection and refuses ordinary startup until an exact successful retry has been completely reconciled under the current owner. Deleting this evidence is not a recovery procedure. A V2 source always receives complete validation, even after CLEAN; V1 clean-start behavior is unchanged. The experimental follower administration path describes restricted pre-cutover retries and the required fresh source fence proof.

Private restore materialization retains the original primary, journal and format-marker file identities before their first engine open. A same-byte replacement of one of those files or its parent directory refuses the private open. This identity check does not substitute for complete integrity validation.

Archive image validation preserves the immutable backup’s exact history fence. It performs complete physical, journal and structural/catalog verification without appending ordinary source compatibility receipts before selected replay. Ordinary backup restore retains its required source compatibility transformations. The final receipt-stamped source proof closes without granting a second staged authorization or source activation; fresh ordinary post-publication startup still owns its existing compatibility and readiness checks.

Offline storage compaction

After offline or online retention pruning deletes historical commit, event, and outbox rows, the freed pages remain inside the redb file for internal reuse. To compact those freed pages and lower the file’s on-disk high-water mark while riffdbd is stopped, run:

riffdb storage compact --database-path /var/lib/riffdb/database.redb

Compaction verifies the database format, journal rebase, retained V3 history, and tombstone chain before and after running redb::Database::compact(), compacts the derived sidecar when present, and reports database_id, bytes_before, bytes_after, reclaimed_bytes, compacted, retention_watermark_sequence, and history_incarnation.