Startup Integrity and Clean Restarts

RiffDB chooses its startup-validation mode automatically. There is no flag, configuration setting, API, or maintenance shortcut that can force the faster mode or suppress complete validation.

An ordinary primary open synchronizes its parent directory after the engine file exists and before returning a usable store. A failed directory sync refuses the open. Reopening repeats this check, including when a previous attempt left a valid file and format marker behind; journal creation is not required to make the primary filename durable.

The changelog V3 substrate adds an internal first-activation transition. An inactive database must complete structural and catalog validation before one hardened transaction publishes the V3 registry and all required roots together. The exclusive startup lease remains held through that proof join and the DIRTY transition. A database already claiming the current V3 registry refuses missing roots; deleting them cannot request legacy startup or a fresh activation. No operator switch or application API controls this transition.

After a graceful shutdown, RiffDB writes a private clean-close lifecycle record as the final authoritative database mutation. The record binds the database identity and history incarnation, durable-format registry, exact application and administration frontiers, selected journal header, fixed metadata roots, active contract catalog, active query modules, and initial authority roots. The next process may use bounded startup validation only when all of those facts still match and the storage engine reports no repair. Before any writer, worker, maintenance task, or public service becomes active, RiffDB durably replaces that clean state with its next dirty generation. A crash after readiness therefore cannot leave reusable clean evidence.

Missing, dirty, malformed, stale, copied, or contradictory lifecycle evidence does not select a population walk. Such a startup opens the storage engine, checks the durable-format identity, recovers the complete journal suffix, validates the same bounded roots the clean path verifies, admits the current registry through a retained checkpoint or exact Completed binding, consumes the dirty generation, and only then publishes readiness. It does not enumerate entity, index, history, event, audit, provenance, projection, outbox, or idempotency rows, and repeating it creates no command, event, outcome, provenance, audit, projection, outbox, or allocator advance. The state that recovery itself owns is capped at a compiled 64 MiB; exceeding the cap is a typed internal-defect refusal, never a partial start.

After an abrupt stop the storage engine also repairs its own allocator state on the first open, reading the whole database file. That repair is storage-engine recovery, not a RiffDB population walk; startup reports it as an engine repair, and its duration grows with file size. On the reference benchmark host a kill restart reached readiness in about 0.5 seconds at 65,536 rows and about 53 seconds at the production benchmark population (about 1.1 million rows), almost all of it engine repair; the RiffDB startup stages together took well under a second. RiffDB leaves the engine’s quick-repair mode off because it would change commit behavior. These figures are host-specific observations, not guarantees.

The first write after any restart also rebuilds the process’s in-memory command indexes once, because steady-state writes depend on them. At the production benchmark population that rebuild holds writes for tens of seconds; reads and readiness are unaffected, and later writes run at normal speed.

Complete startup validation is still used when those bounded-recovery preconditions do not hold: a database with neither a loadable validated-prefix checkpoint nor an exact Completed registry binding, an inactive or incompatible registry, or a follower start without a valid clean-close certificate. Pending registry metadata provides no shortcut. A mismatch in a bound root fails closed. None of these cases asks the operator to classify the preceding shutdown.

Complete native structural and catalog-semantic validation publishes the private Completed registry binding through the sole ordered writer, atomically with its prefix-completion receipt. It binds the exact database, history incarnation and current registry. Partial validation, cancellation, migration, a bounded restart and background refresh cannot create that witness. An uncertain commit fences writes until normal recovery resolves it.

Restore advances the history incarnation. Copied completion and checkpoint bindings cannot certify complete validation of that successor. An ordinary self-contained restore instead binds fresh bounded readiness to its durable restore receipt, verified backup, database identity, new incarnation and exact application frontier. It checks private physical integrity and recovers the complete journal before publication, then validates bounded roots and consumes the dirty generation on reopen. It creates no complete scrub proof, checkpoint, completion witness or clean certificate. Its first-access checks and background scrub retain the assurance boundary below. Legacy migration, archive restore, follower bootstrap and promotion retain their required complete validation. The copied Completed binding still names the predecessor incarnation. A later dirty restart without matching clean evidence or another eligible checkpoint therefore uses complete validation under the existing startup rules. Bounded restore does not silently renew that binding; a successful restore alone must not be interpreted as a promise of bounded startup after every later crash. When complete validation runs, it can replace the exact predecessor checkpoint bytes with a new proof chain and publish its own completion binding in the same durable receipt. An ordinary checkpoint refresh cannot perform that renewal.

Online retention still deletes the optional validated-prefix checkpoint in its first prune step. Ordinary writes and prune preserve the Completed binding, so its absence does not require another population walk. This binding proves completed format/catalog validation; it grants no source-reclamation authority or evidence that every current row is intact. First-access checks and the complete rate-bounded background scrub remain required.

With sequence retention enabled, maintenance can independently rebuild the deleted optional checkpoint from complete pinned structural, catalog, and retained-history validation. Publication refuses a changed history binding and yields to busy writers or unfinished checkpoints. It does not reuse the invalidated proof or force a journal drain. See retention configuration for the maintenance bounds. The sustainable TicketDesk milestone separately measures the first successful write after restart.

Assurance boundary

A clean or bounded dirty restart is a continuity proof, not a full database scrub. It verifies the bounded roots needed to establish operational readiness without walking every live or retained population row. Rows skipped by startup retain their local envelope, checksum, canonical-key, schema, size, and required relationship checks before they can affect authorization, execution, output, delivery, projection, or mutation. Latent corruption unrelated to the bounded roots can therefore be reported when the affected row is first accessed.

Those checks validate RiffDB records and their relationships. Ordinary redb reads do not verify every traversed engine page checksum. Bounded readiness and the background semantic scrub therefore do not certify the entire physical engine tree; explicit physical verification belongs to the offline scrub.

Databases created by current binaries carry the V2 authoritative state catalog. A V2 database never takes the clean-certificate fast path; each of its restarts that meets the preconditions above uses the bounded recovery path, so a graceful restart and a crash restart pay the same bounded cost. Before readiness, RiffDB point-validates the database’s source admission and, when it is fenced, the referenced fence record and its exact receipt; a missing or contradictory record refuses startup. The exhaustive search for a fence row written outside any transaction belongs to the scrubs described below. V1-catalog databases retain the bounded clean-certificate path exactly as documented above.

Background and offline scrubs

Because no restart walks the population, exhaustive corruption discovery happens in two places, both read-only: a background scrub after every start, and an offline scrub an operator requests. Neither repairs, deletes, or hides a row, and neither is an application, agent, or MCP operation.

After readiness, one server-owned background scrub reads the complete structural and retained-history evidence through a least-authority reader at a fixed rows-per-step rate. Authenticated health reports it as the storage_scrub member, which holds exactly one state: pending, running, complete, failed, or cancelled. Every process start begins at pending, and shutdown cancels a running scrub. A failure records one incident ID in the member and sets the authoritative_storage component to degraded, so aggregate health reports degraded while service continues and the damaged rows keep failing closed when accessed. On a V2 database a retained-history failure, such as a stray fence row, instead withdraws authoritative readiness. Health carries no diagnostic text, row, key, count, path, or hash. The scrub has no rate, schedule, scope, or suppression setting. It is one pass over the whole database that keeps one core busy until it completes, so throughput just after a start is lower until storage_scrub reports complete; benchmark windows start only after that.

riffdb storage scrub requests the complete exact-end validation on demand. It uses the same global AdministerCapabilities authorization and offline maintenance lifecycle as backup and restore: the server stops admission, drains accepted work, closes the database, streams the same complete structural and catalog-semantic validator that staged restore and format upgrade use, and then reopens through ordinary startup. The command takes no arguments. It generates the operation ID and prints it; observe the operation with riffdb backup operation <maintenance-operation-id>. Its phases are accepted, draining, offline, validating, and then succeeded or failed_closed. A scrub observation has kind storage_scrub and no backup name.

Before the complete record walk, offline scrub captures one private verification copy of the primary under exclusive original-file ownership and runs redb’s complete engine-integrity checker on it. The record walk uses that same copy and reads the original native journals and retained extents. Neither pass replays a journal or repairs the original. A repair callback, an unsuccessful integrity result, source substitution, cancellation or unavailable capacity refuses success. A journal suffix needing recovery must first pass ordinary startup recovery; scrub cannot certify a partially materialized frontier.

Reserve additional local disk space for one fully allocated copy of the primary file, plus filesystem overhead. The copy cannot grow beyond the captured primary size; inability to allocate or verify within that capacity fails closed. The verifier uses a private directory beside the primary, releases the copy after success or ordinary failure, and reclaims its single recognized scratch artifact after process exit. Unexpected files, symlinks, an original-file alias or uncertain custody refuse cleanup and verification. No repaired copy is published.

A validation_failed scrub means the pass refused the database: incomplete, contradictory, corrupt, or over-limit evidence all fail closed. The server process then exits. Its next start opens the database through ordinary startup with the terminal receipt retained, and the background scrub reports the same defect. Keep the incident, and restore from a verified backup when directed. A succeeded scrub creates no clean certificate, forces no readiness, and does not suppress the next background scrub.

Only one maintenance operation of any kind may be incomplete at a time. If the process stops during a scrub, the next start reruns the pass from its beginning before ordinary startup; there is no resume point. Retrying the same operation ID returns its stored observation, and reusing that ID for a backup, restore, or retirement is refused as idempotency-key reuse. The scrub’s receipt is a bounded, checksummed version-4 maintenance receipt beside the other receipts in the maintenance directory. Do not edit or delete lifecycle metadata or receipts to simulate a scrub.

Graceful shutdown

Configured columnar sources remain cold while the authoritative graph reaches core readiness. Startup resolves their bounded checked registrations but does not open or decode a generation artifact, build a snapshot, catch up, publish, or checkpoint. Consequently a no-demand graceful close also performs no columnar population work. The projection health component is degraded while any registered columnar source remains cold or is activating; this derived health signal does not withdraw authoritative storage, catalog, or command readiness.

The first semantic projected read requests activation from the single server-owned columnar worker and immediately receives the existing typed Building result with no rows. Concurrent first reads coalesce onto that same activation, and cancelling a requesting read does not cancel the shared worker operation. There is no prewarm, eager-mode, activation, artifact-selection, or validation-bypass setting in the CLI, configuration, SDKs, MCP, or transports.

Only the current selected generation becomes queryable after its complete bounded validation. A corrupt, mismatched, stale, excessive, incompatible, or unavailable selection fails closed and serves no rows; RiffDB never substitutes another generation, layout, source, frontier, or authoritative scan. Repeated requests do not create an artifact reopen loop. Correct the underlying storage or selection problem and restart the process; a separately published vector generation may also be admitted through its normal durable lifecycle. Do not weaken freshness or emulate the query against authoritative rows.

Shutdown first stops admission and drains writers, workers, and the journal. If the columnar worker is catching up, it observes the internal stop only after finishing its current authoritative page of at most 64 commit records. It may discard later unpublished in-memory columnar work at that boundary. Any safe prefix published before the stop remains valid, but the stop itself does not publish, notify, checkpoint, or advance the durable projection frontier. There is no application, agent, configuration, or operator option for choosing this behavior or changing its page size.

After restart, columnar state resumes from its last durable projection frontier and replays the authoritative log idempotently. Projected reads can temporarily return their existing typed building or lagging outcome until replay catches up; this does not mean authoritative entity or commit data was lost. An operator should retain the authoritative history required by the projection and allow normal catch-up rather than editing projection files or weakening a query’s declared freshness policy.

After the durable journal-suffix barrier, RiffDB inspects the optional validated-prefix checkpoint through fixed metadata and table cardinalities. It never rebuilds, repairs, deletes, or rewrites that proof during shutdown. An exact-current checkpoint and every companion proof row remain byte-identical; an absent checkpoint remains absent, and stale or ineligible bytes remain unchanged. Those states do not prevent a clean close because the lifecycle certificate is independent evidence.

Shutdown finishes journal sealing already in progress and validates the empty active journal, but does not preallocate a new spare extent for future writes. Unused journal scratch is removed before the clean lifecycle record is written.

RiffDB then rereads the bounded lifecycle roots and writes the clean lifecycle record with immediate durability as the final database mutation. Failure before that commit leaves DIRTY; commit uncertainty is resolved on reopen as either the complete prior DIRTY state or the complete CLEAN successor. A later dirty open still verifies the retained prefix and exact suffix or performs complete validation. Shutdown never moves that population work into the close path.

The internal shutdown receipt reports only a closed checkpoint disposition, a closed lifecycle outcome, and three saturating stage durations. It contains no database identity, path, frontier, generation, hash, key, value, row count, or engine diagnostic.

Startup and shutdown durations are observations, not correctness deadlines or comparative database benchmarks. Their cost depends on durability mode, storage engine recovery, dataset shape, filesystem, and whether the preceding close produced eligible clean evidence. The emitted lifecycle observations use only closed stage names, bounded durations, success/failure state, and the automatically selected startup mode. They contain no database path, identifier, frontier, table population, hash, key, or application value, and none is an accepted configuration or command input.

See also Compatibility, Backup and Restore, and Troubleshooting.

Command payload cache

Command lookup keeps complete exact locator metadata for retained history, while its disposable decoded payload cache is capped at 64 entries and a conservative 64 MiB ownership charge. This bounds payload retention, not total database memory: exact locator, manifest and event metadata still grow with retained identities. Startup rebuild streams one bounded segment at a time. The payload cache fills lazily from validated reads, so publication does not encode or decode commands again to populate it.

Eviction can add an exact storage read and decode. Both cold and cached reads validate against the caller’s captured storage view; missing or corrupt evidence is an error, never evidence that a committed command is absent. Follower replacement retains the prior manifest and event membership needed to remove old indexes even after payload eviction. These limits are internal and do not add operator tuning or weaken recovery checks.