Compatibility

RiffDB 0.1.0 is a pre-release POC. Compatibility is checked at explicit boundaries; it is not a blanket promise that arbitrary commits can be mixed.

Storage adapter Rust source compatibility

riffdb_storage_api::DeclaredOutcome shares its immutable canonical value across clones. value() still returns &CanonicalRecord, but it is no longer a const fn. Storage adapters calling it inside a const function must make that caller an ordinary function. Generated application interfaces and stored outcome bytes retain their existing contracts.

Per-write evidence in V25

Grammar, executable IR, and bundle V25 apply to newly compiled commands with field assignments. Every assignment carries validated evidence in its plan hash. Recompilation rejects replacement set writes justified only by unrelated reads or revision increments; migrate to an explicit checked_by business association, a caller-version fence, or an intentional overwrite. Normal application-lock renewal is required, including commands whose execution behavior is unchanged. There is no warning-only compiler opt-out.

Older deployed bundles retain their bytes, identities, validation, and execution. Their historical “guarded” classification is reported as legacy dependency evidence, not retroactively as caller-version protection. See the write cookbook for the proof limits and supported POC shapes.

Boundaries

Generated source, descriptors, schema inventories, wire vectors, contract fixtures, and interface checkpoints are compatibility evidence. The current source release also carries:

RiffQL V9, query IR V12, and query-module V12 are additive identities for the compiler-declared Limit<MAX> page type (and later additive versions through RiffQL V14, query IR V18, and query-module V18 cover relationship algebra, multi-column group by / having / order by, and event/scalar columnar projected queries). Earlier language sources and IR/module artifacts retain their existing writers and strict readers. A bounded maximum is immutable operation identity: changing it requires module/application rotation and invalidates predecessor cursors, but does not migrate entity data or change storage or public Protobuf formats.

The process-local named-query cursor hash domain excludes only compiler-proved ordinary page-cardinality values. Shared parameter hashes, query/IR/module/plan/lock identities, public opaque token bytes, protocols, storage, and durable formats are unchanged. Cursor registries already do not survive process restart, so predecessor tokens are not reinterpreted. A resumed ordinary ordered query may change its valid submitted page size while all invariant parameters, authority, snapshot, order, and provider epoch stay bound.

• release/version-topology-v1.json, the cross-domain map of independently owned reader/writer windows, writer policy, lifecycle, source assertions, and evidence described in Versioning and Retirement;

• release/durable-format-manifest-v1.json, the exact epoch/writer and complete readable/writable record, redb-layout, journal, backup, marker, and named receipt-family inventory plus the minimum/maximum supported source release;

• fixtures/compatibility/release-pairs-v1.json, the closed supported upgrade graph;

• fixtures/compatibility/physical-formats-v1.json, the redb, journal, backup, receipt, history-incarnation, retention, and recovery-validation inventory; and

• fixtures/compatibility/durable-fixture-inventory-v1.txt, whose SHA-256 is bound into the release manifest.

Run ./scripts/check-version-topology to verify all release-significant version domains, ./scripts/check-durable-format-manifest to verify the physical boundary directly, or ./scripts/check-generated to verify both with every generated artifact. A release bundle is rejected if the manifest, upgrade table, fixture digest, export posture, downgrade posture, or known limitations are absent or stale.

The current pre-alpha identity is alpha epoch 1, writer 4. Downgrade is unsupported. A format marker from another epoch or writer is not evidence that its bytes are readable merely because a decoder accepts some records.

RiffDB retains explicit backup-bound offline upgrades from writers 0, 1, 2 and 3. Writers 2 and 3 are accepted only with their exact frozen registry and compatibility inventory; an epoch or writer number alone grants no transition. Use the existing riffdb storage upgrade operation with a verified matching backup, downtime and free space of at least twice the source size. The native upgrade checks this floor before any mutation on initial admission and every retry. It counts logical lengths of the database and its retained journal, extent, marker, receipt and lifecycle working files, excluding the immutable backup and unrelated siblings. Directory discovery is limited to 16,384 entries; missing, ambiguous, nonregular or unreadable source artifacts and arithmetic overflow refuse. An insufficient-space refusal requires provisioning space and retrying with the same immutable backup. Availability is observed for the invoking principal on the target filesystem; it reserves no space, and later I/O failures still use existing fenced recovery. The backup remains immutable. Ordinary startup and older binaries refuse incompatible markers before opening storage; downgrade remains unsupported.

A predecessor physical backup remains valid as immutable upgrade evidence, but current-binary physical restore requires its declared format range to include writer 4. Upgrade the source and create a new verified backup, or use the matching predecessor binary for restoration before its supported offline upgrade. A successor restore refuses an incompatible backup before creating or changing the target. It never promotes or rewrites the frozen backup.

The writer-3 registry adds private StoredRecordRegistryV3 metadata with Pending or Completed state at the existing metadata key. Upgrade reconciles the acknowledged journal and installs Pending through a durable migration receipt. It grants no readiness shortcut. Only complete native structural and catalog-semantic validation can publish a Completed binding for the exact database, history incarnation and current registry, atomically with its validated-prefix completion receipt. Legacy V2 registry bytes remain readable and contain no independent completion witness.

Writer 4 adds logical journal frame V5 and its closed SourceReclamation kind. V2, V3 and V4 remain readable with their frozen semantics; only V5 may encode the new kind. The physical extent format, canonical V3 receipts and application interfaces are unchanged. Online Standard-profile source reclamation publishes a bounded source-prefix cut without first draining the active suffix. The cut preserves application and administration frontiers and remains subject to complete validation, all holds and an independent durable checkpoint. Replay validates its actual preimages and applies it atomically.

The native writer admits frame ownership against the compiled 64-MiB recovery ceiling before publication. Its conservative bound includes nested terminal receipt reconstruction and can refuse work below the 16-MiB wire ceiling with the existing typed limit failure. Recovery checks the same bound before decoding recorded frames or applying mutations. There is no memory-limit override.

A writer-3 source requires a newly verified matching immutable backup for its own upgrade. A completed older upgrade receipt can establish only that exact previous transition; it never substitutes for the new backup or permits a marker-only override. Upgrade preserves acknowledged commands, outcomes and provenance. Writer-3 binaries refuse the successor marker before opening the engine or replaying V5. Downgrade remains unsupported.

Capability records follow a least-successor rule: ordinary grants remain V1, migration authority uses migration-only V2, installation authority uses V3, and a compiler-bound row-policy grant uses V4. A grant with explicit symbolic application-export authority uses V5; V5 is not selected merely because a grant can read entities, events, provenance, audit, or administer capabilities. V5 preserves the exact optional V2 through V4 extensions and adds a bounded, lineage-ordered export extension. Principal-filtered export requires the matching V4 role/policy extension, while whole-application export requires explicit global/all-partition authority. The public export operation uses that authority for its shared-service, gRPC, Rust operator-client, and CLI safe points; it never derives export access from an older permission. Capability V6 adds administrative replication and primary-fencing authority (ReplicateChangelog and FenceReplicationPrimary), and Capability V7 adds compiler-owned application reimport authority (grant.reimport).

The V4 extension binds one exact application-role hash, canonical principal facts, and canonical protected entity/operation selections; omitting the extension under the V4 identity is corruption. V1 through V4 source, descriptor, hash, and wire fixtures remain frozen. V2’s source and schema hash are frozen to the original capability-V2 bytes. A short-lived pre-alpha build accidentally emitted the additive V3 installation payload under V2’s compact identity; current readers recover that exact payload without dropping authority, while every new installation grant is written as V3. No general unknown-field or best-effort durable decoder is enabled by this narrow compatibility repair.

Normal daemon startup performs the source-free format comparison before redb can open the file. An exact mismatch exits with RDB-FORMAT-0101, the retained and binary epoch/writer identities, the sole manifest-authorized action, backup and downtime requirements, and a safe next command. It never falls through to a decoder-shaped error, creates a replacement database, or enters backup recovery merely because the format is incompatible.

For a new path, RiffDB durably publishes current-format initialization intent before creating the redb container. A crash with only that marker (or an empty container) resumes initialization on the next start; it is never mistaken for a predecessor database that needs an offline upgrade.

The current durable-record registry adds the private clean-close lifecycle record through one exact predecessor-to-successor registry migration. That migration does not synthesize clean evidence: the first open performs complete validation and records dirty lifecycle state. A matching clean lifecycle record is eligible only under the exact successor registry digest. Older binaries do not recognize that registry and downgrade remains unsupported.

Clean-close compatibility is exact rather than best effort:

Only an unchanged, completely and gracefully stopped lifecycle unit can reuse a matching record. A dirty startup recovers the journal suffix and validates the bounded roots without a population walk when a validated-prefix checkpoint loads or an exact Completed registry binding is present, and otherwise performs complete validation; see Startup Integrity. The offline storage scrub is the authorized maintenance operation described there, and its receipt is the version-4 maintenance receipt.

The current registry also adds durable tag 67 StoredColumnarProjectionControlV1 and the columnar_projection_controls table. New scalar and vector controls always begin from authoritative state at a BeforeFirst fence; existing scalar manifests, name-derived directories, and tag-65 vector controls are not migrated. Tag 65 remains structurally readable under a 4,096-row startup bound but is inert and unwritable until its mandatory epoch-2 removal. Controlled V1 manifests are immutable and selected only by the exact length and checksum in tag 67.

The same tag-67 control now supports the bounded per-database transition from a selected V1 generation to an additive immutable V2 generation. V1 remains readable and writable until one complete V2 ROOT-V1 is durably selected; an incomplete candidate is never a compatibility signal. Segment V2, encoding-registry V1, Manifest V2, and generation-root V1 retain their exact registered readers, writers, fixtures, bounds, and definition bytes. Once V2 is selected, open requires the exact control-bound generation, history incarnation, frontier, root identity, partition inventory, manifests, segments, and format tuple. Missing, extra, mixed V1/V2, stale-incarnation, unknown, or corrupt selected material fails closed without V1 fallback. Downgrade remains unsupported; a future removal of V1 identities still requires its separately governed retirement package.

The current registry adds a V2 follower-registration policy codec in the existing source-hold record domain (tag 73, revision 2). Frozen V1 follower, archive and bootstrap holds retain their exact bytes. V2 carries an explicit sequence budget, registration predecessor, phase, optional sequence expiry and checked degradation observation. Decoding it does not authorize a registration or fence release. The registry digest changes: prior exact registry markers require their matching binary. Internal registration and retirement now write V2 through the audited coordinator path described below. Archive and bootstrap jobs retain V1. Automatic registry migration remains unavailable. Operator registration and retirement use the shared service, gRPC, Rust client and CLI.

The audit successor adds ServiceAuditRecordV3 (tag 22, revision 3) in a separate schema. It preserves existing target fields and adds one source-lineage-scoped follower target. V1/V2 bytes and readers remain unchanged; existing audits still write V2. The bounded V3 codec and mixed audit reader, registration/retirement and configured expiry are implemented. Primary fencing and follower promotion use dedicated administration records and external receipts. The exact registry digest changes; earlier markers still require their matching binary until an explicit migration is implemented and proven. An older binary refuses the V3 identity.

The registry also registers StoredReplicationAdministrationV1 at tag 75, revision 1, with a 2 KiB payload bound. Tag 74, revision 1 remains the existing export-page record. Checked administration values bind the exact target, generation, source predecessor, policy images and explicit principal/request/approval or original registration receipt for expiry. Malformed transitions fail closed. The mixed hold reader preserves V1 holds, V2 policies and retired tombstones distinctly. The storage administration reader also recognizes these receipts. Complete source validation checks current V2 policies and retired tombstones against their exact registration and release records from one database snapshot. While a lifecycle transaction remains in replication history, its audit bytes must match that transaction’s receipt. Retired tombstones keep their audit links after older physical receipts are reclaimed; they no longer contribute a live retention fence. Missing or substituted evidence fails closed. This validation streams audit history and retains at most the existing 4096 source holds; it is a full validation pass, not a constant-time checkpoint check. Attached followers retain replicated audit records without acquiring source-only holds. Historical source receipts remain valid for older incarnations of the same permanent database identity. Application export identifies these administration rows with the replication kind.

The internal source acknowledgement writer preserves a V2 registration’s generation, budget, expiry and original audit link. Only attached registrations may acknowledge; equal retries allocate no receipt, and awaiting or retired registrations refuse even an equal acknowledgement. Legacy registration cannot overwrite a V2 policy. An advancing acknowledgement recomputes degradation from the validated source head: catch-up clears budget degradation, while a reached configured expiry remains degraded. Neither condition releases the fence.

A pending V2 registration admits bootstrap custody at its exact registered source cut or a later cut. Attachment preserves its policy and generation while atomically replacing the bootstrap job with the attached follower fence. Missing custody, older cuts and retired identities refuse. A follower adopted from a legacy hold retains its exact older attachment retry. Artifact cleanup excludes a retired policy only after validating its registration and release evidence from the same source snapshot; a separate live bootstrap job still retains custody. Transfer composition revalidates current source custody before releasing a manifest, and already-open artifact handles check custody after page I/O before returning bytes. Retirement therefore also refuses subsequent transfers through an old handle. A held artifact does not keep the source engine open. Closing the source invalidates that handle; reopening requires a fresh custody check.

The internal lifecycle authorization preparation binds the entire request and requires current global administrative authority, as offline maintenance does. Permission to consume replication bytes alone does not authorize registration or retention-fence release. The shared-service policy port and production adapter forward the entire lifecycle request through this preparation; ports without that implementation refuse it. The coordinator rechecks current authority and fresh time inside its drained transaction, including on exact retries. Registration and retirement atomically commit the policy, administration record and V3 receipt; their service result links the exact original administration sequence. A retry with the same lineage, identity and policy returns the original registration receipt even after attachment or retirement, without restoring a retired hold. Retirement selects the original generation and removes only its matching bootstrap job; independent archive custody remains held. Legacy adoption keeps the exact existing follower fence. Conflicting policies, stale generations, reached expiry for a new registration and full hold capacity refuse without allocating a sequence. The public RegisterFollower and RetireFollower RPCs expose these transitions through current global administrative authority; automatic registry migration remains unavailable.

The production audit writer selects ServiceAuditRecordV3 only for the accepted follower target; existing targets continue to produce identical V2 bytes. Mixed V2/V3 audit groups use the existing atomic journal and direct-commit paths. Started and terminal records retain the same complete follower target; changing its database, incarnation, epoch or hold ID refuses the terminal append. This success is additionally checked against the exact registration/retirement action and target of its linked authoritative receipt, including new-request-ID replay.

The additive public lifecycle requests select the source lineage, opaque hold ID and immutable policy or exact registration generation. They return a checked receipt or closed refusal. Service operation tags 0x3a and 0x3b are appended; all prior tags and frozen audit schemas retain their identities. The operation registry allows gRPC and in-process comparison only; the CLI and Rust client use gRPC. No MCP tool or application-role grant is added. Strict older clients cannot invoke these methods; regenerate or use matching updated bindings. New success audits require the exact follower target and a control-plane link; linkless success is refused on append and reconstruction.

The coordinator also admits a closed continuation of stored registration policy, with one transition per turn. Reached budget or sequence expiry first persists a checked degradation point without advancing application or administration heads. A later turn may release an expired registration only after revalidating that point, its immutable policy and the original authorized registration receipt. The expiry record, retired hold and retention-attributed receipt commit atomically. Budget exhaustion alone never releases a hold. Crash recovery retains either the complete old or complete new state; retries do not allocate another expiry. The primary schedules this continuation on an owned observer thread, using published snapshots and the existing bounded coordinator queue. It submits at most 16 one-registration steps before a one-second scheduling delay. A policy whose budget degradation is already persisted and has no reached expiry needs no further writer turn. Clock or proven-not-committed storage outages retry after the delay; corrupt, uncertain or stopped-coordinator outcomes stop the worker. An uncertain expiry may already have committed its complete audited release; recovery validates the persisted state before further work. Shutdown cancels capacity waits, drains accepted work and joins the observer before closing the coordinator. These delays schedule work; only application sequences determine expiry.

Primary Health reports degraded replication when a live registration reaches its budget or configured expiry, including a caught-up registration at expiry. Statistics retains its existing exact sequence counters. The bounded observation uses one published source snapshot and grants no retention-release authority. The existing Health status enum and wire fields are unchanged; validators now admit degraded primary health with live registrations and zero sequence lag. Older clients that require Healthy whenever those counters are zero refuse this newly admitted Health shape; use matching updated validators. Follower health and the zero-registration primary case retain their prior rules.

The current registry registers successor command capsule V7 (tag 54, revision 6) and segment V6 (tag 55, revision 6) for exact command-prefix evidence. Earlier record bytes remain readable by their own codecs and do not acquire intermediate values. The registry digest changes: a database or backup carrying the previous exact registry marker is refused by this binary before opening or replacement. Use its matching binary; this addition declares no automatic migration edge. For received successor command groups, the follower checks evidence against the original receipt and the exact starting rows, including keys that disappear from the receipt because later commands restore their original values. A contradiction refuses the frame without advancing durable progress. Newly committed audited commands now write these successor identities and retain their intermediate entity/index state atomically. Their complete evidence copy counts toward the existing pre-sequence capacity reservation, so a command near the previous encoded limit may now be refused before receiving a sequence. Startup, recovery and received successor groups also check that retained entity images, chain heads and index epochs exactly match their owning command facts; missing, extra or substituted rows refuse even when their envelopes are valid. Pending-idempotency mutations may only delete that command’s identity. Followers also join entity, chain-head and epoch transitions to their exact logical prior rows, including earlier commands in the same group. Retained-segment decoding checks known intra-segment predecessors and their raw mutation preconditions. A wrong nested record type is corruption, never a legacy-format fallback. Secondary-index prefix puts also bind their typed key and schema to the command and require the exact owning partition/index epoch. Available prior index rows are checked against that same generation bucket before replacement or deletion. Startup and follower replay also resolve supplied index keys through the retained command contract and derive each put’s key and exact covered fields from its entity post-image. Undeclared or substituted covered fields refuse even when the receipt and prefix agree. Supplied puts must also name the catalog-derived owning partition. Followers derive the complete required index puts and deletes from each entity’s actual predecessor and post-image, refusing missing or extra mutations. Startup proves the same inventory for creates and known intra-segment predecessors; it does not invent unavailable historical entity values. Each available prior index row, including unchanged indexes, must match the key, covered fields and owning partition derived from its predecessor entity. Missing or contradictory known rows refuse before durable follower progress. Expected covers are derived one at a time. Older writer records use the existing bounded lineage proof to materialize eligible optional fields as null before deriving index contents, while transition hashes still bind the raw writer bytes. Every new prefix entity image now requires the complete field set declared by its exact writer schema, including explicit optional nulls. Catalog checks validate value types and bounds, primary-key field values, command partition route, and the retained executable plan. Extra or missing fields refuse even when no index or vector uses them; historical predecessor materialization remains separate. Vector prefix rows now require canonical keys and typed values. Written evidence binds the entity image, version, vector presence, command plan, sequence, provenance and partition. Every evidence mutation requires its reciprocal index mutation and partition/lineage observations; missing, extra or contradictory links refuse. Observation revisions must match the command, and persisted partition counts cannot be empty. Catalog checks reject undeclared vector fields and validate rewritten embedding metadata against the production declaration. Known entity predecessors determine required source/embedding writes and deletes, including refusal when all vector work is omitted. Available prior evidence and its reciprocal index must agree; unchanged fields preserve earlier evidence, and source-only writes preserve embedding stamps exactly. When evidence and observation predecessors are available, validation folds every affected entity into its shared partition counter and checks exact total, stale and per-model counts. It also derives the lineage health summary with the compiler-owned thresholds. Resealed counter and health forgeries refuse before follower progress. Unknown retained predecessors remain unproven. The offline owner can now rebuild an interior command boundary from a fully replayed selection and checked prefix evidence. This private artifact has no replication identity or publication capability, and ordinary database opens refuse it. A distinct private validator runs the full structural/catalog scan and checks reciprocal locators, segment links and the actual frontier. Its authorization snapshot is bound to those validated bytes. After all authorization readers close, a matching durable Offline V3 receipt can authorize a new incarnation and RestoreAnchor at the stopped frontier. The original selection and backup manifest remain unchanged. Ordinary source validation must pass before the existing replacement ceremony publishes that newly anchored database. A shared preparation owner now also handles earlier complete receipt boundaries and the verified backup fence without partial reconstruction. It validates the full selected suffix before preparing any earlier stop; explicit sequence stops exclude subsequent administration-only records, while last-archived includes the selected terminal frontier. The maintenance driver exposes these paths through RestoreArchivedBackup, the Rust operator client and riffdb storage restore --archive; see archive restore for selection, retry and staged-authorization rules. Full CI and write-path archive-presence qualification pass.

Inspect the format while the server is stopped:

riffdb storage preflight --database-path "$HOME/.local/share/riffdb/riffdb.redb"

The supported alpha-1 predecessor edges are offline, backup-required, one-way transitions from writers 0, 1, 2 or 3 to writer 4:

riffdb storage upgrade \
  --database-path "$HOME/.local/share/riffdb/riffdb.redb" \
  --backup "$HOME/.local/share/riffdb/backups/pre-format-upgrade"

The backup must have been produced and verified by the source release after writes stopped. The upgrade binds its exact manifest and database identity in a checksummed sibling receipt, resumes safely across accepted/migrated/marker crash boundaries, and reports success only after the current marker is durable and revalidated. There is deliberately no --force, --ignore, --reset, or downgrade option.

The production storage engine in riffdbd is redb. Any Fjall comparison lives in an isolated nested workspace and cannot be substituted into the server.

Upgrade Rules

• Start a database only with a binary that recognizes its storage and durable format versions.
• Never use a force, ignore, reset, or best-effort decoder option to cross a durable-format mismatch. RiffDB defines no such compatible action.
• Never edit stored envelopes, maintenance receipts, manifests, generated descriptors, or contract IR by hand.
• Do not deploy two independently changed public/durable schemas under the same version.
• A downgrade is unsupported.
• Copying a database file does not grant clean-start eligibility in another configured database. Clean evidence remains bound to the stored database identity, history incarnation, registry, frontiers, journal, and bounded roots; any mismatch selects complete validation.
• A backup should be restored with the release family that created and verifies its declared restore range before any later migration is attempted. A legacy backup without a physical compatibility range is inspected but not restored by the current binary; restore it with its source release, validate it, take the prescribed pre-upgrade backup, then run the exact offline transition.
• Digest-key rotation is compatible only while every key needed to read retained records remains present with its original ID and material.
• A migration targets one exact parent bundle and one exact canonical successor. It is never inferred from version numbers or chained through an intermediate contract.
• Application Source V3 through V7 and Lock V4 through V8 are additive formats. Earlier source and lock decoders remain strict compatibility boundaries.

Restore preserves the backed-up DatabaseId but does not preserve observations from the destroyed suffix. It is therefore not a mechanism for maintaining globally stable post-backup locators. The durable history_incarnation fence makes restore rewinds detectable for clients that send optional observed_history_incarnation; non-participating clients remain unvalidated.

Primary fencing and promotion

Fresh sources use AuthoritativeStateCatalogV2 (catalog tag 69/revision 2) with explicit Active primary admission. V1 catalog bytes retain their meaning; there is no inferred admission state or automatic upgrade. Primary admission, immutable fence administration and promotion administration use accepted tags 76/1, 77/1 and 78/1. Older registry markers require their matching binary. Existing changelog V3, backup manifest and command-prefix formats are unchanged.

V2 sources perform complete ordinary startup validation even after CLEAN. The V1 certificate does not bind primary admission and cannot skip that work. V1 bytes and bounded startup behavior remain unchanged.

The existing administrative replication stream uses audit operation 0x3d: durable establishment/initial denial audit, followed by redacted telemetry on later denial. Exact fence evidence is a bounded terminal stream item. Promotion uses accepted operation 0x3e and external attempt receipts while attached or offline; successful cutover joins normal service audit atomically. No MCP fencing or promotion tool is exposed. The primary-fenced error is RDB-REP-0102 with contact_operator recovery; older clients refuse unknown variants. See follower administration for the experimental path and remaining recovery limits.

Writer 2 introduced logical journal frame version 4 for bounded retention pruning, and current writer 4 adds frame version 5 (SourceReclamation). Frozen version 2, 3, and 4 frames remain readable; the physical extent format and existing record encodings are unchanged. A binary for an earlier writer identity refuses a newer unit instead of attempting a downgrade.

An existing predecessor unit requires the supported offline format upgrade, with an exact immutable backup created by its matching binary before upgrading. The upgrade validates the predecessor’s registry and frozen compatibility corpus, binds its durable receipt to that backup, and replaces the marker atomically. It preserves application and administration frontiers. It has no online or backup-free upgrade mode. The journaled pruning path, crash recovery, and interference qualification pass; physical and all-consumer steady-state limits are recorded in Known limitations.