Repository navigation
Description
Problem
Object-store WAL writes can put remote request latency and small-object overhead on the ingestion ACK path. A memory-only WAL removes that cost but cannot preserve acknowledged writes across process or node loss. We want an optional, high-throughput durable ingestion path implemented in Rust within lance-context, with minimal changes to Lance OSS.
This issue tracks incremental implementation and validation. Explicit Rust service paths now exist for durable append and mutable turn ingestion. They remain opt-in; workload-specific production readiness and large-scale qualification are still required.
Architecture and current visibility contract
The mutable-turn service path now separates immutable bodies from current narrow
annotations. The diagram shows that path for one shard; the lower-level immutable
append API has a different ACK/visibility contract, described below.
flowchart LR
I["Ingress router: forward Arrow bytes"] --> O["Shard owner: ordered apply and point reads"]
O -->|"Submit original request identity and payload"| L
subgraph G["One Raft group: three durable replicas"]
L["Leader + persistent WAL"] --> F1["Follower + persistent WAL"]
L --> F2["Follower + persistent WAL"]
end
L -->|"Quorum-committed records"| O
O --> S["Bounded active / frozen staging"]
S -->|"Frozen batch"| P["Fenced Lance publisher"]
P --> B["Blob: body and narrow-annotation snapshots"]
B -->|"Compatible selected base"| O
O -->|"Success after durable quorum and applied visibility"| I
- Explicit shard placement. The current mutable-turn path hashes the turn ID
within its tenant/table scope to a configured bucket and owner. Forwarding
ingresses share the validated route map and do not keep independent staging
views. Each physical annotation shard has its own body/narrow table pair and
publication/recovery progress. The immutable append contract retains its
session-key placement. Automatic session placement, shard migration and
rebalancing are not supplied by these explicit shard APIs. - Durable and visible mutation ACKs. Three replicas persist the log on
independent durable storage. A successful turn mutation requires majority
persistence and ordered application to the owner's queryable state. A raw WAL
append receipt alone is not that success. Raft supplies election, ordering and
committed-prefix safety; it does not publish Lance manifests. The storage
fault model tolerates one replica's loss, not loss of a durable quorum. - Field updates and immediate point reads. Patches preserve unspecified
fields; status/score have explicit keep/set/clear operations, and labels have
set addition/removal. Applications merge against the ordered current value.
Point reads resolve active, frozen, then the compatible Lance base, projecting
narrow columns only and never scanning the WAL suffix. Owner fencing and
recovery barriers protect cross-ingress reads. Materialized status filters
currently have a different freshness contract: they do not include active or
frozen updates. CAS and cross-shard transactions are not implemented here. - Batch publication without losing later updates. Freeze one bounded
generation at a source cut while later updates enter a new active generation.
Existing annotation rows use native narrow-column rewrites; newly ingested
bodies are appended separately and bound to the selected annotation snapshot.
Confirm the exact fenced publication before installing its base and retiring
only its frozen generation. Never delete active records by turn ID when an
older batch finishes. Compaction is separate bounded maintenance, not a
per-update rewrite of the immutable body. - Recovery, exact retry and reclamation. Recovery uses the selected Lance
state, persisted request outcomes and the required WAL suffix. Retained exact
producer/sequence retries return their original result; an unknown outcome
requires the same identity and bytes, not a newly identified mutation. A
producer has at most one unresolved request, and the bounded latest-result
contract does not retain arbitrary historical retries. Unknown publication
results are reconciled against the original publication. Source GC also
respects readers, recovery and replica/snapshot dependencies; a materialized
cut alone does not permit deleting every preceding log record. - Bounded resources and measured copying. Staging, caches, payloads, native
workspaces, disk admission and replication queues have explicit budgets;
pressure must leave publication and reclamation able to make progress.
Replicas keep Arrow payloads opaque and ingress forwarding shares byte
buffers. The current owner still validates by decoding on preparation and
decodes again during ordered apply; this is not end-to-end zero-copy.
Framing, integrity checks, native storage and kernel I/O also need measurement.
The existing immutable append API remains separate: its durable WAL ACK can
precede Lance visibility. Its readers and publication checkpoints follow that
binding's contract. Neither this issue nor an annotation-owner configuration
silently changes an existing immutable binding's semantics.
The implementation uses Rust raft-rs and raft-engine with Lance's existing
publication APIs. Useful references remain Flink's checkpoint/replay and bounded
flow control, and Apache Fluss's bucket ownership, batched replication and
asynchronous lake publication. Fluss uses leader/follower replication with ISR;
this design does not imply that Fluss uses Raft. Independent shard publication
does not create a global WAL position or a cross-shard atomic snapshot.
Incremental delivery
Use small PRs with Refs #378 in their descriptions. Keep this tracking issue open across individual implementation PRs; merge each only after its checks pass.
-
Lance 13 dependency/API upgrade: chore: upgrade Lance dependencies to 13.0.0 #374.
-
Rust Arrow IPC allocation/buffer-sharing and durable disk experiment tools: perf: add Rust WAL codec and durable disk experiments #375.
-
Stable session placement and scoped receipts: feat: add stable WAL session placement and receipt scopes #376.
-
Shared node-wide proposal admission: feat: add shared WAL proposal admission budget #377.
-
Configurable maintenance headroom and pressure regressions: feat: reserve configurable WAL maintenance headroom #379.
-
Validated Arrow IPC decoding with shared payload ownership: feat: share validated Arrow IPC decoding for WAL materialization #380.
-
Shared opaque record envelopes and bounded payload validation: feat: add shared WAL record envelopes and payload validation #381.
-
Move owned HTTP rollout payloads into core records without a conversion copy: perf: move owned rollout upload buffers into core records #382.
-
Honor registered rollout dataset locations on cold open, metadata and deletion: fix: honor registered rollout dataset locations #383.
-
Immutable ingestion contracts shared by ingress and materializers: feat: define immutable WAL ingestion contracts #384.
-
Prevent concurrent dataset creation from overwriting committed data: fix: prevent racing dataset creation from overwriting data #385.
-
Shared Arrow schema fingerprints for ingress and materializers: feat: share canonical Arrow ingestion schema fingerprints #386.
-
Guarded Rust experiment process runner and optional-feature CI: feat: add guarded Rust experiment process runner #388.
-
Read-only Rust process and cgroup resource observation: feat: add read-only Rust experiment resource sampling #389.
-
Optional process-owned TCP socket observation, with descriptor ownership, namespace and connection-state tests: feat(wal): observe process-owned TCP sockets #438.
-
Bounded vectored frame writes over shared payloads: feat: add bounded vectored WAL frame writes #393.
-
Measure WAL component disk footprint from actual files: fix: measure WAL component disk footprint from files #397.
-
Isolate physical log read caches by engine lifetime, including rewrite reads: fix(wal): isolate raft-engine entry caches by engine #398.
-
Shared durable Raft log storage, sync failure fencing and recovery contracts: feat(wal): add shared durable Raft log storage #399.
-
Replicated append, durable quorum ACK, retry deduplication and leader failover: feat(wal): add durable quorum append and recovery #400.
-
Local durable client journal, shared group commit and recovery tests: feat(wal): add bounded durable client state with group commit #401.
-
Reclaim obsolete client journal rewrite bytes before physical admission fences, with synchronized replacement, failure fencing and SIGKILL recovery: fix: reclaim client journal rewrite bytes before fencing #430.
-
Durable local Raft checkpoint storage boundary: feat(wal): persist local Raft checkpoint boundaries #402 (application snapshots and remote transfer remain separate work).
-
Atomic received Raft snapshot storage: feat(wal): persist received Raft snapshots atomically #403.
-
Application checkpoint files, bounded transfer and retained-replica recovery: feat(wal): recover replicated state through application snapshots #404 (including interrupted transfer and full process restart tests).
-
Durable canonical table bindings and publisher ownership epochs: feat(wal): persist publisher bindings and ownership epochs #405 (external Lance commit fencing remains separate work).
-
Bounded quorum queries for publisher bindings, accepted pages and producer tails: feat(wal): add bounded quorum application queries #406.
-
Immutable publication plans, fixed transaction targets and independent source-group selections retained across takeover and snapshots: feat(wal): persist immutable publication plans #407 (source pins and external commits remain separate work).
-
Catalog quorum attestations and durable source-quorum pins across takeover, snapshot transfer and full process restart: feat(wal): attest catalog selections and retain source pins #409 (source release and external publication remain separate work).
-
Immutable prepared Lance fragment descriptors, exact retries and real-file recovery across publisher takeover and full process restart: feat(wal): persist immutable prepared Lance descriptors #410 (manifest authority and source retirement remain separate work).
-
Select immutable staged Lance manifests through the catalog quorum, with exact head reservations and recovery after lost replies, owner takeover and full process restart: feat(wal): select immutable manifests through the catalog quorum #412 (outcome advancement and source retirement remain separate work).
-
Persist durable publication outcomes from the exact catalog-selected manifest, recovering identical confirmations across takeover and full process restart: feat(wal): persist publication outcomes from selected manifests #414 (source closure and retirement remain separate work).
-
Close source pins with durable publication fences, reject delayed repins and preserve exact closure retries across snapshots and restart: feat(wal): close source pins with durable publication fences #415 (payload retirement and physical reclamation remain separate work).
-
Retire closed source payloads from new checkpoints while retaining original retry receipts and retirement boundaries across restart: feat(wal): retire closed source payloads while retaining retry receipts #416 (physical reclamation and catalog advancement remain separate work).
-
Use optimized SHA-256 for opaque WAL payload validation while preserving persisted digests and record compatibility: perf(wal): use optimized SHA-256 for opaque payload validation #417.
-
Opt-in Lance object-store I/O metrics through the existing recorder, with core builds supported when metrics are disabled: feat(metrics): expose opt-in Lance object-store diagnostics #421.
-
Source-group closure attestations after a quorum read, with separate signing keys and checkpoint/restart recovery: feat(wal): attest committed source closures #423 (catalog advancement and physical reclamation remain separate work).
-
Advance catalog source cursors only after every selected source attests durable closure, preserving exact retries and managed Lance recovery: feat(wal): advance catalog cursors after source closure #424 (physical reclamation remains separate work).
-
Bounded collection of obsolete application snapshot files, preserving installed recovery images and captured readers across shared groups: feat(wal): collect obsolete application snapshot files #426 (native log reclamation and maintenance scheduling remain separate work).
-
Explicit physical log segment reclamation with node-wide sync barriers and failure fencing: feat(wal): reclaim obsolete physical log segments #427 (maintenance scheduling and replica replacement remain separate work).
-
Bounded managed-manifest history retirement with protected publication dependencies and explicit retired-retry boundaries: feat(wal): retire bounded manifest history #428 (metadata only; object deletion and production reader integration remain separate work).
-
Online replica replacement, durable-volume-loss recovery and operational recovery integration.
-
Lance materialization, fenced publication and durable source checkpoints: feat(ingest): publish session-sharded WAL records into Lance #450, building on the catalog/source protocols above. Sustained capacity remains under qualification.
-
Real Lance transaction-proof cleanup, protected version floors and delayed fixed-target commit regressions: test(core): cover publication recovery under version cleanup #408 (coordinated GC remains separate work).
-
Safe logical retirement and physical reclamation under concurrent activity.
-
Bounded durable replica workers with shared node credits, cancellation ownership, responsive cached status and joined shutdown: feat(wal): run bounded durable replica workers off async executors #441 (network transport, snapshot transfer and maintenance integration remain separate work).
-
Producer batch sizing and end-to-end throughput/latency qualification with unchanged correctness gates.
-
Bounded reusable producer slots with durable generation fencing, exact-tail close, and reuse only after signed source retirement: feat(wal): bound producer state with reusable fenced slots #434 (optional for fresh groups; administrative authorization, SDK integration and sustained churn qualification remain separate work).
-
Replica-pressure coordination, producer-state lifecycle and bounded steady-state operation.
-
Reusable Lance manifest reader/writer adapter over catalog quorum reads and fixed-target durable reservations: feat(wal): publish Lance manifests through a reusable catalog adapter #439 (production transport and writer migration remain separate work).
-
Independent managed Lance readers with scoped catalog resolution and rejected write attempts: feat(wal): open managed Lance readers without a publication plan #440
-
Production integration: visibility API, ownership/catalog integration, authentication and migration/rollback documentation.
-
Release engine directory locks explicitly at final owner shutdown, with duplicated-descriptor and pre-exec inheritance regressions: fix(wal): release directory locks when the engine closes #465.
-
Distribute one pinned table publication across bounded authenticated Rust preparer services, keeping complete aggregation and manifest authority with one fenced publisher: feat(ingest): distribute single-table file preparation #466. Independent process recovery has passed; Kubernetes scaling and sustained throughput remain under qualification.
Validation before enabling the path
- Verify acknowledged records after leader crashes, follower lag, snapshot transfer interruption and full process restarts using real Rust processes and durable storage.
- Exercise ambiguous replies, stale publishers, concurrent manifest commits and retries; verify complete Lance payloads and source positions after recovery.
- Run sustained ingestion through publication and repeated reclamation, including controlled publication pauses. Report ACK latency, published throughput, backlog, RSS, allocated disk space and physical write amplification separately.
- Compare batching, serialization/deserialization, shared-buffer ownership, bounded parallel reads and replication-group placement using reproducible workloads. Short component benchmarks are not evidence of sustained end-to-end throughput.
- Publish only synthetic, portable workloads and sanitized results. Do not include private deployment identifiers, storage locations, identity configuration, business workloads or raw operational artifacts.
Related: #333 tracks maintenance scheduling in the existing system; this issue focuses on the optional durable ingestion path and its correctness/performance contract.
References
-
Flink network stack and credit-based flow control (historical design explanation).
-
Fluss LogTablet replication and log lifecycle and Lance lake connector documentation, pinned to the researched revision.
-
Authenticated bounded TLS peer transport shared across Raft groups, with stalled-follower isolation and real three-process crash/recovery qualification: feat(wal): connect durable replica workers over authenticated TLS #442 (snapshot transfer, disk admission and service integration remain follow-up work).
-
Durable runtime checkpoint/snapshot installation and physical-maintenance transitions, with captured image ownership and durable completion tests: feat(wal): expose durable snapshot and maintenance transitions #443 (network image transfer and node-wide disk admission remain follow-up work).
-
Reserved snapshot completion lane with cancellation and saturated-control regression coverage: fix(wal): retain snapshot feedback capacity through cancellation #444.
-
Bounded authenticated TLS snapshot transfer with interrupted-receive and full-restart process coverage: feat(wal): transfer snapshots over bounded authenticated TLS streams #446.
-
Shared node disk admission for logs, producer state, snapshots and reclamation, with restart coverage: feat(wal): share node disk admission across logs and snapshots #447.
-
Bounded TLS client RPCs with shared physical connection pools and real-process recovery: feat(wal): add bounded TLS RPCs for ingress and materializers #448.
-
Rollout ingestion with shared memory admission, session buckets and durable receipts: feat(ingest): add durable rollout ingestion over WAL RPCs #449.
-
Recoverable Lance file preparation, fixed-version publication and source retirement: feat(ingest): publish session-sharded WAL records into Lance #450.
-
Independent WAL node service, automatic maintenance and six-process restart qualification: feat(wal): add independently deployable replica service #451.
-
Bounded HTTP gateway and journaled Rust producers with exact-retry and Lance recovery qualification: feat(ingest): add HTTP gateway and recoverable Rust producers #453.
-
Add bounded materializer workers and independent managed readers (feat(ingest): coordinate bounded materializer workers with etcd leases #455).
-
feat(ingest): bootstrap managed rollout tables idempotently #457 — Idempotent managed rollout bootstrap with persisted genesis identity and crash recovery.
-
Reconnect read-only RPCs after a reused pooled TLS stream disconnects, preserving the original deadline and admission credits: fix(wal): reconnect reads after pooled TLS disconnects #459. Mutating requests still require explicit original-identity reconciliation.
-
Local runtime admission observations with separate request/byte rejection counters and real-quorum pressure regressions: feat(wal): expose runtime admission pressure #460.
-
Expose local runtime request/byte admission observations through node diagnostics without changing readiness: feat(wal): expose runtime admission in node diagnostics #461.
-
Qualify a single managed table with multiple independent session producers, including full per-producer payload scans and an aggregate table row-count check.
-
Parallel file preparation across workers for one table, with one fenced publisher aggregating a complete bounded set of fragment descriptors: feat(ingest): distribute single-table file preparation #466. Scaling and sustained capacity remain under qualification.
-
Handle and qualify larger inline payloads, including the 40/120 MiB cases in blob_columns (blob-v2 offload) silently loses data and permanently breaks WAL merge #222. The current replicated-record limit is 8 MiB; raising a workload size alone does not cover peer framing, bounded memory, journal/snapshot recovery, or materialization.
-
Compose bootstrap, authenticated ingress, materializer worker and managed-reader roles into a Rust application with executable recovery coverage: feat(ingest): compose durable ingestion application roles #462. Dynamic registry discovery and sustained scaling remain separate work.
-
Expose bounded cumulative materializer phase timing and outcome counters, with documented overlapping-duration semantics: feat(ingest): expose materializer phase durations and outcomes #463. Instrumentation only; no throughput improvement claimed.
-
Add opt-in bounded streaming file preparation within one worker, preserving complete descriptor aggregation and cancellation ownership: feat(ingest): add opt-in streamed file preparation #464. Default off; distributed same-table preparation and performance qualification remain separate work.
-
Add opt-in replica spreading for pinned WAL source reads, preserving exact source validation and all-voter fallback: feat: add opt-in replica spreading for WAL source reads #468. Recovery and single-reachable-replica tests pass; controlled throughput comparison is pending, default remains first voter.
-
Overlap bounded replication with leader persistence (default off; no throughput claim): feat(wal): overlap bounded replication with local persistence #469
-
Reuse verified uncompressed engine read blocks to remove one allocation and copy on cache misses: perf(wal): reuse verified uncompressed read blocks #470. Local durability and recovery checks pass; end-to-end performance qualification remains separate.
-
Return already committed WAL fetches before unrelated subsequent persistence, preserving append ACK barriers and complete payload validation: fix(wal): isolate RPC replies from unrelated waits #471. Real-file failure and process recovery checks pass; end-to-end performance qualification remains separate.
-
Preserve bounded source-scan progress across worker claims: fix(ingest): preserve scan progress across worker claims #472
-
feat(wal): expose fetch queue and service timings #473 — Separate local fetch queue and service timings to diagnose ingestion stalls.
-
Validate preparer source pins concurrently within the existing fetch limit, completing every validation before file writers start: perf(ingest): parallelize preparer source-pin validation #474. End-to-end performance qualification remains separate.
-
Bound CI dependency installation and use the official archive so ingestion recovery tests can execute reliably: ci: bound Rust dependency installation and prefer the official archive #475.
-
Rotate source priority using the completed publication version so larger records cannot be repeatedly skipped; cover publisher takeover and durable-log restart: fix(ingest): give every source group a full publication budget #476.
-
Allow explicitly budgeted publication source limits up to 1 GiB while preserving the default and immutable policy identity: perf(wal): allow explicitly budgeted larger publication plans #480. Sustained performance qualification remains separate.
Query visibility and current follow-ups
The optional session-read API now combines acknowledged WAL rows with published Lance rows. It captures committed WAL metadata, reads retained payloads, then opens a sufficiently recent Lance snapshot and removes overlap by immutable row ID. Readers must consume the complete stream and propagate errors. Different sessions and bounded distinct-record fetches can run concurrently; the WAL-to-Lance visibility handoff must remain ordered. This is not a claim of sustained query capacity or a migration of the existing production API.
-
Query acknowledged session rows before Lance publication: feat(ingest): query acknowledged session rows before Lance publication #481.
-
Batch small publications with a bounded flush wait: perf(ingest): batch small publications with a bounded flush wait #482.
-
Expose append execution failure causes: fix(wal): expose append execution failure causes #483.
-
Prefetch distinct session WAL records within shared memory admission: perf(ingest): prefetch session WAL records within the shared byte budget #485.
-
Replay checkpoint compaction after native-log recovery: fix(wal): replay checkpoint compaction after native log recovery #486.
-
Admit native cleanup using live rewrite input: fix(wal): admit native cleanup using live rewrite input #488.
-
Filter session metadata at its WAL group: perf(ingest): filter session metadata at the WAL group #489.
-
Bound published session scan batches and storage prefetch: fix(ingest): bound published session scan batches and prefetch #492.
-
Support compaction publication through the existing catalog protocol: feat(wal): support compaction publication in the catalog #496.
-
Compact eligible small files through managed publication: feat(ingest): compact small files through managed publication #498.
-
Preserve bounded append response diagnostics in the recoverable HTTP client, without changing retry identity or intent advancement: fix(ingest): preserve append response diagnostics #500.
-
Recover compaction after publisher termination: test(ingest): recover compaction after publisher SIGKILL #504.
-
Configure published session scan concurrency: feat(ingest): configure published session scan concurrency #508.
-
Schedule bounded compaction while WAL data is awaiting publication: fix(ingest): schedule bounded compaction during WAL backlog #511.
-
Keep failed maintenance eligible on the next scheduler tick: fix(wal): keep failed maintenance eligible on the next tick #512.
-
Preserve follower disk headroom for maintenance: fix(wal): preserve follower disk headroom for maintenance #514.
-
Reuse verified materializer payloads without another digest pass: perf(ingest): avoid rehashing verified materializer fetches #516.
-
Preserve row identity in newly created managed ingestion tables: feat: preserve row identity in new managed ingestion tables #517.
-
Maintain native session indexes during publication: feat(ingest): maintain native session indexes during publication #518.
-
Identify failed WAL client RPC exchanges: fix(wal): identify failed client RPC exchanges #520.
-
Filter unindexed session fragments before payload reads: perf(ingest): filter unindexed session fragments before payload reads #522.
-
Discard disconnected idle streams before dispatching requests: fix(wal): discard disconnected idle streams before dispatch #524.
-
Reuse native log integrity checks for accepted reads: perf(wal): avoid redundant digest scans on native accepted reads #527. Snapshot pack reads retain payload digest validation.
-
Keep bucket records together within each publication with opt-in byte-balanced preparer assignment: feat(ingest): preserve bucket locality across preparers #535. Task/cache identity includes the assignment version; tests cover complete-aggregate publication, retry reuse and WAL reopen. Default striping is unchanged; sustained performance remains under qualification.
-
Qualify combined ingestion, session queries, publication and compaction under sustained load. Configuration changes require throughput, latency, backlog and memory evidence; merged implementation alone does not establish production readiness.
Current turn annotations and mixed ingestion
The current-annotation path adds a stronger visibility contract to the original immutable-ingestion path described above: a successful mutation response waits for both durable quorum commit and ordered application. Point reads through another ingress resolve the owner's current annotation view. Publishing to Lance remains asynchronous.
-
Define label add/remove and status/score set/clear semantics, preserving omitted fields (feat(api): define turn annotation field mutation semantics #567). Apply mutations in committed order and return retained request results for exact retries (feat(ingest): wait for annotation application before responding #574, feat(ingest): checkpoint annotation results with published values #575). CAS remains outside the first implementation.
-
Bound staging to one active generation and at most one frozen generation, with finite reader/cache retention (feat(core): add bounded Lance annotation staging views #569). Point reads address a turn directly and capture a compatible staged value or Lance base; an annotation-only miss projects narrow columns. Ordinary annotation point reads do not scan the WAL suffix.
-
Publish the frozen generation through the fenced catalog protocol, bind its coverage and request results to the selected Lance state, and retire only that generation (feat(ingest): preserve annotations through managed publication #572, feat(ingest): checkpoint annotation results with published values #575, feat(ingest): drain and recover one annotation shard automatically #579). Later active updates survive publication and old readers retain their captured views. Recovery reconstructs the remaining suffix; unknown outcomes are resolved using the original publication/request identity (test(ingest): recover a lost Prepare reply with acknowledged updates #621, fix(ingest): confirm uncertain publication plans by quorum observation #669, fix(ingest): observe uncertain publication advancement #670, fix(ingest): resolve unknown annotation appends by quorum observation #673).
-
Run authenticated owner/ingress roles with backpressure, bounded file collection and managed small-file compaction (feat(ingest): route annotation HTTP through one fenced owner #580, feat(ingest): run authenticated annotation owner and ingress roles #581, feat(ingest): reclaim files in the exclusive annotation owner #603, feat(ingest): compact annotation tables through managed publication #611). Tests cover concurrent updates, delayed/unknown publication, retained-reader pressure, process recovery and reclamation. These checks do not establish deployment capacity.
-
Add core support for first-create bodies plus current annotations and paired publication/compaction recovery (feat(core): stage new turns alongside annotation updates #656, feat(core): bind body snapshots to frozen annotations #658, feat(core): capture first body origins with annotation views #659, feat(core): preserve annotation views across body compaction #666, feat(ingest): publish paired turn bodies and annotations #672, feat(ingest): recover paired body compaction publications #675). Re-import of an existing turn preserves its original body and current annotation.
-
Integrate the mixed ordered driver (feat(ingest): stage mixed turns with bounded Arrow frames #676), paired draining/compaction runtime (feat(ingest): drain and compact mixed turn staging #677), owner/ingress HTTP roles (feat(ingest): serve mixed turn writes through owner and ingress roles #678), and fresh body/annotation bootstrap (feat(ingest): initialize a fresh mixed body and annotation pair #680). Bounded Arrow frame windows retain shared buffers across staging reads and publication. Implementation and recovery regressions are merged; deployment capacity remains a separate qualification.
-
Preserve read-only quorum observation after unknown replies (fix(wal): reobserve read-only queries after unknown replies #679), and keep annotation owners alive while a failed GC listing is resolved (fix(ingest): keep annotation owners alive after GC listing failures #681). These changes do not resubmit a business mutation after an unknown outcome.
-
Reuse completed compaction checks for an unchanged snapshot (perf(ingest): reuse compaction checks for unchanged snapshots #682) and expose separate annotation observe/fetch/decode/create/patch timing (feat(ingest): expose annotation apply phase timings #683). Stage timings overlap and are not ACK latency percentiles.
-
Bound committed source-metadata pages (perf(ingest): batch committed annotation source metadata #684). Payload admission, fetching, decoding and application remain sequential; merged metadata paging alone does not establish capacity.
-
Use byte targets for managed compaction reads (perf: use byte targets for managed compaction reads #652), avoiding batches sized only by row count.
-
Complete sustained performance qualification of the combined mixed ingestion, point-read, compaction and reclamation path. Merged optimizations alone do not establish deployment capacity.
-
Preserve current annotation state and captured readers across native index publication (feat(core): preserve annotation views across index publication #685). This core view transition does not by itself enable owner index maintenance or establish index/storage performance.
-
Add opt-in bounded native turn-ID index maintenance through the existing fenced publication and recovery path (feat(ingest): maintain native indexes for annotation turn IDs #686). Builds and size-tiered merges preserve the materialized cut, request results, body reference and active updates; the option remains disabled by default.
-
Collect obsolete native ID index page files through the existing bounded, fenced collector (fix(ingest): reclaim obsolete native annotation index pages #687), protecting raw index UUIDs in retained manifests and held read views. Full repository checks and CI passed; long-term index/storage qualification remains separate.
-
Avoid reserving fetch response bodies for nonempty append requests (fix(wal): avoid reserving fetch response bodies for appends #688). Regressions cover concurrent append admission, cancellation credit release and fetch validation. Full repository checks and CI passed; end-to-end throughput qualification remains separate.
-
Include pending image-file reservations in native log reclamation pressure (fix(wal): reclaim expired logs under image reservation pressure #689). The real-engine regression keeps the reservation held while reclaiming obsolete payloads and checks admission plus restart. Full repository checks and CI passed; cloud throughput comparison remains separate.
-
Remove the redundant initial ownership lookup from annotation point reads (perf(ingest): fence point reads after resolving their view #690), retaining the quorum fence after resolving the captured view. A real cold-read takeover regression rejects both in-flight and already-stale predecessor responses. Full repository checks and CI passed; deployment throughput comparison remains separate.
-
Add explicit mixed annotation owner/bootstrap APIs for independent physical shards (feat(ingest): bind annotation owners to independent WAL shards #691), binding each shard to its durable schema contract and validating every turn before WAL submission. A real RF3 regression covers a stopped owner while another shard drains, retained request results, and WAL reopen. Full repository checks and CI passed; configured ingress routing and multi-worker throughput qualification remain separate.
-
Distinguish annotation WAL append and ordered apply wait observations (feat(ingest): distinguish annotation append and apply wait observations #692), exposing the existing cumulative phase counters separately for WAL acceptance and local apply waiting. A real RF3 test pauses the applier after durable acceptance to verify the distinction; cancellation remains explicit. Full repository checks and CI passed; cloud latency attribution and throughput qualification remain separate.
-
Route mixed annotation HTTP requests across independent shard owners (feat(ingest): route annotation HTTP traffic across shard owners #693), using stable turn routing for point reads and verified producer binding for updates, without re-encoding Arrow IPC. A real RF3 regression covers two ingresses, independent owners and owner replacement. Full repository checks and CI passed; deployment configuration and multi-worker throughput qualification remain separate.
-
Configure independent physical annotation shard applications (feat(ingest): configure independent annotation shard applications #694). Each owner has its own body/narrow tables and publication/recovery progress; shared ingresses use a validated route map. Real RF3/etcd process tests cover acknowledged suffix recovery while a peer drains. Full repository checks passed and all CI is green. Automatic client partitioning, migration and sustained multi-owner capacity remain separate.
-
Overlap independent annotation source metadata observations (perf(ingest): overlap annotation source metadata observations #695). Both quorum results are validated before consuming a page; ordered application, bounded admission and closure fencing remain intact. A regression rejects a stale view without fetching or advancing. Full repository checks passed and all CI is green; no workload throughput gain is claimed.
-
Document annotation owner index and admission settings (docs: clarify annotation owner index and admission settings #696). Each role consumes its own configuration; native ID maintenance belongs on the owner, with explicit prerequisites and coverage checks. Documentation only; all CI is green.
-
Reuse admitted annotation upload buffers during ordered apply (perf(ingest): reuse admitted annotation payloads during apply #697). Exact quorum-observed metadata selects the shared buffer after apply admission; cancellation, restart and remote submissions retain WAL fetch fallback. Durability, ordered apply and publication remain intact; all CI is green.
-
Expose annotation upload admission, body read, preparation, point lookup, and owner-authority timing in bounded diagnostic counters (feat(ingest): expose annotation request stage timings #698). Existing admission, durability and consistency checks remain intact; all CI is green.
-
Expose bounded timing for annotation predecessor loading, target publication, selected-target validation, and durable catalog confirmation (feat(ingest): observe annotation publication resolution stages #699). Existing publication ordering, fencing and recovery checks remain intact; all CI is green.
-
Exercise pipelined replication through independent TLS process leader failure, original-directory restart and interrupted snapshot recovery (test(wal): cover pipelined replication process recovery #700). Both pipeline settings retain full payload and exact-request checks; runtime defaults and ACK durability remain unchanged; all CI is green.
-
Report whether ordered annotation apply is blocked by its memory budget, source window or staging capacity (feat(ingest): report ordered apply backpressure reasons #702). Preserve existing admission and publication behavior; observations are sampled states, not refusal counts or blocked durations; all CI is green.
-
Refuse HTTP/2 annotation uploads that lack memory before consuming their bodies, so waiting streams do not stall admitted uploads on the same connection (fix(ingest): avoid blocking shared HTTP/2 uploads on memory admission #703). Preserve existing memory budgets and explicit not-submitted outcomes; shared-connection regression and complete mixed-workload value/recovery checks cover the change; all CI is green.
-
Separate append RPC admission, connection, send and response timing from WAL execution timing (feat(wal): expose append transport and execution timing #705). Controlled RF3 waits/cancellation and a complete mixed-workload data/recovery audit validate the observation boundaries; no ACK, admission or retry policy changes; all CI is green.
-
Preserve committed coverage for each native scalar-index segment before combining matches (fix(index): filter stale matches before combining scalar segments #706). Regression tests cover mutable status values, clearing to NULL, append-mode catch-up and reopening with stable/physical row IDs; this fixes the existing query wrapper and does not add an asynchronous indexing service; all CI is green.
-
Bound request-drain work before Raft progress with a configurable time budget (fix(wal): bound work before polling durable progress #707). Retain command batching, durable quorum/apply ACKs and recovery behavior; a gated RF3 regression checks progress and complete payloads after reopening; sustained mixed-workload qualification and all CI passed.
-
Opt-in bounded native status index maintenance (feat(ingest): maintain bounded native status indexes #708): idle catch-up, physical-byte merge admission, explicit empty-definition retirement, prepared-operation recovery with maintenance disabled, and retained-reader file reclamation. Filtering API/observable lag and status-inclusive mixed-load qualification remain separate work.
-
Bounded materialized status filters (feat(ingest): expose bounded materialized status filters #709): per-shard HTTP queries project IDs from one pinned Lance snapshot, include unindexed fragments, report materialization/index coverage, fence stale owners and retain GC pins through native query work. Active/frozen updates remain immediately point-readable; filters observe materialized state. No pagination or cross-shard snapshot.
Annotation applications now support explicit fresh physical shards, with one body/narrow table pair per owner and a complete ingress route map (#694). Configured routing and independent shard publication/recovery are implemented; the default remains one table and one virtual shard. Automatic session/client placement and migration, asynchronous status/label/score filtering indexes, and large-table index, storage and recovery costs remain open work. These explicit shard APIs do not establish saturation capacity or complete automatic scale-out.
Remaining qualification must run mixed new-body ingestion, point updates, cross-ingress reads, compaction and reclamation together; verify all accepted values and recovery progress; and report rejection rates, latency distributions, backlog/drain, process memory and storage costs. Separate-process and multi-host persistent-storage/blob tests must state their topology and fault boundary. Closed-loop load and small recovery fixtures must not be presented as fixed-arrival-rate capacity.
See the mutation contract, staging view, and owner runtime.
Activity
feat: add shared WAL proposal admission budget (#377)
feat: reserve configurable WAL maintenance headroom (#379)
feat: share validated Arrow IPC decoding for WAL materialization (#380)
feat: add shared WAL record envelopes and payload validation (#381)
perf: move owned rollout upload buffers into core records (#382)
fix: honor registered rollout dataset locations (#383)
feat: define immutable WAL ingestion contracts (#384)
beinan commentedon Oct 19, 2026
Progress update: the foundational PRs through #384 are merged. They provide the Lance 13 upgrade, allocation/ownership improvements, session placement, admission controls and immutable ingestion contracts. The distributed ingestion path remains experimental and is not enabled by these PRs.
A synthetic Rust experiment compared 32 MiB and 64 MiB publication source caps in an ABBA sequence. Each case wrote 8,192 distinct 128 KiB payloads (1 GiB), using 32 writers, 8,000 session keys and three RF3 groups across three nodes, then materialized into Lance on object storage. Other admission, fetch and maintenance settings were held fixed within the cohort.
All four cases passed complete payload verification, maintenance audits and process restart recovery: all nine group replicas restored their checkpoints, the Lance version/data stayed unchanged, and recovery produced zero additional publications.
Pooled published throughput increased from 37.59 to 40.18 MiB/s, about 6.9%. Two short runs per setting are insufficient to establish sustained capacity or a dependable p99 improvement, so this does not justify a default change yet. These measurements exclude HTTP, catalog resolution, receipt authentication and fixture generation; publisher memory includes a resident 1 GiB synthetic fixture pool. They test process recovery on the original volumes, not volume loss or replacement.
Next steps remain service integration, explicit retry/visibility semantics, replica-pressure coordination and longer bounded-retention tests. Future implementation PRs will continue to use
Refs #378; this issue stays open. Only portable synthetic parameters and aggregate results are included here; operational artifacts remain outside the public repository.fix: prevent racing dataset creation from overwriting data (#385)
beinan commentedon Oct 19, 2026
Concurrent provisioning exposed a data-loss race that is addressed by #385: Lance checks an absent dataset before creation, then reopens its URI at commit time. A delayed creator can otherwise overwrite a second writer's newly appended data. The context-side guard permits only manifest version 1 during creation and preserves normal appends through the returned handle.
The deterministic regression delays an actual empty storage-listing result while another creator creates and appends. Removing the guard makes the test fail with zero rows instead of one; restoring it preserves the row and version. A barrier at manifest authorization was too late to reproduce this race, so it was replaced before merging.
A separate synthetic admission-window experiment stopped at its second case because it did not meet the predeclared maintenance-exposure gate. Both completed cases verified all 4 GiB of distinct payloads and passed process restart recovery on their original volumes, but the larger-window case completed only 12 maintenance rounds against a required minimum of 16. The remaining two ABBA cases were not launched. This is not evidence of an accepted sustained-throughput improvement, and no defaults changed.
The next performance comparison needs longer workloads with bounded fixture generation and explicit generation/encoding/admission timing. Service integration, distributed replica disk credits and producer-state lifecycle remain open; the foundational PRs do not enable the distributed writer in production.
beinan commentedon Oct 19, 2026
A four-case synthetic Rust ingestion/publication/recovery comparison has passed all predeclared gates. Each case ingested 98,304 distinct 128 KiB payloads (12 GiB), with 32 writers, 8,000 session keys and three RF3 groups across three nodes. We held the total node proposal budget at 4 GiB and varied only the producer ceiling in an ABBA sequence.
All cases verified every payload before and after a full WAL-process restart. All nine group replicas restored checkpoints; Lance versions stayed unchanged, with zero additional recovery publications and no maintenance failures. All WAL node process incarnations were explicitly stopped and their recorded identities checked.
Pooled published throughput was 38.92 versus 46.18 MiB/s (+18.6%) in this cohort. Tail latency did not consistently improve. External samples showed peak per-node allocated WAL footprints of 426 and 1412 MiB, respectively; peak sampled acknowledged publication backlogs were 198 and 631 MiB. Two trials per setting are insufficient to establish production capacity or a reliable tail-latency gain; defaults remain unchanged.
The streamed fixture generator retains no workload-wide payload pool and bounds concurrently generated raw payloads to 4 MiB for this workload. Generation is included in request/throughput timing. Maximum observed publisher process RSS high-water was approximately 335 MiB; RSS is not total Pod/cgroup memory, which also accounts for cache and other processes. External resource sampling actually ran roughly every 5–8 seconds, so sampled disk/RSS peaks are observations, not hard bounds. HTTP, authentication, catalog lookup and writer connection setup are outside timing; full readback and restart audits follow the timed publication drain. These results are separate from earlier precomputed-fixture or failed cohorts.
The Arrow schema-contract extraction in #386 (
Refs #378) is now merged. Its local Rust/Python checks and all five GitHub workflows, including all packaging platforms and final distribution validation, passed at the reviewed head before merge. Replica disk credits, producer-state lifecycle, incremental checkpoints and production service integration remain open work.feat: share canonical Arrow ingestion schema fingerprints (#386)
beinan commentedon Oct 19, 2026
The local experimental Rust ingestion path now has a separate automatic materializer service. It discovers table generations through bounded catalog pages and claims work with expiring leases. Replicated publisher epochs and pinned Lance versions fence publication; the scheduling lease alone is not a storage fence. Active-table slots, a shared source-byte budget and a publication quantum bound work. There is no worker per session.
Real-process tests with etcd and WAL nodes cover competing workers, catalog changes during discovery, prepared-publication recovery after owner loss, stale-owner rejection, complete payload comparisons, catalog/WAL restarts and interrupted table closure. A separate HTTP/materializer test verifies that writes receive durable ACKs while materialization is paused, a replacement automatically publishes the backlog, and full ingestion/materializer/WAL process restart plus request retries preserves data without duplicate publication. All 36 columns of 34 synthetic rows, each with a 128 KiB binary payload, are compared; graceful service exit also passes. This is local correctness coverage, not sustained HTTP performance or a released service feature.
The object-storage experiment intended to compare serial and parallel file materialization stopped after its first serial case: two of 960 node-budget observations exceeded the predeclared 800 ms timeout. All 98,304 distinct 128 KiB payloads (12 GiB) passed initial readback, but the observation gate failed. The remaining three cases were never started, the workload was not repeated to obtain acceptance, and defaults remain unchanged. There is no accepted concurrency comparison.
Diagnostic recovery of that original data passed: all nine group replicas restored checkpoints, every payload was verified, the Lance version stayed unchanged, and recovery made zero additional publications. The original failed gate remains recorded; the status-timeout cause is still under investigation.
The materializer currently performs logical retirement. Physical checkpoint/compaction/collection orchestration, efficient discovery with many idle tables, replica disk credits, producer-state lifecycle, authorization/migration and sustained HTTP throughput remain open. Upcoming implementation PRs will continue to use
Refs #378and remain small; this issue stays open.beinan commentedon Oct 19, 2026
The local experimental WAL node now has opt-in automatic checkpoint, compaction and physical collection. One scheduler covers the groups sharing a node's engine/budget, including followers. It uses local durable applied cuts, recovers installed-but-uncompacted checkpoints after restart, and avoids rebuilding an unchanged group. Payload retirement still requires the existing replicated Lance publication proofs; scheduling alone does not authorize deletion.
Real-process tests cover interrupted compaction, unchanged idle groups, a held checkpoint builder across observation timeouts, and full process loss. An existing checkpoint cancellation/recovery regression also passes. Scheduling retains the current job identity after an uncertain observation, while existing builder limits, byte reservations and file/read pins protect concurrent work.
A separate local HTTP/materializer/node test writes 1,024 synthetic rows with 128 KiB binary payloads (128 MiB total), in four publication/drain windows. Eight concurrent upload futures bound generated bodies. Each node has a 32 MiB producer ceiling within a 256 MiB total proposal budget. The test never manually invokes checkpoint, compaction or collection. It compares all 36 columns, requires zero retained source payloads in the final compacted checkpoints and WAL footprints below 32 MiB, then restarts all HTTP/materializer/WAL processes and retries every request without changing the Lance version or duplicating rows.
This validates finite local service/reclamation/recovery behavior. It is not a sustained-throughput measurement, a hard filesystem quota, a volume-loss test, or a result on remote blob storage. Full-copy checkpoint cost, producer-state growth, replica disk credits, idle-table discovery, authorization/migration and longer service workloads remain open. The earlier object-storage concurrency cohort remains failed and has not been rerun. The changes remain local experimental work; implementation PRs will continue to use
Refs #378.beinan commentedon Oct 19, 2026
The experimental Rust HTTP ingestion path now has a bounded workload driver with separate prepare, ingest, verify and recovery modes. It streams distinct Arrow payloads, preserves producer/sequence/body across retries, refuses a second ingestion start, and checks complete Lance rows. Request-to-ACK timing includes generation and encoding; durable ACK and Lance visibility are measured separately.
One new K8s/object-storage cohort completed 8,192 synthetic rows with 128 KiB binary payloads (1 GiB), eight writers, 31 sessions and three RF3 groups. All data became visible in 26.620 s: 38.467 MiB/s of binary payload. Request-to-ACK p50/p99/max were 16.167 / 324.863 / 1,121.279 ms. All 153 node-budget and 50 receipt observations passed their declared deadlines, and all 36 columns passed readback. This is one finite initial measurement, not a sustained-capacity or scaling result.
All HTTP, materializer, three WAL and experimental metadata processes were then explicitly killed. New process incarnations reused the same durable directories. Retrying all 248 latest producer/session requests and verifying every row again left Lance version 38 unchanged. Node-owned checkpoint/compaction/collection ran throughout with zero recorded maintenance errors; final observed WAL footprints after draining were below the 256 MiB producer ceiling. Some ingestion-time footprints exceeded that ceiling, which is admission control rather than a hard disk quota. All experiment processes have been stopped.
Each WAL Pod had a 4 CPU / 8 GiB limit on a separate host. HTTP, materializer, load generator and one experimental etcd shared another 4 CPU / 8 GiB Pod. Process resource samples were sparse, and shared cgroup memory reached about 7.08 GiB; without a before-run baseline this does not establish a memory bound or component attribution. The recovery test preserves volumes and does not establish volume-loss recovery or HA metadata.
The earlier failed file-materialization concurrency cohort remains failed and was not rerun. Service-side ACK latency attribution, efficient idle-table discovery, replica disk credits, producer-state lifecycle, incremental checkpoints and production authorization/migration remain open. These service changes are still experimental local work; implementation PRs will continue to use
Refs #378.399 remaining items
perf(ingest): share bounded base lookups across patch requests (#849)
beinan commentedon Nov 12, 2026
A complete mixed-load comparison for #849 is now qualified. Both runs used four materialization workers/four shards, six WAL servers with three replicas per group, two ingresses, Lance 14.0.2, and the same workload/client/resource budgets. Each shard had eight ingestion lanes (256 rows/request), eight update lanes (32 rows/request), and sixteen point readers over 1,048,576 seed turns. The offered grids ran for 600 seconds; ingestion was closed loop and busy background lanes skipped arrivals.
Observed body throughput increased 16.7% and update throughput 29.2%, but point-read p99 worsened 28.5%, fewer point reads completed, and write refusals increased. This is one sequential pair, not a repeatability result or maximum-capacity claim. Latency is per successful request and excludes skipped arrivals; it is never divided by batch width.
The candidate passed all 10,677,504 complete body/current-annotation value checks, 512 post-drain owner-replacement point checks through both ingresses, all eight empty owner drain views, zero final index tails, per-minute materialization progress on every shard, and physical WAL reclamation on all six servers. All 73 original operations, 146 logs and 64 ACK journals were reconciled. One failed object-delete attempt was retained; all final pending queues were empty. Both experiments are closed, with original failures preserved.
Original phase counters show annotation decoding around 1.13 ms/operation versus 487 ms patch apply-wait, 164 ms/shared base lookup, and 964 ms/narrow publication preparation. These are different/overlapping populations, not additive latency components or p99. Moving lookup outside ordered apply alone is not proof of a speedup. The evidence supports further work on base reads, publication and read/write contention before treating deserialization as the primary measured bottleneck.
Next: a fresh sustained mixed workload adds status-filter readers while retaining ingestion, updates, point reads, indexing, compaction, GC and complete value verification. Status filters cover their reported materialized snapshot; point reads include staging. Fault qualification and the full high-throughput objective remain incomplete. Candidate service revision: f680c19; unchanged benchmark client: bf5acfefb3253eea9d51f29471e3445818872fb5.
fix(wal): route read queries past disconnected voters (#850)
perf(ingest): split point-read phase observations (#852)
fix(ingest): wait for bounded status reader admission (#853)
beinan commentedon Nov 12, 2026
Mixed-load qualification update after #864:
A 600-second native Rust run used four workers/four independent shards, six RF3 WAL nodes and two ingresses. Each worker/WAL/ingress had 4 CPU and 8 GiB; the driver had 8 CPU and 8 GiB. The workload seeded 4,194,304 turns with 16 KiB pseudorandom bodies, then ran ingestion, 32-row annotation updates, point reads over the full seed set, and concurrent status queries. Per shard: eight ingestion writers, eight update writers, 16 point readers and two status readers. Elections were automatic.
Write attempts: 50,027, including 854 explicit refusals (1.707%). The update lanes offered 384,000 requests: 26,567 ran and 357,433 were skipped while the lane was busy. Point readers offered 480,000: 293,055 succeeded, 186,945 were skipped, with no refusals/errors. Status readers offered 48,000: 8,157 captured snapshots, 272 refusals and 39,571 skipped arrivals, with no query errors. Logical write ACK latency includes backoff after explicit not_submitted refusals. Point/status latency describes successful requests; refusal and skipped-arrival counts remain separate. Update p99 is per request, not divided by 32.
Correctness and draining passed: 9,981,440 complete bodies/current annotations checked against original inputs and ACK journals; 512 point reads after owner replacement; all eight original/replacement staging views drained; every shard published in each observed full minute; all six WAL nodes showed physical space reclamation. All 65 operations and 130 original logs were reconciled. Internal Append counters separately recorded 49,173 accepts and 277 disk-budget rejections, with no leadership-unknown outcomes; peer exchange, maintenance and snapshot failures were zero. Transport frame drops were nonzero and are retained in diagnostics, not counted as business refusals.
Status queries are materialized snapshots with visible progress. Their maximum observed uncovered tail during load was 1,368,832 physical rows; all four final tails reached zero after drain. Final catch-up does not establish low indexing lag during sustained load. Publication was configured for 192–256 MiB input batches with a 5-second maximum wait, and a 128 MiB compaction target; these are configured targets, not measured output file sizes.
This qualifies this workload's correctness and reclamation, not a maximum-capacity or matched speedup claim. Owner replacement occurred after drain, so it does not prove in-flight crash recovery. Next are a kill during continuous reads/writes and WAL-node/storage fault qualification. Tail latency and missed arrivals still need substantial improvement; ordered application and cold narrow-value lookups remain the measured investigation focus. No throughput gain is claimed for the pending index-only lookup-reuse change.
beinan commentedon Nov 13, 2026
Owner SIGKILL recovery qualification: 20,480 turns
One owner/shard, three durable WAL replicas, two ingresses and eight point readers. After killing the original owner, a native check proved an acknowledged annotation update remained above the selected Lance cut and matched its original WAL payload. Recovery verified all 20,480 turns, including full bodies and current annotations, and 16 repeatedly updated targets. Four requests with unknown client outcomes were resolved as committed using their original identities; recovery submitted/replayed zero client mutations.
Independent audit checked all 16 original processes, 32 sealed process logs and 97 original journals, including 64,620 acknowledged non-seed requests. Both replacement views drained, all services stopped, and test resources were retired with stores and logs preserved. An initial audit comparison failed on JSON representation of native serde defaults; the corrected comparison follows the exact native omission/null rules and retains every other check. The original failure is preserved.
Availability remains a separate failed qualification. The original 600-second reader workload recorded 232,480 attempts: 205,693 successful responses passed the value oracle, and 26,787 returned HTTP 503. Each reader had a main observed first-failure-to-next-success interval of approximately 70 seconds, including the controller's evidence checks before starting the replacement. This is not automatic failover latency. The original continuous-reader phase failed because its after-ready fence capture exceeded the original workload window; later data recovery and trace inspection do not convert it into a pass. These results do not qualify WAL-node loss, durable-storage loss or capacity.
Next is a matched owner Tokio thread-count comparison (2 to 4) using the existing qualified mixed-load source and unchanged resources/workload. PR #867 is prepared as a separate source-only candidate for a subsequent comparison; its 46 native harness tests and service builds passed. No new throughput gain is claimed.
beinan commentedon Nov 13, 2026
Owner-process failure update: the original data and continuous point-read evidence now pass independent verification for a single-shard cloud/blob test (source
f795922965d3521384259602ec00725afcc05c6f; one owner, RF3 WAL, two ingresses, eight readers, 600-second reader window).The owner was killed while an acknowledged annotation update remained above the selected Lance cut. Native verification while that owner was stopped matched the exact retained WAL record, ACK and deterministic input. Both ingresses subsequently returned that update after replacement readiness.
This test includes an outage. Per-reader longest observed failure-to-next-success windows were 114.590–114.634 seconds, including the controller deliberately waiting for original writers to settle and collecting stopped-owner evidence before starting the replacement. These durations are not automatic failover latency or a zero-downtime result. The original load retains exit code 1.
The original controller also exited 1 during post-load immutable ACK-log transfer. That failure is preserved. After all 16 original processes were terminal, a separate read-only supplement completed the exact missing bytes and remaining evidence, preserving every raw/compressed digest and the original reader, request, identity and full-value checks. It launched no service/workload and replayed no business request. Independent qualification passed on that complete original evidence; the original controller has not been relabeled successful.
Scope remains owner-process loss with retained WAL replicas. A following case will keep one WAL replica unavailable through the workload, owner failure and recovery. Physical disk destruction and empty-disk same-ID rejoin remain unqualified. This result makes no throughput-improvement claim; the latest four-worker mixed-load result remains unchanged, and the owner-thread/PR #867 performance comparisons are still pending.
Correction (2026-11-13): The negative unknown outcomes above were not proven definitely absent. The audit observed unchanged committed producer prefixes, but the original producer generations were still open; repeated quorum reads and stopped callers alone do not rule out a delayed request. The two positively identified commits remain confirmed; the two negative outcomes remain unknown. The full-value and ACK checks remain evidence for the observed committed prefixes, and the original failures are preserved. This narrows the qualification: it does not establish definitive resolution of every unknown request. No data loss has been demonstrated. Future recovery tests will use the existing durable producer-generation fence before making an absence claim; the product already retains unknown on a negative read. No original workload or retired store is being replayed.
beinan commentedon Nov 13, 2026
The next fault case exposed a WAL client routing bug: after stopping one voter process and killing the owner, a fresh payload
Fetchselected the unavailable voter and failed withConnection refused, despite the surviving quorum. PR #870 lets read-only Fetch use the existing configured-voter transport fallback within its original deadline and memory budget. It does not replay submitted mutations. All 44 client regressions passed, including two new real TLS/Raft cases; the PR is still awaiting the remaining HA CI.The original fault test remains failed: its stopped-owner check did not complete the exact unmaterialized ACK/WAL/input proof. A separate recovery audit using the fixed client has now completed and independently verified:
This establishes recovery of that original data, not the missing continuous fault-read qualification, physical disk-loss/rejoin behavior, or a throughput improvement.
Fetch(None)remains a local committed read, not a quorum absence proof.A fresh RF3 test retains the original 600-second split-field workload, eight continuous readers, and unavailable-voter-plus-owner-failure sequence. Subsequent owner-thread and #867 cache comparisons will use the same Fetch-fixed baseline. The previously reported four-worker throughput remains unchanged.
Correction (2026-11-13): The negative unknown outcomes above were not proven definitely absent. The audit observed unchanged committed producer prefixes, but the original producer generations were still open; repeated quorum reads and stopped callers alone do not rule out a delayed request. The four negative outcomes therefore remain unknown. The full-value and ACK checks remain evidence for the observed committed prefixes, and the original failures are preserved. This narrows the qualification: it does not establish definitive resolution of every unknown request. No data loss has been demonstrated. Future recovery tests will use the existing durable producer-generation fence before making an absence claim; the product already retains unknown on a negative read. No original workload or retired store is being replayed.
beinan commentedon Nov 13, 2026
A fresh single-shard recovery test now passes independent verification with one WAL voter unavailable and the owner killed during writes. This uses the Fetch fallback merged in #870; the earlier failed tests and their unresolved outcomes remain unchanged.
The current WAL leader was stopped first. After both ingestion lanes advanced on the surviving quorum, the owner was killed. Before starting its replacement, the native audit proved an acknowledged annotation update at WAL position 8770 remained above Lance cut 8566, matching the original ACK and payload. Both ingresses subsequently returned that update.
This qualifies the stated process-fault recovery scenario, including an acknowledged but unmaterialized update. It does not establish zero downtime, automatic failover latency, disk destruction/empty-disk rejoin or maximum capacity.
No new throughput gain is claimed. Next are matched four-worker comparisons of two versus four owner Tokio threads, followed by #867's reuse of bounded lookup results across index-only publications, with unchanged workload and resource budgets and explicit p99/refusal/skipped-arrival/backlog accounting.
beinan commentedon Nov 13, 2026
The fresh four-worker baseline for the owner-runtime and #867 comparisons has completed full verification following the Fetch fix in #870.
Configuration: four workers/shards, six RF3 WAL nodes, two ingresses, 4 CPU / 8 GiB per worker, 4,194,304 seeded turns, and 600 seconds of mixed ingestion, annotation updates, point reads and status queries. Ingestion uses 256-row requests with 16-KiB logical bodies; updates use 32-row requests.
All 282,189 successful point reads passed with zero point errors/refusals. Status queries recorded 7,017 successes, 342 refusals, zero errors and 40,641 skipped arrivals out of 48,000 offered. Status filtering covers materialized snapshots; maximum sampled unindexed lag was 1,591,808 rows. Successful-request latency excludes skipped arrivals, so these results do not establish service capacity at the offered rate.
Correctness and draining checks passed:
Owner CPU evidence supports the next comparison: each owner had a 4-core quota but averaged 2.16–2.18 cores, with two Tokio workers each using about 0.97 core and almost no quota throttling. The next case changes only owner Tokio workers from two to four, followed by #867's bounded lookup reuse. No throughput gain is established yet. Automatic leader placement, shared-cluster variation and physical volume capacity expansions with unchanged configured IOPS/bandwidth remain recorded comparison caveats.
beinan commentedon Nov 13, 2026
The four-Tokio-thread comparison has completed, following the two-thread baseline. Four workers/shards, six RF3 WAL nodes and two ingresses are unchanged; each owner still has a 4-CPU quota. Product binaries, the driver, workload and application budgets are unchanged. The configured change is owner Tokio workers from two to four.
The 600-second mixed load uses 4,194,304 materialized seed turns, 256-row ingestion requests with 16-KiB logical bodies, 32-row annotation requests, point reads and status queries.
These are observed rates, not maximum capacity or isolated causal speedups. The four-thread run still skipped 90.28% of scheduled update requests. Successful-request latency excludes skipped arrivals. Automatic leader placement, shared-cluster variation and physical volume capacity expansion with unchanged configured IOPS/bandwidth remain comparison caveats. Logical body throughput is not physical blob bandwidth.
All 375,353 successful point reads passed with zero point errors/refusals. Status queries had 11,812 successes, 92 refusals, zero errors and 36,096 / 48,000 skipped arrivals. Status filtering covers materialized snapshots. Maximum observed status unindexed physical rows increased to 2,586,112 from 1,591,808 in the baseline, so the ingestion improvement does not resolve index lag.
Validation completed:
Final turn-ID index tails are [0, 0, 2304, 0]. Complete index catch-up is not established. The original offline status analyzer failed its zero-tail assertion. That executable check contradicted the preregistered and primary-analyzer criterion of each tail below 4096 rows. After observing the result, separate corrected status/review copies explicitly relaxed only the zero-tail gate to the registered threshold; all remaining analyses passed. Original scripts, criteria and failed outcome are preserved. No workload or index build was rerun. The earlier point-phase source-binding correction is also retained.
Owners now average 3.71–3.80 process cores. About 53–62% of cgroup periods contain some throttling; that is not a stalled-wall-time percentage. Mean apply decode is 0.87 ms, versus 319.95 ms apply wait and 109.89 ms shared patch-base lookup, measured over overlapping populations rather than additive CPU phases. The next comparison tests #867's existing bounded lookup reuse across validated index-only publications, with four Tokio threads and the same workload. No lookup-reuse result is claimed yet.
beinan commentedon Nov 13, 2026
Completed the three-case mixed ingestion/update comparison for #378. The largest observed throughput increase was between the two-thread and four-thread owner configurations. Annotation update capacity remains insufficient: approximately 90% of scheduled update requests were skipped before submission.
All cases used 4 owners/workers, 4 shards, 6 WAL nodes with RF3, and 2 HTTP ingresses, with a 600-second timed load, 4,194,304 materialized seed turns, 256-row ingestion batches with deterministic pseudorandom 16-KiB bodies, and 32-row annotation update requests. Each owner had a 4-CPU quota. Lance was 14.0.2. The lookup change is #867; all cases include #870. No Lance dependency or workload change separates these cases.
MB/s counts acknowledged logical body bytes, not physical blob or replica bandwidth. P99 values are histogram upper bounds for successful requests; they exclude skipped arrivals and refusals. Point latency is for annotation-only reads; status filtering covers materialized snapshots and does not include every unmaterialized update. Each case scheduled 384,000 update requests and 480,000 point reads. Lookup preservation skipped 345,200 update requests and 97,580 point reads; its 382,420 successful point reads reported 0 errors and 0 refusals. This does not establish sustainable capacity at the offered rate.
In every case, acknowledged new-row publication advanced in all 40 shard-minute windows, all six WAL nodes showed physical space reclamation, and all eight original/replacement owner views drained. Complete original ACK journals, terminal operation logs and full-value audits were checked before accepting these results; final retirement checks completed. Replacement point checks ran after drain and do not replace the separate in-flight fault experiment. ID-index acceptance was preregistered as fewer than 4,096 trailing rows per shard; nonzero tails are not full index catch-up, and status-index lag is a separate measurement. The original runtime-4/lookup-4 analysis failures from the contradictory zero-tail assertion are preserved; the corrected analysis changes only that executable assertion to the registered bound.
Lookup preservation measured 3.10% higher ingestion and 4.03% higher annotation throughput than the four-thread case. This is one run per case, not a statistically established or isolated causal improvement. Leader placement, physical volume expansion and shared-cluster variation remain confounders. The owner CPU consumption supports collecting actual owner stacks next; overlapping apply/lookup phase durations are not CPU attribution. The separate profiling run will use fresh identities, and its instrumented timings will stay outside this comparison.
Sources: product
367cefcd1390d34458afff2095d79c11a49ce245; lookup owner6a0fb31c3ee7d8bd09f426b0a74d681c9b737e51; drivere9951746b8b4e80e38856610554eefe430ca2435.beinan commentedon Nov 13, 2026
CPU follow-up to the unprofiled four-worker comparison. This separate diagnostic used the same workload and topology: four owners/shards with four CPU cores each, six RF3 WAL nodes, and two ingresses. Its instrumented timings are not another throughput comparison.
All four original-owner profiles are sealed, collected, and symbolized against their exact runtime executables/libraries. The original controller's copies exactly match the independently collected copies; all 127,431 samples are conserved.
About 84–90% of the constant-page leaf samples pass through annotation_staging::read_base: cold reads of materialized narrow annotations. About 89–92% of the SHA-256 leaf samples pass through WAL Batch::new. These are coarse sample shares, with uneven per-thread sampling and differing actual capture windows; they are not precise CPU proportions or a causal throughput breakdown. Stripped libc nearest-symbol names are not interpreted as evidence of DNS, timer, or allocator bottlenecks.
Lance 14.0.2 walks preceding rows when seeking a constant page. Upstream Lance #10339 already removes that walk for pages without repetition levels and avoids encoder-only work on decoded arrays. A narrow backport is being built for native regression and point-scan comparison, alongside an explicit late-materialization scan option. Lists still require repetition-level walking. No performance gain from either candidate is established yet, and no WAL format change is proposed here.
The diagnostic's full native audit verified 12,870,144 original-body/current-annotation values. All eight original/replacement owner views drained, and replacement checks passed 512 point reads with no submitted client mutations. Publication advanced in all 40 shard-minute windows; all six WAL nodes reclaimed disk space, and all four final ID-index tails reached zero. Final evidence qualification and idle-process retirement are complete. This does not extend the earlier failure/rejoin guarantees or resolve the roughly 90% scheduled-update skips in the unprofiled capacity test; those remain work to do.
beinan commentedon Nov 13, 2026
Completed mixed-workload qualification of the public owner builds before and after #872 (
d78abc2bandf048d9f4). Both runs are fully audited and retired. Each used four owners/shards, six RF3 WAL nodes, two ingresses and four CPU cores per owner; 4,194,304 seed turns, then 600 seconds per worker of 256-row ingestion with 16-KiB bodies, 32-row annotation updates, point reads and status queries.These are observed differences, not an isolated code-speedup claim: the baseline's fourth worker started 45.50 seconds late, versus 0.09 seconds for the candidate. Common overlap was 554.97 versus 600.43 seconds. MB/s uses each whole run's earliest-start/latest-end interval; it is not restricted to common overlap. There is one run per case, with automatic placement and shared-infrastructure variation. Successful-request latency excludes skipped arrivals.
The candidate's 64 original ACK journals reconcile with 13,942,016 complete body/current-annotation values, including 1,317,568 distinct updated targets. All eight original/replacement views drained, every shard materialized in each observed full minute, and all six WAL nodes physically reclaimed disk. Final ID-index tails were zero on all four shards; the baseline retained one 2,048-row tail within its preregistered bound. There were 512 ACK-derived replacement point reads across both ingresses without resubmitting mutations. One failed object-delete attempt later recovered; no delete paths remained pending at drain. These replacement checks followed drain; repeated-update/freeze races and the separate in-flight fault qualification remain distinct evidence.
Offered update capacity remains unresolved: 89.28% of planned update arrivals were skipped. Candidate owners averaged 3.66–3.71 CPU cores with 45–49% throttled scheduling periods. A bounded next-window read-overlap experiment has passed 281 ingestion-library tests (18 ignored) and Clippy, but has no end-to-end performance result yet and has not been adopted.