Skip to content

Tracking: high-throughput Rust ingestion with a three-replica WAL #378

Open

Description

@beinan

Problem

Object-store WAL writes can put remote request latency and small-object overhead on the ingestion ACK path. A memory-only WAL removes that cost but cannot preserve acknowledged writes across process or node loss. We want an optional, high-throughput durable ingestion path implemented in Rust within lance-context, with minimal changes to Lance OSS.

This issue tracks incremental implementation and validation. Explicit Rust service paths now exist for durable append and mutable turn ingestion. They remain opt-in; workload-specific production readiness and large-scale qualification are still required.

Architecture and current visibility contract

The mutable-turn service path now separates immutable bodies from current narrow
annotations. The diagram shows that path for one shard; the lower-level immutable
append API has a different ACK/visibility contract, described below.

flowchart LR
    I["Ingress router: forward Arrow bytes"] --> O["Shard owner: ordered apply and point reads"]
    O -->|"Submit original request identity and payload"| L
    subgraph G["One Raft group: three durable replicas"]
        L["Leader + persistent WAL"] --> F1["Follower + persistent WAL"]
        L --> F2["Follower + persistent WAL"]
    end
    L -->|"Quorum-committed records"| O
    O --> S["Bounded active / frozen staging"]
    S -->|"Frozen batch"| P["Fenced Lance publisher"]
    P --> B["Blob: body and narrow-annotation snapshots"]
    B -->|"Compatible selected base"| O
    O -->|"Success after durable quorum and applied visibility"| I
Loading
  1. Explicit shard placement. The current mutable-turn path hashes the turn ID
    within its tenant/table scope to a configured bucket and owner. Forwarding
    ingresses share the validated route map and do not keep independent staging
    views. Each physical annotation shard has its own body/narrow table pair and
    publication/recovery progress. The immutable append contract retains its
    session-key placement. Automatic session placement, shard migration and
    rebalancing are not supplied by these explicit shard APIs.
  2. Durable and visible mutation ACKs. Three replicas persist the log on
    independent durable storage. A successful turn mutation requires majority
    persistence and ordered application to the owner's queryable state. A raw WAL
    append receipt alone is not that success. Raft supplies election, ordering and
    committed-prefix safety; it does not publish Lance manifests. The storage
    fault model tolerates one replica's loss, not loss of a durable quorum.
  3. Field updates and immediate point reads. Patches preserve unspecified
    fields; status/score have explicit keep/set/clear operations, and labels have
    set addition/removal. Applications merge against the ordered current value.
    Point reads resolve active, frozen, then the compatible Lance base, projecting
    narrow columns only and never scanning the WAL suffix. Owner fencing and
    recovery barriers protect cross-ingress reads. Materialized status filters
    currently have a different freshness contract: they do not include active or
    frozen updates. CAS and cross-shard transactions are not implemented here.
  4. Batch publication without losing later updates. Freeze one bounded
    generation at a source cut while later updates enter a new active generation.
    Existing annotation rows use native narrow-column rewrites; newly ingested
    bodies are appended separately and bound to the selected annotation snapshot.
    Confirm the exact fenced publication before installing its base and retiring
    only its frozen generation. Never delete active records by turn ID when an
    older batch finishes. Compaction is separate bounded maintenance, not a
    per-update rewrite of the immutable body.
  5. Recovery, exact retry and reclamation. Recovery uses the selected Lance
    state, persisted request outcomes and the required WAL suffix. Retained exact
    producer/sequence retries return their original result; an unknown outcome
    requires the same identity and bytes, not a newly identified mutation. A
    producer has at most one unresolved request, and the bounded latest-result
    contract does not retain arbitrary historical retries. Unknown publication
    results are reconciled against the original publication. Source GC also
    respects readers, recovery and replica/snapshot dependencies; a materialized
    cut alone does not permit deleting every preceding log record.
  6. Bounded resources and measured copying. Staging, caches, payloads, native
    workspaces, disk admission and replication queues have explicit budgets;
    pressure must leave publication and reclamation able to make progress.
    Replicas keep Arrow payloads opaque and ingress forwarding shares byte
    buffers. The current owner still validates by decoding on preparation and
    decodes again during ordered apply; this is not end-to-end zero-copy.
    Framing, integrity checks, native storage and kernel I/O also need measurement.

The existing immutable append API remains separate: its durable WAL ACK can
precede Lance visibility. Its readers and publication checkpoints follow that
binding's contract. Neither this issue nor an annotation-owner configuration
silently changes an existing immutable binding's semantics.

The implementation uses Rust raft-rs and raft-engine with Lance's existing
publication APIs. Useful references remain Flink's checkpoint/replay and bounded
flow control, and Apache Fluss's bucket ownership, batched replication and
asynchronous lake publication. Fluss uses leader/follower replication with ISR;
this design does not imply that Fluss uses Raft. Independent shard publication
does not create a global WAL position or a cross-shard atomic snapshot.

Incremental delivery

Use small PRs with Refs #378 in their descriptions. Keep this tracking issue open across individual implementation PRs; merge each only after its checks pass.

Validation before enabling the path

  • Verify acknowledged records after leader crashes, follower lag, snapshot transfer interruption and full process restarts using real Rust processes and durable storage.
  • Exercise ambiguous replies, stale publishers, concurrent manifest commits and retries; verify complete Lance payloads and source positions after recovery.
  • Run sustained ingestion through publication and repeated reclamation, including controlled publication pauses. Report ACK latency, published throughput, backlog, RSS, allocated disk space and physical write amplification separately.
  • Compare batching, serialization/deserialization, shared-buffer ownership, bounded parallel reads and replication-group placement using reproducible workloads. Short component benchmarks are not evidence of sustained end-to-end throughput.
  • Publish only synthetic, portable workloads and sanitized results. Do not include private deployment identifiers, storage locations, identity configuration, business workloads or raw operational artifacts.

Related: #333 tracks maintenance scheduling in the existing system; this issue focuses on the optional durable ingestion path and its correctness/performance contract.

References

Query visibility and current follow-ups

The optional session-read API now combines acknowledged WAL rows with published Lance rows. It captures committed WAL metadata, reads retained payloads, then opens a sufficiently recent Lance snapshot and removes overlap by immutable row ID. Readers must consume the complete stream and propagate errors. Different sessions and bounded distinct-record fetches can run concurrently; the WAL-to-Lance visibility handoff must remain ordered. This is not a claim of sustained query capacity or a migration of the existing production API.

Current turn annotations and mixed ingestion

The current-annotation path adds a stronger visibility contract to the original immutable-ingestion path described above: a successful mutation response waits for both durable quorum commit and ordered application. Point reads through another ingress resolve the owner's current annotation view. Publishing to Lance remains asynchronous.

Annotation applications now support explicit fresh physical shards, with one body/narrow table pair per owner and a complete ingress route map (#694). Configured routing and independent shard publication/recovery are implemented; the default remains one table and one virtual shard. Automatic session/client placement and migration, asynchronous status/label/score filtering indexes, and large-table index, storage and recovery costs remain open work. These explicit shard APIs do not establish saturation capacity or complete automatic scale-out.

Remaining qualification must run mixed new-body ingestion, point updates, cross-ingress reads, compaction and reclamation together; verify all accepted values and recovery progress; and report rejection rates, latency distributions, backlog/drain, process memory and storage costs. Separate-process and multi-host persistent-storage/blob tests must state their topology and fault boundary. Closed-loop load and small recovery fixtures must not be presented as fixed-arrival-rate capacity.

See the mutation contract, staging view, and owner runtime.

Activity

  1. beinan commented on Oct 19, 2026

    @beinan
    CollaboratorAuthor

    Progress update: the foundational PRs through #384 are merged. They provide the Lance 13 upgrade, allocation/ownership improvements, session placement, admission controls and immutable ingestion contracts. The distributed ingestion path remains experimental and is not enabled by these PRs.

    A synthetic Rust experiment compared 32 MiB and 64 MiB publication source caps in an ABBA sequence. Each case wrote 8,192 distinct 128 KiB payloads (1 GiB), using 32 writers, 8,000 session keys and three RF3 groups across three nodes, then materialized into Lance on object storage. Other admission, fetch and maintenance settings were held fixed within the cohort.

    Source cap / run Published payload MiB/s Encode-to-ACK p99 ms Publications
    32 MiB / A 36.78 3301 44
    64 MiB / A 38.55 2493 30
    64 MiB / B 41.95 3168 28
    32 MiB / B 38.42 3157 46

    All four cases passed complete payload verification, maintenance audits and process restart recovery: all nine group replicas restored their checkpoints, the Lance version/data stayed unchanged, and recovery produced zero additional publications.

    Pooled published throughput increased from 37.59 to 40.18 MiB/s, about 6.9%. Two short runs per setting are insufficient to establish sustained capacity or a dependable p99 improvement, so this does not justify a default change yet. These measurements exclude HTTP, catalog resolution, receipt authentication and fixture generation; publisher memory includes a resident 1 GiB synthetic fixture pool. They test process recovery on the original volumes, not volume loss or replacement.

    Next steps remain service integration, explicit retry/visibility semantics, replica-pressure coordination and longer bounded-retention tests. Future implementation PRs will continue to use Refs #378; this issue stays open. Only portable synthetic parameters and aggregate results are included here; operational artifacts remain outside the public repository.

  2. beinan commented on Oct 19, 2026

    @beinan
    CollaboratorAuthor

    Concurrent provisioning exposed a data-loss race that is addressed by #385: Lance checks an absent dataset before creation, then reopens its URI at commit time. A delayed creator can otherwise overwrite a second writer's newly appended data. The context-side guard permits only manifest version 1 during creation and preserves normal appends through the returned handle.

    The deterministic regression delays an actual empty storage-listing result while another creator creates and appends. Removing the guard makes the test fail with zero rows instead of one; restoring it preserves the row and version. A barrier at manifest authorization was too late to reproduce this race, so it was replaced before merging.

    A separate synthetic admission-window experiment stopped at its second case because it did not meet the predeclared maintenance-exposure gate. Both completed cases verified all 4 GiB of distinct payloads and passed process restart recovery on their original volumes, but the larger-window case completed only 12 maintenance rounds against a required minimum of 16. The remaining two ABBA cases were not launched. This is not evidence of an accepted sustained-throughput improvement, and no defaults changed.

    The next performance comparison needs longer workloads with bounded fixture generation and explicit generation/encoding/admission timing. Service integration, distributed replica disk credits and producer-state lifecycle remain open; the foundational PRs do not enable the distributed writer in production.

  3. beinan commented on Oct 19, 2026

    @beinan
    CollaboratorAuthor

    A four-case synthetic Rust ingestion/publication/recovery comparison has passed all predeclared gates. Each case ingested 98,304 distinct 128 KiB payloads (12 GiB), with 32 writers, 8,000 session keys and three RF3 groups across three nodes. We held the total node proposal budget at 4 GiB and varied only the producer ceiling in an ABBA sequence.

    Producer ceiling / run Published payload MiB/s Request-to-ACK p99 ms Successful maintenance rounds
    252 MiB / A 38.98 3040 98
    1020 MiB / A 47.37 2326 35
    1020 MiB / B 45.04 3582 35
    252 MiB / B 38.87 2633 99

    All cases verified every payload before and after a full WAL-process restart. All nine group replicas restored checkpoints; Lance versions stayed unchanged, with zero additional recovery publications and no maintenance failures. All WAL node process incarnations were explicitly stopped and their recorded identities checked.

    Pooled published throughput was 38.92 versus 46.18 MiB/s (+18.6%) in this cohort. Tail latency did not consistently improve. External samples showed peak per-node allocated WAL footprints of 426 and 1412 MiB, respectively; peak sampled acknowledged publication backlogs were 198 and 631 MiB. Two trials per setting are insufficient to establish production capacity or a reliable tail-latency gain; defaults remain unchanged.

    The streamed fixture generator retains no workload-wide payload pool and bounds concurrently generated raw payloads to 4 MiB for this workload. Generation is included in request/throughput timing. Maximum observed publisher process RSS high-water was approximately 335 MiB; RSS is not total Pod/cgroup memory, which also accounts for cache and other processes. External resource sampling actually ran roughly every 5–8 seconds, so sampled disk/RSS peaks are observations, not hard bounds. HTTP, authentication, catalog lookup and writer connection setup are outside timing; full readback and restart audits follow the timed publication drain. These results are separate from earlier precomputed-fixture or failed cohorts.

    The Arrow schema-contract extraction in #386 (Refs #378) is now merged. Its local Rust/Python checks and all five GitHub workflows, including all packaging platforms and final distribution validation, passed at the reviewed head before merge. Replica disk credits, producer-state lifecycle, incremental checkpoints and production service integration remain open work.

  4. beinan commented on Oct 19, 2026

    @beinan
    CollaboratorAuthor

    The local experimental Rust ingestion path now has a separate automatic materializer service. It discovers table generations through bounded catalog pages and claims work with expiring leases. Replicated publisher epochs and pinned Lance versions fence publication; the scheduling lease alone is not a storage fence. Active-table slots, a shared source-byte budget and a publication quantum bound work. There is no worker per session.

    Real-process tests with etcd and WAL nodes cover competing workers, catalog changes during discovery, prepared-publication recovery after owner loss, stale-owner rejection, complete payload comparisons, catalog/WAL restarts and interrupted table closure. A separate HTTP/materializer test verifies that writes receive durable ACKs while materialization is paused, a replacement automatically publishes the backlog, and full ingestion/materializer/WAL process restart plus request retries preserves data without duplicate publication. All 36 columns of 34 synthetic rows, each with a 128 KiB binary payload, are compared; graceful service exit also passes. This is local correctness coverage, not sustained HTTP performance or a released service feature.

    The object-storage experiment intended to compare serial and parallel file materialization stopped after its first serial case: two of 960 node-budget observations exceeded the predeclared 800 ms timeout. All 98,304 distinct 128 KiB payloads (12 GiB) passed initial readback, but the observation gate failed. The remaining three cases were never started, the workload was not repeated to obtain acceptance, and defaults remain unchanged. There is no accepted concurrency comparison.

    Diagnostic recovery of that original data passed: all nine group replicas restored checkpoints, every payload was verified, the Lance version stayed unchanged, and recovery made zero additional publications. The original failed gate remains recorded; the status-timeout cause is still under investigation.

    The materializer currently performs logical retirement. Physical checkpoint/compaction/collection orchestration, efficient discovery with many idle tables, replica disk credits, producer-state lifecycle, authorization/migration and sustained HTTP throughput remain open. Upcoming implementation PRs will continue to use Refs #378 and remain small; this issue stays open.

  5. beinan commented on Oct 19, 2026

    @beinan
    CollaboratorAuthor

    The local experimental WAL node now has opt-in automatic checkpoint, compaction and physical collection. One scheduler covers the groups sharing a node's engine/budget, including followers. It uses local durable applied cuts, recovers installed-but-uncompacted checkpoints after restart, and avoids rebuilding an unchanged group. Payload retirement still requires the existing replicated Lance publication proofs; scheduling alone does not authorize deletion.

    Real-process tests cover interrupted compaction, unchanged idle groups, a held checkpoint builder across observation timeouts, and full process loss. An existing checkpoint cancellation/recovery regression also passes. Scheduling retains the current job identity after an uncertain observation, while existing builder limits, byte reservations and file/read pins protect concurrent work.

    A separate local HTTP/materializer/node test writes 1,024 synthetic rows with 128 KiB binary payloads (128 MiB total), in four publication/drain windows. Eight concurrent upload futures bound generated bodies. Each node has a 32 MiB producer ceiling within a 256 MiB total proposal budget. The test never manually invokes checkpoint, compaction or collection. It compares all 36 columns, requires zero retained source payloads in the final compacted checkpoints and WAL footprints below 32 MiB, then restarts all HTTP/materializer/WAL processes and retries every request without changing the Lance version or duplicating rows.

    This validates finite local service/reclamation/recovery behavior. It is not a sustained-throughput measurement, a hard filesystem quota, a volume-loss test, or a result on remote blob storage. Full-copy checkpoint cost, producer-state growth, replica disk credits, idle-table discovery, authorization/migration and longer service workloads remain open. The earlier object-storage concurrency cohort remains failed and has not been rerun. The changes remain local experimental work; implementation PRs will continue to use Refs #378.

  6. beinan commented on Oct 19, 2026

    @beinan
    CollaboratorAuthor

    The experimental Rust HTTP ingestion path now has a bounded workload driver with separate prepare, ingest, verify and recovery modes. It streams distinct Arrow payloads, preserves producer/sequence/body across retries, refuses a second ingestion start, and checks complete Lance rows. Request-to-ACK timing includes generation and encoding; durable ACK and Lance visibility are measured separately.

    One new K8s/object-storage cohort completed 8,192 synthetic rows with 128 KiB binary payloads (1 GiB), eight writers, 31 sessions and three RF3 groups. All data became visible in 26.620 s: 38.467 MiB/s of binary payload. Request-to-ACK p50/p99/max were 16.167 / 324.863 / 1,121.279 ms. All 153 node-budget and 50 receipt observations passed their declared deadlines, and all 36 columns passed readback. This is one finite initial measurement, not a sustained-capacity or scaling result.

    All HTTP, materializer, three WAL and experimental metadata processes were then explicitly killed. New process incarnations reused the same durable directories. Retrying all 248 latest producer/session requests and verifying every row again left Lance version 38 unchanged. Node-owned checkpoint/compaction/collection ran throughout with zero recorded maintenance errors; final observed WAL footprints after draining were below the 256 MiB producer ceiling. Some ingestion-time footprints exceeded that ceiling, which is admission control rather than a hard disk quota. All experiment processes have been stopped.

    Each WAL Pod had a 4 CPU / 8 GiB limit on a separate host. HTTP, materializer, load generator and one experimental etcd shared another 4 CPU / 8 GiB Pod. Process resource samples were sparse, and shared cgroup memory reached about 7.08 GiB; without a before-run baseline this does not establish a memory bound or component attribution. The recovery test preserves volumes and does not establish volume-loss recovery or HA metadata.

    The earlier failed file-materialization concurrency cohort remains failed and was not rerun. Service-side ACK latency attribution, efficient idle-table discovery, replica disk credits, producer-state lifecycle, incremental checkpoints and production authorization/migration remain open. These service changes are still experimental local work; implementation PRs will continue to use Refs #378.

  7. 399 remaining items

  8. beinan commented on Nov 12, 2026

    @beinan
    CollaboratorAuthor

    A complete mixed-load comparison for #849 is now qualified. Both runs used four materialization workers/four shards, six WAL servers with three replicas per group, two ingresses, Lance 14.0.2, and the same workload/client/resource budgets. Each shard had eight ingestion lanes (256 rows/request), eight update lanes (32 rows/request), and sixteen point readers over 1,048,576 seed turns. The offered grids ran for 600 seconds; ingestion was closed loop and busy background lanes skipped arrivals.

    Metric, aggregate across four workers Before #849 With #849
    Actual combined window 600.972 s 602.014 s
    ACKed logical body throughput 151.20 MB/s 176.44 MB/s
    Update requests/s 37.47 48.40
    Updated rows/s 1,198.97 1,548.67
    Ingestion ACK p99 upper bound 2,809 ms 3,052 ms
    Update ACK p99 upper bound 2,750 ms 2,749 ms
    Successful point-read p99 upper bound 207 ms 266 ms
    Write refusals / attempts 263 / 44,445 756 / 55,216
    Busy-skipped update opportunities 361,483 / 384,000 354,865 / 384,000
    Busy-skipped point opportunities 120,824 / 480,000 190,104 / 480,000

    Observed body throughput increased 16.7% and update throughput 29.2%, but point-read p99 worsened 28.5%, fewer point reads completed, and write refusals increased. This is one sequential pair, not a repeatability result or maximum-capacity claim. Latency is per successful request and excludes skipped arrivals; it is never divided by batch width.

    The candidate passed all 10,677,504 complete body/current-annotation value checks, 512 post-drain owner-replacement point checks through both ingresses, all eight empty owner drain views, zero final index tails, per-minute materialization progress on every shard, and physical WAL reclamation on all six servers. All 73 original operations, 146 logs and 64 ACK journals were reconciled. One failed object-delete attempt was retained; all final pending queues were empty. Both experiments are closed, with original failures preserved.

    Original phase counters show annotation decoding around 1.13 ms/operation versus 487 ms patch apply-wait, 164 ms/shared base lookup, and 964 ms/narrow publication preparation. These are different/overlapping populations, not additive latency components or p99. Moving lookup outside ordered apply alone is not proof of a speedup. The evidence supports further work on base reads, publication and read/write contention before treating deserialization as the primary measured bottleneck.

    Next: a fresh sustained mixed workload adds status-filter readers while retaining ingestion, updates, point reads, indexing, compaction, GC and complete value verification. Status filters cover their reported materialized snapshot; point reads include staging. Fault qualification and the full high-throughput objective remain incomplete. Candidate service revision: f680c19; unchanged benchmark client: bf5acfefb3253eea9d51f29471e3445818872fb5.

  9. beinan commented on Nov 12, 2026

    @beinan
    CollaboratorAuthor

    Mixed-load qualification update after #864:

    A 600-second native Rust run used four workers/four independent shards, six RF3 WAL nodes and two ingresses. Each worker/WAL/ingress had 4 CPU and 8 GiB; the driver had 8 CPU and 8 GiB. The workload seeded 4,194,304 turns with 16 KiB pseudorandom bodies, then ran ingestion, 32-row annotation updates, point reads over the full seed set, and concurrent status queries. Per shard: eight ingestion writers, eight update writers, 16 point readers and two status readers. Elections were automatic.

    Result Measured value
    Logical body ingestion 157.690 MB/s
    Annotation updates 1413.885 rows/s
    Ingestion request p99 histogram upper bound 3467 ms
    32-row update request p99 histogram upper bound 3035 ms
    Point-read p99 histogram upper bound 263 ms
    Captured status-query p99 histogram upper bound 2292 ms

    Write attempts: 50,027, including 854 explicit refusals (1.707%). The update lanes offered 384,000 requests: 26,567 ran and 357,433 were skipped while the lane was busy. Point readers offered 480,000: 293,055 succeeded, 186,945 were skipped, with no refusals/errors. Status readers offered 48,000: 8,157 captured snapshots, 272 refusals and 39,571 skipped arrivals, with no query errors. Logical write ACK latency includes backoff after explicit not_submitted refusals. Point/status latency describes successful requests; refusal and skipped-arrival counts remain separate. Update p99 is per request, not divided by 32.

    Correctness and draining passed: 9,981,440 complete bodies/current annotations checked against original inputs and ACK journals; 512 point reads after owner replacement; all eight original/replacement staging views drained; every shard published in each observed full minute; all six WAL nodes showed physical space reclamation. All 65 operations and 130 original logs were reconciled. Internal Append counters separately recorded 49,173 accepts and 277 disk-budget rejections, with no leadership-unknown outcomes; peer exchange, maintenance and snapshot failures were zero. Transport frame drops were nonzero and are retained in diagnostics, not counted as business refusals.

    Status queries are materialized snapshots with visible progress. Their maximum observed uncovered tail during load was 1,368,832 physical rows; all four final tails reached zero after drain. Final catch-up does not establish low indexing lag during sustained load. Publication was configured for 192–256 MiB input batches with a 5-second maximum wait, and a 128 MiB compaction target; these are configured targets, not measured output file sizes.

    This qualifies this workload's correctness and reclamation, not a maximum-capacity or matched speedup claim. Owner replacement occurred after drain, so it does not prove in-flight crash recovery. Next are a kill during continuous reads/writes and WAL-node/storage fault qualification. Tail latency and missed arrivals still need substantial improvement; ordered application and cold narrow-value lookups remain the measured investigation focus. No throughput gain is claimed for the pending index-only lookup-reuse change.

  10. beinan commented on Nov 13, 2026

    @beinan
    CollaboratorAuthor

    Owner SIGKILL recovery qualification: 20,480 turns

    One owner/shard, three durable WAL replicas, two ingresses and eight point readers. After killing the original owner, a native check proved an acknowledged annotation update remained above the selected Lance cut and matched its original WAL payload. Recovery verified all 20,480 turns, including full bodies and current annotations, and 16 repeatedly updated targets. Four requests with unknown client outcomes were resolved as committed using their original identities; recovery submitted/replayed zero client mutations.

    Independent audit checked all 16 original processes, 32 sealed process logs and 97 original journals, including 64,620 acknowledged non-seed requests. Both replacement views drained, all services stopped, and test resources were retired with stores and logs preserved. An initial audit comparison failed on JSON representation of native serde defaults; the corrected comparison follows the exact native omission/null rules and retains every other check. The original failure is preserved.

    Availability remains a separate failed qualification. The original 600-second reader workload recorded 232,480 attempts: 205,693 successful responses passed the value oracle, and 26,787 returned HTTP 503. Each reader had a main observed first-failure-to-next-success interval of approximately 70 seconds, including the controller's evidence checks before starting the replacement. This is not automatic failover latency. The original continuous-reader phase failed because its after-ready fence capture exceeded the original workload window; later data recovery and trace inspection do not convert it into a pass. These results do not qualify WAL-node loss, durable-storage loss or capacity.

    Next is a matched owner Tokio thread-count comparison (2 to 4) using the existing qualified mixed-load source and unchanged resources/workload. PR #867 is prepared as a separate source-only candidate for a subsequent comparison; its 46 native harness tests and service builds passed. No new throughput gain is claimed.

  11. beinan commented on Nov 13, 2026

    @beinan
    CollaboratorAuthor

    Owner-process failure update: the original data and continuous point-read evidence now pass independent verification for a single-shard cloud/blob test (source f795922965d3521384259602ec00725afcc05c6f; one owner, RF3 WAL, two ingresses, eight readers, 600-second reader window).

    The owner was killed while an acknowledged annotation update remained above the selected Lance cut. Native verification while that owner was stopped matched the exact retained WAL record, ACK and deterministic input. Both ingresses subsequently returned that update after replacement readiness.

    • 16,220 acknowledged non-seed requests reconciled; four unknown outcomes resolved as 2 committed and 2 absent. No client mutation was replayed.
    • 20,480 complete turn bodies/current annotations and 16 repeatedly updated targets verified. Replacement and final staging views drained.
    • All 16 original process identities, 32 sealed process logs and 97 original journals verified.
    • 229,649 actual point-read attempts: 185,675 correct, 43,970 HTTP errors and 4 refusals; 10,351 of 240,000 offered arrivals skipped while busy. Successful results satisfy the per-field acknowledged bounds and monotonicity checks; no incorrect successful value was observed.

    This test includes an outage. Per-reader longest observed failure-to-next-success windows were 114.590–114.634 seconds, including the controller deliberately waiting for original writers to settle and collecting stopped-owner evidence before starting the replacement. These durations are not automatic failover latency or a zero-downtime result. The original load retains exit code 1.

    The original controller also exited 1 during post-load immutable ACK-log transfer. That failure is preserved. After all 16 original processes were terminal, a separate read-only supplement completed the exact missing bytes and remaining evidence, preserving every raw/compressed digest and the original reader, request, identity and full-value checks. It launched no service/workload and replayed no business request. Independent qualification passed on that complete original evidence; the original controller has not been relabeled successful.

    Scope remains owner-process loss with retained WAL replicas. A following case will keep one WAL replica unavailable through the workload, owner failure and recovery. Physical disk destruction and empty-disk same-ID rejoin remain unqualified. This result makes no throughput-improvement claim; the latest four-worker mixed-load result remains unchanged, and the owner-thread/PR #867 performance comparisons are still pending.

    Correction (2026-11-13): The negative unknown outcomes above were not proven definitely absent. The audit observed unchanged committed producer prefixes, but the original producer generations were still open; repeated quorum reads and stopped callers alone do not rule out a delayed request. The two positively identified commits remain confirmed; the two negative outcomes remain unknown. The full-value and ACK checks remain evidence for the observed committed prefixes, and the original failures are preserved. This narrows the qualification: it does not establish definitive resolution of every unknown request. No data loss has been demonstrated. Future recovery tests will use the existing durable producer-generation fence before making an absence claim; the product already retains unknown on a negative read. No original workload or retired store is being replayed.

  12. beinan commented on Nov 13, 2026

    @beinan
    CollaboratorAuthor

    The next fault case exposed a WAL client routing bug: after stopping one voter process and killing the owner, a fresh payload Fetch selected the unavailable voter and failed with Connection refused, despite the surviving quorum. PR #870 lets read-only Fetch use the existing configured-voter transport fallback within its original deadline and memory budget. It does not replay submitted mutations. All 44 client regressions passed, including two new real TLS/Raft cases; the PR is still awaiting the remaining HA CI.

    The original fault test remains failed: its stopped-owner check did not complete the exact unmaterialized ACK/WAL/input proof. A separate recovery audit using the fixed client has now completed and independently verified:

    • 16,035 acknowledged non-seed requests; four unknown outcomes resolved absent using the original request identities and quorum metadata.
    • All 20,480 complete turn bodies/current annotations, including 16 repeatedly updated targets.
    • Empty final staging; zero client mutations replayed.
    • All 17 native operations terminal, 96 original journals and 34 sealed process logs reconciled; fresh input/identity/idle checks completed before retiring the experiment. Original stores and failures are retained.

    This establishes recovery of that original data, not the missing continuous fault-read qualification, physical disk-loss/rejoin behavior, or a throughput improvement. Fetch(None) remains a local committed read, not a quorum absence proof.

    A fresh RF3 test retains the original 600-second split-field workload, eight continuous readers, and unavailable-voter-plus-owner-failure sequence. Subsequent owner-thread and #867 cache comparisons will use the same Fetch-fixed baseline. The previously reported four-worker throughput remains unchanged.

    Correction (2026-11-13): The negative unknown outcomes above were not proven definitely absent. The audit observed unchanged committed producer prefixes, but the original producer generations were still open; repeated quorum reads and stopped callers alone do not rule out a delayed request. The four negative outcomes therefore remain unknown. The full-value and ACK checks remain evidence for the observed committed prefixes, and the original failures are preserved. This narrows the qualification: it does not establish definitive resolution of every unknown request. No data loss has been demonstrated. Future recovery tests will use the existing durable producer-generation fence before making an absence claim; the product already retains unknown on a negative read. No original workload or retired store is being replayed.

  13. beinan commented on Nov 13, 2026

    @beinan
    CollaboratorAuthor

    A fresh single-shard recovery test now passes independent verification with one WAL voter unavailable and the owner killed during writes. This uses the Fetch fallback merged in #870; the earlier failed tests and their unresolved outcomes remain unchanged.

    The current WAL leader was stopped first. After both ingestion lanes advanced on the surviving quorum, the owner was killed. Before starting its replacement, the native audit proved an acknowledged annotation update at WAL position 8770 remained above Lance cut 8566, matching the original ACK and payload. Both ingresses subsequently returned that update.

    • 20,480 complete bodies/current annotations and 16 repeatedly updated targets verified; 8,599 acknowledged non-seed requests reconciled.
    • Ten original producer generations were durably closed before interpreting their final receipts. Four unknown outcomes resolved as two committed and two definitely absent. Recovery replayed zero client mutations.
    • All 206,067 successful point reads passed the value checks. The original 600-second workload also recorded 27,526 HTTP errors, four refusals and 6,403 skipped arrivals out of 240,000 offered reads. Its exit code remains 1.
    • Replacement and final staging/source/retained backlog drained to zero. All 16 original processes, 108 journals and 32 sealed logs passed independent verification; fresh identity/input/idle checks completed before retiring the test. Stores and evidence are retained.

    This qualifies the stated process-fault recovery scenario, including an acknowledged but unmaterialized update. It does not establish zero downtime, automatic failover latency, disk destruction/empty-disk rejoin or maximum capacity.

    No new throughput gain is claimed. Next are matched four-worker comparisons of two versus four owner Tokio threads, followed by #867's reuse of bounded lookup results across index-only publications, with unchanged workload and resource budgets and explicit p99/refusal/skipped-arrival/backlog accounting.

  14. beinan commented on Nov 13, 2026

    @beinan
    CollaboratorAuthor

    The fresh four-worker baseline for the owner-runtime and #867 comparisons has completed full verification following the Fetch fix in #870.

    Configuration: four workers/shards, six RF3 WAL nodes, two ingresses, 4 CPU / 8 GiB per worker, 4,194,304 seeded turns, and 600 seconds of mixed ingestion, annotation updates, point reads and status queries. Ingestion uses 256-row requests with 16-KiB logical bodies; updates use 32-row requests.

    Measurement Result
    Acknowledged logical ingestion 146.33 MB/s aggregate
    Annotation updates 1,325.39 rows/s aggregate
    Ingestion / update p99 histogram upper bound 4,013 / 3,675 ms
    Point / status p99 histogram upper bound 272 / 2,613 ms
    Writes refused 735 / 46,574 (1.58%)
    Scheduled update requests skipped while busy 359,119 / 384,000 (93.52%)
    Scheduled point reads skipped while busy 197,811 / 480,000 (41.21%)

    All 282,189 successful point reads passed with zero point errors/refusals. Status queries recorded 7,017 successes, 342 refusals, zero errors and 40,641 skipped arrivals out of 48,000 offered. Status filtering covers materialized snapshots; maximum sampled unindexed lag was 1,591,808 rows. Successful-request latency excludes skipped arrivals, so these results do not establish service capacity at the offered rate.

    Correctness and draining checks passed:

    • 9,559,552 complete bodies/current annotations verified, including 796,192 updated old turns; all 64 original ACK journals reconciled.
    • 512 cross-ingress replacement point checks passed with zero client mutations submitted. Replacement followed drain; the separate in-flight fault result remains here.
    • All eight original/replacement views drained, all four index tails reached zero, and every sampled shard-minute made materialization progress. Each of the six WAL nodes showed physical reclamation; no observed maintenance/snapshot failures.
    • All 65 native operations and 130 sealed process logs verified; the original controller, final analysis and fresh retirement checks completed successfully. The first offline phase-analysis attempt failed on an obsolete source-version assertion; that failure is preserved. Its source binding was corrected to the preregistered version after verifying identical ingestion code, with all metric and acceptance assertions unchanged. No workload was rerun.

    Owner CPU evidence supports the next comparison: each owner had a 4-core quota but averaged 2.16–2.18 cores, with two Tokio workers each using about 0.97 core and almost no quota throttling. The next case changes only owner Tokio workers from two to four, followed by #867's bounded lookup reuse. No throughput gain is established yet. Automatic leader placement, shared-cluster variation and physical volume capacity expansions with unchanged configured IOPS/bandwidth remain recorded comparison caveats.

  15. beinan commented on Nov 13, 2026

    @beinan
    CollaboratorAuthor

    The four-Tokio-thread comparison has completed, following the two-thread baseline. Four workers/shards, six RF3 WAL nodes and two ingresses are unchanged; each owner still has a 4-CPU quota. Product binaries, the driver, workload and application budgets are unchanged. The configured change is owner Tokio workers from two to four.

    The 600-second mixed load uses 4,194,304 materialized seed turns, 256-row ingestion requests with 16-KiB logical bodies, 32-row annotation requests, point reads and status queries.

    Measurement Two threads Four threads
    Acknowledged logical ingestion, aggregate 146.33 MB/s 241.58 MB/s
    Annotation updates, aggregate 1,325.39 rows/s 1,986.82 rows/s
    Ingestion / update p99 histogram upper bound 4,013 / 3,675 ms 2,036 / 1,879 ms
    Point / status p99 histogram upper bound 272 / 2,613 ms 199 / 1,282 ms
    Writes refused 735 / 46,574 965 / 72,888
    Scheduled updates skipped while lanes were busy 359,119 / 384,000 346,689 / 384,000
    Scheduled point reads skipped 197,811 / 480,000 104,647 / 480,000

    These are observed rates, not maximum capacity or isolated causal speedups. The four-thread run still skipped 90.28% of scheduled update requests. Successful-request latency excludes skipped arrivals. Automatic leader placement, shared-cluster variation and physical volume capacity expansion with unchanged configured IOPS/bandwidth remain comparison caveats. Logical body throughput is not physical blob bandwidth.

    All 375,353 successful point reads passed with zero point errors/refusals. Status queries had 11,812 successes, 92 refusals, zero errors and 36,096 / 48,000 skipped arrivals. Status filtering covers materialized snapshots. Maximum observed status unindexed physical rows increased to 2,586,112 from 1,591,808 in the baseline, so the ingestion improvement does not resolve index lag.

    Validation completed:

    • 13,054,976 complete bodies/current annotations verified, including 1,193,952 updated old turns; all 64 ACK journals reconciled.
    • All eight original/replacement views drained. All 40 shard-minute windows published new rows, and all six WAL nodes reclaimed physical storage. No observed maintenance/snapshot failures.
    • 512 cross-ingress replacement point reads passed with no client mutations submitted. Replacement followed drain; this is separate from the in-flight fault experiment.
    • All 65 native operations, 130 sealed logs and four status archives verified. The original controller and final retirement checks completed; original stores and blob tables are preserved.

    Final turn-ID index tails are [0, 0, 2304, 0]. Complete index catch-up is not established. The original offline status analyzer failed its zero-tail assertion. That executable check contradicted the preregistered and primary-analyzer criterion of each tail below 4096 rows. After observing the result, separate corrected status/review copies explicitly relaxed only the zero-tail gate to the registered threshold; all remaining analyses passed. Original scripts, criteria and failed outcome are preserved. No workload or index build was rerun. The earlier point-phase source-binding correction is also retained.

    Owners now average 3.71–3.80 process cores. About 53–62% of cgroup periods contain some throttling; that is not a stalled-wall-time percentage. Mean apply decode is 0.87 ms, versus 319.95 ms apply wait and 109.89 ms shared patch-base lookup, measured over overlapping populations rather than additive CPU phases. The next comparison tests #867's existing bounded lookup reuse across validated index-only publications, with four Tokio threads and the same workload. No lookup-reuse result is claimed yet.

  16. beinan commented on Nov 13, 2026

    @beinan
    CollaboratorAuthor

    Completed the three-case mixed ingestion/update comparison for #378. The largest observed throughput increase was between the two-thread and four-thread owner configurations. Annotation update capacity remains insufficient: approximately 90% of scheduled update requests were skipped before submission.

    All cases used 4 owners/workers, 4 shards, 6 WAL nodes with RF3, and 2 HTTP ingresses, with a 600-second timed load, 4,194,304 materialized seed turns, 256-row ingestion batches with deterministic pseudorandom 16-KiB bodies, and 32-row annotation update requests. Each owner had a 4-CPU quota. Lance was 14.0.2. The lookup change is #867; all cases include #870. No Lance dependency or workload change separates these cases.

    Case ACKed logical ingestion MB/s Annotation rows/s p99 ingestion / update / point / status (ms) Write refusals Scheduled updates skipped
    2 Tokio threads/owner 146.33 1325.39 4013 / 3675 / 272 / 2613 735/46,574 (1.58%) 93.52%
    4 Tokio threads/owner 241.58 1986.82 2036 / 1879 / 199 / 1282 965/72,888 (1.32%) 90.28%
    4 threads + lookup preservation 249.08 2066.89 1973 / 1779 / 185 / 969 1,167/75,640 (1.54%) 89.90%

    MB/s counts acknowledged logical body bytes, not physical blob or replica bandwidth. P99 values are histogram upper bounds for successful requests; they exclude skipped arrivals and refusals. Point latency is for annotation-only reads; status filtering covers materialized snapshots and does not include every unmaterialized update. Each case scheduled 384,000 update requests and 480,000 point reads. Lookup preservation skipped 345,200 update requests and 97,580 point reads; its 382,420 successful point reads reported 0 errors and 0 refusals. This does not establish sustainable capacity at the offered rate.

    Case Complete bodies/current annotations verified Final ID-index tails, by shard Maximum sampled status-unindexed rows Mean CPU cores per owner
    2 Tokio threads/owner 9,559,552 0, 0, 0, 0 1,591,808 2.159–2.175
    4 Tokio threads/owner 13,054,976 0, 0, 2304, 0 2,586,112 3.713–3.800
    4 threads + lookup preservation 13,326,592 0, 0, 0, 2048 1,391,104 3.702–3.820

    In every case, acknowledged new-row publication advanced in all 40 shard-minute windows, all six WAL nodes showed physical space reclamation, and all eight original/replacement owner views drained. Complete original ACK journals, terminal operation logs and full-value audits were checked before accepting these results; final retirement checks completed. Replacement point checks ran after drain and do not replace the separate in-flight fault experiment. ID-index acceptance was preregistered as fewer than 4,096 trailing rows per shard; nonzero tails are not full index catch-up, and status-index lag is a separate measurement. The original runtime-4/lookup-4 analysis failures from the contradictory zero-tail assertion are preserved; the corrected analysis changes only that executable assertion to the registered bound.

    Lookup preservation measured 3.10% higher ingestion and 4.03% higher annotation throughput than the four-thread case. This is one run per case, not a statistically established or isolated causal improvement. Leader placement, physical volume expansion and shared-cluster variation remain confounders. The owner CPU consumption supports collecting actual owner stacks next; overlapping apply/lookup phase durations are not CPU attribution. The separate profiling run will use fresh identities, and its instrumented timings will stay outside this comparison.

    Sources: product 367cefcd1390d34458afff2095d79c11a49ce245; lookup owner 6a0fb31c3ee7d8bd09f426b0a74d681c9b737e51; driver e9951746b8b4e80e38856610554eefe430ca2435.

  17. beinan commented on Nov 13, 2026

    @beinan
    CollaboratorAuthor

    CPU follow-up to the unprofiled four-worker comparison. This separate diagnostic used the same workload and topology: four owners/shards with four CPU cores each, six RF3 WAL nodes, and two ingresses. Its instrumented timings are not another throughput comparison.

    All four original-owner profiles are sealed, collected, and symbolized against their exact runtime executables/libraries. The original controller's copies exactly match the independently collected copies; all 127,431 samples are conserved.

    Owner Samples ConstantPageDecoder::take_row, flat SHA-256 AVX, flat
    A 41,986 14.0% 8.2%
    B 28,318 11.9% 11.6%
    C 29,138 11.1% 11.6%
    D 27,989 15.5% 10.0%

    About 84–90% of the constant-page leaf samples pass through annotation_staging::read_base: cold reads of materialized narrow annotations. About 89–92% of the SHA-256 leaf samples pass through WAL Batch::new. These are coarse sample shares, with uneven per-thread sampling and differing actual capture windows; they are not precise CPU proportions or a causal throughput breakdown. Stripped libc nearest-symbol names are not interpreted as evidence of DNS, timer, or allocator bottlenecks.

    Lance 14.0.2 walks preceding rows when seeking a constant page. Upstream Lance #10339 already removes that walk for pages without repetition levels and avoids encoder-only work on decoded arrays. A narrow backport is being built for native regression and point-scan comparison, alongside an explicit late-materialization scan option. Lists still require repetition-level walking. No performance gain from either candidate is established yet, and no WAL format change is proposed here.

    The diagnostic's full native audit verified 12,870,144 original-body/current-annotation values. All eight original/replacement owner views drained, and replacement checks passed 512 point reads with no submitted client mutations. Publication advanced in all 40 shard-minute windows; all six WAL nodes reclaimed disk space, and all four final ID-index tails reached zero. Final evidence qualification and idle-process retirement are complete. This does not extend the earlier failure/rejoin guarantees or resolve the roughly 90% scheduled-update skips in the unprofiled capacity test; those remain work to do.

  18. beinan commented on Nov 13, 2026

    @beinan
    CollaboratorAuthor

    Completed mixed-workload qualification of the public owner builds before and after #872 (d78abc2b and f048d9f4). Both runs are fully audited and retired. Each used four owners/shards, six RF3 WAL nodes, two ingresses and four CPU cores per owner; 4,194,304 seed turns, then 600 seconds per worker of 256-row ingestion with 16-KiB bodies, 32-row annotation updates, point reads and status queries.

    Measurement Before After
    Aggregate acknowledged logical body MB/s 225.37 265.89
    Completed annotation rows/s 1858.18 2193.58
    Successful ingestion ACK p99, ms 1875 1688
    Successful update ACK p99, ms 1722 1519
    Successful point-read p99, ms 189 181
    Successful status-query p99, ms 1042 958
    Write attempts refused 1.33% 1.09%
    Planned update arrivals skipped while lanes were busy 90.23% 89.28%

    These are observed differences, not an isolated code-speedup claim: the baseline's fourth worker started 45.50 seconds late, versus 0.09 seconds for the candidate. Common overlap was 554.97 versus 600.43 seconds. MB/s uses each whole run's earliest-start/latest-end interval; it is not restricted to common overlap. There is one run per case, with automatic placement and shared-infrastructure variation. Successful-request latency excludes skipped arrivals.

    The candidate's 64 original ACK journals reconcile with 13,942,016 complete body/current-annotation values, including 1,317,568 distinct updated targets. All eight original/replacement views drained, every shard materialized in each observed full minute, and all six WAL nodes physically reclaimed disk. Final ID-index tails were zero on all four shards; the baseline retained one 2,048-row tail within its preregistered bound. There were 512 ACK-derived replacement point reads across both ingresses without resubmitting mutations. One failed object-delete attempt later recovered; no delete paths remained pending at drain. These replacement checks followed drain; repeated-update/freeze races and the separate in-flight fault qualification remain distinct evidence.

    Offered update capacity remains unresolved: 89.28% of planned update arrivals were skipped. Candidate owners averaged 3.66–3.71 CPU cores with 45–49% throttled scheduling periods. A bounded next-window read-overlap experiment has passed 281 ingestion-library tests (18 ignored) and Clippy, but has no end-to-end performance result yet and has not been adopted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions