Daily performance baselining for Opteryx.
Eight workloads, one c8g.4xlarge, daily. Corpora are built once
and stored, never regenerated. Results land in opteryx.benchmarks.telemetry
and on a GitHub Pages site.
We run and publish these benchmarks to track the engine's performance over time across a number of different benchmarks, in order to:
- identify regressions as they are introduced;
- confirm improvements to the engine; and
- demonstrate that we are not over-fitting to any single benchmark.
This is an engineering tracker for Opteryx's own development. We provide no mechanism as part of this tracking to compare or contrast Opteryx with other systems.
| Line | Format | Corpus | Queries | Iterations | Size |
|---|---|---|---|---|---|
| TPC-H SF1 | Parquet (rugo) · zstd | tpch_1_rugo |
22 | 5 | 0.3 GB |
| TPC-H SF10 | Parquet (rugo) · zstd | tpch_10_rugo |
22 | 5 | 3.1 GB |
| TPC-H SF100 | Parquet (rugo) · zstd | tpch_100_rugo |
22 | 3 | 32.4 GB |
| JOB | Parquet (rugo) · zstd | job_rugo |
113 | 3 | 1.5 GB |
| H2O (medium) | Parquet (rugo) · zstd | h2o_rugo |
15 | 3 | 7.9 GB |
| ClickBench partitioned | Parquet · zstd | hits_partitioned |
43 | 5 | 14.8 GB |
| ClickBench rewritten | Parquet (rugo) · zstd | hits_rugo |
43 | 5 | 8.8 GB |
| ClickBench | Skene · lz4 | hits_skene |
43 | 5 | 12.4 GB |
323 queries, ≈ 22 minutes per run. Stock CPython 3.14.
ClickBench rewritten is the canonical files rewritten through rugo's own writer at its defaults (zstd, 64k-row row groups, column-major blocks of 4) — the data the ClickBench entry "Opteryx (Parquet, rewritten)" runs on. It sits beside the canonical line, not in place of it.
ClickBench parquet is the canonical upstream corpus — the 100 objects from
datasets.clickhouse.com/hits_compatible/athena_partitioned/, used unmodified. This line exists as the unmodified reference for the rewritten and Skene lines; the Skene line is where our format is tested. No comparison with other systems is made.
TPC-H, JOB and H2O do not stipulate a storage format. None of the three upstream benchmarks ships one — they distribute rows (
dbgen, CSV from CWI, anddatagen.R), not files. The format is this harness's choice, so it is made once and made the same everywhere: rugo parquet, and it is not part of a line's name. They were Skene mirrors until 2026-10-21; the lines were renamed (tpch_sf1_skene→tpch_sf1, …) so each starts a clean baseline rather than being compared against a different format. ClickBench keeps its three formats, because there the format is the point.
The box installs the published wheel rather than building from source, so a number is attributable to an exact opteryx-core release rather than to "whatever main was". The version is recorded on every run.
Each benchmark's job is to make this run comparable to the previous run.
Results are compared against the trailing baseline (four non-suspect runs).
- Codec postures differ by design. The ClickBench Skene mirror is
lz4(the local benchmark posture); parquet corpora arezstd/snappy(the storage posture). ClickBench Parquet and ClickBench Skene therefore differ by codec as well as by format and are not a like-for-like format comparison. The codec travels on every result row for exactly this reason. - Bookend calibration. TPC-H SF1 runs first and last. A divergence over 10%
means the machine moved under the run; it is stamped
suspect, kept, and excluded from trend baselines. On a shared-tenancy instance this is worth more than any amount of pinning. - Corpus hashes. Every run records the manifest hash of each corpus it read. A trend break where the hash moved is attributed to the data, not the engine.
- Hard failures on silent degradation. A missing allocator preload, an unverified corpus, or a free-threaded interpreter each stop the run rather than quietly producing a number that cannot be compared to anything.
- Regressions are a query more than 15% over the median of the trailing
four non-suspect runs, confirmed on two consecutive runs — or 25% on a single
run. A status transition
ok → error/timeoutis reported immediately and needs no statistics.
This repository's own code (the harness, infrastructure, site and scripts) is
licensed under the Apache License 2.0, see LICENSE. That licence
does not cover the third-party benchmark material: the queries under
queries/ and the corpora derived from the benchmarks' data keep the terms of
their upstream sources, as set out below and in a NOTICE.md beside each set
of queries. All of it is used only for the purpose set out under
Purpose, and the corpora are held in private storage, not
published for download.
| Benchmark | Source | Terms | Notice |
|---|---|---|---|
| ClickBench | ClickHouse, Inc. | CC BY-NC-SA 4.0 | queries/clickbench/NOTICE.md |
| TPC-H | Transaction Processing Performance Council | TPC fair use policy | queries/tpch/NOTICE.md |
| JOB | Leis et al., VLDB 2015; data from IMDb | IMDb non-commercial terms | queries/job/NOTICE.md |
| H2O | H2O.ai db-benchmark | MPL-2.0 | queries/h2o/NOTICE.md |
- ClickBench. The queries have
hitsreplaced by a{DATASET}placeholder and some expressions adapted to Opteryx's SQL dialect; the corpora (hits_rugo,hits_skene) are the upstreamhitsdata rewritten into other storage formats. These materials stay under CC BY-NC-SA 4.0. - TPC-H. These are not TPC-H results. They are derived from TPC-H, are not audited or reviewed by the TPC, and are not comparable to published TPC-H results. "TPC-H" identifies the workload the queries derive from; it does not claim a TPC result.
- JOB. The data is derived from IMDb. Information courtesy of IMDb (https://www.imdb.com). Used with permission.
- H2O. The
.sqlfiles derived from db-benchmark remain under MPL-2.0.
Building, running, and operating this harness — the AWS/GitHub pipeline, corpus builds, local runs, and infrastructure setup — is covered in docs/DEVELOPMENT.md.