[alexqyle]

What this changes

A partitioning compaction builds one series set per source block and merges them all in a single flat merge. Advancing that merge reads the winning block's series entry off disk, and every one of those reads is issued from the one goroutine that advances the merge heap. The disk therefore sees a queue depth near one however much IOPS the volume has provisioned, and a compaction over a large number of blocks ends up bound by read latency rather than by disk capacity. Provisioning more IOPS does not help it.

This adds -compactor.block-merge-fan-in, which groups the per-block series sets into contiguous subtrees before the final merge. Each subtree is wrapped in a backgrounChunkSeriesSet, so it advances in its own goroutine and the reads behind the subtrees overlap.

Disabled by default (0), which keeps the existing single flat merge unchanged.

Benchmark

BenchmarkMergeFanIn injects latency per set advance to model the index read that backs a merge advance. 64 sets, 1024 series replicated 3 ways, so 3136 advances per op.

go test -run XXX -bench BenchmarkMergeFanIn -benchtime 5x ./pkg/compactor/

latency	fan-in	merge goroutines	ns/op	ns/advance	vs flat
0	0 (off)	1	7,519,760	2,398	—
0	4	16	8,696,905	2,773	0.86x
0	8	8	9,024,411	2,878	0.83x
0	16	4	8,390,016	2,675	0.90x
1ms	0 (off)	1	3,451,540,907	1,100,619	—
1ms	16	4	914,257,427	291,536	3.8x
1ms	8	8	496,123,534	158,203	6.9x
1ms	4	16	287,013,931	91,522	12.0x

goos: linux, goarch: amd64, cpu: AMD EPYC 7R13, 8 cores.

The ns/advance column is the mechanism. A flat merge pays the full 1.10 ms per advance because every read queues behind one goroutine; 16 goroutines bring that to 91.5 us without changing the read volume at all. Speedup tracks roughly 0.75x the goroutine count, and the shortfall is the root merge's own serial handoff.

Note the fan-in is inverted against goroutine count: a smaller fan-in produces more subtrees, so fan_in=4 over 64 sets gives 16 goroutines.

With no read latency to hide, the change costs 10-17%, which is why it defaults to off. A workload whose blocks fit in page cache should leave it off. The flag is worth setting when a compaction merges many blocks off a volume with real read latency.

Two measurement notes baked into the benchmark, since both bit me while writing it:

• time.Sleep cannot resolve latencies below roughly 100 us, and pinning to one CPU inflates every sleep. Either makes all rows collapse onto the timer floor and look identical. The benchmark reports ns/advance so a row whose sleeps never resolved to the requested latency is visible rather than silently misleading.
• merge_goroutines reports 1 for the flat case. A flat merge has 64 inputs but one goroutine advancing them, and the concurrency is what matters.

Correctness

TestMergeSubtreesMatchesFlatMerge asserts the grouped merge yields identical labels and samples to the flat merge, across fan-ins 2, 3, 4 and 8 over 9 sets, so both the evenly-divided case and the short trailing group are covered.

It compares samples flattened across chunk boundaries on purpose: a merge tree may lay the same data out in different chunks than a flat merge, which changes the output block's chunk count but not its data. The fixture gives a given timestamp the same value in every replica, which is how replicas present a series; with conflicting values at one timestamp, flat and tree order would legitimately differ and asserting equality would be testing undefined behaviour.

TestMergeSubtreesDisabled pins that fan-in 0, 1 and negative return the sets untouched. TestMergeSubtreesGrouping pins the subtree counts.

Verified with a sabotage check: changing the grouping loop bound so it drops the trailing partial group fails exactly the expected subtests, while leaving fan_in=3 passing since 9 divides by 3.

Memory stays comparable to the flat merge. The existing 1000-series lookahead is split across subtrees (1000/subtrees, floor 8) rather than given to each one, so the buffered series count does not grow with the subtree count.

Notes for review

• CompactionLifecycleCallbackFactory gained a blockMergeFanIn int parameter. Existing tests needed no change, since they construct the callback with field names and assign the factory by type.
• NewBackgroundChunkSeriesSet keeps its signature and delegates to a new NewBackgroundChunkSeriesSetWithBuffer, so current callers are unaffected.
• docs/ and schemas/cortex-config-schema.json are regenerated output from the doc-generator, for make check-doc.
• The CHANGELOG entry carries #TODO for the PR number; I'll follow up with the real number once this is open.
• golangci-lint was not available in my environment, so lint coverage here is go vet and gofmt -l. The full ./pkg/compactor/ suite passes.

[yeya24]

Is it possible for this to work for even non partitioning compaction?

[alexqyle]

We can. The original code is mainly copied from Prometheus. We can either change Prometheus or create new populator in cortex for non partitioning