Skip to content

vulkan chain: the hybrid steps leave the events where a frame does not wait for the host's, and their barriers - #2588

Open
JustVugg wants to merge 3 commits into
devfrom
fix/vk-hybrid-event-barriers
Open

JustVugg wants to merge 3 commits into
devfrom
fix/vk-hybrid-event-barriers

Conversation

@JustVugg

@JustVugg JustVugg commented Nov 4, 2026 •

Copy link
Copy Markdown
Owner

Fixes #2583. The Vulkan chain's hybrid steps: with experts in the host pool, a decode step's (or a short verify's) pool experts are computed by the CPU inside the chain's frame while the device computes its own, the two meeting on Vulkan events (vk_dmoe.h, vkc_ev_signal / vkc_ev_wait in vk_chain.c). qwen36, qwen38 and OLMoE take them.

What was wrong

  • The frame did not have to wait for the host. A hybrid step's frame waits on an event the host sets once its experts are done, while that frame runs. The specification now asks for an event set before the frame that waits for it is submitted (VUID-vkSetEvent-event-09543; "the application must signal the event before submitting the command buffer"), so a driver may let the frame through, and the step's sum then reads the CPU's rows before they are written. That is [Bug]: qwen36 Vulkan degradation when all experts are resident. #2583: on a Radeon RX 9060 XT (AMD's own driver) the answers fell into loops after the barrier fixes below as before them. Its table fits a race the CPU loses more often the more host pool work a step has: every expert resident (2,592 pool pairs, 48 us a step in its log) breaks, a 5 GiB pool (642 pairs, 19 us a step) does not, COLI_VK_HYBRID_GATE=0 and an empty usage history break sooner, and NVIDIA and Lavapipe, which hold the frame, never show it.
  • Two barriers of the same steps. vkc_ev_wait reset the hazard tracker after a wait that orders only the host's writes, so the step's sum read the device's experts' rows with no barrier after the dispatch that wrote them; vkc_ev_signal's event stage mask held VK_PIPELINE_STAGE_HOST_BIT, which vkCmdSetEvent may not (VUID-vkCmdSetEvent-stageMask-01149).

Change

  • The wait is probed once, at the primary device's vkc_init, where no frame is open: a frame writes 1, waits for the event, writes 2; the host sees the 1, spins 20 ms and must still find the 1 before it sets the event (a frame that does not start in 2 s counts as one that does not wait). Where it fails the events stay off for the run, the chain says [VK] chain: this driver does not hold a frame at an event the host sets while it runs, and the hybrid steps run on the timelines (frames A/B/C, the existing COLI_VK_HYBRID_EVENTS=0 path), which the specification allows. A device that holds keeps the events and its speed. COLI_VK_TEST_EV_NOWAIT=1 (tests) drops the waits as such a driver would.
  • vkc_ev_signal: the barrier's second scope holds the device's stages beside the host, the event's stage mask the device's stages only. vkc_ev_wait: the tracker is kept, so the op after the wait that reads what the device wrote before it gets its barrier.
  • CI (tests/vulkan_engines.sh, the Vulkan engine matrix installs vulkan-validationlayers): sync_gate runs the qwen36 GGUF hybrid case under the validation layer's synchronization check and fails on any hazard or forbidden event or barrier use (SKIP without the layer); VUID-vkSetEvent-event-09543 is counted and said, not failed, since a driver that holds the frame is what the probe keeps the events for. A second gate runs the same case with COLI_VK_TEST_EV_NOWAIT=1: the probe must see it, the steps must be on the timelines, the tokens the CPU's.

Measured

RTX PRO 4000, Qwen3.6-35B-A3B int4 (q36c), #2583's placement (COLI_VK_TIER_GB=11.84: 7087 experts on the card, the other 3153 in a 5.25 GiB host pool, every expert resident), cooperative matrix off as on AMD's driver, 8 independent requests at temperature 0.7, 400 tokens each, #2583's unique-word ratio:

ratios tok/s
waits dropped, no probe (a driver that lets the frame through) 0.24 ("1 1 1 1 1 1 1 1"), 0.62, 0.88 ("snakedleeredledled..."), 0.60 ("He was a,,了。"), 0.61, 0.65 ("steeredred"), 0.69, 0.68 70.9 to 105.6
waits dropped, the probe 0.64 to 0.68 56.7 to 59.0
waits kept, the probe (passes, events kept) 0.64 to 0.73 64.9 to 68.1
waits kept, the barrier fixes alone (no probe) 0.64 to 0.73, the same texts 64.4 to 68.1
  • Lavapipe, the qwen36 GGUF hybrid fixture with the waits dropped: other tokens without the probe (95 110 93 50 277 ...), the CPU's with it (115 75 75 112 ...).
  • Validation layer 1.3.275 on Lavapipe, the same fixture: dev 16 read-after-write hazards (the sum's two bindings of d->y, every step) and 24 VUID-vkCmdSetEvent-stageMask-01149; this branch none. Layer 1.4.341 (the runner's): dev fails on 16 VUID-vkCmdSetEvent-stageMask-01149, this branch passes with 10 late host sets counted.
  • Earlier on the same card with the barrier fixes alone: COLI_VK_HYBRID_GATE=0 68.8 tok/s on dev, 68.7 here; the default gate 68.4 on both; the same text.

Not measured here: the Radeon itself (the reporter's card; the probe's line in its log says what it saw), and the timelines' speed against the device reading the host pool itself on that card (the hybrid gate measures that pair at run time).

@JustVugg
JustVugg force-pushed the fix/vk-hybrid-event-barriers branch from 7e8a917 to c6b1b85 Compare November 4, 2026 14:43
…ound them

vkc_ev_signal's barrier reached the host only and vkc_ev_wait's wait orders
only the host's writes, yet both reset the hazard tracker: a hybrid step's sum
read the experts' rows with no barrier (the validation layer's synchronization
check, a read-after-write hazard every step), and vkCmdSetEvent's stage mask
held HOST, which it may not. The signal's barrier now covers the device's
stages too and the event its stages only; the wait keeps the tracker.
sync_gate runs the qwen36 hybrid case under that check in CI.
… on them

The validation layer from 1.4 on (ubuntu-26.04's 1.4.341) reports
VUID-vkSetEvent-event-09543: the specification now wants an event the host
sets to be set before the command buffer that waits on it is submitted, and
a hybrid step sets it from the CPU while its frame runs, by design and on
dev as well. The gate lists it with the device's objects left at exit,
counts it in its OK line and fails on everything else: on Lavapipe with
1.4.341, dev fails on 16 VUID-vkCmdSetEvent-stageMask-01149 and this branch
passes with 10 late sets. A timeline semaphore between two submits is the
form the specification allows; that is a change of the chain's frames.
…t wait for the host's

A hybrid step's frame waits on an event the host sets once its pool experts
are done, while that frame runs. The specification now asks for an event set
before the frame that waits for it is submitted (VUID-vkSetEvent-event-09543,
"the application must signal the event before submitting the command
buffer"), so a driver may let the frame through: the step's sum then reads
the CPU's rows before they are written. #2583 kept falling into loops on a
Radeon RX 9060 XT (AMD's own driver) with the barriers fixed; an RTX PRO 4000
made to skip the wait (Qwen3.6-35B int4, every expert resident, the 8
stories at temperature 0.7) answered the same way: "1 1 1 1 ...", broken
words, 99.6 tok/s where healthy prose runs at 72.

The chain now probes the wait once, at the primary device's vkc_init where no
frame is open: a frame writes 1, waits for the event, writes 2; the host sees
the 1, spins 20 ms, and must still find the 1 before it sets the event (a
frame that does not start in 2 s counts as one that does not wait). Where the
probe fails the events stay off for the run, the chain says so, and the
hybrid steps run on the timelines (frames A/B/C, COLI_VK_HYBRID_EVENTS=0's),
which the specification allows. COLI_VK_TEST_EV_NOWAIT=1 (tests) drops the
waits as such a driver would.

Measured on the RTX PRO 4000, the issue's placement (7087 experts on the
card, 3153 in the host pool), cooperative matrix off as on AMD's driver:
  waits dropped, no probe: story 1 "1 1 1 1" (ratio 0.24, 99.6 tok/s), three
  more with broken words
  waits dropped, the probe: it sees them; the steps on the timelines, ratios
  0.64 to 0.68, 57 to 59 tok/s
  waits kept: the probe passes, the events stay, 65 to 68 tok/s and the same
  texts as without the probe
On Lavapipe, the hybrid fixture with the waits dropped gives other tokens
without the probe and the CPU's with it; vulkan_engines.sh holds the second.
@JustVugg JustVugg changed the title vulkan chain: the hybrid steps' events order the device's own work around them vulkan chain: the hybrid steps leave the events where a frame does not wait for the host's, and their barriers Nov 4, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant