Repository navigation
Conversation
JustVugg
force-pushed
the
fix/vk-hybrid-event-barriers
branch
from
November 4, 2026 14:43
7e8a917 to
c6b1b85
Compare
…ound them vkc_ev_signal's barrier reached the host only and vkc_ev_wait's wait orders only the host's writes, yet both reset the hazard tracker: a hybrid step's sum read the experts' rows with no barrier (the validation layer's synchronization check, a read-after-write hazard every step), and vkCmdSetEvent's stage mask held HOST, which it may not. The signal's barrier now covers the device's stages too and the event its stages only; the wait keeps the tracker. sync_gate runs the qwen36 hybrid case under that check in CI.
JustVugg
force-pushed
the
fix/vk-hybrid-event-barriers
branch
from
November 4, 2026 17:09
c6b1b85 to
56c23b1
Compare
… on them The validation layer from 1.4 on (ubuntu-26.04's 1.4.341) reports VUID-vkSetEvent-event-09543: the specification now wants an event the host sets to be set before the command buffer that waits on it is submitted, and a hybrid step sets it from the CPU while its frame runs, by design and on dev as well. The gate lists it with the device's objects left at exit, counts it in its OK line and fails on everything else: on Lavapipe with 1.4.341, dev fails on 16 VUID-vkCmdSetEvent-stageMask-01149 and this branch passes with 10 late sets. A timeline semaphore between two submits is the form the specification allows; that is a change of the chain's frames.
…t wait for the host's A hybrid step's frame waits on an event the host sets once its pool experts are done, while that frame runs. The specification now asks for an event set before the frame that waits for it is submitted (VUID-vkSetEvent-event-09543, "the application must signal the event before submitting the command buffer"), so a driver may let the frame through: the step's sum then reads the CPU's rows before they are written. #2583 kept falling into loops on a Radeon RX 9060 XT (AMD's own driver) with the barriers fixed; an RTX PRO 4000 made to skip the wait (Qwen3.6-35B int4, every expert resident, the 8 stories at temperature 0.7) answered the same way: "1 1 1 1 ...", broken words, 99.6 tok/s where healthy prose runs at 72. The chain now probes the wait once, at the primary device's vkc_init where no frame is open: a frame writes 1, waits for the event, writes 2; the host sees the 1, spins 20 ms, and must still find the 1 before it sets the event (a frame that does not start in 2 s counts as one that does not wait). Where the probe fails the events stay off for the run, the chain says so, and the hybrid steps run on the timelines (frames A/B/C, COLI_VK_HYBRID_EVENTS=0's), which the specification allows. COLI_VK_TEST_EV_NOWAIT=1 (tests) drops the waits as such a driver would. Measured on the RTX PRO 4000, the issue's placement (7087 experts on the card, 3153 in the host pool), cooperative matrix off as on AMD's driver: waits dropped, no probe: story 1 "1 1 1 1" (ratio 0.24, 99.6 tok/s), three more with broken words waits dropped, the probe: it sees them; the steps on the timelines, ratios 0.64 to 0.68, 57 to 59 tok/s waits kept: the probe passes, the events stay, 65 to 68 tok/s and the same texts as without the probe On Lavapipe, the hybrid fixture with the waits dropped gives other tokens without the probe and the CPU's with it; vulkan_engines.sh holds the second.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2583. The Vulkan chain's hybrid steps: with experts in the host pool, a decode step's (or a short verify's) pool experts are computed by the CPU inside the chain's frame while the device computes its own, the two meeting on Vulkan events (
vk_dmoe.h,vkc_ev_signal/vkc_ev_waitinvk_chain.c). qwen36, qwen38 and OLMoE take them.What was wrong
COLI_VK_HYBRID_GATE=0and an empty usage history break sooner, and NVIDIA and Lavapipe, which hold the frame, never show it.vkc_ev_waitreset the hazard tracker after a wait that orders only the host's writes, so the step's sum read the device's experts' rows with no barrier after the dispatch that wrote them;vkc_ev_signal's event stage mask heldVK_PIPELINE_STAGE_HOST_BIT, whichvkCmdSetEventmay not (VUID-vkCmdSetEvent-stageMask-01149).Change
vkc_init, where no frame is open: a frame writes 1, waits for the event, writes 2; the host sees the 1, spins 20 ms and must still find the 1 before it sets the event (a frame that does not start in 2 s counts as one that does not wait). Where it fails the events stay off for the run, the chain says[VK] chain: this driver does not hold a frame at an event the host sets while it runs, and the hybrid steps run on the timelines (frames A/B/C, the existingCOLI_VK_HYBRID_EVENTS=0path), which the specification allows. A device that holds keeps the events and its speed.COLI_VK_TEST_EV_NOWAIT=1(tests) drops the waits as such a driver would.vkc_ev_signal: the barrier's second scope holds the device's stages beside the host, the event's stage mask the device's stages only.vkc_ev_wait: the tracker is kept, so the op after the wait that reads what the device wrote before it gets its barrier.tests/vulkan_engines.sh, the Vulkan engine matrix installsvulkan-validationlayers):sync_gateruns the qwen36 GGUF hybrid case under the validation layer's synchronization check and fails on any hazard or forbidden event or barrier use (SKIP without the layer); VUID-vkSetEvent-event-09543 is counted and said, not failed, since a driver that holds the frame is what the probe keeps the events for. A second gate runs the same case withCOLI_VK_TEST_EV_NOWAIT=1: the probe must see it, the steps must be on the timelines, the tokens the CPU's.Measured
RTX PRO 4000, Qwen3.6-35B-A3B int4 (
q36c), #2583's placement (COLI_VK_TIER_GB=11.84: 7087 experts on the card, the other 3153 in a 5.25 GiB host pool, every expert resident), cooperative matrix off as on AMD's driver, 8 independent requests at temperature 0.7, 400 tokens each, #2583's unique-word ratio:d->y, every step) and 24 VUID-vkCmdSetEvent-stageMask-01149; this branch none. Layer 1.4.341 (the runner's): dev fails on 16 VUID-vkCmdSetEvent-stageMask-01149, this branch passes with 10 late host sets counted.COLI_VK_HYBRID_GATE=068.8 tok/s on dev, 68.7 here; the default gate 68.4 on both; the same text.Not measured here: the Radeon itself (the reporter's card; the probe's line in its log says what it saw), and the timelines' speed against the device reading the host pool itself on that card (the hybrid gate measures that pair at run time).