Repository navigation
Conversation
…checkpoint with its own template Tune rendered every qwen36-family checkpoint's prompts with one template that ends in an open <think>. The family holds checkpoints with different templates (Qwen3.6, Qwen3-Coder, Flint, Muse Spark, LFM3, the dense Qwen3), and Qwen3-Coder has no thinking mode: after that <think> it ended its turn on the first token, the engine reported no decode speed and the tune failed at its baseline (#2590). Measured on Qwen3-Coder-30B-A3B int4 with CUDA (RTX PRO 4000), the same message as a raw completion: tune's form 0 tokens, finish stop; the gateway's rendering 16 tokens of an answer. openai_server.tune_chat_prompt renders one user message as the gateway does for a chat request that sets nothing: the checkpoint's template (detect_chat_flavor) and its default reasoning (default_reasoning_effort and template_thinks, factored out of the request handler, which uses them). The qwen36 family says tune_renders_chat and `coli tune` uses it there; the other families keep their template. On the RTX PRO 4000: `coli tune` on Qwen3-Coder-30B-A3B fails at its baseline on dev and runs to the end here (baseline 2 requests, omp-14, omp-7, baseline kept). On Qwen3.6-35B-A3B Q4_K_M both run to the end; dev times a request that thinks (114.2 tok/s), this branch the one coli serve is sent by default, without (121.8 tok/s).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2590.
What was wrong
coli tunerendered every qwen36-family checkpoint's prompts with one template,<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n<think>\n. The family holds checkpoints with different templates (Qwen3.6, Qwen3-Coder, Flint, Muse Spark, LFM3, the dense Qwen3; the gateway tells them apart withdetect_chat_flavor). Qwen3-Coder has no thinking mode: after that open<think>it ends its turn on the first token, so the engine did one forward, generated nothing, reported 0 tok/s, and the tune stopped at its baseline. Serving worked because the gateway renders the Coder's own template.Measured on Qwen3-Coder-30B-A3B int4 (
Justvugg/Qwen3-Coder-30B-A3B-colibri-int4) with CUDA on an RTX PRO 4000, the same message as a raw completion, greedy:<think>\n)coli tuneon dev fails the same way on Linux as in the report on Windows:[chain] qwen36: 1 forwards ... (0 tokens), thentuning failed: baseline: engine reported no decode speed.Change
openai_server.tune_chat_prompt(model, text): one user message rendered as the gateway renders a chat request that sets nothing else, with the checkpoint's template and its default reasoning.default_reasoning_effort()(Qwen3.8 and Muse Spark xhigh, MiMo and LFM3 high,COLI_THINK=1, else none) andtemplate_thinks()(false for Qwen3-Coder and OLMoE). The handler behaves as before.FamilyDescriptor.tune_renders_chat, true for qwen36 only:coli tunerenders that family's prompts withtune_chat_prompt. The other families keep theirtune_prompt_template.tests/test_tune_chat_prompt.py: the Coder gets its template and no<think>(equal torender_chat_qwen3_coder); Qwen3.6 gets the pre-closed block a chat request gets;COLI_THINK=1opens it for Qwen3.6 and still not for the Coder; only qwen36 renders through the gateway.Measured
coli tuneon the RTX PRO 4000, CUDA build of dev's engine for both sides (the change touches no engine code):The two Qwen3.6 numbers are not the same request: dev timed a request that thinks, this branch the one
coli serveis sent by default, without thinking (the template's pre-closed block). Gateway, chat template and autotune tests: 543 pass.