Skip to content

coli tune: the qwen36 family's prompts rendered by the gateway, each checkpoint with its own template - #2594

Open
JustVugg wants to merge 1 commit into
devfrom
fix/tune-gateway-prompt
Open

JustVugg wants to merge 1 commit into
devfrom
fix/tune-gateway-prompt

Conversation

@JustVugg

@JustVugg JustVugg commented Nov 4, 2026

Copy link
Copy Markdown
Owner

Fixes #2590.

What was wrong

coli tune rendered every qwen36-family checkpoint's prompts with one template, <|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n<think>\n. The family holds checkpoints with different templates (Qwen3.6, Qwen3-Coder, Flint, Muse Spark, LFM3, the dense Qwen3; the gateway tells them apart with detect_chat_flavor). Qwen3-Coder has no thinking mode: after that open <think> it ends its turn on the first token, so the engine did one forward, generated nothing, reported 0 tok/s, and the tune stopped at its baseline. Serving worked because the gateway renders the Coder's own template.

Measured on Qwen3-Coder-30B-A3B int4 (Justvugg/Qwen3-Coder-30B-A3B-colibri-int4) with CUDA on an RTX PRO 4000, the same message as a raw completion, greedy:

prompt tokens finish
tune's form (ends in <think>\n) 0 stop
the gateway's rendering (Qwen3-Coder template) 16 length, "Here's a Python function that checks whether a string is a palindrome: ..."

coli tune on dev fails the same way on Linux as in the report on Windows: [chain] qwen36: 1 forwards ... (0 tokens), then tuning failed: baseline: engine reported no decode speed.

Change

  • openai_server.tune_chat_prompt(model, text): one user message rendered as the gateway renders a chat request that sets nothing else, with the checkpoint's template and its default reasoning.
  • The request handler's defaults are factored out so both paths share them: default_reasoning_effort() (Qwen3.8 and Muse Spark xhigh, MiMo and LFM3 high, COLI_THINK=1, else none) and template_thinks() (false for Qwen3-Coder and OLMoE). The handler behaves as before.
  • FamilyDescriptor.tune_renders_chat, true for qwen36 only: coli tune renders that family's prompts with tune_chat_prompt. The other families keep their tune_prompt_template.
  • tests/test_tune_chat_prompt.py: the Coder gets its template and no <think> (equal to render_chat_qwen3_coder); Qwen3.6 gets the pre-closed block a chat request gets; COLI_THINK=1 opens it for Qwen3.6 and still not for the Coder; only qwen36 renders through the gateway.

Measured

coli tune on the RTX PRO 4000, CUDA build of dev's engine for both sides (the change touches no engine code):

model dev this branch
Qwen3-Coder-30B-A3B int4 fails at the baseline (0 tokens) runs to the end: baseline, omp-14, omp-7, baseline kept
Qwen3.6-35B-A3B Q4_K_M runs to the end, baseline 114.2 tok/s runs to the end, baseline 121.8 tok/s

The two Qwen3.6 numbers are not the same request: dev timed a request that thinks, this branch the one coli serve is sent by default, without thinking (the template's pre-closed block). Gateway, chat template and autotune tests: 543 pass.

…checkpoint with its own template

Tune rendered every qwen36-family checkpoint's prompts with one template that
ends in an open <think>. The family holds checkpoints with different templates
(Qwen3.6, Qwen3-Coder, Flint, Muse Spark, LFM3, the dense Qwen3), and
Qwen3-Coder has no thinking mode: after that <think> it ended its turn on the
first token, the engine reported no decode speed and the tune failed at its
baseline (#2590). Measured on Qwen3-Coder-30B-A3B int4 with CUDA (RTX PRO
4000), the same message as a raw completion: tune's form 0 tokens, finish
stop; the gateway's rendering 16 tokens of an answer.

openai_server.tune_chat_prompt renders one user message as the gateway does
for a chat request that sets nothing: the checkpoint's template
(detect_chat_flavor) and its default reasoning (default_reasoning_effort and
template_thinks, factored out of the request handler, which uses them). The
qwen36 family says tune_renders_chat and `coli tune` uses it there; the other
families keep their template.

On the RTX PRO 4000: `coli tune` on Qwen3-Coder-30B-A3B fails at its
baseline on dev and runs to the end here (baseline 2 requests, omp-14,
omp-7, baseline kept). On Qwen3.6-35B-A3B Q4_K_M both run to the end; dev
times a request that thinks (114.2 tok/s), this branch the one coli serve is
sent by default, without (121.8 tok/s).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant