Skip to content

Latest commit

 

History

History
252 lines (188 loc) · 18.8 KB

task3.md

File metadata and controls

252 lines (188 loc) · 18.8 KB

Task 3 · Hallucination turn prediction

← back to the README

Where did this conversation go off-track?

A guardrail watching a live conversation should step in as soon as the agent leaves its intended process, not after the damage is done. It should also stay silent while the agent behaves. That's exactly what Task 3 asks: at which turn does the conversation first go off-track? Or none, if it never does.

Input one dialogue (turn by turn, or complete; see the regimes below)
Output the first turn where the conversation goes off-track, given as its position in the conversation (turns counted from 0), or none
Ground truth for a fork, the turn where the deviation starts: the first assistant turn under the new instruction (the dataset's fork_turn_index); none for conformant dialogues
Metrics hallucination MAE and coverage on forks; MAE and false-alarm rate on conformant dialogues; a composite Score (0–100)
Units the 18,241 dialogues of the test split: 1,073 conformant dialogues and their 17,168 forks
Training data only the 1,061 conformant dialogues of the train split, for methods that learn from data (not its forks or labels); see the Task 3 split

Three regimes

regime what the method sees realism
Streaming turns arrive one at a time; it must decide after each turn, with no access to future turns, and with no workflow the most realistic, the hardest
Non-streaming the complete dialogue, with no workflow
Non-streaming + workflow the complete dialogue and its ground-truth workflow an oracle reference: the workflow is rarely available in practice

In every regime, a method may also learn from the conformant dialogues of the train split (see the Task 3 split).

Methods in the paper

Streaming detectors.

  1. Cluster the turns of the train split's conformant dialogues into workflow-step nodes with a sentence encoder, as in FlowKL.
  2. Fit an N-gram (N = 6) language model over the node sequences.
  3. While a new dialogue streams in, map each turn to a node and track how probable each transition is.

On top of this, two detectors decide when to flag:

  • IT (Improbable Transition) flags the first transition whose probability falls below a threshold.
  • SS (Surprisal Spike) flags the first sudden jump in surprisal.

Both thresholds are calibrated on the same train conformant dialogues, for a target false-alarm rate of about 5% on them. The +NC variants weight each transition by a node confidence, a softmax over the turn's similarity to the nearest centroids, which discounts turns that lie between clusters.

Non-streaming (LLMs). Zero-shot prompting of open-weight LLMs (Gemma4-31B, Qwen3.6-35B, GPT-OSS-20B, Mistral-Small-3.2-24B, Llama-3.3-70B) asks for the system turn where the dialogue first deviates, or none. In the workflow regime, the prompt also includes the ground-truth workflow.

Baselines. Always predict the first, the last, or a random turn.

Metrics

metric computed on definition better
Hallucination MAE forks |predicted − true turn|, in turns. An unflagged fork counts as predicted at n, just past its last turn (n = number of turns). Averaged per (task, type), then over tasks, then over the 4 types lower
Hallucination coverage forks % of forks flagged higher
Conformant MAE flagged conformant dialogues |predicted − n|: how early the false alarm fires. Averaged per task, over the tasks with at least one flag lower
Conformant coverage conformant dialogues % flagged: the false-alarm rate lower
Score all 100 × weighted geometric mean of four desirabilities in [0, 1]: 1 − MAE/anchor for both MAEs (weights 0.35 each), hallucination coverage, and 1 − false-alarm rate (0.15 each). The anchors are the worst credible MAEs, those of the trivial baselines: last turn on forks, first turn on conformant dialogues higher

Localization is the defining output of the task, so the MAEs carry the most weight. The coverage terms mainly rule out degenerate strategies: always flag, or never flag. A geometric mean means a method must do well on every term. Always flagging the first turn scores 0, no matter how good its hallucination MAE is.

Results

As reported in the paper, on the 18,241 dialogues of the Task 3 test split; MAEs are in turns. The streaming rows use the encoder with the best Score, LaBSE, and a 3-gram model, learned from the train split's conformant dialogues.

regime method CSC GR SSPC WIC MAE ↓ Cov. ↑ Conf. MAE ↓ False alarms ↓ Score ↑
baseline Random 6.31 7.92 5.74 6.44 6.60 100.00 7.50 100.00% 0.94
baseline First turn 5.64 5.64 5.64 5.64 5.64 100.00 14.12 100.00% 0.00
baseline Last turn 12.01 15.57 10.26 12.22 12.52 100.00 1.00 100.00% 0.00
streaming SS 4.21 1.56 1.30 1.68 2.19 93.24 6.33 12.02% 73.69
streaming SS + NC 4.13 1.61 1.33 1.71 2.20 93.56 6.58 15.47% 72.45
streaming IT 4.13 0.93 1.25 1.63 1.98 93.86 6.52 12.49% 73.60
streaming IT + NC 3.94 0.92 1.29 1.60 1.94 94.80 6.55 16.31% 73.23
non-streaming GPT-OSS-20B 10.10 6.70 4.77 4.34 6.48 72.38 4.40 20.04% 62.65
non-streaming Qwen3.6-35B 8.45 6.46 3.28 4.28 5.62 82.94 4.86 29.92% 64.56
non-streaming Gemma4-31B 10.13 7.12 2.26 3.88 5.85 73.33 4.63 9.79% 65.62
non-streaming Llama-3.3-70B 8.10 5.00 2.04 2.85 4.50 80.62 5.69 18.36% 67.09
non-streaming Mistral-Small-3.2-24B 8.51 5.56 2.88 3.98 5.23 78.20 4.60 12.12% 68.14
+ workflow GPT-OSS-20B 5.18 1.37 1.40 2.45 2.60 96.88 4.45 81.64% 62.31
+ workflow Qwen3.6-35B 4.96 1.14 1.36 1.85 2.33 96.71 4.57 53.96% 71.87
+ workflow Mistral-Small-3.2-24B 5.03 1.36 1.74 2.14 2.57 92.91 4.94 16.68% 76.38
+ workflow Gemma4-31B 5.30 0.51 1.31 1.28 2.10 93.03 4.41 30.48% 77.06
+ workflow Llama-3.3-70B 3.83 0.69 1.19 1.61 1.83 93.89 5.18 21.06% 77.09

Best streaming Score, by encoder (paper appendix): LaBSE 73.69 (SS) · D2F 73.28 (IT+NC) · Qwen3-Emb. 72.06 (IT) · EmbGemma 71.93 (IT+NC) · SBERT 70.61 (IT) · BGE-M3 66.20 (IT) · DSE 66.02 (IT).

What the numbers say.

  • Streaming detectors run turn by turn, with no future context and no ground-truth workflow, learning only from past conformant dialogues. They localize the fork within about 1.9–2.2 turns, flagging 93–95% of forks at a false-alarm rate of 12–16%. IT localizes better than SS; NC weighting sharpens IT further (MAE 1.94, the lowest of the methods that need no ground-truth workflow) and flags more forks, at the cost of more false alarms.
  • LLMs without the workflow localize far worse (MAE 4.5–6.5) and flag only 72–83% of forks, despite seeing the whole dialogue.
  • Given the workflow, LLMs localize about as well as streaming (MAE 1.83–2.60; Llama-3.3-70B localizes best of all methods, 1.83) but raise more false alarms on conformant dialogues: from 17% (Mistral-Small-3.2-24B) and 21% (Llama-3.3-70B) up to 82%. Llama-3.3-70B + workflow has the best overall Score (77.09), tied with Gemma4-31B (77.06).
  • A lightweight detector that learns the workflow from past dialogues thus localizes deviations as they happen nearly as well as the best LLM given the true workflow and the whole dialogue, with fewer false alarms (12–16%). No method is close to a perfect Score: real-time localization of procedural hallucination is open.

To compare a new method with the paper's, score the paper's prediction files in predictions/task3/ with evaluate.py. Every (detector, encoder) pair and every LLM has one.

Beyond the MAE: when the flag fires

The MAE counts a flag one turn early and one turn late alike, and it measures both against the fork turn, where the instruction is injected. A guardrail cares about the difference: an early flag fires before anything went wrong, and a late one after the user has seen the deviation. Two supplementary views refine the picture. Both are computed from the released prediction files, and neither is part of the Score or the leaderboard.

Signed detection delay

evaluate.py --delays also reports where each fork's flag falls relative to the fork turn: early (before it, on the still-conformant part of the dialogue), on time, or 1, 2, 3 or more turns late. On the test split:

method early on time +1 +2 +3 or more not flagged median delay
Random 32.1% 5.8% 5.3% 5.1% 51.6% 0.0% +3
First turn 75.0% 25.0% 0.0% 0.0% 0.0% 0.0% −4
Last turn 0.6% 4.5% 0.0% 6.6% 88.3% 0.0% +14
SS 3.6% 43.8% 13.1% 11.4% 21.4% 6.8% +0
SS + NC 4.9% 42.7% 13.1% 11.5% 21.4% 6.4% +0
IT 3.9% 48.2% 12.5% 10.6% 18.8% 6.1% +0
IT + NC 5.1% 48.2% 12.3% 10.8% 18.4% 5.2% +0
GPT-OSS-20B 3.6% 29.5% 0.0% 10.6% 28.7% 27.6% +2
Qwen3.6-35B 5.3% 31.9% 0.0% 13.5% 32.2% 17.1% +2
Gemma4-31B 2.0% 34.2% 0.0% 12.0% 25.2% 26.7% +2
Llama-3.3-70B 3.6% 42.2% 0.0% 13.1% 21.7% 19.4% +0
Mistral-Small-3.2-24B 4.4% 32.4% 0.0% 16.2% 25.2% 21.8% +2
GPT-OSS-20B + workflow 12.3% 49.5% 0.0% 15.3% 19.8% 3.1% +0
Qwen3.6-35B + workflow 7.0% 57.6% 0.0% 13.7% 18.4% 3.3% +0
Gemma4-31B + workflow 2.1% 64.1% 0.0% 12.6% 14.2% 7.0% +0
Llama-3.3-70B + workflow 2.2% 62.3% 0.0% 13.6% 15.8% 6.1% +0
Mistral-Small-3.2-24B + workflow 3.0% 55.0% 0.0% 15.7% 19.2% 7.1% +0

Delays are in turns of either speaker, as the MAE. The LLMs only flag assistant turns, so their +1 (a user turn) is always empty and +2 is the next assistant turn; the streaming detectors can flag either.

  • The streaming detectors rarely fire early (4–5% of forks) and flag 43–48% of the forks right at the fork turn.
  • Without the workflow, the LLMs tend to be one assistant turn late (median +2 for four of the five) and miss 17–28% of the forks. Given the workflow, they flag 50–64% on time.

Against the onset people saw

An injected behaviour can take a turn or two to become visible, for instance when the substituted workflow starts with the same steps as the intended one. In the human study, annotators marked the first assistant turn where the behaviour starts: 96% of these onsets fall at or after the fork turn. On the 123 forks of the test split that annotators confirmed and placed (the onset is after the fork turn in 41 of them), scripts/task3_human_onset.py compares each flag with both references:

method MAE vs fork turn MAE vs human onset before fork turn fork turn → onset at onset after onset not flagged
Random 6.62 5.77 25.2% 8.1% 6.5% 60.2% 0.0%
First turn 4.96 6.49 73.2% 20.3% 6.5% 0.0% 0.0%
Last turn 13.15 11.63 0.0% 0.0% 2.4% 97.6% 0.0%
SS 2.46 1.76 1.6% 8.1% 39.8% 46.3% 4.1%
SS + NC 2.39 1.74 1.6% 8.9% 39.8% 45.5% 4.1%
IT 2.32 1.65 1.6% 8.1% 43.9% 42.3% 4.1%
IT + NC 2.15 1.60 2.4% 8.9% 45.5% 39.0% 4.1%
GPT-OSS-20B 8.27 6.74 6.5% 1.6% 28.5% 38.2% 25.2%
Qwen3.6-35B 6.70 5.43 12.2% 4.1% 30.9% 36.6% 16.3%
Gemma4-31B 7.76 6.30 4.1% 1.6% 39.8% 27.6% 26.8%
Llama-3.3-70B 6.44 5.33 7.3% 4.1% 36.6% 34.1% 17.9%
Mistral-Small-3.2-24B 7.81 6.51 1.6% 2.4% 30.9% 38.2% 26.8%
GPT-OSS-20B + workflow 2.50 1.82 4.9% 9.8% 62.6% 21.1% 1.6%
Qwen3.6-35B + workflow 2.57 1.69 4.9% 8.1% 69.1% 13.0% 4.9%
Gemma4-31B + workflow 2.37 1.50 1.6% 8.9% 73.2% 11.4% 4.9%
Llama-3.3-70B + workflow 2.32 1.24 3.3% 5.7% 74.8% 12.2% 4.1%
Mistral-Small-3.2-24B + workflow 2.98 1.88 2.4% 6.5% 62.6% 19.5% 8.9%
  • Every method of the paper is closer to the onset people saw than to the fork turn, by 0.5–1.5 turns: part of the error measured against the fork turn is the deviation not being visible yet.
  • Few flags fall between the fork turn and the visible onset (2–10%): methods mostly wait for the deviation to show in the text, as a text-only method should.

The fork turn stays the reference of Task 3: it is known for every dialogue, while human onsets cover 123 test forks from five tasks, so read this table as indicative.

python evaluate.py --task 3 my_method.csv.gz --delays    # signed delay on the test split
python scripts/task3_human_onset.py my_method.csv.gz     # against the human onsets

Evaluate your method

Got a detector of your own? Write one row per dialogue of the Task 3 test split: 18,241 rows (rows for train dialogues are ignored, so a file covering all 36,278 is fine too). If it learns anything from data, it may learn only from the conformant dialogues of the train split, not from its forks or labels (see the Task 3 split). Each dialogue is identified by its dataset id (<dialog_id>-conformant or <dialog_id>-<TYPE>-k<k>). predicted_turn_index is the position of the first turn your method flags as off-track, counting the conversation's turns from 0, or empty if it flags nothing. It uses the same convention as the dataset's fork_turn_index: on Hugging Face, it indexes a row's turns list.

id,predicted_turn_index
30-conformant,
30-GR-k1,0
30-GR-k2,3
30-WIC-k3,
...
python evaluate.py --task 3 my_method.csv.gz              # the test split
python evaluate.py --task 3 my_method.csv.gz --split all  # every dialogue (not comparable with the leaderboard)

The Score's default weights are the paper's and the leaderboard's. If your setting weighs errors differently, for example a guardrail where a false alarm costs more than a late detection, pass your own with --weights, four numbers in this order:

position weight of default
1 hallucination MAE 0.35
2 conformant MAE 0.35
3 hallucination coverage 0.15
4 conformant coverage (the false-alarm rate) 0.15
python evaluate.py --task 3 my_method.csv.gz --weights 0.3,0.2,0.1,0.4   # false alarms weigh most

The four components are always printed next to the Score. Add --delays to also see when your flags fire relative to the fork turn (Beyond the MAE).

  • Any turn can be flagged, the assistant's or the user's: user turns are evidence too (a user who keeps asking for the forecast is a strong hint). A streaming method reports the turn at which it fires; in deployment, the guardrail would act on the next assistant turn.
  • Every test dialogue needs a row. A prediction must fall in 0…n−1, where n is the dialogue's number of turns. --allow-missing scores a subset, for development only.
  • Use the dialogue, not its id. The id tells conformant dialogues from forks and gives away the fork position, so don't let your method see it.
  • Declare your regime. The evaluator cannot check whether a method used future turns or the workflow. When reporting, say whether it is streaming, non-streaming, or non-streaming + workflow, and which conformant dialogues it was calibrated on.

When you're happy with the result, add it to the leaderboard. We'd love to see it there! 🏆

Reproducing the paper

Want to rerun the paper's Task 3 experiments? Everything is in Reproducing the paper → Task 3: the setup, the local and Slurm commands, all the options, and the script that builds the paper's tables and tests.

Optional: tuning with cross-validation on train

A suggestion for those who tune hyperparameters, or want an estimate of how well a configuration generalizes before looking at test.

If you choose among configurations (an encoder, an n-gram order, a threshold), one way to do it without the test split is k-fold cross-validation over the train sources:

  1. Split each task's train sources into k = 4 folds, keeping each source with its 16 forks (a fork replays its source before the fork turn).
  2. For each fold, learn from the conformant dialogues of the other three folds, and predict every dialogue of the held-out fold, conformant dialogues and forks, so that false alarms count too.
  3. Put the four folds' predictions in one file: evaluate.py --split train then gives the cross-validated Score.
  4. Learn the chosen configuration from all train conformant dialogues and report its test Score.

Why k = 4: the smallest task, ride_status, has 4 train sources, so k = 4 is the largest k that leaves no fold empty, while each model still learns from 75% of the training data (k = 2 would leave only two dialogues to learn from in the smallest tasks). The forks are used only to compare configurations, never to train a method. A large gap between the cross-validated and the test Score suggests the configuration does not generalize.

import csv, random
from collections import defaultdict

K = 4
train = defaultdict(list)                          # task -> its train source dialogues
for r in csv.DictReader(open("datasets/dialfork/task3_splits.csv")):
    if r["split"] == "train":
        train[r["task"]].append(int(r["dialog_id"]))

fold = {}                                          # dialog_id -> 0..K-1
for task, ids in sorted(train.items()):
    ids.sort()
    random.Random(f"dialfork-cv:{task}").shuffle(ids)
    for i, dialog_id in enumerate(ids):
        fold[dialog_id] = i % K                    # every task has >= 4 train sources: no empty fold

# fold f: learn from the conformant dialogues with fold[dialog_id] != f,
#         predict every dialogue (conformant and forks) with fold[dialog_id] == f

Questions, ideas, something not working? Open an issue. We're happy to help. 🔱