JCM Route Benchmark Corpus

JCM is a neutral shorthand for this dataset's scope. It is not a product name, an abbreviation used by the JCodeMunch project, or a claim of affiliation with it.

Supporting material developed after jcodemunch-mcp issue #398. The issue addressed adaptive data paths for large repositories; this corpus supports a narrower follow-up question: when an agent turns a naturalistic request into a JCodeMunch retrieval step, does it select an appropriate first action from the available catalog? These are different routing layers: the corpus does not evaluate issue #398's internal adaptive data-path router.

The intended audience is the JCodeMunch maintainer (jgravelle) and contributors evaluating prompt-to-action routing, tool discovery, or catalog-search changes. They may adopt the data, disagree with individual labels, extend the coverage, or run the same prompts through a different normalization and routing pipeline.

Dataset version 0.1.0 is a published seed, not an official JCodeMunch benchmark and not endorsed by that project. The labels are agent-curated and human-reviewed, so treat them as contestable. Disagreement on individual labels is expected and useful.

This is a dataset, not a harness. There is no runner, no scorer, and no results. Presenting prompts to a router, capturing its choice, and scoring it against the labels is work you would build yourself. The validate.py and tools/ scripts here check the integrity and provenance of the data; they do not run the benchmark.

What is included

• 197 naturalistic prompts retained from a larger private request corpus.
• A frozen 91-action JCodeMunch call-metadata catalog from version 1.108.241.
• One strict primary action and up to two ranked alternatives per prompt.
• 22 higher-suitability pilot rows with router_test_score >= 90.
• A deterministic data validator, its regression tests, and a catalog reproduction tool.

The retained prompt_text values are preserved exactly as written instead of being rewritten into polished benchmark language. Mid-conversation fragments, frustration, typos, emphatic capitalization, and profanity are intentional test conditions: the benchmark measures whether routing survives how people actually write while working through a problem.

The full private corpus, excluded rows, partial rows, source sessions, and private review ledger are not included.

Start here

Windows:

py -3 validate.py
py -3 -m unittest discover -s tests -v

Linux or macOS:

python3 validate.py
python3 -m unittest discover -s tests -v

A validator exit code of 0 means the shipped CSV bytes match the manifest, the schemas and identifiers satisfy the documented contracts, the structured and flat labels agree, and no configured hard privacy marker was found. It does not prove that a gold label is semantically correct, that every contextual name is safe to publish, or that the frozen catalog still matches another installed JCodeMunch version.

How the cases and labels were produced

The provenance is intentionally explicit because this is an agent-curated benchmark, not an independently human-labeled reference set.

1. Historical requests from private development work were extracted, cleaned, normalized, deduplicated, and screened for single-action code-retrieval suitability.
2. An OpenAI Codex agent personally read every candidate prompt and assigned router_test_score from 0 through 100. It did not use a scoring script, classifier, sampling process, or helper agent to assign those scores.
3. Rows scoring at least 70 were eligible for the retained review set. Rows scoring at least 90 were eligible for the pilot set.
4. The same Codex agent assigned the primary and alternative actions after reading the complete frozen 91-action catalog. It used prompt-only reasoning about the requested retrieval target and required action inputs. For ambiguous action families, it also ran representative action comparisons to measure answerability, call count, and returned text size. Results from those representative repositories were used only to understand generic tool behavior, not to infer hidden facts about a private prompt.
5. The retained labels underwent human review for obvious problems, with selected cases corrected before the set was accepted.

The Codex agent itself is a language model. No separately configured model API and no local LLM were invoked for scoring, labeling, catalog generation, or package preparation. The same agent performed most of the initial labeling and re-review, so version 0.1.0 should be treated as a contestable benchmark seed rather than independent expert consensus.

The exact repositories and serialized arguments used for the exploratory representative action comparisons were not retained. Those comparisons therefore cannot be independently replayed from this package and should not be treated as reproducible benchmark evidence.

Suitability score

router_test_score measures how useful a prompt is for testing one first JCodeMunch code-search or retrieval action. It is not a confidence probability and is not a router result.

Score	Meaning
90-100	Excellent: clear, standalone, and strongly favors one action.
70-89	Good: usable with minor ambiguity or harmless context.
40-69	Weak: some retrieval intent exists, but it is contextual, mixed, or ambiguous.
1-39	Poor: little useful single-action intent can be recovered.
0	No useful code-search or retrieval request.

Scores rise for natural wording, one clear intent, sufficient standalone context, and an obvious retrieval target. Scores fall for missing context, multiple tasks, implementation requests, broad workflows, generated/status text, or a prompt that already names the action. The candidate file contains only scores from 70 through 95 and is ordered by non-increasing score. Original candidate_rank values from the 223-row review are retained, so gaps are expected.

Labeling objective

Actions were ranked using this order:

1. Task completion and factual accuracy.
2. Directness of the returned evidence.
3. Whether the prompt supplies the action's required handle, file, symbol, snapshot, or runtime data.
4. Number of calls needed to reach answerable evidence.
5. For the dominant search_symbols versus search_text choice, the form of evidence requested: use search_symbols when the first useful evidence should be a named code entity with its signature or summary; use search_text when the first useful evidence should be occurrences with surrounding file content, especially literals, comments, configuration values, data files, or logic that symbol search may miss. Naming an identifier does not by itself decide the route.
6. Returned token cost when the preceding criteria tie.

The label represents the best first action from the prompt, not a downstream action that requires a handle the prompt does not provide. Accuracy wins over token savings. Lower token use is a tie-breaker only when actions are equally capable of completing the task.

The search_symbols versus search_text rule above is the intended boundary, not a claim that every close case was labeled consistently. The corpus contains materially similar intents with opposite strict primaries. Those cases remain useful as contestable seed labels, but they make this the least stable part of the answer key.

How to score a router

Use gold_primary_action for strict top-1 accuracy. Report top-two or top-three agreement separately if alternatives are evaluated. Do not silently count an alternative as a strict win.

The primary answer key is highly concentrated:

Primary action	Cases	Share
search_symbols	82	41.6%
search_text	82	41.6%
get_file_tree	8	4.1%
get_project_intel	7	3.6%
get_file_outline	4	2.0%
get_signal_chains	4	2.0%
get_symbol_source	3	1.5%
find_importers	2	1.0%
get_ranked_context	2	1.0%
get_parity_map	1	0.5%
get_repo_outline	1	0.5%
get_symbol_diff	1	0.5%

search_text and search_symbols together account for 164 of 197 primary labels, or 83.2%. On the full set, either constant answer scores 82 of 197, or 41.6%, without reading a prompt. Every published result should therefore report, beside strict accuracy:

• the best constant-action baseline calculated on the exact evaluated subset: 41.6% for all 197 rows and 36.4% for the 22-row pilot;
• absolute percentage-point improvement over that baseline;
• the input condition used;
• top-two or top-three agreement only as separate secondary metrics.

An accuracy result without the subset-specific best constant-action baseline and input condition is not interpretable as evidence of useful routing.

If a small score difference comes mostly from disagreements between search_symbols and search_text, do not treat it as proof that one router is better without reviewing those cases by hand. Report this pair separately. Also report how often the router chose either of the first two listed actions; choosing the listed alternative may have been equally defensible.

Input conditions

The primary naturalistic condition presents only prompt_text to the router. source_intent is a curator-authored prompt-only normalization used to state the retrieval task and construct the gold clause. It is not part of the primary router input.

Because source_intent was available during label review, a second normalized-intent condition may present it instead of prompt_text. Results from the two conditions must not be pooled. Several raw prompts are conversational fragments, so normalized-intent results will usually be easier and answer a different question.

All 197 current rows contain exactly one clause. Every clause_id is source-1, every acceptable_actions list contains one strict winner, and every source_intent is byte-identical to that clause's text. Consequently, gold_clauses_json and gold_label_summary add no independent answer-key information in version 0.1.0. Consumers should use the flat primary and alternative columns unless they specifically need the extensible clause schema.

Coverage and limitations

• Twelve of 91 catalog actions appear as primary labels, or 13.2%.
• Eighteen of 91 catalog actions appear anywhere in the ranked labels, or 19.8%.
• This is naturalistic, search-focused coverage, not a balanced or exhaustive full-catalog benchmark.
• Indexing, mutation, runtime-trace, session, diff, multi-repository, and resolved-handle actions may legitimately have no eligible prompt.
• Missing actions are coverage opportunities, not negative assessments of those tools.
• The corpus evaluates first-action selection. It does not evaluate retrieved answer quality, server correctness, latency, or end-to-end task success.
• No harness run results are included.

Do not force uncovered actions onto unsuitable prompts to make the histogram look balanced. Extend coverage with independently scorable naturalistic prompts.

Repository contents

Path	Purpose
data/route_candidates_v0.1.csv	The single authoritative candidate table.
data/action_catalog_v1.csv	Frozen action catalog used for labeling.
manifest.json	Release version, source revision, file paths, counts, and SHA-256 digests.
validate.py	Standard-library corpus and privacy-marker validator.
tools/build_action_catalog.py	Rebuilds or checks the catalog against an installed pinned JCodeMunch package.
tests/test_validate.py	Positive and negative contract tests.
README.md	Package scope, provenance, interpretation, and review instructions.
CONTRIBUTING.md	Exact contribution and validation rules.
THIRD_PARTY.md	Upstream material, license boundary, and public-redistribution gate.
LICENSE	MIT-0 terms for the repository's original material.
.gitattributes	Reviewable LF treatment for the canonical CSV files.
.gitignore	Local environment, cache, and generated-output exclusions.

Candidate schema

Column	Meaning
candidate_rank	Original rank in the 223-row reviewed candidate pool. Unique and increasing, but not contiguous.
case_id	Stable UUID for this benchmark case.
prompt_id	SHA-256 of the normalized prompt formula documented below.
router_test_score	Agent-assigned suitability score described above.
review_status	Every retained row must be scorable.
runnable_full	Must be True for every retained row.
pilot_raw_score	Whether router_test_score >= 90.
pilot_runnable	Whether the retained row is eligible for the pilot run; currently identical to pilot_raw_score.
gold_primary_action	Strict best estimated first action.
gold_alternative_2	Optional second-ranked action.
gold_alternative_3	Optional third-ranked action.
gold_label_summary	Exact clause mapping serialization documented in CONTRIBUTING.md.
gold_clauses_json	Clause text, status, one acceptable action, and ranked alternatives.
source_intent	Curator-authored normalized intent; not primary-condition model input.
exclusion_reason	Empty for every retained row.
prompt_text	Naturalistic router input.

Recomputing prompt_id

All 197 identifiers are recomputable from the shipped prompt:

hashlib.sha256(" ".join(prompt_text.casefold().split()).encode("utf-8")).hexdigest()

Case-folding and whitespace normalization happen before hashing. A direct sha256(prompt_text) happens to match 15 rows whose text is already in normalized form; it is not the identifier contract. The validator checks the documented formula for every row.

Action catalog schema and provenance

The catalog was captured from jcodemunch-mcp v1.108.241 at commit d4616740. All 91 action names match that frozen server surface.

This file carries call metadata only. Upstream action summaries, descriptions, and call examples are authored text under a separate license and are deliberately not included here. Labeling used the upstream descriptions as shipped in v1.108.241; they are reproducible on demand rather than redistributed. See THIRD_PARTY.md.

Column	Meaning
catalog_rank	Stable 1-based row position in the captured catalog.
action	Exact JCodeMunch action name.
required	Compact JSON array of required argument names.
state_changing	True when the action can mutate state, otherwise False.

The snapshot is generated from JCodeMunch's server._catalog_rows() with JCODEMUNCH_TOOL_SURFACE=counter, AI summaries disabled, and an empty temporary code index. The catalog builder refuses any installed version other than 1.108.241, and fails if upstream authored text reaches the CSV.

Install the pinned package first:

Windows:

py -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install jcodemunch-mcp==1.108.241

Linux or macOS:

python3 -m venv .venv
.venv/bin/python -m pip install jcodemunch-mcp==1.108.241

Getting the action descriptions

A router needs a tool surface to choose from, and this repository does not ship one. Reproduce it locally from the package installed above, exactly as the labeling did.

Windows:

.\.venv\Scripts\python.exe tools\build_action_catalog.py --descriptions

Linux or macOS:

.venv/bin/python tools/build_action_catalog.py --descriptions

That prints the upstream summary and description for all 91 actions to stdout. It is separately licensed material: use it locally, do not commit or redistribute it. You can also read it at the pinned upstream commit.

Check the shipped snapshot:

Windows:

.\.venv\Scripts\python.exe tools\build_action_catalog.py --check data\action_catalog_v1.csv

Linux or macOS:

.venv/bin/python tools/build_action_catalog.py --check data/action_catalog_v1.csv

Rebuild to a separate file for inspection:

Windows:

.\.venv\Scripts\python.exe tools\build_action_catalog.py --output rebuilt_action_catalog.csv

Linux or macOS:

.venv/bin/python tools/build_action_catalog.py --output rebuilt_action_catalog.csv

The command requires the exact jcodemunch-mcp==1.108.241 package. manifest.json records the upstream URL, full commit, source version, catalog digest, and the declaration that no upstream text is included.

What validation covers

The validator checks:

• manifest paths remain inside the repository;
• both CSV digests and row counts match the manifest;
• exact column order, identifier formats, uniqueness, score order, and pilot threshold;
• the documented prompt_id formula;
• catalog ranks, required-argument JSON, state-change flags, and action uniqueness;
• flat and structured gold-label agreement;
• catalog membership of every ranked action;
• exact clause, source-intent, and summary serialization rules;
• obvious email, credential, local-user-path, profanity, slur, and forbidden-punctuation markers;
• generic absolute Windows paths as manual-review warnings.

It does not check:

• whether a gold action is semantically the best answer;
• whether every catalog action was fairly represented;
• whether the catalog matches a live server other than through the separate builder command;
• whether a project name, game path, unpublished feature, or contextual detail is acceptable for public release;
• whether a contributor owns or may license newly added material;
• router output quality, token use, latency, or end-to-end completion.

Passing validation is an integrity gate, not a release-readiness verdict.

Content notes

Automated leak scans are clean for credentials, emails, user-home paths, IP addresses, and UNC paths. The following contextual identifiers appear in the retained prompts and intents, and were reviewed and cleared for publication by the dataset owner:

• ProxiCraft, UT_CORE, 00_ModProfiler, Open Brain, AudibleBreakingGlassJars;
• doc_explorer, entity_documentor.py, launch_doc_explorer.py;
• ConstraintScorer, AdaptiveWorkerPool, VirtualInventoryProvider, NetworkPacketObserver, schedule-bucket;
• cost_per_fte_hour, hourly_rate, locker_number, seat_assignment.

Two rows contain absolute game-install paths, which carry no user name:

• C:\Steam\steamapps\common\7 Days To Die\Mods;
• C:\Steam\steamapps\common\7 Days To Die\7DaysToDie_Data\Resources\unity default resources.

Three prompts contain profanity and one of those also contains an ableist slur. Prompts are published as written rather than cleaned up, for the reason given at the top of this document: handling real phrasing is part of what the benchmark measures.

See THIRD_PARTY.md for the upstream attribution position.

Version naming

• dataset_version: 0.1.0 is the candidate dataset version. It does not mean public-release review is complete.
• route_candidates_v0.1.csv names the compatible major-minor dataset line. Patch releases may correct metadata or labels without creating another candidate sidecar file.
• action_catalog_v1.csv uses v1 for the catalog CSV schema, not the JCodeMunch product version.
• Catalog provenance records JCodeMunch version 1.108.241 and commit d4616740a0e27d078559353ac3c53461ae874006. The reproduction command enforces the installed package version, but the installed package does not expose a commit identity for the command to verify independently.

Expanding the dataset

This seed is designed to grow through real, independently scorable prompts. Keep all candidate rows and labels in the single authoritative CSV. Do not create correction, exclusion, or gold-label sidecar candidate files.

See CONTRIBUTING.md for the exact invariants and review requirements.

License

The repository is licensed under MIT-0, the MIT No Attribution license. Every file in it is original material: the prompts, the labels, the validator, the tests, and the tooling. No upstream text is redistributed here; see THIRD_PARTY.md.

JCodeMunch is a separate project. Action names appear as identifiers so labels can reference the tool surface, not as a claim that this repository is an official distribution or endorsement of JCodeMunch.