From cc4ff5f9aad5fd951d4490664178e9bf9295789b Mon Sep 17 00:00:00 2001 From: Dmitry Romanov Date: Tue, 3 Nov 2026 17:24:53 -0500 Subject: [PATCH 01/27] Plan: CCDB 2.5 AI-readiness program (PHASE-A0 to A5) The readiness profile, the record of the constants a job resolved, tag properties and `export tag`, the property vocabulary with UCUM checks and JSON-LD, and a Great Expectations example. Read with web-agents-common.md and ccdb-2.5.md. Co-Authored-By: Claude Opus 5.5 --- ccdb-ai-ready.md | 191 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100644 ccdb-ai-ready.md diff --git a/ccdb-ai-ready.md b/ccdb-ai-ready.md new file mode 100644 index 00000000..91d35336 --- /dev/null +++ b/ccdb-ai-ready.md @@ -0,0 +1,191 @@ +# CCDB 2.5 — AI-readiness program plan + +To start a run, the user gives you one line: `Read {repo}/web-agents-common.md, {repo}/ccdb-2.5.md and {repo}/ccdb-ai-ready.md first. Run PHASE-A.` + +Read the three files in that order before anything else: + +1. `web-agents-common.md` — the work protocol, the FIX / ASK / PARK / STOP protocol, the memory + layout of `{space}`, the findings index, and the documentation and coding rules. +2. `ccdb-2.5.md` — the testing rule and the shared ruler of the 2.5 program. Both apply here + unchanged: every work item lands with a test that fails without the change, and the gate of a + phase is the full 2.5 ruler plus the phase's own numbers. +3. This file — what belongs to this program: the goal, the decisions already made, the defaults + for open decisions and the phases. + +Three rules of the common file are overridden here by the user's decisions of 2026-11-03, and only +in the way A-D6 to A-D8 state: git (commits on a branch), the core scope (the files A-D7 names), +and the run mode. Everywhere else the common rules stand. + +The program follows the 2.5 program and lands in the same release: v2.5.0 is not tagged. Nothing +here changes the database schema. Every item is a command, an endpoint, an MCP tool, a GUI page or +a library call over schema 6 as it is. + +## Paths + +| name | path | what lives there | +|---|---|---| +| {repo} | /home/romanov/dev/ccdb-ai-ready | a git worktree of the CCDB checkout on branch `ai-ready`, created from `origin/v2-main` (b16f3b6, 2.5.0 merged by #122) | +| {space} | /home/romanov/dev/ccdb/space-ccdb-2.5 | the 2.5 workspace in the main checkout, outside version control. This program's reports go to `{space}/reports/ai-ready/` | +| {py} | {repo}/python/ccdb | the Python package: `provider.py`, `model.py`, `cmd/commands/`, `api/`, `mcp/`, `webgui/` | +| {cpp} | {repo}/cpp | the C++ library (`src/CCDB`), Catch2 tests (`src/tests`) | +| {jana2} | {repo}/jana2 | the JANA2 service | +| {scripts} | {repo}/scripts | `api/` (the API ruler), `docs/`, `mcp/` | +| {docs} | {repo}/docs | docsify site | +| {ci} | {repo}/.github/workflows | the workflows named in `ccdb-2.5.md` | +| {examples} | {repo}/python/examples | the examples `python-examples.yml` runs | + +## Goal + +Make CCDB 2.5 the calibration database whose readiness for AI use can be measured, whose +constants carry their provenance into every job that reads them, and whose tags name a +processing pass completely. The source of the requirements is the assessment framework of the +GlueX AI-readiness article (Panta, Morean, Romanov, 2026): seven categories, 26 sub-criteria, and +the rule that every readiness claim is reproducible, comparable and defensible. The report +`{space}/reports/ai-readiness.md` maps the article onto CCDB and lists the code changes; this +plan implements changes 1, 2, 3, 4, 5, 6, 8 and 10 of that list. Changes 7 (value expectations on +`add`), 9 (RCDB and Rucio links), 11 (locks and roles) and 12 (agents that write) are not in this +program. + +What the program delivers, in the words a user meets: + +- `ccdb db readiness`: a deterministic JSON profile of an instance, the same numbers at + `/api/v1/readiness`, as the MCP tool `readiness`, and as a GUI page. +- A record of the constants a job resolved: in the C++ and Python libraries, written by the JANA2 + service, checked by `ccdb provenance`. +- Properties on tags, so a tag names the software, container, data and DOI of a pass. +- `ccdb export tag`: the constants of a tag as one SQLite file with a DataCite metadata file and a + hash, ready for a repository deposit. CCDB makes no network call. +- schema.org `Dataset` metadata on every data card. +- A property vocabulary that `prop set` checks, served by the API. +- UCUM unit checks in `mktbl` and `edit`, counted by the readiness profile. +- A Great Expectations example on the Parquet export, run in CI. + +Everything that works in 2.5.0 keeps working. No flag changes meaning; new flags and commands only +add. + +## Decisions already made by the user + +| id | decision | +|---|---| +| A-D1 | Everything in this plan goes into 2.5; there is no 2.5.x or 2.6 split (2026-11-03). | +| A-D2 | CCDB never contacts an outside service by itself: no Zenodo upload, no hooks. `export tag` writes files; a person or a CI job uploads them (2026-11-03). | +| A-D3 | Collaboration data work (units, descriptions, licenses, uncertainty tables, cut values) is out of scope. The program measures these gaps; it does not fill them (2026-11-03). | +| A-D4 | Recommendations and docs name cheap, gradual steps in plain language (feedback of 2026-07-20 on the article). | +| A-D5 | Confirmation flags keep 2.4's `---i-am-sure` spelling; no flag name implies an automatic action (standing). | +| A-D6 | Git: `please-git` for this program (2026-11-03). Work on branch `ai-ready` in the worktree `{repo}`; one commit per work item once its test and gate checks pass, so commits replace the checkpoints of the common file. No pull request. No push until PHASE-A5 is done, because `documentation.yml` publishes the docs site from any pushed branch. The main checkout (`/home/romanov/dev/ccdb`, branch `fix-56-edit-comment`, PR #125) is not touched. | +| A-D7 | Core scope (default taken 2026-11-03, the user stated no preference). These existing core files may be edited, with their tests: A1 `{py}/cmd/commands/db.py`; A2 `{cpp}/src/CCDB/Calibration.h`, `Calibration.cc` and `{py}/provider.py` (the Python package has no `Calibration` class, so the Python record of resolved assignments lives on `AlchemyProvider`); A3 `{py}/provider.py` (`PROPERTY_OWNER_TYPES`, property lookup by tag), `{py}/cmd/commands/prop.py`, `tag.py`, `info.py`, `export.py`; A4 `prop.py`, `mktbl.py`, `edit.py`; plus command registration, `pyproject.toml` and `{ci}`. New files need no decision. A change to any other existing core file goes the core-patch PARK route. | +| A-D8 | Run mode (default taken 2026-11-03). PHASE-A0 and PHASE-A1 run, then the run stops so the user checks the readiness profile the article prints. After the user's OK, A2 to A5 run without stopping between phases: a gate is a check, and a non-blocking question takes its default. | +| A-D9 | The docs written on 2026-11-03 describe every phase as done; they wait uncommitted in `{repo}`. Each phase commits only the sections it implements, corrected to the code it shipped (default taken 2026-11-03). | + +## Defaults for open decisions + +The user answers these in chat. Until then, the default applies. + +| id | question | default | +|---|---|---| +| A-Q1 | Change 2, light or full? | Both, in this order: the light item (the JANA2 service logs the pinned time or tag and warns when a production job has neither) is PHASE-A2 item 1; the full record is items 2–4. The user may stop the phase after item 1. | +| A-Q2 | Where does the JANA2 service write the job record? | Parameter `ccdb:provenance=PATH`; empty by default, so nothing is written unless asked. | +| A-Q3 | Is an unknown property name an error? | A warning. `--free` silences it. An instance may extend the vocabulary with the instance property `property_names` (a comma-separated list). | +| A-Q4 | UCUM units: error or warning? | A warning in `mktbl` and `edit`; never an error. Physics units outside UCUM are written as annotations (`{ADC}`), which UCUM allows. | +| A-Q5 | The staleness window of the readiness profile | 5 years by default (`--stale-years`), echoed in the profile's `config`. | +| A-Q6 | Which engines does `export tag` support? | MySQL, MariaDB and SQLite sources; the output is always SQLite, built by the bundle builder of the API with the run range opened to every run. | + +## Phase plan + +Phases run in order. PHASE-A0 is the baseline. Each later phase has a Step 0, work items, a gate, +a done criterion. The next phase starts when the previous report is in `{space}/reports/ai-ready/` +and its BLOCKING questions are answered in chat. + +### PHASE-A0 — baseline + +Task: a green tree and the numbers the program will change. + +- **Step 0**: the branch `ai-ready` from `v2-main` in the worktree `{repo}`; the only uncommitted changes are this plan and the docs of A-D9. Commit the plan first. +- Work items: + 1. Run the whole 2.5 ruler; record the counts in `{space}/reports/ai-ready/a0_baseline.md`. + 2. Record the Hall D profile by hand once more from the schema 6 trial copy (read-only): units, descriptions, comments, open-ended ranges, stale tables, anonymous authors, duplicate sets, instance properties. These are the numbers `ccdb db readiness` must reproduce in PHASE-A1. +- Gate: the ruler green. +- Done: `a0_baseline.md`. + +### PHASE-A1 — the readiness profile (change 1) + +Task: one deterministic computation of an instance's readiness, served four ways. + +- **Step 0**: the A0 gate. +- Work items: + 1. `{py}/readiness.py`: `profile(provider, config) -> dict`. The dict holds `config` (the inputs: `stale_years`, `generated`, the CCDB version, the schema version), `metadata` (columns with units / total; tables with a description / total; versions with a comment / total; tables with a `contact`, `source_url` or `doc_url` property), `history` (open-ended run ranges; versions by `anonymous`; tables with no write for `stale_years`; tables never filled; deprecated tables), `data` (constant sets, duplicate sets merged, bytes), `instance` (`license`, `citation`, `contact`, `experiment`, `url` set or not), `units` (columns whose unit string is valid UCUM, after PHASE-A4; `null` before). Every count is one SQL query; no sampling. `unit_test_readiness.py` builds a fixture and checks every number; a second test runs the profile twice and compares the dicts. + 2. `ccdb db readiness [--stale-years N] [--json]`: the text form is a table of the same numbers; `--json` prints the dict. `integ_test_cli_manager.py` gets the command; `cli-smoke.yml` runs it. + 3. `GET /api/v1/readiness?stale_years=N`: the dict, `Cache-Control: public, max-age=3600`. `{scripts}/api/pages.py` and `golden.py` learn it; the OpenAPI document diff is in the report. + 4. MCP tool `readiness(stale_years)`: the same dict. `{scripts}/mcp/transcript.py` gets one step. + 5. GUI page `/readiness`: the numbers with one bar per coverage item (visual loop); a link from the instance page. + 6. Docs: `{docs}/concepts/ai-readiness.md` (the metric definitions, one line each, and how to read the profile); the `db` section of `{docs}/cli.md`; `{docs}/apis/http.md` and `{docs}/apis/mcp.md` rows. +- Gate: the ruler plus the new tests; the profile of the Hall D trial copy equals the A0 hand count. +- Done: `a1_readiness.md` with the Hall D profile as JSON. + +### PHASE-A2 — what a job resolved (change 2) + +Task: the calibration third of a processing pass, recorded by the code that resolves it. + +- **Step 0**: the A1 gate. +- Work items: + 1. **Light.** `CCDB_service` logs at startup the connection string (password hidden), the variation, and the time or tag it pins; when neither `ccdb:time` nor `ccdb:tag` is set, it logs a warning that the job reads the newest constants and cannot be repeated by its parameters. A `[jana2]` test checks both messages. *The user may end the phase here (A-Q1).* + 2. `Calibration::GetResolvedAssignments()` in `{cpp}`: for every request served since connection, the request string, the assignment id, the SHA-256, the run range, the variation and the creation time, in request order; a `[sqlite]` and an `[http]` Catch2 test. Python: `ccdb.Calibration.resolved_assignments` with the same fields; `unit_test_calibration_resolved.py`. + 3. `CCDB_service` parameter `ccdb:provenance=PATH` (A-Q2): at the end of the job, the list as JSON with a header (`ccdb_version`, `connection` without password, `variation`, `time` or `tag`, `run`, `generated`). A `[jana2]` test reads the file back. + 4. `ccdb provenance FILE.json [--tag]`: checks every assignment id and hash against the connected database and prints the tags whose resolution equals the file's; exit code 1 when a hash differs. Test in `integ_test_cli_manager.py`. + 5. Docs: a section "What a job used" in `{docs}/apis/jana.md` and `{docs}/apis/cpp.md`; `{docs}/cli.md`. +- Gate: the ruler plus the new tests; the JANA2 and C++ suites through `run_in_eic_shell.sh`. +- Done: `a2_provenance.md`. + +### PHASE-A3 — tags as named passes (changes 3 and 10) + +Task: a tag carries what a pass needs, and leaves the database as one file ready for a deposit. + +- **Step 0**: the A2 gate. +- Work items: + 1. `tag` as a property owner type (`PROPERTY_OWNER_TYPES` gains `"tag"`; `owner_id` is the tag's id; no schema change). `ccdb prop set|get|ls|rm @NAME KEY [VALUE]` — the `@` names a tag as it does in requests. Standard names for tags: `software`, `container`, `data`, `rucio_did`, `doi`, `description`, `citation`. `ccdb tag ls -l`, `ccdb info @NAME`, `/api/v1/tags/{name}`, the MCP `list_tags` and the GUI tag page show them. Tests: `unit_test_properties.py` extended; API golden rows. + 2. `ccdb export tag @NAME -o DIR`: writes `DIR/NAME.sqlite` (every table, the tag's variation chain, every assignment the tag sees for every run, the tag itself), `DIR/NAME.sha256`, and `DIR/NAME.datacite.json` built from the instance properties (`citation` → creators and title, `license`, `url`) and the tag's properties (`description`, `software`, `container`, `data`, `rucio_did`) in DataCite 4.5 JSON; `relatedIdentifiers` carries the `doi` of a previous export of the same tag when set. The builder is the API bundle builder with the run clip removed (A-Q6). `unit_test_export_tag.py` opens the file with the Python provider and compares one request per table against the source; it validates the metadata JSON against the DataCite schema stored under `{scripts}/docs/`. + 3. Docs: `{docs}/concepts/tags.md` (properties; the deposit workflow in four lines: export, upload, record the DOI with `prop set @NAME doi`), `{docs}/cli.md`. +- Gate: the ruler plus the new tests; `export tag` of the GUI fixture's tag opened by the C++ library (Catch2 `[sqlite]`). +- Done: `a3_tags.md`. + +### PHASE-A4 — vocabulary and discovery (changes 4, 6, 8) + +Task: metadata that programs can check and catalogs can find. + +- **Step 0**: the A3 gate. +- Work items: + 1. `{py}/vocabulary.py`: the standard property names per owner type as data (the table of `{docs}/cli.md` moves here and the docs page is generated from it by `{scripts}/docs/`), plus the instance extension `property_names` (A-Q3). `prop set` warns on an unknown name unless `--free`. `GET /api/v1/properties/vocabulary` serves it. Tests. + 2. UCUM: `{py}/units.py` validates a unit string against UCUM's grammar with the base units, prefixes and the common derived units in a table (no network, no new dependency); annotations in braces pass. `mktbl` and `edit` warn on an invalid unit (A-Q4). The readiness profile fills `units`. `unit_test_units.py` with the units of the Hall D trial copy's known tables as cases. + 3. schema.org `Dataset` as JSON-LD: a ` {% endblock %} diff --git a/python/ccdb/webgui/views.py b/python/ccdb/webgui/views.py index 7de04a50..dedb3f80 100644 --- a/python/ccdb/webgui/views.py +++ b/python/ccdb/webgui/views.py @@ -470,7 +470,9 @@ def table_card(table_path): """The data card of a table: what a reader needs to use its constants and to cite them (docs/concepts/fair.md).""" db: ccdb.AlchemyProvider = g.db table = db.get_type_table("/" + table_path) # unknown path → ObjectIsNotFoundInDbError → 404 page - return render_template("table_card.html", card=queries.table_card(db, table), table=table, + card = queries.table_card(db, table) + jsonld = queries.card_jsonld(card, url_for('table_card', table_path=table_path, _external=True)) + return render_template("table_card.html", card=card, table=table, jsonld=jsonld, deprecated_by=author_name(db, table.deprecated_by_user_id)) @app.route('/test_request') diff --git a/python/tests/unit_test_api.py b/python/tests/unit_test_api.py index 1fd009ed..e98c73f4 100644 --- a/python/tests/unit_test_api.py +++ b/python/tests/unit_test_api.py @@ -139,6 +139,18 @@ def test_property_vocabulary(self): self.assertIn("doi", data["meanings"]) self.assertEqual(data["instance_additions"], []) + def test_card_as_jsonld(self): + response = self.get("/tables/test/test_vars/test_table/card?format=jsonld") + self.assertEqual(response.status_code, 200, response.text) + self.assertTrue(response.headers["content-type"].startswith("application/ld+json")) + record = response.json() + self.assertEqual((record["@context"], record["@type"]), ("https://schema.org", "Dataset")) + self.assertEqual(record["identifier"], "/test/test_vars/test_table") + self.assertEqual([variable["name"] for variable in record["variableMeasured"]], ["x", "y", "z"]) + self.assertTrue(record["url"].endswith("/api/v1/tables/test/test_vars/test_table/card")) + self.assertIn("dateModified", record) + self.assertEqual(self.get("/tables/test/test_vars/test_table/card?format=xml").status_code, 400) + def test_openapi_document(self): document = self.json("/openapi.json") self.assertTrue(document["openapi"].startswith("3.")) diff --git a/python/tests/unit_test_webgui.py b/python/tests/unit_test_webgui.py index 07ff35e2..461023fd 100644 --- a/python/tests/unit_test_webgui.py +++ b/python/tests/unit_test_webgui.py @@ -229,6 +229,17 @@ def test_tag_page_shows_the_tag_properties(self): self.assertIn('doi', html) self.assertIn('href="https://doi.org/10.5281/zenodo.1234567"', html) + def test_card_page_embeds_jsonld(self): + import json + import re + html = self.make_app(self.connection_string).test_client().get("/tables/test/test_vars/test_table/card").get_data(as_text=True) + block = re.search(r'', html, re.S) + self.assertIsNotNone(block) + record = json.loads(block.group(1)) + self.assertEqual(record["@type"], "Dataset") + self.assertEqual(record["name"], "/test/test_vars/test_table") + self.assertTrue(record["url"].endswith("/tables/test/test_vars/test_table/card")) + def test_health_reports_a_dead_database_as_503_and_recovers(self): import json missing = os.path.join(self.tmp_dir.name, "not_yet.sqlite") diff --git a/scripts/api/golden.py b/scripts/api/golden.py index 39951bc2..1c43e8f9 100644 --- a/scripts/api/golden.py +++ b/scripts/api/golden.py @@ -87,7 +87,7 @@ def get(url): for label, url, kind in pages.PAGES: if not (url.startswith("/api/v1/tables/") and kind == "json" and "/versions" not in url and "/schema" not in url - and not url.endswith("/card")): + and "/card" not in url): continue path = unquote(url[len("/api/v1/tables"):]) info = get(url).json() @@ -164,6 +164,18 @@ def get(url): "HAVING COUNT(*) > 1)")): checks.equal(f"readiness {group}.{field}", profile[group][field], sql.execute(query).fetchone()[0]) + # The card as schema.org JSON-LD: one variableMeasured per column, with its units, in column order + for label, url, kind in pages.PAGES: + if not (kind == "json" and url.endswith("/card?format=jsonld")): + continue + name = unquote(url[len("/api/v1/tables"):-len("/card?format=jsonld")]).rsplit("/", 1)[1] + record = get(url).json() + rows = sql.execute("SELECT c.name, c.units FROM columns c JOIN typeTables t ON t.id = c.typeId WHERE t.name = ? " + "ORDER BY c.\"order\"", (name,)).fetchall() + checks.equal(f"{label} variableMeasured", [(each["name"], each.get("unitText")) for each in record["variableMeasured"]], + [(column, units or None) for column, units in rows]) + checks.equal(f"{label} type", (record["@context"], record["@type"]), ("https://schema.org", "Dataset")) + # The change feed: the newest log records, newest first from xml.etree import ElementTree feed = ElementTree.fromstring(get("/api/v1/logs.atom").content) diff --git a/scripts/api/openapi.json b/scripts/api/openapi.json index bcf6e711..f8f656bc 100644 --- a/scripts/api/openapi.json +++ b/scripts/api/openapi.json @@ -2064,6 +2064,24 @@ "title": "Path", "type": "string" } + }, + { + "description": "`jsonld`: the card as a schema.org Dataset in JSON-LD", + "in": "query", + "name": "format", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "`jsonld`: the card as a schema.org Dataset in JSON-LD", + "title": "Format" + } } ], "responses": { diff --git a/scripts/api/pages.py b/scripts/api/pages.py index 983cdf30..a5157cfe 100644 --- a/scripts/api/pages.py +++ b/scripts/api/pages.py @@ -42,6 +42,8 @@ ("table_missing", "/api/v1/tables/no/such", "edge"), ("schema_channels", "/api/v1/tables" + CHANNELS_TABLE + "/schema", "json"), ("card_big", "/api/v1/tables" + BIG_TABLE + "/card", "json"), + ("card_big_jsonld", "/api/v1/tables" + BIG_TABLE + "/card?format=jsonld", "json"), + ("card_bad_format", "/api/v1/tables" + BIG_TABLE + "/card?format=xml", "edge"), ("card_shared", "/api/v1/tables" + SHARED_TABLE + "/card", "json"), ("card_unicode", "/api/v1/tables" + quote(UNICODE_TABLE) + "/card", "json"), ("card_retired", "/api/v1/tables" + RETIRED_TABLE + "/card", "json"), From 7f08fddf4503dc177efae154f591fdbfecc7ea23 Mon Sep 17 00:00:00 2001 From: Dmitry Romanov Date: Tue, 3 Nov 2026 19:58:51 -0500 Subject: [PATCH 21/27] Docs: UCUM units, the property vocabulary and JSON-LD cards fair.md: F1 (tag export for a DOI), F2 (vocabulary), F4 (JSON-LD cards), I2 (UCUM and the vocabulary) and R1.2 (the job record) move to today; the data card section shows the JSON-LD form. cli.md: the UCUM check of mktbl and set_column_units for old tables. http.md: /properties/vocabulary and the card's jsonld form. The AI readiness page: the units field, and the tools that fix each gap. Co-Authored-By: Claude Opus 5.5 --- docs/apis/http.md | 2 ++ docs/cli.md | 13 ++++++++++++- docs/concepts/ai-readiness.md | 11 ++++++----- docs/concepts/fair.md | 16 +++++++++++----- 4 files changed, 31 insertions(+), 11 deletions(-) diff --git a/docs/apis/http.md b/docs/apis/http.md index 35b24cbf..ec942b99 100644 --- a/docs/apis/http.md +++ b/docs/apis/http.md @@ -34,12 +34,14 @@ variations unless you add `deprecated=all` ([Deprecation](concepts/deprecation.m |---|---| | `/health` | `{"status": "ok", "database": "ok", "version": ...}`, or 503 when the database does not answer | | `/instance` | the CCDB version, the schema version, the instance properties | +| `/properties/vocabulary` | the standard [property names](cli.md#properties) per owner type, their meanings and the names the instance adds | | `/readiness?stale_years=5` | the [readiness profile](concepts/ai-readiness.md#the-readiness-profile) of the instance, the JSON of `ccdb db readiness --json` | | `/directories`, `/directories/{path}` | every directory; one directory with its subdirectories and tables | | `/tables?search=gain*` | the tables whose name matches a pattern with `*` and `?` | | `/tables/{path}` | one table: columns with types, units and comments, properties, variations with data, the assignment count and the newest assignment | | `/tables/{path}/versions` | a page of the table's assignments, as `ccdb vers`: `page`, `per_page` (25, 50, 100, 200), `sort` (`created`, `run_min`, `run_max`, `variation`, `author`, `id`), `order`, `variation`, `run`, `run_min`, `run_max` | | `/tables/{path}/schema` | a [JSON Schema](https://json-schema.org/) of the `rows` of the table's `data.json` | +| `/tables/{path}/card`, `/card?format=jsonld` | the [data card](concepts/fair.md#data-cards) of the table as JSON, or as a schema.org `Dataset` in JSON-LD | | `/variations`, `/variations/{name}` | every variation; one variation with its ancestry | | `/run-ranges` | every run range | | `/tags`, `/tags/{name}` | every tag; one tag; each with its [properties](concepts/tags.md#tag-properties) ([Tags](concepts/tags.md)) | diff --git a/docs/cli.md b/docs/cli.md index 9200733b..dfeefc51 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -416,7 +416,18 @@ $ ccdb mktbl /calib/gains -r 2 channel=int gain=double:MeV offset=double:ns time ``` The units part is optional, and `name=type` stays valid. An empty type keeps the default `double`: -`time=:ns` is a `double` column in `ns`. `info` shows the units in brackets: +`time=:ns` is a `double` column in `ns`. + +Units are checked against [UCUM](https://ucum.org), the Unified Code for Units of Measure, which programs +can parse and convert: `MeV`, `ns`, `cm`, `cm2`, `1/s`, `GeV/[c]` pass. `mktbl` warns on a string UCUM can't +read (`Mev`, `nsec`, `GeV/c`) and stores it as typed. A quantity without a standard unit, such as ADC counts +or a channel number, is a UCUM annotation in braces, `{ADC}` or `{channel}`, which passes. The +[readiness profile](concepts/ai-readiness.md) counts the columns whose units UCUM reads. + +A table made before units existed gets them column by column in Python, with a log record for each: +`provider.set_column_units("/calib/gains", "gain", "MeV")`. + +`info` shows the units in brackets: ``` $ ccdb info /calib/gains diff --git a/docs/concepts/ai-readiness.md b/docs/concepts/ai-readiness.md index 145c75eb..0c58a655 100644 --- a/docs/concepts/ai-readiness.md +++ b/docs/concepts/ai-readiness.md @@ -16,7 +16,7 @@ FAIR principles; this page maps readiness. | requirement | CCDB 2.5 | |---|---| | Identity of the data | Every version has a permanent id and the SHA-256 of its data; a [tag](concepts/tags.md) names a whole calibration and, with its properties, the software, container and data of the pass; `ccdb export tag` makes a tag a file with a hash and a DataCite record, ready for a DOI ([Publishing a tag](concepts/tags.md#publishing-a-tag)) | -| Understandable without a human | Typed columns with units and comments, [properties](cli.md#properties), a JSON Schema per table, a [data card](concepts/fair.md#data-cards) in HTML and JSON | +| Understandable without a human | Typed columns with [UCUM](cli.md#column-units) units and comments, properties from a checked [vocabulary](cli.md#properties), a JSON Schema per table, a [data card](concepts/fair.md#data-cards) in HTML, JSON and schema.org JSON-LD | | Provenance of every value | Author, UTC time and comment on every version; the change log of every operation; the hash of every data set | | Provenance of every use | The C++ and Python libraries record, when a program asks, which versions it resolved; the JANA2 service writes the record as a file with `ccdb:provenance`; `ccdb provenance` checks it against the database ([JANA2 service](apis/jana.md#what-a-job-used)) | | Access by programs and agents | The [HTTP API](apis/http.md) with its OpenAPI document, run bundles, the read-only [MCP server](apis/mcp.md), [Parquet and data frames](apis/analysis.md) | @@ -48,7 +48,7 @@ The same profile is served at `/api/v1/readiness` by the [HTTP API](apis/http.md | | `tables_with_description` / `tables` | tables with a non-empty comment | | | `versions_with_comment` / `versions` | assignments with a non-empty comment | | | `tables_with_contact`, `tables_with_source_url`, `tables_with_doc_url` | tables with that property | -| `units` | `columns_with_ucum_units` | not measured yet: `null` | +| `units` | `columns_with_ucum_units` | columns whose unit string is valid [UCUM](cli.md#column-units) | | `history` | `open_ended_run_ranges` | assignments whose run range starts at a run above 0 and has no upper bound: later runs receive these constants without anyone deciding so | | | `versions_for_all_runs` | assignments whose run range covers every run, 0 to infinity | | | `versions_by_anonymous` | assignments authored by `anonymous` | @@ -80,6 +80,7 @@ It took 7 seconds: "metadata": {"columns": 7700, "columns_with_units": 0, "tables": 688, "tables_with_description": 318, "versions": 821394, "versions_with_comment": 290787, "tables_with_contact": 0, "tables_with_source_url": 0, "tables_with_doc_url": 0}, + "units": {"columns_with_ucum_units": 0}, "history": {"open_ended_run_ranges": 41350, "versions_for_all_runs": 1212, "versions_by_anonymous": 4675, "stale_tables": 390, "empty_tables": 9, "deprecated_tables": 0, "tags": 1}, "data": {"constant_sets": 585786, "shared_sets": 66146, "unused_sets": 0, "bytes": 2870802276}, @@ -87,6 +88,6 @@ It took 7 seconds: } ``` -Each line names one thing a collaboration can change with the tools CCDB has: `mktbl` and `edit` for -units, `prop set` for descriptions and the instance license, a comment on every `add`, `deprecate` for -the tables nobody writes to. The next profile shows whether it happened. +Each line names one thing a collaboration can change with the tools CCDB has: units in `mktbl` for new +tables and `set_column_units` for old ones, `prop set` for descriptions and the instance license, a +comment on every `add`, `deprecate` for the tables nobody writes to. The next profile shows whether it happened. diff --git a/docs/concepts/fair.md b/docs/concepts/fair.md index 674b76ed..5a5811f3 100644 --- a/docs/concepts/fair.md +++ b/docs/concepts/fair.md @@ -8,20 +8,20 @@ covers most of FAIR by itself. The table maps each principle to what CCDB does t | principle | today | roadmap | |---|---|---| -| **F1** a globally unique, persistent identifier | Every version of every table has a permanent id, and a request with a time names constants for good: `/path:run:variation:time`, or `/api/v1/assignments/` of the [HTTP API](apis/http.md). The data of each version has a SHA-256 ([constant sets](concepts/constant-sets.md)), so identical data has an identical name. [Tags](concepts/tags.md) name a whole calibration campaign. | A DOI per release tag through Zenodo, so a paper cites the constants it used. | -| **F2** rich metadata | Tables have typed columns with units and comments, properties such as purpose, procedure and source code (`ccdb prop`), and a [data card](#data-cards) that gathers them. | A recommended set of property names for every experiment. | +| **F1** a globally unique, persistent identifier | Every version of every table has a permanent id, and a request with a time names constants for good: `/path:run:variation:time`, or `/api/v1/assignments/` of the [HTTP API](apis/http.md). The data of each version has a SHA-256 ([constant sets](concepts/constant-sets.md)), so identical data has an identical name. [Tags](concepts/tags.md) name a whole calibration campaign. `ccdb export tag` writes a tag as a file with its SHA-256 and a DataCite record, ready for a DOI ([Publishing a tag](concepts/tags.md#publishing-a-tag)). | | +| **F2** rich metadata | Tables have typed columns with units and comments, properties from a [standard vocabulary](cli.md#properties) such as description, contact and source code (`ccdb prop`), and a [data card](#data-cards) that gathers them. A tag names the software, container and data of a pass. | | | **F3** metadata names the data's identifier | The data card names the table, its versions and their ids; every row of a [Parquet export](apis/analysis.md) carries its version's id and hash. | | -| **F4** registered or indexed in a searchable resource | The web GUI and the API search tables by name and the change log by text; the [MCP server](apis/mcp.md) lets AI agents search and read; the [Atom feed](#change-feed) announces every change. | schema.org `Dataset` metadata on data cards, for search engines and data catalogs. | +| **F4** registered or indexed in a searchable resource | The web GUI and the API search tables by name and the change log by text; the [MCP server](apis/mcp.md) lets AI agents search and read; the [Atom feed](#change-feed) announces every change; every data card is also a schema.org `Dataset` in JSON-LD, for search engines and data catalogs. | | | **A1** retrievable by identifier over a standard protocol | HTTP and JSON with an OpenAPI description ([HTTP API](apis/http.md)), SQL (MySQL, MariaDB, SQLite), and run bundles: one SQLite file with the constants of one run. | | | **A1.1** open, free, universal protocol | HTTP, SQL and SQLite are open and free; CCDB itself is LGPL-3.0-or-later. | | | **A1.2** authentication and authorization where necessary | Reading needs no password; writing needs a database account ([deployment](deployment/new-experiment.md)). | | | **A2** metadata stays when the data is retired | Nothing is deleted: retired tables, variations and directories are [deprecated](concepts/deprecation.md) and stay readable, and the change log keeps every operation. | | | **I1** a formal, shared language | JSON with a JSON Schema per table (`/api/v1/tables//schema`), Parquet and Arrow with typed columns, pandas and polars [data frames](apis/analysis.md#data-frames). | | -| **I2** FAIR vocabularies | Units are free text per column. | Units from a standard vocabulary (UCUM), checked when a table is made. | +| **I2** FAIR vocabularies | Column units are checked against [UCUM](cli.md#column-units), the Unified Code for Units of Measure, when a table is made; property names come from a [vocabulary](cli.md#properties) that `prop set` checks. | | | **I3** qualified references to other data | Table properties link the code that reads the constants (`source_url`); tags reference a variation and a moment; [resources tables](apis/jana.md) reference files by URL and hash. | Links to the run conditions database for each run range. | | **R1** many accurate, relevant attributes | Every version has its author, creation time to the microsecond, comment and run range; every column its type, units and comment. | | | **R1.1** a clear data usage license | The instance properties `license` and `citation` (`ccdb prop set --instance`) appear on every data card. | | -| **R1.2** detailed provenance | Author, time and reason of every version, the change log of every operation in the GUI, the API, the Atom feed and the MCP server, and the hash of every data set. | | +| **R1.2** detailed provenance | Author, time and reason of every version, the change log of every operation in the GUI, the API, the Atom feed and the MCP server, and the hash of every data set. A job records which versions it resolved ([What a job used](apis/jana.md#what-a-job-used)), and `ccdb provenance` checks the record against a database. | | | **R1.3** domain-relevant community standards | Parquet, Arrow, JSON Schema, OpenAPI and Atom; a [JANA2 service](apis/jana.md) and [DD4hep plugins](apis/dd4hep.md) for the frameworks of nuclear and particle physics. | | ## Data cards @@ -34,8 +34,14 @@ table's versions page); the HTTP API serves it as JSON at `/api/v1/tables/ ```bash curl https://ccdb.example.org/api/v1/tables/epic/HCal/adc_calib/card +curl https://ccdb.example.org/api/v1/tables/epic/HCal/adc_calib/card?format=jsonld ``` +The `jsonld` form is a schema.org `Dataset`: `name` and `identifier` (the table path), `description`, `url`, +`creator` (the authors of its versions), `dateModified`, `license`, `citation`, `keywords`, and one +`variableMeasured` per column with its units. The GUI's card page embeds the same record in a +`', html, re.S) self.assertIsNotNone(block) record = json.loads(block.group(1)) self.assertEqual(record["@type"], "Dataset") self.assertEqual(record["name"], "/test/test_vars/test_table") - self.assertTrue(record["url"].endswith("/tables/test/test_vars/test_table/card")) + self.assertEqual(record["url"], "https://ccdb.example.org/tables/test/test_vars/test_table/card") def test_health_reports_a_dead_database_as_503_and_recovers(self): import json From 3021116365b251de04392a3464dab9281893d59e Mon Sep 17 00:00:00 2001 From: Dmitry Romanov Date: Tue, 3 Nov 2026 21:12:14 -0500 Subject: [PATCH 27/27] Docs: the GlueX AI-readiness study cited by its arXiv authors and id (arXiv:2609.34987) Co-Authored-By: Claude Opus 5.5 --- ccdb-ai-ready.md | 2 +- docs/concepts/ai-readiness.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/ccdb-ai-ready.md b/ccdb-ai-ready.md index 4c1936a3..ec16d188 100644 --- a/ccdb-ai-ready.md +++ b/ccdb-ai-ready.md @@ -39,7 +39,7 @@ a library call over schema 6 as it is. Make CCDB 2.5 the calibration database whose readiness for AI use can be measured, whose constants carry their provenance into every job that reads them, and whose tags name a processing pass completely. The source of the requirements is the assessment framework of the -GlueX AI-readiness article (Panta, Morean, Romanov, 2026): seven categories, 26 sub-criteria, and +GlueX AI-readiness article (Panta, Sawatzky, Morean, Romanov, Higinbotham, 2026, arXiv:2609.34987): seven categories, 26 sub-criteria, and the rule that every readiness claim is reproducible, comparable and defensible. The report `{space}/reports/ai-readiness.md` maps the article onto CCDB and lists the code changes; this plan implements changes 1, 2, 3, 4, 5, 6, 8 and 10 of that list. Changes 7 (value expectations on diff --git a/docs/concepts/ai-readiness.md b/docs/concepts/ai-readiness.md index 0c58a655..404d3879 100644 --- a/docs/concepts/ai-readiness.md +++ b/docs/concepts/ai-readiness.md @@ -5,8 +5,8 @@ constants, understand them without a human, trust where they came from, and meas numbers that come out the same every time. CCDB 2.5 covers each of these in a feature of its own, and `ccdb db readiness` measures how far an instance has come. -The requirements follow the assessment framework of the GlueX AI-readiness study (Panta, Morean, -Romanov, 2026): seven categories of criteria, and the rule that a readiness claim is **reproducible** +The requirements follow the assessment framework of the GlueX AI-readiness study (Panta, Sawatzky, +Morean, Romanov, Higinbotham, 2026, [arXiv:2609.34987](https://arxiv.org/abs/2609.34987)): seven categories of criteria, and the rule that a readiness claim is **reproducible** (the same inputs give the same numbers), **comparable** (across instances and across time) and **defensible** (every number traces to a query or a recorded rule). [FAIR data](concepts/fair.md) maps the FAIR principles; this page maps readiness.