Repository navigation
Stream sampling [merge after chunk loading] #45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
mads-bertelsen
merged 11 commits into
mccode-dev:main
from
mads-bertelsen-agentic:stream_sampling
Oct 16, 2026
Merged
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
c10bdfa
Add stream sampling to scipp exports
mads-bertelsen-agentic ed7a36c
Add effective duration to sampled exports
mads-bertelsen-agentic abae101
Rename variable transfers
mads-bertelsen-agentic 554734c
Propagate sampling duration variance
mads-bertelsen-agentic dd73486
Document and validate sampled event data
mads-bertelsen-agentic 6e52fb3
Generate McStas fixture for chunk tests
mads-bertelsen-agentic c44bec5
Avoid parallel documentation database collisions
mads-bertelsen-agentic d89703e
Merge chunk_processing into stream_sampling
mads-bertelsen-agentic 95a3d9d
Merge branch 'main' into stream_sampling
mads-bertelsen-agentic d636a3a
Normalize notebook formatting
mads-bertelsen-agentic 7c819d7
Support custom sampling weight keys
mads-bertelsen-agentic File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,4 +8,5 @@ maxdepth: 1 | |
| installation | ||
| overall_demo.ipynb | ||
| scipp_demo.ipynb | ||
| stream_sampling_demo.ipynb | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,244 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "0", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Weighted event sampling to `scipp`\n", | ||
| "\n", | ||
| "McStas event files contain weighted events. This notebook focuses on converting those weighted events to a fixed number of normal, unit-weight events while exporting to the simple `scipp` representation." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "1", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Imports" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "2", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from pathlib import Path\n", | ||
| "\n", | ||
| "import numpy as np\n", | ||
| "\n", | ||
| "import mcstastox\n", | ||
| "from read_example import make_instrument" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "3", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Run a small McStas simulation\n", | ||
| "\n", | ||
| "As in the other user-guide notebooks, use `read_example` to create a small instrument and generate a McStas NeXus event file. The `Read` interface takes the output folder, while McStas writes the file as `mccode.h5` inside that folder." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "4", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "instr = make_instrument()\n", | ||
| "instr.set_parameters(wavelength=1.8, delta_wavelength=1.3)\n", | ||
| "instr.settings(ncount=1E6)\n", | ||
| "data = instr.backengine()\n", | ||
| "data_folder = Path(data[0].original_data_location)\n", | ||
| "data_folder" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "5", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Inspect the weighted `scipp_simple` export\n", | ||
| "\n", | ||
| "`export_scipp_simple` returns one `scipp.DataArray`. Before sampling, its data values are the McStas event weights (`p`). Reading in chunks avoids assembling the complete event file while it is loaded." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "6", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "chunk_size = 1_000\n", | ||
| "wavelength = mcstastox.Transfer(\n", | ||
| " mcstas_variable=\"L\",\n", | ||
| " scipp_coord=\"sim_wavelength\",\n", | ||
| " unit=\"angstrom\",\n", | ||
| ")\n", | ||
| "\n", | ||
| "with mcstastox.Read(data_folder) as loaded_data:\n", | ||
| " weighted = loaded_data.export_scipp_simple(\n", | ||
| " source_name=\"source\",\n", | ||
| " sample_name=\"sample_position\",\n", | ||
| " component_name=\"Square_1\",\n", | ||
| " extra_variables=wavelength,\n", | ||
| " chunk_size=chunk_size,\n", | ||
| " )\n", | ||
| "\n", | ||
| "weighted" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "7", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "weighted.sizes, weighted.values[:5], weighted.coords[\"sim_wavelength\"].values[:5]" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "8", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Configure sampling\n", | ||
| "\n", | ||
| "`SamplingSettings` uses weighted reservoir sampling with replacement. The result contains exactly `n_samples` events, so an input event may appear more than once. A seed makes the result reproducible. The weight column defaults to `p`; use `weight_key` when the input stores weights under another name.\n", | ||
| "\n", | ||
| "Here, the **input stream** is the sequence of event records read from the selected McStas component. Each record contains the configured weight column (`p` by default), time (`t`), pixel ID (`id`), and any requested extra variables. With chunked loading, the stream is the same logical sequence as a full read: rows are visited in file order, and selected components are visited in component order.\n", | ||
| "\n", | ||
| "By default, the reservoir is returned in its internal sampling order, so the output order is not meaningful. Set `ordered=True` to retain each sampled record's position in the input stream and sort the result back into that order. This changes only the arrangement of the sampled records, not which weighted sample is selected." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "9", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "sampling = mcstastox.SamplingSettings(\n", | ||
| " n_samples=1_000,\n", | ||
| " seed=42,\n", | ||
| " ordered=True,\n", | ||
| ")\n", | ||
| "sampling" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "10", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Export sampled normal events\n", | ||
| "\n", | ||
| "Pass the settings with the `sampling=` keyword. Sampling happens while the input chunks are read, and the exported data values are all one count. Extra event coordinates are sampled with the same events and remain aligned. The output also has a scalar `effective_duration` coordinate: when the configured weight column is a detector count rate, it is `n_samples / sum(weight)` seconds, with variance propagated from `sum(weight**2)`." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "11", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "with mcstastox.Read(data_folder) as loaded_data:\n", | ||
| " sampled = loaded_data.export_scipp_simple(\n", | ||
| " source_name=\"source\",\n", | ||
| " sample_name=\"sample_position\",\n", | ||
| " component_name=\"Square_1\",\n", | ||
| " extra_variables=wavelength,\n", | ||
| " chunk_size=chunk_size,\n", | ||
| " sampling=sampling,\n", | ||
| " )\n", | ||
| "\n", | ||
| "sampled" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "12", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "assert sampled.sizes[\"events\"] == sampling.n_samples\n", | ||
| "np.testing.assert_array_equal(sampled.values, np.ones(sampling.n_samples))\n", | ||
| "sampled.coords[\"effective_duration\"]\n", | ||
| "\n", | ||
| "sampled.sizes, sampled.values[:5], sampled.coords[\"sim_wavelength\"].values[:5]" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "13", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "The same settings can be used to reproduce the same event selection. Change `seed` to obtain another statistically weighted sample." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "14", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "with mcstastox.Read(data_folder) as loaded_data:\n", | ||
| " sampled_again = loaded_data.export_scipp_simple(\n", | ||
| " source_name=\"source\",\n", | ||
| " sample_name=\"sample_position\",\n", | ||
| " component_name=\"Square_1\",\n", | ||
| " extra_variables=wavelength,\n", | ||
| " chunk_size=chunk_size,\n", | ||
| " sampling=sampling,\n", | ||
| " )\n", | ||
| "\n", | ||
| "np.testing.assert_array_equal(sampled.values, sampled_again.values)\n", | ||
| "np.testing.assert_array_equal(\n", | ||
| " sampled.coords[\"t\"].values, sampled_again.coords[\"t\"].values\n", | ||
| ")\n", | ||
| "print(\"Sampling is reproducible with the same seed.\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "15", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "sampled" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3 (ipykernel)", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.12.13" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 5 | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this to make the instrument unique?
Then we can use
uuidinstead maybe.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since multiple documentation notebooks build that instrument and execute it in parallel, it is to avoid the data files being named the same. In sequence, it is detected and avoided, but not in parallel.
I think this problem can be solved in a nicer way by only making the data folder name funny, not also the instrument name. Would be nice to merge and get it in a later round!