Skip to content

Chunk processing [merge after additional variables] - #44

Merged
mads-bertelsen merged 6 commits into
mccode-dev:mainfrom
mads-bertelsen-agentic:chunk_processing
Oct 13, 2026
Merged

mads-bertelsen merged 6 commits into
mccode-dev:mainfrom
mads-bertelsen-agentic:chunk_processing

Conversation

@mads-bertelsen-agentic

Copy link
Copy Markdown
Contributor

Understood that it would be desirable to include chunk loading and exporting, I believe this is a possible implementation of the feature, it is quite manual, perhaps there are better built in tools that would improve the performance.

This branch is made from the additional variables branch, so it should be merged first.

@YooSunYoung YooSunYoung left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just minor comments

Comment thread src/mcstastox/ReadNeXus.py Outdated
Comment on lines +583 to +587
def iter_event_data(self, variables, component_name=None, chunk_size=None):
"""Yield requested event data in bounded chunks."""
_validate_chunk_size(chunk_size)
if chunk_size is None:
raise ValueError("chunk_size is required when iterating event data")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def iter_event_data(self, variables, component_name=None, chunk_size=None):
"""Yield requested event data in bounded chunks."""
_validate_chunk_size(chunk_size)
if chunk_size is None:
raise ValueError("chunk_size is required when iterating event data")
def iter_event_data(self, variables, *, component_name=None, chunk_size):
"""Yield requested event data in bounded chunks."""
_validate_chunk_size(chunk_size)

If the chucnk_size is a must have keyword, maybe it's easier to not have the default value then it doesn't need the validation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes thats a bit silly, that was changed and the doc string improved

Comment thread src/mcstastox/ReadNeXus.py Outdated
Comment on lines +621 to +632
if chunk_size is not None:
chunks = {var: [] for var in variables}
for event_data in self.iter_event_data(
variables, component_name, chunk_size
):
for var in variables:
chunks[var].append(event_data[var])
return {
var: np.concatenate(values) if values else np.empty(0)
for var, values in chunks.items()
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to iterate chunk by chunk here?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really, now it just set the chunk size to the whole array inside the get_event_data path to reuse the logic in the iterator

Comment thread src/mcstastox/ReadNeXus.py Outdated
for var in variables
}

def get_event_data(self, variables, component_name=None, *, chunk_size=None):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

chunk_size may not be so useful here?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No thats true, it was a bit messy. This function, get_event_data is the simple version thats not chunked and the iter_event_data is the chunked version. They have been refactored a bit to share functionality they both need.

@YooSunYoung YooSunYoung left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this branch needs to be rebased... I'll wait a bit for that...!

Comment thread tests/chunk_processing_test.py Outdated
Comment on lines +187 to +192
data = Data.__new__(Data)
data.component_pixel_order = ["Square_1"]
data.pixel_range = {"Square_1": [0, 0]}
data._iter_event_chunks = lambda *args: iter(())
data.get_id_to_global_coordinates = lambda component_name=None: np.zeros((1, 3))
data.get_global_component_coordinates = lambda component_name: np.zeros(3)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think overwriting methods in tests is not recommended unless you really have to use Mock object/methods.

Can we just make an empty file in a temp_path and use it instead?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, fixed!

@mads-bertelsen
mads-bertelsen merged commit 2c073db into mccode-dev:main Oct 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants