Repository navigation
Chunk processing [merge after additional variables] - #44
Conversation
YooSunYoung
left a comment
There was a problem hiding this comment.
Just minor comments
| def iter_event_data(self, variables, component_name=None, chunk_size=None): | ||
| """Yield requested event data in bounded chunks.""" | ||
| _validate_chunk_size(chunk_size) | ||
| if chunk_size is None: | ||
| raise ValueError("chunk_size is required when iterating event data") |
There was a problem hiding this comment.
| def iter_event_data(self, variables, component_name=None, chunk_size=None): | |
| """Yield requested event data in bounded chunks.""" | |
| _validate_chunk_size(chunk_size) | |
| if chunk_size is None: | |
| raise ValueError("chunk_size is required when iterating event data") | |
| def iter_event_data(self, variables, *, component_name=None, chunk_size): | |
| """Yield requested event data in bounded chunks.""" | |
| _validate_chunk_size(chunk_size) |
If the chucnk_size is a must have keyword, maybe it's easier to not have the default value then it doesn't need the validation.
There was a problem hiding this comment.
Yes thats a bit silly, that was changed and the doc string improved
| if chunk_size is not None: | ||
| chunks = {var: [] for var in variables} | ||
| for event_data in self.iter_event_data( | ||
| variables, component_name, chunk_size | ||
| ): | ||
| for var in variables: | ||
| chunks[var].append(event_data[var]) | ||
| return { | ||
| var: np.concatenate(values) if values else np.empty(0) | ||
| for var, values in chunks.items() | ||
| } | ||
|
|
There was a problem hiding this comment.
Why do we need to iterate chunk by chunk here?
There was a problem hiding this comment.
Not really, now it just set the chunk size to the whole array inside the get_event_data path to reuse the logic in the iterator
| for var in variables | ||
| } | ||
|
|
||
| def get_event_data(self, variables, component_name=None, *, chunk_size=None): |
There was a problem hiding this comment.
chunk_size may not be so useful here?
There was a problem hiding this comment.
No thats true, it was a bit messy. This function, get_event_data is the simple version thats not chunked and the iter_event_data is the chunked version. They have been refactored a bit to share functionality they both need.
YooSunYoung
left a comment
There was a problem hiding this comment.
It looks like this branch needs to be rebased... I'll wait a bit for that...!
| data = Data.__new__(Data) | ||
| data.component_pixel_order = ["Square_1"] | ||
| data.pixel_range = {"Square_1": [0, 0]} | ||
| data._iter_event_chunks = lambda *args: iter(()) | ||
| data.get_id_to_global_coordinates = lambda component_name=None: np.zeros((1, 3)) | ||
| data.get_global_component_coordinates = lambda component_name: np.zeros(3) |
There was a problem hiding this comment.
I think overwriting methods in tests is not recommended unless you really have to use Mock object/methods.
Can we just make an empty file in a temp_path and use it instead?
Understood that it would be desirable to include chunk loading and exporting, I believe this is a possible implementation of the feature, it is quite manual, perhaps there are better built in tools that would improve the performance.
This branch is made from the additional variables branch, so it should be merged first.