Repository navigation
fix(python): support buffer and line reads on S3 readers - #12452
pcbeingused333 wants to merge 2 commits into
Conversation
Signed-off-by: Alex Castillo González <alex.castillog33@gmail.com>
Signed-off-by: Alex Castillo González <alex.castillog33@gmail.com>
|
Thank you for taking the time to open this PR. Please note that we are not seeking outside contribution at this time. Small, trivially-verified PRs that fix a problem are still accepted, but low-value PRs (e.g. typo fixes) and PRs longer than a dozen or so lines will be closed. If you have a bigger idea, please open a feature request instead. See CONTRIBUTING.md for the full policy. If your PR falls within that scope, we ask that you sign our Contributor License Agreement before we can accept it. You can sign the CLA by just posting a Pull Request Comment same as the below format. I have read the CLA Document and I hereby sign the CLA You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
There was a problem hiding this comment.
1 issue found across 2 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="python-client/wmill/wmill/s3_reader.py">
<violation number="1" location="python-client/wmill/wmill/s3_reader.py:93">
P3: `readline(None)` now raises `TypeError`, unlike `BufferedReader.readline` and this class’s `read(None)`. Normalize `None` to `-1` before the size comparisons.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
| view[:len(data)] = data | ||
| return len(data) | ||
|
|
||
| def readline(self, size=-1): |
There was a problem hiding this comment.
P3: readline(None) now raises TypeError, unlike BufferedReader.readline and this class’s read(None). Normalize None to -1 before the size comparisons.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At python-client/wmill/wmill/s3_reader.py, line 93:
<comment>`readline(None)` now raises `TypeError`, unlike `BufferedReader.readline` and this class’s `read(None)`. Normalize `None` to `-1` before the size comparisons.</comment>
<file context>
@@ -79,6 +79,44 @@ def read(self, size=-1):
+ view[:len(data)] = data
+ return len(data)
+
+ def readline(self, size=-1):
+ start = 0
+ while True:
</file context>
| def readline(self, size=-1): | |
| def readline(self, size=-1): | |
| if size is None: | |
| size = -1 |
|
Thanks a lot for tracking this down and for the thorough tests! We took a slightly different route in #12456: instead of overriding the missing methods one by one, the reader now passes the HTTP stream to the standard |
Reading an S3 CSV line by line with
for line in readerorreadline()raisesValueError: I/O operation on uninitialized object. Reading into a reusable buffer withreadinto()fails the same way, as doeshashlib.file_digest(reader, "sha256").S3BufferedReadermanages its own HTTP stream and byte buffer without initializing the inheritedBufferedReader.Override these read operations to use the existing buffer and
_fill()method, and report the reader as readable. Line reads stop at a newline or size limit and preserve the remaining bytes without draining the entire object. Reject read-only destinations before consuming data.Regression tests in
test_s3_reader.pyuse the existing fake HTTP stream and client. They cover partial buffer reads, memoryviews, EOF, empty destinations, hashing, limited line reads across chunks, and incremental CSV line iteration. On main, five regressions fail with the uninitialized-object error; with the fix, all ten tests pass.This change was prepared with assistance from Codex.
Summary by cubic
Fixes
S3BufferedReadersoreadinto(),readline(), and line iteration work instead of raisingValueError: I/O operation on uninitialized object. The reader now uses its existing buffer and_fill()method, reports itself asreadable(), and rejects read-only destinations before consuming data.Adds five regression tests covering partial buffer reads, memoryviews, EOF, empty destinations, hashing, limited line reads across chunks, and incremental CSV line iteration.
Written for commit da8d113. Summary will update on new commits.