Skip to content

fix(python): support buffer and line reads on S3 readers - #12452

Closed
pcbeingused333 wants to merge 2 commits into
windmill-labs:mainfrom
pcbeingused333:fix/s3-readinto-c3464
Closed

pcbeingused333 wants to merge 2 commits into
windmill-labs:mainfrom
pcbeingused333:fix/s3-readinto-c3464

Conversation

@pcbeingused333

@pcbeingused333 pcbeingused333 commented Nov 15, 2026 •

Copy link
Copy Markdown

Reading an S3 CSV line by line with for line in reader or readline() raises ValueError: I/O operation on uninitialized object. Reading into a reusable buffer with readinto() fails the same way, as does hashlib.file_digest(reader, "sha256"). S3BufferedReader manages its own HTTP stream and byte buffer without initializing the inherited BufferedReader.

Override these read operations to use the existing buffer and _fill() method, and report the reader as readable. Line reads stop at a newline or size limit and preserve the remaining bytes without draining the entire object. Reject read-only destinations before consuming data.

Regression tests in test_s3_reader.py use the existing fake HTTP stream and client. They cover partial buffer reads, memoryviews, EOF, empty destinations, hashing, limited line reads across chunks, and incremental CSV line iteration. On main, five regressions fail with the uninitialized-object error; with the fix, all ten tests pass.

This change was prepared with assistance from Codex.


Summary by cubic

Fixes S3BufferedReader so readinto(), readline(), and line iteration work instead of raising ValueError: I/O operation on uninitialized object. The reader now uses its existing buffer and _fill() method, reports itself as readable(), and rejects read-only destinations before consuming data.

Adds five regression tests covering partial buffer reads, memoryviews, EOF, empty destinations, hashing, limited line reads across chunks, and incremental CSV line iteration.

Written for commit da8d113. Summary will update on new commits.

View guided diff

Signed-off-by: Alex Castillo González <alex.castillog33@gmail.com>
Signed-off-by: Alex Castillo González <alex.castillog33@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for taking the time to open this PR.

Please note that we are not seeking outside contribution at this time. Small, trivially-verified PRs that fix a problem are still accepted, but low-value PRs (e.g. typo fixes) and PRs longer than a dozen or so lines will be closed. If you have a bigger idea, please open a feature request instead. See CONTRIBUTING.md for the full policy.

If your PR falls within that scope, we ask that you sign our Contributor License Agreement before we can accept it. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 2 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="python-client/wmill/wmill/s3_reader.py">

<violation number="1" location="python-client/wmill/wmill/s3_reader.py:93">
P3: `readline(None)` now raises `TypeError`, unlike `BufferedReader.readline` and this class’s `read(None)`. Normalize `None` to `-1` before the size comparisons.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

view[:len(data)] = data
return len(data)

def readline(self, size=-1):

@cubic-dev-ai cubic-dev-ai Bot Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: readline(None) now raises TypeError, unlike BufferedReader.readline and this class’s read(None). Normalize None to -1 before the size comparisons.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At python-client/wmill/wmill/s3_reader.py, line 93:

<comment>`readline(None)` now raises `TypeError`, unlike `BufferedReader.readline` and this class’s `read(None)`. Normalize `None` to `-1` before the size comparisons.</comment>

<file context>
@@ -79,6 +79,44 @@ def read(self, size=-1):
+        view[:len(data)] = data
+        return len(data)
+
+    def readline(self, size=-1):
+        start = 0
+        while True:
</file context>
Suggested change
def readline(self, size=-1):
def readline(self, size=-1):
if size is None:
size = -1
Fix with cubic

@rubenfiszel

Copy link
Copy Markdown
Contributor

Thanks a lot for tracking this down and for the thorough tests! We took a slightly different route in #12456: instead of overriding the missing methods one by one, the reader now passes the HTTP stream to the standard BufferedReader, so every method works, including text mode (csv.reader(io.TextIOWrapper(reader))), which was still failing here. Since that replaces this approach, we're closing this one. Also, a reminder for next time: please sign the CLA via the CLAassistant link on the PR so we can merge your contributions directly.

@github-actions github-actions Bot locked and limited conversation to collaborators Nov 16, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants