Skip to content

fix(python): back S3BufferedReader with a real BufferedReader - #12456

Merged
rubenfiszel merged 1 commit into
mainfrom
fix/python-s3-reader-real-buffered-reader
Nov 16, 2026
Merged

rubenfiszel merged 1 commit into
mainfrom
fix/python-s3-reader-real-buffered-reader

Conversation

@rubenfiszel

@rubenfiszel rubenfiszel commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

S3BufferedReader subclasses io.BufferedReader but never calls BufferedReader.__init__, so every method it does not override fails with ValueError: I/O operation on uninitialized object. That covers reading a file from wmill.load_s3_file_reader line by line (for line in reader, readline), into a buffer (readinto, hashlib.file_digest), and in text mode (csv.reader(io.TextIOWrapper(reader))).

Rather than overriding those methods one by one, the HTTP response's chunks now feed a small RawIOBase, and S3BufferedReader passes it to BufferedReader.__init__. The standard implementation then provides the whole file API, and the hand-written read, read1, peek and _fill go away. The behaviours the existing tests pin are the standard ones and still hold: read(0) does not touch the stream, and peek/read1 read it at most once.

Reported in #12452, which overrides the missing methods individually; text mode and readline(None) still failed there.

Changes

  • s3_reader.py: _HttpChunkStream (RawIOBase) yields the response chunks; S3BufferedReader wraps it with super().__init__ and closes itself on __exit__. Reading before with raises a ValueError naming the problem instead of an AttributeError.
  • test_s3_reader.py: text-mode CSV across chunk boundaries, and line/readinto reads that stop without draining the stream. Both fail on main; the 5 existing tests are unchanged.

Test plan

  • PYTHONPATH=. uv run --frozen --python 3.14 pytest tests/: 137 passed.
  • Over a real httpx chunked stream from a local server (4 MB, 100k-row CSV in odd-sized chunks): read(), for line in reader, first line without reading the rest, readline(5), readline(None), a readinto loop, hashlib.file_digest, csv.reader(TextIOWrapper), and closed after the with block all behave as expected. On main only read() worked.
  • Python job on a worker (backend quickjs,private,parquet,python, workspace storage on SeaweedFS) reading a 4.9 MB, 100k-row CSV through wmill.load_s3_file_reader. With the published wmill 1.875.0 only read() works; readline, for line in reader, csv.reader(TextIOWrapper), readinto and hashlib.file_digest all raise I/O operation on uninitialized object. With this s3_reader.py injected into the worker's package cache, all six succeed: 100,001 lines and CSV rows, 4,966,683 bytes through readinto, and a SHA-256 matching the uploaded object.

🤖 Generated with Claude Code


Summary by cubic

Fixes S3BufferedReader so the whole file API works instead of failing with "I/O operation on uninitialized object". The class subclassed io.BufferedReader without initializing it, so only the hand-written read, read1, and peek worked; readline, iteration, readinto, and text mode all broke.

The HTTP response chunks now feed a small RawIOBase, and S3BufferedReader passes it to BufferedReader.__init__, which provides the standard implementation and replaces the hand-written read methods. Adds tests for text-mode CSV across chunk boundaries and reads that stop without draining the stream; both fail on main.

Written for commit f5ba0b5. Summary will update on new commits.

View guided diff Turn on auto-fix

S3BufferedReader subclassed BufferedReader without initializing it, so
every method it did not override (readline, iteration, readinto, closed,
TextIOWrapper) failed with "I/O operation on uninitialized object".
The HTTP chunks now feed a RawIOBase that BufferedReader wraps, which
gives the whole file API and replaces the hand-written read/read1/peek.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rubenfiszel

Copy link
Copy Markdown
Contributor Author

/review

@claude

claude Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rubenfiszel's task in 50s —— View job


Good to merge

Review checklist

  • Read the PR diff (git diff origin/main...HEAD) and AGENTS.md
  • Read the new s3_reader.py and tests/test_s3_reader.py, and checked the callers in client.py
  • Checked the change for correctness, behaviour changes and AGENTS.md compliance
  • Ran pytest tests/test_s3_reader.py: not run, because this runner's permissions blocked python3 -m pytest

Findings

None at P0, P1 or P2.

What I checked:

  • _HttpChunkStream.readinto fills buffer from a memoryview of the pending chunk. It loops past empty chunks, so an empty b"" from iter_bytes() does not look like EOF. It returns 0 only when the iterator is exhausted. This is the RawIOBase contract.
  • Existing tests still hold under the standard BufferedReader:
    • read(0) does not call raw.readinto.
    • peek reads the raw stream at most once.
    • read1 reads it at most once.
    • read(-1) goes through RawIOBase.readall.
  • Order in __init__ is fine. super().__init__(self._raw) runs before windmill_client.stream(...), and BufferedReader.__init__ does not read from the raw stream. Reading before with now hits the ValueError guard in readinto instead of an AttributeError.
  • __exit__ closes the reader first, then exits the httpx context. BufferedReader.close() also closes _raw, and both are safe to call more than once. The 4xx path in __enter__ is unchanged.
  • Callers keep working. load_s3_file, at client.py:1072, only calls .read(), and the BufferedReader return annotation on load_s3_file_reader is now actually true.
  • Memory: for a full read(), readall collects the data in pieces of about DEFAULT_BUFFER_SIZE and then joins them. The old _fill loop also made a copy, so peak memory is about the same.

AGENTS.md compliance

  • The one new comment, in S3BufferedReader.__init__, explains why the class must stay a real BufferedReader. It gives a reason rather than narrating the code, which follows "Comments record constraints, not narration."
  • The two new tests each pin a behaviour that fails on main: text-mode CSV across chunk boundaries, and line/readinto reads that stop without draining the stream. No other test code was added, which follows "Ship only the tests the PR needs."

Test coverage

  • Unit tests: coverage fits the change. It is a Python client fix, and the new tests cover the paths that were broken: TextIOWrapper, iterating over lines, and readinto.
  • Still to check by hand: the unchecked item in the test plan. On a worker with workspace object storage configured, run a Python job that does
    with wmill.load_s3_file_reader(obj) as r:
        rows = list(csv.reader(io.TextIOWrapper(r, encoding="utf-8")))
    on a CSV larger than one HTTP chunk (more than 64 KB). It passes if the job returns every row, with no row split at a chunk boundary, and without ValueError: I/O operation on uninitialized object. Also run for line in r: with a break after the first line: the job should finish quickly, without downloading the whole object.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying windmill with  Cloudflare Pages  Cloudflare Pages

Latest commit: f5ba0b5
Status: ✅  Deploy successful!
Preview URL: https://b904bf23.windmill.pages.dev
Branch Preview URL: https://fix-python-s3-reader-real-bu.windmill.pages.dev

View logs

@github-actions

Copy link
Copy Markdown
Contributor

Codex Review

Good to merge

No confident introduced issues found.

Test coverage

Independently ran the Python SDK suite on Python 3.12.3: 137 passed. The PR reports the same result on Python 3.14. The added tests cover CSV parsing across chunks and bounded line/buffer reads.

Also verified reads, iteration, readline, readinto, hashlib.file_digest, text-mode CSV, early exit and closed state against a real localhost chunked HTTP stream.

The PR’s unchecked worker test remains needed: run a Python job using wmill.load_s3_file_reader against actual workspace object storage and confirm correct content through line or text-mode reads. That integration path was not exercised during this review.

@github-actions

Copy link
Copy Markdown
Contributor

Pi Review

Good to merge

This is a clean, well-scoped fix. S3BufferedReader previously subclassed io.BufferedReader without ever calling super().__init__, so only its hand-written read/read1/peek worked and the rest of the file API blew up on an uninitialized object. The PR backs it with a real RawIOBase (_HttpChunkStream) and lets the standard BufferedReader provide readline, iteration, readinto, and text mode. I read the full before/after of s3_reader.py, the new tests, and the callers in client.py; I also ran the test file locally (7 passed on Python 3.12). No P0/P1/P2 findings.

Considered and dismissed

  • readinto with an empty buffer pulls from the chunk iterator — _HttpChunkStream.readinto advances the iterator before computing n, so an empty buffer would consume a chunk as a side effect. Not reachable: BufferedReader never issues a zero-length readinto, and the raw stream is private to this class.
  • readable() returns True unconditionally (even after close()) — BufferedReader consults readable() only at construction; closed-ness is tracked on BufferedReader/RawIOBase itself, so this has no observable effect. Also close() is never overridden to flip it, but nothing in the read path re-checks it.
  • peek(size)/read1 semantics changed vs the old hand-written methods — the old peek(size) ignored size and returned everything buffered; standard BufferedReader.peek honors it. This is the intended fix (and the documented "reads at most once" behavior still holds). No in-repo caller uses peek/read1 on this reader, and the existing tests pin the new, standard semantics.
  • __exit__ now calls self.close() before the httpx context manager exits — close() on a read-only buffered reader only flushes (no-op) and closes the raw, so it cannot raise and therefore cannot leak the httpx response.
  • read(-1) buffers the whole object — pre-existing behavior, not a regression; the reader still streams for sized reads.
  • memoryview slicing retains the last chunk until the next pull or GC — at most one ~64KB httpx chunk is retained; negligible.
  • __enter__ error path (status ≥ 400) leaks the entered httpx response because it raises before the with machinery can call __exit__ — this is pre-existing (unchanged by this PR) and out of scope here.

Test coverage

  • Python client logic — appropriate. Two new tests cover the previously broken paths (test_text_mode_csv_reads_rows_across_chunks, test_line_and_buffer_reads_stop_where_asked), and the five pre-existing tests pin read(0), read(-1), peek, read1, and bytes_generator semantics against the standard implementation. The CountingIterator assertions verify the "reads the stream at most once / stops without draining" contract is still met. No further unit tests needed for this change.

Manual verification still needed

The PR's remaining unchecked item is the one real end-to-end gap: run a Python job on an actual worker that reads an object from workspace storage through wmill.load_s3_file_reader (e.g. readline loop and io.TextIOWrapper/csv.reader over the streamed response), and confirm the bytes come back intact and closed is True after the with block. The author already exercised the equivalent path against a local httpx chunked server (4 MB, odd-sized chunks), so this is a confirmation of the worker wiring rather than of the reader logic itself.

@rubenfiszel

Copy link
Copy Markdown
Contributor Author

✅ Review round clean @ f5ba0b5

@rubenfiszel
rubenfiszel marked this pull request as ready for review November 16, 2026 09:25
@rubenfiszel
rubenfiszel merged commit 9a074b1 into main Nov 16, 2026
20 checks passed
@rubenfiszel
rubenfiszel deleted the fix/python-s3-reader-real-buffered-reader branch November 16, 2026 09:30
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 16, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant