Repository navigation
fix(python): back S3BufferedReader with a real BufferedReader - #12456
Conversation
S3BufferedReader subclassed BufferedReader without initializing it, so every method it did not override (readline, iteration, readinto, closed, TextIOWrapper) failed with "I/O operation on uninitialized object". The HTTP chunks now feed a RawIOBase that BufferedReader wraps, which gives the whole file API and replaces the hand-written read/read1/peek. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
/review |
|
Claude finished @rubenfiszel's task in 50s —— View job Good to merge Review checklist
FindingsNone at P0, P1 or P2. What I checked:
AGENTS.md compliance
Test coverage
|
Deploying windmill with
|
| Latest commit: |
f5ba0b5
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://b904bf23.windmill.pages.dev |
| Branch Preview URL: | https://fix-python-s3-reader-real-bu.windmill.pages.dev |
Codex ReviewGood to merge No confident introduced issues found. Test coverageIndependently ran the Python SDK suite on Python 3.12.3: 137 passed. The PR reports the same result on Python 3.14. The added tests cover CSV parsing across chunks and bounded line/buffer reads. Also verified reads, iteration, The PR’s unchecked worker test remains needed: run a Python job using |
Pi ReviewGood to merge This is a clean, well-scoped fix. Considered and dismissed
Test coverage
Manual verification still needed The PR's remaining unchecked item is the one real end-to-end gap: run a Python job on an actual worker that reads an object from workspace storage through |
|
✅ Review round clean @ f5ba0b5 |
Summary
S3BufferedReadersubclassesio.BufferedReaderbut never callsBufferedReader.__init__, so every method it does not override fails withValueError: I/O operation on uninitialized object. That covers reading a file fromwmill.load_s3_file_readerline by line (for line in reader,readline), into a buffer (readinto,hashlib.file_digest), and in text mode (csv.reader(io.TextIOWrapper(reader))).Rather than overriding those methods one by one, the HTTP response's chunks now feed a small
RawIOBase, andS3BufferedReaderpasses it toBufferedReader.__init__. The standard implementation then provides the whole file API, and the hand-writtenread,read1,peekand_fillgo away. The behaviours the existing tests pin are the standard ones and still hold:read(0)does not touch the stream, andpeek/read1read it at most once.Reported in #12452, which overrides the missing methods individually; text mode and
readline(None)still failed there.Changes
s3_reader.py:_HttpChunkStream(RawIOBase) yields the response chunks;S3BufferedReaderwraps it withsuper().__init__and closes itself on__exit__. Reading beforewithraises aValueErrornaming the problem instead of anAttributeError.test_s3_reader.py: text-mode CSV across chunk boundaries, and line/readintoreads that stop without draining the stream. Both fail onmain; the 5 existing tests are unchanged.Test plan
PYTHONPATH=. uv run --frozen --python 3.14 pytest tests/: 137 passed.httpxchunked stream from a local server (4 MB, 100k-row CSV in odd-sized chunks):read(),for line in reader, first line without reading the rest,readline(5),readline(None), areadintoloop,hashlib.file_digest,csv.reader(TextIOWrapper), andclosedafter thewithblock all behave as expected. Onmainonlyread()worked.quickjs,private,parquet,python, workspace storage on SeaweedFS) reading a 4.9 MB, 100k-row CSV throughwmill.load_s3_file_reader. With the publishedwmill1.875.0 onlyread()works;readline,for line in reader,csv.reader(TextIOWrapper),readintoandhashlib.file_digestall raiseI/O operation on uninitialized object. With thiss3_reader.pyinjected into the worker's package cache, all six succeed: 100,001 lines and CSV rows, 4,966,683 bytes throughreadinto, and a SHA-256 matching the uploaded object.🤖 Generated with Claude Code
Summary by cubic
Fixes
S3BufferedReaderso the whole file API works instead of failing with "I/O operation on uninitialized object". The class subclassedio.BufferedReaderwithout initializing it, so only the hand-writtenread,read1, andpeekworked;readline, iteration,readinto, and text mode all broke.The HTTP response chunks now feed a small
RawIOBase, andS3BufferedReaderpasses it toBufferedReader.__init__, which provides the standard implementation and replaces the hand-written read methods. Adds tests for text-mode CSV across chunk boundaries and reads that stop without draining the stream; both fail onmain.Written for commit f5ba0b5. Summary will update on new commits.