Skip to content

Preserve encoding and error handling of the replaced streams - #15059

Open
SemTiOne wants to merge 11 commits into
pytest-dev:mainfrom
SemTiOne:fix/4389-capture-encoding
Open

SemTiOne wants to merge 11 commits into
pytest-dev:mainfrom
SemTiOne:fix/4389-capture-encoding

Conversation

@SemTiOne

@SemTiOne SemTiOne commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Captured sys.stdout and sys.stderr now use the encoding and error handler of the streams they replace. When either setting is missing or is not a string, capture falls back to UTF-8 and strict.

Text writes use those settings. Bytes written directly through .buffer are decoded using the selected encoding when captured output is read back, with undecodable bytes replaced.

stdin capture is unchanged. Binary fixtures still return bytes; text written through them is encoded with the replaced streams' encoding and error handler.

Closes #4389.

  • Allow maintainers to push and squash when merging my commits. Please uncheck this if you prefer to squash the commits yourself.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
@pyba-chronographer pyba-chronographer Bot added the bot:chronographer:provided (automation) changelog entry is part of PR label Sep 18, 2026
@SemTiOne
SemTiOne marked this pull request as ready for review November 6, 2026 14:41
@SemTiOne SemTiOne changed the title [WIP] Fix captured streams reporting UTF-8 instead of original encoding Fix captured streams reporting UTF-8 instead of original encoding Nov 6, 2026

Copy link
Copy Markdown
Member

🤖 Written by Claude Opus 5.5 via Claude Code for the pytest maintainers; I prompted it, it did the work, I read it.

This makes .encoding report the original stream's codec while the buffer still encodes and decodes UTF-8. The attribute then describes a codec the stream does not use. The tests only print() text, which never reads the attribute. One prints 日本語 under PYTHONIOENCODING=latin-1 and expects it to pass, which the real stream would refuse.

#4389 is about code that acts on the attribute. With PYTHONIOENCODING=latin-1:

import sys

def test_it(capsys):
    sys.stdout.buffer.write("café\n".encode(sys.stdout.encoding))
    sys.stdout.flush()
    assert capsys.readouterr().out == "café\n"
  • main: passes (the attribute says UTF-8 and the buffer is UTF-8).
  • this PR, --capture=sys/tee-sys or capsys: UnicodeDecodeError in CaptureIO.getvalue at teardown, and every following test in the session errors.
  • this PR, --capture=fd without capsys: the output silently becomes caf�.

The attribute and the codec have to agree. That means building the capture stream with the replaced stream's encoding/errors and decoding readback with the same codec. This is how cot.capture does it (design note, fd-level install). It is also the direction of #4398.


Generated by Claude Code

@SemTiOne SemTiOne changed the title Fix captured streams reporting UTF-8 instead of original encoding Align captured stream codec with the replaced stream Nov 7, 2026
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
@SemTiOne SemTiOne changed the title Align captured stream codec with the replaced stream Preserve encoding and error handling of the replaced streams Nov 7, 2026
SemTiOne and others added 2 commits November 7, 2026 08:40
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
@SemTiOne

SemTiOne commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks, I think this addresses your concern. Could you also rerun the ubuntu-pypy3-xdist and ubuntu-py315 jobs? The PyPy job failed in testing/python/raises.py::TestRaises::test_raises_cyclic_reference[with_group], while the Py315 job timed out in testing/test_debugging.py::TestPDB::test_pdb_unittest_postmortem. Both tests are in unchanged files.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided (automation) changelog entry is part of PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

stdout capturing breaks sys.stdout.encoding

2 participants