Repository navigation
Type put destinations and document path addressing (GH-1404) - #1426
Open
laughingman7743 wants to merge 2 commits into
Open
laughingman7743 wants to merge 2 commits into
laughingman7743 wants to merge 2 commits into
Conversation
put() and the async _put() named the destination of a local file by joining its name to the destination string, which was parsed again: a file named x?versionId=v1 was taken for version v1 of the key x, and the upload raised that a version cannot be written. - _put_pairs() returns an S3Path for a destination named after a local file: the destination root parsed, with the rest of the local name added to its key as it is (S3Path.with_key_suffix(), which s3_copy_pairs() of the pairing uses as well). A destination that is the root itself, or one of a list, stays the caller's string. - put_file() and the async _put_file(), also in a transaction, write an S3Path by its bucket and key; _open_path() opens it with open()'s transaction handling, and get_file() reads through it. - _S3LockedCallback names the branches of S3Path sources and destinations by their strings, so that the async _get() and _put() run fsspec's own loops for lists of paths again; _get_pairs() is removed. - docs/filesystem.md documents the addressing of keys that look like version paths (the S3Path methods, the string compositions, and the path string of a file opened with an S3Path) and of bucket//. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review repairs: - url(), metadata(), getxattr() and setxattr() of a file address its bucket, key and version: sign(), metadata(), getxattr() and setxattr() of the filesystems accept an S3Path. A file opened with the S3Path of a key such as a/x?versionId=null signed and looked up the null version of a/x. Only a pickled copy still reopens the path string, which the docs say. - The async _get() and _put() name the branches of their callback after their paths as strings in a transfer loop of their own, as fsspec runs them, instead of taking the lock of _S3LockedCallback on the event loop, which a callback that calls the filesystem could deadlock. - copy() and put() to a destination root that names a version raise their own messages again, and put() and _put() say so. - _put_pairs() returns early when no destination is named after a local file, put_file() parses its destination once, and the async transaction write invalidates through the filesystem. - The append test of a typed path moves to TestS3File, and the sync put test compares the labels of the callback exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
laughingman7743
commented
Nov 17, 2026
| ) | ||
| ) | ||
|
|
||
| async def _transfer_pairs( |
Member
Author
There was a problem hiding this comment.
Self-review round one (implementation behavior): FINDINGS, repaired in e9bc32d
Scope: the full diff 89ffcd97..522d2cab.
Passes:
/code-reviewat high effort./simplify, with four cleanup agents: reuse (CLEAN), simplification, efficiency (CLEAN) and altitude.
Findings and repairs:
- The async
_get()/_put()took athreading.Lockon the event loop (/code-review). The wrapped callback took it on the loop thread while worker threads held it for their child callbacks. That stalls the loop, and deadlocks it if a child callback calls a sync method of the filesystem, which waits on the loop.- Repair:
_S3LockedCallbackno longer converts labels. The syncget()andput()passstr()of their S3 path, as Act on listed identities in pairing and copy/mv/get (GH-1403) #1421 did. - The async
_get()/_put()run their transfers through_transfer_pairs(), which works as fsspec's loop for lists of paths and labels each branch with the string of its path, without a lock, as before Type put destinations and document path addressing (GH-1404) #1426. It replaces the_get_pairs()of Act on listed identities in pairing and copy/mv/get (GH-1403) #1421. - Mutations of each conversion fail the label tests (sync
put, async_put, async_get).
- Repair:
- The path methods of a file opened with an
S3Pathused its path string (altitude).- For
S3Path("bucket", "a/x?versionId=null"),url(),metadata()andgetxattr()signed and read thenullversion ofa/x, andsetxattr()rejected the file as a version. - The docs listed this as a limit, and named
sign(), whichS3Filedoes not have (its method isurl()). - Repair: the filesystems'
sign(),metadata(),getxattr()andsetxattr()accept anS3Path, andS3Filepasses its bucket, key and version (_s3_path). The docs now name only pickling as the limit. - Test:
TestS3File::test_path_methods_address_path_as_it_is; all four cases fail at 522d2ca.
- For
- The versioned-root errors changed (
/code-review, altitude).copy()of listed sources andput()raisedwith_key_suffix()'s generic message.- Repair:
s3_copy_pairs()raisesCannot copy to a versioned file.again, and_put_pairs()raisesCannot write to the file with the version specified., the message that the string path gave atopen(). TheRaisesofput()and_put()say so.
- Repair:
- Cleanups (simplification):
_put_pairs()returns early when no destination is named after a local file.put_file()and the async transaction write parse their destination once.- The latter invalidates through
self._invalidate().
- Tests (
/code-review):- The append test of a typed path moved to
TestS3File. - The sync
puttest compares the callback's labels exactly (dirincludes the directory'sbucket/up).
- The append test of a typed path moved to
Deferred, with reasons:
- The duplicated upload body of
put_file()and the async transaction write, and the async_put()source check and part windowing: these predate this PR, and are filed as Share the upload body of put_file() with the async transaction write, and check put sources once #1427. - Slicing
other_paths()destinations to derive the suffix. As in Act on listed identities in pairing and copy/mv/get (GH-1403) #1421,test_s3_copy_pairsand theputtests would fail if fsspec named destinations differently. - A
put()of only empty directories under a versioned root now raises. This is intended: such a root names an object. - The
stringify_pathguard repeated input_file/cp_file/get_file. Kept deliberately (Act on listed identities in pairing and copy/mv/get (GH-1403) #1421):stringify_pathwould convert anS3Pathif it ever gained__fspath__.
Validation:
just lint,just docs lintandjust benchmark testpass.- The filesystem suite against AWS, in parts: 1043 + 715 + 103 passed, 6 skipped.
laughingman7743
commented
Nov 17, 2026
| """ | ||
| return self.with_version_id(None) if self.version_id == "null" else self | ||
|
|
||
| def with_key_suffix(self, suffix: str) -> S3Path: |
Member
Author
There was a problem hiding this comment.
Self-review round two (claims, callers, AWS operator, evidence): CLEAN
Scope:
- the full diff
89ffcd97..e9bc32df7202230fbd41f84c03583a8a06e67d7b; - the PR body, the commit messages, the docstrings and the new "Path addressing" section;
- Share the upload body of put_file() with the async transaction write, and check put sources once #1427.
Claims checked:
- The before-state. At 89ffcd9,
putof the filex?versionId=v1raisesValueError: Cannot write to the file with the version specified.This holds for a single file, a recursive directory andmode="create", sync and async, reproduced against the fake store. - "A destination that is the root itself, or one item of a list, stays the caller's string."
_put_pairs()returns the pairs unchanged whenrpathis a list or every destination equals it. Otherwise only the destinations other thanrpathare typed.- So
put(file, "s3://bucket/up/x?versionId=v1")still names that version and fails atopen(), as documented.
- So
- "The async callbacks stay unlocked, as before." At 89ffcd9:
- the async
_get()ran Act on listed identities in pairing and copy/mv/get (GH-1403) #1421's_get_pairs(), without a lock; - the async
_put()passed the callback to fsspec's_put(), without a lock.
_transfer_pairs()keeps both, and callsset_size,branched(..., **kwargs), the transfer and_run_coros_in_chunks(..., callback=)as fsspec 2026.9.0 does. The local directories are created as fsspec's_get()creates them, and_put()filters the directories out before the transfers, as before.
- the async
- The widened path methods behave the same for strings:
sign(),metadata()andgetxattr()parse a string as before.setxattr()now passes its parsed path tometadata()and invalidates it with_invalidate(). Its path never has a version (it raises first), and the name walk evicts at least whatinvalidate_cache()did.- The async
sign,metadata,getxattrandsetxattrhand the path to the sync methods unchanged.
S3Filenow always passesS3Path(bucket, key, version_id)to these methods. For a file opened by a string, that is the parse of itspath, including the pinned?versionId=ofversion_aware, so the requests are the same. No test patches these methods.- The
bucket//section matches the probe at this commit:info→ HeadBucket;cat_file/rm_file/chmod→ the key/;ls/find→ the bucket listing;- a recursive
chmod→ PutObjectAcl of/anda, without PutBucketAcl; pipe_file→ValueError.
- The other docs statements were checked against the code:
- the list of methods that accept an
S3Path, includingS3File; - the compositions through strings (
fsspec.open(),open_files(),get_mapper()); - the pickling limit: fsspec's
__reduce__reopensself.path.
- the list of methods that accept an
Existing callers:
- No public signature was narrowed.
put_file,sign,metadata,getxattrandsetxattr(and the async ones) only accept more.S3Path.with_key_suffix()is new. - Error messages for strings are unchanged.
- The new raise for a versioned
put()root replaces the same message, which each file raised atopen(). It now comes before any upload.
AWS operator:
put()sends the same requests to the literal keys.mode="create"sends HeadObject and a conditional PutObject as before.- The file methods address the key instead of a version.
- No new permission.
Evidence:
laughingman7743
commented
Nov 17, 2026
| """ | ||
| return self.with_version_id(None) if self.version_id == "null" else self | ||
|
|
||
| def with_key_suffix(self, suffix: str) -> S3Path: |
Member
Author
There was a problem hiding this comment.
Independent review (relayed result, Codex): CLEAN
Reviewer and run:
- Reviewer: Codex CLI
codex exec, modelgpt-6-astra, reasoning effortmax, read-only sandbox. Session01a1ea17-a96c-7f72-8213-df19aeecd475. - Scope: a detached snapshot of e9bc32d, reviewing
89ffcd97..e9bc32df7202230fbd41f84c03583a8a06e67d7b. - The prompt stated the agreed contract of Type generated put destinations and document path addressing limits (#1381 step 4) #1404 and the out-of-scope items filed as Share the upload body of put_file() with the async transaction write, and check put sources once #1427, but not the PR, the commit messages or the self-review findings.
- The snapshot and the PR worktree were unchanged afterwards.
- This was a static review: the reviewer ran no tests.
Coverage, in the reviewer's words:
- "
put()/_put()pairing against fsspec 2026.9.0: globs, recursion,maxdepth, lists, trailing slashes, bucket roots and versioned roots." - "Typed upload destinations, create mode and both filesystems' transactions."
- "Async batching, callback labels and contexts, return values, and
_S3BatchTaskscancellation cleanup." - "File and filesystem URL/metadata methods, including
setxattr()lookups and cache invalidation." - "Path-addressing documentation, new regression assertions and existing compatibility coverage."
Result: "CLEAN — No actionable findings."
The independent review is complete. The PR moves to Ready for CI.
laughingman7743
marked this pull request as ready for review
November 17, 2026 15:28
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WHAT
put()now writes a local file to the key named after the file as it is, on both filesystems. This includes a file whose name ends in what looks like a version ID query. The docs now describe how keys that look like version paths, andbucket//, are addressed.Behavior before this change (89ffcd9)
The examples use a local file named
x?versionId=v1:put(<file>, "s3://bucket/up/")built the destinationbucket/up/x?versionId=v1, which was parsed as versionv1ofup/x.ValueError: Cannot write to the file with the version specified.putof its directory, formode="create", and for the async_put().Changes
_put_pairs()returns anS3Pathfor each destination named after a local file. It is the parsed destination root with the rest of the local name added to its key as it is, through the newS3Path.with_key_suffix().with_key_suffix()raisesValueErrorfor a root that names a version.put()raises its own message before anything is uploaded:Cannot write to the file with the version specified.S3PathPairing.s3_copy_pairs()(Act on listed identities in pairing and copy/mv/get (GH-1403) #1421) uses the same method, and keeps its messageCannot copy to a versioned file.put_file()and the async_put_file()accept anS3Path, also in a transaction of either filesystem._open_path()opens anS3Pathwith the transaction handling ofopen(), andget_file()reads through it too.S3Path.url(),metadata(),getxattr()andsetxattr()of a file now address its bucket, key and version. The filesystems'sign(),metadata(),getxattr()andsetxattr()accept anS3Path.S3Pathof a key such asa/x?versionId=nullsigned, looked up and rewrote thenullversion ofa/x.get()andput()are named after their paths as strings, on both filesystems. The async_get()and_put()run their transfers through one loop,_transfer_pairs(), which works as fsspec's loop for lists of paths and labels each branch with the string of its path. It replaces the_get_pairs()that Act on listed identities in pairing and copy/mv/get (GH-1403) #1421 added. The async callbacks stay unlocked, as before.docs/filesystem.mdcovers decisions 7, 8 and 10 of the Settle the response models, S3Object, and the listing cache #1341 audit:S3Pathnames it, and the section lists the methods that accept one.fsspec.open(),open_files(),get_mapper().S3Path(bucket, "")is the bucket.S3Pathof such a key has the key's name as its path string. Its methods address the key, but a pickled copy reopens the path string.s3://bucket//names the key/forcat_file(),rm_file()and a non-recursivechmod(), and the bucket forinfo()and the listings. A recursivechmod()changes the ACLs of the listed objects, including/, and not that of the bucket. A write raisesValueError. Each of these was reproduced against the fake store at this commit.This is step 4 of 4 of #1381 (plan in this comment), and completes it. Two older costs found in the review are filed as #1427: the duplicated upload body, and the source checks and part windowing of the async
put().WHY
Closes #1404.
TEST
Tested commit: e9bc32d.
just lintandjust docs lint;just benchmark test(363 passed, 1 skipped).test_put_names_destinations_after_local_files, sync and async, casesfile,dirandcreate. They compare the stored keys, the PutObject and HeadObject keys,IfNoneMatch, and the callback's destination labels.test_put_in_transaction_names_destinations_after_local_files, sync and async: nothing is written before the transaction ends, and the key afterwards.TestS3Path::test_with_key_suffixandtest_with_key_suffix_of_version.TestS3File::test_append_to_path_as_it_is: append mode of anS3Path, which Act on listed identities in pairing and copy/mv/get (GH-1403) #1421 made possible but no test exercised (carried over from Act on listed identities in pairing and copy/mv/get (GH-1403) #1421).TestS3File::test_path_methods_address_path_as_it_is:url(),metadata(),getxattr()andsetxattr()of a file opened with anS3Path. All four fail with the source at 522d2ca.S3Pathpath to the callback as a string: the syncput()destination, the async_put()destination and the async_get()source (3, 3 and 1 tests);S3Pathfile to the transaction (the sync transaction test).tests/pyathena/filesystem/suite against AWS, run locally in parts with a per-test timeout:test_s3.py, including the fsspec conformance tests: 1043 passed, 3 skipped;TestS3FileSystem: 715 passed, 3 skipped;TestS3FileandTestCompressedBuffer: 103 passed.🤖 Generated with Claude Code