Skip to content

Type put destinations and document path addressing (GH-1404) - #1426

Open
laughingman7743 wants to merge 2 commits into
masterfrom
fix/1404-typed-put-destinations
Open

laughingman7743 wants to merge 2 commits into
masterfrom
fix/1404-typed-put-destinations

Conversation

@laughingman7743

@laughingman7743 laughingman7743 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

WHAT

put() now writes a local file to the key named after the file as it is, on both filesystems. This includes a file whose name ends in what looks like a version ID query. The docs now describe how keys that look like version paths, and bucket//, are addressed.

Behavior before this change (89ffcd9)

The examples use a local file named x?versionId=v1:

  • put(<file>, "s3://bucket/up/") built the destination bucket/up/x?versionId=v1, which was parsed as version v1 of up/x.
  • The upload then raised ValueError: Cannot write to the file with the version specified.
  • The same happened for a recursive put of its directory, for mode="create", and for the async _put().

Changes

  • Destinations. _put_pairs() returns an S3Path for each destination named after a local file. It is the parsed destination root with the rest of the local name added to its key as it is, through the new S3Path.with_key_suffix().
    • with_key_suffix() raises ValueError for a root that names a version. put() raises its own message before anything is uploaded: Cannot write to the file with the version specified.
    • S3PathPairing.s3_copy_pairs() (Act on listed identities in pairing and copy/mv/get (GH-1403) #1421) uses the same method, and keeps its message Cannot copy to a versioned file.
    • A destination that is the root itself, or one item of a list of destinations, stays the caller's string, with string semantics.
  • Writes. put_file() and the async _put_file() accept an S3Path, also in a transaction of either filesystem.
    • The new _open_path() opens an S3Path with the transaction handling of open(), and get_file() reads through it too.
    • The transaction opener of the async filesystem also accepts an S3Path.
  • File methods. url(), metadata(), getxattr() and setxattr() of a file now address its bucket, key and version. The filesystems' sign(), metadata(), getxattr() and setxattr() accept an S3Path.
  • Callback labels. The branches of the callback of get() and put() are named after their paths as strings, on both filesystems. The async _get() and _put() run their transfers through one loop, _transfer_pairs(), which works as fsspec's loop for lists of paths and labels each branch with the string of its path. It replaces the _get_pairs() that Act on listed identities in pairing and copy/mv/get (GH-1403) #1421 added. The async callbacks stay unlocked, as before.
  • Docs. A new "Path addressing" section in docs/filesystem.md covers decisions 7, 8 and 10 of the Settle the response models, S3Object, and the listing cache #1341 audit:
    • A key that ends in a version ID query cannot be named by a path string, and there is no escape syntax. An S3Path names it, and the section lists the methods that accept one.
    • The listings and the operations built on them act on such keys. The compositions that pass path strings back do not keep this identity: fsspec.open(), open_files(), get_mapper().
    • S3Path(bucket, "") is the bucket.
    • A file opened with an S3Path of such a key has the key's name as its path string. Its methods address the key, but a pickled copy reopens the path string.
    • s3://bucket// names the key / for cat_file(), rm_file() and a non-recursive chmod(), and the bucket for info() and the listings. A recursive chmod() changes the ACLs of the listed objects, including /, and not that of the bucket. A write raises ValueError. Each of these was reproduced against the fake store at this commit.

This is step 4 of 4 of #1381 (plan in this comment), and completes it. Two older costs found in the review are filed as #1427: the duplicated upload body, and the source checks and part windowing of the async put().

WHY

Closes #1404.

TEST

Tested commit: e9bc32d.

  • These pass:
    • just lint and just docs lint;
    • a Sphinx build of the working tree, whose only warnings are the duplicate object descriptions that the other pages already have;
    • just benchmark test (363 passed, 1 skipped).
  • New tests against the offline fake S3 store:
    • test_put_names_destinations_after_local_files, sync and async, cases file, dir and create. They compare the stored keys, the PutObject and HeadObject keys, IfNoneMatch, and the callback's destination labels.
    • test_put_in_transaction_names_destinations_after_local_files, sync and async: nothing is written before the transaction ends, and the key afterwards.
    • TestS3Path::test_with_key_suffix and test_with_key_suffix_of_version.
    • TestS3File::test_append_to_path_as_it_is: append mode of an S3Path, which Act on listed identities in pairing and copy/mv/get (GH-1403) #1421 made possible but no test exercised (carried over from Act on listed identities in pairing and copy/mv/get (GH-1403) #1421).
    • TestS3File::test_path_methods_address_path_as_it_is: url(), metadata(), getxattr() and setxattr() of a file opened with an S3Path. All four fail with the source at 522d2ca.
  • Of the 17 new tests, 16 fail with the source at 89ffcd9. The append test passes there, as noted above.
  • Mutations, each of which fails the tests that protect it:
    • not passing an S3Path path to the callback as a string: the sync put() destination, the async _put() destination and the async _get() source (3, 3 and 1 tests);
    • not joining a written S3Path file to the transaction (the sync transaction test).
  • The full tests/pyathena/filesystem/ suite against AWS, run locally in parts with a per-test timeout:
    • the files other than test_s3.py, including the fsspec conformance tests: 1043 passed, 3 skipped;
    • TestS3FileSystem: 715 passed, 3 skipped;
    • TestS3File and TestCompressedBuffer: 103 passed.

🤖 Generated with Claude Code

laughingman7743 and others added 2 commits November 17, 2026 23:46
put() and the async _put() named the destination of a local file by
joining its name to the destination string, which was parsed again: a
file named x?versionId=v1 was taken for version v1 of the key x, and
the upload raised that a version cannot be written.

- _put_pairs() returns an S3Path for a destination named after a local
  file: the destination root parsed, with the rest of the local name
  added to its key as it is (S3Path.with_key_suffix(), which
  s3_copy_pairs() of the pairing uses as well). A destination that is
  the root itself, or one of a list, stays the caller's string.
- put_file() and the async _put_file(), also in a transaction, write an
  S3Path by its bucket and key; _open_path() opens it with open()'s
  transaction handling, and get_file() reads through it.
- _S3LockedCallback names the branches of S3Path sources and
  destinations by their strings, so that the async _get() and _put()
  run fsspec's own loops for lists of paths again; _get_pairs() is
  removed.
- docs/filesystem.md documents the addressing of keys that look like
  version paths (the S3Path methods, the string compositions, and the
  path string of a file opened with an S3Path) and of bucket//.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review repairs:

- url(), metadata(), getxattr() and setxattr() of a file address its
  bucket, key and version: sign(), metadata(), getxattr() and
  setxattr() of the filesystems accept an S3Path. A file opened with
  the S3Path of a key such as a/x?versionId=null signed and looked up
  the null version of a/x. Only a pickled copy still reopens the path
  string, which the docs say.
- The async _get() and _put() name the branches of their callback
  after their paths as strings in a transfer loop of their own, as
  fsspec runs them, instead of taking the lock of _S3LockedCallback on
  the event loop, which a callback that calls the filesystem could
  deadlock.
- copy() and put() to a destination root that names a version raise
  their own messages again, and put() and _put() say so.
- _put_pairs() returns early when no destination is named after a
  local file, put_file() parses its destination once, and the async
  transaction write invalidates through the filesystem.
- The append test of a typed path moves to TestS3File, and the sync put
  test compares the labels of the callback exactly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)
)

async def _transfer_pairs(

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round one (implementation behavior): FINDINGS, repaired in e9bc32d

Scope: the full diff 89ffcd97..522d2cab.

Passes:

  • /code-review at high effort.
  • /simplify, with four cleanup agents: reuse (CLEAN), simplification, efficiency (CLEAN) and altitude.

Findings and repairs:

  1. The async _get()/_put() took a threading.Lock on the event loop (/code-review). The wrapped callback took it on the loop thread while worker threads held it for their child callbacks. That stalls the loop, and deadlocks it if a child callback calls a sync method of the filesystem, which waits on the loop.
  2. The path methods of a file opened with an S3Path used its path string (altitude).
    • For S3Path("bucket", "a/x?versionId=null"), url(), metadata() and getxattr() signed and read the null version of a/x, and setxattr() rejected the file as a version.
    • The docs listed this as a limit, and named sign(), which S3File does not have (its method is url()).
    • Repair: the filesystems' sign(), metadata(), getxattr() and setxattr() accept an S3Path, and S3File passes its bucket, key and version (_s3_path). The docs now name only pickling as the limit.
    • Test: TestS3File::test_path_methods_address_path_as_it_is; all four cases fail at 522d2ca.
  3. The versioned-root errors changed (/code-review, altitude). copy() of listed sources and put() raised with_key_suffix()'s generic message.
    • Repair: s3_copy_pairs() raises Cannot copy to a versioned file. again, and _put_pairs() raises Cannot write to the file with the version specified., the message that the string path gave at open(). The Raises of put() and _put() say so.
  4. Cleanups (simplification):
    • _put_pairs() returns early when no destination is named after a local file.
    • put_file() and the async transaction write parse their destination once.
    • The latter invalidates through self._invalidate().
  5. Tests (/code-review):
    • The append test of a typed path moved to TestS3File.
    • The sync put test compares the callback's labels exactly (dir includes the directory's bucket/up).

Deferred, with reasons:

Validation:

  • just lint, just docs lint and just benchmark test pass.
  • The filesystem suite against AWS, in parts: 1043 + 715 + 103 passed, 6 skipped.

"""
return self.with_version_id(None) if self.version_id == "null" else self

def with_key_suffix(self, suffix: str) -> S3Path:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round two (claims, callers, AWS operator, evidence): CLEAN

Scope:

Claims checked:

  1. The before-state. At 89ffcd9, put of the file x?versionId=v1 raises ValueError: Cannot write to the file with the version specified. This holds for a single file, a recursive directory and mode="create", sync and async, reproduced against the fake store.
  2. "A destination that is the root itself, or one item of a list, stays the caller's string." _put_pairs() returns the pairs unchanged when rpath is a list or every destination equals it. Otherwise only the destinations other than rpath are typed.
    • So put(file, "s3://bucket/up/x?versionId=v1") still names that version and fails at open(), as documented.
  3. "The async callbacks stay unlocked, as before." At 89ffcd9:
    • the async _get() ran Act on listed identities in pairing and copy/mv/get (GH-1403) #1421's _get_pairs(), without a lock;
    • the async _put() passed the callback to fsspec's _put(), without a lock.
      _transfer_pairs() keeps both, and calls set_size, branched(..., **kwargs), the transfer and _run_coros_in_chunks(..., callback=) as fsspec 2026.9.0 does. The local directories are created as fsspec's _get() creates them, and _put() filters the directories out before the transfers, as before.
  4. The widened path methods behave the same for strings:
    • sign(), metadata() and getxattr() parse a string as before.
    • setxattr() now passes its parsed path to metadata() and invalidates it with _invalidate(). Its path never has a version (it raises first), and the name walk evicts at least what invalidate_cache() did.
    • The async sign, metadata, getxattr and setxattr hand the path to the sync methods unchanged.
  5. S3File now always passes S3Path(bucket, key, version_id) to these methods. For a file opened by a string, that is the parse of its path, including the pinned ?versionId= of version_aware, so the requests are the same. No test patches these methods.
  6. The bucket// section matches the probe at this commit:
    • info → HeadBucket;
    • cat_file/rm_file/chmod → the key /;
    • ls/find → the bucket listing;
    • a recursive chmod → PutObjectAcl of / and a, without PutBucketAcl;
    • pipe_file → ValueError.
  7. The other docs statements were checked against the code:
    • the list of methods that accept an S3Path, including S3File;
    • the compositions through strings (fsspec.open(), open_files(), get_mapper());
    • the pickling limit: fsspec's __reduce__ reopens self.path.

Existing callers:

  • No public signature was narrowed. put_file, sign, metadata, getxattr and setxattr (and the async ones) only accept more. S3Path.with_key_suffix() is new.
  • Error messages for strings are unchanged.
  • The new raise for a versioned put() root replaces the same message, which each file raised at open(). It now comes before any upload.

AWS operator:

  • put() sends the same requests to the literal keys.
  • mode="create" sends HeadObject and a conditional PutObject as before.
  • The file methods address the key instead of a version.
  • No new permission.

Evidence:

  • Local runs, in the round-one record.
  • The 16 new tests that fail at 89ffcd9, and the four file-method tests that fail at 522d2ca.
  • CI against AWS has not run yet (Draft).

"""
return self.with_version_id(None) if self.version_id == "null" else self

def with_key_suffix(self, suffix: str) -> S3Path:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (relayed result, Codex): CLEAN

Reviewer and run:

Coverage, in the reviewer's words:

  • "put()/_put() pairing against fsspec 2026.9.0: globs, recursion, maxdepth, lists, trailing slashes, bucket roots and versioned roots."
  • "Typed upload destinations, create mode and both filesystems' transactions."
  • "Async batching, callback labels and contexts, return values, and _S3BatchTasks cancellation cleanup."
  • "File and filesystem URL/metadata methods, including setxattr() lookups and cache invalidation."
  • "Path-addressing documentation, new regression assertions and existing compatibility coverage."

Result: "CLEAN — No actionable findings."

The independent review is complete. The PR moves to Ready for CI.

@laughingman7743
laughingman7743 marked this pull request as ready for review November 17, 2026 15:28

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Type generated put destinations and document path addressing limits (#1381 step 4)

1 participant