Skip to content

Look up every conflicting source of a move, as of a copy (GH-1397) - #1423

Open
laughingman7743 wants to merge 5 commits into
masterfrom
fix/1397-mv-prefix-only-source
Open

laughingman7743 wants to merge 5 commits into
masterfrom
fix/1397-mv-prefix-only-source

Conversation

@laughingman7743

@laughingman7743 laughingman7743 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

WHAT

mv() looks up the same conflict candidates as copy(): every source without a version whose destination conflicts by name, not only a source with another source below it. A source without an object at its key writes no destination and does not conflict, so a key prefix given in a list without its keys is skipped, as the move skips it, instead of raising ValueError. AioS3FileSystem._mv() pairs its paths with S3FileSystem._move_pairs() and gets the same change.

  • S3PathPairing.conflict_candidates() has one rule for moves and copies. Its copy keyword, added in Run the batch operations of S3FileSystem concurrently (GH-1322) #1344 and released in 4.3.0, is removed, and so is the copy parameter of S3FileSystem._conflict_lookups().
  • docs/filesystem.md and the docstrings of mv(), copy(), AioS3FileSystem._mv(), conflict_candidates(), move_pairs(), and check_copy_pairs() state that mv raises for the same pairs as cp, that any source without an object at its key, not only a directory, does not conflict, and that the conflicting sources are looked up; mv() names the FileNotFoundError of a source that does not exist.
  • S3PathPairing._check() no longer intersects missing with the candidates: with one rule, a source in missing that is not a candidate stays in place, has a version (and always writes), or writes a destination that nothing else writes and that is no source, so skipping it changes no check.
  • Tests: test_mv_skipped_directory_does_not_conflict reproduces the issue. 14 existing tests changed, none of which pinned the rejection as intended:
    • 7 unit tests of S3PathPairing (test_move_pairs_*, including Act on listed identities in pairing and copy/mv/get (GH-1403) #1421's test_move_pairs_path_conflict_candidates): their candidate lists now include the conflicting sources without another source below them, and move_pairs() gets missing=set() where the pairs conflict. What they check is unchanged: the null version distinction, the conflict messages, the skip of a source without an object, that a path with a version is never looked up, and that the candidates of S3Path sources are S3Path objects.
    • 5 tests that a conflicting move raises without any request (test_mv_conflicting_destinations ×4, test_mv_conflicting_destinations_of_listed_sources): they now check that only the sources are looked up (and GetBucketVersioning where it was sent) and nothing is copied or deleted.
    • 2 Stubber sequences of Pass the source's SSE-C and RequestPayer parameters to the lookups of copies and setxattr #1333 (test_mv_source_lookup_and_delete_params, sync and aio, through stub_move_with_source_params): the lookup of bucket/a/x, with the source parameters, moves before the copies, whose lookup of it is then cached.

Release notes:

  • Breaking: S3PathPairing.conflict_candidates() no longer accepts copy. For a move it returns what copy=True returned: every source without a version whose destination conflicts. A caller of move_pairs() must pass missing whenever the pairs conflict by name.
  • mv() no longer raises ValueError for a source without an object at its key, such as a key prefix given in a list, whose destination conflicts with another pair; it skips that source as it already did without the conflict (mv() rejects a prefix-only source that it would skip #1397).
  • A move whose pairs conflict by name looks up each conflicting source without a version with HeadObject, with the source's parameters, before it raises or copies, and a failed lookup, such as one that S3 denies, raises its own error. It used to raise ValueError without a request unless the source had another source below it. Moves without such pairs send no extra request.
  • A conflicting source that does not exist at all now fails with FileNotFoundError when it is copied, and nothing is deleted, as with copy(); the copies that ran before or alongside it stay in place. It used to raise ValueError before anything was copied.

WHY

Closes #1397. The issue left two questions open: whether moves use the candidates of a copy, which made 11 existing tests fail during #1344 (13 on b720199), and the cost of one more lookup per conflicting source. Each failing test was checked as listed above, and the extra lookups are limited to moves whose pairs conflict by name, which used to raise. The maintainer chose to use the candidates of a copy for moves and to remove the copy keyword.

TEST

Tested commit: fb16382 (17bd9dd plus a docs wording fix and an order-independent assertion), rebased onto master 89ffcd9 (#1421, which compares the pairs as S3Path targets). #1421's test_move_pairs_path_conflict_candidates now expects both of its S3Path sources as candidates.

  • just lint and just docs lint: passed.

  • With the source changes of the branch reverted at 17bd9dd, test_mv_skipped_directory_does_not_conflict, the 5 request tests, the 2 Stubber tests, and 4 of the 7 S3PathPairing tests fail (the other 3 only gained missing=set()); with them, all pass.

  • The tests of test_s3_path_pairing.py, test_s3.py, and test_s3_async.py selected by move, copy, conflict, version, listed-identity, get, and put names, without conftest and with dummy credentials: 410 passed, 19 failed; the same 19 AWS integration tests fail at 89ffcd9 with dummy credentials.

  • uv run --with 'botocore[crt]' --env-file .env pytest -n 4 tests/pyathena/filesystem/ (local, AWS) at fb16382: 1845 passed, 6 skipped.

  • Reviews: self-review rounds 1 and 2, Codex independent review (2 findings) and a follow-up (CLEAN); records inline.

AWS CI starts when the PR is marked Ready.

🤖 Generated with Claude Code

laughingman7743 and others added 3 commits November 17, 2026 23:34
mv() looked up only the conflicting sources with another source below
them, so a key prefix given in a list without its keys was taken for an
object and rejected, although the move skips it. Moves now use the
conflict candidates of a copy: every source without a version whose
destination conflicts is looked up, and one without an object at its
key writes no destination. The copy keyword of
S3PathPairing.conflict_candidates(), which selected that rule, is
removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)

The docstrings of move_pairs(), check_copy_pairs(), mv(), copy(), and
AioS3FileSystem._mv() still exempted only a directory, while any source
without an object at its key is exempt, and mv() did not name the
FileNotFoundError of a source that does not exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every source without a version whose destination conflicts is now a
candidate, so a source in missing that is not one writes a destination
that nothing else writes, and skipping it changes no check. The mv()
docstring states the skip once, under Raises, and the move stub binds
the 404 lookup of bucket/a once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@laughingman7743
laughingman7743 force-pushed the fix/1397-mv-prefix-only-source branch from d942f59 to 17bd9dd Compare November 17, 2026 14:40
copy: bool = False,
) -> list[str | S3Path]:
"""Return the sources of a move whose conflicts depend on an object at their key.
"""Return the sources of a move or a copy whose conflicts depend on an object at their key.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round 1 (behavior and implementation; author: Claude Opus 5.5)

  • Base b720199003ff8e84d7b574334343b1202cf9d7bc, initial head 104d04f79131198a891606dff4724e3fb7535e9c. Repairs: b8ab83eb, d942f59f; rebased onto 89ffcd9705b3d7f9adffba44f24f773868c10773 (Act on listed identities in pairing and copy/mv/get (GH-1403) #1421) as fe5fe00f6ffd1026c4cc64ec07d86b675002d1af, e97904f96b8e663a5f757375308d6bc24cfc965e, 17bd9ddb02a1e672d540e7aaa8f36f45aa3bc838.
  • Covered: S3PathPairing.conflict_candidates(), _moves(), _check(); S3FileSystem._conflict_lookups(), _move_pairs(), mv(), copy(); AioS3FileSystem._mv() through _move_pairs(); docs/filesystem.md; the 13 changed tests and the new regression test. Manual pass, /code-review (high), and /simplify (reuse, simplification, efficiency, altitude).
  • Result: FINDINGS, repaired:
    1. The docstrings of move_pairs(), check_copy_pairs(), mv(), copy(), and AioS3FileSystem._mv() exempted only a directory, while any source without an object at its key is exempt, and mv() did not name the FileNotFoundError of a source that does not exist (e97904f9).
    2. _check() intersected missing with the candidates; with one rule, a source in missing that is not a candidate writes a destination that nothing else writes, so the filter changed nothing. Removed, with the skip stated once in mv() and the 404 lookup of the move stub bound once (17bd9ddb).
    3. Rebase onto Act on listed identities in pairing and copy/mv/get (GH-1403) #1421: the pairs may hold S3Path sources (_NamedPair); the unified rule keeps S3Path candidates, and Act on listed identities in pairing and copy/mv/get (GH-1403) #1421's test_move_pairs_path_conflict_candidates now expects both of its S3Path sources.

Comment thread pyathena/filesystem/s3.py
GetBucketVersioning once per bucket during the move's planning and
requires ``s3:GetBucketVersioning``. The result is not cached across
moves. Directory buckets do not support versioning and need no lookup.
A source without a version whose destination conflicts with another

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round 1, not changed (with reasons)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread docs/filesystem.md
into it one after another, in order.
`MetadataDirective="REPLACE"`, is no conflict. `mv` raises for the same pairs as `cp`.
A source of `cp` or `mv` without an object at its key, such as a key prefix given in a
list, writes no destination and does not conflict; the sources without a version whose

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round 2 (claims, callers, operations; author: Claude Opus 5.5)

  • Base 89ffcd9705b3d7f9adffba44f24f773868c10773, head 17bd9ddb02a1e672d540e7aaa8f36f45aa3bc838; repair 788aa81cf76c63afe795b581c0d682f71e988b1e.
  • Claims checked against the code and runs: the issue's reproduction (test_mv_skipped_directory_does_not_conflict); AioS3FileSystem._mv() pairs through _move_pairs() (s3_async.py:778); the copy keyword shipped in v4.3.0 (git tag --contains); moves without name conflicts send no extra request (candidates need a shared or source destination); _head_object() returns None only for FileNotFoundError, so a denied lookup raises; mv raises for the same pairs as cp (one _check()); the docstrings of mv(), copy(), _mv(), move_pairs(), check_copy_pairs(), conflict_candidates(); the commit messages.
  • Result: FINDINGS, corrected:
    1. This paragraph said the conflicting sources are looked up; only those without a version are (788aa81c).
    2. PR body: 14 existing tests changed, not 13 (the rebase added Act on listed identities in pairing and copy/mv/get (GH-1403) #1421's test_move_pairs_path_conflict_candidates); the _check() rationale and the FileNotFoundError timing were imprecise (the copies that ran before or alongside it stay in place; with a failure, the sync move starts no further copy).
  • Evidence: with the branch's source changes reverted at 17bd9ddb, 12 of the changed and new tests fail (the regression test, the 5 request tests, the 2 Stubber tests, 4 S3PathPairing tests; the other 3 only gained missing=set()). Local runs are mock or Stubber runs with dummy credentials (410 passed; the 19 failures are AWS integration tests that fail the same way at 89ffcd97). Not run yet: the AWS suite (local login expired) and AWS CI.
  • Operators: the extra lookups are HeadObject requests with the source's parameters, through the core's retries, only for pairs that conflict by name; s3:GetObject on the sources is already needed to copy them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
copy: bool = False,
) -> list[str | S3Path]:
"""Return the sources of a move whose conflicts depend on an object at their key.
"""Return the sources of a move or a copy whose conflicts depend on an object at their key.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (relayed result; reviewer: OpenAI Codex CLI 0.169.0, ChatGPT login, codex exec -s read-only, default model and effort; static review, no tests run)

  • Base 89ffcd9705b3d7f9adffba44f24f773868c10773, head 788aa81cf76c63afe795b581c0d682f71e988b1e, from a detached snapshot that stayed unchanged; prompt without the PR text, commit messages, or prior findings.
  • Covered (reviewer): sync move/copy planning, lookups, caching, source parameters, deletion after copies; async planning, typed paths, failures, cancellation; S3PathPairing callers, typed keys, versions, null-version buckets; the changed docs and every changed test with the stub's two consumers.
  • Result: FINDINGS (2 introduced). The conflict computation and the prose about missing sources otherwise agree with the code; explicit versions stay writers, and typed keys keep their literal identity. A cancellation edge of _mv() (gather cancelling already-cancelling children) was reported as pre-existing, excluded by the existing test matrix and documented in docs/aio.md.
  1. (P2) Removing the copy keyword of the public conflict_candidates() makes conflict_candidates(pairs, copy=True) raise TypeError; keeping it as an accepted argument was suggested. Not changed: the maintainer decided to remove it, with a release note, as 4.x minors have carried breaking changes with notes (Run the batch operations of S3FileSystem concurrently (GH-1322) #1344, Raise FileNotFoundError from ls() of a missing key path (GH-1330) #1347, Inject the filesystem's S3 client with client= and retry_config= (GH-1363) #1378).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review follow-up 1 (Codex CLI 0.169.0, codex exec -s read-only, static; repair diff 788aa81cf76c63afe795b581c0d682f71e988b1e..fb1638214dc6e9642074d3ed74b043598e7fc9b3, merge-base 89ffcd9705b3d7f9adffba44f24f773868c10773, with the full change for context; snapshot unchanged): CLEAN. Sorting keeps the method, key, and call-count checks while dropping the order; no docstring inaccuracies, vacuous assertions, timing dependencies, or regressions found. Self-review of the repair (behavior and claims): the sorted comparison still fails without the lookups (an empty call list), and no claim changed.

fs.mv(["s3://bucket/x/a", "s3://bucket/y/a"], "s3://bucket/out", recursive=True)
fs.isdir.assert_not_called()
fs._call.assert_not_called()
assert sorted((c.args[0], c.kwargs["Key"]) for c in fs._call.call_args_list) == [

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review, finding 2 (Codex, see the record on s3_path_pairing.py:204)

(P3) test_mv_conflicting_destinations_of_listed_sources required the lookups in pair order, which the contract of mv() does not state. Fixed in fb163821: the lookups are compared sorted, with the no-write assertions kept.

@laughingman7743
laughingman7743 marked this pull request as ready for review November 17, 2026 15:51

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mv() rejects a prefix-only source that it would skip

1 participant