Repository navigation
Look up every conflicting source of a move, as of a copy (GH-1397) - #1423
laughingman7743 wants to merge 5 commits into
Conversation
mv() looked up only the conflicting sources with another source below them, so a key prefix given in a list without its keys was taken for an object and rejected, although the move skips it. Moves now use the conflict candidates of a copy: every source without a version whose destination conflicts is looked up, and one without an object at its key writes no destination. The copy keyword of S3PathPairing.conflict_candidates(), which selected that rule, is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
) The docstrings of move_pairs(), check_copy_pairs(), mv(), copy(), and AioS3FileSystem._mv() still exempted only a directory, while any source without an object at its key is exempt, and mv() did not name the FileNotFoundError of a source that does not exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every source without a version whose destination conflicts is now a candidate, so a source in missing that is not one writes a destination that nothing else writes, and skipping it changes no check. The mv() docstring states the skip once, under Raises, and the move stub binds the 404 lookup of bucket/a once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d942f59 to
17bd9dd
Compare
| copy: bool = False, | ||
| ) -> list[str | S3Path]: | ||
| """Return the sources of a move whose conflicts depend on an object at their key. | ||
| """Return the sources of a move or a copy whose conflicts depend on an object at their key. |
There was a problem hiding this comment.
Self-review round 1 (behavior and implementation; author: Claude Opus 5.5)
- Base
b720199003ff8e84d7b574334343b1202cf9d7bc, initial head104d04f79131198a891606dff4724e3fb7535e9c. Repairs:b8ab83eb,d942f59f; rebased onto89ffcd9705b3d7f9adffba44f24f773868c10773(Act on listed identities in pairing and copy/mv/get (GH-1403) #1421) asfe5fe00f6ffd1026c4cc64ec07d86b675002d1af,e97904f96b8e663a5f757375308d6bc24cfc965e,17bd9ddb02a1e672d540e7aaa8f36f45aa3bc838. - Covered:
S3PathPairing.conflict_candidates(),_moves(),_check();S3FileSystem._conflict_lookups(),_move_pairs(),mv(),copy();AioS3FileSystem._mv()through_move_pairs();docs/filesystem.md; the 13 changed tests and the new regression test. Manual pass,/code-review(high), and/simplify(reuse, simplification, efficiency, altitude). - Result: FINDINGS, repaired:
- The docstrings of
move_pairs(),check_copy_pairs(),mv(),copy(), andAioS3FileSystem._mv()exempted only a directory, while any source without an object at its key is exempt, andmv()did not name theFileNotFoundErrorof a source that does not exist (e97904f9). _check()intersectedmissingwith the candidates; with one rule, a source inmissingthat is not a candidate writes a destination that nothing else writes, so the filter changed nothing. Removed, with the skip stated once inmv()and the 404 lookup of the move stub bound once (17bd9ddb).- Rebase onto Act on listed identities in pairing and copy/mv/get (GH-1403) #1421: the pairs may hold
S3Pathsources (_NamedPair); the unified rule keepsS3Pathcandidates, and Act on listed identities in pairing and copy/mv/get (GH-1403) #1421'stest_move_pairs_path_conflict_candidatesnow expects both of itsS3Pathsources.
- The docstrings of
| GetBucketVersioning once per bucket during the move's planning and | ||
| requires ``s3:GetBucketVersioning``. The result is not cached across | ||
| moves. Directory buckets do not support versioning and need no lookup. | ||
| A source without a version whose destination conflicts with another |
There was a problem hiding this comment.
Self-review round 1, not changed (with reasons)
- A conflicting source that does not exist at all fails with
FileNotFoundErrorwhen it is copied, after the running copies, and nothing is deleted, instead ofValueErrorup front. Same ascopy()since Run the batch operations of S3FileSystem concurrently (GH-1322) #1344; the maintainer chose this over confirming a prefix with a listing. In the release notes. - A failed lookup, such as a 403 without
s3:ListBucket, now raises instead of theValueErrorof a conflicting move. Same ascopy(); added to the release notes. - The lookups run one at a time, a 404 is looked up again by the copy's
info(), and a source given twice is looked up twice (test_conflict_candidates_one_per_pair). All already incopy(), and only for pairs that conflict by name. - Removing the
copykeyword of the publicconflict_candidates()without a deprecation: the maintainer's decision; 4.x minors have carried breaking changes with notes (Run the batch operations of S3FileSystem concurrently (GH-1322) #1344, Raise FileNotFoundError from ls() of a missing key path (GH-1330) #1347, Inject the filesystem's S3 client with client= and retry_config= (GH-1363) #1378). test_mv_conflicting_destinationscompares the set of methods only: the exact candidates, including the exclusion of destinations and versions, are pinned by theS3PathPairingtests, and the exact lookups by the Stubber sequences.- Renaming
_moves(): private naming churn.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| into it one after another, in order. | ||
| `MetadataDirective="REPLACE"`, is no conflict. `mv` raises for the same pairs as `cp`. | ||
| A source of `cp` or `mv` without an object at its key, such as a key prefix given in a | ||
| list, writes no destination and does not conflict; the sources without a version whose |
There was a problem hiding this comment.
Self-review round 2 (claims, callers, operations; author: Claude Opus 5.5)
- Base
89ffcd9705b3d7f9adffba44f24f773868c10773, head17bd9ddb02a1e672d540e7aaa8f36f45aa3bc838; repair788aa81cf76c63afe795b581c0d682f71e988b1e. - Claims checked against the code and runs: the issue's reproduction (
test_mv_skipped_directory_does_not_conflict);AioS3FileSystem._mv()pairs through_move_pairs()(s3_async.py:778); thecopykeyword shipped in v4.3.0 (git tag --contains); moves without name conflicts send no extra request (candidates need a shared or source destination);_head_object()returns None only forFileNotFoundError, so a denied lookup raises;mvraises for the same pairs ascp(one_check()); the docstrings ofmv(),copy(),_mv(),move_pairs(),check_copy_pairs(),conflict_candidates(); the commit messages. - Result: FINDINGS, corrected:
- This paragraph said the conflicting sources are looked up; only those without a version are (
788aa81c). - PR body: 14 existing tests changed, not 13 (the rebase added Act on listed identities in pairing and copy/mv/get (GH-1403) #1421's
test_move_pairs_path_conflict_candidates); the_check()rationale and theFileNotFoundErrortiming were imprecise (the copies that ran before or alongside it stay in place; with a failure, the sync move starts no further copy).
- This paragraph said the conflicting sources are looked up; only those without a version are (
- Evidence: with the branch's source changes reverted at
17bd9ddb, 12 of the changed and new tests fail (the regression test, the 5 request tests, the 2 Stubber tests, 4S3PathPairingtests; the other 3 only gainedmissing=set()). Local runs are mock or Stubber runs with dummy credentials (410 passed; the 19 failures are AWS integration tests that fail the same way at89ffcd97). Not run yet: the AWS suite (local login expired) and AWS CI. - Operators: the extra lookups are HeadObject requests with the source's parameters, through the core's retries, only for pairs that conflict by name;
s3:GetObjecton the sources is already needed to copy them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| copy: bool = False, | ||
| ) -> list[str | S3Path]: | ||
| """Return the sources of a move whose conflicts depend on an object at their key. | ||
| """Return the sources of a move or a copy whose conflicts depend on an object at their key. |
There was a problem hiding this comment.
Independent review (relayed result; reviewer: OpenAI Codex CLI 0.169.0, ChatGPT login, codex exec -s read-only, default model and effort; static review, no tests run)
- Base
89ffcd9705b3d7f9adffba44f24f773868c10773, head788aa81cf76c63afe795b581c0d682f71e988b1e, from a detached snapshot that stayed unchanged; prompt without the PR text, commit messages, or prior findings. - Covered (reviewer): sync move/copy planning, lookups, caching, source parameters, deletion after copies; async planning, typed paths, failures, cancellation;
S3PathPairingcallers, typed keys, versions,null-version buckets; the changed docs and every changed test with the stub's two consumers. - Result: FINDINGS (2 introduced). The conflict computation and the prose about missing sources otherwise agree with the code; explicit versions stay writers, and typed keys keep their literal identity. A cancellation edge of
_mv()(gathercancelling already-cancelling children) was reported as pre-existing, excluded by the existing test matrix and documented indocs/aio.md.
- (P2) Removing the
copykeyword of the publicconflict_candidates()makesconflict_candidates(pairs, copy=True)raiseTypeError; keeping it as an accepted argument was suggested. Not changed: the maintainer decided to remove it, with a release note, as 4.x minors have carried breaking changes with notes (Run the batch operations of S3FileSystem concurrently (GH-1322) #1344, Raise FileNotFoundError from ls() of a missing key path (GH-1330) #1347, Inject the filesystem's S3 client with client= and retry_config= (GH-1363) #1378).
There was a problem hiding this comment.
Independent review follow-up 1 (Codex CLI 0.169.0, codex exec -s read-only, static; repair diff 788aa81cf76c63afe795b581c0d682f71e988b1e..fb1638214dc6e9642074d3ed74b043598e7fc9b3, merge-base 89ffcd9705b3d7f9adffba44f24f773868c10773, with the full change for context; snapshot unchanged): CLEAN. Sorting keeps the method, key, and call-count checks while dropping the order; no docstring inaccuracies, vacuous assertions, timing dependencies, or regressions found. Self-review of the repair (behavior and claims): the sorted comparison still fails without the lookups (an empty call list), and no claim changed.
| fs.mv(["s3://bucket/x/a", "s3://bucket/y/a"], "s3://bucket/out", recursive=True) | ||
| fs.isdir.assert_not_called() | ||
| fs._call.assert_not_called() | ||
| assert sorted((c.args[0], c.kwargs["Key"]) for c in fs._call.call_args_list) == [ |
There was a problem hiding this comment.
Independent review, finding 2 (Codex, see the record on s3_path_pairing.py:204)
(P3) test_mv_conflicting_destinations_of_listed_sources required the lookups in pair order, which the contract of mv() does not state. Fixed in fb163821: the lookups are compared sorted, with the no-write assertions kept.
WHAT
mv()looks up the same conflict candidates ascopy(): every source without a version whose destination conflicts by name, not only a source with another source below it. A source without an object at its key writes no destination and does not conflict, so a key prefix given in a list without its keys is skipped, as the move skips it, instead of raisingValueError.AioS3FileSystem._mv()pairs its paths withS3FileSystem._move_pairs()and gets the same change.S3PathPairing.conflict_candidates()has one rule for moves and copies. Itscopykeyword, added in Run the batch operations of S3FileSystem concurrently (GH-1322) #1344 and released in 4.3.0, is removed, and so is thecopyparameter ofS3FileSystem._conflict_lookups().docs/filesystem.mdand the docstrings ofmv(),copy(),AioS3FileSystem._mv(),conflict_candidates(),move_pairs(), andcheck_copy_pairs()state thatmvraises for the same pairs ascp, that any source without an object at its key, not only a directory, does not conflict, and that the conflicting sources are looked up;mv()names theFileNotFoundErrorof a source that does not exist.S3PathPairing._check()no longer intersectsmissingwith the candidates: with one rule, a source inmissingthat is not a candidate stays in place, has a version (and always writes), or writes a destination that nothing else writes and that is no source, so skipping it changes no check.test_mv_skipped_directory_does_not_conflictreproduces the issue. 14 existing tests changed, none of which pinned the rejection as intended:S3PathPairing(test_move_pairs_*, including Act on listed identities in pairing and copy/mv/get (GH-1403) #1421'stest_move_pairs_path_conflict_candidates): their candidate lists now include the conflicting sources without another source below them, andmove_pairs()getsmissing=set()where the pairs conflict. What they check is unchanged: thenullversion distinction, the conflict messages, the skip of a source without an object, that a path with a version is never looked up, and that the candidates ofS3Pathsources areS3Pathobjects.test_mv_conflicting_destinations×4,test_mv_conflicting_destinations_of_listed_sources): they now check that only the sources are looked up (and GetBucketVersioning where it was sent) and nothing is copied or deleted.test_mv_source_lookup_and_delete_params, sync and aio, throughstub_move_with_source_params): the lookup ofbucket/a/x, with the source parameters, moves before the copies, whose lookup of it is then cached.Release notes:
S3PathPairing.conflict_candidates()no longer acceptscopy. For a move it returns whatcopy=Truereturned: every source without a version whose destination conflicts. A caller ofmove_pairs()must passmissingwhenever the pairs conflict by name.mv()no longer raisesValueErrorfor a source without an object at its key, such as a key prefix given in a list, whose destination conflicts with another pair; it skips that source as it already did without the conflict (mv() rejects a prefix-only source that it would skip #1397).ValueErrorwithout a request unless the source had another source below it. Moves without such pairs send no extra request.FileNotFoundErrorwhen it is copied, and nothing is deleted, as withcopy(); the copies that ran before or alongside it stay in place. It used to raiseValueErrorbefore anything was copied.WHY
Closes #1397. The issue left two questions open: whether moves use the candidates of a copy, which made 11 existing tests fail during #1344 (13 on b720199), and the cost of one more lookup per conflicting source. Each failing test was checked as listed above, and the extra lookups are limited to moves whose pairs conflict by name, which used to raise. The maintainer chose to use the candidates of a copy for moves and to remove the
copykeyword.TEST
Tested commit: fb16382 (17bd9dd plus a docs wording fix and an order-independent assertion), rebased onto master 89ffcd9 (#1421, which compares the pairs as
S3Pathtargets). #1421'stest_move_pairs_path_conflict_candidatesnow expects both of itsS3Pathsources as candidates.just lintandjust docs lint: passed.With the source changes of the branch reverted at 17bd9dd,
test_mv_skipped_directory_does_not_conflict, the 5 request tests, the 2 Stubber tests, and 4 of the 7S3PathPairingtests fail (the other 3 only gainedmissing=set()); with them, all pass.The tests of
test_s3_path_pairing.py,test_s3.py, andtest_s3_async.pyselected by move, copy, conflict, version, listed-identity, get, and put names, without conftest and with dummy credentials: 410 passed, 19 failed; the same 19 AWS integration tests fail at 89ffcd9 with dummy credentials.uv run --with 'botocore[crt]' --env-file .env pytest -n 4 tests/pyathena/filesystem/(local, AWS) at fb16382: 1845 passed, 6 skipped.Reviews: self-review rounds 1 and 2, Codex independent review (2 findings) and a follow-up (CLEAN); records inline.
AWS CI starts when the PR is marked Ready.
🤖 Generated with Claude Code