Repository navigation
Look up every conflicting source of a move, as of a copy (GH-1397) #1423
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
fe5fe00
e97904f
17bd9dd
788aa81
fb16382
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2709,6 +2709,8 @@ def mv(self, path1, path2, recursive=False, maxdepth=None, batch_size=None, **kw | |
| GetBucketVersioning once per bucket during the move's planning and | ||
| requires ``s3:GetBucketVersioning``. The result is not cached across | ||
| moves. Directory buckets do not support versioning and need no lookup. | ||
| A source without a version whose destination conflicts with another | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Self-review round 1, not changed (with reasons)
|
||
| pair is looked up with HeadObject during the planning too. | ||
|
|
||
| The files are copied in parallel as ``copy()`` copies them, and are | ||
| deleted once every copy has succeeded; nothing is deleted after a | ||
|
|
@@ -2731,9 +2733,12 @@ def mv(self, path1, path2, recursive=False, maxdepth=None, batch_size=None, **kw | |
| Raises: | ||
| ValueError: If two sources have the same destination, or a | ||
| destination is another source, including one left in place, | ||
| which is checked before anything is copied. A directory with | ||
| no object at its key, which is not copied, does not conflict. | ||
| Also if ``batch_size`` is neither positive, -1, nor None. | ||
| which is checked before anything is copied. A source without | ||
| an object at its key, such as a directory, writes no | ||
| destination and does not conflict. Also if ``batch_size`` is | ||
| neither positive, -1, nor None. | ||
| FileNotFoundError: If a source does not exist, once the running | ||
| copies have ended; nothing is deleted. | ||
| OSError: If bucket versioning cannot be read. | ||
| """ | ||
| path1, path2 = self._stringify_paths(path1), self._stringify_paths(path2) | ||
|
|
@@ -2810,7 +2815,6 @@ def _conflict_lookups( | |
| pairing: S3PathPairing, | ||
| pairs: list[tuple[str | S3Path, str | S3Path]], | ||
| source_params: Mapping[str, Any] | None = None, | ||
| copy: bool = False, | ||
| ) -> tuple[set[str | S3Path], set[str]]: | ||
| """Look up what the conflict checks of a move or a copy need. | ||
|
|
||
|
|
@@ -2822,7 +2826,6 @@ def _conflict_lookups( | |
| source_params: The parameters of the requests that read the | ||
| sources (see :meth:`S3Core.copy_source_params`), sent with | ||
| the lookups of the sources. | ||
| copy: Whether the pairs are those of a copy. | ||
|
|
||
| Returns: | ||
| The conflict candidates with no object at their key, and the | ||
|
|
@@ -2846,7 +2849,7 @@ def _conflict_lookups( | |
| missing = { | ||
| source | ||
| for source in pairing.conflict_candidates( | ||
| pairs, versioning_enabled_buckets=versioned_buckets, copy=copy | ||
| pairs, versioning_enabled_buckets=versioned_buckets | ||
| ) | ||
| if self._head_object(S3Path.of(source), lookup_kwargs=lookup_kwargs) is None | ||
| } | ||
|
|
@@ -2937,9 +2940,9 @@ def copy( | |
|
|
||
| Raises: | ||
| ValueError: If two sources have the same destination, or a | ||
| destination is another source, except for a directory with no | ||
| object at its key, which is not copied. Also if | ||
| ``batch_size`` is neither positive, -1, nor None. | ||
| destination is another source, except for a source without an | ||
| object at its key, such as a directory, which is not copied. | ||
| Also if ``batch_size`` is neither positive, -1, nor None. | ||
| OSError: If bucket versioning cannot be read for the checks. | ||
| """ | ||
| path1, path2 = self._stringify_paths(path1), self._stringify_paths(path2) | ||
|
|
@@ -2951,7 +2954,7 @@ def copy( | |
| if not pairs: | ||
| return | ||
| missing, versioned_buckets = self._conflict_lookups( | ||
| pairing, pairs, self.core.copy_source_params(kwargs), copy=True | ||
| pairing, pairs, self.core.copy_source_params(kwargs) | ||
| ) | ||
| pairing.check_copy_pairs( | ||
| pairs, missing=missing, versioning_enabled_buckets=versioned_buckets | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -200,42 +200,39 @@ def conflict_candidates( | |
| pairs: Sequence[tuple[str | S3Path, str | S3Path]], | ||
| *, | ||
| versioning_enabled_buckets: Collection[str] = (), | ||
| copy: bool = False, | ||
| ) -> list[str | S3Path]: | ||
| """Return the sources of a move whose conflicts depend on an object at their key. | ||
| """Return the sources of a move or a copy whose conflicts depend on an object at their key. | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Self-review round 1 (behavior and implementation; author: Claude Opus 5.5)
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Independent review (relayed result; reviewer: OpenAI Codex CLI 0.169.0, ChatGPT login,
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Independent review follow-up 1 (Codex CLI 0.169.0, |
||
|
|
||
| A source with another source below it may be a directory. If no object | ||
| exists at its key, it is not copied, so it writes no destination and | ||
| is left out of the conflict checks of :meth:`move_pairs`. A path with | ||
| a version always names an object. For a copy, any source may be a | ||
| directory, such as a prefix given in a list, which the copy skips | ||
| (see :meth:`check_copy_pairs`). | ||
| A source without a version may be a directory, such as a prefix given | ||
| in a list or one with other sources below it. If no object exists at | ||
| its key, it is not copied, so it writes no destination and is left | ||
| out of the conflict checks of :meth:`move_pairs` and | ||
| :meth:`check_copy_pairs`. A path with a version always names an | ||
| object. | ||
|
|
||
| Args: | ||
| pairs: The sources and destinations of the move, as | ||
| pairs: The sources and destinations of the move or the copy, as | ||
| :meth:`copy_pairs` or :meth:`s3_copy_pairs` of this pairing | ||
| returns them. A string is parsed; an ``S3Path`` is used as | ||
| it is. | ||
| versioning_enabled_buckets: Buckets whose versioning is enabled. | ||
| Their ``null`` versions are distinct from their unversioned | ||
| keys. Use the same collection for :meth:`move_pairs`. | ||
| copy: Whether the pairs are those of a copy, whose candidates | ||
| include every source without a version whose destination | ||
| conflicts. | ||
| keys. Use the same collection for :meth:`move_pairs` or | ||
| :meth:`check_copy_pairs`. | ||
|
|
||
| Returns: | ||
| The sources that have another source below them and a | ||
| destination that conflicts, one per pair in the order of the | ||
| pairs, as their :attr:`~pyathena.filesystem.s3_path.S3Path.target`: | ||
| in ``bucket/key`` form for a source string, and as an ``S3Path`` | ||
| for an ``S3Path`` source. Empty if nothing needs to be looked up. | ||
| The sources without a version whose destination conflicts, one | ||
| per pair in the order of the pairs, as their | ||
| :attr:`~pyathena.filesystem.s3_path.S3Path.target`: in | ||
| ``bucket/key`` form for a source string, and as an ``S3Path`` for | ||
| an ``S3Path`` source. Empty if nothing needs to be looked up. | ||
|
|
||
| Raises: | ||
| TypeError: If ``versioning_enabled_buckets`` is a string. | ||
| """ | ||
| return [ | ||
| pair.source if isinstance(pair.path1, S3Path) else str(pair.source) | ||
| for pair in self._moves(pairs, versioning_enabled_buckets, copy)[2] | ||
| for pair in self._moves(pairs, versioning_enabled_buckets)[2] | ||
| ] | ||
|
|
||
| def move_pairs( | ||
|
|
@@ -266,8 +263,8 @@ def move_pairs( | |
| Raises: | ||
| ValueError: If two sources have the same destination, or a | ||
| destination is another source, including one left in place, | ||
| except for a directory with no object at its key, which is not | ||
| copied. Also if ``missing`` is needed and None. | ||
| except for a source without a version in ``missing``, which | ||
| writes no destination. Also if ``missing`` is needed and None. | ||
| TypeError: If ``missing`` is a string instead of a collection. | ||
| Also if ``versioning_enabled_buckets`` is a string. | ||
| """ | ||
|
|
@@ -301,9 +298,9 @@ def check_copy_pairs( | |
|
|
||
| Raises: | ||
| ValueError: If two sources have the same destination, or a | ||
| destination is another source, except for a directory with no | ||
| object at its key, which is not copied. Also if ``missing`` is | ||
| needed and None. | ||
| destination is another source, except for a source without a | ||
| version in ``missing``, which writes no destination. Also if | ||
| ``missing`` is needed and None. | ||
| TypeError: If ``missing`` is a string instead of a collection. | ||
| Also if ``versioning_enabled_buckets`` is a string. | ||
| """ | ||
|
|
@@ -335,15 +332,14 @@ def _check( | |
| """ | ||
| if isinstance(missing, str): | ||
| raise TypeError("missing is a collection of paths, not a path.") | ||
| named, sources, candidates = self._moves( | ||
| pairs, versioning_enabled_buckets, operation == "copy" | ||
| ) | ||
| named, sources, candidates = self._moves(pairs, versioning_enabled_buckets) | ||
| if missing is None and candidates: | ||
| raise ValueError("missing is needed to check the pairs.") | ||
| # A directory without an object at its key writes no destination. | ||
| skipped = {pair.source for pair in candidates}.intersection( | ||
| # A source without an object at its key, such as a directory, writes | ||
| # no destination. | ||
| skipped = { | ||
| self._target(S3Path.of(path), versioning_enabled_buckets) for path in missing or () | ||
| ) | ||
| } | ||
| # A path with a version always names an object, so it writes its | ||
| # destination even when the key has no current object. | ||
| writers = Counter( | ||
|
|
@@ -416,14 +412,12 @@ def _target(path: S3Path, versioning_enabled_buckets: Collection[str]) -> S3Path | |
| def _moves( | ||
| pairs: Sequence[tuple[str | S3Path, str | S3Path]], | ||
| versioning_enabled_buckets: Collection[str] = (), | ||
| copy: bool = False, | ||
| ) -> tuple[list[_NamedPair], set[S3Path], list[_NamedPair]]: | ||
| """Compare the paths of a move by what they name. | ||
| """Compare the paths of a move or a copy by what they name. | ||
|
|
||
| Args: | ||
| pairs: The sources and destinations of the move. | ||
| pairs: The sources and destinations of the move or the copy. | ||
| versioning_enabled_buckets: Buckets whose versioning is enabled. | ||
| copy: Whether every conflicting source is a candidate. | ||
|
|
||
| Returns: | ||
| Each pair with whether its source has a version and the targets | ||
|
|
@@ -452,21 +446,12 @@ def _moves( | |
| ) | ||
| # The sources left in place count too; a copy onto one overwrites it. | ||
| sources = {pair.source for pair in named} | ||
| # A source with another source below it may be a directory, named | ||
| # as it is listed: the parents of a name are cut at its last slash. | ||
| directories: set[str] = set() | ||
| for source in sources: | ||
| parent = source.name.rpartition("/")[0] | ||
| while parent and parent not in directories: | ||
| directories.add(parent) | ||
| parent = parent.rpartition("/")[0] | ||
| counts = Counter(pair.destination for pair in named if pair.source != pair.destination) | ||
| candidates = [ | ||
| pair | ||
| for pair in named | ||
| if pair.source != pair.destination | ||
| and (counts[pair.destination] > 1 or pair.destination in sources) | ||
| and (copy or pair.source.name in directories) | ||
| and not pair.versioned | ||
| ] | ||
| return named, sources, candidates | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2329,20 +2329,23 @@ def test_cp_file_directory(self): | |
| ], | ||
| ) | ||
| def test_mv_conflicting_destinations(self, path1, path2): | ||
| # The sources whose destinations conflict are looked up, and objects | ||
| # at their keys raise before anything is copied. | ||
| fs = self._make_fs() | ||
| fs.info = mock.MagicMock() | ||
| fs._call.return_value = {} | ||
|
|
||
| with pytest.raises(ValueError, match="Cannot move"): | ||
| fs.mv(path1, path2, recursive=True) | ||
| fs.info.assert_not_called() | ||
| expected = {fs._client.head_object} | ||
| if any(S3Path.parse(p).version_id == "null" for p in path1): | ||
| fs._call.assert_called_once_with(fs._client.get_bucket_versioning, Bucket="bucket") | ||
| else: | ||
| fs._call.assert_not_called() | ||
| expected.add(fs._client.get_bucket_versioning) | ||
| assert {c.args[0] for c in fs._call.call_args_list} == expected | ||
|
|
||
| def test_mv_conflicting_destinations_of_listed_sources(self): | ||
| # A list of sources is mapped without looking up the destination. | ||
| # A list of sources is mapped without looking up the destination; | ||
| # only the sources are looked up. | ||
| fs = self._make_fs() | ||
| fs._expand_paths = mock.MagicMock( | ||
| return_value=[(p, S3Path.parse(p)) for p in ("bucket/x/a", "bucket/y/a")] | ||
|
|
@@ -2352,7 +2355,21 @@ def test_mv_conflicting_destinations_of_listed_sources(self): | |
| with pytest.raises(ValueError, match="same destination"): | ||
| fs.mv(["s3://bucket/x/a", "s3://bucket/y/a"], "s3://bucket/out", recursive=True) | ||
| fs.isdir.assert_not_called() | ||
| fs._call.assert_not_called() | ||
| assert sorted((c.args[0], c.kwargs["Key"]) for c in fs._call.call_args_list) == [ | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Independent review, finding 2 (Codex, see the record on (P3) |
||
| (fs._client.head_object, "x/a"), | ||
| (fs._client.head_object, "y/a"), | ||
| ] | ||
|
|
||
| def test_mv_skipped_directory_does_not_conflict(self): | ||
| # GH-1397: a listed source without an object at its key, a prefix | ||
| # that the move skips, writes no destination, so its destination | ||
| # being another source is no conflict, as with copy(). It used to | ||
| # be looked up only with another source below it. | ||
| fs = self._make_fs() | ||
| store = self._serve_keys(fs, {"dir/a", "x"}) | ||
|
|
||
| fs.mv(["s3://bucket/dir", "s3://bucket/x"], ["s3://bucket/x", "s3://bucket/y"]) | ||
| assert store == {"dir/a", "y"} | ||
|
|
||
| def test_mv_keeps_given_paths(self): | ||
| # The destination keeps its trailing slash, as with copy(), and the | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Self-review round 2 (claims, callers, operations; author: Claude Opus 5.5)
89ffcd9705b3d7f9adffba44f24f773868c10773, head17bd9ddb02a1e672d540e7aaa8f36f45aa3bc838; repair788aa81cf76c63afe795b581c0d682f71e988b1e.test_mv_skipped_directory_does_not_conflict);AioS3FileSystem._mv()pairs through_move_pairs()(s3_async.py:778); thecopykeyword shipped in v4.3.0 (git tag --contains); moves without name conflicts send no extra request (candidates need a shared or source destination);_head_object()returns None only forFileNotFoundError, so a denied lookup raises;mvraises for the same pairs ascp(one_check()); the docstrings ofmv(),copy(),_mv(),move_pairs(),check_copy_pairs(),conflict_candidates(); the commit messages.788aa81c).test_move_pairs_path_conflict_candidates); the_check()rationale and theFileNotFoundErrortiming were imprecise (the copies that ran before or alongside it stay in place; with a failure, the sync move starts no further copy).17bd9ddb, 12 of the changed and new tests fail (the regression test, the 5 request tests, the 2 Stubber tests, 4S3PathPairingtests; the other 3 only gainedmissing=set()). Local runs are mock or Stubber runs with dummy credentials (410 passed; the 19 failures are AWS integration tests that fail the same way at89ffcd97). Not run yet: the AWS suite (local login expired) and AWS CI.s3:GetObjecton the sources is already needed to copy them.