Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions docs/filesystem.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,8 +147,11 @@ and a `cp` destination that is another source. Local paths that do not exist yet
compared as strings, after `os.path.normcase`, so on a case-insensitive filesystem other
than Windows, such as the default one of macOS, give `get` destinations that differ by
more than case. A `cp` of an object onto itself, such as with
`MetadataDirective="REPLACE"`, is no conflict. `get` with `outfile` writes the objects
into it one after another, in order.
`MetadataDirective="REPLACE"`, is no conflict. `mv` raises for the same pairs as `cp`.
A source of `cp` or `mv` without an object at its key, such as a key prefix given in a
list, writes no destination and does not conflict; the sources without a version whose

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round 2 (claims, callers, operations; author: Claude Opus 5.5)

  • Base 89ffcd9705b3d7f9adffba44f24f773868c10773, head 17bd9ddb02a1e672d540e7aaa8f36f45aa3bc838; repair 788aa81cf76c63afe795b581c0d682f71e988b1e.
  • Claims checked against the code and runs: the issue's reproduction (test_mv_skipped_directory_does_not_conflict); AioS3FileSystem._mv() pairs through _move_pairs() (s3_async.py:778); the copy keyword shipped in v4.3.0 (git tag --contains); moves without name conflicts send no extra request (candidates need a shared or source destination); _head_object() returns None only for FileNotFoundError, so a denied lookup raises; mv raises for the same pairs as cp (one _check()); the docstrings of mv(), copy(), _mv(), move_pairs(), check_copy_pairs(), conflict_candidates(); the commit messages.
  • Result: FINDINGS, corrected:
    1. This paragraph said the conflicting sources are looked up; only those without a version are (788aa81c).
    2. PR body: 14 existing tests changed, not 13 (the rebase added Act on listed identities in pairing and copy/mv/get (GH-1403) #1421's test_move_pairs_path_conflict_candidates); the _check() rationale and the FileNotFoundError timing were imprecise (the copies that ran before or alongside it stay in place; with a failure, the sync move starts no further copy).
  • Evidence: with the branch's source changes reverted at 17bd9ddb, 12 of the changed and new tests fail (the regression test, the 5 request tests, the 2 Stubber tests, 4 S3PathPairing tests; the other 3 only gained missing=set()). Local runs are mock or Stubber runs with dummy credentials (410 passed; the 19 failures are AWS integration tests that fail the same way at 89ffcd97). Not run yet: the AWS suite (local login expired) and AWS CI.
  • Operators: the extra lookups are HeadObject requests with the source's parameters, through the core's retries, only for pairs that conflict by name; s3:GetObject on the sources is already needed to copy them.

destinations conflict are looked up with HeadObject before anything is copied. `get`
with `outfile` writes the objects into it one after another, in order.

`rm` deletes up to 1,000 objects per DeleteObjects request, `batch_size` requests at a
time. Its `max_workers` argument is a deprecated alias of `batch_size`.
Expand Down
23 changes: 13 additions & 10 deletions pyathena/filesystem/s3.py
Original file line number Diff line number Diff line change
Expand Up @@ -2709,6 +2709,8 @@ def mv(self, path1, path2, recursive=False, maxdepth=None, batch_size=None, **kw
GetBucketVersioning once per bucket during the move's planning and
requires ``s3:GetBucketVersioning``. The result is not cached across
moves. Directory buckets do not support versioning and need no lookup.
A source without a version whose destination conflicts with another

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round 1, not changed (with reasons)

pair is looked up with HeadObject during the planning too.

The files are copied in parallel as ``copy()`` copies them, and are
deleted once every copy has succeeded; nothing is deleted after a
Expand All @@ -2731,9 +2733,12 @@ def mv(self, path1, path2, recursive=False, maxdepth=None, batch_size=None, **kw
Raises:
ValueError: If two sources have the same destination, or a
destination is another source, including one left in place,
which is checked before anything is copied. A directory with
no object at its key, which is not copied, does not conflict.
Also if ``batch_size`` is neither positive, -1, nor None.
which is checked before anything is copied. A source without
an object at its key, such as a directory, writes no
destination and does not conflict. Also if ``batch_size`` is
neither positive, -1, nor None.
FileNotFoundError: If a source does not exist, once the running
copies have ended; nothing is deleted.
OSError: If bucket versioning cannot be read.
"""
path1, path2 = self._stringify_paths(path1), self._stringify_paths(path2)
Expand Down Expand Up @@ -2810,7 +2815,6 @@ def _conflict_lookups(
pairing: S3PathPairing,
pairs: list[tuple[str | S3Path, str | S3Path]],
source_params: Mapping[str, Any] | None = None,
copy: bool = False,
) -> tuple[set[str | S3Path], set[str]]:
"""Look up what the conflict checks of a move or a copy need.

Expand All @@ -2822,7 +2826,6 @@ def _conflict_lookups(
source_params: The parameters of the requests that read the
sources (see :meth:`S3Core.copy_source_params`), sent with
the lookups of the sources.
copy: Whether the pairs are those of a copy.

Returns:
The conflict candidates with no object at their key, and the
Expand All @@ -2846,7 +2849,7 @@ def _conflict_lookups(
missing = {
source
for source in pairing.conflict_candidates(
pairs, versioning_enabled_buckets=versioned_buckets, copy=copy
pairs, versioning_enabled_buckets=versioned_buckets
)
if self._head_object(S3Path.of(source), lookup_kwargs=lookup_kwargs) is None
}
Expand Down Expand Up @@ -2937,9 +2940,9 @@ def copy(

Raises:
ValueError: If two sources have the same destination, or a
destination is another source, except for a directory with no
object at its key, which is not copied. Also if
``batch_size`` is neither positive, -1, nor None.
destination is another source, except for a source without an
object at its key, such as a directory, which is not copied.
Also if ``batch_size`` is neither positive, -1, nor None.
OSError: If bucket versioning cannot be read for the checks.
"""
path1, path2 = self._stringify_paths(path1), self._stringify_paths(path2)
Expand All @@ -2951,7 +2954,7 @@ def copy(
if not pairs:
return
missing, versioned_buckets = self._conflict_lookups(
pairing, pairs, self.core.copy_source_params(kwargs), copy=True
pairing, pairs, self.core.copy_source_params(kwargs)
)
pairing.check_copy_pairs(
pairs, missing=missing, versioning_enabled_buckets=versioned_buckets
Expand Down
7 changes: 5 additions & 2 deletions pyathena/filesystem/s3_async.py
Original file line number Diff line number Diff line change
Expand Up @@ -763,8 +763,11 @@ async def _mv(self, path1, path2, recursive=False, maxdepth=None, **kwargs) -> N
Raises:
ValueError: If two sources have the same destination, or a
destination is another source, including one left in place,
which is checked before anything is copied. A directory with
no object at its key, which is not copied, does not conflict.
which is checked before anything is copied. A source without
an object at its key, such as a directory, writes no
destination and does not conflict.
FileNotFoundError: If a source does not exist, once the other
copies have ended; nothing is deleted.
OSError: If bucket versioning cannot be read.
"""
path1, path2 = self._sync_fs._stringify_paths(path1), self._sync_fs._stringify_paths(path2)
Expand Down
71 changes: 28 additions & 43 deletions pyathena/filesystem/s3_path_pairing.py
Original file line number Diff line number Diff line change
Expand Up @@ -200,42 +200,39 @@ def conflict_candidates(
pairs: Sequence[tuple[str | S3Path, str | S3Path]],
*,
versioning_enabled_buckets: Collection[str] = (),
copy: bool = False,
) -> list[str | S3Path]:
"""Return the sources of a move whose conflicts depend on an object at their key.
"""Return the sources of a move or a copy whose conflicts depend on an object at their key.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round 1 (behavior and implementation; author: Claude Opus 5.5)

  • Base b720199003ff8e84d7b574334343b1202cf9d7bc, initial head 104d04f79131198a891606dff4724e3fb7535e9c. Repairs: b8ab83eb, d942f59f; rebased onto 89ffcd9705b3d7f9adffba44f24f773868c10773 (Act on listed identities in pairing and copy/mv/get (GH-1403) #1421) as fe5fe00f6ffd1026c4cc64ec07d86b675002d1af, e97904f96b8e663a5f757375308d6bc24cfc965e, 17bd9ddb02a1e672d540e7aaa8f36f45aa3bc838.
  • Covered: S3PathPairing.conflict_candidates(), _moves(), _check(); S3FileSystem._conflict_lookups(), _move_pairs(), mv(), copy(); AioS3FileSystem._mv() through _move_pairs(); docs/filesystem.md; the 13 changed tests and the new regression test. Manual pass, /code-review (high), and /simplify (reuse, simplification, efficiency, altitude).
  • Result: FINDINGS, repaired:
    1. The docstrings of move_pairs(), check_copy_pairs(), mv(), copy(), and AioS3FileSystem._mv() exempted only a directory, while any source without an object at its key is exempt, and mv() did not name the FileNotFoundError of a source that does not exist (e97904f9).
    2. _check() intersected missing with the candidates; with one rule, a source in missing that is not a candidate writes a destination that nothing else writes, so the filter changed nothing. Removed, with the skip stated once in mv() and the 404 lookup of the move stub bound once (17bd9ddb).
    3. Rebase onto Act on listed identities in pairing and copy/mv/get (GH-1403) #1421: the pairs may hold S3Path sources (_NamedPair); the unified rule keeps S3Path candidates, and Act on listed identities in pairing and copy/mv/get (GH-1403) #1421's test_move_pairs_path_conflict_candidates now expects both of its S3Path sources.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (relayed result; reviewer: OpenAI Codex CLI 0.169.0, ChatGPT login, codex exec -s read-only, default model and effort; static review, no tests run)

  • Base 89ffcd9705b3d7f9adffba44f24f773868c10773, head 788aa81cf76c63afe795b581c0d682f71e988b1e, from a detached snapshot that stayed unchanged; prompt without the PR text, commit messages, or prior findings.
  • Covered (reviewer): sync move/copy planning, lookups, caching, source parameters, deletion after copies; async planning, typed paths, failures, cancellation; S3PathPairing callers, typed keys, versions, null-version buckets; the changed docs and every changed test with the stub's two consumers.
  • Result: FINDINGS (2 introduced). The conflict computation and the prose about missing sources otherwise agree with the code; explicit versions stay writers, and typed keys keep their literal identity. A cancellation edge of _mv() (gather cancelling already-cancelling children) was reported as pre-existing, excluded by the existing test matrix and documented in docs/aio.md.
  1. (P2) Removing the copy keyword of the public conflict_candidates() makes conflict_candidates(pairs, copy=True) raise TypeError; keeping it as an accepted argument was suggested. Not changed: the maintainer decided to remove it, with a release note, as 4.x minors have carried breaking changes with notes (Run the batch operations of S3FileSystem concurrently (GH-1322) #1344, Raise FileNotFoundError from ls() of a missing key path (GH-1330) #1347, Inject the filesystem's S3 client with client= and retry_config= (GH-1363) #1378).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review follow-up 1 (Codex CLI 0.169.0, codex exec -s read-only, static; repair diff 788aa81cf76c63afe795b581c0d682f71e988b1e..fb1638214dc6e9642074d3ed74b043598e7fc9b3, merge-base 89ffcd9705b3d7f9adffba44f24f773868c10773, with the full change for context; snapshot unchanged): CLEAN. Sorting keeps the method, key, and call-count checks while dropping the order; no docstring inaccuracies, vacuous assertions, timing dependencies, or regressions found. Self-review of the repair (behavior and claims): the sorted comparison still fails without the lookups (an empty call list), and no claim changed.


A source with another source below it may be a directory. If no object
exists at its key, it is not copied, so it writes no destination and
is left out of the conflict checks of :meth:`move_pairs`. A path with
a version always names an object. For a copy, any source may be a
directory, such as a prefix given in a list, which the copy skips
(see :meth:`check_copy_pairs`).
A source without a version may be a directory, such as a prefix given
in a list or one with other sources below it. If no object exists at
its key, it is not copied, so it writes no destination and is left
out of the conflict checks of :meth:`move_pairs` and
:meth:`check_copy_pairs`. A path with a version always names an
object.

Args:
pairs: The sources and destinations of the move, as
pairs: The sources and destinations of the move or the copy, as
:meth:`copy_pairs` or :meth:`s3_copy_pairs` of this pairing
returns them. A string is parsed; an ``S3Path`` is used as
it is.
versioning_enabled_buckets: Buckets whose versioning is enabled.
Their ``null`` versions are distinct from their unversioned
keys. Use the same collection for :meth:`move_pairs`.
copy: Whether the pairs are those of a copy, whose candidates
include every source without a version whose destination
conflicts.
keys. Use the same collection for :meth:`move_pairs` or
:meth:`check_copy_pairs`.

Returns:
The sources that have another source below them and a
destination that conflicts, one per pair in the order of the
pairs, as their :attr:`~pyathena.filesystem.s3_path.S3Path.target`:
in ``bucket/key`` form for a source string, and as an ``S3Path``
for an ``S3Path`` source. Empty if nothing needs to be looked up.
The sources without a version whose destination conflicts, one
per pair in the order of the pairs, as their
:attr:`~pyathena.filesystem.s3_path.S3Path.target`: in
``bucket/key`` form for a source string, and as an ``S3Path`` for
an ``S3Path`` source. Empty if nothing needs to be looked up.

Raises:
TypeError: If ``versioning_enabled_buckets`` is a string.
"""
return [
pair.source if isinstance(pair.path1, S3Path) else str(pair.source)
for pair in self._moves(pairs, versioning_enabled_buckets, copy)[2]
for pair in self._moves(pairs, versioning_enabled_buckets)[2]
]

def move_pairs(
Expand Down Expand Up @@ -266,8 +263,8 @@ def move_pairs(
Raises:
ValueError: If two sources have the same destination, or a
destination is another source, including one left in place,
except for a directory with no object at its key, which is not
copied. Also if ``missing`` is needed and None.
except for a source without a version in ``missing``, which
writes no destination. Also if ``missing`` is needed and None.
TypeError: If ``missing`` is a string instead of a collection.
Also if ``versioning_enabled_buckets`` is a string.
"""
Expand Down Expand Up @@ -301,9 +298,9 @@ def check_copy_pairs(

Raises:
ValueError: If two sources have the same destination, or a
destination is another source, except for a directory with no
object at its key, which is not copied. Also if ``missing`` is
needed and None.
destination is another source, except for a source without a
version in ``missing``, which writes no destination. Also if
``missing`` is needed and None.
TypeError: If ``missing`` is a string instead of a collection.
Also if ``versioning_enabled_buckets`` is a string.
"""
Expand Down Expand Up @@ -335,15 +332,14 @@ def _check(
"""
if isinstance(missing, str):
raise TypeError("missing is a collection of paths, not a path.")
named, sources, candidates = self._moves(
pairs, versioning_enabled_buckets, operation == "copy"
)
named, sources, candidates = self._moves(pairs, versioning_enabled_buckets)
if missing is None and candidates:
raise ValueError("missing is needed to check the pairs.")
# A directory without an object at its key writes no destination.
skipped = {pair.source for pair in candidates}.intersection(
# A source without an object at its key, such as a directory, writes
# no destination.
skipped = {
self._target(S3Path.of(path), versioning_enabled_buckets) for path in missing or ()
)
}
# A path with a version always names an object, so it writes its
# destination even when the key has no current object.
writers = Counter(
Expand Down Expand Up @@ -416,14 +412,12 @@ def _target(path: S3Path, versioning_enabled_buckets: Collection[str]) -> S3Path
def _moves(
pairs: Sequence[tuple[str | S3Path, str | S3Path]],
versioning_enabled_buckets: Collection[str] = (),
copy: bool = False,
) -> tuple[list[_NamedPair], set[S3Path], list[_NamedPair]]:
"""Compare the paths of a move by what they name.
"""Compare the paths of a move or a copy by what they name.

Args:
pairs: The sources and destinations of the move.
pairs: The sources and destinations of the move or the copy.
versioning_enabled_buckets: Buckets whose versioning is enabled.
copy: Whether every conflicting source is a candidate.

Returns:
Each pair with whether its source has a version and the targets
Expand Down Expand Up @@ -452,21 +446,12 @@ def _moves(
)
# The sources left in place count too; a copy onto one overwrites it.
sources = {pair.source for pair in named}
# A source with another source below it may be a directory, named
# as it is listed: the parents of a name are cut at its last slash.
directories: set[str] = set()
for source in sources:
parent = source.name.rpartition("/")[0]
while parent and parent not in directories:
directories.add(parent)
parent = parent.rpartition("/")[0]
counts = Counter(pair.destination for pair in named if pair.source != pair.destination)
candidates = [
pair
for pair in named
if pair.source != pair.destination
and (counts[pair.destination] > 1 or pair.destination in sources)
and (copy or pair.source.name in directories)
and not pair.versioned
]
return named, sources, candidates
Expand Down
27 changes: 22 additions & 5 deletions tests/pyathena/filesystem/test_s3.py
Original file line number Diff line number Diff line change
Expand Up @@ -2329,20 +2329,23 @@ def test_cp_file_directory(self):
],
)
def test_mv_conflicting_destinations(self, path1, path2):
# The sources whose destinations conflict are looked up, and objects
# at their keys raise before anything is copied.
fs = self._make_fs()
fs.info = mock.MagicMock()
fs._call.return_value = {}

with pytest.raises(ValueError, match="Cannot move"):
fs.mv(path1, path2, recursive=True)
fs.info.assert_not_called()
expected = {fs._client.head_object}
if any(S3Path.parse(p).version_id == "null" for p in path1):
fs._call.assert_called_once_with(fs._client.get_bucket_versioning, Bucket="bucket")
else:
fs._call.assert_not_called()
expected.add(fs._client.get_bucket_versioning)
assert {c.args[0] for c in fs._call.call_args_list} == expected

def test_mv_conflicting_destinations_of_listed_sources(self):
# A list of sources is mapped without looking up the destination.
# A list of sources is mapped without looking up the destination;
# only the sources are looked up.
fs = self._make_fs()
fs._expand_paths = mock.MagicMock(
return_value=[(p, S3Path.parse(p)) for p in ("bucket/x/a", "bucket/y/a")]
Expand All @@ -2352,7 +2355,21 @@ def test_mv_conflicting_destinations_of_listed_sources(self):
with pytest.raises(ValueError, match="same destination"):
fs.mv(["s3://bucket/x/a", "s3://bucket/y/a"], "s3://bucket/out", recursive=True)
fs.isdir.assert_not_called()
fs._call.assert_not_called()
assert sorted((c.args[0], c.kwargs["Key"]) for c in fs._call.call_args_list) == [

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review, finding 2 (Codex, see the record on s3_path_pairing.py:204)

(P3) test_mv_conflicting_destinations_of_listed_sources required the lookups in pair order, which the contract of mv() does not state. Fixed in fb163821: the lookups are compared sorted, with the no-write assertions kept.

(fs._client.head_object, "x/a"),
(fs._client.head_object, "y/a"),
]

def test_mv_skipped_directory_does_not_conflict(self):
# GH-1397: a listed source without an object at its key, a prefix
# that the move skips, writes no destination, so its destination
# being another source is no conflict, as with copy(). It used to
# be looked up only with another source below it.
fs = self._make_fs()
store = self._serve_keys(fs, {"dir/a", "x"})

fs.mv(["s3://bucket/dir", "s3://bucket/x"], ["s3://bucket/x", "s3://bucket/y"])
assert store == {"dir/a", "y"}

def test_mv_keeps_given_paths(self):
# The destination keeps its trailing slash, as with copy(), and the
Expand Down
Loading
Loading