Skip to content

HDDS-16364. Exclude multipart open keys from Recon open key reports - #11685

Open
lpavanvenkat wants to merge 4 commits into
apache:masterfrom
lpavanvenkat:HDDS-16364
Open

lpavanvenkat wants to merge 4 commits into
apache:masterfrom
lpavanvenkat:HDDS-16364

Conversation

@lpavanvenkat

@lpavanvenkat lpavanvenkat commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Recon reports incomplete S3 multipart uploads (MPUs) twice. When an MPU is initiated, OM writes a multipart open key
(isMultipartKey=true, size 0) to the open key/file table, and tracks the upload in multipartInfoTable and
multipartPartsTable. Recon counted the multipart open key as a regular open key, so the same upload appeared both
in the open key endpoints, with 0 bytes, and in /keys/open/mpu/summary, with the size of its committed parts.

Following the review discussion, multipart open keys are now excluded from the open key endpoints, and incomplete
MPUs are reported only by /keys/open/mpu/summary. For one MPU with 10 MiB of committed parts:

Endpoint Before After
/keys/open lists the multipart open key with size 0 not listed
/keys/open/summary 1 open key, 0 bytes 0 open keys, 0 bytes
/keys/open/mpu/summary 1 MPU, 10 MiB 1 MPU, 10 MiB (unchanged)

Open keys of parts that are still being written (isMultipartKey=false) are regular open keys and are still listed
and counted.

Changes:

  • /keys/open: ReconUtils.extractKeysFromTable has a new overload that takes a filter, and
    OMDBInsightEndpoint passes !OMMultipartUploadUtils.isMultipartKeySet(keyInfo) for the OBS scan and both FSO
    scans. The filter is applied during the scan, so excluded keys do not count towards limit and paging is not
    affected.
  • /keys/open/summary: OpenKeysInsightHandler skips multipart open keys on put and delete events and in the
    full recount (getTableSizeAndCount).
  • Upgrade: open key totals stored by an older Recon include multipart open keys. A new Recon version
    OPEN_KEYS_EXCLUDE_MULTIPART_UPLOADS (7) and its upgrade action OpenKeysExcludeMultipartUploadsUpgradeAction
    queue a task reinitialization to recompute them. If the reinitialization cannot be queued at startup (for example,
    because Recon has not loaded its OM DB copy yet), the controller retries it on the next OM sync, the same way as
    the NSSummary rebuild upgrade action.
  • MPU size stats after a Recon restart: MultipartInfoInsightHandler caches each split-layout MPU's replication
    config to compute the replicated size of part events (HDDS-16363). The cache was filled only on
    multipartInfoTable PUT/UPDATE, so after a Recon restart, aborting or completing an MPU that was already open
    skipped every part delete (No cached replication config ... skipping part size update), leaving its bytes in
    /keys/open/mpu/summary. OM deletes the multipartInfoTable row before its multipartPartsTable rows in the
    same batch, so the handler now also caches the config on the multipartInfoTable DELETE event.

The earlier approach in this PR (adding committed part sizes to the open key endpoints) has been removed.

Known limitation, not changed here:

  • MPUs initiated before HDDS-9017 do not have the multipart flag on their open key (OM falls back to
    multipartInfoTable for them), so their multipart open keys are still reported as open keys.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16364

How was this patch tested?

  • New unit tests:
    • TestOmDBInsightEndPoint#testGetOpenKeyInfoExcludesMultipartOpenKeys: OBS and FSO listings exclude multipart
      open keys, and paging with limit still returns full pages.
    • TestOmTableInsightTask#testProcessExcludesMultipartOpenKeys: put and delete events of multipart open keys do
      not change the open key stats, while regular open keys are still counted.
    • TestOmTableInsightTask#testReprocessExcludesMultipartOpenKeys: the full recount skips multipart open keys.
    • TestOmTableInsightTask#testProcessForSplitSchemaMPUDeletedAfterRestart: a new task instance (empty cache)
      processes an MPU delete followed by its part deletes and the sizes return to 0. It fails without the fix.
    • TestOpenKeysExcludeMultipartUploadsUpgradeAction and TestReconVersionManager: the upgrade action queues the
      reinitialization, does not fail the upgrade if queueing fails, and is registered for version 7.
  • Changed test classes: 130 tests, 0 failures. checkstyle.sh: 0 violations. rat.sh: no unapproved files.
  • Manual upgrade test on the ozone compose cluster (3 datanodes, FSO bucket, 2 MPUs with 2 parts of 5 MiB each),
    starting on master and then upgrading only Recon to this patch with its data kept:
    • On master, /keys/open/summary reported 2 open keys with 0 bytes and /keys/open listed both MPUs.
    • After the upgrade, without triggering an OM sync manually, /keys/open/summary reported 0 open keys and
      /keys/open returned 204. /keys/open/mpu/summary still reported 2 MPUs and 20 MiB.
    • Aborting one MPU kept the open key count at 0 and reduced the MPU summary to 1 MPU and 10 MiB.
    • After a Recon restart, completing the other MPU reduced the MPU summary to 0.
    • While a 20 MiB part was uploading, /keys/open listed only the part's open key and /keys/open/summary
      reported 1 open key and 20 MiB. After the part was committed, the open key count returned to 0 and the MPU
      summary showed 20 MiB.

Co-authored-by: with Cursor (Claude Opus 5.5)

@lpavanvenkat

Copy link
Copy Markdown
Contributor Author

@priyeshkaratha @ArafatKhan2198 @devmadhuu , pls do review this pr, Thanks.

@github-actions github-actions Bot added documentation Improvements or additions to documentation recon labels Oct 22, 2026
smengcl
smengcl previously approved these changes Oct 22, 2026

@smengcl smengcl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@adoroszlai adoroszlai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @lpavanvenkat for the patch. Now that CI checks have completed, please keep this PR the way it is, unless someone requests code changes. I definitely do not want another round of CI to be triggered for this alone. However, in the future please add documentation in ozone-site. Leftover docs are being removed from this repo in #10829.

@chungen0126 chungen0126 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @lpavanvenkat for working on this. Just took a quick look and left a couple of comments.

* Returns the (unreplicated, replicated) size of the committed parts of a single MPU, read from the embedded
* parts for legacy-schema MPUs and from {@code multipartPartsTable} for split-schema MPUs.
*/
public static Pair<Long, Long> getCommittedPartsSize(OmMultipartKeyInfo multipartKeyInfo,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to HDDS-15942, please avoid using Pair.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can now use records in server-side code.

* Returns the (unreplicated, replicated) size of the committed parts of the multipart upload that a multipart
* open key belongs to, or zero if the upload can no longer be found.
*/
private Pair<Long, Long> getMultipartUploadSize(String dbKey, OmKeyInfo keyInfo) throws IOException {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to HDDS-15942, please avoid using Pair.

@adoroszlai

Copy link
Copy Markdown
Contributor

please add documentation in ozone-site

Since changes to avoid using Pair are suggested, please drop the unnecessary doc update as well.

@devmadhuu

Copy link
Copy Markdown
Contributor

@lpavanvenkat largely changes looks fine, just one comment and a nit.

Could we handle older FSO multipart open-key records whose keyName contains only the filename?

Before HDDS-10649, the open-file write path stored the leaf name. For an unfinished upload of dir/test1.txt, this lookup therefore constructs /vol/bucket/test1.txt/uploadId, while the multipart info entry is under /vol/bucket/dir/test1.txt/uploadId.

The lookup returns null and reports zero despite committed parts being present. The new FSO test uses the full dir/mpu-legacy name, so it does not cover this older record format.

Could we resolve the full key path for these records and add a test? If support is intentionally deferred, please document this additional limitation and track the follow-up.

One optional test suggestion: cover erasure-coded uploads.

@chungen0126

Copy link
Copy Markdown
Contributor

Hi @devmadhuu , thanks for catching this edge case! However, do we really need to handle this for backward compatibility? These records only exist on clusters upgraded from a release before ozone 2.0 with incomplete FSO multipart uploads. MultipartUploadCleanupService aborts any MPU older than ozone.om.open.mpu.expire.threshold.

These uploads were initiated before the upgrade, they are gone within about 31 days after upgrade. Is it worth adding more code complexity just to avoid a monitoring inaccuracy during that window?

@sumitagrawl

sumitagrawl commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

@lpavanvenkat @devmadhuu Normally if key is part of open key / open file table, its never recorded about datasize, as it comes only after its committed.

And for MPU, its counted only after part is committed. Till that point, that open part is never counted to MPU. I think, after part is committed to MPU, it will show the bytes used as MPU.

IMO, open key/ open file, should not be counted as part of MPU in summary, but be counted as open key/file only.

  1. OpenKey/OpenFile usages can be approx only, and there we can have approx size based on blocks allocated. This can not give actual used size as during commit, may failed / partial failed blocks can be removed. We can give recheck how replicated/non-replicated size calculated and is correct as per block or not.

@errose28

Copy link
Copy Markdown
Contributor

I have similar concerns to Sumit. Recon is a debugging tool so it is most effective when it reports what is literally stored. Open key stats should be for anything in the open key table, and MPU stats should be for anything in the MPU tables. This helps us effectively troubleshoot issues with background services that work on those tables.

Specifically for this PR, that would mean all open keys (including uncommitted MPU parts) are reported as open key usage. Once a part is committed, it is no longer tracked as an open key and instead tracked using separate stats for committed but incomplete MPU parts.

@devmadhuu

Copy link
Copy Markdown
Contributor

I agree with both Sumit and Ethan on this point that they both should be kept separate.

@lpavanvenkat

Copy link
Copy Markdown
Contributor Author

@errose28 @sumitagrawl agreed on keeping open key and MPU stats separate. My original proposal for HDDS-16364 follows that:

Problem: at MPU initiate, OM writes a multipart open key with size 0 and no blocks, which stays until the upload is completed or aborted. Recon counts it as a regular open key, while committed parts are reported by /keys/open/mpu/summary.

For example, start 2 multipart uploads and commit 2 parts of 5 MiB to each, without completing either upload. Recon then reports:

/keys/open/summary: 2 open keys, 0 bytes
/keys/open/mpu/summary: 2 uploads, 20971520 bytes (20 MiB)
Showing these as 2 open keys with 0 bytes is misleading, since the uploads already hold 20 MiB of committed parts.

Fix (Recon only): exclude multipart open keys (isMultipartKey=true) from /keys/open and /keys/open/summary, and report uploads only via /keys/open/mpu/summary.

With the same 2 uploads, while a third part of one upload is still being written, Recon would report:

/keys/open/summary: 1 open key (the part being written)
/keys/open/mpu/summary: 2 uploads, 20971520 bytes (20 MiB)
Behavior for parts being written is unchanged: they are counted as open keys until they commit, then move to the MPU total.

This matches OM: getExpiredOpenKeys (used by OpenKeyCleanupService) skips these keys, and expired ones are cleaned up only by MultipartUploadCleanupService. The tradeoff is that the stored open key totals must be recomputed after upgrade, via a Recon upgrade action that queues a task reinitialization (as existing upgrade actions do).

Would you prefer this, or leaving the open key view as is?

CC: @devmadhuu @priyeshkaratha

@adoroszlai
adoroszlai dismissed smengcl’s stale review October 27, 2026 15:02

further changes requested by others

@devmadhuu

Copy link
Copy Markdown
Contributor

@errose28 @sumitagrawl agreed on keeping open key and MPU stats separate. My original proposal for HDDS-16364 follows that:

Problem: at MPU initiate, OM writes a multipart open key with size 0 and no blocks, which stays until the upload is completed or aborted. Recon counts it as a regular open key, while committed parts are reported by /keys/open/mpu/summary.

For example, start 2 multipart uploads and commit 2 parts of 5 MiB to each, without completing either upload. Recon then reports:

/keys/open/summary: 2 open keys, 0 bytes /keys/open/mpu/summary: 2 uploads, 20971520 bytes (20 MiB) Showing these as 2 open keys with 0 bytes is misleading, since the uploads already hold 20 MiB of committed parts.

Fix (Recon only): exclude multipart open keys (isMultipartKey=true) from /keys/open and /keys/open/summary, and report uploads only via /keys/open/mpu/summary.

With the same 2 uploads, while a third part of one upload is still being written, Recon would report:

/keys/open/summary: 1 open key (the part being written) /keys/open/mpu/summary: 2 uploads, 20971520 bytes (20 MiB) Behavior for parts being written is unchanged: they are counted as open keys until they commit, then move to the MPU total.

This matches OM: getExpiredOpenKeys (used by OpenKeyCleanupService) skips these keys, and expired ones are cleaned up only by MultipartUploadCleanupService. The tradeoff is that the stored open key totals must be recomputed after upgrade, via a Recon upgrade action that queues a task reinitialization (as existing upgrade actions do).

Would you prefer this, or leaving the open key view as is?

CC: @devmadhuu @priyeshkaratha

Thanks @lpavanvenkat for giving clarity and explanation. IMO above makes sense to filter out committed key parts from openkey summary using flag isMultipartKey=true.

@sumitagrawl @errose28 any concerns on this ?

@priyeshkaratha priyeshkaratha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @lpavanvenkat Please check the inline comments provided. Also the two endpoints can report different totals.

  • /keys/open adds up live per-row lookups.
  • /keys/open/summary reads the counters maintained by Recon's background stats task.

Please check this.

* Returns the (unreplicated, replicated) size of the committed parts of a single MPU, read from the embedded
* parts for legacy-schema MPUs and from {@code multipartPartsTable} for split-schema MPUs.
*/
public static Pair<Long, Long> getCommittedPartsSize(OmMultipartKeyInfo multipartKeyInfo,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method re-implements what applyLegacyPartSizes and applyPartSize do, and its replicated-size logic differs (OMMultipartUploadUtils.getReplicatedSize versus QuotaUtil per part). It's easy for the two to drift apart, for example in how EC padding is handled. One shared helper would fix this.

public Map<String, Long> getOpenKeySummaryWithMPU() {
Map<String, Long> keysSummary = getOpenKeySummary();
Map<String, Long> mpuSummary = getMPUKeySummary();
keysSummary.merge("totalReplicatedDataSize", mpuSummary.get("totalReplicatedDataSize"), Long::sum);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only /keys/open/summary uses getOpenKeySummaryWithMPU. Other callers of getOpenKeySummary(), such as the cluster-state and overview pages, still leave out multipart part sizes. Users would see different open-key totals depending on the page.

@lpavanvenkat

Copy link
Copy Markdown
Contributor Author

@errose28 @sumitagrawl agreed on keeping open key and MPU stats separate. My original proposal for HDDS-16364 follows that:
Problem: at MPU initiate, OM writes a multipart open key with size 0 and no blocks, which stays until the upload is completed or aborted. Recon counts it as a regular open key, while committed parts are reported by /keys/open/mpu/summary.
For example, start 2 multipart uploads and commit 2 parts of 5 MiB to each, without completing either upload. Recon then reports:
/keys/open/summary: 2 open keys, 0 bytes /keys/open/mpu/summary: 2 uploads, 20971520 bytes (20 MiB) Showing these as 2 open keys with 0 bytes is misleading, since the uploads already hold 20 MiB of committed parts.
Fix (Recon only): exclude multipart open keys (isMultipartKey=true) from /keys/open and /keys/open/summary, and report uploads only via /keys/open/mpu/summary.
With the same 2 uploads, while a third part of one upload is still being written, Recon would report:
/keys/open/summary: 1 open key (the part being written) /keys/open/mpu/summary: 2 uploads, 20971520 bytes (20 MiB) Behavior for parts being written is unchanged: they are counted as open keys until they commit, then move to the MPU total.
This matches OM: getExpiredOpenKeys (used by OpenKeyCleanupService) skips these keys, and expired ones are cleaned up only by MultipartUploadCleanupService. The tradeoff is that the stored open key totals must be recomputed after upgrade, via a Recon upgrade action that queues a task reinitialization (as existing upgrade actions do).
Would you prefer this, or leaving the open key view as is?
CC: @devmadhuu @priyeshkaratha

Thanks @lpavanvenkat for giving clarity and explanation. IMO above makes sense to filter out committed key parts from openkey summary using flag isMultipartKey=true.

@sumitagrawl @errose28 any concerns on this ?

Thanks @devmadhuu. Small clarification: committed parts are already out of the open key table, since OM deletes a part's open key on commit. The filter would drop only the size-0 multipart open key that OM writes at initiate (isMultipartKey=true). Parts still being written stay counted as open keys.

@errose28

errose28 commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Thanks @lpavanvenkat for the clarification. That explanation makes sense. It looks different than the PR description though:

for one MPU with 10 MiB of committed parts:

Endpoint Before After
/keys/open row 0 10 MiB
/keys/open/summary 0 10 MiB
/keys/open/mpu/summary 10 MiB 10 MiB (unchanged)

This seems to indicate that the 10mb of committed but not completed parts were not counted towards open key size before this PR, but that they are after the PR. Your latest comment says that the 10mb of completed parts would not count towards the open key size.

For example, start 2 multipart uploads and commit 2 parts of 5 MiB to each, without completing either upload. Recon then reports:

/keys/open/summary: 2 open keys, 0 bytes

@lpavanvenkat

Copy link
Copy Markdown
Contributor Author

Thanks @lpavanvenkat for the clarification. That explanation makes sense. It looks different than the PR description though:

for one MPU with 10 MiB of committed parts:

Endpoint Before After
/keys/open row 0 10 MiB
/keys/open/summary 0 10 MiB
/keys/open/mpu/summary 10 MiB 10 MiB (unchanged)
This seems to indicate that the 10mb of committed but not completed parts were not counted towards open key size before this PR, but that they are after the PR. Your latest comment says that the 10mb of completed parts would not count towards the open key size.

For example, start 2 multipart uploads and commit 2 parts of 5 MiB to each, without completing either upload. Recon then reports:

/keys/open/summary: 2 open keys, 0 bytes

Thanks @errose28, you're right, they don't match. The PR description describes the current code, which adds committed part sizes to /keys/open and /keys/open/summary. The proposal replaces that: committed parts are counted only in /keys/open/mpu/summary, and the size-0 multipart open key is excluded from /keys/open and /keys/open/summary.

For one MPU with 10 MiB of committed parts and no part being written, the proposal would give:

Endpoint Before After
/keys/open/summary 1 open key, 0 bytes 0 open keys, 0 bytes
/keys/open 1 row, 0 bytes no row
/keys/open/mpu/summary 1 upload, 10 MiB 1 upload, 10 MiB (unchanged)

If this works for you, I'll rework the PR to this approach and update the description.

@errose28

errose28 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Looks good to me. I think we can bring the code and descriptions in sync with this proposal.

sumitagrawl
sumitagrawl previously approved these changes Nov 5, 2026
@adoroszlai adoroszlai assigned adoroszlai and unassigned adoroszlai Nov 5, 2026
@adoroszlai
adoroszlai marked this pull request as draft November 5, 2026 19:03
@smengcl
smengcl marked this pull request as ready for review November 5, 2026 20:18
Copilot AI lite review requested due to automatic review settings November 5, 2026 20:19
@smengcl
smengcl marked this pull request as draft November 5, 2026 20:19
Conflicts:
	hadoop-hdds/docs/content/interface/ReconApi.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues
What changed in this PR

Align Recon open key reporting by including committed multipart upload part sizes in /keys/open rows and /keys/open/summary, and fix MPU size stats correctness across Recon restarts.

Changes:

  • Add committed MPU part sizes to /keys/open row sizes and page totals by resolving upload IDs and summing committed parts.
  • Update /keys/open/summary to include MPU committed part sizes without changing open-key counts.
  • Fix split-schema MPU stats after Recon restart by caching replication config on multipartInfoTable DELETE events; add targeted unit tests.
File Description
hadoop-ozone/​recon/​src/​test/​java/​org/​apache/​hadoop/​ozone/​recon/​tasks/​TestOmTableInsightTask.java Adds unit test covering split-schema MPU delete processing after Recon restart (empty cache).
hadoop-ozone/​recon/​src/​test/​java/​org/​apache/​hadoop/​ozone/​recon/​api/​TestOmDBInsightEndPoint.java Adds unit tests validating open key summary + MPU sizes and multipart open key row sizing behavior.
hadoop-ozone/​recon/​src/​main/​java/​org/​apache/​hadoop/​ozone/​recon/​tasks/​MultipartInfoInsightHandler.java Adds committed-parts size helper and caches replication config on delete to keep MPU stats correct after restart.
hadoop-ozone/​recon/​src/​main/​java/​org/​apache/​hadoop/​ozone/​recon/​api/​ReconGlobalMetricsService.java Introduces getOpenKeySummaryWithMPU() to add MPU bytes to open key summary sizes.
hadoop-ozone/​recon/​src/​main/​java/​org/​apache/​hadoop/​ozone/​recon/​api/​OMDBInsightEndpoint.java Uses the new summary method and computes multipart open key row size by summing committed parts.

Review effort: Lite

Comment on lines +112 to +113
keysSummary.merge("totalReplicatedDataSize", mpuSummary.get("totalReplicatedDataSize"), Long::sum);
keysSummary.merge("totalUnreplicatedDataSize", mpuSummary.get("totalDataSize"), Long::sum);
Comment on lines 1175 to 1182
Pair<Long, Long> uploadSize = getMultipartUploadSize(dbKey, keyInfo);
keyEntityInfo.setSize(uploadSize.getLeft());
keyEntityInfo.setReplicatedSize(uploadSize.getRight());
} else {
keyEntityInfo.setSize(keyInfo.getDataSize());
keyEntityInfo.setReplicatedSize(keyInfo.getReplicatedSize());
}
keyEntityInfo.setCreationTime(keyInfo.getCreationTime());
Comment on lines +354 to +357
SortedMap<Integer, OmMultipartPartInfo> parts =
OMMultipartUploadUtils.scanParts(omMetadataManager, multipartKeyInfo.getUploadID());
long dataSize = parts.values().stream().mapToLong(OmMultipartPartInfo::getDataSize).sum();
return Pair.of(dataSize, OMMultipartUploadUtils.getReplicatedSize(parts, multipartKeyInfo.getReplicationConfig()));
@adoroszlai
adoroszlai dismissed sumitagrawl’s stale review November 6, 2026 07:53

author said: "I'll rework the PR to this approach", it will need to be re-reviewed afterwards

@lpavanvenkat lpavanvenkat changed the title HDDS-16364. Include committed part sizes of incomplete multipart uploads in Recon open key reports HDDS-16364. Exclude multipart open keys from Recon open key reports Nov 11, 2026
@lpavanvenkat
lpavanvenkat marked this pull request as ready for review November 11, 2026 06:59
@lpavanvenkat

Copy link
Copy Markdown
Contributor Author

@errose28 @sumitagrawl @devmadhuu @priyeshkaratha @smengcl, I've reworked the PR to the approach agreed above and updated the description. Please take another look.

Summary of the change:

  • Multipart open keys (isMultipartKey=true, size 0) are excluded from /keys/open and /keys/open/summary. Incomplete MPUs are reported only by /keys/open/mpu/summary.
  • Open keys of parts still being written are not multipart open keys, so they are still listed and counted as open keys until they commit.
  • A Recon upgrade action (version 7) queues a task reinitialization so the open key totals stored by older versions are recomputed.
  • The restart fix in MultipartInfoInsightHandler (caching the replication config on the multipartInfoTable DELETE event) is kept.
  • The earlier approach (adding committed part sizes to the open key endpoints) has been removed.

On the earlier review comments:

  • @devmadhuu: the lookup of older FSO multipart open keys by key name no longer applies, since the PR no longer resolves the upload from the key path. Keys are filtered by the isMultipartKey flag only. The remaining limitation, for keys created before HDDS-9017 without the flag, is noted in the description.
  • @priyeshkaratha: /keys/open and /keys/open/summary now exclude the same keys with the same check, so their totals no longer diverge because of MPUs.
  • The Copilot comments refer to getOpenKeySummaryWithMPU and getCommittedPartsSize, which have been removed.

I tested an upgrade from master on a compose cluster. The details are in the "How was this patch tested?" section. Thanks!

@smengcl smengcl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @lpavanvenkat . It almost looks good to me

private static final Logger LOG =
LoggerFactory.getLogger(OMDBInsightEndpoint.class);
/** Excludes multipart open keys, which are reported by {@code /keys/open/mpu/summary} instead. */
private static final Predicate<OmKeyInfo> NOT_MULTIPART_OPEN_KEY =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A question, not blocking: with this filter, Recon no longer lists individual incomplete MPUs anywhere. Before this change, /keys/open at least showed each upload's key name and creation time (with 0 bytes), while /keys/open/mpu/summary only gives totals. If an operator sees a large MPU total, how would they find which uploads are holding the space? If a per-upload listing is planned as a follow-up, could we file a JIRA for it and link it here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @smengcl. Yes, this is intentional: with this PR, incomplete MPUs are reported only by /keys/open/mpu/summary, as described above. A per-upload view is planned in HDDS-17802: a toggle on the Overview open keys card to switch between open keys and multipart uploads, and an "Open Multipart Uploads" tab on OM DB Insights that lists each upload with its key, upload ID, parts, committed size and age, backed by a new list endpoint. Until then, uploads can be listed per bucket through the S3 ListMultipartUploads API.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation recon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants