Skip to content

HDDS-16805. Verify peer chunk checksums during container reconciliation #11903

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andyhuangdev wants to merge 3 commits into
base: master
Choose a base branch
from
Open

HDDS-16805. Verify peer chunk checksums during container reconciliation #11903

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e384c71
HDDS-16805. Verify peer chunk checksums during container reconciliation
andyhuangdev Nov 2, 2026
b254ee5
Merge commit 'c359bf72ea6aa98147fd1935952c805242f1d18c' into HDDS-16805
adoroszlai Nov 13, 2026
c7b14dc
Remove empty line
adoroszlai Nov 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion ...ner-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1977,9 +1977,12 @@ BlockRepairResult reconcileChunksPerBlock(KeyValueContainer container, Pipeline
.build();
// Under construction is set here, during BlockInputStream#initialize() it is used to update the block length.
blkInfo.setUnderConstruction(true);
// Always verify peer data before using it to repair this replica, regardless of the client setting.
OzoneClientConfig clientConfig = conf.getObject(OzoneClientConfig.class);
clientConfig.setChecksumVerify(true);
try (BlockInputStream blockInputStream = blockInputStreamFactory.createBlockInputStream(
blkInfo, pipeline, blockToken, dnClient.getXceiverClientManager(),
null, conf.getObject(OzoneClientConfig.class))) {
null, clientConfig)) {
// Initialize the BlockInputStream. Gets the blockData from the peer, sets the block length and
// initializes ChunkInputStream for each chunk.
blockInputStream.initialize();
Expand Down
33 changes: 33 additions & 0 deletions .../apache/hadoop/ozone/container/keyvalue/TestContainerReconciliationWithMockDatanodes.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,6 +311,39 @@ public void testContainerReconciliationWithPeerFailure(FailureLocation failureLo
mockContainerProtocolCalls();
}

@Test
public void testContainerReconciliationVerifiesPeerDataWhenClientChecksumVerifyDisabled() throws Exception {
long containerID = CONTAINER_ID + 2;
for (MockDatanode dn : datanodes) {
dn.addContainerWithBlocks(containerID, 5, false);
dn.scanContainer(containerID);
}
long healthyChecksum = assertUniqueChecksumCount(containerID, datanodes, 1);
MockDatanode laggingNode = datanodes.get(0);
MockDatanode corruptPeer = datanodes.get(1);
MockDatanode healthyPeer = datanodes.get(2);

laggingNode.introduceCorruption(containerID, 1, 0, false);
laggingNode.scanContainer(containerID);
// Corrupt the peer after its last scan so its Merkle tree still advertises healthy data.
corruptPeer.introduceCorruption(containerID, 0, 1, false);

String verifyChecksumKey = "ozone.client.verify.checksum";
laggingNode.conf.setBoolean(verifyChecksumKey, false);
Comment on lines +331 to +332

@errose28 errose28 Nov 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
String verifyChecksumKey = "ozone.client.verify.checksum";
laggingNode.conf.setBoolean(verifyChecksumKey, false);
// The datanode should override this configuration value to true and always verify checksums during reconciliation.
String verifyChecksumKey = "ozone.client.verify.checksum";
laggingNode.conf.setBoolean(verifyChecksumKey, false);

try {
laggingNode.reconcileContainerSuccess(dnClient,
Arrays.asList(corruptPeer.getDnDetails(), healthyPeer.getDnDetails()), containerID);
} finally {
laggingNode.conf.unset(verifyChecksumKey);
}

// Reject the corrupt peer's bytes and repair from the healthy peer in the same reconciliation task.
laggingNode.scanContainer(containerID);
assertEquals(healthyChecksum, laggingNode.checkAndGetDataChecksum(containerID));
assertEquals(ContainerProtos.ContainerDataProto.State.CLOSED,
laggingNode.getContainer(containerID).getContainerData().getState());
}

@Test
public void testContainerReconciliationFailureContainerScan()
throws Exception {
Expand Down