Skip to content

fix(table): keep v2 position deletes when a v3 delete writes a file's first DV - #2669

Open
AynSiddiqui wants to merge 1 commit into
apache:mainfrom
AynSiddiqui:fix/2604-v3-upgrade-pos-deletes
Open

AynSiddiqui wants to merge 1 commit into
apache:mainfrom
AynSiddiqui:fix/2604-v3-upgrade-pos-deletes

Conversation

@AynSiddiqui

Copy link
Copy Markdown
Contributor

Fixes #2604

What

On a table upgraded from v2 to v3, a merge-on-read Delete wrote a deletion vector (DV) with only the new deletes. Once a data file has a DV, readers skip its v2 position delete files, so rows deleted before the upgrade came back.

How

When a v3 Delete gives a data file its first DV, the positions from all of that file's v2 position delete files (matched like scan planning does) are merged into the DV. As in Java's BaseDVFileWriter, only file-scoped position delete files (referenced_data_file is that data file) are removed in the same snapshot. Partition-scoped files that can cover several data files are kept: readers ignore them for files that have a DV and still apply them to the others. Equality deletes are not touched. Removing a delete file makes the commit non-replayable, as for existing DV replacement, so a concurrent commit fails with ErrCommitFailed and the caller retries.

Notes:

  • Matching the v2 deletes reuses scan planning, so a table with inconsistent v2 deletes (for example a delete file whose spec or partition does not match its data file) now makes the Delete fail with ErrInvalidMetadata, as a scan of that table already does. Before, the Delete did not check this.
  • Each matched position delete file is read in full once per Delete, so memory grows with its number of positions; a large partition-wide v2 file costs more.

Tests

New tests in table/mor_v3_upgrade_deletes_test.go: the issue's scenario with a second delete after the upgrade, several position delete files on one data file, a shared position delete file, a partitioned table, a partition spec change before the upgrade, equality deletes kept, a delete that matches no row still writing the seeded DV, and a concurrent commit failure. They fail on main and pass with the fix. go test ./table/... and make lint pass.

… first DV

When a v3 merge-on-read Delete gives a data file its first deletion vector,
merge the positions from that file's v2 position delete files into the DV
and remove the file-scoped ones in the same snapshot, as Java's
BaseDVFileWriter does. Partition-scoped files are kept.

Fixes apache#2604

Signed-off-by: Ayaan <ayaansiddiqui68@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Transaction: merge-on-read Delete on a table upgraded to v3 brings back rows deleted before the upgrade

1 participant