Repository navigation
fix(table): update the table's name mapping on schema changes - #2646
ShivamSharma43 wants to merge 2 commits into
Conversation
UpdateSchema only updated schema.name-mapping.default when the caller passed a mapping with WithNameMapping, which nothing in the repo does. After AddColumn or RenameColumn the property still described the old schema, so files without field IDs could not be resolved for the new or renamed column. AddDataFilesByPath sets the mapping on first use, and a later file carrying an added column failed with "missing field_id". When no mapping is passed, BuildUpdates now updates the table's own mapping if it has one, as Java's SchemaUpdate and PyIceberg's UpdateSchema do. Tables without the property are left unchanged, and an explicit WithNameMapping still takes precedence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Shivam Sharma <192242476+ShivamSharma43@users.noreply.github.com>
badalprasadsingh
left a comment
There was a problem hiding this comment.
few comments @ShivamSharma43
| if err != nil { | ||
| return nil, nil, err | ||
| } | ||
| mappingJSON, err := json.Marshal(updatedNameMapping) |
There was a problem hiding this comment.
when the mapping is [], ore all entries have empty names (something which is allowed by the spec), a rename or delete gets nil from "UpdateNameMapping" -> so this writes null
go scans and add-files then fail with invalid
'schema.name-mapping.default'
better:
if updatedNameMapping == nil {
// UpdateNameMapping returns nil when no field keeps a name.
updatedNameMapping = iceberg.NameMapping{}
}also, add a test for it
There was a problem hiding this comment.
Fixed in 44ff0fa: BuildUpdates now writes [] when UpdateNameMapping returns nil. Added TestUpdateSchemaKeepsEmptyNameMappingValid, which covers rename and delete on [] and on entries with empty names, and checks the property stays valid. It fails without the fix with invalid schema.name-mapping.default.
| // field IDs cannot be resolved for added or renamed columns. | ||
| nameMapping := u.nameMapping | ||
| if nameMapping == nil { | ||
| nameMapping = meta.NameMapping() |
There was a problem hiding this comment.
this changes what users see: schema updates now rewrite schema.name-mapping.default, and when a new col reuses a name freed by "DeleteColumn" or "RenameColumn" -> older files without field ids show the old values in it
better: add an unreleased compatibility note in releases.md
There was a problem hiding this comment.
Added an unreleased compatibility note in website/src/releases.md (44ff0fa) covering the mapping rewrite and the reused-name case for older files without field IDs.
There was a problem hiding this comment.
also, there is no requirement of creating a new file, we can add the tests in "update_schema_test.go"
There was a problem hiding this comment.
Moved the tests into update_schema_test.go and removed update_schema_name_mapping_test.go in 44ff0fa. They reuse the existing countingCatalog.
UpdateNameMapping returns nil when no mapping entry keeps a name, as for [] or entries with empty names, which the spec allows. A rename or delete then wrote null to schema.name-mapping.default, and scans and add-files rejected the property as invalid. Write [] instead. Move the name mapping tests into update_schema_test.go, and add a compatibility note: schema updates now rewrite the table's name mapping, so a new column that reuses a name freed by DeleteColumn or RenameColumn maps to that name in older files without field IDs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Shivam Sharma <192242476+ShivamSharma43@users.noreply.github.com>
UpdateSchemanever updated the table'sschema.name-mapping.defaultproperty unless the caller passed a mapping withWithNameMapping, and nothing in the repo does. AfterAddColumnorRenameColumnthe mapping still described the old schema, so files without Parquet field IDs could not be resolved for the new or renamed column. SinceAddDataFilesByPathandAddDataFilesset the mapping on first use, any table that iceberg-go has added files to hits this: after adding a column, adding a file that contains it fails withinvalid schema: cannot convert category: type=utf8, nullable metadata: ["PARQUET:field_id": "-1"] to Iceberg field, missing field_id(see the issue for the reproduction).Cause
BuildUpdatesonly raniceberg.UpdateNameMappingwhenu.nameMappingwas set, and that field is only set by theWithNameMappingoption. It never read the mapping stored in the table's properties.Fix
When no mapping is passed,
BuildUpdatesnow uses the table's own mapping (meta.NameMapping(), read from the transaction's staged metadata) and writes the updated mapping back toschema.name-mapping.default. That matches Java'sSchemaUpdate.applyChangesToMetadata, which updates the property on every schema change, and PyIceberg, whoseTransaction.update_schemapassestable_metadata.name_mapping()toUpdateSchema._commit.UpdateNameMappingalready does (same as Java'sMappingUtil.update).WithNameMappingstill takes precedence.meta.NameMapping()returns nil and the update goes ahead without touching it. Java does the same: it logs a warning and keeps the old value.Testing
New
table/update_schema_name_mapping_test.go:TestUpdateSchemaUpdatesTableNameMappingadds a column and renames another on a table with a mapping, then checks the committed mapping (categoryfor the new field,dataandpayloadfor the renamed one).TestUpdateSchemaWithoutNameMappingLeavesPropertyUnsetchecks that a table without the property doesn't get one.TestAddFilesAfterAddColumnReadsNewColumnByNameruns the reproduction from the issue end to end:AddDataFilesByPath,AddColumn,AddDataFilesByPathwith a file that has the new column and no field IDs, then a scan that readsbooksfrom it.On main, the first and third fail (the mapping has no entry for field 3, and the second
AddDataFilesByPathfails with themissing field_iderror above). The second passes on main and with the fix.go vet ./table/is clean.go test ./table/...passes except for tests that fail the same way on unmodified main on my Windows machine (path handling):TestEmptyStringIdentityPartitionScans,TestScanImportedSignedZeroBounds,TestDeleteImportedSignedZerosWithIn,TestDeleteNormallyWrittenSignedZeroWithIn,TestScanLegacyEmptyStringPartitionDeleteBounds,TestScanWithIDLessNameMappingFields, twoTestRewriteDataFiles_PartialProgress*tests andTestWriteMapType.go test ./catalog/...gives the same results as main.Closes #2645
This PR was prepared with AI assistance (Claude Code).
🤖 Generated with Claude Code