Skip to content

feat(catalog): accept a scoped FileIO registry when loading catalogs - #2664

Open
ViktorBystrov72 wants to merge 1 commit into
apache:mainfrom
ViktorBystrov72:issue-iceberg-go-2634-catalog-io
Open

ViktorBystrov72 wants to merge 1 commit into
apache:mainfrom
ViktorBystrov72:issue-iceberg-go-2634-catalog-io

Conversation

@ViktorBystrov72

@ViktorBystrov72 ViktorBystrov72 commented Nov 20, 2026 •

Copy link
Copy Markdown

catalog.LoadWithOptions and (*catalog.Registry).LoadWithOptions accept catalog.WithIORegistry. Catalogs in the same process can use different FileIO handlers for the same URI scheme. The selected registry is used when opening the catalog and when reading its tables. The existing Load entry points retain their signatures.

The built-in catalogs accept the selected registry. Custom factories can opt in through IORegistryRegistrarFunc. Requesting a scoped registry for a factory that lacks this capability returns an error.

Tests

go test -short -mod=readonly passed in seven catalog packages. Coverage includes two registrations for the same URI scheme, reads through HTTP/REST, behavior for custom factories, and compatibility with the existing Load signatures.

This patch was prepared with assistance from OpenAI Codex.

Fixes #2634

@laskoviymishka

Copy link
Copy Markdown
Contributor

@ViktorBystrov72 - can you localize this PR?
Generally we are not accepting PR-s that in languages other than english.

Signed-off-by: Viktor Bystrov <18964918+ViktorBystrov72@users.noreply.github.com>
@ViktorBystrov72
ViktorBystrov72 force-pushed the issue-iceberg-go-2634-catalog-io branch from 2e86a6b to 9dde50e Compare November 20, 2026 08:10
@ViktorBystrov72 ViktorBystrov72 changed the title feat(catalog): выбирать отдельный реестр FileIO при загрузке каталога feat(catalog): accept a scoped FileIO registry when loading catalogs Nov 20, 2026

@badalprasadsingh badalprasadsingh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

welcome @ViktorBystrov72, first pass

Comment thread catalog/registry.go
Comment on lines +219 to +222
// LoadWithOptions resolves a catalog like Load and applies options to its factory.
func LoadWithOptions(ctx context.Context, name string, props iceberg.Properties, opts ...LoadOption) (Catalog, error) {
return defaultRegistry.LoadWithOptions(ctx, name, props, opts...)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#2612 chose a trailing variadic option over a separate ...WithOptions function (ig you can check its note in releases.md), can "Load" and "Registry.Load" take opts ...LoadOption instead of these two new functions? direct calls still compile.

it needs a ! title and a compatibility note like #2612's.

Comment thread catalog/sql/sql.go
// Here, name is the loader-supplied catalog name and is persisted as the
// catalog_name partition key on every row in the iceberg_tables and iceberg_namespace_properties.
cat, err := NewCatalog(ctx, name, sqldb, SupportedDialect(dialect), p)
cat, err := NewCatalog(ctx, name, sqldb, SupportedDialect(dialect), p, WithIORegistry(registry))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only the hadoop and rest registrars are tested on this path: the sql, glue, hive and dynamodb registrars could drop the registry and every test would still pass

can we add a LoadWithOptions case for each? for dynamodb: TestDynamoDBRegistry could create, commit and load a table

Comment thread catalog/registry.go

// WithIORegistry selects the FileIO factories for this catalog and its tables.
// A nil registry uses the process-wide default. Custom catalogs that accept a
// scoped registry can register an IORegistryRegistrarFunc.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in "configuration.md", we still say: a scoped registry "does not affect catalog.Load"

please document the new option there, noting that dynamodb gets it only this way. please also say, here and in its "Catalog type registry" section, that other factories fail with iceberg.ErrNotImplemented

Comment on lines +34 to +40
// Load remains usable as a function value and through a caller-defined interface.
var (
_ func(context.Context, string, iceberg.Properties) (catalog.Catalog, error) = catalog.Load
_ interface {
Load(context.Context, string, iceberg.Properties) (catalog.Catalog, error)
} = (*catalog.Registry)(nil)
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the api compatibility check already fails any pr that changes Load's signature without a ! title -> so these lines repeat it

better: drop them, if Load becomes variadic, they have to go anyway

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants