Repository navigation
feat(catalog): accept a scoped FileIO registry when loading catalogs - #2664
ViktorBystrov72 wants to merge 1 commit into
Conversation
|
@ViktorBystrov72 - can you localize this PR? |
Signed-off-by: Viktor Bystrov <18964918+ViktorBystrov72@users.noreply.github.com>
2e86a6b to
9dde50e
Compare
badalprasadsingh
left a comment
There was a problem hiding this comment.
welcome @ViktorBystrov72, first pass
| // LoadWithOptions resolves a catalog like Load and applies options to its factory. | ||
| func LoadWithOptions(ctx context.Context, name string, props iceberg.Properties, opts ...LoadOption) (Catalog, error) { | ||
| return defaultRegistry.LoadWithOptions(ctx, name, props, opts...) | ||
| } |
There was a problem hiding this comment.
| // Here, name is the loader-supplied catalog name and is persisted as the | ||
| // catalog_name partition key on every row in the iceberg_tables and iceberg_namespace_properties. | ||
| cat, err := NewCatalog(ctx, name, sqldb, SupportedDialect(dialect), p) | ||
| cat, err := NewCatalog(ctx, name, sqldb, SupportedDialect(dialect), p, WithIORegistry(registry)) |
There was a problem hiding this comment.
only the hadoop and rest registrars are tested on this path: the sql, glue, hive and dynamodb registrars could drop the registry and every test would still pass
can we add a LoadWithOptions case for each? for dynamodb: TestDynamoDBRegistry could create, commit and load a table
|
|
||
| // WithIORegistry selects the FileIO factories for this catalog and its tables. | ||
| // A nil registry uses the process-wide default. Custom catalogs that accept a | ||
| // scoped registry can register an IORegistryRegistrarFunc. |
There was a problem hiding this comment.
in "configuration.md", we still say: a scoped registry "does not affect catalog.Load"
please document the new option there, noting that dynamodb gets it only this way. please also say, here and in its "Catalog type registry" section, that other factories fail with iceberg.ErrNotImplemented
| // Load remains usable as a function value and through a caller-defined interface. | ||
| var ( | ||
| _ func(context.Context, string, iceberg.Properties) (catalog.Catalog, error) = catalog.Load | ||
| _ interface { | ||
| Load(context.Context, string, iceberg.Properties) (catalog.Catalog, error) | ||
| } = (*catalog.Registry)(nil) | ||
| ) |
There was a problem hiding this comment.
nit: the api compatibility check already fails any pr that changes Load's signature without a ! title -> so these lines repeat it
better: drop them, if Load becomes variadic, they have to go anyway
catalog.LoadWithOptionsand(*catalog.Registry).LoadWithOptionsacceptcatalog.WithIORegistry. Catalogs in the same process can use different FileIO handlers for the same URI scheme. The selected registry is used when opening the catalog and when reading its tables. The existingLoadentry points retain their signatures.The built-in catalogs accept the selected registry. Custom factories can opt in through
IORegistryRegistrarFunc. Requesting a scoped registry for a factory that lacks this capability returns an error.Tests
go test -short -mod=readonlypassed in seven catalog packages. Coverage includes two registrations for the same URI scheme, reads through HTTP/REST, behavior for custom factories, and compatibility with the existingLoadsignatures.This patch was prepared with assistance from OpenAI Codex.
Fixes #2634