Commit Graph
4 Commits
Author SHA1 Message Date
Nguyễn Lộc Phúcandchrislusf b62bbb99fc fix(iceberg): serve a registered table's metadata on load and commit (#12159)
* iceberg: serve a registered table's metadata on load and commit

RegisterTable saved only the metadata location, and loads and commits
never read the file there. They served a placeholder with no schema, no
snapshots and a new random table UUID on every load, so a registered
table read as empty, an INSERT failed against the empty schema, and
every other commit failed its assert-table-uuid requirement. Metadata
uploaded through the S3 gateway, as a migrated table's is, is stored in
chunks that registration did not read, so it registered an empty table.

Read the file following its chunks, reject one that is not Iceberg
metadata with a 400, and keep its content and table UUID in the record
the way CreateTable does.

Records registered before this change hold only the location. Loads and
commits now read the file there, and the first commit stores it in the
record under the file's table UUID.

* iceberg: transactions read a location-only registration's metadata file

* iceberg: cover transactions on a location-only registration

* iceberg: refresh the default name mapping on transaction commits

* iceberg: cover name-mapping refresh in transactions

* iceberg: roll a transaction back to a registration's metadata file

A transaction that failed after flipping a table registered before the
record kept its metadata could not undo the flip. The rollback restored
the record's prior metadata, which for such a table is only a location,
and UpdateTable keeps the full metadata a flip stored when the restore
sends none. Loads went on serving the change the client was told had
failed, and a client retrying the transaction could apply it twice.

Restore the metadata loads served from the file instead, under the
file's table UUID.

* iceberg: register oversized metadata by its location

Registration sends the metadata file's content to the S3 Tables
manager, which refuses a request over its 10MB body limit, so a table
whose metadata is larger could no longer be registered at all.

Register such a file by its location and table UUID alone; loads read
it from that location, as they do for a record that never kept its
metadata. Export the limit as s3tables.MaxRequestBodySize so the
catalog can check against it. A commit to such a table still sends the
whole metadata to UpdateTable and stays over the limit.

* iceberg: commit oversized metadata by its location

* iceberg: let maintenance follow a location-only table record

* admin: read a table's metadata file when the record keeps none

* iceberg: bound the metadata file a location-only record reads

* iceberg: bound only chunked metadata reads in the worker

Inline content is what the catalog itself wrote and the REST load path
accepts it at any size, so rejecting it in loadMetadata stranded tables
over 64 MiB from every maintenance scan. Cap only the chunked read, where
the recorded location can name any object in the bucket.

---------

Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
2026-10-28 08:15:42 -07:00
Chris Lu 8d2b417c69 iceberg, lance: vend read-only credentials to callers that may only read (#11859)
* s3tables: share the UpdateTable authorization gate

Extract the validation, load, and permission check from handleUpdateTable
into authorizeUpdateTable so the table an update was authorized against is
the same snapshot the mutation writes to. Manager gains
AuthorizeUpdateTable, mirroring AuthorizeCreateTable, so callers can ask the
question without updating anything.

* s3tables: share the UpdateView authorization gate

Extract the lookup, policy load, and permission check from handleUpdateView
into authorizeUpdateView, mirroring authorizeUpdateTable, and expose it as
Manager.AuthorizeUpdateView for callers that need the answer without the
update.

* s3api: scope vended table credentials to read-only on request

VendTableCredentials takes a readOnly flag and tablePrefixSessionPolicy
emits only s3:GetObject for the table prefix when it is set, keeping the
conditioned ListBucket and unconditioned GetBucketLocation statements that
reads still need.

* iceberg: vend read-only credentials when the caller may not write

LoadTable and friends vend storage credentials only for the authority the
caller holds: the vend now checks UpdateTable/UpdateView permission and a
caller that may only read the catalog entry gets a read-only session. Paths
that just passed a write gate (stage-create, view commit) keep the full
session.

* lance: vend read-only credentials when the caller may not write

storage_options vends the same read-only session as the Iceberg catalog
when the caller lacks UpdateTable on the table.

* lance: keep write credentials for callers authorized to declare

A Lance commit is a direct S3 write that never revisits the catalog, so a
caller that just passed CreateTable still gets a writable session; the
UpdateTable gate applies to describe, the read path.

* iceberg, lance: log a failed write check instead of only reading it as denied
2026-10-20 07:16:27 -07:00
Chris Lu 7522e17b6d iceberg: vend table-scoped credentials to clients that ask for delegation (#10777)
* iceberg: vend table-scoped credentials to clients that ask for delegation

The catalog recognised X-Iceberg-Access-Delegation: vended-credentials
and then deliberately said nothing, because it had nothing to vend: it
withheld even the S3 endpoint so the client would keep the credentials it
was configured with. That left every engine expecting the catalog to hand
out access - Snowflake, Databricks, Trino with vending, any multi-tenant
setup - needing static S3 keys distributed out of band.

Mint an STS session per request instead, scoped by a session policy to
the table's own prefix plus the bucket listing needed to resolve it, and
return it in the load response config and storage-credentials. The role
to assume is named by -s3.iceberg.credentialRole; its trust policy is
what decides whether a caller may assume it, and vending stays off until
it is set. A failed mint falls back to the old silence rather than
handing back an endpoint the client cannot sign for.

* iceberg: keep vended credentials inside the table prefix

Review follow-ups on credential vending:

Listing was granted on the bucket ARN with no condition, so a credential
vended for one table could enumerate every other table's object names.
Constrain s3:prefix to the table's own prefix, which the S3 gateway
already populates for list requests.

A table location carrying * or ? would have gone into the policy's
resource pattern unescaped and widened the session to sibling prefixes.
Refuse to vend for such a location rather than escaping it; nothing the
catalog generates contains those characters.

DurationSeconds skipped the 900..43200 bounds the other assume-role paths
enforce, so -s3.iceberg.credentialDurationSeconds could ask for a session
outside them. The check is now shared by all three entry points.

* iceberg: return the vended credentials from buildFileIOConfig itself

buildStorageConfig was a second name for what buildFileIOConfig already
did; it now returns the storage credentials alongside the properties, and
callers that only want the properties drop them.

* iceberg: split the vended bucket grants, and refuse a whole-bucket scope

The prefix condition sat on a statement that also granted
GetBucketLocation and ListBucketMultipartUploads, neither of which carries
an s3:prefix to satisfy it, so both were denied for every vended
credential. GetBucketLocation moves to its own unconditioned statement.
ListBucketMultipartUploads is dropped: Iceberg writers complete and abort
by upload id, and granting it either leaks in-flight keys bucket-wide or
breaks on the same missing prefix.

A table whose location has no prefix - one registered at the bucket root -
would have been vended read and write over every other table in the
bucket. Refuse, the way a location with wildcards is refused.
2026-08-16 12:57:12 -07:00
Chris Lu d448e9db7b iceberg: withhold the S3 endpoint from credential-vending clients (#10570)
* iceberg: withhold the S3 endpoint from credential-vending clients

A client that sends X-Iceberg-Access-Delegation: vended-credentials builds
its storage credential out of the LoadTable config and drops the one it was
configured with. We vend no credentials, so the endpoint we advertised left
DuckDB signing nothing: every metadata and data file came back 403, and its
attempt to refresh the empty credential 404ed on stage-created tables.

Answer those clients with no config at all so they keep their own
credentials. Clients that do not ask for delegation still get the endpoint.

* iceberg: mark load responses as varying on the delegation header

The FileIO config in a table or view load response now depends on whether
the client asked for vended credentials, so a cache between us and the
client must key on that header rather than on the URL alone.

* test: cover the DuckDB vended-credentials access pattern

Runs weed mini with -s3.externalUrl, which is what makes the catalog
advertise an endpoint at all, and checks both halves: a plain LoadTable
still gets the endpoint, while one asking for vended credentials never gets
an endpoint without the credentials to sign with. The DuckDB round trip
creates a table from a query and reads it back, which is the flow that
failed with 403 on every data file.
2026-08-04 18:14:34 -07:00