4 Commits

Include renames

Author	SHA1	Message	Date
Nguyễn Lộc Phúc	195f8156de	fix(iceberg): keep a losing table create from deleting the winner's metadata (#12474) ... * iceberg: keep a losing table create from deleting the winner's metadata * iceberg: keep a staged file whose published record was renamed away A create answers a conflict when its record was already visible and another request changed it before the create could back it out. The cleanup then looked only at the original name, so a record renamed in between read as absent and the file it still points at was deleted. S3 Tables now reports that case with a conflict of its own, and both create paths keep the staged file on it instead of probing the name. A conflict over a still-hidden record, lost to another create, still removes the file. * iceberg: discard a losing create's metadata only while it is still its own The cleanup deleted the staged file by name. The path can be reused before it runs: the table is dropped and purged, and another create stages the same v1.metadata.json and registers it, which the losing request then deleted. Staged files now carry a per-request token in an internal extended attribute, and the cleanup deletes with that token as the filer's ExpectedExtended precondition, checked under the path lock the delete runs under. A file that took the path since is left in place. * iceberg: classify a typed S3 Tables error by its code, not its message isS3TablesAlreadyExists matched "already exists" in the message before looking at the code, so an InternalError whose text mentioned an existing entry read as the name being taken: CreateTable answered 409 instead of 500, and registrationRefused discarded a staged file the catalog may still point at. A typed error now decides by its code; only an untyped one is matched by message. * iceberg: pin stage-create on in the stage-create conflict test Stage-create is on by default, but a shell that sets ICEBERG_ENABLE_STAGE_CREATE=false turned the expected 409 into a 501. Setting it in the test keeps the result independent of the environment. * iceberg: test two creates overlapping at the canonical metadata name The concurrent-creators test only lines the creators up before they start, so whether two of them ever collide at v1.metadata.json is up to the scheduler. The new test fixes the order: the second create runs while the first has staged v1.metadata.json and not yet registered it. It checks that every staging attempt is exclusive, that the second falls back to v1-<uuid>, that both files hold their own metadata while both are staged, and that the loser removes only its own file. * s3tables: a create whose unmark write never went out stayed hidden confirmChild counted any failure to take a create's mark off as possibly published, the lookup that precedes the write included. That lookup failing leaves the mark on, so the record stayed hidden; if another create then adopted it, the back-out reported a published record that changed, and the Iceberg catalog kept a staged file no record names. The mark-removal calls now go through a wrapper that notes whether an UpdateEntry went out without the filer refusing it. Only then may the record have been seen. A refused compare, a missing entry, or a failure before any write leaves the create's record hidden. * s3tables: pass mayBeVisible into backOutChild Fixes the build after rebasing onto the backOutChild split.	2026-11-07 10:54:04 -08:00
Chris Lu	d044839ab2	iceberg: make a table commit a compare-and-swap (#10775) ... * iceberg: make a table commit a compare-and-swap The catalog validated the caller's version token, ran its authorization checks, and only then wrote the new metadata xattr. Two engines committing against the same base both passed that check and both wrote, so the second silently dropped the first one's snapshot. Both also derive the same v{N}.metadata.json name and the file write overwrote, leaving the surviving pointer aimed at the loser's metadata - and the loser's conflict cleanup then deleted the winner's file. Write the metadata file with an exclusive create and update the xattr conditionally on the bytes the handler read, the way the maintenance worker already commits. A writer that lost the race re-reads and retries, and reports 409 CommitFailedException once out of attempts. * iceberg: stage a commit under a unique name when the versioned one is taken Two follow-ups from review of the commit compare-and-swap: Refusing to overwrite v{N}.metadata.json also refused to get past a file left behind by a commit that died between staging and updating the pointer. Every later commit derived the same name, saw the collision, and reported a conflict, so the table stayed uncommittable until an orphan sweep removed the file. Stage under v{N}-{uuid} instead: neither writer's file is overwritten and the catalog pointer still decides who won, which is how the maintenance worker has always staged its own metadata. metadataVersionFromLocation learned to read the version back out of that name. The conditional update guarded only the metadata attribute while the write replaced the whole entry, so a policy or tag written in the same window was silently reverted. Guard every catalog attribute, which turns that into a conflict the caller retries on fresh state. * iceberg: give saveMetadataFile the exclusive flag instead of a second name saveNewMetadataFile, saveMetadataBlobExclusive and uniqueMetadataFileName were three new names around one existing helper. The flag now rides on saveMetadataFile and saveMetadataBlob, and the unique-name construction sits where it is used. * iceberg: reuse the filer CAS helpers #10773 added, and stage transactions exclusively #10773 landed mutateEntryExtended, which already writes an entry back under a whole-entry precondition and retries. Drop the helper this branch added and route the table commit through it: the check that the metadata is still the one this request read now lives in the mutation, where it sees current state. The policy the request was authorized against is asserted too, so an administrator restricting it mid-commit sends the caller back through authorization instead of having a stale decision applied. Bucket and namespace policies live on other entries and a single-entry precondition cannot cover them. Multi-table transactions stage their metadata exclusively for the same reason single-table commits do, and carry the name they landed on into the pointer flip.	2026-08-16 12:55:42 -07:00
Chris Lu	b4b0346f95	iceberg maintenance: resolve table files from the recorded location (#10418) ... The worker assumed every file of a table sits under its catalog path, so loadFileByIcebergPath stripped the scheme off a recorded location and joined the remainder onto /buckets/<bucket>/<ns>/<table>. A table the REST catalog placed elsewhere in the bucket — which is what a client gets whenever the catalog path is already occupied — then resolves to a doubled path: lookup /buckets/lake/source/t/lake/source/t-0cd81bca-.../metadata/snap-.avro so the very first manifest list read fails and the job fails again on every scan interval, indefinitely. Resolve absolute references (s3:// URIs and /buckets paths) from the bucket root and keep relative ones under the table's own directory; the bucket-relative form is now the canonical key everywhere references are compared. That directory comes from the metadata location the catalog stores, so reads, writes and deletes all land where the table's other files are instead of splitting it across two trees. References outside the table's bucket are rejected rather than silently misresolved. Rewritten position-delete files now name their data file by absolute URI, the way the table itself names it, instead of a path relative to the table.	2026-07-24 02:40:14 -07:00
Chris Lu	c95401b11a	iceberg: support table rename (#10068) ... * s3tables: add RenameTable operation * iceberg: support table rename * iceberg: test table rename * s3tables: keep table data in place on rename rename is catalog-only: drop the source's catalog xattrs in place instead of recursively deleting its directory, which wiped the metadata.json and data files the renamed destination still points at. treat a missing table-metadata xattr as NoSuchTable in GetTable so the soft-deleted source name stops resolving. * s3tables: test rename preserves data make the in-memory filer honor recursive data deletion and seed the source table's metadata/ and data/ children, then assert a rename leaves them intact, the source name resolves to NoSuchTable, and the destination resolves to the preserved location. * iceberg: map rename errors through wrapped manager error * s3tables: authorize rename destination namespace rename moved a table into the destination namespace after only checking the source, letting a source-authorized caller place tables in namespaces they don't control. require CreateTable on the destination namespace and bucket before writing. * s3tables: purge renamed table data on drop * s3tables: test table data dir derivation	2026-06-23 20:18:11 -07:00