2 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandClaude Opus 5.5	4f24b756ba	s3tables, iceberg: give tables created without a location their own directory (#12475) ... * s3tables: give a table created without a location its own directory An Iceberg table created without a location kept its files under its name path, so a rename left them where the next table of that name would land. Such a table now gets s3://<bucket>/<namespace>/<name>-<uuid>, stored as the record's warehouseLocation and returned by GetTable; its first commit writes there without moving the root. Records without one keep resolving to their name path, so existing tables need no rewrite. Lance datasets stay at their name path, where Lance clients and the vector engine look for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: create tables and views in a directory of their own A table or view created without a location took its name path, or a unique one only when that path was already taken. It now always gets <name>-<uuid>, so a later rename never leaves its files under a name another table can take. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: load and commit a table at its warehouse location A table created through S3 Tables without a location has no metadata until its first commit; loading or committing it now uses the directory recorded for it, and only a record without one falls back to its name path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: fail a stage-create whose marker is not saved The marker is how the commit finds the location the stage-create handed out. Without it the commit fell back to the name path while the client's files sat under the returned location. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: expect tables created without a location in their own directory Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: rename a table that keeps its files elsewhere before reusing its name Enterprise keeps the storage marker on a renamed table's files, so the name of a table whose files sit under its name path stays taken there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: remove the table directory when a create backs out Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: answer a create that lost the race with the registered table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: answer a view create that lost the race with the registered view Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: fail a staged commit whose name another create registered Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: remove the directory of a create whose name another table took Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: keep the metadata of a staged commit whose name is taken Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: write the lakekeeper table's data under its warehouse location Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: commit a staged table at the location of its own stage Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: never continue another stage when a commit's own marker is gone Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: test filer honors exclusive creates and token-guarded deletes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 16:37:31 -08:00
4 people	d6ec6bc4e7	fix(s3tables): let DeleteTableBucket remove empty leftover directories (#12307) ... * s3tables: let DeleteTableBucket remove empty leftover directories Page through the bucket instead of reading one page, and check each bare directory a deleted namespace left behind for files: empty ones go with the bucket, one holding files refuses with 409 naming it. The delete claims the bucket before scanning, and CreateNamespace refuses a claimed bucket and confirms it again after its write, so a namespace created meanwhile is seen by the scan or turned away instead of deleted with the bucket. The bucket is removed only while it still carries that claim, otherwise 409. A delete that refuses lifts the claim, retrying a failed release, since a claim left behind refuses every create in the bucket. The new namespace stays marked unfinished until that check passes, so no table or view is created in one that gets backed out; a create takes over a namespace still carrying the mark. Both checks accept only the bucket the create was authorized against, by its creation time: one recreated under the same name answers 404 and gets nothing written into it. The removal is not recursive. Each leftover is claimed first, only while it is still the directory the scan saw, and goes only while it holds no file or catalog record; the bucket goes only while nothing is left in it. A file uploaded or a namespace taken back after the scan is refused with 409 instead of deleted. * s3tables: keep directory-backed S3 objects in DeleteTableBucket * s3tables: keep a file that replaced a leftover directory in DeleteTableBucket * s3tables: claim only the authorized bucket in DeleteTableBucket * s3tables: assert the prefix-object mark when claiming a leftover directory * s3tables: hide unfinished namespaces from readers and DeleteNamespace * s3tables: stop CreateNamespace on any failed bucket check * s3tables: retry lifting claims on leftover directories * s3vectors: declare the index namespace again after losing it to a concurrent declare CreateNamespace now writes a namespace marked unfinished and confirms it after checking the bucket. A second CreateIndex that declares the namespace in that window takes the marked record over, and the first one's confirmation answers 409, which failed its CreateIndex. Declare it again, a bounded number of times, when the create answers a conflict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: name an unfinished namespace that keeps a bucket from being deleted A CreateNamespace that dies between its write and its confirmation leaves a namespace marked unfinished. Listings and GetNamespace hide it and DeleteNamespace answers that it does not exist, while DeleteTableBucket counts it, so the bucket could not be deleted and nothing said why. Name it in the BucketNotEmpty answer; creating it again takes it over. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: read a refused empty-directory delete with filer.DeleteEntryError Use the filer's own reading of a non-recursive delete's answer instead of a copy of its message prefix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: release the held create when the concurrent-declare test fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: buffer the held create's result so it can exit after a failure Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: retry lifting a refused namespace delete's claim A claim left on a namespace refuses every create in it, the same harm the bucket's claim does, so its release retries too. With no one-shot caller left, the retrying release takes the releaseDeleteClaim name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: keep the data under a hidden directory on bucket delete The scan filed every hidden entry by name for a recursive delete, so .hidden/data went with the bucket. A hidden directory waits with the leftovers instead: the walk refuses one that holds a file or an object, and removeEmptyTree takes only one that is still bare and empty. A hidden file is removed by name as before. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-11-01 13:43:15 -08:00