mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-27 07:20:44 +01:00
Lance catalog, and a Rust plugin worker to maintain it (#10841)
* iceberg: skip tables the maintenance worker does not own A Lance dataset registered through the Lance namespace's Iceberg REST adapter arrives as an Iceberg table with a placeholder schema and table_type=lance, and keeps its fragments under data/ - the same subdirectory the orphan cleaner walks. Every fragment is unreferenced by the Iceberg metadata, so a maintenance pass deletes the dataset. Views share the entry shape and were only skipped because parsing their metadata happened to fail first. Gate the scan and the execution path on the entry actually being an Iceberg table. Maintenance is off by default, so this was latent rather than live. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: let a table declare a format the catalog does not interpret CreateTable accepted ICEBERG and nothing else. A Lance table has no metadata file for the catalog to maintain - the entry records a name and the dataset root, and the client owns everything under it - so accept LANCE, and carry the declared format on the entry instead of hardcoding it back on the way out. ListTables now reports format and metadataLocation, so listing a catalog that holds both kinds takes one pass rather than a GetTable per row. AWS omits both fields; adding them is additive. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: move the in-memory filer into its own package The Lance namespace tests need the same harness, and copying it would leave two of them to keep in step. Extracted as it was, plus the two fidelity gaps that only surface once a paginating caller uses it: ListEntries ignored startFromFileName and limit, so a caller that paginates re-read the first page until it hit its own cap and reported the same entry over and over, and GetFilerConfiguration was missing, which CreateTableBucket needs to resolve the buckets directory. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: serve the Lance Namespace REST spec A second catalog surface beside the Iceberg one, over the same table buckets: the namespace and table metadata operations, the $-delimited identifier codec, the spec's numeric error model, the directory-catalog marker files, and storage_options vending through the STS path the Iceberg catalog already uses. Listens on -port.lance, 9101 by default, and inherits ARNs, policies and tags from the storage layer, so a Lance table needs no second permission model. Identifiers map bucket / namespace / table onto the three levels Lance clients already use, which is why there is no warehouse selector to invent. The data plane needs Lance format support that does not exist in Go and answers with the spec's Unsupported code rather than a bare 404. Two things it deliberately will not do: create a table bucket as a side effect of creating a namespace inside one, since a bucket carries its own policy and lifecycle, and resolve an Iceberg table's location for a Lance client, which would hand it a table another engine owns. The design note this follows is in design-lance-catalog.md, including the .lance directory suffix it proposed and this does not implement. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * mini: give the Lance port the same treatment as the Iceberg one The flag was registered but nothing else knew about it, so mini would start the server without reserving its port, waiting for it, or saying where it is. Adds it to the startup service list, the conflict resolver, the gRPC allocator's reserved set, the readiness wait, the stop reporting and the banner. The admin server still takes only the Iceberg port, because there is no Lance page for it to link to. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: stop deregister and repoint from deleting the dataset Deregistering preserves data by definition, and this did the opposite: the catalog entry is the dataset directory, so DeleteTable took the files with it. Registering over an existing name had the same shape, destroying the dataset the name used to hold. Found by driving the running server rather than the in-memory filer, where both looked like success because the table did stop being listed. Deregistering is now a state on the entry - the marker file hides it, and declaring or registering the name again brings it back. Repointing a name at another dataset is an UpdateTable against the version token, so neither dataset loses files. Drop is left alone; it is the operation that does remove data. The storage endpoint now falls back to the advertised -ip where the Iceberg derivation gives up. An Iceberg client brings its own s3.endpoint and advertising the wrong one hijacks it, but storage_options is the only place a Lance client learns where the store is, and without it object_store quietly talks to real AWS. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: refuse to create a table over one of another format Creating a table that already exists is idempotent, and that path returned the existing table without looking at its format. A Lance declare over an Iceberg table answered 200 and handed back a directory Iceberg owns, so the client would write its dataset on top. The view check immediately above it already guards the same class of collision. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: let a table bucket hold a format other than Iceberg The S3 door validated every object written into a table bucket against Iceberg's file layout, so a Lance client could not write its dataset at all: it got 403 on data/*.lance, on _versions/, and on the _transactions/ directory it turned out to write as well. Table buckets were only neutral containers by intention; in practice they were Iceberg-shaped and enforced as such. The allowed set is now the union of what the supported formats write, because the validator runs where the table's format is not in hand. Underscore-prefixed directories are treated as belonging to the format, since enumerating them means guessing at the next one - _transactions is exactly the one this missed - and their contents are checked only for traversal. Iceberg writes none of them, so it loses nothing. Marker files at the table root are admitted too, which the namespace/table/dir/file shape had rejected as too shallow. Describe also honours the request-body spellings of with_table_uri, load_detailed_metadata and check_declared. The spec puts them in the query string, but real clients send both. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record what the implementation found The table bucket being an Iceberg-shaped container, enforced at the S3 door, was the premise this design never questioned and the one that had to change before anything worked end to end. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: prove the data loss the foreign-format guard prevents The guard landed with a unit test for the predicate and nothing showing what it saves. These seed what the Lance namespace's Iceberg REST adapter actually leaves behind - an Iceberg table with a placeholder schema and table_type=lance whose directory holds a Lance dataset - and assert both halves: orphan collection does flag the dataset's fragments, because the Iceberg metadata beside them references nothing, and the scan never reaches the table. An ordinary Iceberg table in the same shape is still scanned, so the guard is not just skipping everything. Confirmed against a running gateway first: our Iceberg catalog accepts the adapter's registration, and a real Lance client then writes a dataset into that table's location. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tablestest: make the in-memory filer safe to race against Two gaps that only matter once a test drives concurrent writers, which is what an exclusive create has to be tested with: the entry map had no lock, and CreateEntry ignored O_EXCL entirely, so both writers of the same name would have won and the test would have passed while proving nothing. The BeforeUpdate hook runs before the lock is taken. Its whole purpose is to land a competing write in a handler's read-to-write window, and that write needs the lock the hook would otherwise be holding. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: make the namespace an external manifest store Lance commits a version by writing _versions/{v}.manifest with put-if-not-exists. The S3 layer in front of this same filer evaluates If-None-Match by looking the entry up and then writing without a precondition, so two writers can both pass the check and one commit is lost. The filer itself has the primitive: CreateEntry with o_excl. Adds the four version operations a Lance client actually calls - create, list, describe and batch-delete - recording one entry per version under _lance_versions/, and advertises managed_versioning so the client routes its commits here. Reserving a version is the exclusive create, so exactly one of several racing writers wins and the rest rebase. Off by default, behind -lance.managedVersioning. Turning it on moves where a table's version history lives, and a reader that does not come through this namespace no longer sees all of it; that is the operator's call, not a default. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record what managed versioning does and does not reach The first commit through a namespace-backed store works and is recorded the way the protocol specifies. Later commits do not, because lance 4.0.0 refuses put_if_exists on that path in its own code, so the feature is capped upstream rather than here. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: integration tests for the Lance namespace Everything this surface got wrong so far - a deregister that deleted the dataset, an S3 door that refused every Lance file, a version reservation that could not actually be exclusive - passed against an in-memory filer first. So these run against a live gateway, and where the claim is about data they check storage rather than visibility. Five Go tests on the shared harness: namespace and table lifecycle including that deregister keeps the bytes and drop removes them, that a Lance client cannot resolve or declare over an Iceberg table, that a Lance dataset's files get past the table-bucket layout guard while junk still does not, and that eight writers racing for one version produce exactly one winner. One Docker-gated test drives the real Lance client, which is the only way to check that the location and storage_options the namespace vends are between them enough to write and read a dataset. It overrides the endpoint with the container's view of the same gateway, because the shared harness binds a wildcard address and so vends none. The harness gains a Lance port and turns managed versioning on; the flag touches nothing outside that surface. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: a directory with no namespace metadata is a missing namespace Three callers resolved a namespace by reading its metadata attribute and each tested only for a missing entry, so a directory that carried no metadata came back as an internal error saying "attribute not found". Creating a table under a namespace that does not exist answered 500. Collapses the three copies into one helper that reports both conditions as absent, which is what they are: a directory without namespace metadata is not a namespace. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: stop reporting storage-layer refusals as server faults writeManagerError recognised a missing table bucket and sent everything else to 500, so a missing namespace, a duplicate name and a commit conflict all reached the client as InternalServerError with nothing to act on. Creating a table in a namespace that does not exist is the case that turned up: 500 where the spec wants 404 NoSuchNamespaceException. Maps the storage error types onto the exception names this package already uses, and keeps the existing bucket message, which explains how to select a table bucket. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: skip a foreign-format table by name, not by failing to parse it A table the namespace created as LANCE carries no Iceberg metadata, so the worker skipped it only because the parse failed, and logged that as damaged metadata. The catalog records the format on the entry and this never read it. Reading it turns an accident into a decision, and separates a mixed catalog from a corrupt one in the logs. The property check beside it still covers the other shape: a real Iceberg table wearing table_type=lance, which is what the Lance namespace's Iceberg REST adapter writes. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: answer whether a Lance table needs maintenance It does, and index optimization has no Iceberg equivalent: rows written after an index was built are not covered by it, so a vector search quietly misses them. None of the three jobs can run in the Go worker, and there is no useful subset, because deciding what an old version still references means parsing Lance manifests. Version cleanup at least has an answer that needs nothing from us - Lance can enable it on the dataset itself. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: the Lance maintenance worker is a plugin worker, in Rust Framing it as a sidecar was wrong. plugin.proto already defines a language-agnostic gRPC contract for external maintenance workers, and "weed worker -admin=..." is the Go reference implementation of it from outside the admin process. seaweed-volume already compiles protos out of weed/pb with tonic_build, so a Lance worker is that build plus plugin.proto and the lance crate. Scheduling, retries, dedupe, progress and the admin settings page all come from the protocol: a worker that answers RequestConfigSchema with a descriptor gets its configuration form rendered without a line of Go. The data plane is the part that genuinely does need a process answering HTTP, and this had the two conflated. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: Rust plugin worker workspace, with Lance as the first one plugin.proto is language-agnostic and the Rust toolchain was already in the tree, so a Lance maintenance worker needs no new integration surface: core is the contract and nothing else, and a worker crate beside it supplies handlers and a binary. A second worker is a new member here rather than a fork of the protocol, which is why this is seaweed-worker and not seaweed-lance-worker. Verified against a running admin: it connects, is accepted, and admin prefetches descriptors for lance_compact, lance_optimize_indices and lance_cleanup_versions, so their settings pages render from the Rust side without a line of Go. The stream stays up across heartbeats. The job bodies are stubs that report failure. Doing the work means adding the lance crate and opening the dataset, and claiming success before that would be worse than saying so. Two things running it caught that reading the proto did not: the admin address has to be converted to the gRPC port the way pb.ServerToGrpcAddress does, or the dial fails as an h2 frame error; and the generated field names differ from the Go ones in several places, so JobCompleted carries success rather than a state enum. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: implement compaction Detection lists tables from the namespace, opens each one, and proposes a job for any with more fragments than the policy allows; opening a dataset reads its manifest and not its data, so a sweep stays cheap. Execution re-resolves the table rather than trusting what detection saw - it may have been repointed, and the vended credentials expire - then compacts and reports the fragment counts either side. Verified against a live gateway: a twelve-fragment dataset became one fragment with all twelve rows intact. The test drives the handler directly and skips unless WEED_LANCE_NAMESPACE names a namespace, the way the Go integration tests skip without Docker. Running it turned up a gap the design had not: a gateway without STS vends no credentials at all, so the worker could not open anything and detection quietly proposed nothing. --access-key/--secret-key are the fallback, and whatever the namespace vends still wins over them. Two API assumptions did not survive contact either. Datasets open through DatasetBuilder::with_storage_options, not ReadParams, and lance 10's ObjectStoreParams has no storage_options field at all. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: implement index optimization and version cleanup Index optimization is the job with no Iceberg equivalent: rows appended after an index was built are invisible to a search of it until this runs. Detection reads num_unindexed_rows from each index's statistics and proposes a table once more rows sit outside its indices than the budget allows; a table with no indices is skipped, which is different from one whose indices have fallen behind. Cleanup applies a retention window, refusing rather than silently dropping a tagged version, and leaving unverified files alone because they may belong to a commit still in flight. Both verified against a live gateway: 512 uncovered rows became 0, and a fourteen-version table lost its old ones. Each test now seeds what it needs, including building an IVF_PQ index and appending rows outside it. The first version of these depended on state a script had left, so the second run found the work already done and asserted nothing - a test that passes by doing nothing is worse than no test. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: answer an empty catalog with an empty list, not null ListAllTables built its result from a nil slice, so a namespace holding no tables answered {"tables":null} on a field the spec marks required. A generated client may decode that differently from an empty list. Found running the namespace on a dev box, where the catalog was empty. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: give Lance maintenance its own scheduler lane Lane assignment is a hardcoded map, so the three lance_* job types fell through to the default lane. That lane serialises its work under the cluster admin lock because volume management shares global state, which would queue a table's compaction behind volume balancing for no reason - Iceberg has its own lock-free lane for exactly this. Adds the lane, maps the three job types to it, and puts it in the sidebar beside Iceberg and Lifecycle. The lane routes were already generic, so only the nav was hand-written. The lane-coverage test spelled out the three known lanes, so a fourth failed it. It now checks against AllLanes(), which is the property it was reaching for and does not need editing next time. Found by connecting the Rust worker to a real admin: it registered fine and its job types were known, but they were filed under "default" and had no page. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: log what detection saw "Detection proposed nothing" and "the worker could not read the table" look identical from the admin side, and the second is what a missing credential produces. One line per table separates them. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: fix a leaked heartbeat and a silent reconnect loop spawn_heartbeat returned a handle to an empty task rather than the ticker it had just spawned, so aborting it aborted nothing and every reconnect left another heartbeat running against a dead channel. A stream that admin closes cleanly is not an error, but reconnecting in silence hides why. Two workers sharing an id evict each other forever and the log shows nothing but a login every five seconds - which is exactly how this presented on a dev box, and it took a look at the admin's own log to see it. The message now names the id to check. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: a namespace cannot be created without its parent Storage keeps a namespace's parts flattened, so creating "a.b" with no "a" was accepted and left an intermediate that only existed inside a name. Listing derives child names by slicing those parts, so it reported "a", while describe and exists on "a" both answered 404 - a client walking the tree got a 404 on something the listing had just handed it. The spec asks for NamespaceNotFound when the parent is missing, which is also what keeps listing and describe telling the same story. Namespaces created through the S3 Tables API still bypass this, so listing keeps deriving intermediates rather than hiding whatever is already there. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: say why a non-Iceberg table shows no schema The table pages read Iceberg metadata for schema and snapshots, and a Lance table has none, so both panels rendered "No schema available" - which reads as an empty table rather than a table this page cannot describe. The dataset behind the one that prompted this holds 1024 rows. The format is already on the entry and shown two rows above, so the empty states now use it: the catalog records where a LANCE table lives, not what is in it. Reading the schema for real needs Lance format code, which is the same wall as the data plane. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: run rustfmt over the workspace Committed the crates unformatted, so `cargo fmt --all --check` failed on files nothing had touched since. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * plugin: let a worker report what it saw about an object Admin cannot read a Lance table: it knows where the dataset lives and nothing else, so the details page had a location and two empty panels. The worker already opens every dataset during detection to decide whether it needs compacting, so it knows the schema, the row count and the fragment count at that moment. It just had no way to say so. Add a WorkerObservations body to the worker stream. Admin caches the last observation per object and serves it back, timestamped, for display; nothing schedules from it. The Lance compaction sweep reports what it opened, and the S3 Tables details page fills its schema panel from the cache when it has no metadata of its own, badged with when the worker looked and which worker it was. Nothing about this is Lance-specific past the reporting side, which is the point: any format admin cannot parse can describe itself the same way. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record the observation channel Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * plugin: ask a worker for sample rows of a table admin cannot read Browse Data reads an Iceberg table's Parquet files directly, so it shows real rows. For a Lance table it showed "Table has no Iceberg metadata" and an empty grid, because there is no Go Lance reader and never will be one worth maintaining. The worker has the reader. Add RequestObjectPreview / ObjectPreviewResponse to the stream, mirroring the config-schema round trip that already exists, and give the Rust worker a PreviewProvider that scans the dataset and formats the rows with Arrow's own formatter, so a vector column reads as a vector. Admin picks the worker from the observation store: whichever one last described this table is the one that can read it. Unlike an observation the rows are not cached. They are the table's data rather than a description of it, and a copy sitting in admin would be both stale and nobody's business. The page fetches on load, bounded at 200 rows and a 15 second round trip, and drops the snapshot and data-file panels that only mean something for Iceberg. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record the preview channel Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: disable the lance listener when two gateways share a host * test: keep AllocatePorts away from the lance default port * s3tables: let a table bucket declare the format it holds A bucket is a catalog, and a catalog serves one protocol. Format was recorded per table, so nothing could answer "where do I point a client at this bucket" without opening a table first, and an empty bucket had no answer at all. CreateTableBucket takes an optional format, stored with the rest of the bucket metadata and returned by Get and List. Empty means ICEBERG, which is what AWS S3 Tables serves and therefore what an SDK that has never heard of the field means. CreateTable refuses a table of another format, and CreateView refuses outright in a bucket that is not Iceberg, since a view is Iceberg metadata. Buckets that already exist carry no declaration and keep accepting anything, so nothing is migrated and nothing that worked stops working. The Lance namespace declares LANCE for the buckets it creates, which is what stops one of them being described to a client as an Iceberg catalog. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: take the Lance port the way it takes the Iceberg one The UI cannot name the endpoint that serves a Lance bucket without it, and every format-aware page below needs to. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: show which format a table bucket holds The bucket list printed an Iceberg endpoint for every bucket, including ones holding Lance datasets, where that endpoint serves nothing. It was the most visible place the UI assumed one format. The list gains a Format column and its endpoint column follows the bucket's declaration. The banner names both endpoints rather than asserting everything is Iceberg, and says so only for the servers that are actually running. Create Bucket picks a format with two cards rather than a dropdown, since what matters is not the name but which clients can read the result, and the endpoint under them updates as you choose so the operator leaves the modal knowing where to point one. A bucket from before the declaration existed shows "unset" in an outline badge, explained on hover. It is a fact about the bucket's age, not a fault, so nothing nags about it. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: carry the bucket's format into the pages inside it Namespaces and tables are reached through a bucket, so both now say which catalog they belong to rather than making you go back up to find out. The tables list gains a Format column and a Rows column filled from what a worker last observed, since for a format admin cannot read that is the only row count there is; a table nothing has looked at shows a dash, not a zero. Create Table stops offering a choice the bucket has already made: in a declared bucket the format is fixed and says why, and only an undeclared one still offers both. Before this the select had exactly one option, hardcoded, which made a Lance table impossible to create from the UI at all. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: let the table page speak the table's own format Partitions and Snapshot History are Iceberg's shape. Rendering them empty for a Lance table reads as a fault; a Lance table has neither, and says so by not showing them. In their place is a Versions panel, which is what that format calls its history, carrying the worker's timestamp so it is clear the numbers are a cached look rather than something read live. The breadcrumb carries the format badge, so the page names what it is looking at before you read a panel and wonder why it is empty. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: show how to connect to either catalog, and group the two format workers The client examples on the buckets page were Iceberg's alone, so the one thing an operator wants after creating a Lance bucket - what to type to reach it - was not written down anywhere in the UI. Both formats now get a pair of snippets, and only for a server that is running. In the Workers menu, Iceberg moves below Lifecycle so it sits next to Lance: the two table-format workers together, the two cluster-wide ones above them. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * shell: create a table bucket of either format s3tables.bucket -create takes -format, so a Lance bucket can be made without going through the UI. The integration harness passes it too: its Lance tests were creating Iceberg buckets and getting away with it only because nothing checked. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record that a bucket declares its format Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: drop managed versioning; the store already orders commits The namespace offered itself as an external manifest store, so that a commit could reserve a version through a real put-if-not-exists. That was designed around a gateway that no longer exists: If-None-Match: * is reduced to a filer WriteCondition and evaluated at the object's owner under its per-path lock, or under the object write lock on the fallback path. Sixteen writers racing one fresh key get a single 200 and fifteen 412s, every time. Lance needs nothing else. commit_handler_from_url hands every s3:// dataset a ConditionalPutCommitHandler, which puts with PutMode::Create, which object_store sends as If-None-Match: *. So the feature solved a problem this store does not have, while moving a table's version history out of the dataset and into the catalog - and lance could not use it past the first commit anyway, since its own namespace-backed store answers "put_if_not_exists is not supported" to the second. The version operations answer Unsupported with the rest, managed_versioning is false, and the flag is gone. In place of the reserve-once test there is one that races eight writers at the manifest key through S3, which is the path a commit actually takes. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: honour the version floor, the slot limits, and a shutdown Five findings from review, all of them things the worker claimed to do and did not. The version floor was checked when a cleanup job was proposed and ignored when it ran, so a table whose versions had aged past the retention window in between could be taken below the count the operator asked to keep. Execution now computes the floor itself and passes it as before_version; CleanupPolicy ANDs its clauses, so a version has to be both too old and below the floor to go. Both settings are clamped to the range the form offers, since Duration::hours panics on a large enough value and a negative min-versions wraps to a huge usize. Admin's shutdown was answered by returning from the stream, which the reconnect loop read as a healthy close and logged straight back in: the worker could not be stopped. serve_once now says which of the two happened. The advertised concurrency limits bounded nothing - every request spawned a task - and the heartbeat reported zero slots in use whatever was running. Both now go through semaphores sized from the limits, with the permits held for the life of the request and reported in the heartbeat. A namespace call had no timeout, so a gateway that accepted the connection and went quiet held a detection slot forever. And one table whose stats could not be read failed the whole sweep, losing the proposals for every table already scanned; it is now skipped and warned about, like a table that cannot be opened. The tests drove one shared catalog concurrently, which is why one of them asserted "no proposals at all" and passed by luck. They now take a lock and judge only their own tables. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: fix the review findings on the format-aware pages The endpoint hint in Create Bucket built its HTML by concatenating the bucket name the operator is typing, so a name like <img onerror=...> ran in the admin origin as they typed it. It is built from DOM nodes now. A preview reply looked its channel up under the lock and then sent outside it, which Shutdown can close in between: a Gosched in that gap panics with "send on closed channel" every time. The send now happens under the lock. Observations were looked up by path alone, so a table dropped and remade in another format at the same path was described by the observation left behind. Lookups now have to agree on the format. Also: the Lance namespace caps a request body rather than reading whatever arrives; the details action no longer says "Iceberg" over a Lance table; mini stops advertising a catalog port when it is not running S3; a format whose server this cluster does not run cannot be picked in the modal or accepted by the API, since a bucket nothing can reach is not worth creating; and the unused catalogPortFor helper is gone. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: let the control stream use mTLS The channel was hardcoded to http://, so off loopback the stream carried preview rows and execution commands in the clear - and a cluster with grpc TLS turned on would refuse the worker outright. --tls-ca, --tls-cert and --tls-key take the same certificates the Go worker reads from the [grpc.worker] section of security.toml, and must be given together: a CA on its own would quietly mean one-way TLS, which a mutual setup rejects anyway. Without them the stream stays plaintext, which is what the Go worker also does when nothing is configured. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: answer null properties rather than an empty map The catalog does not keep a table's properties. Declare echoed the request's back and describe answered {}, both of which claim they were stored and are empty. Null says the catalog does not keep them, which is what the spec distinguishes and what is true here. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: test the slot accounting The heartbeat reporting and the waiting are the two things the semaphores are for, and neither is observable from outside without catching a sweep mid-flight. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: fix the mixed-format catalog test, and name the binary it drives The integration suite passed locally and failed in CI on TestLanceRefusesIcebergTables. Both were right: CI builds the binary first, my tree had one from the day before, so locally the test drove a gateway with no format enforcement at all. The test itself no longer holds as written. It made a bucket, put an Iceberg table in it, and checked the Lance surface hid it - but a bucket that declares LANCE now refuses the Iceberg table outright. The invariant still matters from the other side, so it starts from an Iceberg bucket instead: Lance must not describe or list a table whose format it does not serve, and must refuse to declare one beside it. The harness now prints which weed binary it is about to run and when that was built. `make test` rebuilds first; a plain `go test` will happily drive a weeks-old binary and report a pass for code it never ran, which is exactly what happened here. Also make the row-limit conversion in the preview request explicitly bounded: CodeQL flagged the int-to-int32 conversion, and clamping by reassignment beforehand is not a form it recognises. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: prove concurrent commits are kept, and preselect the only format on offer Two more from review. The commit test asserted that exactly one writer wins the conditional PUT, which is the mechanism, not the claim. The claim is that nothing is lost: the losers see the conflict, rebase and commit again. So there is now a test that has eight writers append to one dataset at once and counts the rows afterwards - all eight batches survive. That is also the sequence managed versioning could not finish, since its store refuses the second commit outright. And when Iceberg's endpoint is not running, the format picker offered two options with neither selected, so Create Bucket submitted no format at all, fell back to ICEBERG, and was refused by the guard added last round. Lance is preselected when it is the only format this cluster serves. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * Clamp the remaining worker settings, and bootstrap buckets in a served format Compaction and index optimization read their thresholds and cast straight to usize and u64, so a negative arrives as an enormous number and turns the threshold into "never": compaction and reindexing both go quiet with nothing to say. The cleanup job was fixed last round; these are the same bug. Clamped to the values that stay meaningful rather than to what the form offers - zero uncovered rows is a real setting, meaning reindex as soon as anything is not covered, so the floor there is zero and not the form's thousand. mini pre-creates the buckets named by -tableBucket, and did so without a format, which now means Iceberg. Started with the Iceberg endpoint off and the Lance one on, that left buckets nothing could reach and which refused every Lance table. It takes the format from the endpoint that is actually running, and creates nothing when neither is. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3: allow-unordered is a listing parameter, not an unimplemented subresource The guard that stops a bucket GET with an unknown subresource from being answered with a listing does not know about allow-unordered, so it answers 501 NotImplemented - to a parameter the listing handlers already read and already validate against delimiter. This is why test_bucket_list_unordered and test_bucket_listv2_unordered fail in the Ceph s3-tests suite. They fail on master too; this is not a Lance change and can be taken on its own. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
This commit is contained in:
Chris Lu authored and GitHub committed
1 parent
814ee75af4
commit
8c7d714d5e
110 files changed
+18542
-2040
No files matched your search
@@ -69,6 +69,11 @@ func (s3a *S3ApiServer) validateTableBucketObjectPath(bucket, object string) err
|
||||
return err
|
||||
}
|
||||
parts := strings.SplitN(cleanObject, "/", 4)
|
||||
// A table's marker files sit at namespace/table/<marker>, one level above
|
||||
// everything else, so they are three parts rather than four.
|
||||
if len(parts) == 3 && s3tables.IsTableMarkerFile(parts[2]) {
|
||||
return nil
|
||||
}
|
||||
if len(parts) < 4 {
|
||||
return &s3tables.IcebergLayoutError{
|
||||
Code: s3tables.ErrCodeInvalidIcebergLayout,
|
||||
|
||||
@@ -169,14 +169,38 @@ func writeManagerError(w http.ResponseWriter, err error) {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
|
||||
return
|
||||
}
|
||||
// A missing table bucket means the catalog the client selected does not
|
||||
// exist, not a server fault. The storage-layer message names the resolved
|
||||
// bucket, which for a client that sent no warehouse at all is the default
|
||||
// one it never asked for, so say how to select a real table bucket.
|
||||
// Storage-layer failures are mostly the client's, not the server's. Reporting
|
||||
// a missing namespace or a name conflict as a 500 makes the catalog look
|
||||
// broken and gives the client nothing to act on.
|
||||
var tableErr *s3tables.S3TablesError
|
||||
if errors.As(err, &tableErr) && tableErr.Type == s3tables.ErrCodeNoSuchBucket {
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException",
|
||||
fmt.Sprintf("%s: each table bucket is a separate catalog, select one with warehouse=s3://<table-bucket>/ or /v1/<table-bucket>/", tableErr.Message))
|
||||
if errors.As(err, &tableErr) {
|
||||
switch tableErr.Type {
|
||||
case s3tables.ErrCodeNoSuchBucket:
|
||||
// The storage-layer message names the resolved bucket, which for a
|
||||
// client that sent no warehouse at all is the default one it never
|
||||
// asked for, so say how to select a real table bucket.
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException",
|
||||
fmt.Sprintf("%s: each table bucket is a separate catalog, select one with warehouse=s3://<table-bucket>/ or /v1/<table-bucket>/", tableErr.Message))
|
||||
case s3tables.ErrCodeNoSuchNamespace:
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", tableErr.Message)
|
||||
case s3tables.ErrCodeNoSuchTable:
|
||||
writeError(w, http.StatusNotFound, "NoSuchTableException", tableErr.Message)
|
||||
case s3tables.ErrCodeNoSuchView:
|
||||
writeError(w, http.StatusNotFound, "NoSuchViewException", tableErr.Message)
|
||||
case s3tables.ErrCodeNamespaceAlreadyExists, s3tables.ErrCodeBucketAlreadyExists,
|
||||
s3tables.ErrCodeTableAlreadyExists, s3tables.ErrCodeViewAlreadyExists:
|
||||
writeError(w, http.StatusConflict, "AlreadyExistsException", tableErr.Message)
|
||||
case s3tables.ErrCodeNamespaceNotEmpty, s3tables.ErrCodeBucketNotEmpty:
|
||||
writeError(w, http.StatusConflict, "AlreadyExistsException", tableErr.Message)
|
||||
case s3tables.ErrCodeConflict:
|
||||
writeError(w, http.StatusConflict, "CommitFailedException", tableErr.Message)
|
||||
case s3tables.ErrCodeAccessDenied:
|
||||
writeError(w, http.StatusForbidden, "ForbiddenException", tableErr.Message)
|
||||
case s3tables.ErrCodeInvalidRequest, s3tables.ErrCodeInvalidIcebergLayout:
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", tableErr.Message)
|
||||
default:
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
|
||||
}
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
package lance
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// Lance Namespace error codes. The spec numbers them, so a client switches on
|
||||
// the code rather than parsing the message.
|
||||
const (
|
||||
codeUnsupported = 0
|
||||
codeNamespaceNotFound = 1
|
||||
codeNamespaceAlreadyExists = 2
|
||||
codeNamespaceNotEmpty = 3
|
||||
codeTableNotFound = 4
|
||||
codeTableAlreadyExists = 5
|
||||
codeTableVersionNotFound = 11
|
||||
codeInvalidInput = 13
|
||||
codeConcurrentModification = 14
|
||||
codePermissionDenied = 15
|
||||
codeUnauthenticated = 16
|
||||
codeInternal = 18
|
||||
)
|
||||
|
||||
type errorResponse struct {
|
||||
Error string `json:"error,omitempty"`
|
||||
Code int `json:"code"`
|
||||
Detail string `json:"detail,omitempty"`
|
||||
Instance string `json:"instance,omitempty"`
|
||||
}
|
||||
|
||||
func writeJSON(w http.ResponseWriter, status int, body interface{}) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
if body == nil {
|
||||
return
|
||||
}
|
||||
if err := json.NewEncoder(w).Encode(body); err != nil {
|
||||
glog.Warningf("lance: failed to encode response: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func writeError(w http.ResponseWriter, r *http.Request, status, code int, message string) {
|
||||
instance := ""
|
||||
if r != nil {
|
||||
instance = r.URL.Path
|
||||
}
|
||||
writeJSON(w, status, errorResponse{Error: message, Code: code, Instance: instance})
|
||||
}
|
||||
|
||||
// writeStorageError translates an S3 Tables storage error into the Lance error
|
||||
// model. A table bucket is the first namespace level here, so a missing bucket
|
||||
// is a missing namespace rather than a missing catalog.
|
||||
func writeStorageError(w http.ResponseWriter, r *http.Request, err error) {
|
||||
var storageErr *s3tables.S3TablesError
|
||||
if !errors.As(err, &storageErr) {
|
||||
writeError(w, r, http.StatusInternalServerError, codeInternal, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
status, code := http.StatusInternalServerError, codeInternal
|
||||
switch storageErr.Type {
|
||||
case s3tables.ErrCodeNoSuchBucket, s3tables.ErrCodeNoSuchNamespace:
|
||||
status, code = http.StatusNotFound, codeNamespaceNotFound
|
||||
case s3tables.ErrCodeNoSuchTable, s3tables.ErrCodeNoSuchView:
|
||||
status, code = http.StatusNotFound, codeTableNotFound
|
||||
case s3tables.ErrCodeBucketAlreadyExists, s3tables.ErrCodeNamespaceAlreadyExists:
|
||||
status, code = http.StatusConflict, codeNamespaceAlreadyExists
|
||||
case s3tables.ErrCodeTableAlreadyExists, s3tables.ErrCodeViewAlreadyExists:
|
||||
status, code = http.StatusConflict, codeTableAlreadyExists
|
||||
case s3tables.ErrCodeBucketNotEmpty, s3tables.ErrCodeNamespaceNotEmpty:
|
||||
status, code = http.StatusConflict, codeNamespaceNotEmpty
|
||||
case s3tables.ErrCodeConflict:
|
||||
status, code = http.StatusConflict, codeConcurrentModification
|
||||
case s3tables.ErrCodeAccessDenied:
|
||||
status, code = http.StatusForbidden, codePermissionDenied
|
||||
case s3tables.ErrCodeInvalidRequest, s3tables.ErrCodeInvalidIcebergLayout:
|
||||
status, code = http.StatusBadRequest, codeInvalidInput
|
||||
}
|
||||
writeError(w, r, status, code, storageErr.Message)
|
||||
}
|
||||
|
||||
// isNotFound reports whether a storage error means the object is absent, so
|
||||
// exists-style handlers can answer without a second lookup.
|
||||
func isNotFound(err error) bool {
|
||||
var storageErr *s3tables.S3TablesError
|
||||
if !errors.As(err, &storageErr) {
|
||||
return false
|
||||
}
|
||||
switch storageErr.Type {
|
||||
case s3tables.ErrCodeNoSuchBucket, s3tables.ErrCodeNoSuchNamespace,
|
||||
s3tables.ErrCodeNoSuchTable, s3tables.ErrCodeNoSuchView:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,341 @@
|
||||
package lance
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// handleCreateNamespace creates a table bucket for a one-part identifier and a
|
||||
// namespace inside one for anything deeper. The root cannot be created.
|
||||
func (s *Server) handleCreateNamespace(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
var req CreateNamespaceRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
if len(id) == 0 {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, "the root namespace always exists and cannot be created")
|
||||
return
|
||||
}
|
||||
|
||||
mode := normalizeMode(req.Mode, modeCreate)
|
||||
switch mode {
|
||||
case modeCreate, modeExistOk, modeOverwrite:
|
||||
default:
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, "mode must be Create, ExistOk or Overwrite")
|
||||
return
|
||||
}
|
||||
|
||||
bucket, ns := id.namespace()
|
||||
exists, err := s.namespaceExists(r, bucket, ns)
|
||||
if err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
if exists {
|
||||
switch mode {
|
||||
case modeExistOk:
|
||||
writeJSON(w, http.StatusOK, CreateNamespaceResponse{Properties: req.Properties})
|
||||
return
|
||||
case modeCreate:
|
||||
writeError(w, r, http.StatusConflict, codeNamespaceAlreadyExists, "namespace already exists")
|
||||
return
|
||||
}
|
||||
// Overwrite replaces the namespace with an empty one, so the drop has to
|
||||
// succeed first. A namespace holding tables refuses, which is the point.
|
||||
if err := s.dropNamespace(r, bucket, ns); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if err := s.createNamespace(r, bucket, ns, req.Properties); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, CreateNamespaceResponse{Properties: normalizeProperties(req.Properties)})
|
||||
}
|
||||
|
||||
// handleListNamespaces lists the children of a namespace: table buckets at the
|
||||
// root, and the next path component below that.
|
||||
func (s *Server) handleListNamespaces(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
bucket, ns := id.namespace()
|
||||
|
||||
if bucket == "" {
|
||||
var resp s3tables.ListTableBucketsResponse
|
||||
req := &s3tables.ListTableBucketsRequest{
|
||||
ContinuationToken: r.URL.Query().Get("page_token"),
|
||||
MaxBuckets: pageSize(r),
|
||||
}
|
||||
if err := s.execute(r, "ListTableBuckets", req, &resp); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
names := make([]string, 0, len(resp.TableBuckets))
|
||||
for _, b := range resp.TableBuckets {
|
||||
names = append(names, b.Name)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, ListNamespacesResponse{Namespaces: names, PageToken: resp.ContinuationToken})
|
||||
return
|
||||
}
|
||||
|
||||
var resp s3tables.ListNamespacesResponse
|
||||
req := &s3tables.ListNamespacesRequest{
|
||||
TableBucketARN: bucketARN(bucket),
|
||||
ContinuationToken: r.URL.Query().Get("page_token"),
|
||||
MaxNamespaces: pageSize(r),
|
||||
}
|
||||
if len(ns) > 0 {
|
||||
req.Prefix = strings.Join(ns, ".") + "."
|
||||
}
|
||||
if err := s.execute(r, "ListNamespaces", req, &resp); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
|
||||
// Storage namespaces are full paths; the spec wants the child name relative
|
||||
// to the parent, so take the next component and drop repeats.
|
||||
children := make([]string, 0, len(resp.Namespaces))
|
||||
seen := make(map[string]struct{}, len(resp.Namespaces))
|
||||
for _, summary := range resp.Namespaces {
|
||||
if len(summary.Namespace) <= len(ns) {
|
||||
continue
|
||||
}
|
||||
child := summary.Namespace[len(ns)]
|
||||
if _, done := seen[child]; done {
|
||||
continue
|
||||
}
|
||||
seen[child] = struct{}{}
|
||||
children = append(children, child)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, ListNamespacesResponse{Namespaces: children, PageToken: resp.ContinuationToken})
|
||||
}
|
||||
|
||||
// handleDescribeNamespace returns a namespace's properties.
|
||||
func (s *Server) handleDescribeNamespace(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req DescribeNamespaceRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
|
||||
bucket, ns := id.namespace()
|
||||
if bucket == "" {
|
||||
writeJSON(w, http.StatusOK, DescribeNamespaceResponse{Properties: map[string]string{}})
|
||||
return
|
||||
}
|
||||
if len(ns) == 0 {
|
||||
var resp s3tables.GetTableBucketResponse
|
||||
if err := s.execute(r, "GetTableBucket", &s3tables.GetTableBucketRequest{TableBucketARN: bucketARN(bucket)}, &resp); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, DescribeNamespaceResponse{Properties: map[string]string{}})
|
||||
return
|
||||
}
|
||||
|
||||
var resp s3tables.GetNamespaceResponse
|
||||
req2 := &s3tables.GetNamespaceRequest{TableBucketARN: bucketARN(bucket), Namespace: ns}
|
||||
if err := s.execute(r, "GetNamespace", req2, &resp); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, DescribeNamespaceResponse{Properties: normalizeProperties(resp.Properties)})
|
||||
}
|
||||
|
||||
// handleDropNamespace removes a namespace or table bucket.
|
||||
func (s *Server) handleDropNamespace(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req DropNamespaceRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
if len(id) == 0 {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, "the root namespace cannot be dropped")
|
||||
return
|
||||
}
|
||||
|
||||
if normalizeMode(req.Behavior, behaviorRestrict) == behaviorCascade {
|
||||
writeError(w, r, http.StatusNotImplemented, codeUnsupported,
|
||||
"cascade drop is not supported; drop the tables in the namespace first")
|
||||
return
|
||||
}
|
||||
|
||||
bucket, ns := id.namespace()
|
||||
err := s.dropNamespace(r, bucket, ns)
|
||||
if err == nil {
|
||||
writeJSON(w, http.StatusOK, DropNamespaceResponse{})
|
||||
return
|
||||
}
|
||||
if isNotFound(err) {
|
||||
// Skip reports success on a missing namespace; Fail reports 400 rather
|
||||
// than 404, which is what the spec asks for on this operation alone.
|
||||
if normalizeMode(req.Mode, modeFail) == modeSkip {
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
return
|
||||
}
|
||||
writeError(w, r, http.StatusBadRequest, codeNamespaceNotFound, "namespace does not exist")
|
||||
return
|
||||
}
|
||||
writeStorageError(w, r, err)
|
||||
}
|
||||
|
||||
// handleNamespaceExists answers with the status code and no body.
|
||||
func (s *Server) handleNamespaceExists(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req NamespaceExistsRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
|
||||
bucket, ns := id.namespace()
|
||||
if bucket == "" {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
exists, err := s.namespaceExists(r, bucket, ns)
|
||||
if err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
if !exists {
|
||||
writeError(w, r, http.StatusNotFound, codeNamespaceNotFound, "namespace does not exist")
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}
|
||||
|
||||
func (s *Server) namespaceExists(r *http.Request, bucket string, ns []string) (bool, error) {
|
||||
var err error
|
||||
if len(ns) == 0 {
|
||||
var resp s3tables.GetTableBucketResponse
|
||||
err = s.execute(r, "GetTableBucket", &s3tables.GetTableBucketRequest{TableBucketARN: bucketARN(bucket)}, &resp)
|
||||
} else {
|
||||
var resp s3tables.GetNamespaceResponse
|
||||
err = s.execute(r, "GetNamespace", &s3tables.GetNamespaceRequest{TableBucketARN: bucketARN(bucket), Namespace: ns}, &resp)
|
||||
}
|
||||
if err == nil {
|
||||
return true, nil
|
||||
}
|
||||
if isNotFound(err) {
|
||||
return false, nil
|
||||
}
|
||||
return false, err
|
||||
}
|
||||
|
||||
func (s *Server) createNamespace(r *http.Request, bucket string, ns []string, properties map[string]string) error {
|
||||
if len(ns) == 0 {
|
||||
// A bucket made through this surface holds Lance tables. Saying so is
|
||||
// what stops it from being described to a client as an Iceberg catalog.
|
||||
var resp s3tables.CreateTableBucketResponse
|
||||
return s.execute(r, "CreateTableBucket", &s3tables.CreateTableBucketRequest{
|
||||
Name: bucket,
|
||||
Format: s3tables.FormatLance,
|
||||
}, &resp)
|
||||
}
|
||||
// A table bucket is a tenant resource with its own policy and lifecycle, so
|
||||
// it is created deliberately, never as a side effect of naming a namespace
|
||||
// inside it.
|
||||
// The immediate parent has to exist too. Storage keeps a namespace's parts
|
||||
// flattened, so creating "a.b" without "a" leaves an intermediate that
|
||||
// listing derives from the name and describe then denies exists. The spec
|
||||
// asks for NamespaceNotFound here, which also keeps the two consistent.
|
||||
if len(ns) > 1 {
|
||||
parent := ns[:len(ns)-1]
|
||||
if exists, err := s.namespaceExists(r, bucket, parent); err != nil {
|
||||
return err
|
||||
} else if !exists {
|
||||
return &s3tables.S3TablesError{
|
||||
Type: s3tables.ErrCodeNoSuchNamespace,
|
||||
Message: "parent namespace " + strings.Join(parent, ".") + " does not exist",
|
||||
}
|
||||
}
|
||||
}
|
||||
if exists, err := s.namespaceExists(r, bucket, nil); err != nil {
|
||||
return err
|
||||
} else if !exists {
|
||||
return &s3tables.S3TablesError{
|
||||
Type: s3tables.ErrCodeNoSuchBucket,
|
||||
Message: "table bucket " + bucket + " does not exist",
|
||||
}
|
||||
}
|
||||
var resp s3tables.CreateNamespaceResponse
|
||||
req := &s3tables.CreateNamespaceRequest{
|
||||
TableBucketARN: bucketARN(bucket),
|
||||
Namespace: ns,
|
||||
Properties: properties,
|
||||
}
|
||||
return s.execute(r, "CreateNamespace", req, &resp)
|
||||
}
|
||||
|
||||
func (s *Server) dropNamespace(r *http.Request, bucket string, ns []string) error {
|
||||
if len(ns) == 0 {
|
||||
return s.execute(r, "DeleteTableBucket", &s3tables.DeleteTableBucketRequest{TableBucketARN: bucketARN(bucket)}, nil)
|
||||
}
|
||||
req := &s3tables.DeleteNamespaceRequest{TableBucketARN: bucketARN(bucket), Namespace: ns}
|
||||
return s.execute(r, "DeleteNamespace", req, nil)
|
||||
}
|
||||
|
||||
// normalizeMode folds a spec mode or behavior value, which is case-insensitive
|
||||
// and spelled either PascalCase or snake_case, onto its lowercase form.
|
||||
func normalizeMode(value, fallback string) string {
|
||||
value = strings.ToLower(strings.ReplaceAll(strings.TrimSpace(value), "_", ""))
|
||||
if value == "" {
|
||||
return fallback
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
func normalizeProperties(properties map[string]string) map[string]string {
|
||||
if properties == nil {
|
||||
return map[string]string{}
|
||||
}
|
||||
return properties
|
||||
}
|
||||
@@ -0,0 +1,566 @@
|
||||
package lance
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// handleListTables lists the Lance tables under a namespace. The spec asks for
|
||||
// full string identifiers, not bare names, so a recursive listing stays
|
||||
// unambiguous.
|
||||
func (s *Server) handleListTables(w http.ResponseWriter, r *http.Request) {
|
||||
id, delimiter, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
bucket, ns := id.namespace()
|
||||
if bucket == "" {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, "listing tables needs a namespace, not the root")
|
||||
return
|
||||
}
|
||||
|
||||
req := &s3tables.ListTablesRequest{
|
||||
TableBucketARN: bucketARN(bucket),
|
||||
Namespace: ns,
|
||||
ContinuationToken: r.URL.Query().Get("page_token"),
|
||||
MaxTables: pageSize(r),
|
||||
}
|
||||
var resp s3tables.ListTablesResponse
|
||||
if err := s.execute(r, "ListTables", req, &resp); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
|
||||
includeDeclared := true
|
||||
if raw := r.URL.Query().Get("include_declared"); raw != "" {
|
||||
includeDeclared = boolQuery(r, "include_declared")
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, ListTablesResponse{
|
||||
Tables: s.lanceTableIDs(r, bucket, resp.Tables, delimiter, includeDeclared),
|
||||
PageToken: resp.ContinuationToken,
|
||||
})
|
||||
}
|
||||
|
||||
// handleListAllTables lists every Lance table the caller can see, across every
|
||||
// table bucket.
|
||||
func (s *Server) handleListAllTables(w http.ResponseWriter, r *http.Request) {
|
||||
delimiter := requestDelimiter(r)
|
||||
|
||||
var buckets s3tables.ListTableBucketsResponse
|
||||
if err := s.execute(r, "ListTableBuckets", &s3tables.ListTableBucketsRequest{MaxBuckets: pageSize(r)}, &buckets); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
|
||||
// The spec marks `tables` required, so an empty catalog answers with an
|
||||
// empty list rather than null.
|
||||
all := []string{}
|
||||
for _, bucket := range buckets.TableBuckets {
|
||||
var tables s3tables.ListTablesResponse
|
||||
req := &s3tables.ListTablesRequest{TableBucketARN: bucketARN(bucket.Name), MaxTables: pageSize(r)}
|
||||
if err := s.execute(r, "ListTables", req, &tables); err != nil {
|
||||
// One unreadable bucket must not hide the rest; a caller with access
|
||||
// to some buckets still gets those.
|
||||
glog.V(2).Infof("lance: skipping bucket %s in ListAllTables: %v", bucket.Name, err)
|
||||
continue
|
||||
}
|
||||
all = append(all, s.lanceTableIDs(r, bucket.Name, tables.Tables, delimiter, true)...)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, ListTablesResponse{Tables: all})
|
||||
}
|
||||
|
||||
// lanceTableIDs keeps the Lance tables out of a listing that also carries
|
||||
// Iceberg tables, and drops deregistered ones because they are meant to be
|
||||
// invisible until re-registered.
|
||||
func (s *Server) lanceTableIDs(r *http.Request, bucket string, summaries []s3tables.TableSummary, delimiter string, includeDeclared bool) []string {
|
||||
ids := make([]string, 0, len(summaries))
|
||||
for _, summary := range summaries {
|
||||
if summary.Format != s3tables.FormatLance {
|
||||
continue
|
||||
}
|
||||
location := summary.MetadataLocation
|
||||
if location == "" {
|
||||
location = tableLocation(bucket, summary.Namespace, summary.Name)
|
||||
}
|
||||
deregistered, hasData, err := s.datasetState(r, location)
|
||||
if err != nil {
|
||||
glog.V(2).Infof("lance: cannot read dataset state for %s: %v", location, err)
|
||||
continue
|
||||
}
|
||||
if deregistered {
|
||||
continue
|
||||
}
|
||||
if !hasData && !includeDeclared {
|
||||
continue
|
||||
}
|
||||
parts := append([]string{bucket}, summary.Namespace...)
|
||||
ids = append(ids, identifier(append(parts, summary.Name)).String(delimiter))
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
// handleDeclareTable records a table that does not exist on storage yet. This
|
||||
// is what a Lance client calls on CREATE TABLE, before it writes any data.
|
||||
func (s *Server) handleDeclareTable(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req DeclareTableRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
bucket, ns, name, err := id.table()
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
location := strings.TrimSuffix(req.Location, "/")
|
||||
if location == "" {
|
||||
location = tableLocation(bucket, ns, name)
|
||||
}
|
||||
|
||||
if err := s.createTable(r, bucket, ns, name, location); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
if err := s.writeMarker(r, location, reservedMarker); err != nil {
|
||||
// The catalog entry is the authority; the marker only mirrors it for
|
||||
// clients that read the storage prefix directly.
|
||||
glog.V(1).Infof("lance: could not write %s for %s: %v", reservedMarker, location, err)
|
||||
}
|
||||
// Declaring a name that was deregistered brings it back, the same way
|
||||
// registering it does.
|
||||
if err := s.removeMarker(r, location, deregisteredMarker); err != nil {
|
||||
glog.V(2).Infof("lance: could not clear %s for %s: %v", deregisteredMarker, location, err)
|
||||
}
|
||||
|
||||
options, err := s.storageOptions(r, bucket, location, wants(r, "vend_credentials", req.VendCredentials))
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusInternalServerError, codeInternal, err.Error())
|
||||
return
|
||||
}
|
||||
// Properties are not persisted for a table, so neither response carries
|
||||
// them: null says "this catalog does not keep them", where echoing the
|
||||
// request back or answering {} would say they were stored and are empty.
|
||||
writeJSON(w, http.StatusOK, DeclareTableResponse{
|
||||
Location: location,
|
||||
StorageOptions: options,
|
||||
})
|
||||
}
|
||||
|
||||
// handleDescribeTable resolves a table to a location, and to credentials when
|
||||
// the caller asks for them.
|
||||
func (s *Server) handleDescribeTable(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req DescribeTableRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
bucket, ns, name, err := id.table()
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if req.Version != nil || req.Tag != "" || req.Branch != "" {
|
||||
writeError(w, r, http.StatusNotImplemented, codeUnsupported,
|
||||
"this namespace does not resolve versions, tags or branches; the dataset owns them")
|
||||
return
|
||||
}
|
||||
|
||||
table, err := s.loadLanceTable(r, bucket, ns, name)
|
||||
if err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
location := table.location
|
||||
|
||||
deregistered, hasData, err := s.datasetState(r, location)
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusInternalServerError, codeInternal, err.Error())
|
||||
return
|
||||
}
|
||||
if deregistered {
|
||||
writeError(w, r, http.StatusNotFound, codeTableNotFound, "table is deregistered")
|
||||
return
|
||||
}
|
||||
|
||||
options, err := s.storageOptions(r, bucket, location, wants(r, "vend_credentials", req.VendCredentials))
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusInternalServerError, codeInternal, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
resp := DescribeTableResponse{
|
||||
Location: location,
|
||||
}
|
||||
if len(options) > 0 {
|
||||
resp.StorageOptions = options
|
||||
}
|
||||
if wants(r, "with_table_uri", req.WithTableURI) {
|
||||
resp.TableURI = location
|
||||
}
|
||||
if wants(r, "check_declared", req.CheckDeclared) {
|
||||
onlyDeclared := !hasData
|
||||
resp.IsOnlyDeclared = &onlyDeclared
|
||||
}
|
||||
if wants(r, "load_detailed_metadata", req.LoadDetailedMetadata) {
|
||||
resp.Table = name
|
||||
resp.Namespace = ns
|
||||
}
|
||||
writeJSON(w, http.StatusOK, resp)
|
||||
}
|
||||
|
||||
// handleTableExists answers with the status code and no body.
|
||||
func (s *Server) handleTableExists(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req TableExistsRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
bucket, ns, name, err := id.table()
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
table, err := s.loadLanceTable(r, bucket, ns, name)
|
||||
if err != nil {
|
||||
if isNotFound(err) {
|
||||
writeError(w, r, http.StatusNotFound, codeTableNotFound, "table does not exist")
|
||||
return
|
||||
}
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
deregistered, _, err := s.datasetState(r, table.location)
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusInternalServerError, codeInternal, err.Error())
|
||||
return
|
||||
}
|
||||
if deregistered {
|
||||
writeError(w, r, http.StatusNotFound, codeTableNotFound, "table is deregistered")
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}
|
||||
|
||||
// handleRegisterTable points a table name at an existing dataset.
|
||||
func (s *Server) handleRegisterTable(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req RegisterTableRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
bucket, ns, name, err := id.table()
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
location := strings.TrimSuffix(req.Location, "/")
|
||||
if location == "" {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, "location is required")
|
||||
return
|
||||
}
|
||||
|
||||
mode := normalizeMode(req.Mode, modeCreate)
|
||||
if mode != modeCreate && mode != modeOverwrite {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, "mode must be Create or Overwrite")
|
||||
return
|
||||
}
|
||||
|
||||
existing, err := s.loadLanceTable(r, bucket, ns, name)
|
||||
switch {
|
||||
case err != nil && !isNotFound(err):
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
case err == nil:
|
||||
// A deregistered table is absent as far as the spec is concerned, so
|
||||
// registering over it is a re-registration rather than a conflict.
|
||||
deregistered, _, stateErr := s.datasetState(r, existing.location)
|
||||
if stateErr != nil {
|
||||
writeError(w, r, http.StatusInternalServerError, codeInternal, stateErr.Error())
|
||||
return
|
||||
}
|
||||
if !deregistered && mode == modeCreate {
|
||||
writeError(w, r, http.StatusConflict, codeTableAlreadyExists, "table already exists")
|
||||
return
|
||||
}
|
||||
if existing.location != location {
|
||||
// Repointing the name at another dataset is an update. Dropping and
|
||||
// recreating the entry would take the old dataset's files with it,
|
||||
// because the entry is the directory holding them.
|
||||
if err := s.repointTable(r, bucket, ns, name, location, existing.versionToken); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
default:
|
||||
if err := s.createTable(r, bucket, ns, name, location); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// Registering a deregistered dataset brings it back.
|
||||
if err := s.removeMarker(r, location, deregisteredMarker); err != nil {
|
||||
glog.V(2).Infof("lance: could not clear %s for %s: %v", deregisteredMarker, location, err)
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, RegisterTableResponse{
|
||||
Location: location,
|
||||
Properties: normalizeProperties(req.Properties),
|
||||
})
|
||||
}
|
||||
|
||||
// handleDeregisterTable forgets a table without touching its data.
|
||||
func (s *Server) handleDeregisterTable(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req DeregisterTableRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
bucket, ns, name, err := id.table()
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
table, err := s.loadLanceTable(r, bucket, ns, name)
|
||||
if err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
|
||||
// Deregistering is a state, not a deletion. Dropping the catalog entry would
|
||||
// take the dataset with it, because the entry is the dataset directory and
|
||||
// DeleteTable purges what it holds. The marker is what hides the table, and
|
||||
// it is also what a directory-catalog client reads.
|
||||
if err := s.writeMarker(r, table.location, deregisteredMarker); err != nil {
|
||||
writeError(w, r, http.StatusInternalServerError, codeInternal,
|
||||
"could not mark the table deregistered: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, DeregisterTableResponse{
|
||||
ID: append(append([]string{bucket}, ns...), name),
|
||||
Location: table.location,
|
||||
Properties: map[string]string{},
|
||||
})
|
||||
}
|
||||
|
||||
// handleDropTable removes the table and its data.
|
||||
func (s *Server) handleDropTable(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req DropTableRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
bucket, ns, name, err := id.table()
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
table, err := s.loadLanceTable(r, bucket, ns, name)
|
||||
if err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
if err := s.dropTableEntry(r, bucket, ns, name); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, DropTableResponse{
|
||||
ID: append(append([]string{bucket}, ns...), name),
|
||||
Location: table.location,
|
||||
Properties: map[string]string{},
|
||||
})
|
||||
}
|
||||
|
||||
// handleRenameTable moves a table's catalog entry. The dataset stays put, which
|
||||
// is what the storage layer already does for a renamed Iceberg table.
|
||||
func (s *Server) handleRenameTable(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req RenameTableRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
bucket, ns, name, err := id.table()
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
destBucket, destNS, destName, err := identifier(req.NewID).table()
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, "new_id: "+err.Error())
|
||||
return
|
||||
}
|
||||
if destBucket != bucket {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput,
|
||||
"a table cannot move between table buckets")
|
||||
return
|
||||
}
|
||||
|
||||
if _, err := s.loadLanceTable(r, bucket, ns, name); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
|
||||
renameReq := &s3tables.RenameTableRequest{
|
||||
TableBucketARN: bucketARN(bucket),
|
||||
SourceNamespace: ns,
|
||||
SourceName: name,
|
||||
DestNamespace: destNS,
|
||||
DestName: destName,
|
||||
}
|
||||
var renameResp s3tables.RenameTableResponse
|
||||
if err := s.execute(r, "RenameTable", renameReq, &renameResp); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, RenameTableResponse{})
|
||||
}
|
||||
|
||||
// lanceTable is the catalog's record of one Lance table.
|
||||
type lanceTable struct {
|
||||
location string
|
||||
versionToken string
|
||||
}
|
||||
|
||||
// loadLanceTable reads a table and refuses one that is not a Lance table, so a
|
||||
// Lance client never resolves an Iceberg table's location and writes over it.
|
||||
func (s *Server) loadLanceTable(r *http.Request, bucket string, ns []string, name string) (*lanceTable, error) {
|
||||
req := &s3tables.GetTableRequest{
|
||||
TableBucketARN: bucketARN(bucket),
|
||||
Namespace: ns,
|
||||
Name: name,
|
||||
}
|
||||
var resp s3tables.GetTableResponse
|
||||
if err := s.execute(r, "GetTable", req, &resp); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if resp.Format != s3tables.FormatLance {
|
||||
return nil, &s3tables.S3TablesError{
|
||||
Type: s3tables.ErrCodeNoSuchTable,
|
||||
Message: "table " + name + " is not a lance table",
|
||||
}
|
||||
}
|
||||
location := resp.MetadataLocation
|
||||
if location == "" {
|
||||
location = tableLocation(bucket, resp.Namespace, resp.Name)
|
||||
}
|
||||
return &lanceTable{location: location, versionToken: resp.VersionToken}, nil
|
||||
}
|
||||
|
||||
func (s *Server) createTable(r *http.Request, bucket string, ns []string, name, location string) error {
|
||||
req := &s3tables.CreateTableRequest{
|
||||
TableBucketARN: bucketARN(bucket),
|
||||
Namespace: ns,
|
||||
Name: name,
|
||||
Format: s3tables.FormatLance,
|
||||
// A Lance table has no metadata file, so the location the catalog stores
|
||||
// is the dataset root itself.
|
||||
MetadataLocation: location,
|
||||
}
|
||||
var resp s3tables.CreateTableResponse
|
||||
return s.execute(r, "CreateTable", req, &resp)
|
||||
}
|
||||
|
||||
// repointTable moves an existing entry to another dataset location without
|
||||
// touching either dataset's files.
|
||||
func (s *Server) repointTable(r *http.Request, bucket string, ns []string, name, location, versionToken string) error {
|
||||
req := &s3tables.UpdateTableRequest{
|
||||
TableBucketARN: bucketARN(bucket),
|
||||
Namespace: ns,
|
||||
Name: name,
|
||||
VersionToken: versionToken,
|
||||
MetadataLocation: location,
|
||||
}
|
||||
var resp s3tables.UpdateTableResponse
|
||||
return s.execute(r, "UpdateTable", req, &resp)
|
||||
}
|
||||
|
||||
// dropTableEntry removes the catalog entry and the dataset under it. Only
|
||||
// DropTable wants this; deregistering and repointing must leave the files.
|
||||
func (s *Server) dropTableEntry(r *http.Request, bucket string, ns []string, name string) error {
|
||||
req := &s3tables.DeleteTableRequest{
|
||||
TableBucketARN: bucketARN(bucket),
|
||||
Namespace: ns,
|
||||
Name: name,
|
||||
}
|
||||
return s.execute(r, "DeleteTable", req, nil)
|
||||
}
|
||||
@@ -0,0 +1,359 @@
|
||||
package lance
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest"
|
||||
)
|
||||
|
||||
// openAuthenticator stands in for a gateway with no IAM configured, which is the
|
||||
// mode the namespace falls open in.
|
||||
type openAuthenticator struct{}
|
||||
|
||||
func (openAuthenticator) AuthenticateRequest(*http.Request) (string, interface{}, s3err.ErrorCode) {
|
||||
return s3tables.DefaultAccountID, nil, s3err.ErrNone
|
||||
}
|
||||
|
||||
func (openAuthenticator) DefaultAllow() bool { return true }
|
||||
|
||||
type testHarness struct {
|
||||
router *mux.Router
|
||||
filer *s3tablestest.MemFiler
|
||||
admin *s3tables.Manager
|
||||
server *Server
|
||||
}
|
||||
|
||||
func newTestHarness(t *testing.T) *testHarness {
|
||||
t.Helper()
|
||||
filer := s3tablestest.Start(t)
|
||||
server := NewServer(s3tables.NewManagerClient(filer.Client), openAuthenticator{})
|
||||
server.SetS3Endpoint("http://127.0.0.1:8333")
|
||||
server.SetS3Region(s3tables.DefaultRegion)
|
||||
router := mux.NewRouter().SkipClean(true)
|
||||
server.RegisterRoutes(router)
|
||||
|
||||
// Table buckets are created by an operator, not by a namespace client, so
|
||||
// the harness seeds them the way the shell and admin console would.
|
||||
admin := s3tables.NewManager()
|
||||
admin.SetTrusted(true)
|
||||
return &testHarness{router: router, filer: filer, admin: admin, server: server}
|
||||
}
|
||||
|
||||
// createBucket seeds a table bucket the Lance namespace can then be pointed at.
|
||||
// An empty format strips the declaration afterwards, which is the only way to
|
||||
// get the shape a bucket made before formats existed has: one that still
|
||||
// accepts either format.
|
||||
func (h *testHarness) createBucket(t *testing.T, name, format string) {
|
||||
t.Helper()
|
||||
var resp s3tables.CreateTableBucketResponse
|
||||
declared := format
|
||||
if declared == "" {
|
||||
declared = s3tables.FormatIceberg
|
||||
}
|
||||
err := h.admin.Execute(t.Context(), s3tables.NewManagerClient(h.filer.Client), "CreateTableBucket",
|
||||
&s3tables.CreateTableBucketRequest{Name: name, Format: declared}, &resp, "")
|
||||
if err != nil {
|
||||
t.Fatalf("create table bucket %s: %v", name, err)
|
||||
}
|
||||
if format == "" {
|
||||
h.undeclareBucket(t, name)
|
||||
}
|
||||
}
|
||||
|
||||
// undeclareBucket removes a bucket's format from its metadata, ageing it back to
|
||||
// what the filer holds for a bucket created before the field existed.
|
||||
func (h *testHarness) undeclareBucket(t *testing.T, name string) {
|
||||
t.Helper()
|
||||
entry := h.filer.Get(s3tables.TablesPath, name)
|
||||
if entry == nil {
|
||||
t.Fatalf("table bucket %s is not in the filer", name)
|
||||
}
|
||||
var metadata map[string]any
|
||||
if err := json.Unmarshal(entry.Extended[s3tables.ExtendedKeyMetadata], &metadata); err != nil {
|
||||
t.Fatalf("read bucket metadata: %v", err)
|
||||
}
|
||||
delete(metadata, "format")
|
||||
updated, err := json.Marshal(metadata)
|
||||
if err != nil {
|
||||
t.Fatalf("write bucket metadata: %v", err)
|
||||
}
|
||||
extended := map[string][]byte{}
|
||||
for key, value := range entry.Extended {
|
||||
extended[key] = value
|
||||
}
|
||||
extended[s3tables.ExtendedKeyMetadata] = updated
|
||||
h.filer.Put(s3tables.TablesPath, name, extended)
|
||||
}
|
||||
|
||||
func (h *testHarness) bucketARN(t *testing.T, name string) string {
|
||||
t.Helper()
|
||||
arn, err := s3tables.BuildBucketARN(s3tables.DefaultRegion, s3tables.DefaultAccountID, name)
|
||||
if err != nil {
|
||||
t.Fatalf("build arn: %v", err)
|
||||
}
|
||||
return arn
|
||||
}
|
||||
|
||||
func (h *testHarness) do(t *testing.T, method, target, body string) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
var reader *strings.Reader
|
||||
if body == "" {
|
||||
reader = strings.NewReader("")
|
||||
} else {
|
||||
reader = strings.NewReader(body)
|
||||
}
|
||||
req := httptest.NewRequest(method, target, reader)
|
||||
recorder := httptest.NewRecorder()
|
||||
h.router.ServeHTTP(recorder, req)
|
||||
return recorder
|
||||
}
|
||||
|
||||
func (h *testHarness) mustDo(t *testing.T, method, target, body string, want int) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
recorder := h.do(t, method, target, body)
|
||||
if recorder.Code != want {
|
||||
t.Fatalf("%s %s = %d (%s), want %d", method, target, recorder.Code, recorder.Body.String(), want)
|
||||
}
|
||||
return recorder
|
||||
}
|
||||
|
||||
func decode[T any](t *testing.T, recorder *httptest.ResponseRecorder) T {
|
||||
t.Helper()
|
||||
var out T
|
||||
if err := json.Unmarshal(recorder.Body.Bytes(), &out); err != nil {
|
||||
t.Fatalf("decode %s: %v", recorder.Body.String(), err)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// The lifecycle a Lance client drives: create the namespace, declare the table
|
||||
// before any data exists, resolve it to a location, then deregister and bring it
|
||||
// back by registering the same location.
|
||||
func TestTableLifecycle(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
|
||||
h.createBucket(t, "analytics", s3tables.FormatLance)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
|
||||
|
||||
declared := decode[DeclareTableResponse](t,
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/declare", `{}`, http.StatusOK))
|
||||
if declared.Location != "s3://analytics/sales/orders" {
|
||||
t.Fatalf("declared location = %q", declared.Location)
|
||||
}
|
||||
// The endpoint is plaintext, so a client that does not get allow_http fails
|
||||
// with what looks like a credential error.
|
||||
if declared.StorageOptions["aws_endpoint"] != "http://127.0.0.1:8333" ||
|
||||
declared.StorageOptions["allow_http"] != "true" {
|
||||
t.Fatalf("storage options = %v", declared.StorageOptions)
|
||||
}
|
||||
|
||||
described := decode[DescribeTableResponse](t,
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/describe?check_declared=true", `{}`, http.StatusOK))
|
||||
if described.Location != declared.Location {
|
||||
t.Fatalf("described location = %q, want %q", described.Location, declared.Location)
|
||||
}
|
||||
if described.IsOnlyDeclared == nil || !*described.IsOnlyDeclared {
|
||||
t.Fatalf("is_only_declared = %v, want true for a table with no data", described.IsOnlyDeclared)
|
||||
}
|
||||
|
||||
listed := decode[ListTablesResponse](t,
|
||||
h.mustDo(t, http.MethodGet, "/v1/namespace/analytics$sales/table/list", "", http.StatusOK))
|
||||
if len(listed.Tables) != 1 || listed.Tables[0] != "analytics$sales$orders" {
|
||||
t.Fatalf("listed tables = %v, want the full identifier", listed.Tables)
|
||||
}
|
||||
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/exists", `{}`, http.StatusOK)
|
||||
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/deregister", `{}`, http.StatusOK)
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/exists", `{}`, http.StatusNotFound)
|
||||
afterDrop := decode[ListTablesResponse](t,
|
||||
h.mustDo(t, http.MethodGet, "/v1/namespace/analytics$sales/table/list", "", http.StatusOK))
|
||||
if len(afterDrop.Tables) != 0 {
|
||||
t.Fatalf("deregistered table still listed: %v", afterDrop.Tables)
|
||||
}
|
||||
// Deregistering preserves the data. The catalog entry is the dataset
|
||||
// directory, so dropping it would take the dataset with it.
|
||||
if h.filer.Get(s3tables.GetNamespacePath("analytics", "sales"), "orders") == nil {
|
||||
t.Fatal("deregister deleted the dataset directory")
|
||||
}
|
||||
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/register",
|
||||
`{"location":"s3://analytics/sales/orders"}`, http.StatusOK)
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/exists", `{}`, http.StatusOK)
|
||||
|
||||
// Dropping is the operation that does remove the data.
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/drop", `{}`, http.StatusOK)
|
||||
if h.filer.Get(s3tables.GetNamespacePath("analytics", "sales"), "orders") != nil {
|
||||
t.Fatal("drop left the dataset directory behind")
|
||||
}
|
||||
}
|
||||
|
||||
// Repointing a registered name at another dataset must not take the dataset it
|
||||
// used to name with it.
|
||||
func TestRegisterOverwriteKeepsTheOldDataset(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
h.createBucket(t, "analytics", s3tables.FormatLance)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/declare", `{}`, http.StatusOK)
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$archive/declare", `{}`, http.StatusOK)
|
||||
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/register",
|
||||
`{"location":"s3://analytics/sales/archive","mode":"Overwrite"}`, http.StatusOK)
|
||||
|
||||
described := decode[DescribeTableResponse](t,
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/describe", `{}`, http.StatusOK))
|
||||
if described.Location != "s3://analytics/sales/archive" {
|
||||
t.Fatalf("location after repointing = %q", described.Location)
|
||||
}
|
||||
if h.filer.Get(s3tables.GetNamespacePath("analytics", "sales"), "orders") == nil {
|
||||
t.Fatal("repointing deleted the dataset the name used to hold")
|
||||
}
|
||||
}
|
||||
|
||||
// A required list field answers empty rather than null, which a generated
|
||||
// client may decode differently.
|
||||
func TestListAllTablesIsNeverNull(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
h.createBucket(t, "analytics", s3tables.FormatLance)
|
||||
|
||||
body := h.mustDo(t, http.MethodGet, "/v1/table", "", http.StatusOK).Body.String()
|
||||
if strings.Contains(body, `"tables":null`) {
|
||||
t.Fatalf("ListAllTables returned null for a required field: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNamespaceListing(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
h.createBucket(t, "analytics", s3tables.FormatLance)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$finance/create", `{}`, http.StatusOK)
|
||||
|
||||
roots := decode[ListNamespacesResponse](t,
|
||||
h.mustDo(t, http.MethodGet, "/v1/namespace/$/list", "", http.StatusOK))
|
||||
if len(roots.Namespaces) != 1 || roots.Namespaces[0] != "analytics" {
|
||||
t.Fatalf("root listing = %v, want the table buckets", roots.Namespaces)
|
||||
}
|
||||
|
||||
children := decode[ListNamespacesResponse](t,
|
||||
h.mustDo(t, http.MethodGet, "/v1/namespace/analytics/list", "", http.StatusOK))
|
||||
if len(children.Namespaces) != 2 {
|
||||
t.Fatalf("bucket listing = %v, want two namespaces", children.Namespaces)
|
||||
}
|
||||
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/exists", `{}`, http.StatusOK)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$nope/exists", `{}`, http.StatusNotFound)
|
||||
|
||||
// Fail mode reports a missing namespace as 400 on this operation, which is
|
||||
// what the spec asks for; Skip reports success.
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$nope/drop", `{}`, http.StatusBadRequest)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$nope/drop", `{"mode":"Skip"}`, http.StatusNoContent)
|
||||
}
|
||||
|
||||
// Storage flattens a namespace's parts, so creating a$b without a would leave
|
||||
// an intermediate that listing derives from the name and describe denies
|
||||
// exists. The spec asks for NamespaceNotFound, which keeps them consistent.
|
||||
func TestCreateNamespaceRequiresItsParent(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
h.createBucket(t, "analytics", s3tables.FormatLance)
|
||||
|
||||
recorder := h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$missing$child/create", `{}`, http.StatusNotFound)
|
||||
if got := decode[errorResponse](t, recorder); got.Code != codeNamespaceNotFound {
|
||||
t.Fatalf("error code = %d, want %d", got.Code, codeNamespaceNotFound)
|
||||
}
|
||||
|
||||
// With the parent in place the child is fine, and both are then listed and
|
||||
// describable.
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$parent/create", `{}`, http.StatusOK)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$parent$child/create", `{}`, http.StatusOK)
|
||||
|
||||
listed := decode[ListNamespacesResponse](t,
|
||||
h.mustDo(t, http.MethodGet, "/v1/namespace/analytics/list", "", http.StatusOK))
|
||||
for _, name := range listed.Namespaces {
|
||||
path := "/v1/namespace/analytics$" + name
|
||||
h.mustDo(t, http.MethodPost, path+"/exists", `{}`, http.StatusOK)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateNamespaceModes(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
h.createBucket(t, "analytics", s3tables.FormatLance)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
|
||||
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusConflict)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{"mode":"ExistOk"}`, http.StatusOK)
|
||||
// snake_case is the other spelling the spec accepts.
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{"mode":"exist_ok"}`, http.StatusOK)
|
||||
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/$/create", `{}`, http.StatusBadRequest)
|
||||
}
|
||||
|
||||
// A Lance client must never resolve an Iceberg table's location, or it would
|
||||
// write a dataset over a table another engine owns.
|
||||
func TestIcebergTablesAreInvisible(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
// Undeclared, because a bucket that declares one format cannot hold the
|
||||
// other - and mixing is exactly what this test needs to prove is hidden.
|
||||
h.createBucket(t, "analytics", "")
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$vectors/declare", `{}`, http.StatusOK)
|
||||
|
||||
manager := h.admin
|
||||
var created s3tables.CreateTableResponse
|
||||
err := manager.Execute(t.Context(), s3tables.NewManagerClient(h.filer.Client), "CreateTable", &s3tables.CreateTableRequest{
|
||||
TableBucketARN: h.bucketARN(t, "analytics"),
|
||||
Namespace: []string{"sales"},
|
||||
Name: "ledger",
|
||||
Format: s3tables.FormatIceberg,
|
||||
MetadataLocation: "s3://analytics/sales/ledger/metadata/v1.metadata.json",
|
||||
}, &created, "")
|
||||
if err != nil {
|
||||
t.Fatalf("create iceberg table: %v", err)
|
||||
}
|
||||
|
||||
listed := decode[ListTablesResponse](t,
|
||||
h.mustDo(t, http.MethodGet, "/v1/namespace/analytics$sales/table/list", "", http.StatusOK))
|
||||
if len(listed.Tables) != 1 || listed.Tables[0] != "analytics$sales$vectors" {
|
||||
t.Fatalf("listing = %v, want only the lance table", listed.Tables)
|
||||
}
|
||||
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$ledger/describe", `{}`, http.StatusNotFound)
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$ledger/exists", `{}`, http.StatusNotFound)
|
||||
|
||||
// Declaring over the Iceberg table must not quietly succeed and hand the
|
||||
// Lance client a directory another format owns.
|
||||
recorder := h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$ledger/declare", `{}`, http.StatusConflict)
|
||||
if got := decode[errorResponse](t, recorder); got.Code != codeTableAlreadyExists {
|
||||
t.Fatalf("error code = %d, want %d", got.Code, codeTableAlreadyExists)
|
||||
}
|
||||
}
|
||||
|
||||
// The route and the body naming different objects is a bad request, not a silent
|
||||
// preference for one of them.
|
||||
func TestRouteAndBodyMustAgree(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
h.createBucket(t, "analytics", s3tables.FormatLance)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
|
||||
|
||||
recorder := h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/declare",
|
||||
`{"id":["analytics","sales","other"]}`, http.StatusBadRequest)
|
||||
if got := decode[errorResponse](t, recorder); got.Code != codeInvalidInput {
|
||||
t.Fatalf("error code = %d, want %d", got.Code, codeInvalidInput)
|
||||
}
|
||||
}
|
||||
|
||||
// The data plane needs Lance format support that does not exist in Go, so it
|
||||
// answers with the spec's own Unsupported code rather than a bare 404.
|
||||
func TestDataPlaneIsUnsupported(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
recorder := h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/query", `{}`, http.StatusNotImplemented)
|
||||
if got := decode[errorResponse](t, recorder); got.Code != codeUnsupported {
|
||||
t.Fatalf("error code = %d, want %d", got.Code, codeUnsupported)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
package lance
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// defaultDelimiter joins the parts of a Lance string identifier when the caller
|
||||
// does not pass ?delimiter=. An id equal to the delimiter is the root namespace,
|
||||
// so /v1/namespace/$/list lists the root's children.
|
||||
const defaultDelimiter = "$"
|
||||
|
||||
// identifier is a Lance object identifier: zero parts is the root namespace, one
|
||||
// part names a table bucket, and the rest are namespace parts with a table name
|
||||
// last. The mapping onto storage is bucket / namespace / table, which is the
|
||||
// three-level shape Lance clients already use.
|
||||
type identifier []string
|
||||
|
||||
func requestDelimiter(r *http.Request) string {
|
||||
if d := r.URL.Query().Get("delimiter"); d != "" {
|
||||
return d
|
||||
}
|
||||
return defaultDelimiter
|
||||
}
|
||||
|
||||
// parseIdentifier decodes the {id} route variable. Empty parts are rejected
|
||||
// rather than dropped so that "a$$b" cannot silently resolve to "a$b".
|
||||
func parseIdentifier(encoded, delimiter string) (identifier, error) {
|
||||
if encoded == "" || encoded == delimiter {
|
||||
return nil, nil
|
||||
}
|
||||
parts := strings.Split(encoded, delimiter)
|
||||
for _, part := range parts {
|
||||
if part == "" {
|
||||
return nil, fmt.Errorf("identifier %q has an empty part", encoded)
|
||||
}
|
||||
}
|
||||
return parts, nil
|
||||
}
|
||||
|
||||
func (id identifier) String(delimiter string) string {
|
||||
if len(id) == 0 {
|
||||
return delimiter
|
||||
}
|
||||
return strings.Join(id, delimiter)
|
||||
}
|
||||
|
||||
// namespace splits a namespace identifier into the table bucket and the parts of
|
||||
// the namespace inside it. The root and a bare bucket both return an empty
|
||||
// namespace, so callers check the bucket to tell them apart.
|
||||
func (id identifier) namespace() (bucket string, ns []string) {
|
||||
if len(id) == 0 {
|
||||
return "", nil
|
||||
}
|
||||
return id[0], id[1:]
|
||||
}
|
||||
|
||||
// table splits a table identifier. A table needs a bucket, at least one
|
||||
// namespace part and a name, because storage has no unnamespaced tables.
|
||||
func (id identifier) table() (bucket string, ns []string, name string, err error) {
|
||||
if len(id) < 3 {
|
||||
return "", nil, "", fmt.Errorf("table identifier needs a bucket, a namespace and a name")
|
||||
}
|
||||
return id[0], id[1 : len(id)-1], id[len(id)-1], nil
|
||||
}
|
||||
|
||||
// matchesBody reports whether an identifier carried in the request body agrees
|
||||
// with the one in the route. The spec requires 400 when both are present and
|
||||
// differ, and requires the route to win when the body omits it.
|
||||
func (id identifier) matchesBody(body []string) bool {
|
||||
if len(body) == 0 {
|
||||
return true
|
||||
}
|
||||
if len(body) != len(id) {
|
||||
return false
|
||||
}
|
||||
for i := range body {
|
||||
if body[i] != id[i] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
package lance
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestParseIdentifier(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
encoded string
|
||||
delimiter string
|
||||
want []string
|
||||
wantErr bool
|
||||
}{
|
||||
{name: "root is the delimiter", encoded: "$", delimiter: "$"},
|
||||
{name: "empty is the root", encoded: "", delimiter: "$"},
|
||||
{name: "bucket", encoded: "analytics", delimiter: "$", want: []string{"analytics"}},
|
||||
{name: "namespace", encoded: "analytics$sales", delimiter: "$", want: []string{"analytics", "sales"}},
|
||||
{name: "table", encoded: "analytics$sales$orders", delimiter: "$", want: []string{"analytics", "sales", "orders"}},
|
||||
{name: "custom delimiter", encoded: "a.b.c", delimiter: ".", want: []string{"a", "b", "c"}},
|
||||
{name: "empty part is rejected", encoded: "a$$b", delimiter: "$", wantErr: true},
|
||||
{name: "trailing delimiter is rejected", encoded: "a$", delimiter: "$", wantErr: true},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
got, err := parseIdentifier(c.encoded, c.delimiter)
|
||||
if c.wantErr {
|
||||
if err == nil {
|
||||
t.Fatalf("parseIdentifier(%q) error = nil, want an error", c.encoded)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("parseIdentifier(%q) error = %v", c.encoded, err)
|
||||
}
|
||||
if len(got) != len(c.want) {
|
||||
t.Fatalf("parseIdentifier(%q) = %v, want %v", c.encoded, got, c.want)
|
||||
}
|
||||
for i := range got {
|
||||
if got[i] != c.want[i] {
|
||||
t.Fatalf("parseIdentifier(%q) = %v, want %v", c.encoded, got, c.want)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentifierNamespace(t *testing.T) {
|
||||
bucket, ns := identifier(nil).namespace()
|
||||
if bucket != "" || len(ns) != 0 {
|
||||
t.Fatalf("root namespace() = %q %v, want empty", bucket, ns)
|
||||
}
|
||||
|
||||
bucket, ns = identifier{"analytics"}.namespace()
|
||||
if bucket != "analytics" || len(ns) != 0 {
|
||||
t.Fatalf("bucket namespace() = %q %v", bucket, ns)
|
||||
}
|
||||
|
||||
bucket, ns = identifier{"analytics", "sales", "eu"}.namespace()
|
||||
if bucket != "analytics" || len(ns) != 2 || ns[0] != "sales" || ns[1] != "eu" {
|
||||
t.Fatalf("nested namespace() = %q %v", bucket, ns)
|
||||
}
|
||||
}
|
||||
|
||||
// A table always needs a bucket, a namespace and a name: storage has no
|
||||
// unnamespaced tables, so a two-part identifier is a client error rather than a
|
||||
// table at the top of a bucket.
|
||||
func TestIdentifierTable(t *testing.T) {
|
||||
if _, _, _, err := (identifier{"analytics", "orders"}).table(); err == nil {
|
||||
t.Fatal("table() on a two-part identifier error = nil, want an error")
|
||||
}
|
||||
|
||||
bucket, ns, name, err := identifier{"analytics", "sales", "eu", "orders"}.table()
|
||||
if err != nil {
|
||||
t.Fatalf("table() error = %v", err)
|
||||
}
|
||||
if bucket != "analytics" || name != "orders" || len(ns) != 2 || ns[0] != "sales" || ns[1] != "eu" {
|
||||
t.Fatalf("table() = %q %v %q", bucket, ns, name)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentifierMatchesBody(t *testing.T) {
|
||||
id := identifier{"a", "b", "c"}
|
||||
if !id.matchesBody(nil) {
|
||||
t.Fatal("an absent body identifier must defer to the route")
|
||||
}
|
||||
if !id.matchesBody([]string{"a", "b", "c"}) {
|
||||
t.Fatal("an equal body identifier must match")
|
||||
}
|
||||
if id.matchesBody([]string{"a", "b"}) {
|
||||
t.Fatal("a shorter body identifier must not match")
|
||||
}
|
||||
if id.matchesBody([]string{"a", "b", "d"}) {
|
||||
t.Fatal("a different body identifier must not match")
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentifierString(t *testing.T) {
|
||||
if got := identifier(nil).String("$"); got != "$" {
|
||||
t.Fatalf("root String() = %q, want %q", got, "$")
|
||||
}
|
||||
if got := (identifier{"a", "b"}).String("$"); got != "a$b" {
|
||||
t.Fatalf("String() = %q, want %q", got, "a$b")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
package lance
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// Marker files the Lance Directory Catalog reads. Writing them beside the
|
||||
// dataset keeps a client that bypasses this namespace, and lists the storage
|
||||
// prefix directly, seeing the same table states the catalog reports.
|
||||
const (
|
||||
reservedMarker = ".lance-reserved"
|
||||
deregisteredMarker = ".lance-deregistered"
|
||||
versionsDir = "_versions"
|
||||
)
|
||||
|
||||
// datasetDir maps a table location onto the filer directory holding it.
|
||||
func datasetDir(location string) string {
|
||||
return s3tables.TableDataDirFromMetadataLocation(location)
|
||||
}
|
||||
|
||||
func (s *Server) entryExists(r *http.Request, dir, name string) (bool, error) {
|
||||
found := false
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
_, lookupErr := filer_pb.LookupEntry(r.Context(), client, &filer_pb.LookupDirectoryEntryRequest{
|
||||
Directory: dir,
|
||||
Name: name,
|
||||
})
|
||||
if lookupErr == filer_pb.ErrNotFound {
|
||||
return nil
|
||||
}
|
||||
if lookupErr != nil {
|
||||
return lookupErr
|
||||
}
|
||||
found = true
|
||||
return nil
|
||||
})
|
||||
return found, err
|
||||
}
|
||||
|
||||
// writeMarker drops a zero-length marker into the dataset directory. The
|
||||
// directory has to exist already: a table declared through this namespace has
|
||||
// one, and a table registered at a location that does not yet exist has no
|
||||
// storage to mark.
|
||||
func (s *Server) writeMarker(r *http.Request, location, name string) error {
|
||||
dir := datasetDir(location)
|
||||
if dir == "" {
|
||||
return nil
|
||||
}
|
||||
now := time.Now().Unix()
|
||||
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
return filer_pb.CreateEntry(r.Context(), client, &filer_pb.CreateEntryRequest{
|
||||
Directory: dir,
|
||||
Entry: &filer_pb.Entry{
|
||||
Name: name,
|
||||
Attributes: &filer_pb.FuseAttributes{
|
||||
Mtime: now,
|
||||
Crtime: now,
|
||||
FileMode: uint32(0644),
|
||||
},
|
||||
},
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) removeMarker(r *http.Request, location, name string) error {
|
||||
dir := datasetDir(location)
|
||||
if dir == "" {
|
||||
return nil
|
||||
}
|
||||
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
return filer_pb.DoRemove(r.Context(), client, dir, name, true, false, true, false, nil)
|
||||
})
|
||||
}
|
||||
|
||||
// datasetState reports the two things the spec asks about a table's storage:
|
||||
// whether it has been deregistered, and whether it holds data yet. A dataset
|
||||
// holds data once it has a version manifest, which is how a directory catalog
|
||||
// decides the same question.
|
||||
func (s *Server) datasetState(r *http.Request, location string) (deregistered, hasData bool, err error) {
|
||||
dir := datasetDir(location)
|
||||
if dir == "" {
|
||||
return false, false, nil
|
||||
}
|
||||
if deregistered, err = s.entryExists(r, dir, deregisteredMarker); err != nil {
|
||||
return false, false, err
|
||||
}
|
||||
hasData, err = s.entryExists(r, dir, versionsDir)
|
||||
return deregistered, hasData, err
|
||||
}
|
||||
@@ -0,0 +1,188 @@
|
||||
package lance
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// FilerClient provides access to the filer for storage operations.
|
||||
type FilerClient interface {
|
||||
WithFilerClient(streamingMode bool, fn func(client filer_pb.SeaweedFilerClient) error) error
|
||||
}
|
||||
|
||||
type S3Authenticator interface {
|
||||
AuthenticateRequest(r *http.Request) (string, interface{}, s3err.ErrorCode)
|
||||
DefaultAllow() bool
|
||||
}
|
||||
|
||||
// VendedCredentials are short-lived S3 credentials scoped to one table.
|
||||
type VendedCredentials struct {
|
||||
AccessKeyID string
|
||||
SecretAccessKey string
|
||||
SessionToken string
|
||||
Expiration time.Time
|
||||
}
|
||||
|
||||
// CredentialVendor mints credentials limited to a single table's prefix for a
|
||||
// caller the catalog has already authenticated and authorized. A nil result with
|
||||
// no error means the deployment has vending switched off.
|
||||
type CredentialVendor interface {
|
||||
VendTableCredentials(ctx context.Context, principal, bucket, prefix string) (*VendedCredentials, error)
|
||||
}
|
||||
|
||||
// Server implements the Lance Namespace REST spec.
|
||||
type Server struct {
|
||||
filerClient FilerClient
|
||||
tablesManager *s3tables.Manager
|
||||
authenticator S3Authenticator
|
||||
credentialVendor CredentialVendor
|
||||
s3Endpoint string
|
||||
s3Region string
|
||||
}
|
||||
|
||||
// NewServer creates a Lance namespace server over the given filer.
|
||||
func NewServer(filerClient FilerClient, authenticator S3Authenticator) *Server {
|
||||
manager := s3tables.NewManager()
|
||||
// Mirror the S3 port: fall open by default only when the gateway itself is
|
||||
// open, so an authenticated caller still passes the normal permission check.
|
||||
if authenticator != nil {
|
||||
manager.SetDefaultAllow(authenticator.DefaultAllow())
|
||||
}
|
||||
return &Server{
|
||||
filerClient: filerClient,
|
||||
tablesManager: manager,
|
||||
authenticator: authenticator,
|
||||
}
|
||||
}
|
||||
|
||||
// SetCredentialVendor enables storage_options credential vending for clients
|
||||
// that ask for it with vend_credentials.
|
||||
func (s *Server) SetCredentialVendor(vendor CredentialVendor) {
|
||||
s.credentialVendor = vendor
|
||||
}
|
||||
|
||||
// SetS3Endpoint configures the S3 endpoint advertised in storage_options so a
|
||||
// client can reach the dataset without separately discovering the S3 address.
|
||||
func (s *Server) SetS3Endpoint(endpoint string) {
|
||||
s.s3Endpoint = endpoint
|
||||
}
|
||||
|
||||
// SetS3Region configures the region advertised in storage_options.
|
||||
func (s *Server) SetS3Region(region string) {
|
||||
s.s3Region = region
|
||||
}
|
||||
|
||||
// RegisterRoutes registers the Lance Namespace REST routes.
|
||||
//
|
||||
// The spec puts the identifier in the path rather than the body so a reverse
|
||||
// proxy can route and authorize without deserializing the request.
|
||||
func (s *Server) RegisterRoutes(router *mux.Router) {
|
||||
router.Use(loggingMiddleware)
|
||||
|
||||
router.HandleFunc("/v1/namespace/{id}/create", s.Auth(s.handleCreateNamespace)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/namespace/{id}/list", s.Auth(s.handleListNamespaces)).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/namespace/{id}/describe", s.Auth(s.handleDescribeNamespace)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/namespace/{id}/drop", s.Auth(s.handleDropNamespace)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/namespace/{id}/exists", s.Auth(s.handleNamespaceExists)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/namespace/{id}/table/list", s.Auth(s.handleListTables)).Methods(http.MethodGet)
|
||||
|
||||
router.HandleFunc("/v1/table", s.Auth(s.handleListAllTables)).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/table/{id}/declare", s.Auth(s.handleDeclareTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/table/{id}/describe", s.Auth(s.handleDescribeTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/table/{id}/exists", s.Auth(s.handleTableExists)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/table/{id}/register", s.Auth(s.handleRegisterTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/table/{id}/deregister", s.Auth(s.handleDeregisterTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/table/{id}/drop", s.Auth(s.handleDropTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/table/{id}/rename", s.Auth(s.handleRenameTable)).Methods(http.MethodPost)
|
||||
|
||||
// The data plane needs Lance format support that does not exist in Go. Say
|
||||
// so with the spec's own code instead of returning a bare 404.
|
||||
for _, action := range []string{"create", "insert", "merge_insert", "update", "delete",
|
||||
"query", "count_rows", "explain_plan", "analyze_plan", "restore",
|
||||
"add_columns", "alter_columns", "drop_columns", "backfill_column",
|
||||
"create_index", "create_scalar_index", "stats", "schema_metadata/update"} {
|
||||
router.HandleFunc("/v1/table/{id}/"+action, s.Auth(s.handleUnsupported)).Methods(http.MethodPost)
|
||||
}
|
||||
// Version ops exist in the spec for stores that cannot order commits
|
||||
// themselves. Ours can: a Lance commit is a put-if-not-exists, and this S3
|
||||
// evaluates that precondition at the object's owner filer under a per-path
|
||||
// lock, so the dataset keeps its own version history and the catalog stays
|
||||
// out of the commit path.
|
||||
for _, action := range []string{
|
||||
"version/create", "version/list", "version/describe", "version/delete",
|
||||
"index/list", "tags/list", "tags/version", "tags/create", "tags/delete", "tags/update",
|
||||
"branches/list", "branches/create", "branches/delete"} {
|
||||
router.HandleFunc("/v1/table/{id}/"+action, s.Auth(s.handleUnsupported)).Methods(http.MethodPost)
|
||||
}
|
||||
|
||||
router.PathPrefix("/").HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
glog.V(2).Infof("lance: no route for %s %s", r.Method, r.RequestURI)
|
||||
writeError(w, r, http.StatusNotFound, codeUnsupported, "no such operation")
|
||||
})
|
||||
|
||||
glog.V(2).Infof("Registered Lance Namespace routes")
|
||||
}
|
||||
|
||||
func (s *Server) handleUnsupported(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, r, http.StatusNotImplemented, codeUnsupported,
|
||||
"this namespace records table metadata only; run data operations through a Lance client against the table location")
|
||||
}
|
||||
|
||||
func loggingMiddleware(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
glog.V(2).Infof("lance request: %s %s from %s", r.Method, r.RequestURI, r.RemoteAddr)
|
||||
next.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
// Auth authenticates the caller and puts the identity in the request context.
|
||||
// The Lance spec maps identity onto the same headers the S3 authenticator
|
||||
// already understands, so SigV4 and bearer tokens both keep working.
|
||||
func (s *Server) Auth(handler http.HandlerFunc) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if s.authenticator == nil {
|
||||
writeError(w, r, http.StatusUnauthorized, codeUnauthenticated, "authentication required")
|
||||
return
|
||||
}
|
||||
|
||||
identityName, identity, errCode := s.authenticator.AuthenticateRequest(r)
|
||||
if errCode != s3err.ErrNone {
|
||||
if !s.authenticator.DefaultAllow() {
|
||||
apiErr := s3err.GetAPIError(errCode)
|
||||
code := codeInternal
|
||||
switch apiErr.HTTPStatusCode {
|
||||
case http.StatusForbidden:
|
||||
code = codePermissionDenied
|
||||
case http.StatusUnauthorized:
|
||||
code = codeUnauthenticated
|
||||
case http.StatusBadRequest:
|
||||
code = codeInvalidInput
|
||||
}
|
||||
writeError(w, r, apiErr.HTTPStatusCode, code, apiErr.Description)
|
||||
return
|
||||
}
|
||||
glog.V(2).Infof("lance: authentication failed (%v) but the gateway is open, proceeding", errCode)
|
||||
}
|
||||
|
||||
if identityName != "" || identity != nil {
|
||||
ctx := r.Context()
|
||||
if identityName != "" {
|
||||
ctx = s3_constants.SetIdentityNameInContext(ctx, identityName)
|
||||
}
|
||||
if identity != nil {
|
||||
ctx = s3_constants.SetIdentityInContext(ctx, identity)
|
||||
}
|
||||
r = r.WithContext(ctx)
|
||||
}
|
||||
|
||||
handler(w, r)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,158 @@
|
||||
package lance
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
const defaultPageSize = 1000
|
||||
|
||||
// execute runs one S3 Tables operation as the request's authenticated caller.
|
||||
func (s *Server) execute(r *http.Request, operation string, req, resp interface{}) error {
|
||||
identityName := s3_constants.GetIdentityNameFromContext(r)
|
||||
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
return s.tablesManager.Execute(r.Context(), s3tables.NewManagerClient(client), operation, req, resp, identityName)
|
||||
})
|
||||
}
|
||||
|
||||
func bucketARN(bucket string) string {
|
||||
arn, _ := s3tables.BuildBucketARN(s3tables.DefaultRegion, s3_constants.AccountAdminId, bucket)
|
||||
return arn
|
||||
}
|
||||
|
||||
// maxRequestBody bounds what one call can make the catalog hold. Every request
|
||||
// this surface takes is a small JSON envelope; the largest carries a set of
|
||||
// properties, not data.
|
||||
const maxRequestBody = 4 << 20
|
||||
|
||||
// decodeBody reads an optional JSON request body. Every Lance operation carries
|
||||
// one, but the fields that matter are also in the route, so an empty body is
|
||||
// not an error.
|
||||
func decodeBody(r *http.Request, into interface{}) error {
|
||||
body, err := io.ReadAll(io.LimitReader(r.Body, maxRequestBody+1))
|
||||
if err != nil {
|
||||
return fmt.Errorf("read request body: %w", err)
|
||||
}
|
||||
if len(body) > maxRequestBody {
|
||||
return fmt.Errorf("request body is larger than %d bytes", maxRequestBody)
|
||||
}
|
||||
if len(strings.TrimSpace(string(body))) == 0 {
|
||||
return nil
|
||||
}
|
||||
if err := json.Unmarshal(body, into); err != nil {
|
||||
return fmt.Errorf("invalid request body: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// routeIdentifier parses the {id} route variable, writing the error response
|
||||
// itself when the identifier is malformed.
|
||||
func routeIdentifier(w http.ResponseWriter, r *http.Request) (identifier, string, bool) {
|
||||
delimiter := requestDelimiter(r)
|
||||
id, err := parseIdentifier(mux.Vars(r)["id"], delimiter)
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return nil, delimiter, false
|
||||
}
|
||||
return id, delimiter, true
|
||||
}
|
||||
|
||||
// checkBodyIdentifier enforces the spec rule that a route and a body naming
|
||||
// different objects is a bad request rather than a silent preference.
|
||||
func checkBodyIdentifier(w http.ResponseWriter, r *http.Request, id identifier, body []string) bool {
|
||||
if id.matchesBody(body) {
|
||||
return true
|
||||
}
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput,
|
||||
"the identifier in the request body does not match the one in the route")
|
||||
return false
|
||||
}
|
||||
|
||||
func pageSize(r *http.Request) int {
|
||||
if raw := r.URL.Query().Get("limit"); raw != "" {
|
||||
if parsed, err := strconv.Atoi(raw); err == nil && parsed > 0 {
|
||||
return parsed
|
||||
}
|
||||
}
|
||||
return defaultPageSize
|
||||
}
|
||||
|
||||
func boolQuery(r *http.Request, name string) bool {
|
||||
value, err := strconv.ParseBool(r.URL.Query().Get(name))
|
||||
return err == nil && value
|
||||
}
|
||||
|
||||
// wants resolves a tri-state request flag: the query parameter the REST spec
|
||||
// adds, else the body field, else the implementation's own choice.
|
||||
func wants(r *http.Request, name string, body *bool) bool {
|
||||
if boolQuery(r, name) {
|
||||
return true
|
||||
}
|
||||
return body != nil && *body
|
||||
}
|
||||
|
||||
// tableLocation is where a table's dataset lives when the caller does not name
|
||||
// a location. It mirrors the Iceberg catalog's layout so both catalogs put a
|
||||
// table of the same name in the same place.
|
||||
func tableLocation(bucket string, ns []string, name string) string {
|
||||
return fmt.Sprintf("s3://%s/%s/%s", bucket, strings.Join(ns, "."), name)
|
||||
}
|
||||
|
||||
// storageOptions builds the object_store settings a Lance client needs to reach
|
||||
// the dataset. The key names are the aws_-prefixed forms Lance clients pass
|
||||
// through to object_store.
|
||||
func (s *Server) storageOptions(r *http.Request, bucket, location string, vend bool) (map[string]string, error) {
|
||||
options := map[string]string{}
|
||||
if s.s3Endpoint != "" {
|
||||
options["aws_endpoint"] = s.s3Endpoint
|
||||
if strings.HasPrefix(s.s3Endpoint, "http://") {
|
||||
// object_store refuses a plaintext endpoint unless told to allow it,
|
||||
// and the resulting failure reads like a credential problem.
|
||||
options["allow_http"] = "true"
|
||||
}
|
||||
}
|
||||
if s.s3Region != "" {
|
||||
options["aws_region"] = s.s3Region
|
||||
}
|
||||
|
||||
if !vend || s.credentialVendor == nil {
|
||||
return options, nil
|
||||
}
|
||||
|
||||
principal := s3_constants.GetIdentityNameFromContext(r)
|
||||
credentials, err := s.credentialVendor.VendTableCredentials(r.Context(), principal, bucket, locationPrefix(location))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if credentials == nil {
|
||||
return options, nil
|
||||
}
|
||||
options["aws_access_key_id"] = credentials.AccessKeyID
|
||||
options["aws_secret_access_key"] = credentials.SecretAccessKey
|
||||
if credentials.SessionToken != "" {
|
||||
options["aws_session_token"] = credentials.SessionToken
|
||||
}
|
||||
if !credentials.Expiration.IsZero() {
|
||||
options["expires_at_millis"] = strconv.FormatInt(credentials.Expiration.UnixMilli(), 10)
|
||||
}
|
||||
return options, nil
|
||||
}
|
||||
|
||||
// locationPrefix strips the s3://bucket/ part of a location, leaving the key
|
||||
// prefix a credential is scoped to.
|
||||
func locationPrefix(location string) string {
|
||||
trimmed := strings.TrimPrefix(location, "s3://")
|
||||
if _, prefix, found := strings.Cut(trimmed, "/"); found {
|
||||
return prefix
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,149 @@
|
||||
// Package lance serves the Lance Namespace REST spec over the same table
|
||||
// buckets the Iceberg REST catalog uses. A Lance table is a catalog entry with
|
||||
// format LANCE: the namespace records where the dataset lives and vends
|
||||
// credentials for it, and the Lance client owns everything under that location.
|
||||
package lance
|
||||
|
||||
// Modes shared by create and register. The spec matches them case-insensitively
|
||||
// and accepts both PascalCase and snake_case.
|
||||
const (
|
||||
modeCreate = "create"
|
||||
modeExistOk = "existok"
|
||||
modeOverwrite = "overwrite"
|
||||
modeFail = "fail"
|
||||
modeSkip = "skip"
|
||||
|
||||
behaviorRestrict = "restrict"
|
||||
behaviorCascade = "cascade"
|
||||
)
|
||||
|
||||
type CreateNamespaceRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
Mode string `json:"mode,omitempty"`
|
||||
Properties map[string]string `json:"properties,omitempty"`
|
||||
}
|
||||
|
||||
type CreateNamespaceResponse struct {
|
||||
Properties map[string]string `json:"properties"`
|
||||
}
|
||||
|
||||
type ListNamespacesResponse struct {
|
||||
Namespaces []string `json:"namespaces"`
|
||||
PageToken string `json:"page_token,omitempty"`
|
||||
}
|
||||
|
||||
type DescribeNamespaceRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
}
|
||||
|
||||
type DescribeNamespaceResponse struct {
|
||||
Properties map[string]string `json:"properties"`
|
||||
}
|
||||
|
||||
type DropNamespaceRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
Mode string `json:"mode,omitempty"`
|
||||
Behavior string `json:"behavior,omitempty"`
|
||||
}
|
||||
|
||||
type DropNamespaceResponse struct {
|
||||
Properties map[string]string `json:"properties,omitempty"`
|
||||
}
|
||||
|
||||
type NamespaceExistsRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
}
|
||||
|
||||
type ListTablesResponse struct {
|
||||
Tables []string `json:"tables"`
|
||||
PageToken string `json:"page_token,omitempty"`
|
||||
}
|
||||
|
||||
type DeclareTableRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
Location string `json:"location,omitempty"`
|
||||
VendCredentials *bool `json:"vend_credentials,omitempty"`
|
||||
Properties map[string]string `json:"properties,omitempty"`
|
||||
}
|
||||
|
||||
type DeclareTableResponse struct {
|
||||
Location string `json:"location"`
|
||||
StorageOptions map[string]string `json:"storage_options,omitempty"`
|
||||
// Properties are null rather than {}: the catalog does not keep a table's
|
||||
// properties, and {} would claim it kept them and found none.
|
||||
Properties map[string]string `json:"properties"`
|
||||
// ManagedVersioning stays false: the dataset owns its version history,
|
||||
// because this store can order commits without the catalog in the path.
|
||||
ManagedVersioning bool `json:"managed_versioning"`
|
||||
}
|
||||
|
||||
type DescribeTableRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
Version *int64 `json:"version,omitempty"`
|
||||
Tag string `json:"tag,omitempty"`
|
||||
Branch string `json:"branch,omitempty"`
|
||||
// The REST spec carries these as query parameters, but clients also put
|
||||
// them in the body, so honour both.
|
||||
WithTableURI *bool `json:"with_table_uri,omitempty"`
|
||||
LoadDetailedMetadata *bool `json:"load_detailed_metadata,omitempty"`
|
||||
CheckDeclared *bool `json:"check_declared,omitempty"`
|
||||
VendCredentials *bool `json:"vend_credentials,omitempty"`
|
||||
}
|
||||
|
||||
type DescribeTableResponse struct {
|
||||
Table string `json:"table,omitempty"`
|
||||
Namespace []string `json:"namespace,omitempty"`
|
||||
Version *int64 `json:"version,omitempty"`
|
||||
Location string `json:"location"`
|
||||
TableURI string `json:"table_uri,omitempty"`
|
||||
StorageOptions map[string]string `json:"storage_options,omitempty"`
|
||||
// Null, not {}: this catalog does not keep a table's properties.
|
||||
Properties map[string]string `json:"properties"`
|
||||
// ManagedVersioning stays false: the dataset owns its version history,
|
||||
// because this store can order commits without the catalog in the path.
|
||||
ManagedVersioning bool `json:"managed_versioning"`
|
||||
IsOnlyDeclared *bool `json:"is_only_declared,omitempty"`
|
||||
}
|
||||
|
||||
type TableExistsRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
}
|
||||
|
||||
type RegisterTableRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
Location string `json:"location"`
|
||||
Mode string `json:"mode,omitempty"`
|
||||
Properties map[string]string `json:"properties,omitempty"`
|
||||
}
|
||||
|
||||
type RegisterTableResponse struct {
|
||||
Location string `json:"location"`
|
||||
Properties map[string]string `json:"properties"`
|
||||
}
|
||||
|
||||
type DeregisterTableRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
}
|
||||
|
||||
type DeregisterTableResponse struct {
|
||||
ID []string `json:"id"`
|
||||
Location string `json:"location"`
|
||||
Properties map[string]string `json:"properties"`
|
||||
}
|
||||
|
||||
type DropTableRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
}
|
||||
|
||||
type DropTableResponse struct {
|
||||
ID []string `json:"id"`
|
||||
Location string `json:"location"`
|
||||
Properties map[string]string `json:"properties"`
|
||||
}
|
||||
|
||||
type RenameTableRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
NewID []string `json:"new_id"`
|
||||
}
|
||||
|
||||
type RenameTableResponse struct{}
|
||||
@@ -3,6 +3,7 @@ package s3tables
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
@@ -247,6 +248,31 @@ func (h *S3TablesHandler) getExtendedAttribute(ctx context.Context, client filer
|
||||
return data, nil
|
||||
}
|
||||
|
||||
// loadNamespaceMetadata resolves a namespace to its metadata. A directory that
|
||||
// carries no namespace metadata is not a namespace, so a missing attribute
|
||||
// reports the same absence as a missing entry and every caller tests one
|
||||
// condition instead of forgetting the second.
|
||||
func (h *S3TablesHandler) loadNamespaceMetadata(ctx context.Context, filerClient FilerClient, bucketName, namespaceName string) (*namespaceMetadata, error) {
|
||||
var metadata namespaceMetadata
|
||||
err := filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
data, err := h.getExtendedAttribute(ctx, client, GetNamespacePath(bucketName, namespaceName), ExtendedKeyMetadata)
|
||||
if err != nil {
|
||||
if errors.Is(err, ErrAttributeNotFound) {
|
||||
return filer_pb.ErrNotFound
|
||||
}
|
||||
return err
|
||||
}
|
||||
if err := json.Unmarshal(data, &metadata); err != nil {
|
||||
return fmt.Errorf("failed to unmarshal namespace metadata: %w", err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &metadata, nil
|
||||
}
|
||||
|
||||
// lookupEntry returns the filer entry at the given path.
|
||||
func (h *S3TablesHandler) lookupEntry(ctx context.Context, client filer_pb.SeaweedFilerClient, path string) (*filer_pb.Entry, error) {
|
||||
dir, name := splitPath(path)
|
||||
|
||||
@@ -26,6 +26,19 @@ func (h *S3TablesHandler) handleCreateTableBucket(w http.ResponseWriter, r *http
|
||||
return err
|
||||
}
|
||||
|
||||
// A bucket is a catalog, and a catalog serves one protocol. Saying which one
|
||||
// at creation is what lets everything downstream - the endpoint the UI
|
||||
// shows, the tables the bucket accepts - be answered without opening a table.
|
||||
bucketFormat := FormatIceberg
|
||||
if req.Format != "" {
|
||||
normalized, ok := NormalizeFormat(req.Format)
|
||||
if !ok {
|
||||
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, fmt.Sprintf("unsupported format %q", req.Format))
|
||||
return fmt.Errorf("invalid format")
|
||||
}
|
||||
bucketFormat = normalized
|
||||
}
|
||||
|
||||
principal := h.getAccountID(r)
|
||||
identityActions := getIdentityActions(r)
|
||||
identityPolicyNames := getIdentityPolicyNames(r)
|
||||
@@ -115,6 +128,7 @@ func (h *S3TablesHandler) handleCreateTableBucket(w http.ResponseWriter, r *http
|
||||
Name: req.Name,
|
||||
CreatedAt: now,
|
||||
OwnerAccountID: principal,
|
||||
Format: bucketFormat,
|
||||
}
|
||||
|
||||
metadataBytes, err := json.Marshal(metadata)
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
package s3tables
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
const formatTestBucket = "formats"
|
||||
|
||||
// bucketWithFormat lays down a table bucket holding one namespace, declared as
|
||||
// the given format. An empty format is a bucket from before the declaration
|
||||
// existed.
|
||||
func bucketWithFormat(t *testing.T, format string) (*s3tablestest.MemFiler, *Manager) {
|
||||
t.Helper()
|
||||
fs := s3tablestest.Start(t)
|
||||
m := NewManager()
|
||||
|
||||
bucketMeta, _ := json.Marshal(tableBucketMetadata{
|
||||
Name: formatTestBucket,
|
||||
OwnerAccountID: DefaultAccountID,
|
||||
Format: format,
|
||||
})
|
||||
fs.Put(TablesPath, formatTestBucket, map[string][]byte{
|
||||
ExtendedKeyTableBucket: []byte("{}"),
|
||||
ExtendedKeyMetadata: bucketMeta,
|
||||
})
|
||||
|
||||
nsMeta, _ := json.Marshal(namespaceMetadata{Namespace: []string{"ns"}, OwnerAccountID: DefaultAccountID})
|
||||
fs.Put(GetTableBucketPath(formatTestBucket), "ns", map[string][]byte{ExtendedKeyMetadata: nsMeta})
|
||||
|
||||
return fs, m
|
||||
}
|
||||
|
||||
func createTableOfFormat(t *testing.T, m *Manager, fs *s3tablestest.MemFiler, name, format string) error {
|
||||
t.Helper()
|
||||
return m.Execute(context.Background(), NewManagerClient(fs.Client), "CreateTable", &CreateTableRequest{
|
||||
TableBucketARN: "arn:aws:s3tables:::bucket/" + formatTestBucket,
|
||||
Namespace: []string{"ns"},
|
||||
Name: name,
|
||||
Format: format,
|
||||
}, nil, "")
|
||||
}
|
||||
|
||||
// The declaration is the point: a bucket that says it holds Lance cannot be
|
||||
// handed an Iceberg table, because the catalog serving it would never show one.
|
||||
func TestCreateTableRefusesAForeignFormat(t *testing.T) {
|
||||
fs, m := bucketWithFormat(t, FormatLance)
|
||||
|
||||
require.NoError(t, createTableOfFormat(t, m, fs, "vectors", FormatLance))
|
||||
|
||||
err := createTableOfFormat(t, m, fs, "events", FormatIceberg)
|
||||
require.Error(t, err, "an Iceberg table in a Lance bucket should be refused")
|
||||
require.Contains(t, err.Error(), "holds LANCE")
|
||||
}
|
||||
|
||||
func TestCreateTableRefusesLanceInAnIcebergBucket(t *testing.T) {
|
||||
fs, m := bucketWithFormat(t, FormatIceberg)
|
||||
|
||||
err := createTableOfFormat(t, m, fs, "vectors", FormatLance)
|
||||
require.Error(t, err)
|
||||
require.Contains(t, err.Error(), "holds ICEBERG")
|
||||
}
|
||||
|
||||
// A bucket made before formats were declared keeps taking anything. Nothing is
|
||||
// migrated, so nothing that worked stops working.
|
||||
func TestUndeclaredBucketAcceptsEitherFormat(t *testing.T) {
|
||||
fs, m := bucketWithFormat(t, "")
|
||||
|
||||
require.NoError(t, createTableOfFormat(t, m, fs, "events", FormatIceberg))
|
||||
require.NoError(t, createTableOfFormat(t, m, fs, "vectors", FormatLance))
|
||||
}
|
||||
|
||||
// A view is Iceberg metadata, so it has no meaning in a bucket of another format.
|
||||
func TestCreateViewRefusedInALanceBucket(t *testing.T) {
|
||||
fs, m := bucketWithFormat(t, FormatLance)
|
||||
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.Client), "CreateView", &CreateViewRequest{
|
||||
TableBucketARN: "arn:aws:s3tables:::bucket/" + formatTestBucket,
|
||||
Namespace: []string{"ns"},
|
||||
Name: "v",
|
||||
}, nil, "")
|
||||
require.Error(t, err)
|
||||
require.Contains(t, err.Error(), "cannot hold views")
|
||||
}
|
||||
|
||||
func TestNormalizeFormat(t *testing.T) {
|
||||
cases := []struct {
|
||||
in string
|
||||
want string
|
||||
ok bool
|
||||
}{
|
||||
{"ICEBERG", FormatIceberg, true},
|
||||
{"iceberg", FormatIceberg, true},
|
||||
{" Lance ", FormatLance, true},
|
||||
{"delta", "", false},
|
||||
{"", "", false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
got, ok := NormalizeFormat(c.in)
|
||||
if got != c.want || ok != c.ok {
|
||||
t.Errorf("NormalizeFormat(%q) = %q,%v; want %q,%v", c.in, got, ok, c.want, c.ok)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -83,6 +83,7 @@ func (h *S3TablesHandler) handleGetTableBucket(w http.ResponseWriter, r *http.Re
|
||||
Name: metadata.Name,
|
||||
OwnerAccountID: metadata.OwnerAccountID,
|
||||
CreatedAt: metadata.CreatedAt,
|
||||
Format: metadata.Format,
|
||||
}
|
||||
|
||||
h.writeJSON(w, http.StatusOK, resp)
|
||||
@@ -202,6 +203,7 @@ func (h *S3TablesHandler) handleListTableBuckets(w http.ResponseWriter, r *http.
|
||||
ARN: bucketARN,
|
||||
Name: entry.Entry.Name,
|
||||
CreatedAt: metadata.CreatedAt,
|
||||
Format: metadata.Format,
|
||||
})
|
||||
|
||||
if len(buckets) >= maxBuckets {
|
||||
|
||||
@@ -5,13 +5,15 @@ import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func runDeleteTable(t *testing.T, m *Manager, fs *memFilerServer, namespace, name string) error {
|
||||
func runDeleteTable(t *testing.T, m *Manager, fs *s3tablestest.MemFiler, namespace, name string) error {
|
||||
t.Helper()
|
||||
return m.Execute(context.Background(), NewManagerClient(fs.client), "DeleteTable", &DeleteTableRequest{
|
||||
return m.Execute(context.Background(), NewManagerClient(fs.Client), "DeleteTable", &DeleteTableRequest{
|
||||
TableBucketARN: mustBucketARN(t),
|
||||
Namespace: []string{namespace},
|
||||
Name: name,
|
||||
@@ -33,24 +35,24 @@ func TestDeleteTableDecoupledKeepsReusedNamePath(t *testing.T) {
|
||||
MetadataLocation: "s3://" + renameTestBucket + "/ns/newt-x/metadata/v1.metadata.json",
|
||||
})
|
||||
markerKeys := []string{ExtendedKeyMetadata, ExtendedKeyMetadataVersion, ExtendedKeyPolicy, ExtendedKeyTags, ExtendedKeyEntryType}
|
||||
fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), "newt", map[string][]byte{
|
||||
fs.Put(GetNamespacePath(renameTestBucket, "ns"), "newt", map[string][]byte{
|
||||
ExtendedKeyMetadata: newtMeta,
|
||||
ExtendedKeyMetadataVersion: []byte("v1"),
|
||||
ExtendedKeyPolicy: []byte(`{"Version":"2012-10-17"}`),
|
||||
ExtendedKeyTags: []byte(`{"k":"v"}`),
|
||||
ExtendedKeyEntryType: []byte(EntryTypeTable),
|
||||
})
|
||||
fs.putEntry(GetTablePath(renameTestBucket, "ns", "newt"), "leftover", nil) // another table's data under the name path
|
||||
fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), "newt-x", nil) // this table's own (decoupled) data
|
||||
fs.putEntry(GetTablePath(renameTestBucket, "ns", "newt-x"), "metadata", nil)
|
||||
fs.Put(GetTablePath(renameTestBucket, "ns", "newt"), "leftover", nil) // another table's data under the name path
|
||||
fs.Put(GetNamespacePath(renameTestBucket, "ns"), "newt-x", nil) // this table's own (decoupled) data
|
||||
fs.Put(GetTablePath(renameTestBucket, "ns", "newt-x"), "metadata", nil)
|
||||
|
||||
require.NoError(t, runDeleteTable(t, m, fs, "ns", "newt"))
|
||||
|
||||
assert.Nil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "newt-x"),
|
||||
assert.Nil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "newt-x"),
|
||||
"the table's own data location must be purged")
|
||||
assert.NotNil(t, fs.getEntry(GetTablePath(renameTestBucket, "ns", "newt"), "leftover"),
|
||||
assert.NotNil(t, fs.Get(GetTablePath(renameTestBucket, "ns", "newt"), "leftover"),
|
||||
"data under the reused name path must survive")
|
||||
marker := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "newt")
|
||||
marker := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "newt")
|
||||
require.NotNil(t, marker)
|
||||
for _, key := range markerKeys {
|
||||
_, present := marker.Extended[key]
|
||||
@@ -72,14 +74,14 @@ func TestDeleteTableRefusesAncestorDataPath(t *testing.T) {
|
||||
OwnerAccountID: DefaultAccountID,
|
||||
MetadataLocation: "s3://" + renameTestBucket + "/ns/metadata/v1.metadata.json",
|
||||
})
|
||||
fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), "badt", map[string][]byte{ExtendedKeyMetadata: badMeta})
|
||||
fs.Put(GetNamespacePath(renameTestBucket, "ns"), "badt", map[string][]byte{ExtendedKeyMetadata: badMeta})
|
||||
|
||||
require.Error(t, runDeleteTable(t, m, fs, "ns", "badt"))
|
||||
|
||||
// The sibling table seeded by startRenameManager and its data must survive.
|
||||
assert.NotNil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t"),
|
||||
assert.NotNil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t"),
|
||||
"sibling table marker must survive a refused delete")
|
||||
assert.NotNil(t, fs.getEntry(GetTablePath(renameTestBucket, "ns", "t"), "data"),
|
||||
assert.NotNil(t, fs.Get(GetTablePath(renameTestBucket, "ns", "t"), "data"),
|
||||
"sibling table data must survive a refused delete")
|
||||
}
|
||||
|
||||
@@ -89,6 +91,6 @@ func TestDeleteTableColocatedRemovesData(t *testing.T) {
|
||||
|
||||
require.NoError(t, runDeleteTable(t, m, fs, "ns", "t"))
|
||||
|
||||
assert.Nil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t"), "colocated table entry must be deleted")
|
||||
assert.Nil(t, fs.getEntry(GetTablePath(renameTestBucket, "ns", "t"), "metadata"), "colocated table data must be deleted")
|
||||
assert.Nil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t"), "colocated table entry must be deleted")
|
||||
assert.Nil(t, fs.Get(GetTablePath(renameTestBucket, "ns", "t"), "metadata"), "colocated table data must be deleted")
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
package s3tables
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest"
|
||||
)
|
||||
|
||||
// A directory under a table bucket that carries no namespace metadata is not a
|
||||
// namespace. Reporting that as an internal error, which is what the three
|
||||
// callers of this did while each testing only for a missing entry, turns a
|
||||
// client mistake into a 500 and hides it behind "attribute not found".
|
||||
func TestLoadNamespaceMetadataTreatsMissingAttributeAsAbsent(t *testing.T) {
|
||||
filer := s3tablestest.Start(t)
|
||||
handler := NewS3TablesHandler()
|
||||
client := NewManagerClient(filer.Client)
|
||||
ctx := context.Background()
|
||||
|
||||
filer.Put(TablesPath, "bkt", map[string][]byte{ExtendedKeyTableBucket: []byte("{}")})
|
||||
|
||||
// A stray directory where a namespace would live, with no metadata on it.
|
||||
filer.Put(GetTableBucketPath("bkt"), "stray", nil)
|
||||
|
||||
if _, err := handler.loadNamespaceMetadata(ctx, client, "bkt", "stray"); !errors.Is(err, filer_pb.ErrNotFound) {
|
||||
t.Fatalf("loadNamespaceMetadata on a directory without metadata = %v, want ErrNotFound", err)
|
||||
}
|
||||
|
||||
if _, err := handler.loadNamespaceMetadata(ctx, client, "bkt", "absent"); !errors.Is(err, filer_pb.ErrNotFound) {
|
||||
t.Fatalf("loadNamespaceMetadata on a missing directory = %v, want ErrNotFound", err)
|
||||
}
|
||||
|
||||
filer.Put(GetTableBucketPath("bkt"), "real", map[string][]byte{
|
||||
ExtendedKeyMetadata: []byte(`{"namespace":["real"],"ownerAccountId":"000000000000"}`),
|
||||
})
|
||||
metadata, err := handler.loadNamespaceMetadata(ctx, client, "bkt", "real")
|
||||
if err != nil {
|
||||
t.Fatalf("loadNamespaceMetadata on a real namespace: %v", err)
|
||||
}
|
||||
if metadata.OwnerAccountID != "000000000000" {
|
||||
t.Fatalf("owner = %q, want the stored one", metadata.OwnerAccountID)
|
||||
}
|
||||
}
|
||||
@@ -1,165 +1,18 @@
|
||||
package s3tables
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net"
|
||||
"path"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"google.golang.org/grpc"
|
||||
"google.golang.org/grpc/codes"
|
||||
"google.golang.org/grpc/credentials/insecure"
|
||||
"google.golang.org/grpc/status"
|
||||
)
|
||||
|
||||
// memFilerServer is an in-memory filer used to drive Manager operations
|
||||
// end-to-end without a live cluster.
|
||||
type memFilerServer struct {
|
||||
filer_pb.UnimplementedSeaweedFilerServer
|
||||
entries map[string]map[string]*filer_pb.Entry // dir -> name -> entry
|
||||
client filer_pb.SeaweedFilerClient
|
||||
// beforeUpdate runs once, at the start of the next UpdateEntry, so a test
|
||||
// can land a competing write in a handler's read-to-write window.
|
||||
beforeUpdate func()
|
||||
}
|
||||
|
||||
func newMemFilerServer() *memFilerServer {
|
||||
return &memFilerServer{entries: make(map[string]map[string]*filer_pb.Entry)}
|
||||
}
|
||||
|
||||
func (f *memFilerServer) getEntry(dir, name string) *filer_pb.Entry {
|
||||
if d, ok := f.entries[dir]; ok {
|
||||
return d[name]
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *memFilerServer) putEntry(dir, name string, extended map[string][]byte) {
|
||||
if _, ok := f.entries[dir]; !ok {
|
||||
f.entries[dir] = make(map[string]*filer_pb.Entry)
|
||||
}
|
||||
f.entries[dir][name] = &filer_pb.Entry{Name: name, IsDirectory: true, Extended: extended}
|
||||
}
|
||||
|
||||
func (f *memFilerServer) LookupDirectoryEntry(_ context.Context, req *filer_pb.LookupDirectoryEntryRequest) (*filer_pb.LookupDirectoryEntryResponse, error) {
|
||||
if e := f.getEntry(req.Directory, req.Name); e != nil {
|
||||
return &filer_pb.LookupDirectoryEntryResponse{Entry: e}, nil
|
||||
}
|
||||
// Carry the sentinel text so filer_pb.LookupEntry maps it to ErrNotFound.
|
||||
return nil, status.Errorf(codes.NotFound, "%s: %s/%s", filer_pb.ErrNotFound.Error(), req.Directory, req.Name)
|
||||
}
|
||||
|
||||
func (f *memFilerServer) ListEntries(req *filer_pb.ListEntriesRequest, stream grpc.ServerStreamingServer[filer_pb.ListEntriesResponse]) error {
|
||||
d, ok := f.entries[req.Directory]
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
names := make([]string, 0, len(d))
|
||||
for name := range d {
|
||||
names = append(names, name)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, name := range names {
|
||||
if err := stream.Send(&filer_pb.ListEntriesResponse{Entry: d[name]}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *memFilerServer) CreateEntry(_ context.Context, req *filer_pb.CreateEntryRequest) (*filer_pb.CreateEntryResponse, error) {
|
||||
if _, ok := f.entries[req.Directory]; !ok {
|
||||
f.entries[req.Directory] = make(map[string]*filer_pb.Entry)
|
||||
}
|
||||
f.entries[req.Directory][req.Entry.Name] = req.Entry
|
||||
return &filer_pb.CreateEntryResponse{}, nil
|
||||
}
|
||||
|
||||
func (f *memFilerServer) UpdateEntry(_ context.Context, req *filer_pb.UpdateEntryRequest) (*filer_pb.UpdateEntryResponse, error) {
|
||||
if hook := f.beforeUpdate; hook != nil {
|
||||
f.beforeUpdate = nil
|
||||
hook()
|
||||
}
|
||||
// The real filer validates ExpectedExtended under the per-path lock; without
|
||||
// it here a lost update would look like a success.
|
||||
for key, expected := range req.ExpectedExtended {
|
||||
var actual []byte
|
||||
if existing := f.getEntry(req.Directory, req.Entry.Name); existing != nil {
|
||||
actual = existing.Extended[key]
|
||||
}
|
||||
if !bytes.Equal(actual, expected) {
|
||||
return nil, status.Errorf(codes.FailedPrecondition, "extended attribute %q changed", key)
|
||||
}
|
||||
}
|
||||
if _, ok := f.entries[req.Directory]; !ok {
|
||||
f.entries[req.Directory] = make(map[string]*filer_pb.Entry)
|
||||
}
|
||||
f.entries[req.Directory][req.Entry.Name] = req.Entry
|
||||
return &filer_pb.UpdateEntryResponse{}, nil
|
||||
}
|
||||
|
||||
func (f *memFilerServer) DeleteEntry(_ context.Context, req *filer_pb.DeleteEntryRequest) (*filer_pb.DeleteEntryResponse, error) {
|
||||
if d, ok := f.entries[req.Directory]; ok {
|
||||
delete(d, req.Name)
|
||||
}
|
||||
// Honor recursive data deletion so a regression that wipes the table directory
|
||||
// also drops its metadata/ and data/ children (the data-loss this guards against).
|
||||
if req.IsRecursive && req.IsDeleteData {
|
||||
child := path.Join(req.Directory, req.Name)
|
||||
for dir := range f.entries {
|
||||
if dir == child || strings.HasPrefix(dir, child+"/") {
|
||||
delete(f.entries, dir)
|
||||
}
|
||||
}
|
||||
}
|
||||
return &filer_pb.DeleteEntryResponse{}, nil
|
||||
}
|
||||
|
||||
func (f *memFilerServer) Ping(_ context.Context, _ *filer_pb.PingRequest) (*filer_pb.PingResponse, error) {
|
||||
now := time.Now().UnixNano()
|
||||
return &filer_pb.PingResponse{StartTimeNs: now, RemoteTimeNs: now, StopTimeNs: now}, nil
|
||||
}
|
||||
|
||||
func startMemFiler(t *testing.T) *memFilerServer {
|
||||
t.Helper()
|
||||
fs := newMemFilerServer()
|
||||
|
||||
listener, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
require.NoError(t, err)
|
||||
|
||||
server := grpc.NewServer()
|
||||
filer_pb.RegisterSeaweedFilerServer(server, fs)
|
||||
go func() { _ = server.Serve(listener) }()
|
||||
t.Cleanup(server.GracefulStop)
|
||||
|
||||
conn, err := grpc.NewClient(listener.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = conn.Close() })
|
||||
|
||||
fs.client = filer_pb.NewSeaweedFilerClient(conn)
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
pingCtx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
_, err := fs.client.Ping(pingCtx, &filer_pb.PingRequest{})
|
||||
cancel()
|
||||
if err == nil {
|
||||
break
|
||||
}
|
||||
require.False(t, time.Now().After(deadline), "filer not ready: %v", err)
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
return fs
|
||||
}
|
||||
|
||||
const renameTestBucket = "renamebkt"
|
||||
|
||||
func mustBucketARN(t *testing.T) string {
|
||||
@@ -170,18 +23,18 @@ func mustBucketARN(t *testing.T) string {
|
||||
}
|
||||
|
||||
// startRenameManager seeds a bucket/namespace/table and returns a trusted Manager.
|
||||
func startRenameManager(t *testing.T) (*memFilerServer, *Manager) {
|
||||
func startRenameManager(t *testing.T) (*s3tablestest.MemFiler, *Manager) {
|
||||
t.Helper()
|
||||
fs := startMemFiler(t)
|
||||
fs := s3tablestest.Start(t)
|
||||
|
||||
bucketMeta, _ := json.Marshal(tableBucketMetadata{Name: renameTestBucket, OwnerAccountID: DefaultAccountID})
|
||||
fs.putEntry(TablesPath, renameTestBucket, map[string][]byte{
|
||||
fs.Put(TablesPath, renameTestBucket, map[string][]byte{
|
||||
ExtendedKeyTableBucket: []byte("{}"),
|
||||
ExtendedKeyMetadata: bucketMeta,
|
||||
})
|
||||
|
||||
nsMeta, _ := json.Marshal(namespaceMetadata{Namespace: []string{"ns"}, OwnerAccountID: DefaultAccountID})
|
||||
fs.putEntry(GetTableBucketPath(renameTestBucket), "ns", map[string][]byte{ExtendedKeyMetadata: nsMeta})
|
||||
fs.Put(GetTableBucketPath(renameTestBucket), "ns", map[string][]byte{ExtendedKeyMetadata: nsMeta})
|
||||
|
||||
tableMeta, _ := json.Marshal(tableMetadataInternal{
|
||||
Name: "t",
|
||||
@@ -191,31 +44,31 @@ func startRenameManager(t *testing.T) (*memFilerServer, *Manager) {
|
||||
MetadataVersion: 3,
|
||||
MetadataLocation: "s3://" + renameTestBucket + "/ns/t/metadata/v3.metadata.json",
|
||||
})
|
||||
fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), "t", map[string][]byte{
|
||||
fs.Put(GetNamespacePath(renameTestBucket, "ns"), "t", map[string][]byte{
|
||||
ExtendedKeyMetadata: tableMeta,
|
||||
ExtendedKeyMetadataVersion: []byte("3"),
|
||||
})
|
||||
|
||||
// Physical metadata.json and data files live under the table directory.
|
||||
tablePath := GetTablePath(renameTestBucket, "ns", "t")
|
||||
fs.putEntry(tablePath, "metadata", nil)
|
||||
fs.putEntry(tablePath, "data", nil)
|
||||
fs.putEntry(path.Join(tablePath, "metadata"), "v3.metadata.json", nil)
|
||||
fs.Put(tablePath, "metadata", nil)
|
||||
fs.Put(tablePath, "data", nil)
|
||||
fs.Put(path.Join(tablePath, "metadata"), "v3.metadata.json", nil)
|
||||
|
||||
m := NewManager()
|
||||
m.SetTrusted(true)
|
||||
return fs, m
|
||||
}
|
||||
|
||||
func runRename(t *testing.T, m *Manager, fs *memFilerServer, req *RenameTableRequest) error {
|
||||
func runRename(t *testing.T, m *Manager, fs *s3tablestest.MemFiler, req *RenameTableRequest) error {
|
||||
t.Helper()
|
||||
return m.Execute(context.Background(), NewManagerClient(fs.client), "RenameTable", req, nil, "")
|
||||
return m.Execute(context.Background(), NewManagerClient(fs.Client), "RenameTable", req, nil, "")
|
||||
}
|
||||
|
||||
func runGetTable(t *testing.T, m *Manager, fs *memFilerServer, namespace, name string) (*GetTableResponse, error) {
|
||||
func runGetTable(t *testing.T, m *Manager, fs *s3tablestest.MemFiler, namespace, name string) (*GetTableResponse, error) {
|
||||
t.Helper()
|
||||
resp := &GetTableResponse{}
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.client), "GetTable", &GetTableRequest{
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.Client), "GetTable", &GetTableRequest{
|
||||
TableBucketARN: mustBucketARN(t),
|
||||
Namespace: []string{namespace},
|
||||
Name: name,
|
||||
@@ -238,12 +91,12 @@ func TestRenameTablePreservesData(t *testing.T) {
|
||||
// The source directory and its metadata.json/data children must survive: rename
|
||||
// is catalog-only and the destination still points at the original location.
|
||||
srcPath := GetTablePath(renameTestBucket, "ns", "t")
|
||||
assert.NotNil(t, fs.getEntry(srcPath, "metadata"), "source metadata dir must survive")
|
||||
assert.NotNil(t, fs.getEntry(srcPath, "data"), "source data dir must survive")
|
||||
assert.NotNil(t, fs.getEntry(path.Join(srcPath, "metadata"), "v3.metadata.json"), "metadata.json must survive")
|
||||
assert.NotNil(t, fs.Get(srcPath, "metadata"), "source metadata dir must survive")
|
||||
assert.NotNil(t, fs.Get(srcPath, "data"), "source data dir must survive")
|
||||
assert.NotNil(t, fs.Get(path.Join(srcPath, "metadata"), "v3.metadata.json"), "metadata.json must survive")
|
||||
|
||||
// Source catalog xattrs are dropped so the name stops resolving.
|
||||
src := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
src := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
require.NotNil(t, src, "source directory must remain to hold the data children")
|
||||
_, hasMeta := src.Extended[ExtendedKeyMetadata]
|
||||
assert.False(t, hasMeta, "source table-metadata xattr must be removed")
|
||||
@@ -260,7 +113,7 @@ func TestRenameTablePreservesData(t *testing.T) {
|
||||
assert.Equal(t, "t2", got.Name)
|
||||
assert.Equal(t, "s3://"+renameTestBucket+"/ns/t/metadata/v3.metadata.json", got.MetadataLocation)
|
||||
|
||||
dest := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t2")
|
||||
dest := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t2")
|
||||
require.NotNil(t, dest)
|
||||
assert.Equal(t, []byte("3"), dest.Extended[ExtendedKeyMetadataVersion])
|
||||
}
|
||||
@@ -283,7 +136,7 @@ func TestRenameTableSourceMissing(t *testing.T) {
|
||||
func TestRenameTableDestExists(t *testing.T) {
|
||||
fs, m := startRenameManager(t)
|
||||
existing, _ := json.Marshal(tableMetadataInternal{Name: "t2", Namespace: "ns", OwnerAccountID: DefaultAccountID})
|
||||
fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), "t2", map[string][]byte{ExtendedKeyMetadata: existing})
|
||||
fs.Put(GetNamespacePath(renameTestBucket, "ns"), "t2", map[string][]byte{ExtendedKeyMetadata: existing})
|
||||
|
||||
err := runRename(t, m, fs, &RenameTableRequest{
|
||||
TableBucketARN: mustBucketARN(t),
|
||||
@@ -296,7 +149,7 @@ func TestRenameTableDestExists(t *testing.T) {
|
||||
var s3Err *S3TablesError
|
||||
require.ErrorAs(t, err, &s3Err)
|
||||
assert.Equal(t, ErrCodeTableAlreadyExists, s3Err.Type)
|
||||
assert.NotNil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched on conflict")
|
||||
assert.NotNil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched on conflict")
|
||||
}
|
||||
|
||||
func TestRenameTableDestNamespaceMissing(t *testing.T) {
|
||||
@@ -312,7 +165,7 @@ func TestRenameTableDestNamespaceMissing(t *testing.T) {
|
||||
var s3Err *S3TablesError
|
||||
require.ErrorAs(t, err, &s3Err)
|
||||
assert.Equal(t, ErrCodeNoSuchNamespace, s3Err.Type)
|
||||
assert.NotNil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched")
|
||||
assert.NotNil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched")
|
||||
}
|
||||
|
||||
// A principal allowed to rename the source must still be denied when it cannot
|
||||
@@ -332,16 +185,16 @@ func TestRenameTableDestNamespaceUnauthorized(t *testing.T) {
|
||||
"Resource": "*",
|
||||
}},
|
||||
})
|
||||
srcEntry := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
srcEntry := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
require.NotNil(t, srcEntry)
|
||||
srcEntry.Extended[ExtendedKeyPolicy] = srcPolicy
|
||||
|
||||
destNsMeta, _ := json.Marshal(namespaceMetadata{Namespace: []string{"dest"}, OwnerAccountID: DefaultAccountID})
|
||||
fs.putEntry(GetTableBucketPath(renameTestBucket), "dest", map[string][]byte{ExtendedKeyMetadata: destNsMeta})
|
||||
fs.Put(GetTableBucketPath(renameTestBucket), "dest", map[string][]byte{ExtendedKeyMetadata: destNsMeta})
|
||||
|
||||
mover := &testIdentity{Name: "mover", Account: &testIdentityAccount{Id: "mover"}}
|
||||
ctx := s3_constants.SetIdentityInContext(context.Background(), mover)
|
||||
err := m.Execute(ctx, NewManagerClient(fs.client), "RenameTable", &RenameTableRequest{
|
||||
err := m.Execute(ctx, NewManagerClient(fs.Client), "RenameTable", &RenameTableRequest{
|
||||
TableBucketARN: mustBucketARN(t),
|
||||
SourceNamespace: []string{"ns"},
|
||||
SourceName: "t",
|
||||
@@ -353,8 +206,8 @@ func TestRenameTableDestNamespaceUnauthorized(t *testing.T) {
|
||||
require.ErrorAs(t, err, &s3Err)
|
||||
assert.Equal(t, ErrCodeAccessDenied, s3Err.Type)
|
||||
|
||||
assert.NotNil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched")
|
||||
assert.Nil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "dest"), "t2"), "destination must not be written")
|
||||
assert.NotNil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched")
|
||||
assert.Nil(t, fs.Get(GetNamespacePath(renameTestBucket, "dest"), "t2"), "destination must not be written")
|
||||
}
|
||||
|
||||
func TestRenameTableInvalidName(t *testing.T) {
|
||||
@@ -378,7 +231,7 @@ func TestRenameTableInvalidName(t *testing.T) {
|
||||
func TestRenameTableCarriesMaintenanceConfiguration(t *testing.T) {
|
||||
fs, m := startRenameManager(t)
|
||||
|
||||
src := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
src := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
require.NotNil(t, src)
|
||||
config := []byte(`{"icebergSnapshotManagement":{"status":"disabled"}}`)
|
||||
status := []byte(`{"icebergCompaction":{"status":"Successful"}}`)
|
||||
@@ -393,13 +246,13 @@ func TestRenameTableCarriesMaintenanceConfiguration(t *testing.T) {
|
||||
DestName: "t2",
|
||||
}))
|
||||
|
||||
dest := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t2")
|
||||
dest := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t2")
|
||||
require.NotNil(t, dest)
|
||||
assert.Equal(t, config, dest.Extended[ExtendedKeyMaintenance], "the disable must move with the table")
|
||||
assert.Equal(t, status, dest.Extended[ExtendedKeyMaintenanceStatus])
|
||||
|
||||
// And must not linger on the old name, where a reused name would inherit it.
|
||||
moved := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
moved := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
require.NotNil(t, moved)
|
||||
_, hasConfig := moved.Extended[ExtendedKeyMaintenance]
|
||||
assert.False(t, hasConfig, "source maintenance configuration must be cleared")
|
||||
@@ -413,7 +266,7 @@ func TestRenameTableCarriesMaintenanceConfiguration(t *testing.T) {
|
||||
func TestRenameTableRejectsConcurrentMaintenanceWrite(t *testing.T) {
|
||||
fs, _ := startRenameManager(t)
|
||||
|
||||
src := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
src := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
require.NotNil(t, src)
|
||||
copied := []byte(`{"icebergCompaction":{"status":"enabled"}}`)
|
||||
src.Extended[ExtendedKeyMaintenance] = copied
|
||||
@@ -425,7 +278,7 @@ func TestRenameTableRejectsConcurrentMaintenanceWrite(t *testing.T) {
|
||||
src.Extended[ExtendedKeyMaintenance] = landedLate
|
||||
|
||||
h := NewS3TablesHandler()
|
||||
err := NewManagerClient(fs.client).WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
err := NewManagerClient(fs.Client).WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
return h.removeExtendedAttributesIf(context.Background(),
|
||||
client, GetTablePath(renameTestBucket, "ns", "t"), expected, renamedTableAttributes...)
|
||||
})
|
||||
|
||||
@@ -45,8 +45,8 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque
|
||||
}
|
||||
|
||||
// Validate format
|
||||
if req.Format != "ICEBERG" {
|
||||
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, "only ICEBERG format is supported")
|
||||
if req.Format != FormatIceberg && req.Format != FormatLance {
|
||||
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, fmt.Sprintf("unsupported format %q", req.Format))
|
||||
return fmt.Errorf("invalid format")
|
||||
}
|
||||
|
||||
@@ -65,18 +65,7 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque
|
||||
|
||||
// Check if namespace exists
|
||||
namespacePath := GetNamespacePath(bucketName, namespaceName)
|
||||
var namespaceMetadata namespaceMetadata
|
||||
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
data, err := h.getExtendedAttribute(r.Context(), client, namespacePath, ExtendedKeyMetadata)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := json.Unmarshal(data, &namespaceMetadata); err != nil {
|
||||
return fmt.Errorf("failed to unmarshal namespace metadata: %w", err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
namespaceMetadata, err := h.loadNamespaceMetadata(r.Context(), filerClient, bucketName, namespaceName)
|
||||
if err != nil {
|
||||
if errors.Is(err, filer_pb.ErrNotFound) {
|
||||
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchNamespace, fmt.Sprintf("namespace %s not found", namespaceName))
|
||||
@@ -135,6 +124,15 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque
|
||||
return err
|
||||
}
|
||||
|
||||
// A bucket declares the format it holds, and a table of another format would
|
||||
// be invisible to the catalog serving it. A bucket made before the
|
||||
// declaration existed has none, and keeps taking anything.
|
||||
if bucketMetadata.Format != "" && bucketMetadata.Format != req.Format {
|
||||
message := fmt.Sprintf("table bucket %s holds %s tables", bucketName, bucketMetadata.Format)
|
||||
h.writeError(w, http.StatusConflict, ErrCodeConflict, message)
|
||||
return fmt.Errorf("%s", message)
|
||||
}
|
||||
|
||||
bucketARN := h.generateTableBucketARN(bucketMetadata.OwnerAccountID, bucketName)
|
||||
identityActions := getIdentityActions(r)
|
||||
nsAllowed := CheckPermissionWithContext("CreateTable", accountID, namespaceMetadata.OwnerAccountID, namespacePolicy, bucketARN, &PolicyContext{
|
||||
@@ -174,7 +172,7 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if entryType(entry.Extended) == EntryTypeView {
|
||||
if EntryType(entry.Extended) == EntryTypeView {
|
||||
existingIsView = true
|
||||
return nil
|
||||
}
|
||||
@@ -193,6 +191,14 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque
|
||||
h.writeError(w, http.StatusConflict, ErrCodeTableAlreadyExists, fmt.Sprintf("a view named %s already exists", tableName))
|
||||
return fmt.Errorf("view name conflict: %s", tableName)
|
||||
}
|
||||
// Creating a table that already exists is idempotent, but only for the
|
||||
// same format. Handing a Lance client an Iceberg table's location, or the
|
||||
// reverse, has it write one format's files into the other's directory.
|
||||
if existingMetadata.Format != "" && existingMetadata.Format != req.Format {
|
||||
h.writeError(w, http.StatusConflict, ErrCodeTableAlreadyExists,
|
||||
fmt.Sprintf("a %s table named %s already exists", existingMetadata.Format, tableName))
|
||||
return fmt.Errorf("format conflict: %s", tableName)
|
||||
}
|
||||
tableARN := h.generateTableARN(existingMetadata.OwnerAccountID, bucketName, namespaceName+"/"+tableName)
|
||||
h.writeJSON(w, http.StatusOK, &CreateTableResponse{
|
||||
TableARN: tableARN,
|
||||
@@ -357,17 +363,7 @@ func (h *S3TablesHandler) handleRegisterTable(w http.ResponseWriter, r *http.Req
|
||||
|
||||
// Namespace must exist.
|
||||
namespacePath := GetNamespacePath(bucketName, namespaceName)
|
||||
var namespaceMetadata namespaceMetadata
|
||||
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
data, err := h.getExtendedAttribute(r.Context(), client, namespacePath, ExtendedKeyMetadata)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := json.Unmarshal(data, &namespaceMetadata); err != nil {
|
||||
return fmt.Errorf("failed to unmarshal namespace metadata: %w", err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
namespaceMetadata, err := h.loadNamespaceMetadata(r.Context(), filerClient, bucketName, namespaceName)
|
||||
if err != nil {
|
||||
if errors.Is(err, filer_pb.ErrNotFound) {
|
||||
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchNamespace, fmt.Sprintf("namespace %s not found", namespaceName))
|
||||
@@ -468,7 +464,7 @@ func (h *S3TablesHandler) handleRegisterTable(w http.ResponseWriter, r *http.Req
|
||||
metadata := &tableMetadataInternal{
|
||||
Name: tableName,
|
||||
Namespace: namespaceName,
|
||||
Format: "ICEBERG",
|
||||
Format: FormatIceberg,
|
||||
CreatedAt: now,
|
||||
ModifiedAt: now,
|
||||
OwnerAccountID: namespaceMetadata.OwnerAccountID,
|
||||
@@ -551,7 +547,7 @@ func (h *S3TablesHandler) handleGetTable(w http.ResponseWriter, r *http.Request,
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if entryType(entry.Extended) == EntryTypeView {
|
||||
if EntryType(entry.Extended) == EntryTypeView {
|
||||
return filer_pb.ErrNotFound
|
||||
}
|
||||
data, ok := entry.Extended[ExtendedKeyMetadata]
|
||||
@@ -923,7 +919,7 @@ func (h *S3TablesHandler) listTablesWithClient(r *http.Request, client filer_pb.
|
||||
}
|
||||
|
||||
// Views share the table layout; exclude them from table listings.
|
||||
if entryType(entry.Entry.Extended) == EntryTypeView {
|
||||
if EntryType(entry.Entry.Extended) == EntryTypeView {
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -945,11 +941,13 @@ func (h *S3TablesHandler) listTablesWithClient(r *http.Request, client filer_pb.
|
||||
tableARN := h.generateTableARN(metadata.OwnerAccountID, bucketName, namespaceName+"/"+entry.Entry.Name)
|
||||
|
||||
tables = append(tables, TableSummary{
|
||||
Name: entry.Entry.Name,
|
||||
TableARN: tableARN,
|
||||
Namespace: expandNamespace(namespaceName),
|
||||
CreatedAt: metadata.CreatedAt,
|
||||
ModifiedAt: metadata.ModifiedAt,
|
||||
Name: entry.Entry.Name,
|
||||
TableARN: tableARN,
|
||||
Namespace: expandNamespace(namespaceName),
|
||||
Format: metadata.Format,
|
||||
CreatedAt: metadata.CreatedAt,
|
||||
ModifiedAt: metadata.ModifiedAt,
|
||||
MetadataLocation: metadata.MetadataLocation,
|
||||
})
|
||||
|
||||
if len(tables) >= maxTables {
|
||||
@@ -1435,7 +1433,7 @@ func (h *S3TablesHandler) renameCatalogEntry(w http.ResponseWriter, r *http.Requ
|
||||
|
||||
// Tables and views share the namespace directory, so a rename must not pick
|
||||
// up the other kind under the same name.
|
||||
if entryType(srcExtended) != kind.entryType {
|
||||
if EntryType(srcExtended) != kind.entryType {
|
||||
h.writeError(w, http.StatusNotFound, kind.notFoundCode, fmt.Sprintf("%s %s not found", kind.noun, srcName))
|
||||
return fmt.Errorf("%s %s not found", kind.noun, srcName)
|
||||
}
|
||||
|
||||
@@ -26,7 +26,7 @@ func TestUpdateTableRejectsLostUpdate(t *testing.T) {
|
||||
fs, m := startRenameManager(t)
|
||||
|
||||
winnerLocation := "s3://" + renameTestBucket + "/ns/t/metadata/v4-winner.metadata.json"
|
||||
fs.beforeUpdate = func() {
|
||||
fs.BeforeUpdate = func() {
|
||||
winner, err := json.Marshal(tableMetadataInternal{
|
||||
Name: "t",
|
||||
Namespace: "ns",
|
||||
@@ -37,13 +37,13 @@ func TestUpdateTableRejectsLostUpdate(t *testing.T) {
|
||||
VersionToken: generateVersionToken(),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
entry := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
entry := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
require.NotNil(t, entry)
|
||||
entry.Extended[ExtendedKeyMetadata] = winner
|
||||
}
|
||||
|
||||
loserLocation := "s3://" + renameTestBucket + "/ns/t/metadata/v4-loser.metadata.json"
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.client), "UpdateTable",
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.Client), "UpdateTable",
|
||||
updateTableRequest(t, loserLocation, 4), nil, "")
|
||||
|
||||
require.Error(t, err)
|
||||
@@ -64,14 +64,14 @@ func TestUpdateTableRejectsWhenTheAuthorizingPolicyChanges(t *testing.T) {
|
||||
fs, m := startRenameManager(t)
|
||||
|
||||
policy := []byte(`{"Version":"2012-10-17","Statement":[]}`)
|
||||
fs.beforeUpdate = func() {
|
||||
entry := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
fs.BeforeUpdate = func() {
|
||||
entry := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
require.NotNil(t, entry)
|
||||
entry.Extended[ExtendedKeyPolicy] = policy
|
||||
}
|
||||
|
||||
location := "s3://" + renameTestBucket + "/ns/t/metadata/v4.metadata.json"
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.client), "UpdateTable",
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.Client), "UpdateTable",
|
||||
updateTableRequest(t, location, 4), nil, "")
|
||||
|
||||
require.Error(t, err)
|
||||
@@ -79,7 +79,7 @@ func TestUpdateTableRejectsWhenTheAuthorizingPolicyChanges(t *testing.T) {
|
||||
require.ErrorAs(t, err, &s3Err)
|
||||
assert.Equal(t, ErrCodeConflict, s3Err.Type)
|
||||
|
||||
entry := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
entry := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
require.NotNil(t, entry)
|
||||
assert.Equal(t, policy, entry.Extended[ExtendedKeyPolicy], "the concurrent policy write must survive")
|
||||
}
|
||||
@@ -88,7 +88,7 @@ func TestUpdateTableAppliesWithoutContention(t *testing.T) {
|
||||
fs, m := startRenameManager(t)
|
||||
|
||||
location := "s3://" + renameTestBucket + "/ns/t/metadata/v4.metadata.json"
|
||||
require.NoError(t, m.Execute(context.Background(), NewManagerClient(fs.client), "UpdateTable",
|
||||
require.NoError(t, m.Execute(context.Background(), NewManagerClient(fs.Client), "UpdateTable",
|
||||
updateTableRequest(t, location, 4), nil, ""))
|
||||
|
||||
got, err := runGetTable(t, m, fs, "ns", "t")
|
||||
|
||||
@@ -52,14 +52,7 @@ func (h *S3TablesHandler) handleCreateView(w http.ResponseWriter, r *http.Reques
|
||||
|
||||
// Check if namespace exists
|
||||
namespacePath := GetNamespacePath(bucketName, namespaceName)
|
||||
var namespaceMetadata namespaceMetadata
|
||||
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
data, err := h.getExtendedAttribute(r.Context(), client, namespacePath, ExtendedKeyMetadata)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return json.Unmarshal(data, &namespaceMetadata)
|
||||
})
|
||||
namespaceMetadata, err := h.loadNamespaceMetadata(r.Context(), filerClient, bucketName, namespaceName)
|
||||
if err != nil {
|
||||
if errors.Is(err, filer_pb.ErrNotFound) {
|
||||
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchNamespace, fmt.Sprintf("namespace %s not found", namespaceName))
|
||||
@@ -77,6 +70,14 @@ func (h *S3TablesHandler) handleCreateView(w http.ResponseWriter, r *http.Reques
|
||||
return err
|
||||
}
|
||||
|
||||
// A view is Iceberg metadata, so it belongs only in a bucket that holds
|
||||
// Iceberg tables.
|
||||
if bucketMetadata.Format != "" && bucketMetadata.Format != FormatIceberg {
|
||||
message := fmt.Sprintf("table bucket %s holds %s tables and cannot hold views", bucketName, bucketMetadata.Format)
|
||||
h.writeError(w, http.StatusConflict, ErrCodeConflict, message)
|
||||
return fmt.Errorf("%s", message)
|
||||
}
|
||||
|
||||
bucketARN := h.generateTableBucketARN(bucketMetadata.OwnerAccountID, bucketName)
|
||||
if !h.authorizeViewOp(r, "CreateView", accountID, namespaceMetadata.OwnerAccountID, bucketMetadata.OwnerAccountID, namespacePolicy, bucketPolicy, bucketARN, bucketName, namespaceName, viewName, bucketTags) {
|
||||
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to create view in this namespace")
|
||||
@@ -93,7 +94,7 @@ func (h *S3TablesHandler) handleCreateView(w http.ResponseWriter, r *http.Reques
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if entryType(entry.Extended) != EntryTypeView {
|
||||
if EntryType(entry.Extended) != EntryTypeView {
|
||||
existingIsTable = true
|
||||
return nil
|
||||
}
|
||||
@@ -125,7 +126,7 @@ func (h *S3TablesHandler) handleCreateView(w http.ResponseWriter, r *http.Reques
|
||||
metadata := &tableMetadataInternal{
|
||||
Name: viewName,
|
||||
Namespace: namespaceName,
|
||||
Format: "ICEBERG",
|
||||
Format: FormatIceberg,
|
||||
CreatedAt: now,
|
||||
ModifiedAt: now,
|
||||
OwnerAccountID: namespaceMetadata.OwnerAccountID,
|
||||
@@ -185,7 +186,7 @@ func (h *S3TablesHandler) handleGetView(w http.ResponseWriter, r *http.Request,
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if entryType(entry.Extended) != EntryTypeView {
|
||||
if EntryType(entry.Extended) != EntryTypeView {
|
||||
return filer_pb.ErrNotFound
|
||||
}
|
||||
data, ok := entry.Extended[ExtendedKeyMetadata]
|
||||
@@ -354,7 +355,7 @@ func (h *S3TablesHandler) listViewsInNamespace(r *http.Request, client filer_pb.
|
||||
continue
|
||||
}
|
||||
// Only include view entries; skip tables and untagged entries.
|
||||
if entryType(entry.Entry.Extended) != EntryTypeView {
|
||||
if EntryType(entry.Entry.Extended) != EntryTypeView {
|
||||
continue
|
||||
}
|
||||
data, ok := entry.Entry.Extended[ExtendedKeyMetadata]
|
||||
@@ -411,7 +412,7 @@ func (h *S3TablesHandler) handleUpdateView(w http.ResponseWriter, r *http.Reques
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if entryType(entry.Extended) != EntryTypeView {
|
||||
if EntryType(entry.Extended) != EntryTypeView {
|
||||
return filer_pb.ErrNotFound
|
||||
}
|
||||
data, ok := entry.Extended[ExtendedKeyMetadata]
|
||||
@@ -516,7 +517,7 @@ func (h *S3TablesHandler) handleDeleteView(w http.ResponseWriter, r *http.Reques
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if entryType(entry.Extended) != EntryTypeView {
|
||||
if EntryType(entry.Extended) != EntryTypeView {
|
||||
return filer_pb.ErrNotFound
|
||||
}
|
||||
data, ok := entry.Extended[ExtendedKeyMetadata]
|
||||
|
||||
@@ -6,12 +6,14 @@ import (
|
||||
"path"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// seedView adds a view alongside the table the rename harness already creates.
|
||||
func seedView(t *testing.T, fs *memFilerServer, name string) {
|
||||
func seedView(t *testing.T, fs *s3tablestest.MemFiler, name string) {
|
||||
t.Helper()
|
||||
|
||||
viewMeta, err := json.Marshal(tableMetadataInternal{
|
||||
@@ -24,19 +26,19 @@ func seedView(t *testing.T, fs *memFilerServer, name string) {
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), name, map[string][]byte{
|
||||
fs.Put(GetNamespacePath(renameTestBucket, "ns"), name, map[string][]byte{
|
||||
ExtendedKeyMetadata: viewMeta,
|
||||
ExtendedKeyMetadataVersion: []byte("1"),
|
||||
ExtendedKeyEntryType: []byte(EntryTypeView),
|
||||
})
|
||||
viewPath := GetTablePath(renameTestBucket, "ns", name)
|
||||
fs.putEntry(viewPath, "metadata", nil)
|
||||
fs.putEntry(path.Join(viewPath, "metadata"), "v1.metadata.json", nil)
|
||||
fs.Put(viewPath, "metadata", nil)
|
||||
fs.Put(path.Join(viewPath, "metadata"), "v1.metadata.json", nil)
|
||||
}
|
||||
|
||||
func runRenameView(t *testing.T, m *Manager, fs *memFilerServer, sourceName, destName string) error {
|
||||
func runRenameView(t *testing.T, m *Manager, fs *s3tablestest.MemFiler, sourceName, destName string) error {
|
||||
t.Helper()
|
||||
return m.Execute(context.Background(), NewManagerClient(fs.client), "RenameView", &RenameTableRequest{
|
||||
return m.Execute(context.Background(), NewManagerClient(fs.Client), "RenameView", &RenameTableRequest{
|
||||
TableBucketARN: mustBucketARN(t),
|
||||
SourceNamespace: []string{"ns"},
|
||||
SourceName: sourceName,
|
||||
@@ -51,9 +53,9 @@ func TestRenameViewMovesCatalogPointer(t *testing.T) {
|
||||
|
||||
require.NoError(t, runRenameView(t, m, fs, "v", "v2"))
|
||||
|
||||
dest := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "v2")
|
||||
dest := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "v2")
|
||||
require.NotNil(t, dest)
|
||||
assert.Equal(t, EntryTypeView, entryType(dest.Extended), "destination must stay a view")
|
||||
assert.Equal(t, EntryTypeView, EntryType(dest.Extended), "destination must stay a view")
|
||||
|
||||
var moved tableMetadataInternal
|
||||
require.NoError(t, json.Unmarshal(dest.Extended[ExtendedKeyMetadata], &moved))
|
||||
@@ -61,11 +63,11 @@ func TestRenameViewMovesCatalogPointer(t *testing.T) {
|
||||
assert.Equal(t, "s3://"+renameTestBucket+"/ns/v/metadata/v1.metadata.json", moved.MetadataLocation,
|
||||
"rename is catalog-only, the metadata stays where it was written")
|
||||
|
||||
src := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "v")
|
||||
src := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "v")
|
||||
require.NotNil(t, src)
|
||||
_, stillListed := src.Extended[ExtendedKeyMetadata]
|
||||
assert.False(t, stillListed, "source name must stop resolving")
|
||||
assert.NotNil(t, fs.getEntry(path.Join(GetTablePath(renameTestBucket, "ns", "v"), "metadata"), "v1.metadata.json"),
|
||||
assert.NotNil(t, fs.Get(path.Join(GetTablePath(renameTestBucket, "ns", "v"), "metadata"), "v1.metadata.json"),
|
||||
"the view's metadata file must survive")
|
||||
}
|
||||
|
||||
@@ -110,18 +112,18 @@ func TestRenameViewAuthorizesAgainstTheViewARN(t *testing.T) {
|
||||
viewARN := "arn:aws:s3tables:" + DefaultRegion + ":" + DefaultAccountID + ":bucket/" + renameTestBucket + "/view/ns/v"
|
||||
viewPolicy := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"` + principal +
|
||||
`","Action":"s3tables:RenameView","Resource":"` + viewARN + `"}]}`
|
||||
view := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "v")
|
||||
view := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "v")
|
||||
require.NotNil(t, view)
|
||||
view.Extended[ExtendedKeyPolicy] = []byte(viewPolicy)
|
||||
|
||||
// Landing in the namespace needs create permission there as well.
|
||||
namespacePolicy := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"` + principal +
|
||||
`","Action":"s3tables:CreateView","Resource":"` + mustBucketARN(t) + `"}]}`
|
||||
namespace := fs.getEntry(GetTableBucketPath(renameTestBucket), "ns")
|
||||
namespace := fs.Get(GetTableBucketPath(renameTestBucket), "ns")
|
||||
require.NotNil(t, namespace)
|
||||
namespace.Extended[ExtendedKeyPolicy] = []byte(namespacePolicy)
|
||||
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.client), "RenameView", &RenameTableRequest{
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.Client), "RenameView", &RenameTableRequest{
|
||||
TableBucketARN: mustBucketARN(t),
|
||||
SourceNamespace: []string{"ns"},
|
||||
SourceName: "v",
|
||||
@@ -130,5 +132,5 @@ func TestRenameViewAuthorizesAgainstTheViewARN(t *testing.T) {
|
||||
}, nil, principal)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.NotNil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "v2"))
|
||||
assert.NotNil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "v2"))
|
||||
}
|
||||
@@ -16,8 +16,8 @@ func TestEntryType(t *testing.T) {
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
if got := entryType(c.extended); got != c.want {
|
||||
t.Fatalf("entryType() = %q, want %q", got, c.want)
|
||||
if got := EntryType(c.extended); got != c.want {
|
||||
t.Fatalf("EntryType() = %q, want %q", got, c.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@@ -20,12 +20,29 @@ import (
|
||||
const uuidPattern = `[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}`
|
||||
|
||||
var (
|
||||
// Allowed directories in an Iceberg table
|
||||
icebergAllowedDirs = map[string]bool{
|
||||
// Allowed directories in a table. The set is the union of what every
|
||||
// supported format writes, because this runs on the S3 door where the
|
||||
// table's format is not in hand: metadata/ and data/ for Iceberg,
|
||||
// _versions/ and _indices/ for Lance, which also puts its fragments in
|
||||
// data/.
|
||||
tableAllowedDirs = map[string]bool{
|
||||
"metadata": true,
|
||||
"data": true,
|
||||
}
|
||||
|
||||
// A format's own bookkeeping lives in underscore-prefixed directories that
|
||||
// the catalog does not interpret: Lance writes _versions, _transactions,
|
||||
// _indices and _deletions, and enumerating them here would mean guessing at
|
||||
// the next one. Iceberg writes none, so admitting them costs it nothing.
|
||||
formatOwnedDirPattern = regexp.MustCompile(`^_[a-z][a-z0-9_]*$`)
|
||||
|
||||
// Marker files a Lance table keeps at its root to record that it is
|
||||
// reserved but not yet written, or deregistered but still on storage.
|
||||
lanceMarkerFiles = map[string]bool{
|
||||
".lance-reserved": true,
|
||||
".lance-deregistered": true,
|
||||
}
|
||||
|
||||
// Patterns for valid metadata files.
|
||||
//
|
||||
// Note: Iceberg engines (Flink/Spark/Trino, plus different Iceberg versions)
|
||||
@@ -58,6 +75,7 @@ var (
|
||||
regexp.MustCompile(`^[^/]+\.parquet$`), // Parquet files
|
||||
regexp.MustCompile(`^[^/]+\.orc$`), // ORC files
|
||||
regexp.MustCompile(`^[^/]+\.avro$`), // Avro files
|
||||
regexp.MustCompile(`^[^/]+\.lance$`), // Lance fragments
|
||||
}
|
||||
|
||||
// Data file partition path pattern (e.g., year=2024/month=01/)
|
||||
@@ -91,11 +109,16 @@ func (v *IcebergLayoutValidator) ValidateFilePath(relativePath string) error {
|
||||
|
||||
topDir := parts[0]
|
||||
|
||||
// A Lance table's marker files sit at its root rather than in a directory.
|
||||
if len(parts) == 1 && lanceMarkerFiles[topDir] {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Check if top-level directory is allowed
|
||||
if !icebergAllowedDirs[topDir] {
|
||||
if !tableAllowedDirs[topDir] && !formatOwnedDirPattern.MatchString(topDir) {
|
||||
return &IcebergLayoutError{
|
||||
Code: ErrCodeInvalidIcebergLayout,
|
||||
Message: "files must be placed in 'metadata/' or 'data/' directories",
|
||||
Message: "files must be placed in 'metadata/', 'data/' or a format-owned '_' directory",
|
||||
}
|
||||
}
|
||||
|
||||
@@ -119,6 +142,28 @@ func (v *IcebergLayoutValidator) ValidateFilePath(relativePath string) error {
|
||||
return v.validateDataFile(remainingPath)
|
||||
}
|
||||
|
||||
// Everything else is a format-owned directory: the only guard is that the
|
||||
// path stays inside the table.
|
||||
return validateFileSegments(remainingPath)
|
||||
}
|
||||
|
||||
// IsTableMarkerFile reports whether a name is one of the marker files a table
|
||||
// keeps at its root.
|
||||
func IsTableMarkerFile(name string) bool {
|
||||
return lanceMarkerFiles[name]
|
||||
}
|
||||
|
||||
// validateFileSegments rejects traversal and empty segments in a path whose
|
||||
// contents the catalog does not otherwise interpret.
|
||||
func validateFileSegments(path string) error {
|
||||
for _, segment := range strings.Split(path, "/") {
|
||||
if segment == "" || segment == "." || segment == ".." {
|
||||
return &IcebergLayoutError{
|
||||
Code: ErrCodeInvalidIcebergLayout,
|
||||
Message: "invalid path segment in: " + path,
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -100,3 +100,37 @@ func TestIcebergLayoutValidator_AcceptsRealWorldDataFiles(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A Lance dataset writes fragments into data/ and keeps its own bookkeeping in
|
||||
// underscore-prefixed directories. The validator runs on the S3 door, where the
|
||||
// table's format is not in hand, so it has to admit both formats' layouts.
|
||||
func TestValidateFilePathAcceptsLanceLayout(t *testing.T) {
|
||||
v := NewIcebergLayoutValidator()
|
||||
allowed := []string{
|
||||
"data/01111110101110001101011164cadc43919eace9c608107bd9.lance",
|
||||
"_versions/1.manifest",
|
||||
"_versions/9223372036854775806.manifest",
|
||||
"_transactions/0-ddb27ab7-2e5c-42c7-b4bf-265d8a3ff636.txn",
|
||||
"_indices/85814508-ed9a-41f2-b939-2050bb7a0ed5-fts/index.idx",
|
||||
"_deletions/_deletions-1.arrow",
|
||||
".lance-reserved",
|
||||
".lance-deregistered",
|
||||
}
|
||||
for _, path := range allowed {
|
||||
if err := v.ValidateFilePath(path); err != nil {
|
||||
t.Errorf("ValidateFilePath(%q) = %v, want nil", path, err)
|
||||
}
|
||||
}
|
||||
|
||||
rejected := []string{
|
||||
"_versions/../../escape",
|
||||
"_versions//empty",
|
||||
"notadir/file.lance",
|
||||
"random.txt",
|
||||
}
|
||||
for _, path := range rejected {
|
||||
if err := v.ValidateFilePath(path); err == nil {
|
||||
t.Errorf("ValidateFilePath(%q) = nil, want an error", path)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,234 @@
|
||||
// Package s3tablestest provides an in-memory filer for driving S3 Tables and
|
||||
// Lance namespace operations end-to-end without a live cluster.
|
||||
package s3tablestest
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"net"
|
||||
"path"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"google.golang.org/grpc"
|
||||
"google.golang.org/grpc/codes"
|
||||
"google.golang.org/grpc/credentials/insecure"
|
||||
"google.golang.org/grpc/status"
|
||||
)
|
||||
|
||||
// MemFiler is an in-memory filer used to drive Manager operations
|
||||
// end-to-end without a live cluster.
|
||||
type MemFiler struct {
|
||||
filer_pb.UnimplementedSeaweedFilerServer
|
||||
// mu guards entries. The real filer serves concurrent RPCs, and a test that
|
||||
// races writers against each other - the point of an exclusive create - hits
|
||||
// this map from several goroutines at once.
|
||||
mu sync.RWMutex
|
||||
entries map[string]map[string]*filer_pb.Entry // dir -> name -> entry
|
||||
Client filer_pb.SeaweedFilerClient
|
||||
// BeforeUpdate runs once, at the start of the next UpdateEntry, so a test
|
||||
// can land a competing write in a handler's read-to-write window.
|
||||
BeforeUpdate func()
|
||||
}
|
||||
|
||||
func newMemFiler() *MemFiler {
|
||||
return &MemFiler{entries: make(map[string]map[string]*filer_pb.Entry)}
|
||||
}
|
||||
|
||||
func (f *MemFiler) Get(dir, name string) *filer_pb.Entry {
|
||||
f.mu.RLock()
|
||||
defer f.mu.RUnlock()
|
||||
if d, ok := f.entries[dir]; ok {
|
||||
return d[name]
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *MemFiler) Put(dir, name string, extended map[string][]byte) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if _, ok := f.entries[dir]; !ok {
|
||||
f.entries[dir] = make(map[string]*filer_pb.Entry)
|
||||
}
|
||||
f.entries[dir][name] = &filer_pb.Entry{Name: name, IsDirectory: true, Extended: extended}
|
||||
}
|
||||
|
||||
// PutFile adds a file entry with an explicit modification time, which callers
|
||||
// that age entries out (orphan cleanup, expiry) need in order to see them.
|
||||
func (f *MemFiler) PutFile(dir, name string, mtime time.Time) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if _, ok := f.entries[dir]; !ok {
|
||||
f.entries[dir] = make(map[string]*filer_pb.Entry)
|
||||
}
|
||||
f.entries[dir][name] = &filer_pb.Entry{
|
||||
Name: name,
|
||||
Attributes: &filer_pb.FuseAttributes{Mtime: mtime.Unix(), Crtime: mtime.Unix()},
|
||||
}
|
||||
}
|
||||
|
||||
func (f *MemFiler) LookupDirectoryEntry(_ context.Context, req *filer_pb.LookupDirectoryEntryRequest) (*filer_pb.LookupDirectoryEntryResponse, error) {
|
||||
if e := f.Get(req.Directory, req.Name); e != nil {
|
||||
return &filer_pb.LookupDirectoryEntryResponse{Entry: e}, nil
|
||||
}
|
||||
// Carry the sentinel text so filer_pb.LookupEntry maps it to ErrNotFound.
|
||||
return nil, status.Errorf(codes.NotFound, "%s: %s/%s", filer_pb.ErrNotFound.Error(), req.Directory, req.Name)
|
||||
}
|
||||
|
||||
// ListEntries honours prefix, start-from and limit the way the real filer does.
|
||||
// A harness that ignores them makes a paginating caller re-read the first page
|
||||
// forever, which looks like duplicated entries rather than a broken listing.
|
||||
func (f *MemFiler) ListEntries(req *filer_pb.ListEntriesRequest, stream grpc.ServerStreamingServer[filer_pb.ListEntriesResponse]) error {
|
||||
f.mu.RLock()
|
||||
d, ok := f.entries[req.Directory]
|
||||
names := make([]string, 0, len(d))
|
||||
snapshot := make(map[string]*filer_pb.Entry, len(d))
|
||||
for name, entry := range d {
|
||||
names = append(names, name)
|
||||
snapshot[name] = entry
|
||||
}
|
||||
f.mu.RUnlock()
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
sort.Strings(names)
|
||||
|
||||
sent := uint32(0)
|
||||
for _, name := range names {
|
||||
if req.Prefix != "" && !strings.HasPrefix(name, req.Prefix) {
|
||||
continue
|
||||
}
|
||||
if req.StartFromFileName != "" {
|
||||
if name < req.StartFromFileName {
|
||||
continue
|
||||
}
|
||||
if name == req.StartFromFileName && !req.InclusiveStartFrom {
|
||||
continue
|
||||
}
|
||||
}
|
||||
if err := stream.Send(&filer_pb.ListEntriesResponse{Entry: snapshot[name]}); err != nil {
|
||||
return err
|
||||
}
|
||||
sent++
|
||||
if req.Limit > 0 && sent >= req.Limit {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *MemFiler) CreateEntry(_ context.Context, req *filer_pb.CreateEntryRequest) (*filer_pb.CreateEntryResponse, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if _, ok := f.entries[req.Directory]; !ok {
|
||||
f.entries[req.Directory] = make(map[string]*filer_pb.Entry)
|
||||
}
|
||||
// O_EXCL is the filer's put-if-not-exists. Ignoring it here would let a test
|
||||
// that races two writers see both of them win.
|
||||
if _, exists := f.entries[req.Directory][req.Entry.Name]; exists && req.OExcl {
|
||||
return &filer_pb.CreateEntryResponse{ErrorCode: filer_pb.FilerError_ENTRY_ALREADY_EXISTS}, nil
|
||||
}
|
||||
f.entries[req.Directory][req.Entry.Name] = req.Entry
|
||||
return &filer_pb.CreateEntryResponse{}, nil
|
||||
}
|
||||
|
||||
func (f *MemFiler) UpdateEntry(_ context.Context, req *filer_pb.UpdateEntryRequest) (*filer_pb.UpdateEntryResponse, error) {
|
||||
// The hook runs before the lock is taken: its whole purpose is to land a
|
||||
// competing write in the read-to-write window, and that write needs the lock.
|
||||
if hook := f.BeforeUpdate; hook != nil {
|
||||
f.BeforeUpdate = nil
|
||||
hook()
|
||||
}
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
// The real filer validates ExpectedExtended under the per-path lock; without
|
||||
// it here a lost update would look like a success.
|
||||
for key, expected := range req.ExpectedExtended {
|
||||
var actual []byte
|
||||
if d, ok := f.entries[req.Directory]; ok {
|
||||
if existing := d[req.Entry.Name]; existing != nil {
|
||||
actual = existing.Extended[key]
|
||||
}
|
||||
}
|
||||
if !bytes.Equal(actual, expected) {
|
||||
return nil, status.Errorf(codes.FailedPrecondition, "extended attribute %q changed", key)
|
||||
}
|
||||
}
|
||||
if _, ok := f.entries[req.Directory]; !ok {
|
||||
f.entries[req.Directory] = make(map[string]*filer_pb.Entry)
|
||||
}
|
||||
f.entries[req.Directory][req.Entry.Name] = req.Entry
|
||||
return &filer_pb.UpdateEntryResponse{}, nil
|
||||
}
|
||||
|
||||
func (f *MemFiler) DeleteEntry(_ context.Context, req *filer_pb.DeleteEntryRequest) (*filer_pb.DeleteEntryResponse, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if d, ok := f.entries[req.Directory]; ok {
|
||||
delete(d, req.Name)
|
||||
}
|
||||
// Honor recursive data deletion so a regression that wipes the table directory
|
||||
// also drops its metadata/ and data/ children (the data-loss this guards against).
|
||||
if req.IsRecursive && req.IsDeleteData {
|
||||
child := path.Join(req.Directory, req.Name)
|
||||
for dir := range f.entries {
|
||||
if dir == child || strings.HasPrefix(dir, child+"/") {
|
||||
delete(f.entries, dir)
|
||||
}
|
||||
}
|
||||
}
|
||||
return &filer_pb.DeleteEntryResponse{}, nil
|
||||
}
|
||||
|
||||
// GetFilerConfiguration answers with the defaults, so operations that resolve
|
||||
// the buckets directory before touching an entry work against this filer.
|
||||
func (f *MemFiler) GetFilerConfiguration(_ context.Context, _ *filer_pb.GetFilerConfigurationRequest) (*filer_pb.GetFilerConfigurationResponse, error) {
|
||||
return &filer_pb.GetFilerConfigurationResponse{DirBuckets: s3_constants.DefaultBucketsPath}, nil
|
||||
}
|
||||
|
||||
func (f *MemFiler) Ping(_ context.Context, _ *filer_pb.PingRequest) (*filer_pb.PingResponse, error) {
|
||||
now := time.Now().UnixNano()
|
||||
return &filer_pb.PingResponse{StartTimeNs: now, RemoteTimeNs: now, StopTimeNs: now}, nil
|
||||
}
|
||||
|
||||
func Start(t *testing.T) *MemFiler {
|
||||
t.Helper()
|
||||
fs := newMemFiler()
|
||||
|
||||
listener, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatalf("start filer: %v", err)
|
||||
}
|
||||
|
||||
server := grpc.NewServer()
|
||||
filer_pb.RegisterSeaweedFilerServer(server, fs)
|
||||
go func() { _ = server.Serve(listener) }()
|
||||
t.Cleanup(server.GracefulStop)
|
||||
|
||||
conn, err := grpc.NewClient(listener.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
|
||||
if err != nil {
|
||||
t.Fatalf("start filer: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = conn.Close() })
|
||||
|
||||
fs.Client = filer_pb.NewSeaweedFilerClient(conn)
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
pingCtx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
_, err := fs.Client.Ping(pingCtx, &filer_pb.PingRequest{})
|
||||
cancel()
|
||||
if err == nil {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("filer not ready: %v", err)
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
return fs
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package s3tables
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
@@ -12,11 +13,17 @@ type TableBucket struct {
|
||||
Name string `json:"name"`
|
||||
OwnerAccountID string `json:"ownerAccountId"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
Format string `json:"format,omitempty"`
|
||||
}
|
||||
|
||||
type CreateTableBucketRequest struct {
|
||||
Name string `json:"name"`
|
||||
Tags map[string]string `json:"tags,omitempty"`
|
||||
// Format is the table format this bucket holds. A bucket is a catalog and a
|
||||
// catalog serves one protocol, so declaring it here is what lets a caller be
|
||||
// told where to connect. Empty means ICEBERG, which is what AWS S3 Tables
|
||||
// serves and therefore what an SDK that has never heard of this field means.
|
||||
Format string `json:"format,omitempty"`
|
||||
}
|
||||
|
||||
type CreateTableBucketResponse struct {
|
||||
@@ -32,6 +39,9 @@ type GetTableBucketResponse struct {
|
||||
Name string `json:"name"`
|
||||
OwnerAccountID string `json:"ownerAccountId"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
// Format is empty for a bucket created before formats were declared. Such a
|
||||
// bucket accepts any format, which is what it did when it was made.
|
||||
Format string `json:"format,omitempty"`
|
||||
}
|
||||
|
||||
type ListTableBucketsRequest struct {
|
||||
@@ -44,6 +54,7 @@ type TableBucketSummary struct {
|
||||
ARN string `json:"arn"`
|
||||
Name string `json:"name"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
Format string `json:"format,omitempty"`
|
||||
}
|
||||
|
||||
type ListTableBucketsResponse struct {
|
||||
@@ -235,9 +246,12 @@ type ListTablesRequest struct {
|
||||
}
|
||||
|
||||
type TableSummary struct {
|
||||
Name string `json:"name"`
|
||||
TableARN string `json:"tableARN"`
|
||||
Namespace []string `json:"namespace"`
|
||||
Name string `json:"name"`
|
||||
TableARN string `json:"tableARN"`
|
||||
Namespace []string `json:"namespace"`
|
||||
// Format lets a caller tell an Iceberg table from a catalog-only one without
|
||||
// a GetTable per row. AWS omits it; listing a mixed catalog needs it.
|
||||
Format string `json:"format,omitempty"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
ModifiedAt time.Time `json:"modifiedAt"`
|
||||
MetadataLocation string `json:"metadataLocation,omitempty"`
|
||||
@@ -562,6 +576,36 @@ func (e *S3TablesError) Error() string {
|
||||
return e.Message
|
||||
}
|
||||
|
||||
// Table formats a catalog entry may declare.
|
||||
//
|
||||
// ICEBERG tables carry metadata the catalog maintains and the maintenance
|
||||
// worker rewrites. LANCE is catalog-only: the entry records a name and the
|
||||
// dataset root in MetadataLocation, and the Lance client owns every byte under
|
||||
// it. Nothing in this package interprets a catalog-only table's files.
|
||||
const (
|
||||
FormatIceberg = "ICEBERG"
|
||||
FormatLance = "LANCE"
|
||||
)
|
||||
|
||||
// IsCatalogOnlyFormat reports whether the catalog only records where a table of
|
||||
// this format lives, without understanding its files.
|
||||
func IsCatalogOnlyFormat(format string) bool {
|
||||
return format == FormatLance
|
||||
}
|
||||
|
||||
// NormalizeFormat folds a caller's spelling onto the canonical one and reports
|
||||
// whether it names a format this catalog serves.
|
||||
func NormalizeFormat(format string) (string, bool) {
|
||||
switch strings.ToUpper(strings.TrimSpace(format)) {
|
||||
case FormatIceberg:
|
||||
return FormatIceberg, true
|
||||
case FormatLance:
|
||||
return FormatLance, true
|
||||
default:
|
||||
return "", false
|
||||
}
|
||||
}
|
||||
|
||||
// Error codes
|
||||
const (
|
||||
ErrCodeBucketAlreadyExists = "BucketAlreadyExists"
|
||||
|
||||
@@ -142,6 +142,7 @@ type tableBucketMetadata struct {
|
||||
Name string `json:"name"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
OwnerAccountID string `json:"ownerAccountId"`
|
||||
Format string `json:"format,omitempty"`
|
||||
}
|
||||
|
||||
// namespaceMetadata stores metadata for a namespace
|
||||
@@ -175,10 +176,10 @@ func IsTableBucketEntry(entry *filer_pb.Entry) bool {
|
||||
return ok
|
||||
}
|
||||
|
||||
// entryType returns the entry-type marker for a catalog entry. Tables and views
|
||||
// EntryType returns the entry-type marker for a catalog entry. Tables and views
|
||||
// share the same on-disk layout; the marker distinguishes them. An absent marker
|
||||
// means table for back-compat.
|
||||
func entryType(extended map[string][]byte) string {
|
||||
func EntryType(extended map[string][]byte) string {
|
||||
if extended == nil {
|
||||
return EntryTypeTable
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user