Files
seaweedfs/weed/s3api/lance/handlers_test.go
T
Chris LuandClaude Fable 5.1 2fe480a6e2 S3 Vectors: the gateway resolves the table it sends to weed-vector (#12468)
* lance catalog: in-process DescribeTable for trusted callers

DescribeTable and DescribeTableInternal resolve a table's location and
storage options without the HTTP layer; the internal form runs the lookup
through a trusted manager so the caller's catalog grants do not gate a
resolution its own permission already authorized. Same code as the
Enterprise repository.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector.proto: Table carries its location and storage options

The gateway resolves the table it addresses, so the engine can open it
without a describe round trip of its own. Both fields are optional; an
empty location means the engine looks the table up as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: test the in-process describe

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: resolve the table before calling the engine

The gateway describes the table in process, through the trusted catalog
manager, and sends its location and storage options with every engine
call. Storage options are vended for the caller's principal, as the
Enterprise router does for queries. A catalog failure answers with the
code the engine would have given.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: open a table at the location the gateway sent

A request whose table carries a location is opened there, with the
storage options the gateway vended over the engine's fallback; the
catalog is described only for a table that comes without one, as from
an older gateway. dataset::open_at and dataset::drop_at take a
description the caller already holds.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors e2e: run the engine without a catalog

VECTOR_ENGINE_NO_CATALOG=1 gives the engine an unreachable catalog URL;
every CI leg sets it, so the suite proves the gateway resolves each table
it sends.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: an internal describe may write a vector index

With credential vending on, the options vended to the engine followed the
caller's s3tables grants, which never allow writing the vector namespace,
so every write the gateway sent would have run with read-only credentials
and reads would have lost their version lease. The engine is that
namespace's writer and the caller's vector permission authorized the
operation, so an internal describe of a vector table vends write-capable
options; other namespaces keep the UpdateTable check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: resolve tables for the engine only while the catalog runs

With -port.lance=0 the gateway has no catalog to resolve through; the
engine then keeps looking tables up itself, as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: a deregistered table describes as NoSuchTable

So the gateway answers NotFoundException for it, not InternalFailure.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: trust a sent location only for a colocated vector index

A caller that reaches the engine could otherwise name any storage the
engine's credentials cover. A location is taken from the request only
when the id names a vector index and the location is its colocated path,
which the engine writes already; anything else is described through the
catalog as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: scope the read cache by storage endpoint

The cache was keyed by bucket and path alone, so the same path reached
through another endpoint would have shared entries. Each open wraps the
store with the cache scoped to the endpoint its options name.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep a configured endpoint over one a request names

A caller that reaches the engine could otherwise send its fallback
credentials to an endpoint of the caller's choosing. When the engine's
--storage-option names an aws_endpoint, a table whose request names
another one is described through the catalog instead.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: use request options only where credentials cannot leave

A request's storage options are taken only when they hold the keys a
gateway's describe produces, so a proxy or an endpoint alias cannot route
the engine's credentials elsewhere, and an endpoint only when it is the
configured one or the engine holds no credentials to send. Anything else
sends the table to the catalog.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors e2e: give the engine its endpoint with its credentials

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: a drop checks the request's options like an open

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: an unpinned endpoint needs the request to sign for itself

Without configured credentials the S3 store still signs with the AWS
default chain, so an endpoint from a request is taken only when the
request skips signing or brings its own credentials.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: a trailing slash does not unpin the endpoint, and say so

A configured aws_endpoint written with a trailing slash matched nothing
the gateway sends, so every table fell back to the catalog in silence.
The comparison ignores the slash, and a fallback for a resolved table is
logged at debug.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: resolved by the gateway, or refused

The engine no longer has a catalog: --namespace and --filer-signing-key
are gone, a request whose table names no location is InvalidArgument, and
one whose location is not the index's own or whose storage options the
engine's credentials must not be used with is PermissionDenied. A drop
removes the colocated path it was given.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: the catalog resolves vector tables whether or not its port is served

newLanceServer builds the Lance catalog for the S3 Vectors API; the
listener on -port.lance is optional.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors tests: start the engine without a catalog

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: the aux listener test builds the catalog before serving it

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: an unpinned endpoint takes only credentials the request signs with alone

Lance fills an option a request leaves out from the engine's environment,
under any of object_store's names for it (TOKEN, SESSION_TOKEN, ...), and
asks the AWS default chain when the key pair is incomplete. A request that
named its own endpoint with only an access key id, or a key pair while the
engine's environment held a session token, would have sent the engine's
credentials there. Such a request now needs a full key pair, and its own
session token whenever the environment would supply one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: accept --namespace and --filer-signing-key, ignored

Both flags shipped in 4.52 and are documented, so dropping them made an
existing command line exit at startup with an unexpected-argument error.
They are kept hidden and ignored, with a warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors design: the engine is started without a catalog

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: the trust check takes the environment it reads

The options test read the process environment through signs_alone, so a
host exporting AWS_SESSION_TOKEN or TOKEN failed it. The caller now
passes the variable names in; describe passes the engine's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: log both endpoints when a request's options are refused

The gateway turns the PermissionDenied into a bare AccessDenied and logs
nothing, so a pinned aws_endpoint written in another form than the one the
gateway sends (host case, a default port, an IP for a name) failed every
call with no trace of which value differed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: the Lance catalog sets the credential role whether or not Iceberg runs

-iceberg.credentialRole reached the S3 server only while the Iceberg
listener started, so with -port.iceberg=0 the Lance catalog installed its
vendor but vending stayed off, and the vector engine got no credentials for
the tables the gateway resolves. Both catalogs now set the role through
one helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: set the credential role once, before either catalog serves

Both catalogs wrote the role as they were built; the second write landed
while the first was already serving requests that read it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-06 19:39:09 -08:00

556 lines
23 KiB
Go

package lance
import (
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest"
)
// openAuthenticator stands in for a gateway with no IAM configured, which is the
// mode the namespace falls open in.
type openAuthenticator struct{}
func (openAuthenticator) AuthenticateRequest(*http.Request) (string, interface{}, s3err.ErrorCode) {
return s3tables.DefaultAccountID, nil, s3err.ErrNone
}
func (openAuthenticator) DefaultAllow() bool { return true }
func (openAuthenticator) AuthEnabled() bool { return false }
// signedAuthenticator admits the same caller on a gateway that checks
// credentials.
type signedAuthenticator struct{ openAuthenticator }
func (signedAuthenticator) AuthEnabled() bool { return true }
type testHarness struct {
router *mux.Router
filer *s3tablestest.MemFiler
admin *s3tables.Manager
server *Server
}
func newTestHarness(t *testing.T) *testHarness {
t.Helper()
filer := s3tablestest.Start(t)
server := NewServer(s3tables.NewManagerClient(filer.Client), openAuthenticator{})
server.SetS3Endpoint("http://127.0.0.1:8333")
server.SetS3Region(s3tables.DefaultRegion)
router := mux.NewRouter().SkipClean(true)
server.RegisterRoutes(router)
// Table buckets are created by an operator, not by a namespace client, so
// the harness seeds them the way the shell and admin console would.
admin := s3tables.NewManager()
admin.SetTrusted(true)
return &testHarness{router: router, filer: filer, admin: admin, server: server}
}
// createBucket seeds a table bucket the Lance namespace can then be pointed at.
// An empty format strips the declaration afterwards, which is the only way to
// get the shape a bucket made before formats existed has: one that still
// accepts either format.
func (h *testHarness) createBucket(t *testing.T, name, format string) {
t.Helper()
var resp s3tables.CreateTableBucketResponse
declared := format
if declared == "" {
declared = s3tables.FormatIceberg
}
err := h.admin.Execute(t.Context(), s3tables.NewManagerClient(h.filer.Client), "CreateTableBucket",
&s3tables.CreateTableBucketRequest{Name: name, Format: declared}, &resp, "")
if err != nil {
t.Fatalf("create table bucket %s: %v", name, err)
}
if format == "" {
h.undeclareBucket(t, name)
}
}
// undeclareBucket removes a bucket's format from its metadata, ageing it back to
// what the filer holds for a bucket created before the field existed.
func (h *testHarness) undeclareBucket(t *testing.T, name string) {
t.Helper()
entry := h.filer.Get(s3tables.TablesPath, name)
if entry == nil {
t.Fatalf("table bucket %s is not in the filer", name)
}
var metadata map[string]any
if err := json.Unmarshal(entry.Extended[s3tables.ExtendedKeyMetadata], &metadata); err != nil {
t.Fatalf("read bucket metadata: %v", err)
}
delete(metadata, "format")
updated, err := json.Marshal(metadata)
if err != nil {
t.Fatalf("write bucket metadata: %v", err)
}
extended := map[string][]byte{}
for key, value := range entry.Extended {
extended[key] = value
}
extended[s3tables.ExtendedKeyMetadata] = updated
h.filer.Put(s3tables.TablesPath, name, extended)
}
func (h *testHarness) bucketARN(t *testing.T, name string) string {
t.Helper()
arn, err := s3tables.BuildBucketARN(s3tables.DefaultRegion, s3tables.DefaultAccountID, name)
if err != nil {
t.Fatalf("build arn: %v", err)
}
return arn
}
func (h *testHarness) do(t *testing.T, method, target, body string) *httptest.ResponseRecorder {
t.Helper()
var reader *strings.Reader
if body == "" {
reader = strings.NewReader("")
} else {
reader = strings.NewReader(body)
}
req := httptest.NewRequest(method, target, reader)
recorder := httptest.NewRecorder()
h.router.ServeHTTP(recorder, req)
return recorder
}
func (h *testHarness) mustDo(t *testing.T, method, target, body string, want int) *httptest.ResponseRecorder {
t.Helper()
recorder := h.do(t, method, target, body)
if recorder.Code != want {
t.Fatalf("%s %s = %d (%s), want %d", method, target, recorder.Code, recorder.Body.String(), want)
}
return recorder
}
func decode[T any](t *testing.T, recorder *httptest.ResponseRecorder) T {
t.Helper()
var out T
if err := json.Unmarshal(recorder.Body.Bytes(), &out); err != nil {
t.Fatalf("decode %s: %v", recorder.Body.String(), err)
}
return out
}
// While the S3 port takes unsigned requests the catalog says so, and a client
// with no credentials of its own still reaches the data. Once auth is on it
// has to sign again.
func TestStorageOptionsSkipSignatureOnlyWhileS3AuthIsOff(t *testing.T) {
h := newTestHarness(t)
h.createBucket(t, "open", s3tables.FormatLance)
h.mustDo(t, http.MethodPost, "/v1/namespace/open$ns/create", `{}`, http.StatusOK)
declared := decode[DeclareTableResponse](t,
h.mustDo(t, http.MethodPost, "/v1/table/open$ns$t/declare", `{}`, http.StatusOK))
if declared.StorageOptions["aws_skip_signature"] != "true" {
t.Fatalf("auth off: storage options = %v, want aws_skip_signature", declared.StorageOptions)
}
h.server.authenticator = signedAuthenticator{}
described := decode[DescribeTableResponse](t,
h.mustDo(t, http.MethodPost, "/v1/table/open$ns$t/describe", `{}`, http.StatusOK))
if _, ok := described.StorageOptions["aws_skip_signature"]; ok {
t.Fatalf("auth on: storage options = %v, want no aws_skip_signature", described.StorageOptions)
}
}
// The lifecycle a Lance client drives: create the namespace, declare the table
// before any data exists, resolve it to a location, then deregister and bring it
// back by registering the same location.
func TestTableLifecycle(t *testing.T) {
h := newTestHarness(t)
h.createBucket(t, "analytics", s3tables.FormatLance)
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
declared := decode[DeclareTableResponse](t,
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/declare", `{}`, http.StatusOK))
if declared.Location != "s3://analytics/sales/orders" {
t.Fatalf("declared location = %q", declared.Location)
}
// The endpoint is plaintext, so a client that does not get allow_http fails
// with what looks like a credential error.
if declared.StorageOptions["aws_endpoint"] != "http://127.0.0.1:8333" ||
declared.StorageOptions["allow_http"] != "true" {
t.Fatalf("storage options = %v", declared.StorageOptions)
}
described := decode[DescribeTableResponse](t,
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/describe?check_declared=true", `{}`, http.StatusOK))
if described.Location != declared.Location {
t.Fatalf("described location = %q, want %q", described.Location, declared.Location)
}
if described.IsOnlyDeclared == nil || !*described.IsOnlyDeclared {
t.Fatalf("is_only_declared = %v, want true for a table with no data", described.IsOnlyDeclared)
}
listed := decode[ListTablesResponse](t,
h.mustDo(t, http.MethodGet, "/v1/namespace/analytics$sales/table/list", "", http.StatusOK))
if len(listed.Tables) != 1 || listed.Tables[0] != "analytics$sales$orders" {
t.Fatalf("listed tables = %v, want the full identifier", listed.Tables)
}
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/exists", `{}`, http.StatusOK)
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/deregister", `{}`, http.StatusOK)
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/exists", `{}`, http.StatusNotFound)
afterDrop := decode[ListTablesResponse](t,
h.mustDo(t, http.MethodGet, "/v1/namespace/analytics$sales/table/list", "", http.StatusOK))
if len(afterDrop.Tables) != 0 {
t.Fatalf("deregistered table still listed: %v", afterDrop.Tables)
}
// Deregistering preserves the data. The catalog entry is the dataset
// directory, so dropping it would take the dataset with it.
if h.filer.Get(s3tables.GetNamespacePath("analytics", "sales"), "orders") == nil {
t.Fatal("deregister deleted the dataset directory")
}
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/register",
`{"location":"s3://analytics/sales/orders"}`, http.StatusOK)
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/exists", `{}`, http.StatusOK)
// Dropping is the operation that does remove the data.
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/drop", `{}`, http.StatusOK)
if h.filer.Get(s3tables.GetNamespacePath("analytics", "sales"), "orders") != nil {
t.Fatal("drop left the dataset directory behind")
}
}
// Repointing a registered name at another dataset must not take the dataset it
// used to name with it.
func TestRegisterOverwriteKeepsTheOldDataset(t *testing.T) {
h := newTestHarness(t)
h.createBucket(t, "analytics", s3tables.FormatLance)
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/declare", `{}`, http.StatusOK)
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$archive/declare", `{}`, http.StatusOK)
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/register",
`{"location":"s3://analytics/sales/archive","mode":"Overwrite"}`, http.StatusOK)
described := decode[DescribeTableResponse](t,
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/describe", `{}`, http.StatusOK))
if described.Location != "s3://analytics/sales/archive" {
t.Fatalf("location after repointing = %q", described.Location)
}
if h.filer.Get(s3tables.GetNamespacePath("analytics", "sales"), "orders") == nil {
t.Fatal("repointing deleted the dataset the name used to hold")
}
}
// A required list field answers empty rather than null, which a generated
// client may decode differently.
func TestListAllTablesIsNeverNull(t *testing.T) {
h := newTestHarness(t)
h.createBucket(t, "analytics", s3tables.FormatLance)
body := h.mustDo(t, http.MethodGet, "/v1/table", "", http.StatusOK).Body.String()
if strings.Contains(body, `"tables":null`) {
t.Fatalf("ListAllTables returned null for a required field: %s", body)
}
}
func TestNamespaceListing(t *testing.T) {
h := newTestHarness(t)
h.createBucket(t, "analytics", s3tables.FormatLance)
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$finance/create", `{}`, http.StatusOK)
roots := decode[ListNamespacesResponse](t,
h.mustDo(t, http.MethodGet, "/v1/namespace/$/list", "", http.StatusOK))
if len(roots.Namespaces) != 1 || roots.Namespaces[0] != "analytics" {
t.Fatalf("root listing = %v, want the table buckets", roots.Namespaces)
}
children := decode[ListNamespacesResponse](t,
h.mustDo(t, http.MethodGet, "/v1/namespace/analytics/list", "", http.StatusOK))
if len(children.Namespaces) != 2 {
t.Fatalf("bucket listing = %v, want two namespaces", children.Namespaces)
}
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/exists", `{}`, http.StatusOK)
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$nope/exists", `{}`, http.StatusNotFound)
// Fail mode reports a missing namespace as 400 on this operation, which is
// what the spec asks for; Skip reports success.
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$nope/drop", `{}`, http.StatusBadRequest)
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$nope/drop", `{"mode":"Skip"}`, http.StatusNoContent)
}
// Storage flattens a namespace's parts, so creating a$b without a would leave
// an intermediate that listing derives from the name and describe denies
// exists. The spec asks for NamespaceNotFound, which keeps them consistent.
func TestCreateNamespaceRequiresItsParent(t *testing.T) {
h := newTestHarness(t)
h.createBucket(t, "analytics", s3tables.FormatLance)
recorder := h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$missing$child/create", `{}`, http.StatusNotFound)
if got := decode[errorResponse](t, recorder); got.Code != codeNamespaceNotFound {
t.Fatalf("error code = %d, want %d", got.Code, codeNamespaceNotFound)
}
// With the parent in place the child is fine, and both are then listed and
// describable.
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$parent/create", `{}`, http.StatusOK)
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$parent$child/create", `{}`, http.StatusOK)
listed := decode[ListNamespacesResponse](t,
h.mustDo(t, http.MethodGet, "/v1/namespace/analytics/list", "", http.StatusOK))
for _, name := range listed.Namespaces {
path := "/v1/namespace/analytics$" + name
h.mustDo(t, http.MethodPost, path+"/exists", `{}`, http.StatusOK)
}
}
func TestCreateNamespaceModes(t *testing.T) {
h := newTestHarness(t)
h.createBucket(t, "analytics", s3tables.FormatLance)
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusConflict)
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{"mode":"ExistOk"}`, http.StatusOK)
// snake_case is the other spelling the spec accepts.
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{"mode":"exist_ok"}`, http.StatusOK)
h.mustDo(t, http.MethodPost, "/v1/namespace/$/create", `{}`, http.StatusBadRequest)
}
// A Lance client must never resolve an Iceberg table's location, or it would
// write a dataset over a table another engine owns.
func TestIcebergTablesAreInvisible(t *testing.T) {
h := newTestHarness(t)
// Undeclared, because a bucket that declares one format cannot hold the
// other - and mixing is exactly what this test needs to prove is hidden.
h.createBucket(t, "analytics", "")
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$vectors/declare", `{}`, http.StatusOK)
manager := h.admin
var created s3tables.CreateTableResponse
err := manager.Execute(t.Context(), s3tables.NewManagerClient(h.filer.Client), "CreateTable", &s3tables.CreateTableRequest{
TableBucketARN: h.bucketARN(t, "analytics"),
Namespace: []string{"sales"},
Name: "ledger",
Format: s3tables.FormatIceberg,
MetadataLocation: "s3://analytics/sales/ledger/metadata/v1.metadata.json",
}, &created, "")
if err != nil {
t.Fatalf("create iceberg table: %v", err)
}
listed := decode[ListTablesResponse](t,
h.mustDo(t, http.MethodGet, "/v1/namespace/analytics$sales/table/list", "", http.StatusOK))
if len(listed.Tables) != 1 || listed.Tables[0] != "analytics$sales$vectors" {
t.Fatalf("listing = %v, want only the lance table", listed.Tables)
}
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$ledger/describe", `{}`, http.StatusNotFound)
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$ledger/exists", `{}`, http.StatusNotFound)
// Declaring over the Iceberg table must not quietly succeed and hand the
// Lance client a directory another format owns.
recorder := h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$ledger/declare", `{}`, http.StatusConflict)
if got := decode[errorResponse](t, recorder); got.Code != codeTableAlreadyExists {
t.Fatalf("error code = %d, want %d", got.Code, codeTableAlreadyExists)
}
}
// The route and the body naming different objects is a bad request, not a silent
// preference for one of them.
func TestRouteAndBodyMustAgree(t *testing.T) {
h := newTestHarness(t)
h.createBucket(t, "analytics", s3tables.FormatLance)
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
recorder := h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/declare",
`{"id":["analytics","sales","other"]}`, http.StatusBadRequest)
if got := decode[errorResponse](t, recorder); got.Code != codeInvalidInput {
t.Fatalf("error code = %d, want %d", got.Code, codeInvalidInput)
}
}
// A client-supplied location must resolve inside its own bucket. Without this,
// a "../"-laden or cross-bucket location escapes when the marker path is joined
// under /buckets, letting the request reach another tenant's namespace, or
// outside /buckets entirely, and auto-create the parent directories on the way.
func TestLocationOutsideBucketIsRejected(t *testing.T) {
cases := []struct {
name string
location string
}{
{"another bucket", "s3://victim/secret"},
{"traversal into another bucket", "s3://analytics/../victim/secret"},
{"traversal above the buckets root", "s3://analytics/../../etc/cron.d"},
{"not an s3 uri", "../../etc/cron.d"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
dir := s3tables.TableDataDirFromMetadataLocation(tc.location)
body := `{"location":"` + tc.location + `"}`
// declare writes .lance-reserved at the location: rejection must leave
// nothing behind at the escaped directory.
t.Run("declare", func(t *testing.T) {
h := newTestHarness(t)
h.createBucket(t, "analytics", s3tables.FormatLance)
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
recorder := h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/declare", body, http.StatusBadRequest)
if got := decode[errorResponse](t, recorder); got.Code != codeInvalidInput {
t.Fatalf("error code = %d, want %d", got.Code, codeInvalidInput)
}
if dir != "" && h.filer.Get(dir, reservedMarker) != nil {
t.Fatalf("declare wrote %s into %s despite rejection", reservedMarker, dir)
}
})
// register removes .lance-deregistered at the location: rejection must
// leave a victim's marker in place rather than un-hiding their table.
t.Run("register", func(t *testing.T) {
h := newTestHarness(t)
h.createBucket(t, "analytics", s3tables.FormatLance)
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
if dir != "" {
h.filer.PutFile(dir, deregisteredMarker, time.Unix(1, 0))
}
recorder := h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/register", body, http.StatusBadRequest)
if got := decode[errorResponse](t, recorder); got.Code != codeInvalidInput {
t.Fatalf("error code = %d, want %d", got.Code, codeInvalidInput)
}
if dir != "" && h.filer.Get(dir, deregisteredMarker) == nil {
t.Fatalf("register removed %s from %s despite rejection", deregisteredMarker, dir)
}
})
})
}
}
// The data plane needs Lance format support that does not exist in Go, so it
// answers with the spec's own Unsupported code rather than a bare 404.
func TestDataPlaneIsUnsupported(t *testing.T) {
h := newTestHarness(t)
recorder := h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/query", `{}`, http.StatusNotImplemented)
if got := decode[errorResponse](t, recorder); got.Code != codeUnsupported {
t.Fatalf("error code = %d, want %d", got.Code, codeUnsupported)
}
}
// createOnlyAuthenticator signs in an identity that may create and read tables
// but not update them.
type createOnlyAuthenticator struct{}
func (createOnlyAuthenticator) AuthenticateRequest(*http.Request) (string, interface{}, s3err.ErrorCode) {
return "creator", &struct {
Name string
Actions []string
}{Name: "creator", Actions: []string{"s3tables:CreateTable", "s3tables:GetTable"}}, s3err.ErrNone
}
func (createOnlyAuthenticator) DefaultAllow() bool { return false }
func (createOnlyAuthenticator) AuthEnabled() bool { return true }
// The right to create a name is the write evidence for a table the declare
// creates, but not for one it only finds.
func TestDeclareVendsWriteOnlyForATableItCreates(t *testing.T) {
h := newTestHarness(t)
h.createBucket(t, "lnc", s3tables.FormatLance)
h.mustDo(t, http.MethodPost, "/v1/namespace/lnc$ns/create", `{}`, http.StatusOK)
h.mustDo(t, http.MethodPost, "/v1/table/lnc$ns$theirs/declare", `{}`, http.StatusOK)
h.mustDo(t, http.MethodPost, "/v1/table/lnc$ns$theirs/deregister", `{}`, http.StatusOK)
server := NewServer(s3tables.NewManagerClient(h.filer.Client), createOnlyAuthenticator{})
server.SetCredentialVendor(recordingVendor{})
router := mux.NewRouter().SkipClean(true)
server.RegisterRoutes(router)
creator := &testHarness{router: router, filer: h.filer, admin: h.admin, server: server}
for _, tc := range []struct{ name, want string }{
{"theirs", "read-only"},
{"mine", "read-write"},
} {
t.Run(tc.name, func(t *testing.T) {
if got := creator.vendedKey(t, "/v1/table/lnc$ns$"+tc.name+"/declare", ""); got != tc.want {
t.Errorf("vended %q, want %q", got, tc.want)
}
})
}
if h.filer.Get("/buckets/lnc/ns/theirs", deregisteredMarker) == nil {
t.Error("a declare that may not write the table brought it back")
}
}
// An in-process describe for a trusted caller resolves the table without the
// caller's catalog grants, and still refuses a deregistered one.
func TestDescribeTableInternalIgnoresCatalogGrants(t *testing.T) {
h := newTestHarness(t)
h.createBucket(t, "lnc", s3tables.FormatLance)
h.mustDo(t, http.MethodPost, "/v1/namespace/lnc$ns/create", `{}`, http.StatusOK)
h.mustDo(t, http.MethodPost, "/v1/table/lnc$ns$t/declare", `{}`, http.StatusOK)
server := NewServer(s3tables.NewManagerClient(h.filer.Client), createOnlyAuthenticator{})
server.SetS3Endpoint("http://127.0.0.1:8333")
anonymous := httptest.NewRequest(http.MethodPost, "/", nil)
anonymous = anonymous.WithContext(s3_constants.SetIdentityNameInContext(anonymous.Context(), "stranger"))
if _, _, err := server.DescribeTable(anonymous, []string{"lnc", "ns", "t"}); err == nil {
t.Fatal("DescribeTable without grants resolved the table")
}
location, options, err := server.DescribeTableInternal(anonymous, []string{"lnc", "ns", "t"})
if err != nil {
t.Fatalf("DescribeTableInternal: %v", err)
}
if location != "s3://lnc/ns/t" || options["aws_endpoint"] != "http://127.0.0.1:8333" {
t.Fatalf("resolved %q %v", location, options)
}
h.mustDo(t, http.MethodPost, "/v1/table/lnc$ns$t/deregister", `{}`, http.StatusOK)
_, _, err = server.DescribeTableInternal(anonymous, []string{"lnc", "ns", "t"})
var apiErr *s3tables.S3TablesError
if !errors.As(err, &apiErr) || apiErr.Type != s3tables.ErrCodeNoSuchTable {
t.Fatalf("a deregistered table resolved: %v, want %s", err, s3tables.ErrCodeNoSuchTable)
}
}
// The options an internal describe vends may write a vector index, whose
// writer is the engine, but not a table whose writes s3tables grants gate.
func TestDescribeTableInternalVendsWriteForTheVectorNamespace(t *testing.T) {
h := newTestHarness(t)
h.seedVectorIndex(t, "vec", s3tables.FormatLance)
h.createBucket(t, "lnc", s3tables.FormatLance)
h.mustDo(t, http.MethodPost, "/v1/namespace/lnc$ns/create", `{}`, http.StatusOK)
h.mustDo(t, http.MethodPost, "/v1/table/lnc$ns$t/declare", `{}`, http.StatusOK)
server := NewServer(s3tables.NewManagerClient(h.filer.Client), createOnlyAuthenticator{})
server.SetCredentialVendor(recordingVendor{})
stranger := httptest.NewRequest(http.MethodPost, "/", nil)
stranger = stranger.WithContext(s3_constants.SetIdentityNameInContext(stranger.Context(), "stranger"))
for _, tc := range []struct {
id []string
want string
}{
{[]string{"vec", "vector", "idx"}, "read-write"},
{[]string{"lnc", "ns", "t"}, "read-only"},
} {
_, options, err := server.DescribeTableInternal(stranger, tc.id)
if err != nil {
t.Fatalf("%v: %v", tc.id, err)
}
if got := options["aws_access_key_id"]; got != tc.want {
t.Errorf("%v: vended %q, want %q", tc.id, got, tc.want)
}
}
}