Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-11-07 12:37:41 +01:00
Code Issues Packages Projects Releases Wiki Activity

S3 Vectors: the gateway resolves the table it sends to weed-vector (#12468)

Browse Source 
* lance catalog: in-process DescribeTable for trusted callers

DescribeTable and DescribeTableInternal resolve a table's location and
storage options without the HTTP layer; the internal form runs the lookup
through a trusted manager so the caller's catalog grants do not gate a
resolution its own permission already authorized. Same code as the
Enterprise repository.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector.proto: Table carries its location and storage options

The gateway resolves the table it addresses, so the engine can open it
without a describe round trip of its own. Both fields are optional; an
empty location means the engine looks the table up as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: test the in-process describe

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: resolve the table before calling the engine

The gateway describes the table in process, through the trusted catalog
manager, and sends its location and storage options with every engine
call. Storage options are vended for the caller's principal, as the
Enterprise router does for queries. A catalog failure answers with the
code the engine would have given.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: open a table at the location the gateway sent

A request whose table carries a location is opened there, with the
storage options the gateway vended over the engine's fallback; the
catalog is described only for a table that comes without one, as from
an older gateway. dataset::open_at and dataset::drop_at take a
description the caller already holds.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors e2e: run the engine without a catalog

VECTOR_ENGINE_NO_CATALOG=1 gives the engine an unreachable catalog URL;
every CI leg sets it, so the suite proves the gateway resolves each table
it sends.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: an internal describe may write a vector index

With credential vending on, the options vended to the engine followed the
caller's s3tables grants, which never allow writing the vector namespace,
so every write the gateway sent would have run with read-only credentials
and reads would have lost their version lease. The engine is that
namespace's writer and the caller's vector permission authorized the
operation, so an internal describe of a vector table vends write-capable
options; other namespaces keep the UpdateTable check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: resolve tables for the engine only while the catalog runs

With -port.lance=0 the gateway has no catalog to resolve through; the
engine then keeps looking tables up itself, as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: a deregistered table describes as NoSuchTable

So the gateway answers NotFoundException for it, not InternalFailure.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: trust a sent location only for a colocated vector index

A caller that reaches the engine could otherwise name any storage the
engine's credentials cover. A location is taken from the request only
when the id names a vector index and the location is its colocated path,
which the engine writes already; anything else is described through the
catalog as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: scope the read cache by storage endpoint

The cache was keyed by bucket and path alone, so the same path reached
through another endpoint would have shared entries. Each open wraps the
store with the cache scoped to the endpoint its options name.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep a configured endpoint over one a request names

A caller that reaches the engine could otherwise send its fallback
credentials to an endpoint of the caller's choosing. When the engine's
--storage-option names an aws_endpoint, a table whose request names
another one is described through the catalog instead.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: use request options only where credentials cannot leave

A request's storage options are taken only when they hold the keys a
gateway's describe produces, so a proxy or an endpoint alias cannot route
the engine's credentials elsewhere, and an endpoint only when it is the
configured one or the engine holds no credentials to send. Anything else
sends the table to the catalog.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors e2e: give the engine its endpoint with its credentials

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: a drop checks the request's options like an open

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: an unpinned endpoint needs the request to sign for itself

Without configured credentials the S3 store still signs with the AWS
default chain, so an endpoint from a request is taken only when the
request skips signing or brings its own credentials.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: a trailing slash does not unpin the endpoint, and say so

A configured aws_endpoint written with a trailing slash matched nothing
the gateway sends, so every table fell back to the catalog in silence.
The comparison ignores the slash, and a fallback for a resolved table is
logged at debug.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: resolved by the gateway, or refused

The engine no longer has a catalog: --namespace and --filer-signing-key
are gone, a request whose table names no location is InvalidArgument, and
one whose location is not the index's own or whose storage options the
engine's credentials must not be used with is PermissionDenied. A drop
removes the colocated path it was given.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: the catalog resolves vector tables whether or not its port is served

newLanceServer builds the Lance catalog for the S3 Vectors API; the
listener on -port.lance is optional.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors tests: start the engine without a catalog

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: the aux listener test builds the catalog before serving it

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: an unpinned endpoint takes only credentials the request signs with alone

Lance fills an option a request leaves out from the engine's environment,
under any of object_store's names for it (TOKEN, SESSION_TOKEN, ...), and
asks the AWS default chain when the key pair is incomplete. A request that
named its own endpoint with only an access key id, or a key pair while the
engine's environment held a session token, would have sent the engine's
credentials there. Such a request now needs a full key pair, and its own
session token whenever the environment would supply one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: accept --namespace and --filer-signing-key, ignored

Both flags shipped in 4.52 and are documented, so dropping them made an
existing command line exit at startup with an unexpected-argument error.
They are kept hidden and ignored, with a warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors design: the engine is started without a catalog

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: the trust check takes the environment it reads

The options test read the process environment through signs_alone, so a
host exporting AWS_SESSION_TOKEN or TOKEN failed it. The caller now
passes the variable names in; describe passes the engine's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: log both endpoints when a request's options are refused

The gateway turns the PermissionDenied into a bare AccessDenied and logs
nothing, so a pinned aws_endpoint written in another form than the one the
gateway sends (host case, a default port, an IP for a name) failed every
call with no trace of which value differed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: the Lance catalog sets the credential role whether or not Iceberg runs

-iceberg.credentialRole reached the S3 server only while the Iceberg
listener started, so with -port.iceberg=0 the Lance catalog installed its
vendor but vending stayed off, and the vector engine got no credentials for
the tables the gateway resolves. Both catalogs now set the role through
one helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: set the credential role once, before either catalog serves

Both catalogs wrote the role as they were built; the second write landed
while the first was already serving requests that read it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Chris LuandClaude Fable 5.1 authored and GitHub committed 2026-11-06 19:39:09 -08:00
1 parent 7beac0e8ea
commit 2fe480a6e2
18 files changed
+954 -209
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

weed/pb/vector_pb/vector.pb.go
+68 -42
View File
@@ -75,14 +75,19 @@ func (DistanceMetric) EnumDescriptor() ([]byte, []int) {
}
// Table addresses a Lance table through the namespace: the id parts as the
// namespace catalog joins them.
// namespace catalog joins them. The gateway resolves the table before it
// calls, so the engine opens it without a catalog round trip of its own.
type Table struct {
state protoimpl.MessageState `protogen:"open.v1"`
Id []string `protobuf:"bytes,1,rep,name=id,proto3" json:"id,omitempty"`
// Dataset version to pin reads to; 0 means latest.
Version uint64 `protobuf:"varint,2,opt,name=version,proto3" json:"version,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
Version uint64 `protobuf:"varint,2,opt,name=version,proto3" json:"version,omitempty"`
// The table's Lance location, as the catalog describes it.
Location string `protobuf:"bytes,3,opt,name=location,proto3" json:"location,omitempty"`
// object_store options from describe_table (vended creds / endpoint).
StorageOptions map[string]string `protobuf:"bytes,4,rep,name=storage_options,json=storageOptions,proto3" json:"storage_options,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *Table) Reset() {
@@ -129,6 +134,20 @@ func (x *Table) GetVersion() uint64 {
return 0
}
func (x *Table) GetLocation() string {
if x != nil {
return x.Location
}
return ""
}
func (x *Table) GetStorageOptions() map[string]string {
if x != nil {
return x.StorageOptions
}
return nil
}
type CreateIndexRequest struct {
state protoimpl.MessageState `protogen:"open.v1"`
Table *Table `protobuf:"bytes,1,opt,name=table,proto3" json:"table,omitempty"`
@@ -1061,10 +1080,15 @@ var File_vector_proto protoreflect.FileDescriptor
const file_vector_proto_rawDesc = "" +
"\n" +
"\fvector.proto\x12\tvector_pb\x1a\x1cgoogle/protobuf/struct.proto\"1\n" +
"\fvector.proto\x12\tvector_pb\x1a\x1cgoogle/protobuf/struct.proto\"\xdf\x01\n" +
"\x05Table\x12\x0e\n" +
"\x02id\x18\x01 \x03(\tR\x02id\x12\x18\n" +
"\aversion\x18\x02 \x01(\x04R\aversion\"\xdf\x01\n" +
"\aversion\x18\x02 \x01(\x04R\aversion\x12\x1a\n" +
"\blocation\x18\x03 \x01(\tR\blocation\x12M\n" +
"\x0fstorage_options\x18\x04 \x03(\v2$.vector_pb.Table.StorageOptionsEntryR\x0estorageOptions\x1aA\n" +
"\x13StorageOptionsEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\xdf\x01\n" +
"\x12CreateIndexRequest\x12&\n" +
"\x05table\x18\x01 \x01(\v2\x10.vector_pb.TableR\x05table\x12\x1c\n" +
"\tdimension\x18\x02 \x01(\rR\tdimension\x12B\n" +
@@ -1162,7 +1186,7 @@ func file_vector_proto_rawDescGZIP() []byte {
}
var file_vector_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
var file_vector_proto_msgTypes = make([]protoimpl.MessageInfo, 17)
var file_vector_proto_msgTypes = make([]protoimpl.MessageInfo, 18)
var file_vector_proto_goTypes = []any{
(DistanceMetric)(0), // 0: vector_pb.DistanceMetric
(*Table)(nil), // 1: vector_pb.Table
@@ -1182,42 +1206,44 @@ var file_vector_proto_goTypes = []any{
(*QueryVectorsRequest)(nil), // 15: vector_pb.QueryVectorsRequest
(*QueryResult)(nil), // 16: vector_pb.QueryResult
(*QueryVectorsResponse)(nil), // 17: vector_pb.QueryVectorsResponse
(*structpb.Struct)(nil), // 18: google.protobuf.Struct
nil, // 18: vector_pb.Table.StorageOptionsEntry
(*structpb.Struct)(nil), // 19: google.protobuf.Struct
}
var file_vector_proto_depIdxs = []int32{
1, // 0: vector_pb.CreateIndexRequest.table:type_name -> vector_pb.Table
0, // 1: vector_pb.CreateIndexRequest.distance_metric:type_name -> vector_pb.DistanceMetric
1, // 2: vector_pb.DropIndexRequest.table:type_name -> vector_pb.Table
18, // 3: vector_pb.Vector.metadata:type_name -> google.protobuf.Struct
1, // 4: vector_pb.PutVectorsRequest.table:type_name -> vector_pb.Table
6, // 5: vector_pb.PutVectorsRequest.vectors:type_name -> vector_pb.Vector
1, // 6: vector_pb.GetVectorsRequest.table:type_name -> vector_pb.Table
6, // 7: vector_pb.GetVectorsResponse.vectors:type_name -> vector_pb.Vector
1, // 8: vector_pb.DeleteVectorsRequest.table:type_name -> vector_pb.Table
1, // 9: vector_pb.ListVectorsRequest.table:type_name -> vector_pb.Table
6, // 10: vector_pb.ListVectorsResponse.vectors:type_name -> vector_pb.Vector
1, // 11: vector_pb.QueryVectorsRequest.table:type_name -> vector_pb.Table
18, // 12: vector_pb.QueryResult.metadata:type_name -> google.protobuf.Struct
16, // 13: vector_pb.QueryVectorsResponse.vectors:type_name -> vector_pb.QueryResult
2, // 14: vector_pb.VectorService.CreateIndex:input_type -> vector_pb.CreateIndexRequest
4, // 15: vector_pb.VectorService.DropIndex:input_type -> vector_pb.DropIndexRequest
7, // 16: vector_pb.VectorService.PutVectors:input_type -> vector_pb.PutVectorsRequest
9, // 17: vector_pb.VectorService.GetVectors:input_type -> vector_pb.GetVectorsRequest
11, // 18: vector_pb.VectorService.DeleteVectors:input_type -> vector_pb.DeleteVectorsRequest
13, // 19: vector_pb.VectorService.ListVectors:input_type -> vector_pb.ListVectorsRequest
15, // 20: vector_pb.VectorService.QueryVectors:input_type -> vector_pb.QueryVectorsRequest
3, // 21: vector_pb.VectorService.CreateIndex:output_type -> vector_pb.CreateIndexResponse
5, // 22: vector_pb.VectorService.DropIndex:output_type -> vector_pb.DropIndexResponse
8, // 23: vector_pb.VectorService.PutVectors:output_type -> vector_pb.PutVectorsResponse
10, // 24: vector_pb.VectorService.GetVectors:output_type -> vector_pb.GetVectorsResponse
12, // 25: vector_pb.VectorService.DeleteVectors:output_type -> vector_pb.DeleteVectorsResponse
14, // 26: vector_pb.VectorService.ListVectors:output_type -> vector_pb.ListVectorsResponse
17, // 27: vector_pb.VectorService.QueryVectors:output_type -> vector_pb.QueryVectorsResponse
21, // [21:28] is the sub-list for method output_type
14, // [14:21] is the sub-list for method input_type
14, // [14:14] is the sub-list for extension type_name
14, // [14:14] is the sub-list for extension extendee
0, // [0:14] is the sub-list for field type_name
18, // 0: vector_pb.Table.storage_options:type_name -> vector_pb.Table.StorageOptionsEntry
1, // 1: vector_pb.CreateIndexRequest.table:type_name -> vector_pb.Table
0, // 2: vector_pb.CreateIndexRequest.distance_metric:type_name -> vector_pb.DistanceMetric
1, // 3: vector_pb.DropIndexRequest.table:type_name -> vector_pb.Table
19, // 4: vector_pb.Vector.metadata:type_name -> google.protobuf.Struct
1, // 5: vector_pb.PutVectorsRequest.table:type_name -> vector_pb.Table
6, // 6: vector_pb.PutVectorsRequest.vectors:type_name -> vector_pb.Vector
1, // 7: vector_pb.GetVectorsRequest.table:type_name -> vector_pb.Table
6, // 8: vector_pb.GetVectorsResponse.vectors:type_name -> vector_pb.Vector
1, // 9: vector_pb.DeleteVectorsRequest.table:type_name -> vector_pb.Table
1, // 10: vector_pb.ListVectorsRequest.table:type_name -> vector_pb.Table
6, // 11: vector_pb.ListVectorsResponse.vectors:type_name -> vector_pb.Vector
1, // 12: vector_pb.QueryVectorsRequest.table:type_name -> vector_pb.Table
19, // 13: vector_pb.QueryResult.metadata:type_name -> google.protobuf.Struct
16, // 14: vector_pb.QueryVectorsResponse.vectors:type_name -> vector_pb.QueryResult
2, // 15: vector_pb.VectorService.CreateIndex:input_type -> vector_pb.CreateIndexRequest
4, // 16: vector_pb.VectorService.DropIndex:input_type -> vector_pb.DropIndexRequest
7, // 17: vector_pb.VectorService.PutVectors:input_type -> vector_pb.PutVectorsRequest
9, // 18: vector_pb.VectorService.GetVectors:input_type -> vector_pb.GetVectorsRequest
11, // 19: vector_pb.VectorService.DeleteVectors:input_type -> vector_pb.DeleteVectorsRequest
13, // 20: vector_pb.VectorService.ListVectors:input_type -> vector_pb.ListVectorsRequest
15, // 21: vector_pb.VectorService.QueryVectors:input_type -> vector_pb.QueryVectorsRequest
3, // 22: vector_pb.VectorService.CreateIndex:output_type -> vector_pb.CreateIndexResponse
5, // 23: vector_pb.VectorService.DropIndex:output_type -> vector_pb.DropIndexResponse
8, // 24: vector_pb.VectorService.PutVectors:output_type -> vector_pb.PutVectorsResponse
10, // 25: vector_pb.VectorService.GetVectors:output_type -> vector_pb.GetVectorsResponse
12, // 26: vector_pb.VectorService.DeleteVectors:output_type -> vector_pb.DeleteVectorsResponse
14, // 27: vector_pb.VectorService.ListVectors:output_type -> vector_pb.ListVectorsResponse
17, // 28: vector_pb.VectorService.QueryVectors:output_type -> vector_pb.QueryVectorsResponse
22, // [22:29] is the sub-list for method output_type
15, // [15:22] is the sub-list for method input_type
15, // [15:15] is the sub-list for extension type_name
15, // [15:15] is the sub-list for extension extendee
0, // [0:15] is the sub-list for field type_name
}
func init() { file_vector_proto_init() }
@@ -1231,7 +1257,7 @@ func file_vector_proto_init() {
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: unsafe.Slice(unsafe.StringData(file_vector_proto_rawDesc), len(file_vector_proto_rawDesc)),
NumEnums: 1,
NumMessages: 17,
NumMessages: 18,
NumExtensions: 0,
NumServices: 1,
},