mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-07 12:37:41 +01:00
2fe480a6e2a7cf8de199d12a9bfdd1cfdc6caaaf
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Fable 5.1
|
2fe480a6e2 |
S3 Vectors: the gateway resolves the table it sends to weed-vector (#12468)
* lance catalog: in-process DescribeTable for trusted callers DescribeTable and DescribeTableInternal resolve a table's location and storage options without the HTTP layer; the internal form runs the lookup through a trusted manager so the caller's catalog grants do not gate a resolution its own permission already authorized. Same code as the Enterprise repository. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector.proto: Table carries its location and storage options The gateway resolves the table it addresses, so the engine can open it without a describe round trip of its own. Both fields are optional; an empty location means the engine looks the table up as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: test the in-process describe Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: resolve the table before calling the engine The gateway describes the table in process, through the trusted catalog manager, and sends its location and storage options with every engine call. Storage options are vended for the caller's principal, as the Enterprise router does for queries. A catalog failure answers with the code the engine would have given. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: open a table at the location the gateway sent A request whose table carries a location is opened there, with the storage options the gateway vended over the engine's fallback; the catalog is described only for a table that comes without one, as from an older gateway. dataset::open_at and dataset::drop_at take a description the caller already holds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: run the engine without a catalog VECTOR_ENGINE_NO_CATALOG=1 gives the engine an unreachable catalog URL; every CI leg sets it, so the suite proves the gateway resolves each table it sends. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: an internal describe may write a vector index With credential vending on, the options vended to the engine followed the caller's s3tables grants, which never allow writing the vector namespace, so every write the gateway sent would have run with read-only credentials and reads would have lost their version lease. The engine is that namespace's writer and the caller's vector permission authorized the operation, so an internal describe of a vector table vends write-capable options; other namespaces keep the UpdateTable check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: resolve tables for the engine only while the catalog runs With -port.lance=0 the gateway has no catalog to resolve through; the engine then keeps looking tables up itself, as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: a deregistered table describes as NoSuchTable So the gateway answers NotFoundException for it, not InternalFailure. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: trust a sent location only for a colocated vector index A caller that reaches the engine could otherwise name any storage the engine's credentials cover. A location is taken from the request only when the id names a vector index and the location is its colocated path, which the engine writes already; anything else is described through the catalog as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: scope the read cache by storage endpoint The cache was keyed by bucket and path alone, so the same path reached through another endpoint would have shared entries. Each open wraps the store with the cache scoped to the endpoint its options name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep a configured endpoint over one a request names A caller that reaches the engine could otherwise send its fallback credentials to an endpoint of the caller's choosing. When the engine's --storage-option names an aws_endpoint, a table whose request names another one is described through the catalog instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: use request options only where credentials cannot leave A request's storage options are taken only when they hold the keys a gateway's describe produces, so a proxy or an endpoint alias cannot route the engine's credentials elsewhere, and an endpoint only when it is the configured one or the engine holds no credentials to send. Anything else sends the table to the catalog. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: give the engine its endpoint with its credentials Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a drop checks the request's options like an open Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: an unpinned endpoint needs the request to sign for itself Without configured credentials the S3 store still signs with the AWS default chain, so an endpoint from a request is taken only when the request skips signing or brings its own credentials. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a trailing slash does not unpin the endpoint, and say so A configured aws_endpoint written with a trailing slash matched nothing the gateway sends, so every table fell back to the catalog in silence. The comparison ignores the slash, and a fallback for a resolved table is logged at debug. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: resolved by the gateway, or refused The engine no longer has a catalog: --namespace and --filer-signing-key are gone, a request whose table names no location is InvalidArgument, and one whose location is not the index's own or whose storage options the engine's credentials must not be used with is PermissionDenied. A drop removes the colocated path it was given. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: the catalog resolves vector tables whether or not its port is served newLanceServer builds the Lance catalog for the S3 Vectors API; the listener on -port.lance is optional. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors tests: start the engine without a catalog Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: the aux listener test builds the catalog before serving it Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: an unpinned endpoint takes only credentials the request signs with alone Lance fills an option a request leaves out from the engine's environment, under any of object_store's names for it (TOKEN, SESSION_TOKEN, ...), and asks the AWS default chain when the key pair is incomplete. A request that named its own endpoint with only an access key id, or a key pair while the engine's environment held a session token, would have sent the engine's credentials there. Such a request now needs a full key pair, and its own session token whenever the environment would supply one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: accept --namespace and --filer-signing-key, ignored Both flags shipped in 4.52 and are documented, so dropping them made an existing command line exit at startup with an unexpected-argument error. They are kept hidden and ignored, with a warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors design: the engine is started without a catalog Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: the trust check takes the environment it reads The options test read the process environment through signs_alone, so a host exporting AWS_SESSION_TOKEN or TOKEN failed it. The caller now passes the variable names in; describe passes the engine's own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: log both endpoints when a request's options are refused The gateway turns the PermissionDenied into a bare AccessDenied and logs nothing, so a pinned aws_endpoint written in another form than the one the gateway sends (host case, a default port, an IP for a name) failed every call with no trace of which value differed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: the Lance catalog sets the credential role whether or not Iceberg runs -iceberg.credentialRole reached the S3 server only while the Iceberg listener started, so with -port.iceberg=0 the Lance catalog installed its vendor but vending stayed off, and the vector engine got no credentials for the tables the gateway resolves. Both catalogs now set the role through one helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: set the credential role once, before either catalog serves Both catalogs wrote the role as they were built; the second write landed while the first was already serving requests that read it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
4be191173a |
vector: keep ListVectors pages and segments flat as an index grows; read only keys in GetVectors (#12300)
* vector: scan only the fragments a ListVectors page covers A page pushed its offset into the scan, but the scan still loaded every fragment first, reading each one's deletion file: about 105 S3 GETs per page at 50k rows against 25 at 10k. The page's fragments now come from the live row counts in the manifest, so a page reads only its own. Tokens and row order are unchanged; a manifest without the counts keeps the old scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: list a ListVectors segment in the engine The gateway paged through the whole index for every segment and kept one row in segmentCount, so a segment cost segmentCount engine pages and 16 segments listed slower than one. The engine now takes segment_count and segment_index: it reads keys across the stride, keeps the segment's rows and takes their data and metadata alone, one call per page. A segment is still the rows whose position modulo segmentCount is segmentIndex, and its token still the position to resume from. An engine that predates the fields does not set segmented, and the gateway then pages through it as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: read only the key column across the table in GetVectors The key lookup scanned every column. On S3, lance reads a column during the scan when a row of it is under 1000 bytes, so a 128-dimension vector was read for every row of the table to return 100 of them, and data and metadata were read even when the request did not ask for them. The scan now projects the requested columns and materializes everything but the key late, for the matching rows only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: start the legacy segment fallback from the page already read With an engine that ignores segment_count, the gateway dropped the page it had just read and fetched it again. The fallback now consumes that page first, so a one-row segment page still costs one engine call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: check metadata on segmented ListVectors pages The segment test listed data only. Its rows now carry metadata on two in three keys, and every segmented row, read through take_rows, must equal the same row in a plain listing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: lease the version while a segment takes its data A segmented page with data or metadata reads keys first and then takes the other columns from the same version, so maintenance could reclaim its files in between. That path now holds a lease like GetVectors. A keys-only page is one scan and stays unleased, as plain pages are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: leave GetVectors materialization to lance Forcing every column late made a 100-key GetVectors with data slower at 10k x 128 in the perf cluster: 54.7 ms against 46.3 ms. The take issued a read per row, where lance's heuristic reads a narrow vector column in one pass per fragment. The projection keeps the gain on its own: with neither data nor metadata requested, the call went from 38.6 ms to 25.3 ms and from 150 to 89 S3 GETs. At 50k x 384 the vector is already late under the heuristic, and a full lookup is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandDevin
|
d771a91571 |
vector: VectorService gRPC contract (P1) (#12101)
* vector: VectorService protobuf contract The shared RPC boundary between the Go S3 gateway and the Rust vector engine from the vector design doc: CreateIndex/DropIndex, Put/Get/ List/Delete/Query vectors. A Table reference carries the Lance table id and an optional pinned version, matching the snapshot discipline queries are required to keep. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: generated Go stubs for vector.proto protoc output via weed/pb/Makefile. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: carry return_data through QueryVectors S3 Vectors QueryVectors honours returnData, but the contract had no way to ask for or carry vector data in a query response, so the gateway would need a second GetVectors round-trip per query. Add return_data to the request and data to QueryResult now, while the contract is still new. * vector: pin metadata numbers to double precision in the contract Metadata arrives as JSON, where numbers carry double precision; say so on the field so neither side promises integer fidelity past 2^53. * vector: clarify the query filter is raw Lance SQL over one metadata column The doc said 'metadata columns' but the table stores all metadata in a single JSON column named metadata; the gateway translates S3 Vectors filter expressions onto it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |