mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-04 11:07:35 +01:00
5dfda3645ead2003ff85cd2f236bb296af39fc22
17
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
5063aa9be2 |
weed-vector: check master certificates against the server allowlists, read -config_dir (#12374)
* weed-vector: look for security.toml in -config_dir first weed processes search the directory -config_dir names before ., so a cluster keeping security.toml there left the engine in plaintext. --config-dir (also -config_dir, WEED_VECTOR_CONFIG_DIR) does the same. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: dial the masters the way weed clients do The master's certificate must chain to grpc.ca and is not checked against the host dialed, as weed clients verify (advancedtls.CertVerification), so masters sharing one workload certificate, or dialed by IP, register as they do from weed. Instead, grpc.client.master_allowed_server_commonNames, grpc.client.allowed_server_commonNames or grpc.allowed_server_commonNames (and the wildcard domain likewise) restrict which server identities are accepted: the common name or a DNS SAN, or an exact URI SAN, as security.Authenticator.AuthenticateServer does. These used to be refused at startup; --tls-server-name is gone with the host check. The TLS runs in a connector that reads the credentials asynchronously at every dial, so the connect timeout still covers the reads, the TCP connect and the handshake together. An IPv6 master's brackets are stripped before the dial. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: the hardened leg allowlists server identities for grpc.client The gateway and the engine then accept only servers named localhost, which exercises the engine's allowed_server_commonNames check on the master dial. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: parse the refresh interval with time.ParseDuration's arithmetic The interval was summed in floating-point seconds, so 0.9ns rounded up to 1ns where Go truncates it to 0 and falls back to the 5h default, 0.3s was off by a nanosecond, and 3000000h was accepted where Go rejects anything past the longest time.Duration. Each part is now an integer count of nanoseconds plus the fraction Go adds, with Go's overflow bounds, and the value is no longer trimmed, as Go does not trim it either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: test a master dial whose credential read stalls The bounded-dial tests passed no credentials or a prebuilt endpoint, so moving the credential read out of the connect timeout would have failed none of them. The new test points grpc.ca at a FIFO nobody writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: dial a closed port in the stalled credential read test Against a silent listener, a dial that never read the credentials timed out with the same message, so the test passed without the read. Nothing listens on the port now, so only the stalled read can time out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: test that Security::load searches the config dir Only candidates() was tested, so load dropping config_dir on the way would have passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: expand a leading ~ in -config_dir like weed's ResolvePath Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: hold the port in the stalled credential read test A released port can be taken before the dial, and a dial that skipped the read would then time out and pass. A listener that drops every connection keeps the port and still fails such a dial at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
eeb8aefe70 |
weed-vector: gRPC TLS to the masters and for the gateway (#12368)
* weed-vector: read the cluster's security.toml The engine finds security.toml where weed processes do (., ~/.seaweedfs, /usr/local/etc/seaweedfs, /etc/seaweedfs) with WEED_* environment variables over it, and builds gRPC client and server credentials by the rules of security.LoadClientTLS and LoadServerTLS: a dedicated client certificate is preferred, both halves of a pair must be set, and grpc.ca is required once either is. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: gRPC TLS to the masters and from the gateway The masters are dialed with [grpc.client]'s credentials when the cluster has them, the master's certificate checked against the host dialed or --tls-server-name. The engine's own listener takes its identity from [grpc.vector] and then requires a client certificate chaining to grpc.ca, which is how the gateway, dialing with [grpc.client], reaches it. The rustls provider is pinned before the first TLS use, as the other binaries linking both aws-lc-rs and ring do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * scaffold: document [grpc.vector] Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: put every gRPC hop under mutual TLS in the hardened leg VECTOR_TLS=1 now also writes [grpc.master], [grpc.volume], [grpc.filer], [grpc.s3], [grpc.client] and [grpc.vector], and probes the master's and the engine's gRPC ports: plaintext HTTP/2 and a client without a certificate are refused, the client certificate gets a SETTINGS frame. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: give the test volume server 64 volume slots Every collection grows 7 volumes and test_vector_bucket_encryption opens two buckets while the previous test's bucket is still being torn down, so 16 slots were marginal: the Go legs failed CreateIndex now and then with "failed to find writable volumes" and master.log showing "create 7 volume, created 4: Not enough data nodes found!". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: reload the listener's certificates and enforce allowed_commonNames The listener now runs the way weed servers run theirs. The certificate, key and CA files are re-read every WEED_TLS_CERT_REFRESH_INTERVAL (weed's default of 5h) and on SIGHUP, and new handshakes use the changed files without a restart; a file that no longer parses is logged and the working configuration stays. [grpc.vector] allowed_commonNames and grpc.allowed_wildcard_domain restrict which client certificates are served, checked against the subject common name as security.Authenticator and the Rust volume server's CommonNameVerifier do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * scaffold: allowed_commonNames and the refresh cadence for [grpc.vector] Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the engine's allowlist keeps an outsider certificate out The hardened leg sets [grpc.vector] allowed_commonNames = "localhost" and probes the engine with a second certificate the same CA signed for CN=outsider: the handshake must be refused, while the gateway's certificate still gets an HTTP/2 SETTINGS frame. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: take the master's TLS name from the parsed endpoint A master given without a port (https://master.example) had the whole URL checked against its certificate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: bound the master TLS handshake connect_timeout covers only the TCP connect, so a master that accepted the connection and never answered the handshake held up the other masters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: look up security.toml keys the way viper does An empty WEED_* variable no longer hides the file's value, and keys match regardless of case, as for weed processes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: refuse master allowlists it cannot enforce weed checks a dialed server against allowed_server_commonNames and allowed_server_wildcard_domain on top of grpc.ca. The engine's master dial accepted every certificate the CA signed even when one was set; it now fails at startup and names the key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: read the whole frame header in the gRPC TLS probe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: probe the volume, filer and S3 gRPC listeners too Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read the master dial credentials only when given masters A standalone engine never dials with grpc.client, so its settings no longer stop it from starting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: run each listener TLS handshake on its own task The accept loop awaited every handshake before accepting the next connection, so a peer that connected and sent nothing stopped every gateway connection after it. Handshakes now run concurrently, each under a 10s timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read the master dial's TLS files at every dial The client certificate, key and CA were read once at startup, so after the files rotated a reconnect still presented the old pair. Each dial now reads them, as the listener re-reads its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read every unit Go durations accept in the refresh interval WEED_TLS_CERT_REFRESH_INTERVAL values in ns, us or a leading + fell back to five hours in the engine while weed servers honored them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: fall back on a refresh interval too long to hold An oversized WEED_TLS_CERT_REFRESH_INTERVAL panicked the engine at startup; weed servers reject it and keep the default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: refuse allowed_commonNames without the listener's certificate With [grpc.vector] allowed_commonNames set but no cert and key, the listener served plaintext and checked no client at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read the master dial's credentials inside its connect timeout The CA, certificate and key were read synchronously before the timeout started, so a stalled credential path held up every other master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: let a set key win over an empty spelling of it With allowed_server_commonNames = "" beside allowed_server_commonnames = "master", the lookup stopped at the empty exact match and the allowlist went unnoticed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d72f761d83 |
seaweed-vector: work on its defaults, and without identities or credentials (#12323)
* seaweed-vector: default --namespace to the gateway's Lance port The old default, http://localhost:8888/lance, is a path nothing serves: the Lance Namespace listens on its own port, 9101 by default on weed s3, weed server, weed filer and weed mini. With it, the filer answered the engine's catalog calls with a 500 and every CreateIndex failed with InternalFailure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: default --listen to a port no SeaweedFS server takes 127.0.0.1:8080 is the volume server's default port, so weed-vector next to weed server exited with "Address already in use (os error 98)". 8090 is not among the defaults of master, volume, filer, S3 and its Iceberg and Lance catalogs, IAM, WebDAV, SFTP, the message queue broker and agent, the admin server or weed mini, nor any of their gRPC ports (HTTP + 10000). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance: take any key while the S3 server runs without identities Without identities the S3 port accepts any signature and treats the caller as the admin. The Lance catalog still checked an x-api-key or Bearer token against identities that did not exist, so weed-vector started with --storage-option credentials on such a server got 401 on every catalog call. The catalog now skips both checks when S3 auth is off and leaves the caller to the S3 authenticator, which admits it as the admin, as it already did for a request with no key. With identities configured nothing changes: an unknown key is still 401. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance: tell clients to skip signing while S3 auth is off A client with no credentials of its own could not reach a table's data on a server without identities, though the S3 port takes unsigned requests there: lance fell back to the AWS default credential chain and failed with CredentialsNotLoaded. weed-vector without --storage-option failed every CreateIndex that way. The catalog now adds aws_skip_signature to the storage options it hands out while S3 auth is off, and leaves it out once identities exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: bind the listener before registering with the master The engine spawned its master registration and then bound --listen, so an engine whose port was taken could reach the master before it exited: on weed server with 18080 taken the master logged "vector node 127.0.0.1:18080 added" and removed it in the same millisecond, and the gateway noticed it. Bind first, and name the address in the error instead of "transport error". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-worker: keep an unsigned store off the AWS default chain A store the catalog says takes unsigned requests needs no credentials, but lance builds the AWS default credential chain for any store without them, and the chain probes instance metadata on every dataset open. With no credentials of its own, weed-vector spent a second on each query (1030-1066 ms against 32-46 ms with --storage-option ones). The merge of the operator's fallback and the vended options, which the engine and the Lance worker each did in their own copy, moves into table_options, and gives such a store empty credentials. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): serve an index with no identities and no engine credentials A second cluster runs weed s3 without -config and weed-vector without --storage-option, with no AWS credentials in either environment: weed s3 would turn them into an admin identity, and the engine would find them. On master its CreateIndex fails with InternalFailure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b9bc42c217 |
s3vectors: vend write credentials to index data only to the vector engine and Lance worker (#12316)
* security: share the filer admin token check CheckGrpcAdminPeer verifies an admin Bearer token inline. Move the check into IsFilerAdminToken so an HTTP endpoint can accept the same token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: recognize the cluster's own components by a signed token A request whose X-Seaweedfs-Cluster-Token header carries an admin token signed with jwt.filer_signing.key comes from a component that holds the cluster key: the vector engine or the Lance maintenance worker. Mark such a request as trusted. It still authenticates as an identity, which stays the principal for vended credentials. A dedicated header leaves the Authorization header to Lance OAuth clients. A gateway with no key trusts no one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: vend write credentials to a vector bucket's indexes only to trusted callers An index in a vector bucket's vector namespace is also a Lance table, so a caller granted s3tables:UpdateTable could describe it, or declare it again, and get credentials that write the index data without passing the vector bucket policy. In that namespace both now vend read-only credentials unless the caller is trusted, and a declare from anyone else no longer writes the dataset markers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance worker: prove the cluster key to the Lance namespace The gateway now vends write credentials to a vector bucket's indexes only to callers that send a token signed with jwt.filer_signing.key. The worker already takes that key for the admin stream; its namespace client now mints a token from it for every catalog request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: describe the index the gateway created and prove the cluster key CreateIndex declared a table the gateway had just created, which only worked because declare is idempotent. Describe it instead. The Lance namespace vends write credentials to a vector bucket's indexes only to callers that send a token signed with jwt.filer_signing.key, so take the key with --filer-signing-key, the same flag the Lance worker has. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance worker: keep the cluster key out of Debug output Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: answer a declare of an existing table with its own location CreateTable succeeds for a name that already exists without looking at the location requested, and declare kept using that location for the dataset markers and the vended credentials. Declaring an existing table with another table's location, such as a vector index, got credentials to that table's data. Use the location the catalog holds, which CreateTable now also reports for a new table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: sign cluster tokens with a key derived for the catalog A cluster token was a filer admin token signed with jwt.filer_signing.key and, by default, never expired. Sent to the Lance namespace on every request, often over plain HTTP, a captured one would also authorize the filer's IAM service. Sign it with a key derived from the cluster key for this one use, which the admin services do not accept, and mint it with a five minute lifetime. weed-vector drops its token lifetime flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: follow a reloaded jwt.filer_signing.key The Lance namespace copied the cluster key at startup. A SIGHUP reload refreshes the gateway's filer guard but not that copy, so after a key rotation the restarted vector engine and worker were no longer trusted. Read the key from the guard on every check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: keep a vector bucket's indexes read-only An index in a vector bucket is also a catalog table the Iceberg REST catalog loads, and LoadTable vended credentials that could write it to any caller granted s3tables:UpdateTable. A commit staged its metadata file in the index directory on the same grant. Apply the vector namespace rule to both, after the caller's own UpdateTable check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: confine a created table's location to its namespace before writing Stage-create skips the registration that confines a location to the table's namespace, yet vends write credentials for that location. A caller allowed to create a table in any namespace of a bucket could name another namespace's prefix, a vector index included, and get write credentials to it. The non-staged create also wrote its first metadata file there before the registration refused the location. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: a declare that finds an existing table answers like describe Creating a name that already exists is idempotent, so a caller allowed only to create tables could declare someone else's table and get write credentials to it, and clear its deregistered marker. Describe would have vended that caller read-only credentials. A declare now looks the name up first; when it only finds the table, writing needs UpdateTable, as it does for describe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
55f1e1efc5 |
vector: cache reads of write-once Lance files across requests (#12303)
* lance worker: let dataset::open take an object store wrapper open_with is open with the dataset's object store wrapped, so a caller such as weed-vector can put a read cache in front of the table's files. open keeps its signature and opens unwrapped, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: cache reads of write-once Lance files across requests Every VectorService call opens the table afresh, and Lance keeps no data pages between opens, so each QueryVectors re-read the whole vector column through the S3 gateway: 83 MB and about 228 range GETs per query at 50k x 384. The gateway and volume server spent their CPU streaming it and throughput stayed at 6-8 queries/s whatever the client concurrency. weed-vector now wraps each table's object store with a bounded in-memory cache (--read-cache-mb, 512 MiB by default, 0 disables). It only serves data files, deletion files and index files: Lance writes each once under a fresh random name, so a cached read cannot go stale, even when an index is dropped and recreated at the same location. Manifests, tags and every other path still go to the store, and each call still describes the table, loads the latest version and takes the query lease, so reads see every write committed before them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: cache an owned copy of each fetched read A range from get_ranges can be a slice of a larger response buffer, such as a coalesced fetch, and caching the slice kept the whole buffer alive while the weigher counted only the slice. The cache now copies each fetched range and whole-object read into its own buffer, so --read-cache-mb bounds what it actually holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
4be191173a |
vector: keep ListVectors pages and segments flat as an index grows; read only keys in GetVectors (#12300)
* vector: scan only the fragments a ListVectors page covers A page pushed its offset into the scan, but the scan still loaded every fragment first, reading each one's deletion file: about 105 S3 GETs per page at 50k rows against 25 at 10k. The page's fragments now come from the live row counts in the manifest, so a page reads only its own. Tokens and row order are unchanged; a manifest without the counts keeps the old scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: list a ListVectors segment in the engine The gateway paged through the whole index for every segment and kept one row in segmentCount, so a segment cost segmentCount engine pages and 16 segments listed slower than one. The engine now takes segment_count and segment_index: it reads keys across the stride, keeps the segment's rows and takes their data and metadata alone, one call per page. A segment is still the rows whose position modulo segmentCount is segmentIndex, and its token still the position to resume from. An engine that predates the fields does not set segmented, and the gateway then pages through it as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: read only the key column across the table in GetVectors The key lookup scanned every column. On S3, lance reads a column during the scan when a row of it is under 1000 bytes, so a 128-dimension vector was read for every row of the table to return 100 of them, and data and metadata were read even when the request did not ask for them. The scan now projects the requested columns and materializes everything but the key late, for the matching rows only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: start the legacy segment fallback from the page already read With an engine that ignores segment_count, the gateway dropped the page it had just read and fetched it again. The fallback now consumes that page first, so a one-row segment page still costs one engine call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: check metadata on segmented ListVectors pages The segment test listed data only. Its rows now carry metadata on two in three keys, and every segmented row, read through take_rows, must equal the same row in a plain listing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: lease the version while a segment takes its data A segmented page with data or metadata reads keys first and then takes the other columns from the same version, so maintenance could reclaim its files in between. That path now holds a lease like GetVectors. A keys-only page is one scan and stays unleased, as plain pages are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: leave GetVectors materialization to lance Forcing every column late made a 100-key GetVectors with data slower at 10k x 128 in the perf cluster: 54.7 ms against 46.3 ms. The take issued a read per row, where lance's heuristic reads a narrow vector column in one pass per fragment. The projection keeps the gain on its own: with neither data nor metadata requested, the call went from 38.6 ms to 25.3 ms and from 150 to 89 S3 GETs. At 50k x 384 the vector is already late under the heuristic, and a full lookup is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a7a6482752 |
vector: follow a leader hint at once and skip one that failed this pass (#12295)
* vector: follow a leader hint at once and skip one that failed this pass A follower naming a dead leader stalled master failover. Each pass broke off at the first hint, tried that leader first on the next pass, then got the same hint from the same follower, so the later configured masters were never tried. A pass now works like wdclient.tryAllMasters: it follows a hint at once, does not follow a hint to a master that already failed in that pass, and moves on to the rest. The hops are bounded by the master count, as the volume heartbeat bounds its hints, so two masters naming each other cannot spin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: test a hint naming a failed master in its gRPC form A pass keys failed masters by host:http, so a configured m1:9333 that failed must also stop a hint of m1:9333.19333. Cover that equivalence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
6cdddcd76d |
vector: confine DropIndex to the table's own path, bound master dials, check the gRPC port (#12294)
* vector: refuse to drop a location that is not the table's own DropIndex removed everything under whatever location the catalog described. That location is caller-supplied through the Lance namespace and only confined to the bucket, so a table registered at s3://bkt/vector, or at another index's directory, wiped every index in the bucket or that index. drop now accepts only the colocated path the gateway's tableLocation derives from the table id, after trimming a trailing slash, and the RPC answers FailedPrecondition otherwise. This mirrors the ancestor and live-entry guards on the catalog's DeleteTable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: bound the master dial and try a leader hint once A remembered leader hint was never cleared and the dial had no connect timeout, so a black-holed former leader added the kernel's SYN timeout (about two minutes) to every reconnect pass before the configured masters were tried, and registration lapsed. The dial now gives up after 5s, the same bound wdclient puts on a master call, and a hint is tried once ahead of the configured masters, as wdclient.tryAllMasters keeps nextHintedLeader for one cycle only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: refuse a master port whose gRPC port would pass 65535 grpc_addr derived the gRPC port as http + 10000 in u16, which panics in debug builds and in release silently dials a wrapped port for HTTP ports above 55535. Go's pb.ServerToGrpcAddress adds in int, so the port never wraps and the dial fails instead. The addition is now checked, and weed-vector refuses to start with an error asking for the explicit host:http.grpc form. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: refuse a numeric master port past 65535 A master given as m1:65536 or m1:9333.65536 failed both u16 parses and fell through to the raw address, so startup accepted it and every dial failed. Ports now parse as u64, as Go's ServerToGrpcAddress does with ParseUint, and one outside u16 is refused at startup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: bound the master's first keep-connected answer The connect timeout only covers TCP. A master that accepts the stream and never answers held stream_to forever: grpc-go sends the response headers with the first message, so the call never returned and later seeds were never tried. A healthy master always answers at once (a follower with its leader hint), so the stream open and the first response now share the same 5s bound; after that the long-lived stream stays untimed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris Lu
|
a1f2886647 | vector: accept Go-style single-dash long flags (#12247) | ||
|
Chris Lu
|
15066c8a9f |
vector: implement drop_index and have DeleteIndex drop the data (#12204)
* vector: implement drop_index in the engine The RPC was a stub; it now resolves the table's vended location and removes every object under it through the same credential path open uses. A table the catalog can no longer describe is already dropped, so the call stays idempotent. * s3vectors: drop index data after the catalog delete DeleteTable carries the delete authorization, so it runs first; the engine DropIndex then removes the data under the index's location. NotFound and Unimplemented are tolerated so an index whose data went away with the catalog entry, or an older engine, still answers 200. * s3vectors: let the catalog delete carry the index data removal A vector index is colocated: the catalog entry sits at the dataset root, so DeleteTable's recursive delete already removes entry and data. The engine DropIndex call afterwards could only ever describe a missing or recreated record — a no-op in the common case, and a delete of a replacement index's files when the name was reused in between. |
||
|
Chris Lu
|
511727cf3a |
vector: DeleteIndex format guard, sticky discovery, advertise normalize (#12171)
* vector: DeleteIndex format guard, sticky discovery, advertise normalize - handleDeleteIndex verifies the table carries the Lance format before DeleteTable — an Iceberg table under the same namespace is not an index and must not be removed by it. - discovery keeps the current worker while it is still registered — membership-order churn can no longer cancel in-flight calls, and the resolver closes each master conn inside a 5s deadline. - advertised_addr strips scheme and .grpcPort so the UDP probe reaches every supported master form. * vector: empty-follower retry, format check without read grant - discoverVectorWorkers keeps looking past an empty master reply — followers list nothing when workers registered with the leader. - the DeleteIndex format check uses the internal catalog read, so DeleteTable-only callers don't gain a GetTable requirement, and the format must be Lance exactly — format-less tables no longer pass. |
||
|
Chris Lu
|
b3d0caabe2 |
vector: only follow a leader hint that points elsewhere (#12164)
* vector: only follow a leader hint that points elsewhere The master sends volume_location on every KeepConnected response — the leader always names itself, so the worker exited the stream on the first response and flapped every 3s, leaving ListClusterNodes to see the node intermittently. Only redirect when the hint points at a different master. * vector: leader hints carry host:http.grpc — normalize before comparing A self-hint arrives as host:9333.19333 while --masters gives host:9333, so the raw comparison still redirected and grpc_addr could not parse the dotted port. grpc_addr now prefers the explicit .grpcPort, and the hint comparison normalizes both sides to host:httpPort first. |
||
|
Chris Lu
|
190ab32cfb |
vector: weed-vector registers to master; s3 discovers it there (#12146)
* vector: weed-vector registers with the weed master weed-vector-server --masters holds a KeepConnected stream to the weed master, the same mechanism filer and s3 use via wdclient — the server shows up in ListClusterNodes(client_type=vector). Resends registration and rediscovers on failure. * vector: s3 discovers weed-vector through the master With -vectors.grpc unset the vectors gateway resolves one registered weed-vector via ListClusterNodes on the masters it already derives from filer config — discovery replaces the static dial for the single-worker OSS path. The conformance harness registers the engine to mini's master and drops -vectors.grpc. * vector: default-fill registration fields ..Default::default() keeps the source compiling against proto variants that carry extra fields, so the same file works in either tree. * vector: serve the Vectors API on the main s3 port The vectors router sits on the main -port listener behind a dispatch that checks the SigV4 credential scope for /s3vectors/ — no separate port needed. -port.vectors stays for explicit dedicated-port deployments; the conformance harness now hits vectors through :8334. * vector: drop -port.vectors entirely It was new — no compat to preserve. The main -port listener serves the API whenever a weed-vector backend is reachable. * vector: drop -vectors.grpc; discovery is the only locator weed-vector always registers to master, the gateway always resolves it through ListClusterNodes — a static address just forks membership. * vector: binary is weed-vector Matches the enterprise binary; package and lib names are unchanged. * vector: review fixes — cluster type, grpc port, lazy resolve, leader hint, advertise - cluster: register 'vector' as a node type so ListClusterNodes returns weed-vector members - s3: dial masters via ToGrpcAddress (HTTP port + 10000) - s3: backend resolves lazily through vectorDiscovery — workers joining after gateway startup work on first query, and address changes re-dial - master_client: convert master HTTP addr to grpc +10000, honor http(s) scheme, follow the leader hint in KeepConnectedResponse, and --advertise covers wildcard/unreachable listen addresses * vector: no --advertise flag; derive the outbound address A wildcard listen resolves to the source IP the master sees via a route lookup, so registration always advertises something reachable without another flag. * vector: review fixes — plaintext dispatch, conn leaks, discovery backoff The two plaintext http servers now serve the vectors dispatch too — signed s3vectors calls on the main port reach the right router. discoverVectorWorker closes each master conn and bounds lookups to 5s; a dead or absent worker no longer re-hits master on every RPC — the resolver backs off for 5s between rediscovery attempts. * vector: 1 GiB grpc cap, dead-worker invalidation, CLI rename Valid PutVectors batches exceed the tonic 4 MiB default — match the gateway's 1 GiB cap. A replaced worker no longer sticks: any RPC error resets the cached address and the next call rediscovers through the master. The core CLI bin renames to weed-vector-cli so the two crates do not collide on the same output path. * vector: retry-only-transport reset, periodic re-resolve, cli rename - reset() takes the failed client and runs only on transport errors — an InvalidArgument on a healthy conn no longer interrupts unrelated requests, and a concurrent re-resolve wins the race. - cli() re-resolves once the interval passes even with a live client, so a worker that moves addresses is picked up. - verify.rs spike docs use weed-vector-cli after the bin rename. * vector: only Unavailable counts as a transport failure A request deadline or an application error can fire on a healthy connection — resetting the shared conn on those churned reconnects. |
||
|
Chris LuandDevin
|
af77406cf9 |
vector: index-segment restriction + partition helper (P3 seam) (#12124)
* vector: allow restricting ANN search to index segments Query gains an optional index_segments list; when set and the index path is used, the scanner restricts the vector index search to those segments via with_index_segments. This is the seam the Enterprise planner uses to fan one query out across per-segment workers; OSS keeps passing None. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: enumerate and partition index segments for fanout segments::index_segments lists the physical segment UUIDs covering the vector field at the pinned version; partition() assigns them round-robin to workers. The parity test proves two workers scanning disjoint shares merge back to the full-dataset top-K — the P3 exit gate, in-process. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: let a query carry fragment ids for the unindexed sweep Lance only applies the index-segment restriction to indexed rows — unindexed fragments outside the selected segments are skipped. For a partitioned query to stay correct, the planner needs to hand the unindexed sweep to a worker too, so Query gains a fragments list that the scanner resolves to manifest fragments via with_fragments. A task carrying both searches its segments via the index and its fragments with flat KNN in the same scan. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: empty scope returns no hits; cover the sweep in tests Some([]) fragments or index_segments fell through to an unrestricted scan — a worker assigned nothing would have searched everything. empty_scope short-circuits those queries, and the segment restriction only calls with_index_segments on a non-empty list. segments::segment_fragments also returns each segment's live fragment bitmap, and the parity test now leaves a fragment unindexed so the worker carrying it exercises the flat sweep — the contract the Enterprise planner relies on. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: scope the scan to listed ∪ segment-covered fragments Lance intersects with_fragments with the selected index segments, so a worker given its segment share plus the unindexed sweep lost the share entirely — only the listed fragments were scanned. Expand the fragment list with the live coverage of the selected segments. The list is now also applied when it resolves empty: stale ids or an empty assignment must scan nothing, not fall through to an unrestricted scan. The merge test now leaves a fragment unindexed so the combined path is exercised. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: partition assigns nothing when there are no workers n.max(1) returned every segment in one phantom partition for a nonexistent worker, silently dropping coverage. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
8379ee9251 |
vector: snapshot lease + maintenance safety tests (P2) (#12107)
* vector: version-lease helpers + maintenance safety tests Pinned queries must survive Lance maintenance. maintenance::pin tags the current dataset version so cleanup refuses to reclaim it; release drops the tag. Tests prove the P2 gate: a pinned query returns identical hits across compact_files, a tagged version survives cleanup_old_versions (error_if_tagged protects it, the non-strict run skips it), and a missing data file fails the query instead of returning partial hits. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: lease the query's dataset version in the VectorService query_vectors pins the resolved version with a query tag for the duration of the RPC and releases it after, so concurrent maintenance cleanup cannot reclaim the files out from under an in-flight read. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: release the version lease on every query exit path The lease tag was only released on query_vectors' success path — get_vectors never released and query_vectors leaked its tag on early returns. pin now returns a guard whose drop deletes the tag, so every path frees the version. * vector: unique lease tag names + reap abandoned leases Nanos-since-epoch tag names can collide across concurrent readers and processes, so each lease now carries a unique suffix. pin() also drops query tags older than an hour, since a killed or cancelled request leaves its lease behind and would otherwise block cleanup forever. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: keep live lease tags + throttle abandoned-lease scans An expired tag is only reaped once no live lease in this process owns it, so a query running past LEASE_TTL keeps its pin. Tag scans run at most once per 60s per dataset instead of on every request. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: answer read-only queries without a version lease A namespace may hand the server credentials that can read a table but not update it, so the tag write in pin() fails and the whole query errors out. try_pin() degrades a denied tag write to an unleased query instead of failing the request. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: fail a lease whose pinned version is already gone Cleanup lists tags once at the start, so a tag created mid-cleanup does not protect the version it names. After the tag lands, pin now verifies the version still exists and drops the lease when it does not, failing the read at pin time instead of mid-query. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
e2d899ac25 |
test/vectors: fix the S3 Vectors e2e harness (#12112)
* test/vectors: give the vector server static storage credentials The e2e harness ran weed-vector-server without credentials, so lance's AWS object store fell back to IMDS and CreateIndex died on the runner's 411 from the token endpoint. Point it at the test credentials the s3 gateway now reads from s3.json. * vector: take the list item field from the schema when batching vectors Lance stores the fixed-size list item as nullable even when the write schema declared it non-null, so PutVectors built batches that no longer matched: expected FixedSizeList(4 x Float32) but found FixedSizeList(4 x non-null Float32). * vector: store index metadata as lance jsonb so filters resolve The metadata field was plain Utf8, but the filter functions lance's planner registers (json_get_string, json_get_float, json_exists) take a LargeBinary jsonb column, so QueryVectors with a metadata filter failed to plan. Declare the field with the arrow.json extension — lance encodes it to jsonb on write and back on read — and encode the batch's strings when the stored column is LargeBinary. Also rename the filter's json_get_str call to json_get_string, the name lance registers. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
87f15e4d6c |
vector: weed-vector-server, the VectorService gRPC (P1) (#12102)
* vector: VectorService protobuf contract The shared RPC boundary between the Go S3 gateway and the Rust vector engine from the vector design doc: CreateIndex/DropIndex, Put/Get/ List/Delete/Query vectors. A Table reference carries the Lance table id and an optional pinned version, matching the snapshot discipline queries are required to keep. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: generated Go stubs for vector.proto protoc output via weed/pb/Makefile. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: carry return_data through QueryVectors S3 Vectors QueryVectors honours returnData, but the contract had no way to ask for or carry vector data in a query response, so the gateway would need a second GetVectors round-trip per query. Add return_data to the request and data to QueryResult now, while the contract is still new. * vector: pin metadata numbers to double precision in the contract Metadata arrives as JSON, where numbers carry double precision; say so on the field so neither side promises integer fidelity past 2^53. * vector: declare_table on the lance namespace client CreateIndex needs a way to ask the catalog for a new table's location; expose POST /v1/table/{id}/declare. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: cargo fmt coverage.rs Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: weed-vector-server, the VectorService gRPC A tonic server over weed-vector-core: CreateIndex declares the table through the namespace and writes an empty {key, vector, metadata} dataset; PutVectors upserts by key (delete + append until merge_insert is exercised); Get/List/Delete/Query route through the engine's resolved table, honoring pinned versions and the prefilter default. Metadata travels as a Utf8 JSON column, proto Struct on the wire. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: stamp the metric in the schema, tighten put validation CreateIndex now bounds dimension at the S3 Vectors 4096 maximum and records the distance metric in the vector field's Arrow metadata, so a later reader recovers the index's configured metric from the table itself. batch_of checks every vector's data length individually before flattening — equal totals could still let a short vector borrow floats from its neighbor — and writes against the dataset's own schema. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: upsert vectors in a single merge commit PutVectors ran delete-then-append: a failed append left the old keys deleted, and two concurrent puts could both append a replacement. Use lance's merge_insert on the key column — one commit, no delete window, conflicts fail the request instead of duplicating keys. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: read the index's metric; fetch metadata in one take QueryVectors hardcoded L2 — it now reads the metric the index was created with from the vector field's metadata. return_metadata fetched each hit's row one at a time; collect the row ids into a single take_rows call and join metadata back on the hit key. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: honor return_data in query results QueryResult grew a data field but query_vectors never populated it. Fetch the vector column in the same batched take_rows as metadata and return each hit's stored embedding. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: reject a malformed query as invalid_argument A zero top_k silently became one result and a dim-mismatched query vector surfaced as an internal scan error; reject both before querying. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: reject a malformed list next_token A token that failed to parse silently restarted the listing from offset zero; surface it as invalid_argument instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: answer an empty put without a commit and cover ragged input An empty vectors list would still run a merge commit; return the current version instead. Pin the ragged-vector check with a test. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: keep non_filterable_metadata_keys on the table schema create_index dropped the declared filter policy. Persist it as JSON on the metadata field so the index carries which keys a filter may touch; enforcing the list belongs to the gateway that translates the request filter into Lance SQL. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: reject duplicate keys in one put A repeated key for a new row committed both copies, and for an existing key the merge failure surfaced as internal. Reject duplicates up front as invalid_argument. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: report open failures as not_found or unavailable A catalog 404 or a missing dataset/version is not a client mistake, and a catalog outage is retryable: mapping every open error to invalid_argument made both indistinguishable from a bad request. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: page list_vectors when max_results is unset A proto3 client that omits max_results sends 0, which used to limit the scan to zero rows and return no token. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: drop stray empty service.rs.tmp * vector: rename datas map to data_map codespell flags datas as a misspelling of data; use a distinct map name. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |