mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-04 11:07:35 +01:00
de4a71665e14daae9f34cbee00a885084f0009c2
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
5063aa9be2 |
weed-vector: check master certificates against the server allowlists, read -config_dir (#12374)
* weed-vector: look for security.toml in -config_dir first weed processes search the directory -config_dir names before ., so a cluster keeping security.toml there left the engine in plaintext. --config-dir (also -config_dir, WEED_VECTOR_CONFIG_DIR) does the same. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: dial the masters the way weed clients do The master's certificate must chain to grpc.ca and is not checked against the host dialed, as weed clients verify (advancedtls.CertVerification), so masters sharing one workload certificate, or dialed by IP, register as they do from weed. Instead, grpc.client.master_allowed_server_commonNames, grpc.client.allowed_server_commonNames or grpc.allowed_server_commonNames (and the wildcard domain likewise) restrict which server identities are accepted: the common name or a DNS SAN, or an exact URI SAN, as security.Authenticator.AuthenticateServer does. These used to be refused at startup; --tls-server-name is gone with the host check. The TLS runs in a connector that reads the credentials asynchronously at every dial, so the connect timeout still covers the reads, the TCP connect and the handshake together. An IPv6 master's brackets are stripped before the dial. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: the hardened leg allowlists server identities for grpc.client The gateway and the engine then accept only servers named localhost, which exercises the engine's allowed_server_commonNames check on the master dial. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: parse the refresh interval with time.ParseDuration's arithmetic The interval was summed in floating-point seconds, so 0.9ns rounded up to 1ns where Go truncates it to 0 and falls back to the 5h default, 0.3s was off by a nanosecond, and 3000000h was accepted where Go rejects anything past the longest time.Duration. Each part is now an integer count of nanoseconds plus the fraction Go adds, with Go's overflow bounds, and the value is no longer trimmed, as Go does not trim it either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: test a master dial whose credential read stalls The bounded-dial tests passed no credentials or a prebuilt endpoint, so moving the credential read out of the connect timeout would have failed none of them. The new test points grpc.ca at a FIFO nobody writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: dial a closed port in the stalled credential read test Against a silent listener, a dial that never read the credentials timed out with the same message, so the test passed without the read. Nothing listens on the port now, so only the stalled read can time out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: test that Security::load searches the config dir Only candidates() was tested, so load dropping config_dir on the way would have passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: expand a leading ~ in -config_dir like weed's ResolvePath Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: hold the port in the stalled credential read test A released port can be taken before the dial, and a dial that skipped the read would then time out and pass. A listener that drops every connection keeps the port and still fails such a dial at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
eeb8aefe70 |
weed-vector: gRPC TLS to the masters and for the gateway (#12368)
* weed-vector: read the cluster's security.toml The engine finds security.toml where weed processes do (., ~/.seaweedfs, /usr/local/etc/seaweedfs, /etc/seaweedfs) with WEED_* environment variables over it, and builds gRPC client and server credentials by the rules of security.LoadClientTLS and LoadServerTLS: a dedicated client certificate is preferred, both halves of a pair must be set, and grpc.ca is required once either is. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: gRPC TLS to the masters and from the gateway The masters are dialed with [grpc.client]'s credentials when the cluster has them, the master's certificate checked against the host dialed or --tls-server-name. The engine's own listener takes its identity from [grpc.vector] and then requires a client certificate chaining to grpc.ca, which is how the gateway, dialing with [grpc.client], reaches it. The rustls provider is pinned before the first TLS use, as the other binaries linking both aws-lc-rs and ring do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * scaffold: document [grpc.vector] Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: put every gRPC hop under mutual TLS in the hardened leg VECTOR_TLS=1 now also writes [grpc.master], [grpc.volume], [grpc.filer], [grpc.s3], [grpc.client] and [grpc.vector], and probes the master's and the engine's gRPC ports: plaintext HTTP/2 and a client without a certificate are refused, the client certificate gets a SETTINGS frame. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: give the test volume server 64 volume slots Every collection grows 7 volumes and test_vector_bucket_encryption opens two buckets while the previous test's bucket is still being torn down, so 16 slots were marginal: the Go legs failed CreateIndex now and then with "failed to find writable volumes" and master.log showing "create 7 volume, created 4: Not enough data nodes found!". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: reload the listener's certificates and enforce allowed_commonNames The listener now runs the way weed servers run theirs. The certificate, key and CA files are re-read every WEED_TLS_CERT_REFRESH_INTERVAL (weed's default of 5h) and on SIGHUP, and new handshakes use the changed files without a restart; a file that no longer parses is logged and the working configuration stays. [grpc.vector] allowed_commonNames and grpc.allowed_wildcard_domain restrict which client certificates are served, checked against the subject common name as security.Authenticator and the Rust volume server's CommonNameVerifier do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * scaffold: allowed_commonNames and the refresh cadence for [grpc.vector] Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the engine's allowlist keeps an outsider certificate out The hardened leg sets [grpc.vector] allowed_commonNames = "localhost" and probes the engine with a second certificate the same CA signed for CN=outsider: the handshake must be refused, while the gateway's certificate still gets an HTTP/2 SETTINGS frame. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: take the master's TLS name from the parsed endpoint A master given without a port (https://master.example) had the whole URL checked against its certificate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: bound the master TLS handshake connect_timeout covers only the TCP connect, so a master that accepted the connection and never answered the handshake held up the other masters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: look up security.toml keys the way viper does An empty WEED_* variable no longer hides the file's value, and keys match regardless of case, as for weed processes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: refuse master allowlists it cannot enforce weed checks a dialed server against allowed_server_commonNames and allowed_server_wildcard_domain on top of grpc.ca. The engine's master dial accepted every certificate the CA signed even when one was set; it now fails at startup and names the key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: read the whole frame header in the gRPC TLS probe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: probe the volume, filer and S3 gRPC listeners too Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read the master dial credentials only when given masters A standalone engine never dials with grpc.client, so its settings no longer stop it from starting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: run each listener TLS handshake on its own task The accept loop awaited every handshake before accepting the next connection, so a peer that connected and sent nothing stopped every gateway connection after it. Handshakes now run concurrently, each under a 10s timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read the master dial's TLS files at every dial The client certificate, key and CA were read once at startup, so after the files rotated a reconnect still presented the old pair. Each dial now reads them, as the listener re-reads its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read every unit Go durations accept in the refresh interval WEED_TLS_CERT_REFRESH_INTERVAL values in ns, us or a leading + fell back to five hours in the engine while weed servers honored them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: fall back on a refresh interval too long to hold An oversized WEED_TLS_CERT_REFRESH_INTERVAL panicked the engine at startup; weed servers reject it and keep the default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: refuse allowed_commonNames without the listener's certificate With [grpc.vector] allowed_commonNames set but no cert and key, the listener served plaintext and checked no client at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read the master dial's credentials inside its connect timeout The CA, certificate and key were read synchronously before the timeout started, so a stalled credential path held up every other master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: let a set key win over an empty spelling of it With allowed_server_commonNames = "" beside allowed_server_commonnames = "master", the lookup stopped at the empty exact match and the allowlist went unnoticed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
55f1e1efc5 |
vector: cache reads of write-once Lance files across requests (#12303)
* lance worker: let dataset::open take an object store wrapper open_with is open with the dataset's object store wrapped, so a caller such as weed-vector can put a read cache in front of the table's files. open keeps its signature and opens unwrapped, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: cache reads of write-once Lance files across requests Every VectorService call opens the table afresh, and Lance keeps no data pages between opens, so each QueryVectors re-read the whole vector column through the S3 gateway: 83 MB and about 228 range GETs per query at 50k x 384. The gateway and volume server spent their CPU streaming it and throughput stayed at 6-8 queries/s whatever the client concurrency. weed-vector now wraps each table's object store with a bounded in-memory cache (--read-cache-mb, 512 MiB by default, 0 disables). It only serves data files, deletion files and index files: Lance writes each once under a fresh random name, so a cached read cannot go stale, even when an index is dropped and recreated at the same location. Manifests, tags and every other path still go to the store, and each call still describes the table, loads the latest version and takes the query lease, so reads see every write committed before them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: cache an owned copy of each fetched read A range from get_ranges can be a slice of a larger response buffer, such as a coalesced fetch, and caching the slice kept the whole buffer alive while the weigher counted only the slice. The cache now copies each fetched range and whole-object read into its own buffer, so --read-cache-mb bounds what it actually holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
5a60ce04e3 |
vector: stop re-parsing a filter's JSONPath for every scanned row (#12302)
* vector: parse a filter's JSONPath once per batch instead of per row
Lance's json_extract and json_exists parse their JSONPath argument again
for every row they see. Every translated S3 Vectors filter goes through
them, so a flat scan paid about 1 µs per row before the predicate
compared anything, and a 10%-selective {"category": {"$eq": "c3"}} ran
about 50% slower than the same query unfiltered.
The path is always a literal, so the scanner now parses the filter as
before and swaps in copies of the two functions that parse it once per
batch. They call the same jsonb selector and serializer, so results and
errors do not change. Calls whose path is not a literal keep Lance's
functions. The filter SQL is unchanged, so the gateway, existing tables
and the distributed workers (which run query::ann) need no change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: read the field once per extractor in $in and $nin
$in and $nin rendered one IN per member, each with its own json_extract,
so an N-member list parsed the row's metadata N times. Members that share
an extractor (strings and booleans, or numbers) now go into one IN list,
so a list reads the field at most twice. Matching is unchanged: each
group still only sees its own JSON type and still coalesces NULL before
$nin negates it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: rename the $in literal map so codespell passes
Codespell reads "lits" as a misspelling of "list".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Chris LuandClaude Opus 5.5
|
4be191173a |
vector: keep ListVectors pages and segments flat as an index grows; read only keys in GetVectors (#12300)
* vector: scan only the fragments a ListVectors page covers A page pushed its offset into the scan, but the scan still loaded every fragment first, reading each one's deletion file: about 105 S3 GETs per page at 50k rows against 25 at 10k. The page's fragments now come from the live row counts in the manifest, so a page reads only its own. Tokens and row order are unchanged; a manifest without the counts keeps the old scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: list a ListVectors segment in the engine The gateway paged through the whole index for every segment and kept one row in segmentCount, so a segment cost segmentCount engine pages and 16 segments listed slower than one. The engine now takes segment_count and segment_index: it reads keys across the stride, keeps the segment's rows and takes their data and metadata alone, one call per page. A segment is still the rows whose position modulo segmentCount is segmentIndex, and its token still the position to resume from. An engine that predates the fields does not set segmented, and the gateway then pages through it as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: read only the key column across the table in GetVectors The key lookup scanned every column. On S3, lance reads a column during the scan when a row of it is under 1000 bytes, so a 128-dimension vector was read for every row of the table to return 100 of them, and data and metadata were read even when the request did not ask for them. The scan now projects the requested columns and materializes everything but the key late, for the matching rows only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: start the legacy segment fallback from the page already read With an engine that ignores segment_count, the gateway dropped the page it had just read and fetched it again. The fallback now consumes that page first, so a one-row segment page still costs one engine call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: check metadata on segmented ListVectors pages The segment test listed data only. Its rows now carry metadata on two in three keys, and every segmented row, read through take_rows, must equal the same row in a plain listing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: lease the version while a segment takes its data A segmented page with data or metadata reads keys first and then takes the other columns from the same version, so maintenance could reclaim its files in between. That path now holds a lease like GetVectors. A keys-only page is one scan and stays unleased, as plain pages are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: leave GetVectors materialization to lance Forcing every column late made a 100-key GetVectors with data slower at 10k x 128 in the perf cluster: 54.7 ms against 46.3 ms. The take issued a read per row, where lance's heuristic reads a narrow vector column in one pass per fragment. The projection keeps the gain on its own: with neither data nor metadata requested, the call went from 38.6 ms to 25.3 ms and from 150 to 89 S3 GETs. At 50k x 384 the vector is already late under the heuristic, and a full lookup is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris Lu
|
190ab32cfb |
vector: weed-vector registers to master; s3 discovers it there (#12146)
* vector: weed-vector registers with the weed master weed-vector-server --masters holds a KeepConnected stream to the weed master, the same mechanism filer and s3 use via wdclient — the server shows up in ListClusterNodes(client_type=vector). Resends registration and rediscovers on failure. * vector: s3 discovers weed-vector through the master With -vectors.grpc unset the vectors gateway resolves one registered weed-vector via ListClusterNodes on the masters it already derives from filer config — discovery replaces the static dial for the single-worker OSS path. The conformance harness registers the engine to mini's master and drops -vectors.grpc. * vector: default-fill registration fields ..Default::default() keeps the source compiling against proto variants that carry extra fields, so the same file works in either tree. * vector: serve the Vectors API on the main s3 port The vectors router sits on the main -port listener behind a dispatch that checks the SigV4 credential scope for /s3vectors/ — no separate port needed. -port.vectors stays for explicit dedicated-port deployments; the conformance harness now hits vectors through :8334. * vector: drop -port.vectors entirely It was new — no compat to preserve. The main -port listener serves the API whenever a weed-vector backend is reachable. * vector: drop -vectors.grpc; discovery is the only locator weed-vector always registers to master, the gateway always resolves it through ListClusterNodes — a static address just forks membership. * vector: binary is weed-vector Matches the enterprise binary; package and lib names are unchanged. * vector: review fixes — cluster type, grpc port, lazy resolve, leader hint, advertise - cluster: register 'vector' as a node type so ListClusterNodes returns weed-vector members - s3: dial masters via ToGrpcAddress (HTTP port + 10000) - s3: backend resolves lazily through vectorDiscovery — workers joining after gateway startup work on first query, and address changes re-dial - master_client: convert master HTTP addr to grpc +10000, honor http(s) scheme, follow the leader hint in KeepConnectedResponse, and --advertise covers wildcard/unreachable listen addresses * vector: no --advertise flag; derive the outbound address A wildcard listen resolves to the source IP the master sees via a route lookup, so registration always advertises something reachable without another flag. * vector: review fixes — plaintext dispatch, conn leaks, discovery backoff The two plaintext http servers now serve the vectors dispatch too — signed s3vectors calls on the main port reach the right router. discoverVectorWorker closes each master conn and bounds lookups to 5s; a dead or absent worker no longer re-hits master on every RPC — the resolver backs off for 5s between rediscovery attempts. * vector: 1 GiB grpc cap, dead-worker invalidation, CLI rename Valid PutVectors batches exceed the tonic 4 MiB default — match the gateway's 1 GiB cap. A replaced worker no longer sticks: any RPC error resets the cached address and the next call rediscovers through the master. The core CLI bin renames to weed-vector-cli so the two crates do not collide on the same output path. * vector: retry-only-transport reset, periodic re-resolve, cli rename - reset() takes the failed client and runs only on transport errors — an InvalidArgument on a healthy conn no longer interrupts unrelated requests, and a concurrent re-resolve wins the race. - cli() re-resolves once the interval passes even with a live client, so a worker that moves addresses is picked up. - verify.rs spike docs use weed-vector-cli after the bin rename. * vector: only Unavailable counts as a transport failure A request deadline or an application error can fire on a healthy connection — resetting the shared conn on those churned reconnects. |
||
|
Chris LuandDevin
|
8379ee9251 |
vector: snapshot lease + maintenance safety tests (P2) (#12107)
* vector: version-lease helpers + maintenance safety tests Pinned queries must survive Lance maintenance. maintenance::pin tags the current dataset version so cleanup refuses to reclaim it; release drops the tag. Tests prove the P2 gate: a pinned query returns identical hits across compact_files, a tagged version survives cleanup_old_versions (error_if_tagged protects it, the non-strict run skips it), and a missing data file fails the query instead of returning partial hits. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: lease the query's dataset version in the VectorService query_vectors pins the resolved version with a query tag for the duration of the RPC and releases it after, so concurrent maintenance cleanup cannot reclaim the files out from under an in-flight read. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: release the version lease on every query exit path The lease tag was only released on query_vectors' success path — get_vectors never released and query_vectors leaked its tag on early returns. pin now returns a guard whose drop deletes the tag, so every path frees the version. * vector: unique lease tag names + reap abandoned leases Nanos-since-epoch tag names can collide across concurrent readers and processes, so each lease now carries a unique suffix. pin() also drops query tags older than an hour, since a killed or cancelled request leaves its lease behind and would otherwise block cleanup forever. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: keep live lease tags + throttle abandoned-lease scans An expired tag is only reaped once no live lease in this process owns it, so a query running past LEASE_TTL keeps its pin. Tag scans run at most once per 60s per dataset instead of on every request. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: answer read-only queries without a version lease A namespace may hand the server credentials that can read a table but not update it, so the tag write in pin() fails and the whole query errors out. try_pin() degrades a denied tag write to an unleased query instead of failing the request. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: fail a lease whose pinned version is already gone Cleanup lists tags once at the start, so a tag created mid-cleanup does not protect the version it names. After the tag lands, pin now verifies the version still exists and drops the lease when it does not, failing the read at pin time instead of mid-query. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
e2d899ac25 |
test/vectors: fix the S3 Vectors e2e harness (#12112)
* test/vectors: give the vector server static storage credentials The e2e harness ran weed-vector-server without credentials, so lance's AWS object store fell back to IMDS and CreateIndex died on the runner's 411 from the token endpoint. Point it at the test credentials the s3 gateway now reads from s3.json. * vector: take the list item field from the schema when batching vectors Lance stores the fixed-size list item as nullable even when the write schema declared it non-null, so PutVectors built batches that no longer matched: expected FixedSizeList(4 x Float32) but found FixedSizeList(4 x non-null Float32). * vector: store index metadata as lance jsonb so filters resolve The metadata field was plain Utf8, but the filter functions lance's planner registers (json_get_string, json_get_float, json_exists) take a LargeBinary jsonb column, so QueryVectors with a metadata filter failed to plan. Declare the field with the arrow.json extension — lance encodes it to jsonb on write and back on read — and encode the batch's strings when the stored column is LargeBinary. Also rename the filter's json_get_str call to json_get_string, the name lance registers. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
87f15e4d6c |
vector: weed-vector-server, the VectorService gRPC (P1) (#12102)
* vector: VectorService protobuf contract The shared RPC boundary between the Go S3 gateway and the Rust vector engine from the vector design doc: CreateIndex/DropIndex, Put/Get/ List/Delete/Query vectors. A Table reference carries the Lance table id and an optional pinned version, matching the snapshot discipline queries are required to keep. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: generated Go stubs for vector.proto protoc output via weed/pb/Makefile. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: carry return_data through QueryVectors S3 Vectors QueryVectors honours returnData, but the contract had no way to ask for or carry vector data in a query response, so the gateway would need a second GetVectors round-trip per query. Add return_data to the request and data to QueryResult now, while the contract is still new. * vector: pin metadata numbers to double precision in the contract Metadata arrives as JSON, where numbers carry double precision; say so on the field so neither side promises integer fidelity past 2^53. * vector: declare_table on the lance namespace client CreateIndex needs a way to ask the catalog for a new table's location; expose POST /v1/table/{id}/declare. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: cargo fmt coverage.rs Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: weed-vector-server, the VectorService gRPC A tonic server over weed-vector-core: CreateIndex declares the table through the namespace and writes an empty {key, vector, metadata} dataset; PutVectors upserts by key (delete + append until merge_insert is exercised); Get/List/Delete/Query route through the engine's resolved table, honoring pinned versions and the prefilter default. Metadata travels as a Utf8 JSON column, proto Struct on the wire. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: stamp the metric in the schema, tighten put validation CreateIndex now bounds dimension at the S3 Vectors 4096 maximum and records the distance metric in the vector field's Arrow metadata, so a later reader recovers the index's configured metric from the table itself. batch_of checks every vector's data length individually before flattening — equal totals could still let a short vector borrow floats from its neighbor — and writes against the dataset's own schema. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: upsert vectors in a single merge commit PutVectors ran delete-then-append: a failed append left the old keys deleted, and two concurrent puts could both append a replacement. Use lance's merge_insert on the key column — one commit, no delete window, conflicts fail the request instead of duplicating keys. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: read the index's metric; fetch metadata in one take QueryVectors hardcoded L2 — it now reads the metric the index was created with from the vector field's metadata. return_metadata fetched each hit's row one at a time; collect the row ids into a single take_rows call and join metadata back on the hit key. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: honor return_data in query results QueryResult grew a data field but query_vectors never populated it. Fetch the vector column in the same batched take_rows as metadata and return each hit's stored embedding. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: reject a malformed query as invalid_argument A zero top_k silently became one result and a dim-mismatched query vector surfaced as an internal scan error; reject both before querying. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: reject a malformed list next_token A token that failed to parse silently restarted the listing from offset zero; surface it as invalid_argument instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: answer an empty put without a commit and cover ragged input An empty vectors list would still run a merge commit; return the current version instead. Pin the ragged-vector check with a test. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: keep non_filterable_metadata_keys on the table schema create_index dropped the declared filter policy. Persist it as JSON on the metadata field so the index carries which keys a filter may touch; enforcing the list belongs to the gateway that translates the request filter into Lance SQL. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: reject duplicate keys in one put A repeated key for a new row committed both copies, and for an existing key the merge failure surfaced as internal. Reject duplicates up front as invalid_argument. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: report open failures as not_found or unavailable A catalog 404 or a missing dataset/version is not a client mistake, and a catalog outage is retryable: mapping every open error to invalid_argument made both indistinguishable from a bad request. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: page list_vectors when max_results is unset A proto3 client that omits max_results sends 0, which used to limit the scan to zero rows and return no token. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: drop stray empty service.rs.tmp * vector: rename datas map to data_map codespell flags datas as a misspelling of data; use a distinct map name. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
8498ad716b |
vector: spike CLI, reference-comparison suite, CI (P0 phase 3) (#12098)
* vector: project _distance and _rowid explicitly Lance warns that a knn projection without _distance will stop auto-including it; spell both scoring columns out. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: reference-comparison tests The spike's exit gate: assert the engine's answers equal a bare scan().nearest() on the same pinned version — over a fresh index, over unindexed fragments, over a two-segment index after optimize_indices, and after a delete. A version-pinned handle querying through maintenance sees only its own version. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: weed-vector spike CLI describe / coverage / query / get against a namespace-registered table or a plain dataset URI, so the spike runs against a real SeaweedFS Lance table bucket and against a local fixture alike. Flag and env names mirror weed-worker. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: rust-vector-tests workflow Build and test seaweed-vector in release on pushes and PRs touching it or its path-dependency trees, mirroring rust-worker-tests. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: derive Serialize for the spike CLI's JSON output Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: --postfilter on the spike CLI Track the Query field rename; the flag stays an explicit opt-out of prefiltering for comparison runs. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: assert the pinned query's answers The maintenance test passed even with zero pinned hits: an empty result satisfies all() trivially. Assert the hit count and compare the pinned answers against the reference on the pinned handle. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: vector tests also watch plugin.proto weed-vector-core path-depends on weed-lance-worker, which compiles weed/pb/plugin.proto through seaweed-worker-core; a proto-only change would skip these checks and could break this workspace. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: describe prints layout and coverage The describe doc comment promised the vector layout and index coverage; print both, and propagate count_rows errors instead of printing 0. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
eda4b9efdd |
vector: ANN coverage introspection + prefilter (P0 phase 2) (#12097)
* vector: share the test dataset builder The coverage tests need the same tempdir Lance datasets the query tests build; move them into a cfg(test) helper module. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: prefilter the metadata filter prefilter=false post-filters the top-K rows, which can return fewer than k results and never sees unindexed rows beyond the nearest cut. Expose the choice so the S3 Vectors layer can honour AWS filter semantics. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: index coverage introspection coverage() unions each index's fragment bitmaps against the manifest's fragment list to report which fragments a vector index covers, which it has not reached, and how many rows the flat pass still has to scan. Tests pin the behaviour the spike is for: rows appended after an index build stay in the top-K, and a version-pinned dataset never sees the next commit. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: default filtered queries to prefilter postfilter=false is now the normal path: the metadata filter cuts the eligible rows before ranking, so a query returns k hits whenever k eligible rows exist. Lance's post-filter remains reachable through Query::postfilter for comparison runs. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: intersect index bitmaps with the live manifest A bitmap records fragments the index covered when built; a fragment the manifest dropped afterwards stayed in the reported coverage. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: keep bitmap-less indices in coverage An index whose metadata has no fragment bitmap was dropped from the report entirely, indistinguishable from no index at all. Count its segments and mark its coverage unknown instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: count live rows in unindexed coverage physical_rows counts deleted rows and can be absent in the manifest; sum FileFragment::count_rows for the unindexed fragments instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: error on a null key in a selected key column * vector: compare filtered ANN and flat hits as sets * vector: coverage reports the checked-out version * vector: project _distance and _rowid explicitly --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chris@seaweedfs.com> |