4 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandClaude Opus 5.5	d0b5529a2d	S3 Vectors: weed-vector, weed-worker and the CLI write tables to the filers and volume servers (#12470) ... * sw-client: the filer and volume client library The client the Enterprise repository's VFS and vector engine use, with the same API: filer gRPC for entries, listings, volume locations, assignments and remote caching; volume HTTP, plain or TLS, for needle bytes; chunk resolution for sparse, overlapping and manifest chunks; a location cache; and the write side. Enterprise-only protocol fields (license, conditional mutations, hard links, KV listing) are left out. Compiled against weed/pb/filer.proto like the other worker crates. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-lance-worker: reads straight from the filers and volume servers direct_read::DirectReads wraps the object store Lance opens so that entries, listings and bytes come from the filers (gRPC) and the volume servers (HTTP, signed with the read JWT), whatever the entry holds: inline content, chunk manifests, sparse or overlapping chunks, the filer's volume encryption, the gateway's SSE-S3 (sse.rs, with the KEK from security.toml or the filer) and objects tiered to remote storage. Writes still go to the inner store. The code and its tests are those of the Enterprise repository's engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: read tables from the filers and volume servers The engine asks the masters for the filers (and follows the list), takes jwt.signing.read.key, grpc.client, https.client and the SSE-S3 keys from security.toml, and opens every table with DirectReads innermost and the read cache outside it. --masters is required: the gateway finds the engine through them and the engine finds the filers. Writes still go to the gateway. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sse: chunks out of a manifest keep their own SSE-S3 keys A chunk expanded from a manifest carries its id as fid, so keying the per-chunk key cache by file_id gave every such chunk the first one's key and IV. Keyed by the chunk's id however it is carried. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: a silent filer is passed over like an unreachable one Each filer attempt is bounded, so a filer that accepts the connection and never answers does not hold the read or the startup lookup. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: carry the assignment's fsync to the volume upload The filer's storage rule may want a needle on disk before the volume answers; the assignment says so and the upload now asks for it, as the Go uploader does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: argument tests pass the required masters; connect timeout for filers Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: a failed read is tried once more against fresh locations A fetch that fails forgets the volume locations it was made against; the request now looks them up again and retries instead of failing on a volume that moved. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: an exclusive create that finds its entry, and needle deletion create_entry marks ENTRY_ALREADY_EXISTS with EntryExists, so a caller can tell a lost conditional create from a failure; the transport deletes a needle whose entry was never created. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sse: encode chunks as the gateway stores them EntryEncoder writes SSE-S3 under a fresh object key wrapped by the KEK, serialized with the IV, key id and commitment the gateway's reader expects, each chunk under the IV at its offset; and the filer's volume encryption under a per-chunk key. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: writes, deletes and table removal without the gateway A put uploads its chunks to the volume servers the filer assigns, at the filer's chunk size, encoded as the bucket and cluster ask, and creates the entry the gateway would: md5, mime, size, the ETag attribute, the SSE-S3 attributes. A conditional create is the filer's exclusive create, so Lance's manifest commit conflicts the same way; the needles of a create that failed are discarded. A multipart upload turns parts into chunks and derives its ETag from them. Deletes remove the entry with its data, and remove_table a table directory. Copy and rename, which Lance does not use on S3, are not offered. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector.proto: Table says whether its bucket encrypts with SSE-S3 The engine writes the table's files itself, so it needs what the S3 write path knew: whether the bucket declares SSE-S3. storage_options go: the engine reaches no S3 store. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: send the engine the bucket's SSE-S3 setting The rule the S3 write path applies to a Lance bucket's files, factored out of DeclaredBucketEncryption, decides it from the vector bucket's record. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: every table operation goes to the filers and volume servers Opens, index creation and drops all go through the direct store, so the engine has no S3 client of its own left to protect: --storage-option, the endpoint pin and the request-option allowlist go, and lance's own S3 store is built unsigned and never dialed. The bucket's SSE-S3 setting comes with the table; -encryptVolumeData encrypts chunks for the volume servers as the gateway's flag does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors tests: the engine takes no credentials Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: the engine writes what the gateway would have Every leg checks that the entries the engine wrote carry the gateway's ETag attribute, the hardened leg that their chunks are encrypted for the volume servers, and a new test that an SSE-S3 bucket's files are written under keys the gateway unwraps when it serves them over S3. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * cluster plumbing moves to the worker crate for the engine, worker and CLI security.toml, the gRPC TLS connector and acceptor, the master dialer, the filer discovery and the storage connection built from them now live in weed-lance-worker (cluster, security, grpc_tls), where the worker and the CLI can use them too. The server crate re-exports security and tls under their old paths. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker, weed-vector-cli: tables through the filers and volume servers dataset::Storage replaces the S3 credentials every job and the CLI carried: a table is opened and written through the cluster's storage, encrypted as its bucket declares (read from the bucket's record on the filer), and dropped there. Both take --masters and security.toml like the engine; --access-key and --secret-key go, and with them the catalog's x-api-key credential, which the cluster token replaces. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: the cluster token alone authenticates a cluster process The engine, the maintenance worker and the CLI hold the cluster's filer signing key and reach a table's files themselves. A request presenting the cluster token and no other credential is theirs: it is authenticated by the token, runs through the trusted manager, and is vended no S3 credentials. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * worker chart, README and test design follow the worker's new arguments Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors tests: the worker and the CLI reach the cluster through the masters The cluster gets a filer signing key, which the catalog authenticates the worker and the CLI by; both take --masters in place of S3 keys. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: encrypted chunks each start their own CTR stream The gateway encrypts a multipart part from the IV derived for its offset, from position zero; the encoder encrypted from the exact byte offset of the object's stream instead, which agreed only at block boundaries. A part of 3,000 bytes read back wrong; now tested. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: keep what a lost reply saved, discard what a failed put left A create the filer saved but did not answer, or an exclusive create retried onto this very upload, keeps its chunks and counts as written; the needles of a put that failed midway are discarded. The report prints reads as reads. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: the security.toml it is given also holds its signing key Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * worker chart: the Lance worker mounts grpc.client's certificate Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: the filer deadline bounds a listing page, not the listing Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: security.toml booleans read as viper reads them Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a filer the master lists badly does not hide the others Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: register with the masters only once ready to serve Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sse: every chunk after the first is written under a key of its own Chunks of one object shared a key with IVs derived from their offsets, so two chunks meeting inside a cipher block shared keystream. The first chunk stays under the object's key and IV, which is how the gateway reads an object of one chunk; every other chunk gets a fresh key and IV, as the gateway's multipart parts have and as it reads an object of several. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * worker chart: the Lance worker is given the filer group Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: the gateway serves engine-written SSE-S3 files of one chunk and of several Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: the HTTP transport asks the volume to fsync when the assignment does write_synced fell back to the trait default, which refuses fsync, so every upload under a fsync storage rule failed before it was sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: https.client.ca alone vouches for volume servers With a CA configured, reqwest still trusted the system roots next to it; weed processes trust only the configured pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an empty volume range is read without a request A zero length underflowed the Range end, and a range past u64::MAX overflowed it; the first now returns no bytes, the second an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: a whole-needle answer to a ranged read is cut to the range A server or proxy that ignores Range answers 200 with the whole needle; the read returned its first bytes in place of the ones asked for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: a read size past u64::MAX reads to the end of the file offset + size overflowed before the clamp to the file size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an SSE-encrypted manifest is refused without a decoder Only filer-encrypted manifests were; an SSE one went to the protobuf decoder as ciphertext. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an empty write leaves the staged file size alone A zero-byte write past the end grew the size to its offset, as a zero-byte pwrite does not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: the chunk cache keeps no empty chunks They add no bytes, so nothing evicted them and the map grew with every distinct empty needle, even at capacity zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: flushing with a zero chunk size is an error, not a panic Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: an empty master list fails at startup "" split into one empty address that passed the check, and the engine then waited two minutes for a filer before a generic error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance cluster: an empty master list fails at startup Cluster::new took "" as one master, so weed-vector, weed-worker and the CLI waited two minutes for a filer before a generic error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: a failed create keeps its chunks when the filer cannot be asked The lookup after a failed create is what tells a lost reply from a refusal; when it fails too, the entry may be saved, so its chunks stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: a gateway-described table takes the shared unsigned store options Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector.proto: give sse_s3 a new field number and reserve storage_options Field 4 held the storage_options map on master; a gateway from that build would send a length-delimited value the engine cannot decode as a bool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector-cli: open s3:// URIs on the cluster's storage A table named by its s3:// location skipped the store wrapper and opened Lance's own S3 store, which has no credentials since the fallback flags were dropped. It now opens like a namespace table and needs --masters; local paths keep the plain builder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: an index id is plain path components, and drop trusts describe() `resolved()` now refuses an id whose part is empty, `.`, `..` or holds a `/`, since the colocated path would land elsewhere on the filer; and `drop_index` no longer repeats the colocation check for the location describe() already enforced for the configured schemes, which made a drop on a memory:// table fail in the scheme test configuration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a file delete takes only its entry; a table path is plain names The store's delete of one file no longer asks the filer for a recursive delete, so a path that names a directory does not take its content with it, and `remove_table` refuses a location whose bucket or path component is empty, `.`, `..` or holds a separator, which the filer would resolve outside the table. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: chunks are assigned to the bucket's collection The filer assigns a chunk's volume from the collection the caller names; the gateway names the bucket, prefixed by the filer group when the cluster has one, so a plain assign landed the engine's chunks in the default collection. The store now asks for `<bucket>` or `<group>_<bucket>`, with the group taken from the cluster connection. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a put's content type becomes the entry's mime Lance sets no content type today, so every entry was written as application/octet-stream; a put that does carry one now creates the entry with it, as the gateway does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: an upload whose create went unanswered is done with A create the filer took and never answered (a timeout, or a transport that reports the filer unavailable) may still be applied, so its chunks are kept, as they already were for a create whose reply was lost. The upload then records that it is done: a second completion is refused instead of creating an empty entry, and an abort discards nothing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: an upload dropped unfinished discards its chunks A request cancelled mid-write drops its multipart upload with neither complete nor abort called, leaving its needles on the volume servers. Dropping an upload that still holds chunks now discards them on the runtime; a completed upload, or one whose completion is in doubt, holds none. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: a cluster process registers and deregisters in the vector namespace The Lance catalog ran a cluster-token caller through the trusted manager for every operation but the vector-namespace write check of register and deregister, which still used the caller-checked manager and answered 403 for an index of a LANCE bucket. One `manager(r)` now picks the manager for all three. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: the worker's flags take two dashes The worker parses its flags with clap and, unlike the engine, does not accept Go-style single-dash long flags, so the harness's hardened worker leg and the README's `-encryptVolumeData` did not start it. The README's commands also gain the `--masters` the worker now requires. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: the CLI takes --encryptVolumeData An index build or optimize from the CLI wrote its chunks unencrypted for the volume servers unless the filer itself asked for the cipher; the CLI now takes the flag the engine and the worker have. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a bucket's encryption is read when a table is opened The setting was cached for a minute, so a bucket recreated with SSE-S3 within that minute had its first tables written plain, and a record that could not be decoded was read as "no encryption". A table open now reads the bucket's record from the filer, once per open, and an unreadable record is an error. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a removed table's files leave the object cache `remove_table` asked the cache to forget everything under the table, but the cache was built without invalidation closures, so the call was refused and its error dropped: a data file read before the drop was still served afterwards. The cache now supports the invalidation, and a refusal is an error. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: an abort keeps the chunks it could not discard `discard_all` stopped at the first chunk a volume server would not give up and dropped the rest from the upload's state, so a retried abort found nothing to do and the needles stayed. Every chunk is now attempted, the ones left are kept for the next abort or the drop, and the upload can no longer be completed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a chunk write that fails gives its needle back When the volume server took the chunk but its answer was lost, the needle stayed under a file id nothing would ever reference. The assignment is known before the write, so a failed write now discards it, best effort, before the error is returned. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: the live worker tests require the cluster's masters With WEED_LANCE_NAMESPACE set and WEED_MASTERS not, the tests seeded their tables through lance's own S3 store, which reaches nothing, and failed before a job ran. The missing variable is now named outright. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: only the filer's missing-entry error means not found Matching any "not found" text read failures such as an unresolved remote storage as a missing entry, so a delete or a listing reported success over data that was still there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage test: compare the chunks' SSE-S3 IVs Each chunk's metadata wraps its key under a fresh nonce, so the metadata differed even if the key and IV were reused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: a filer passed over unanswered keeps the outcome open When one filer timed out on a create and the next refused it, the refusal came back without the no-answer mark, so the put discarded chunks the first filer may still reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance test: start every concurrent writer before awaiting them The lazy map spawned each writer only after the previous one finished, so the commits never overlapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: count failed uploads and deletes The storage report counted failed reads and lookups but not a volume upload, an object delete or a table removal that failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * S3 Vectors tests: run weed-vector-cli as the cluster's processes run The CLI ran without the fixture's environment and outside its directory, so under VECTOR_READ_JWT it could not read the volumes and under VECTOR_TLS it found no security.toml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: a filer that fell away leaves the outcome open too An unavailable error (connection reset, deadline, transport failure) can arrive after the filer took the request, so a refusal from the next filer no longer settles a create the first one may still apply: the chunks are kept, as after a timeout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: a chunk write abandoned in flight gives its needle back The needle is assigned before the write, so a request cancelled while the write is in flight left it on the volume server with nothing to name it. `Source::upload` is now `assign` and `write`, and the upload holds the assignment across the write in a guard: a write that fails discards it and one abandoned discards it on the runtime, while a write that returns hands it to the entry. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance tests: sign for the namespace as the worker does The live tests reached the namespace anonymously once the S3 keys went, and an anonymous caller lists no tables, so detection proposed nothing. They now take the cluster key from where the worker reads it (WEED_JWT_FILER_SIGNING_KEY or security.toml) and log at info, so a table a handler skips says why in the harness output. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3tables lifecycle test: run the Lance worker against the cluster The worker leg of TestLanceTableLifecycle still handed the Rust tests S3 credentials and no masters. The mini cluster now has a cluster key, and the tests get WEED_MASTERS and that key; passes with WEED_LANCE_MAINTENANCE=worker (3 fragments became 1, 4 versions removed). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: name no collection outside a filer group, as the gateway Without a filer group the gateway leaves the collection to the filer, so a filer.conf rule for the bucket's path picks it; naming the bucket overrode that rule, and the chunks landed where a bucket delete does not look. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: a read reports the entry's content type A put now stores its ContentType as the entry's mime, but get_opts returned no attributes, so the type did not come back on a read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-11-07 10:51:52 -08:00
Chris LuandClaude Fable 5.1	2fe480a6e2	S3 Vectors: the gateway resolves the table it sends to weed-vector (#12468) ... * lance catalog: in-process DescribeTable for trusted callers DescribeTable and DescribeTableInternal resolve a table's location and storage options without the HTTP layer; the internal form runs the lookup through a trusted manager so the caller's catalog grants do not gate a resolution its own permission already authorized. Same code as the Enterprise repository. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector.proto: Table carries its location and storage options The gateway resolves the table it addresses, so the engine can open it without a describe round trip of its own. Both fields are optional; an empty location means the engine looks the table up as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: test the in-process describe Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: resolve the table before calling the engine The gateway describes the table in process, through the trusted catalog manager, and sends its location and storage options with every engine call. Storage options are vended for the caller's principal, as the Enterprise router does for queries. A catalog failure answers with the code the engine would have given. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: open a table at the location the gateway sent A request whose table carries a location is opened there, with the storage options the gateway vended over the engine's fallback; the catalog is described only for a table that comes without one, as from an older gateway. dataset::open_at and dataset::drop_at take a description the caller already holds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: run the engine without a catalog VECTOR_ENGINE_NO_CATALOG=1 gives the engine an unreachable catalog URL; every CI leg sets it, so the suite proves the gateway resolves each table it sends. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: an internal describe may write a vector index With credential vending on, the options vended to the engine followed the caller's s3tables grants, which never allow writing the vector namespace, so every write the gateway sent would have run with read-only credentials and reads would have lost their version lease. The engine is that namespace's writer and the caller's vector permission authorized the operation, so an internal describe of a vector table vends write-capable options; other namespaces keep the UpdateTable check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: resolve tables for the engine only while the catalog runs With -port.lance=0 the gateway has no catalog to resolve through; the engine then keeps looking tables up itself, as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: a deregistered table describes as NoSuchTable So the gateway answers NotFoundException for it, not InternalFailure. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: trust a sent location only for a colocated vector index A caller that reaches the engine could otherwise name any storage the engine's credentials cover. A location is taken from the request only when the id names a vector index and the location is its colocated path, which the engine writes already; anything else is described through the catalog as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: scope the read cache by storage endpoint The cache was keyed by bucket and path alone, so the same path reached through another endpoint would have shared entries. Each open wraps the store with the cache scoped to the endpoint its options name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep a configured endpoint over one a request names A caller that reaches the engine could otherwise send its fallback credentials to an endpoint of the caller's choosing. When the engine's --storage-option names an aws_endpoint, a table whose request names another one is described through the catalog instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: use request options only where credentials cannot leave A request's storage options are taken only when they hold the keys a gateway's describe produces, so a proxy or an endpoint alias cannot route the engine's credentials elsewhere, and an endpoint only when it is the configured one or the engine holds no credentials to send. Anything else sends the table to the catalog. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: give the engine its endpoint with its credentials Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a drop checks the request's options like an open Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: an unpinned endpoint needs the request to sign for itself Without configured credentials the S3 store still signs with the AWS default chain, so an endpoint from a request is taken only when the request skips signing or brings its own credentials. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a trailing slash does not unpin the endpoint, and say so A configured aws_endpoint written with a trailing slash matched nothing the gateway sends, so every table fell back to the catalog in silence. The comparison ignores the slash, and a fallback for a resolved table is logged at debug. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: resolved by the gateway, or refused The engine no longer has a catalog: --namespace and --filer-signing-key are gone, a request whose table names no location is InvalidArgument, and one whose location is not the index's own or whose storage options the engine's credentials must not be used with is PermissionDenied. A drop removes the colocated path it was given. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: the catalog resolves vector tables whether or not its port is served newLanceServer builds the Lance catalog for the S3 Vectors API; the listener on -port.lance is optional. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors tests: start the engine without a catalog Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: the aux listener test builds the catalog before serving it Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: an unpinned endpoint takes only credentials the request signs with alone Lance fills an option a request leaves out from the engine's environment, under any of object_store's names for it (TOKEN, SESSION_TOKEN, ...), and asks the AWS default chain when the key pair is incomplete. A request that named its own endpoint with only an access key id, or a key pair while the engine's environment held a session token, would have sent the engine's credentials there. Such a request now needs a full key pair, and its own session token whenever the environment would supply one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: accept --namespace and --filer-signing-key, ignored Both flags shipped in 4.52 and are documented, so dropping them made an existing command line exit at startup with an unexpected-argument error. They are kept hidden and ignored, with a warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors design: the engine is started without a catalog Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: the trust check takes the environment it reads The options test read the process environment through signs_alone, so a host exporting AWS_SESSION_TOKEN or TOKEN failed it. The caller now passes the variable names in; describe passes the engine's own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: log both endpoints when a request's options are refused The gateway turns the PermissionDenied into a bare AccessDenied and logs nothing, so a pinned aws_endpoint written in another form than the one the gateway sends (host case, a default port, an IP for a name) failed every call with no trace of which value differed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: the Lance catalog sets the credential role whether or not Iceberg runs -iceberg.credentialRole reached the S3 server only while the Iceberg listener started, so with -port.iceberg=0 the Lance catalog installed its vendor but vending stayed off, and the vector engine got no credentials for the tables the gateway resolves. Both catalogs now set the role through one helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: set the credential role once, before either catalog serves Both catalogs wrote the role as they were built; the second write landed while the first was already serving requests that read it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-11-06 19:39:09 -08:00
Chris LuandClaude Opus 5.5	4be191173a	vector: keep ListVectors pages and segments flat as an index grows; read only keys in GetVectors (#12300) ... * vector: scan only the fragments a ListVectors page covers A page pushed its offset into the scan, but the scan still loaded every fragment first, reading each one's deletion file: about 105 S3 GETs per page at 50k rows against 25 at 10k. The page's fragments now come from the live row counts in the manifest, so a page reads only its own. Tokens and row order are unchanged; a manifest without the counts keeps the old scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: list a ListVectors segment in the engine The gateway paged through the whole index for every segment and kept one row in segmentCount, so a segment cost segmentCount engine pages and 16 segments listed slower than one. The engine now takes segment_count and segment_index: it reads keys across the stride, keeps the segment's rows and takes their data and metadata alone, one call per page. A segment is still the rows whose position modulo segmentCount is segmentIndex, and its token still the position to resume from. An engine that predates the fields does not set segmented, and the gateway then pages through it as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: read only the key column across the table in GetVectors The key lookup scanned every column. On S3, lance reads a column during the scan when a row of it is under 1000 bytes, so a 128-dimension vector was read for every row of the table to return 100 of them, and data and metadata were read even when the request did not ask for them. The scan now projects the requested columns and materializes everything but the key late, for the matching rows only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: start the legacy segment fallback from the page already read With an engine that ignores segment_count, the gateway dropped the page it had just read and fetched it again. The fallback now consumes that page first, so a one-row segment page still costs one engine call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: check metadata on segmented ListVectors pages The segment test listed data only. Its rows now carry metadata on two in three keys, and every segmented row, read through take_rows, must equal the same row in a plain listing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: lease the version while a segment takes its data A segmented page with data or metadata reads keys first and then takes the other columns from the same version, so maintenance could reclaim its files in between. That path now holds a lease like GetVectors. A keys-only page is one scan and stays unleased, as plain pages are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: leave GetVectors materialization to lance Forcing every column late made a 100-key GetVectors with data slower at 10k x 128 in the perf cluster: 54.7 ms against 46.3 ms. The take issued a read per row, where lance's heuristic reads a narrow vector column in one pass per fragment. The projection keeps the gain on its own: with neither data nor metadata requested, the call went from 38.6 ms to 25.3 ms and from 150 to 89 S3 GETs. At 50k x 384 the vector is already late under the heuristic, and a full lookup is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-01 11:57:44 -08:00
Chris LuandDevin	d771a91571	vector: VectorService gRPC contract (P1) (#12101) ... * vector: VectorService protobuf contract The shared RPC boundary between the Go S3 gateway and the Rust vector engine from the vector design doc: CreateIndex/DropIndex, Put/Get/ List/Delete/Query vectors. A Table reference carries the Lance table id and an optional pinned version, matching the snapshot discipline queries are required to keep. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: generated Go stubs for vector.proto protoc output via weed/pb/Makefile. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: carry return_data through QueryVectors S3 Vectors QueryVectors honours returnData, but the contract had no way to ask for or carry vector data in a query response, so the gateway would need a second GetVectors round-trip per query. Add return_data to the request and data to QueryResult now, while the contract is still new. * vector: pin metadata numbers to double precision in the contract Metadata arrives as JSON, where numbers carry double precision; say so on the field so neither side promises integer fidelity past 2^53. * vector: clarify the query filter is raw Lance SQL over one metadata column The doc said 'metadata columns' but the table stores all metadata in a single JSON column named metadata; the gateway translates S3 Vectors filter expressions onto it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-26 13:26:11 -07:00