mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-07 20:47:46 +01:00
985706cc32b27b04e7cce1e6fcd76d6944c1bc4e
16090
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
985706cc32 |
volume: check a needle's CRC the first time a range of it is read (#12467)
* volume: check a needle's CRC the first time a range of it is read A ranged read never checked the CRC, so a replica with a damaged needle served the damaged bytes with 206 and the reader never tried another copy. The first ranged read of a needle now streams the whole needle through the existing CRC check, with the usual pooled page buffers, before writing any byte. A mismatch fails the read, so the reader falls back to another replica. Needles that pass go into a bounded LRU, keyed by volume, compaction revision, id, append time and checksum, so later ranges of the same needle stream as before. Fixes #5959 for ranged reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: key verified ranges by volume load, share concurrent checks From review: the cache key read v.SuperBlock.CompactionRevision without the data file lock, racing a compaction commit that reloads the super block, and a remounted copy of a volume inherited the old copy's verified entries. Key entries by a load generation instead, an atomic counter that changes on every load, including the reload in CommitCompact, and cache a result only if no reload happened during the check. Concurrent first ranges of a needle now share one check through singleflight instead of each reading the whole needle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: keep hot needles verified, leave the io error streak to the range read From review: Contains does not refresh an entry's place in the LRU, so even a constantly read needle was evicted after 64k other needles were verified; look it up with Get instead. The verification pass also cleared the io error streak before the range read ran, so a range read failing with EIO after a passing check never built up a streak; the pass into io.Discard now leaves the streak to the range read, like the meta read before a streamed GET. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume (Rust): check a needle's CRC the first time a range of it is read Mirror of the Go change: a range served from the data file now checks the whole needle's CRC before any byte of it is sent, so a damaged replica answers 500 and the reader tries another copy. Needles that pass go into a bounded LRU keyed by the volume's load generation, needle id, append time and checksum; concurrent first ranges of a needle share one check, and a check that passes leaves the io error streak to the range read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume (Rust): a range over the whole needle checks its CRC Go reads a range that covers the whole needle through the checked full-read path. The Rust volume server read it from the data file unchecked, so "bytes=0-" served a damaged needle with 206. Check the slice's CRC before answering, as the full read does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: check the covered blocks of every range against CRCs recorded at the first check The first ranged read of a needle now records the CRC of each block (64 KiB to 1 MiB, at most 64 blocks below 64 MiB) while it checks the whole needle. Later ranges read the block-aligned span around them and write a block's bytes only once it matches, so damage that appears after the first check is not served. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume (Rust): check the covered blocks of every range against CRCs recorded at the first check verify_checksum now records a CRC per block (same sizes as Go's rangeCheckBlockSize) and returns them from the cache; read_checked_range reads the block-aligned span of a range and answers 500 if a block no longer matches, so damage after the first check is not served. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandClaude Opus 5.5
|
d0b5529a2d |
S3 Vectors: weed-vector, weed-worker and the CLI write tables to the filers and volume servers (#12470)
* sw-client: the filer and volume client library The client the Enterprise repository's VFS and vector engine use, with the same API: filer gRPC for entries, listings, volume locations, assignments and remote caching; volume HTTP, plain or TLS, for needle bytes; chunk resolution for sparse, overlapping and manifest chunks; a location cache; and the write side. Enterprise-only protocol fields (license, conditional mutations, hard links, KV listing) are left out. Compiled against weed/pb/filer.proto like the other worker crates. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-lance-worker: reads straight from the filers and volume servers direct_read::DirectReads wraps the object store Lance opens so that entries, listings and bytes come from the filers (gRPC) and the volume servers (HTTP, signed with the read JWT), whatever the entry holds: inline content, chunk manifests, sparse or overlapping chunks, the filer's volume encryption, the gateway's SSE-S3 (sse.rs, with the KEK from security.toml or the filer) and objects tiered to remote storage. Writes still go to the inner store. The code and its tests are those of the Enterprise repository's engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: read tables from the filers and volume servers The engine asks the masters for the filers (and follows the list), takes jwt.signing.read.key, grpc.client, https.client and the SSE-S3 keys from security.toml, and opens every table with DirectReads innermost and the read cache outside it. --masters is required: the gateway finds the engine through them and the engine finds the filers. Writes still go to the gateway. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sse: chunks out of a manifest keep their own SSE-S3 keys A chunk expanded from a manifest carries its id as fid, so keying the per-chunk key cache by file_id gave every such chunk the first one's key and IV. Keyed by the chunk's id however it is carried. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: a silent filer is passed over like an unreachable one Each filer attempt is bounded, so a filer that accepts the connection and never answers does not hold the read or the startup lookup. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: carry the assignment's fsync to the volume upload The filer's storage rule may want a needle on disk before the volume answers; the assignment says so and the upload now asks for it, as the Go uploader does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: argument tests pass the required masters; connect timeout for filers Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: a failed read is tried once more against fresh locations A fetch that fails forgets the volume locations it was made against; the request now looks them up again and retries instead of failing on a volume that moved. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: an exclusive create that finds its entry, and needle deletion create_entry marks ENTRY_ALREADY_EXISTS with EntryExists, so a caller can tell a lost conditional create from a failure; the transport deletes a needle whose entry was never created. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sse: encode chunks as the gateway stores them EntryEncoder writes SSE-S3 under a fresh object key wrapped by the KEK, serialized with the IV, key id and commitment the gateway's reader expects, each chunk under the IV at its offset; and the filer's volume encryption under a per-chunk key. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: writes, deletes and table removal without the gateway A put uploads its chunks to the volume servers the filer assigns, at the filer's chunk size, encoded as the bucket and cluster ask, and creates the entry the gateway would: md5, mime, size, the ETag attribute, the SSE-S3 attributes. A conditional create is the filer's exclusive create, so Lance's manifest commit conflicts the same way; the needles of a create that failed are discarded. A multipart upload turns parts into chunks and derives its ETag from them. Deletes remove the entry with its data, and remove_table a table directory. Copy and rename, which Lance does not use on S3, are not offered. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector.proto: Table says whether its bucket encrypts with SSE-S3 The engine writes the table's files itself, so it needs what the S3 write path knew: whether the bucket declares SSE-S3. storage_options go: the engine reaches no S3 store. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: send the engine the bucket's SSE-S3 setting The rule the S3 write path applies to a Lance bucket's files, factored out of DeclaredBucketEncryption, decides it from the vector bucket's record. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: every table operation goes to the filers and volume servers Opens, index creation and drops all go through the direct store, so the engine has no S3 client of its own left to protect: --storage-option, the endpoint pin and the request-option allowlist go, and lance's own S3 store is built unsigned and never dialed. The bucket's SSE-S3 setting comes with the table; -encryptVolumeData encrypts chunks for the volume servers as the gateway's flag does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors tests: the engine takes no credentials Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: the engine writes what the gateway would have Every leg checks that the entries the engine wrote carry the gateway's ETag attribute, the hardened leg that their chunks are encrypted for the volume servers, and a new test that an SSE-S3 bucket's files are written under keys the gateway unwraps when it serves them over S3. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * cluster plumbing moves to the worker crate for the engine, worker and CLI security.toml, the gRPC TLS connector and acceptor, the master dialer, the filer discovery and the storage connection built from them now live in weed-lance-worker (cluster, security, grpc_tls), where the worker and the CLI can use them too. The server crate re-exports security and tls under their old paths. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker, weed-vector-cli: tables through the filers and volume servers dataset::Storage replaces the S3 credentials every job and the CLI carried: a table is opened and written through the cluster's storage, encrypted as its bucket declares (read from the bucket's record on the filer), and dropped there. Both take --masters and security.toml like the engine; --access-key and --secret-key go, and with them the catalog's x-api-key credential, which the cluster token replaces. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: the cluster token alone authenticates a cluster process The engine, the maintenance worker and the CLI hold the cluster's filer signing key and reach a table's files themselves. A request presenting the cluster token and no other credential is theirs: it is authenticated by the token, runs through the trusted manager, and is vended no S3 credentials. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * worker chart, README and test design follow the worker's new arguments Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors tests: the worker and the CLI reach the cluster through the masters The cluster gets a filer signing key, which the catalog authenticates the worker and the CLI by; both take --masters in place of S3 keys. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: encrypted chunks each start their own CTR stream The gateway encrypts a multipart part from the IV derived for its offset, from position zero; the encoder encrypted from the exact byte offset of the object's stream instead, which agreed only at block boundaries. A part of 3,000 bytes read back wrong; now tested. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: keep what a lost reply saved, discard what a failed put left A create the filer saved but did not answer, or an exclusive create retried onto this very upload, keeps its chunks and counts as written; the needles of a put that failed midway are discarded. The report prints reads as reads. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: the security.toml it is given also holds its signing key Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * worker chart: the Lance worker mounts grpc.client's certificate Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: the filer deadline bounds a listing page, not the listing Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: security.toml booleans read as viper reads them Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a filer the master lists badly does not hide the others Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: register with the masters only once ready to serve Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sse: every chunk after the first is written under a key of its own Chunks of one object shared a key with IVs derived from their offsets, so two chunks meeting inside a cipher block shared keystream. The first chunk stays under the object's key and IV, which is how the gateway reads an object of one chunk; every other chunk gets a fresh key and IV, as the gateway's multipart parts have and as it reads an object of several. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * worker chart: the Lance worker is given the filer group Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: the gateway serves engine-written SSE-S3 files of one chunk and of several Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: the HTTP transport asks the volume to fsync when the assignment does write_synced fell back to the trait default, which refuses fsync, so every upload under a fsync storage rule failed before it was sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: https.client.ca alone vouches for volume servers With a CA configured, reqwest still trusted the system roots next to it; weed processes trust only the configured pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an empty volume range is read without a request A zero length underflowed the Range end, and a range past u64::MAX overflowed it; the first now returns no bytes, the second an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: a whole-needle answer to a ranged read is cut to the range A server or proxy that ignores Range answers 200 with the whole needle; the read returned its first bytes in place of the ones asked for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: a read size past u64::MAX reads to the end of the file offset + size overflowed before the clamp to the file size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an SSE-encrypted manifest is refused without a decoder Only filer-encrypted manifests were; an SSE one went to the protobuf decoder as ciphertext. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an empty write leaves the staged file size alone A zero-byte write past the end grew the size to its offset, as a zero-byte pwrite does not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: the chunk cache keeps no empty chunks They add no bytes, so nothing evicted them and the map grew with every distinct empty needle, even at capacity zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: flushing with a zero chunk size is an error, not a panic Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: an empty master list fails at startup "" split into one empty address that passed the check, and the engine then waited two minutes for a filer before a generic error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance cluster: an empty master list fails at startup Cluster::new took "" as one master, so weed-vector, weed-worker and the CLI waited two minutes for a filer before a generic error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: a failed create keeps its chunks when the filer cannot be asked The lookup after a failed create is what tells a lost reply from a refusal; when it fails too, the entry may be saved, so its chunks stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: a gateway-described table takes the shared unsigned store options Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector.proto: give sse_s3 a new field number and reserve storage_options Field 4 held the storage_options map on master; a gateway from that build would send a length-delimited value the engine cannot decode as a bool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector-cli: open s3:// URIs on the cluster's storage A table named by its s3:// location skipped the store wrapper and opened Lance's own S3 store, which has no credentials since the fallback flags were dropped. It now opens like a namespace table and needs --masters; local paths keep the plain builder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: an index id is plain path components, and drop trusts describe() `resolved()` now refuses an id whose part is empty, `.`, `..` or holds a `/`, since the colocated path would land elsewhere on the filer; and `drop_index` no longer repeats the colocation check for the location describe() already enforced for the configured schemes, which made a drop on a memory:// table fail in the scheme test configuration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a file delete takes only its entry; a table path is plain names The store's delete of one file no longer asks the filer for a recursive delete, so a path that names a directory does not take its content with it, and `remove_table` refuses a location whose bucket or path component is empty, `.`, `..` or holds a separator, which the filer would resolve outside the table. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: chunks are assigned to the bucket's collection The filer assigns a chunk's volume from the collection the caller names; the gateway names the bucket, prefixed by the filer group when the cluster has one, so a plain assign landed the engine's chunks in the default collection. The store now asks for `<bucket>` or `<group>_<bucket>`, with the group taken from the cluster connection. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a put's content type becomes the entry's mime Lance sets no content type today, so every entry was written as application/octet-stream; a put that does carry one now creates the entry with it, as the gateway does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: an upload whose create went unanswered is done with A create the filer took and never answered (a timeout, or a transport that reports the filer unavailable) may still be applied, so its chunks are kept, as they already were for a create whose reply was lost. The upload then records that it is done: a second completion is refused instead of creating an empty entry, and an abort discards nothing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: an upload dropped unfinished discards its chunks A request cancelled mid-write drops its multipart upload with neither complete nor abort called, leaving its needles on the volume servers. Dropping an upload that still holds chunks now discards them on the runtime; a completed upload, or one whose completion is in doubt, holds none. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: a cluster process registers and deregisters in the vector namespace The Lance catalog ran a cluster-token caller through the trusted manager for every operation but the vector-namespace write check of register and deregister, which still used the caller-checked manager and answered 403 for an index of a LANCE bucket. One `manager(r)` now picks the manager for all three. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: the worker's flags take two dashes The worker parses its flags with clap and, unlike the engine, does not accept Go-style single-dash long flags, so the harness's hardened worker leg and the README's `-encryptVolumeData` did not start it. The README's commands also gain the `--masters` the worker now requires. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: the CLI takes --encryptVolumeData An index build or optimize from the CLI wrote its chunks unencrypted for the volume servers unless the filer itself asked for the cipher; the CLI now takes the flag the engine and the worker have. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a bucket's encryption is read when a table is opened The setting was cached for a minute, so a bucket recreated with SSE-S3 within that minute had its first tables written plain, and a record that could not be decoded was read as "no encryption". A table open now reads the bucket's record from the filer, once per open, and an unreadable record is an error. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a removed table's files leave the object cache `remove_table` asked the cache to forget everything under the table, but the cache was built without invalidation closures, so the call was refused and its error dropped: a data file read before the drop was still served afterwards. The cache now supports the invalidation, and a refusal is an error. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: an abort keeps the chunks it could not discard `discard_all` stopped at the first chunk a volume server would not give up and dropped the rest from the upload's state, so a retried abort found nothing to do and the needles stayed. Every chunk is now attempted, the ones left are kept for the next abort or the drop, and the upload can no longer be completed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a chunk write that fails gives its needle back When the volume server took the chunk but its answer was lost, the needle stayed under a file id nothing would ever reference. The assignment is known before the write, so a failed write now discards it, best effort, before the error is returned. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: the live worker tests require the cluster's masters With WEED_LANCE_NAMESPACE set and WEED_MASTERS not, the tests seeded their tables through lance's own S3 store, which reaches nothing, and failed before a job ran. The missing variable is now named outright. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: only the filer's missing-entry error means not found Matching any "not found" text read failures such as an unresolved remote storage as a missing entry, so a delete or a listing reported success over data that was still there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage test: compare the chunks' SSE-S3 IVs Each chunk's metadata wraps its key under a fresh nonce, so the metadata differed even if the key and IV were reused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: a filer passed over unanswered keeps the outcome open When one filer timed out on a create and the next refused it, the refusal came back without the no-answer mark, so the put discarded chunks the first filer may still reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance test: start every concurrent writer before awaiting them The lazy map spawned each writer only after the previous one finished, so the commits never overlapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: count failed uploads and deletes The storage report counted failed reads and lookups but not a volume upload, an object delete or a table removal that failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * S3 Vectors tests: run weed-vector-cli as the cluster's processes run The CLI ran without the fixture's environment and outside its directory, so under VECTOR_READ_JWT it could not read the volumes and under VECTOR_TLS it found no security.toml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: a filer that fell away leaves the outcome open too An unavailable error (connection reset, deadline, transport failure) can arrive after the filer took the request, so a refusal from the next filer no longer settles a create the first one may still apply: the chunks are kept, as after a timeout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: a chunk write abandoned in flight gives its needle back The needle is assigned before the write, so a request cancelled while the write is in flight left it on the volume server with nothing to name it. `Source::upload` is now `assign` and `write`, and the upload holds the assignment across the write in a guard: a write that fails discards it and one abandoned discards it on the runtime, while a write that returns hands it to the entry. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance tests: sign for the namespace as the worker does The live tests reached the namespace anonymously once the S3 keys went, and an anonymous caller lists no tables, so detection proposed nothing. They now take the cluster key from where the worker reads it (WEED_JWT_FILER_SIGNING_KEY or security.toml) and log at info, so a table a handler skips says why in the harness output. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3tables lifecycle test: run the Lance worker against the cluster The worker leg of TestLanceTableLifecycle still handed the Rust tests S3 credentials and no masters. The mini cluster now has a cluster key, and the tests get WEED_MASTERS and that key; passes with WEED_LANCE_MAINTENANCE=worker (3 fragments became 1, 4 versions removed). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: name no collection outside a filer group, as the gateway Without a filer group the gateway leaves the collection to the filer, so a filer.conf rule for the bucket's path picks it; naming the bucket overrode that rule, and the chunks landed where a bucket delete does not look. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: a read reports the entry's content type A put now stores its ContentType as the entry's mime, but get_opts returned no attributes, so the type did not come back on a read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
5cd38527ba |
filer.remote.sync: skip pre-mount mount-table versions, decide the unmount before admitting later events (#12472)
* filer.remote.sync: judge the mount table before admitting later events The unmount was decided inside a processor job, so events behind the mapping write, such as remote.unmount's purge deletes, were admitted and could delete remote objects before the exit landed. The stream callback now reads each mount-table version before the next event, admits nothing after the one that ends the mount, saves no offset, and drains the admitted jobs before exiting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: ignore mount-table versions from before the mount Without a saved offset the follow starts at the mounted directory's creation second, so it replays mount-table versions written before remote.mount added the directory. Each read as an unmount and the process exited on every start. A version now ends the write-back only when the version it replaced mapped the directory to its location. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: judge the mount table by the mount's own key -dir may lie below a mount, and DetectMountInfo resolves it to the enclosing mount. Looking the table up by -dir found no mapping in any version, so a replaced table never counted as mapped and the parent's unmount was missed, admitting its purge deletes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: a table renamed away ends the mount A rename of /etc/remote/mount.mapping to another name inside /etc/remote was ignored, although the filer no longer has a mount table. The table is now matched by its exact path on both sides of the event, so renaming it away counts as removing it, and a file of that name elsewhere under /etc/remote no longer counts as the table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: a mount on another remote is not the same mount The mount-table check compared only bucket and path, so remounting the directory on another remote storage at the same bucket and path kept writing through the old client, and a replayed unmount from the other remote stopped the write-back. The remote storage name now counts too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: check the drain at the exit itself Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: resolve the mount covering -dir in each table version Judging the table by the startup mount key alone missed a mount added at -dir itself below that key: the key stayed mapped, so the nested mount and its later removal were both admitted, and the removal's purge deletes ran against the startup remote. Each version is now resolved the way DetectMountInfo resolves the live table, and a different covering mount ends the follow as a move. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: an earlier mount's unmount still ends the follow Skipping every version whose replaced table did not map the directory to this mount also skipped the unmount of an earlier mount of the directory, which a -timeAgo replay reaches, and admitted its purge deletes against this mount's remote. Only a version in which no mount covers the directory, replacing one in which none did, is now skipped as history from before the mount. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: map paths below -dir under its place in the mount With -dir below a mount, every remote key was taken relative to -dir and appended to the mount's location, so /mount/sub/x.txt uploaded to x.txt instead of sub/x.txt and could overwrite another object. The processor now gets the remote location of -dir itself; the mount's own location still judges mount-table changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: resume below a mount no earlier than the mount's offset remote.unmount saves the offset of the directory it unmounts, after its purge, and leaves the offsets of directories below it alone. A write-back of -dir=/mount/sub resumed from its own offset after a remount, replayed the unmount and exited, on every start. Without -timeAgo it now starts no earlier than the mount's offset. Whatever that skips was purged, or was synced by a write-back of the whole mount. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
5b1450b06b |
s3tables, iceberg: one catalog record per table location (#12473)
* s3tables: refuse a table or view location another record holds A create, register, update or rename could name a location at, under or above another catalog record's files, including the files a renamed table left under its old name. The new record then loaded that table's metadata, had credentials vended for its files, and could purge them on drop. Each of those writes now scans the bucket's records and refuses a root another record claims, and a drop of a record sharing its files is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: check a table's location before writing to it REST create and stage-create write a metadata file, and stage-create vends write credentials, before the catalog registers the table, so the create gate now also validates the location and refuses one another record holds. RegisterTable runs the same gate before it reads the metadata file, instead of reading first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: keep a dropped location another record holds DropTable and DropView purge the dropped record's location after the catalog delete. A view recorded at a table's location took that table's directory, record included. The purge now leaves a location another record still holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: authorize CreateTable before reporting the namespace A caller without CreateTable got 404 for a missing namespace, 403 for an existing one and 500 with decoder text for an unreadable one. The grant is now evaluated against the bucket first, and only a caller that passes learns the namespace is missing or unreadable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg maintenance: skip a table whose files another record shares Orphan removal and snapshot expiry delete what a table's metadata does not reference. For two records over one location that deletes the other's files, so detection leaves such tables alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: count another record's catalog entry as claimed A table keeping its files elsewhere still owns its entry directory, and a view recorded there would take that entry with it on drop. The ownership check now refuses a root at, above or under another record's entry too. Maintenance still compares files only, so a table created over a renamed table's old name does not stop that table being maintained. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: check a claimed location again once the record is written Two creates, or a create and an update moving a root, could both read the same root free and both land. A create now checks again after writing its still-marked record and backs out if another record claims the root, and an update that moved its root checks again after the write and puts the previous metadata back. Of two racing writers at least one sees the other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: enable stage-create in the location ownership test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg maintenance: check for a shared location again before removing orphans A job can run long after its table was scheduled, or come from a detector that never checked, so remove_orphans reads the bucket's records itself and leaves the files alone when another record shares the table's location. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: check a renamed record's new root again once it is written A record without a location moves its root with its name, and was checked only before the rename wrote the destination, so a create landing on that root in between could share it. The rename now checks again with the destination written and backs out, like a create. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: put a moved record back when its location cannot be checked again An update that moved its root kept the move when the check after the write failed to list the bucket, so a racing writer that relied on that check could share the root. The move is now undone as it is when the root is taken. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: hold a moved record's old root until the move lands An update moving a record's root wrote the new metadata first and put the old metadata back when the check that followed found the new root taken. The old root was free in between, so a create racing for it could pass its own check and then share it with the restored record. The update now stores the metadata it is about to write beside the current one, so the record holds both roots, checks the new root with that reservation in place, and only then switches the metadata. A move that backs out drops the reservation without touching the metadata, and an update that keeps its root clears a reservation a crashed move left behind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: test a move racing another update of the record Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: drop a move's reservation when a rename or delete takes the record A rename or a decoupled delete that took a record while an update held its move reservation left the reservation on the old name, where the move could no longer clear it, and later creates of that name were refused as taken. Both now clear it with the record's other attributes, and a failed rename does not put it back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: register a table from metadata under its own location The Spark register test registered a live table's metadata file under a second name, which now fails because both records would own the source's files. It checks that refusal instead, and registers a copy uploaded under the new table's own directory, with its location pointing there, to cover reading and writing a registered table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b650303aee |
S3 Vectors: weed-vector reads tables from the filers and volume servers (#12469)
* sw-client: the filer and volume client library The client the Enterprise repository's VFS and vector engine use, with the same API: filer gRPC for entries, listings, volume locations, assignments and remote caching; volume HTTP, plain or TLS, for needle bytes; chunk resolution for sparse, overlapping and manifest chunks; a location cache; and the write side. Enterprise-only protocol fields (license, conditional mutations, hard links, KV listing) are left out. Compiled against weed/pb/filer.proto like the other worker crates. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-lance-worker: reads straight from the filers and volume servers direct_read::DirectReads wraps the object store Lance opens so that entries, listings and bytes come from the filers (gRPC) and the volume servers (HTTP, signed with the read JWT), whatever the entry holds: inline content, chunk manifests, sparse or overlapping chunks, the filer's volume encryption, the gateway's SSE-S3 (sse.rs, with the KEK from security.toml or the filer) and objects tiered to remote storage. Writes still go to the inner store. The code and its tests are those of the Enterprise repository's engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: read tables from the filers and volume servers The engine asks the masters for the filers (and follows the list), takes jwt.signing.read.key, grpc.client, https.client and the SSE-S3 keys from security.toml, and opens every table with DirectReads innermost and the read cache outside it. --masters is required: the gateway finds the engine through them and the engine finds the filers. Writes still go to the gateway. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sse: chunks out of a manifest keep their own SSE-S3 keys A chunk expanded from a manifest carries its id as fid, so keying the per-chunk key cache by file_id gave every such chunk the first one's key and IV. Keyed by the chunk's id however it is carried. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: a silent filer is passed over like an unreachable one Each filer attempt is bounded, so a filer that accepts the connection and never answers does not hold the read or the startup lookup. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: carry the assignment's fsync to the volume upload The filer's storage rule may want a needle on disk before the volume answers; the assignment says so and the upload now asks for it, as the Go uploader does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: argument tests pass the required masters; connect timeout for filers Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: a failed read is tried once more against fresh locations A fetch that fails forgets the volume locations it was made against; the request now looks them up again and retries instead of failing on a volume that moved. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: the filer deadline bounds a listing page, not the listing Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: security.toml booleans read as viper reads them Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a filer the master lists badly does not hide the others Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: register with the masters only once ready to serve Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: the HTTP transport asks the volume to fsync when the assignment does write_synced fell back to the trait default, which refuses fsync, so every upload under a fsync storage rule failed before it was sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: https.client.ca alone vouches for volume servers With a CA configured, reqwest still trusted the system roots next to it; weed processes trust only the configured pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an empty volume range is read without a request A zero length underflowed the Range end, and a range past u64::MAX overflowed it; the first now returns no bytes, the second an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: a whole-needle answer to a ranged read is cut to the range A server or proxy that ignores Range answers 200 with the whole needle; the read returned its first bytes in place of the ones asked for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: a read size past u64::MAX reads to the end of the file offset + size overflowed before the clamp to the file size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an SSE-encrypted manifest is refused without a decoder Only filer-encrypted manifests were; an SSE one went to the protobuf decoder as ciphertext. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an empty write leaves the staged file size alone A zero-byte write past the end grew the size to its offset, as a zero-byte pwrite does not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: the chunk cache keeps no empty chunks They add no bytes, so nothing evicted them and the map grew with every distinct empty needle, even at capacity zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: flushing with a zero chunk size is an error, not a panic Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: an empty master list fails at startup "" split into one empty address that passed the check, and the engine then waited two minutes for a filer before a generic error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Fable 5.1
|
2fe480a6e2 |
S3 Vectors: the gateway resolves the table it sends to weed-vector (#12468)
* lance catalog: in-process DescribeTable for trusted callers DescribeTable and DescribeTableInternal resolve a table's location and storage options without the HTTP layer; the internal form runs the lookup through a trusted manager so the caller's catalog grants do not gate a resolution its own permission already authorized. Same code as the Enterprise repository. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector.proto: Table carries its location and storage options The gateway resolves the table it addresses, so the engine can open it without a describe round trip of its own. Both fields are optional; an empty location means the engine looks the table up as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: test the in-process describe Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: resolve the table before calling the engine The gateway describes the table in process, through the trusted catalog manager, and sends its location and storage options with every engine call. Storage options are vended for the caller's principal, as the Enterprise router does for queries. A catalog failure answers with the code the engine would have given. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: open a table at the location the gateway sent A request whose table carries a location is opened there, with the storage options the gateway vended over the engine's fallback; the catalog is described only for a table that comes without one, as from an older gateway. dataset::open_at and dataset::drop_at take a description the caller already holds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: run the engine without a catalog VECTOR_ENGINE_NO_CATALOG=1 gives the engine an unreachable catalog URL; every CI leg sets it, so the suite proves the gateway resolves each table it sends. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: an internal describe may write a vector index With credential vending on, the options vended to the engine followed the caller's s3tables grants, which never allow writing the vector namespace, so every write the gateway sent would have run with read-only credentials and reads would have lost their version lease. The engine is that namespace's writer and the caller's vector permission authorized the operation, so an internal describe of a vector table vends write-capable options; other namespaces keep the UpdateTable check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: resolve tables for the engine only while the catalog runs With -port.lance=0 the gateway has no catalog to resolve through; the engine then keeps looking tables up itself, as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: a deregistered table describes as NoSuchTable So the gateway answers NotFoundException for it, not InternalFailure. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: trust a sent location only for a colocated vector index A caller that reaches the engine could otherwise name any storage the engine's credentials cover. A location is taken from the request only when the id names a vector index and the location is its colocated path, which the engine writes already; anything else is described through the catalog as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: scope the read cache by storage endpoint The cache was keyed by bucket and path alone, so the same path reached through another endpoint would have shared entries. Each open wraps the store with the cache scoped to the endpoint its options name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep a configured endpoint over one a request names A caller that reaches the engine could otherwise send its fallback credentials to an endpoint of the caller's choosing. When the engine's --storage-option names an aws_endpoint, a table whose request names another one is described through the catalog instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: use request options only where credentials cannot leave A request's storage options are taken only when they hold the keys a gateway's describe produces, so a proxy or an endpoint alias cannot route the engine's credentials elsewhere, and an endpoint only when it is the configured one or the engine holds no credentials to send. Anything else sends the table to the catalog. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: give the engine its endpoint with its credentials Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a drop checks the request's options like an open Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: an unpinned endpoint needs the request to sign for itself Without configured credentials the S3 store still signs with the AWS default chain, so an endpoint from a request is taken only when the request skips signing or brings its own credentials. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a trailing slash does not unpin the endpoint, and say so A configured aws_endpoint written with a trailing slash matched nothing the gateway sends, so every table fell back to the catalog in silence. The comparison ignores the slash, and a fallback for a resolved table is logged at debug. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: resolved by the gateway, or refused The engine no longer has a catalog: --namespace and --filer-signing-key are gone, a request whose table names no location is InvalidArgument, and one whose location is not the index's own or whose storage options the engine's credentials must not be used with is PermissionDenied. A drop removes the colocated path it was given. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: the catalog resolves vector tables whether or not its port is served newLanceServer builds the Lance catalog for the S3 Vectors API; the listener on -port.lance is optional. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors tests: start the engine without a catalog Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: the aux listener test builds the catalog before serving it Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: an unpinned endpoint takes only credentials the request signs with alone Lance fills an option a request leaves out from the engine's environment, under any of object_store's names for it (TOKEN, SESSION_TOKEN, ...), and asks the AWS default chain when the key pair is incomplete. A request that named its own endpoint with only an access key id, or a key pair while the engine's environment held a session token, would have sent the engine's credentials there. Such a request now needs a full key pair, and its own session token whenever the environment would supply one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: accept --namespace and --filer-signing-key, ignored Both flags shipped in 4.52 and are documented, so dropping them made an existing command line exit at startup with an unexpected-argument error. They are kept hidden and ignored, with a warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors design: the engine is started without a catalog Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: the trust check takes the environment it reads The options test read the process environment through signs_alone, so a host exporting AWS_SESSION_TOKEN or TOKEN failed it. The caller now passes the variable names in; describe passes the engine's own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: log both endpoints when a request's options are refused The gateway turns the PermissionDenied into a bare AccessDenied and logs nothing, so a pinned aws_endpoint written in another form than the one the gateway sends (host case, a default port, an IP for a name) failed every call with no trace of which value differed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: the Lance catalog sets the credential role whether or not Iceberg runs -iceberg.credentialRole reached the S3 server only while the Iceberg listener started, so with -port.iceberg=0 the Lance catalog installed its vendor but vending stayed off, and the vector engine got no credentials for the tables the gateway resolves. Both catalogs now set the role through one helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: set the credential role once, before either catalog serves Both catalogs wrote the role as they were built; the second write landed while the first was already serving requests that read it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
7beac0e8ea |
admin: show disk usage on the dashboard instead of the data size (#12466)
* admin: show disk usage on the dashboard instead of the data size The dashboard's Total Size card summed the volume and EC shard sizes the volume servers report. That leaves out deleted needles waiting for vacuum, EC generations kept after a vacuum, and anything else on the disks, so on a cluster with a lot of EC vacuum activity it showed far less than the disks held. The card now shows the bytes in use on the volume servers' disks, the same statfs numbers the volume server list uses, with the disks' capacity, and its trend line follows that number. Servers that report no statfs numbers count their data size, as on the volume server list. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * admin: compute the disk usage percentage in float64 current*100 overflows int64 past about 92 PB of used space, so the dashboard card showed a negative percentage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: size the volume server disk bar with calculatePercent64 int(host.DiskUsage) truncates byte counts on 32-bit builds, so the bar width was wrong for any disk over 2 GB there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: test the cluster disk totals across statfs and fallback servers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
fd81a50c57 |
mount: keep retrying writeback chunk uploads through a volume outage (#12465)
* mount: keep retrying writeback chunk uploads through a volume outage With -writebackCache, close() returns before the data is uploaded. When the volume servers stay unreachable past the upload's own retry (~20 s), the chunk was dropped and the file never appeared, with no error to the application. Retry transient upload errors with the same bytes until they succeed, unless the dirty pages were discarded. Permanent errors and the synchronous mode keep failing as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test writeback upload retries and the cases that still fail Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * operation: keep the filer's read-only verdict on assign failures UploadWithRetry flattened AssignVolumeResponse errors into a string, so callers could not tell a read-only rule from an outage. Wrap the filer's coded error so errors.Is(err, filer_pb.ErrReadOnly) works; the message is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: retry writeback uploads the upload layer reassigns, never read-only A filer proxy that cannot reach the volume answers 500 with an empty body, which util.IsTransientError does not match, so that outage still dropped the chunk. Also retry what operation.ShouldReassignUpload takes. A read-only rule stays final even when its path or message contains a transient word such as "unavailable". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: stop a writeback upload once its dirty pages are discarded Destroy() was only seen after a failed attempt, so a truncate during the retry wait still started another upload, and a chunk that finished after its pages were replaced was attached to the new content. Check before each attempt, and drop the chunk instead of attaching it; volume.fsck reclaims it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep Destroy from slipping between the discard check and the append truncateEntry destroys the pages and then installs the new entry, so a Destroy landing after the check but before AddChunks put the old chunk into the replacing entry. Hold one lock across the check and the append, and take it in Destroy while setting the flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: leave the Windows interrupt drain to the capped filesystem hook Interrupt hooks run in reverse registration order, so the Windows hook ran before the filesystem hook and waited on pending writeback flushes without a bound. With writeback uploads now retrying through an outage, Ctrl-C could wait until the volume servers came back. Unmount only, as on Linux, and let the filesystem hook drain for at most 30 s and keep the write cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test a writeback upload whose assign keeps timing out With every volume server gone the assign RPC times out until the filer client gives up; that error must be retried like a refused upload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
github-actions[bot]
|
7d6680c7a4 | docs: regenerate star history chart | ||
|
Chris LuandClaude Opus 5.5
|
99539484b6 |
grafana: chart the remote-deletion tombstone gauges (#12444)
* grafana: chart the remote-deletion tombstone gauges Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: describe the fetch hold that outlasts a full tombstone table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: word the tombstone panel description like the gauge help Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
1c4fe61fcd |
master: stop the resumed single-master leader from checking quorum every 1 ms (#12463)
* deps: bump seaweedfs/raft so the leader quorum check follows the election timeout A single master resuming raft state self-elects on a 1 ms election timeout and restores the configured one afterwards, but the leader kept running its quorum check every 1 ms for the rest of the process. The raft change re-arms that check from the current timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * deps: bump seaweedfs/raft to the raft PR head with the extended test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * deps: bump seaweedfs/raft for the non-positive election timeout guard Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * deps: bump seaweedfs/raft so a timeout change re-arms the leader quorum check Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * deps: pin seaweedfs/raft v1.2.4 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
cdea7e79e0 |
filer: fold small chunks into manifests 256 at a time (#12462)
* filer: fold small chunks into manifests 256 at a time Every metadata event carries the file's whole chunk list, so a file grown by small appends re-sent each chunk record on every append until the list reached ManifestBatch (10000). Chunks under 1 MiB now fold once 256 of them build up; full-size chunks still fold at ManifestBatch, so large files keep their flat lists and read without a manifest fetch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: merge appended small chunks at the small-chunk fold count The append path merges small chunks into full-size ones, but only at 1000 of them; the fold now hides small chunks in a manifest at 256, so the merge would never see enough of them. Merging at the same count keeps rewriting them into full chunks first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
728b476ba2 |
filer.remote.gateway: keep the remote object while a version under a delete marker is locked (#12459)
* filer.remote.gateway: keep the remote object while a version under a delete marker is locked Every version of a versioned bucket uploads to the object's key, so on a remote without versioning that object is the only remote copy of the latest version. A delete marker deleted it even when a version still held a legal hold or unexpired retention. Keep it while any version of the key is locked, the same rule the filer uses before pruning a remote entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway, filer.remote.sync: say that earlier versions need remote versioning Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: delete a kept delete marker's object once the version lock is released A delete marker kept for a locked version stays unstamped. When an update or delete leaves a version that carried a legal hold or retention without an active lock, look up the key's latest version and, if it is still that unstamped marker, run the delete again. Only changes to versions that carried a lock trigger it, so the noncurrent mark and tag updates of unlocked versions do not send repeat remote deletes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: test the kept delete marker retry through the bucket mount The retry reads the key's versions through a filer client parameter, as the rename helpers do, so a test can check the mapped object key and the marker stamp, and that lock-releasing updates and deletes reach it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: check the lock release lookup goes to the gateway's filer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
8c22336944 |
filer.meta.backup: stream from the start of an unfinished walk (#12458)
* filer.meta.backup: stream from the start of an unfinished walk A walk only inserts entries, so when the first walk is interrupted the next start walks again and streams from the second walk's start: deletes and renames between the two walks never reach the backup store. Keep the first walk's start time in the store until a walk completes and stream from it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.meta.backup: start fresh once the stream has passed an earlier walk's start A walk start left in the store by a crash between saving the offset and clearing it, or by an interrupted -restart followed by a plain restart, stops mattering once the stream saves an offset past it. Ignore it then instead of replaying from it on the next walk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.meta.backup: fail the walk on an unreadable offset Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
jsasandjames
|
5ba97b4432 |
filer: a full remote-deletion tombstone table keeps the lazy fetch off (#12440)
* filer: a full remote-deletion tombstone table keeps the lazy fetch off At the cap a new delete under a remote mount was dropped with a log line and isRemoteDeletionPending answered false for it, so the lazy remote fetch resurrected deleted paths. While the table is full every path may be a pending delete: isRemoteDeletionPending answers true until the table has room again. The size and the full state are gauges (filer_remote_deletion_tombstones, filer_remote_deletion_tombstones_full). (cherry picked from commit f3be0ab965cde3bfd3053d3ac720fbbe3f68e98b) * filer: a full tombstone table drains on reads and lifts after the unrecorded deletes age out While the table is full, a lookup miss evicts expired tombstones and those the write-back offset has passed, so the table drains without a mutation. The full state lifts only once the table has room and the newest unrecorded delete is past the TTL: an unrecorded delete has no event to confirm it, so only the TTL makes it safe, as for a pending tombstone. (cherry picked from commit 19394fe5d67798af7b0d755ac18c83c75e20a12b) * filer: a full tombstone table drains per mount, only when its offset advances Offset-based release applies to the tombstones under the mount whose offset was read; another mount's tombstones answer to their own offset. A read sweeps the table only once that offset advanced since the last drain for the mount or the unrecorded deletes aged past the TTL, so a listing of many missing children does not rescan the table per child. (cherry picked from commit 9cf00e2e54d70e6ba530e6ec1b1ff4b502873847) * filer: the tombstones-full gauge help names when it clears --------- Co-authored-by: james <james@medable.com> |
||
|
|
7fc33e7fe3 |
filer: no lazy remote fetch inside a store transaction (#12439)
* filer: no lazy remote fetch inside a store transaction A rename onto an existing target deletes the target inside the store transaction, then CreateEntry's FindEntry misses the row and runs the lazy remote fetch, which persists the archive copy on a second connection. The store wrapper drops that call's deadline, so the UPDATE waits without bound on the row lock of the transaction's own uncommitted DELETE while the rename goroutine holds the path's exclusive locks: every later lookup, rename and delete of the path on that filer waits forever. A lookup whose context carries the store transaction skips the fetch. (cherry picked from commit b69e9535c2597d0c8e856f031d3778fddfd4159c) * filer: materialize remote-only rename paths before the transaction With the lazy fetch off inside a store transaction, a source or target the filer holds only on the remote would read as absent to AtomicRenameEntry and StreamRenameEntry: a rename with FailIfTargetExists could overwrite it, and a remote-only source would skip the deferred copy. PrefetchRemoteEntries runs the fetch for both paths before BeginTransaction, under the path locks. (cherry picked from commit e61f9718018669ef59768cd5eda25b6466d69597) * filer: skip the lazy remote fetch only in CreateEntry's lookup inside a transaction The guard on every lookup inside a store transaction hid a source or target the filer holds only on the remote from the rename, and the prefetch before the transaction did not reach the children of a directory rename. The deadlock needs a write: CreateEntry's lookup after the transaction deleted the path. That lookup alone skips the fetch; reads inside the transaction keep it, and their persist on a second connection holds no lock the transaction waits for. (cherry picked from commit f555092c324780a8fe5314b238694551e3ef6be5) * filer: mark a store transaction independently of the store's own key ArangoDB and FoundationDB keep their transaction under other context keys, so the CreateEntry guard keyed on "tx" missed them. The filer now marks every context returned by BeginTransaction and the guard checks that marker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit |
||
|
Chris LuandClaude Opus 5.5
|
86db60b669 |
s3api: mint a versioned PUT's id at commit, after the lock wait and the upload (#12454)
* s3api: mint a versioned PUT's id after the object write lock putVersionedObject minted the version id before the upload and the lock wait. A DELETE that committed while the PUT waited got a newer id, so a conditional PUT whose If-None-Match passed against the delete marker was stored behind it: accepted, but GET returned NoSuchKey. The finalize now re-mints the id once the lock is held and the precondition passed, so the version orders after everything the check saw. An ambiguous routed attempt keeps its id, since recovery looks for that entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: mint a routed versioned PUT's id after the upload On the routed path the version id was still the one minted before the upload, and RECOMPUTE_LATEST picks the newest name. A DELETE that committed during a long upload left its marker latest over the accepted PUT. Re-mint right before the routed transaction, as the lock path does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1176f975b3 |
filer: release metadata log chunks a failed append leaves unreferenced (#12455)
* filer: stop unbounded metadata-log growth on append-only files Two independent leaks make a filer metadata log grow far faster than the data it describes. On a cluster whose live logs weigh a few hundred megabytes, they produced tens of gigabytes of metadata per day. 1. Metadata events embed the whole old and new entry, chunk lists included. An append-only file — a log or a journal written in small pieces — therefore re-serializes every chunk it ever had on every append, so the cost is quadratic in the number of chunks. A 2,700-chunk file emits a couple of megabytes per append. Cap the carried chunk list at 256 entries and set metadata_event_chunks_truncated so subscribers that need the content read the entry from the filer instead of trusting the event. Subscribers that only track names, attributes, or chunk deltas keep working unchanged. 2. appendToFile uploads the chunk before committing the filer entry. When the commit fails — store error, cancelled context, filer shutting down — the chunk is stranded: no entry references it, so no later delete can mark it, and neither vacuum nor fsck can see it because both work from the entry index. The volume server keeps counting those bytes as live data forever. Release the chunk when the commit fails. * filer: keep chunk lists in metadata events Mount meta caches, peer filers replaying into their own store, and the filer.sync, filer.backup and filer.meta.backup sinks all store the event's entry as the file. Dropping the chunk list past a count left those copies of a large file without content. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: release a metadata log chunk the lookup or commit leaves unreferenced The flush retries a failed append with a new upload, so a store outage left one stranded chunk per retry. A failed lookup strands the chunk too. Use DeleteUncommittedChunks, as the other write paths do: the chunk was never part of the file, so path chunk-deletion rules do not apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: keep a metadata log chunk the failed create stored CreateEntry inserts a new entry before its parent folders, so a parent failure returns an error with the entry already referencing the chunk. Release it only when the store shows no entry, or an entry without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: count a metadata log piece the failed create stored as appended The flush retry would append the same bytes after it, and a piece cut inside a record then breaks replay of that segment. The retry takes the update path, which does not create parent folders either, so returning the error only duplicated the piece. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Martin Morez <martin@morez.org> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
2be4498c04 |
s3 lifecycle: walker deletes carry the listed identity and must target the latest version (#12452)
* s3 lifecycle: send the listed entry's identity on walker deletes The daily walker dispatched LifecycleDelete with no ExpectedIdentity, so the server skipped its compare-and-swap and an object rewritten between listing and dispatch was deleted. The list adapter now records each listed entry's identity, computed the same way as on the event path, and the walker sends it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: current-version expiration must target the latest version The walker addresses the latest version of a versioned key by id. A version file never changes, so its identity still matches after a newer PUT, and the delete went ahead: a delete marker over the new version with versioning enabled, or the new null version deleted while suspended. LifecycleDelete now resolves the key's latest version for EXPIRATION_DAYS/EXPIRATION_DATE requests that carry a version id and answers NOOP_RESOLVED (VERSION_NOT_LATEST) when it is a different one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: check the latest version for unaddressed suspended expirations With versioning suspended, an expiration without a version id reads the plain entry, which is the null version but not necessarily the latest: a newer real version can sit in .versions. The latest-version check now runs in the expiration branch, by versioning state: for every suspended request (an unaddressed one addresses the null version) and for enabled requests that carry a version id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test that each version sibling carries its own identity Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
9af736da62 |
volume: stop serving EC reads from quarantined shards (#12450)
* storage: skip quarantined EC shards on local reads A needle read tried the local shard first even when its disk was quarantined after repeated IO errors, so every read paid the failing disk's EIO latency before falling back. Treat such a shard as not local so the read goes to peers or reconstruction, and read it only as a last resort when no healthy copy answers. Scrub still reads it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: refuse EC shard reads from a quarantined disk VolumeEcShardRead served peers from a shard whose disk was quarantined, and it ignored the .ecx lookup error: a failed index read returns a tombstone size, so the peer was told a live needle was deleted. Refuse quarantined shards and fail the request on an index read error; a needle missing from the index still reads, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: report quarantined EC shards deleted right away Quarantining an EC volume only dropped it from later full EC heartbeats, so the master kept routing reads to the faulty disk for up to 17 heartbeat intervals. The heartbeat that quarantines the volume now carries its shards as deleted, once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: do not announce EC shards mounted onto a quarantined volume MountEcShards and the shard load notifier reused a quarantined EcVolume and announced the new shard, pointing the master back at the faulty disk until the next full EC heartbeat. Skip the announcement, and fail the mount so a caller moving a shard here keeps its source copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: count IO errors on local EC shard reads Go's readLocalEcShardInterval feeds every local shard read into the EC volume's IO error streak; the Rust needle read and recovery gather did not, so a failing disk under local reads never moved toward quarantine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: skip quarantined EC shards on local reads Mirror the Go read path: a needle read and the recovery gather treat a shard on a quarantined disk as not local, and read it only as a last resort once peers and reconstruction fail. Scrub still reads it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse EC shard reads from a quarantined disk Mirror Go's VolumeEcShardRead: answer Unavailable for a shard whose disk is quarantined, so the peer tries the next holder or reconstructs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: report quarantined EC shards deleted right away Mirror Go: the volume heartbeat that quarantines an EC volume carries its shards as deleted, once, instead of leaving the master to drop them at the next full EC heartbeat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: fail EC shard mounts onto a quarantined volume Mirror Go's MountEcShards: the heartbeat already leaves such a shard unannounced, so fail the mount and let a caller moving the shard here keep its source copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: keep reading EC needles when the shard lookup fails A failed or short master lookup aborted the read before any shard was tried, so the quarantined-disk last resort never ran once the master had dropped those shards, and local shards went unread whenever the master was unreachable. Read on with the cached locations, as the Rust server already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: log a failed last-resort read of a quarantined shard When reconstruction and the quarantined read both fail, say why the local read failed too. Go and Rust. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: drop a shard whose mount was refused on a quarantined disk The refused mount left the new shard loaded on the quarantined volume, where it was neither served nor announced, and the in-memory claim drew later copies of that shard back to the same disk. Unload it again, keeping a shard that was already mounted there. Go and Rust. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: finish a mount batch before reporting a failed shard A shard that failed to mount, now including one refused on a quarantined disk, returned before the sidecar reload, and in Rust before the heartbeat notify, so shards mounted earlier in the same batch kept checksum protection off and reached the master only at the next full EC heartbeat. Stop at the failed shard but finish the rest first. Go and Rust. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: keep a peer's deleted answer when its stream fails after it A shard read recorded the holder's deleted answer, and a stream error after it failed the read: Rust discarded the flag, and Go let the next source or the reconstruction result overwrite it. With the last-resort read of a quarantined shard, that could serve a needle a peer had reported deleted. Once the deleted answer is in, end the read on a later stream error instead of failing it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: use the platform media error in the quarantine heartbeat test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: rebuild EC intervals from quarantined shards when healthy ones fall short Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: rebuild EC intervals from quarantined shards when healthy ones fall short Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: serve a peer's EC shard read from a healthy copy on another disk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: serve a peer's EC shard read from a healthy copy on another disk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a quarantined EC shard read instead of serving a sibling copy A sibling disk's copy can have its own deletion journal, and an EC blob delete is journaled on one local copy only, so the sibling could serve a deleted needle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: refuse a quarantined EC shard read instead of serving a sibling copy A sibling disk's copy can have its own deletion journal, and VolumeEcBlobDelete journals a delete on one local copy only, so the sibling could serve a deleted needle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: check the disk an EC shard mounted on for quarantine mount_ec_shard now returns the disk it mounted on, and the mount RPC checks only that disk, like Go's MountEcShards. Before, a quarantined sibling holding the same shard failed the mount while the healthy disk's copy stayed mounted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: test an EC mount beside a quarantined sibling Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: check the quarantined-sibling mount test by its result A server that remounts a shard on the disk already claiming it refuses the mount there instead. The test now only requires that the result matches whether the healthy disk ended up holding the shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: check the quarantined-sibling mount test by its result Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: expect the quarantined-sibling mount result from the disk the store picks The test accepted a refused mount whenever the healthy disk stayed empty, so a handler that refused before mounting passed too. Ask the store which disk it picks first, then require success exactly when that is the healthy disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: expect the quarantined-sibling mount result from the disk the store picks Mirrors the Go test. add_location's cross-disk reconcile mounted shard 1 on the healthy disk too, so the test now unmounts it there first, leaving the quarantined sibling as the only claimant, as in Go. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2df7d435f2 |
filer.remote.sync: bound the stamp and offset writes, startup offset fixes, stamp a copy (#12447)
* filer.remote.sync: bounded filer lookups, one chunk cache per process The filer lookups on the sync path (currentEntry, isSuperseded, uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no deadline. A lookup that never answers keeps its job alive, the job keeps its path in the MetadataProcessor conflict index, AddSyncJob waits on the condition and the daemon stops consuming events with no log line. Each lookup now carries a 30 s deadline, so the job fails and the processor's stop-and-resubscribe path runs. remoteUploadReader built a chunk cache per upload; the cache's worker goroutine is never stopped, so each upload leaked one goroutine and its cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory limit). One chunk cache per process. (cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78) * filer.remote.sync: a failed mount lookup at startup never resolves to now collectLastSyncOffset returned time.Now() when the mounted directory lookup failed, which skips every event since the saved offset. The saved offset is used instead; without one the daemon exits so a restart retries. (cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae) * filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read retriedWriteFile returns the supersession lookup's error instead of treating an unanswered lookup as "not superseded": in a replay an older snapshot could otherwise overwrite a newer remote object. The processor retries the event. isSuperseded keeps its boolean form for the other call sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls back to the saved offset cannot hang on it. TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed lookup; it now asserts the withheld upload and the lookup error. (cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa) * filer.remote.sync test: keep the slices import the other gateway tests use * filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo After a failed upload attempt the supersession lookup's error ended in isSuperseded's boolean and the loop could upload the snapshot again unverified. The lookup error now ends the loop (errLookupUnanswered, not retried) and the processor retries the event. With -timeAgo set, a failed read of the saved offset resolved to now minus timeAgo, skipping everything older; the daemon exits instead so a restart retries. (cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566) * filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting resolveLastSyncOffset returns the lookup or offset-read error; both callers (directory and buckets) hand it to util.RetryUntil, which retries with backoff instead of the process exiting. * filer.remote.sync: keep doc comments on the functions they describe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: isSuperseded returns the lookup error itself Its only production callers needed the error, so the boolean wrapper was left to the tests. The tests now assert the error on a failed lookup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: bound the stamp written after each upload updateLocalEntry's UpdateEntry was the last filer call on the sync path without a deadline. A filer that never answers the stamp keeps the job and its path in the conflict index the same way an unanswered lookup did. The stamp carries IF_ENTRY_EQUAL, so a retried event after a timeout either stamps again or skips a stamp that already landed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: an unread offset never resolves to the mount's creation time With the offset read now bounded, a timed-out read fell through to the directory's creation time and replayed every event already synced. The error goes back to the startup retry loop, as with -timeAgo; a missing offset still reads as zero without an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: stamp a copy of the event entry updateLocalEntry set the RemoteEntry and dropped the rename marker on the caller's entry before the RPC. A transient stamp failure retries the same event in place, which then read as already synced and skipped both the upload and the stamp. The stub filer now applies a successful update the way the filer does, since tests had relied on that shared object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: a directory not created yet starts now, as before A gateway started before the first bucket looked up a missing buckets directory, had no saved offset, and returned an error to the retry loop on every attempt until a bucket appeared. A missing directory with no saved offset has nothing to replay, so it starts now as it did before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: take the start time before the mount lookup A bucket created between a not-found lookup and the end of the offset read had its creation before a start taken afterwards, so the gateway never created its remote bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: save the resume position when a subscription ends A failed event ends the subscription with ErrResubscribe, and the retry loop runs startup again, recomputing -timeAgo from a later now. The periodic checkpoint only runs as events arrive, so a failure before the first one could start the retry after the event it failed on. The resume position is now saved whenever the subscription returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * remote_storage: bound the sync offset write like the read The offset write runs in the subscription's checkpoint callback and, now, when a subscription ends; a filer that never answers it stalled either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: say that any saved offset wins over the creation time The creation time in seconds was compared with the offset in milliseconds, which made any saved offset win. That is the behaviour to keep: remote.unmount recreates the directory and then saves the offset, usually in the same second, so a seconds comparison would start at the creation second and replay the purge. The comparison now states it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: keep the resume position for the retry when its write fails The retry in the same process now resumes from the position the ended subscription returned, as filer.backup does, so a failed offset write followed by an empty read cannot recompute -timeAgo past a failed event. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync test: give the same-second saved offset a fractional part With whole seconds, returning the creation second resolved to the same time and passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: james <james@medable.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
206d7b1418 |
filer.remote.sync: bounded filer lookups, one chunk cache per process (#12441)
* filer.remote.sync: bounded filer lookups, one chunk cache per process The filer lookups on the sync path (currentEntry, isSuperseded, uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no deadline. A lookup that never answers keeps its job alive, the job keeps its path in the MetadataProcessor conflict index, AddSyncJob waits on the condition and the daemon stops consuming events with no log line. Each lookup now carries a 30 s deadline, so the job fails and the processor's stop-and-resubscribe path runs. remoteUploadReader built a chunk cache per upload; the cache's worker goroutine is never stopped, so each upload leaked one goroutine and its cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory limit). One chunk cache per process. (cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78) * filer.remote.sync: a failed mount lookup at startup never resolves to now collectLastSyncOffset returned time.Now() when the mounted directory lookup failed, which skips every event since the saved offset. The saved offset is used instead; without one the daemon exits so a restart retries. (cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae) * filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read retriedWriteFile returns the supersession lookup's error instead of treating an unanswered lookup as "not superseded": in a replay an older snapshot could otherwise overwrite a newer remote object. The processor retries the event. isSuperseded keeps its boolean form for the other call sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls back to the saved offset cannot hang on it. TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed lookup; it now asserts the withheld upload and the lookup error. (cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa) * filer.remote.sync test: keep the slices import the other gateway tests use * filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo After a failed upload attempt the supersession lookup's error ended in isSuperseded's boolean and the loop could upload the snapshot again unverified. The lookup error now ends the loop (errLookupUnanswered, not retried) and the processor retries the event. With -timeAgo set, a failed read of the saved offset resolved to now minus timeAgo, skipping everything older; the daemon exits instead so a restart retries. (cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566) * filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting resolveLastSyncOffset returns the lookup or offset-read error; both callers (directory and buckets) hand it to util.RetryUntil, which retries with backoff instead of the process exiting. * filer.remote.sync: keep doc comments on the functions they describe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit |
||
|
Chris LuandClaude Opus 5.5
|
1406335301 |
volume: a delete cancels writes of its needle still in flight (#12445)
* volume: a delete cancels the writes of its needle still in flight A write queued behind the volume locks, for example behind another write's fsync on a busy disk, is not in the needle map yet. A delete of that needle in the meantime finds nothing, and the write lands afterwards, so the needle comes back on this copy only while the other replicas have it deleted. Each volume now tracks its writes from entry until they return. A delete cancels the ones for its needle, and a cancelled write that landed is tombstoned before it returns, unless the needle was deleted or rewritten since. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: a replicate delete that finds nothing cancels the pending write Rolling back a failed chunk upload sends DELETE ?type=replicate to every holder, including the slow one whose write timed out on the client. That holder has not indexed the write yet, so the delete answered 404 and the write landed live afterwards. Cancel the needle's pending writes, then look again in case one finished in between. A plain delete that finds nothing still answers 404 without cancelling: it is not sent on to the replicas. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: a delete cancels the writes of its needle still in flight Mirrors the Go volume. Volume::track_pending_write registers a write until its guard drops, delete_needle cancels the needle's pending writes, and settle_pending_write tombstones a cancelled write that landed unless the needle was deleted or rewritten since. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: track uploads and cancel them on a replicate delete An upload is tracked from the handler, and a durable one from submission to the write queue, where the guard travels with the request: a handler dropped while the write waits in the queue does not untrack it, and the worker settles it after the batch. A replicate delete that finds nothing cancels the needle's pending writes and looks again, like the Go handler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: take a cancelled write back in the lock span that publishes it The write tombstoned itself only after it returned, with the volume locks released. In that gap a compaction could move the record so the offset check skipped it, a close could drop it, and an identical write sent after the delete could match the live record and report success before the tombstone removed it. The check now runs right after the publish, under the same dataFileSyncLock and dataFileAccessLock hold, in the plain, fsync and batch write paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: a delete cancels only the writes carrying its cookie A delete with the right needle id but the wrong cookie could cancel an in-flight write and get it tombstoned. Each pending write now records its cookie, and a cancel only marks the matching ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: take a cancelled write back in the lock span that publishes it Mirror the Go change: settle_pending_write ran after write_needles_grouped returned, so a compaction, a close or an identical write later in the same batch could slip in before the tombstone. drop_cancelled_write now runs right after the publish, under the same sync and access lock hold, in the synced run, the unsynced run and write_needle_unsynced_or_refused. write_pending_needles carries each write's PendingWrite down to that point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: a delete cancels only the writes carrying its cookie Mirror the Go change: a delete with the right needle id and the wrong cookie no longer cancels an in-flight write. Each tracked write keeps its cookie next to its cancel flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a queued write whose volume was remounted The write queue tracks a write on the volume found at submit, but the worker looks the volume up again by id. After a remount, a delete reaches the new volume and cannot cancel the write registered on the old one, so the worker now refuses such a write instead of landing it on the new volume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: decode the delete's type parameter like Go's FormValue The delete handler matched the raw query text for type=replicate, so a percent-encoded name or value was not treated as a replica delete and did not cancel the write it undoes. Parse the query the way the upload handler does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: test a replica delete landing in a write's fsync window Drive DeleteHandler while a durable write is parked in its .dat fsync, appended but not indexed: the replica delete answers 404 and the write is taken back when it publishes, while a plain delete leaves it live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: flush a take-back tombstone before the index row that points at it A durable write taken back during its fsync appends the tombstone after the .dat sync, and the .idx sync that follows made its row durable while the record itself could still be lost, which reloads the volume read only. The fsync and batch paths now sync the .dat again first when a take-back appended a tombstone; a failure stops the volume taking writes like a failed index sync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: flush a take-back tombstone before the index row that points at it A durable write taken back in write_synced_run appends its tombstone after the .dat sync, so the .idx flush that follows could make the row durable while the tombstone record itself was still in the page cache. The run now syncs the .dat again first when a take-back appended a tombstone; a failure stops the volume taking writes like a failed index flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: a skip-cookie batch delete cancels the writes of the file id it names BatchDelete with SkipCookieCheck left the needle's cookie at zero, so the delete's cancel matched none of the in-flight writes and a retry queued behind it could land afterwards. The non-EC delete now carries the file id's cookie, as the Rust volume server already does; the EC path keeps cookie 0, which DeleteEcShardNeedle reads as "skip the check". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: release a held test write before the store closes A failed wait ended the test before release ran, so the writer stayed parked in its fsync and the store cleanup blocked on its lock until the package timed out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f71c0fe523 |
fix(s3tables): route REST RenameTable and refuse unrouted S3 Tables requests (#12446)
* s3tables: route REST RenameTable and refuse unrouted S3 Tables requests
PUT /tables/<arn>/<ns>/<name>/rename had no route, so it fell through to the
/{bucket} routers and ran as a PutObject into a bucket named "tables",
answering 2xx while the table kept its name.
- Answer any s3tables-signed request no S3 Tables route takes with 404
UnknownOperationException, before the health, domain and bucket routers.
- Route the AWS RenameTable REST call (newNamespaceName, newName, both
optional; versionToken).
- Honor versionToken on RenameTable, checked against the snapshot the claim
asserts so a commit between the two reads cannot slip a stale token through.
* s3tables: match the credential scope's service, not a region named s3tables
The unrouted-request fallback widens what a false match costs: an S3
client whose region is named s3tables had every request answered as an
unknown S3 Tables operation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: refuse an empty newName or newNamespaceName on REST rename
An empty field was read as omitted, so {"newNamespaceName":"other",
"newName":""} moved the table under its old name instead of failing
validation the way AWS does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: check the rename version token after the snapshot comparison
Once the separately read metadata is known to equal the claimed snapshot,
the token compares against the parsed metadata; no second decode of the
snapshot bytes is needed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables test: cover policy and tag changes in the rename snapshot compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables test: give each REST rename subtest its own source table
The token subtests renamed the table the first subtest produced, so run
alone they failed on a missing table instead of testing the token.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Chris LuandClaude Opus 5.5
|
c84c43f3da |
master: export Hashicorp raft metrics through the go-metrics global raft uses (#12448)
* master: keep the hostname out of raft metric names go-metrics prefixes gauge names with the hostname by default, which gives every master its own metric family, e.g. SeaweedFS_<host>_raft_peers. Prometheus already tells masters apart by the scrape target. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: send Hashicorp raft metrics through the go-metrics global raft uses hashicorp/raft and raft-boltdb report through github.com/hashicorp/go-metrics, but the master set up its prometheus sink on the armon/go-metrics global, so a -raftHashicorp master exported no raft metric families. The sink is now registered on the hashicorp/go-metrics global, and armon/go-metrics is no longer a dependency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: chart raft commit time and leader contact Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: keep the cluster in the raft panels Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: average raft commit time and leader contact over the rate window The go-metrics summaries keep quantiles for 10s, so a p99 scraped every 30s misses most observations. The _sum/_count rates count every sample. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: leave raft panels empty when nothing was observed Clamping the count rate plotted 0 ms for windows without commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: let the raft metrics test use its own registry Registering on stats.Gather made go test -count=2 fail with a duplicate collector registration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: register raft metrics before opening the raft stores raft.NewRaft starts its run loop before returning, so registering the sink afterwards could drop the first state and boltdb samples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
7b3983f154 |
s3api: speed up recursive listing of branched directory trees (#12438)
* s3api: look up a trailing-slash prefix's directory instead of listing its parent A prefix such as "account-0001/" names one directory, but the walk found it by listing the parent with the name as a prefix. SQL stores answer that with a LIKE that bounds the scan from below only, so in a wide bucket root the query reads every sibling after the name: seconds for a prefix holding a few keys. Look the directory up by name and serve it as the level's only entry. A miss, an error or a non-directory falls back to the prefix listing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: track failover receives atomically The failover wrapper flags a received reply through a plain bool, which a callback reading from the filer on several goroutines would race on. Use an atomic.Bool so the callback may share its filer client. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: read directories ahead of a recursive listing walk A recursive ListObjects walk enters directories one at a time, so a bucket made of many directories holding one or two keys each costs one filer round trip per directory, in series: seconds per 1000-key page on a store whose reads wait on disk. Read the first window of the directories the walk will enter on a few workers, nearest first in walk order, and hand each window over when the walk gets there. Every window the walk reads itself is drained first so its subdirectories are queued before the walk descends into the first one. The number of windows held ahead grows while directories end inside their first window and falls back when one does not, so a page ending inside a large directory leaves few reads wasted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: drop the prefetcher's record of every directory it was handed A page can cross any number of empty directories without spending its key budget, so a set of every announced directory grows with the tree. A directory is already handled when it is at or behind the walk, held or in flight, or queued; check those, which the queue and window caps bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: end a trailing-slash prefix listing at the file its lookup finds A file named by the prefix lists nothing under it, the same as the sibling listing it fell back to, which threw the lookup's answer away and paid the parent scan anyway. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: read a walk's own listing ahead instead of draining it Draining the window before the walk took its first entry made a page wait for the end of the listing, which a SQL store answering a name prefix reaches only after scanning the rest of the directory; a page that fills before then used to return at once. A background reader now hands the window's directories to the prefetcher as they arrive while the walk takes each entry, holding at most 1 MiB of entries ahead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: bound the bytes a prefetched listing window holds Entries with long chunk lists or inline content made the up to 64 held windows of 32 entries an unbounded amount of memory per listing. A window now stops at 128 KiB of entries and the walk lists the rest of that directory from the window's last entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test recursive listings whose streams break partway Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3e64326f2b |
s3: require their own actions to change a bucket's CORS or versioning (#12433)
* s3: require their own actions to change a bucket's CORS or versioning PUT/DELETE ?cors and PUT ?versioning were authorized as Write, so an identity granted only object writes on a bucket could replace its CORS rules or suspend versioning. They now need PutBucketCors, DeleteBucketCors and PutBucketVersioning (or Admin), and inline policies granting s3:PutBucketCors, s3:DeleteBucketCors or s3:PutBucketVersioning map to those actions instead of Write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test that object Write cannot change bucket CORS or versioning Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: offer the bucket CORS and versioning permissions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test each bucket config action with its own identity Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3a579ece3b |
s3: commit the delete marker in the owner's finalize transaction (#12432)
* s3: commit the delete marker in the owner's finalize transaction createDeleteMarker wrote the marker with mkFile to the gateway's current filer and then sent RECOMPUTE_LATEST to the object's ring owner. When the two differ, the owner scans .versions before peer metadata sync delivers the marker, re-derives the pointer to the previous version, and the marker stays non-current on every filer. routedVersionedFinalize now takes an optional version entry and PUTs it in the same ObjectTransaction ahead of the recompute, the way PutObject and CompleteMultipartUpload already do. The copy destination keeps passing nil: its version file is already written on the owner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test a versioned delete when the gateway's filer is not the owner Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: assert the delete marker and the pointer flip share one transaction Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a8766f5177 |
s3api: apply copy bucket-default SSE after the ACL re-auth (#12436)
* s3api: apply copy bucket-default SSE after the ACL re-auth CopyObject without its own x-amz-server-side-encryption header takes the destination bucket's default encryption, which CopyObjectHandler writes into the request headers. It did so before preparePutObjectACL, which re-authenticates a request that carries an ACL (x-amz-acl or a grant header) for s3:PutObjectAcl. verifySignedHeadersCoverage then found the synthesized x-amz-server-side-encryption* headers outside SignedHeaders and rejected the copy with SignatureDoesNotMatch. Move the bucket-default injection after the ACL check, so the re-authentication sees the request as the client signed it. The destination object still gets the bucket default encryption. * s3api: assert the copy's KMS key and stored ciphertext Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Stefan Scharlott <stefans@Stefans-MacBook-Pro.local> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d127be610f |
admin: re-send the master vacuum disable every poll so it survives leader changes (#12430)
* admin: re-send the master vacuum disable every poll so it survives leader changes
The admin disabled the master's scheduled vacuum only when a vacuum plugin
worker connected or disconnected. The master keeps that flag in memory and it
is not replicated through raft, so after a leader change or a leader restart
the new leader ran its own vacuum next to the plugin worker. When both picked
the same volume the master's commit failed ("compact swap aborted: missing
.cpd/.cpx") and it dropped that replica, leaving the volume under-replicated.
Send the desired state on every poll instead, and log on the master only
when the flag actually changes.
* admin: shorten the vacuum sync comments
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Kevin <kevin@leetify.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
github-actions[bot]
|
0dd33f5f0f | docs: regenerate star history chart | ||
|
Chris LuandClaude Opus 5.5
|
878979af70 |
filer: hide write controls in the UI on the read-only port (#12429)
* filer: hide write controls in the UI on the read-only port The read-only port accepts only GET and HEAD, but its directory page offered New Folder, Upload, drag-and-drop, rename and delete, each of which failed with 405. Requests that come through the read-only handler now render the page without those controls or the upload script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test the directory page controls on each port Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: match the rendered row buttons in the port test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e44c16802d |
weed/server: verify data shards against bitrot sidecar before ec decode (#12422) (#12427)
* weed/server: verify data shards against bitrot sidecar before ec decode (#12422) * ec.decode: verify data shards against the mounted volume's bitrot protection The decode looked up only the generation-0 .ecsum beside the data base and each location's data directory, so it missed a sidecar kept with the index and ignored the generation the volume actually serves. Take the protection the volume's per-disk runtimes resolved instead (On, else Invalid, else Off), and test a corrupt block past the first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.decode: check for live entries before verifying the shards A volume whose needles are all deleted gets its shards purged on the no-live-entries status; a failing checksum must not keep it around. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: verify data shards against the bitrot protection before a decode Mirrors the Go volume server: the decode takes the protection the volume's per-disk runtimes resolved and refuses a data shard that fails its .ecsum checksum, after the no-live-entries check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: a decode refuses a data shard that fails its .ecsum checksum Runs against both volume server implementations: the refused decode keeps every shard and the sidecar, and the needle stays readable through EC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.decode: reload the bitrot sidecar before verifying the shards A rebuild's backfill writes the sidecar without reloading the mounted runtimes, which then still report no protection; resolve it again across the EC metadata directories, as a shard mount does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: reload the bitrot sidecar before verifying the shards Mirrors the Go volume server: the decode job resolves the sidecar again on every runtime of the volume before choosing its protection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: a decode finds a sidecar written after the shards were mounted Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: read through the EC shards after the refused decode The source volume stayed mounted, so the final read never touched the shards it is meant to check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.decode: refuse when a sidecar reload reports another layout ReloadBitrotSidecar logged and dropped the layout-mismatch error, so the runtime stayed Off and the decode skipped the data-shard check. It now returns that error and the decode refuses on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a decode when a sidecar reload reports another layout Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: a decode refuses a late sidecar that records another layout Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.decode: refuse when an unmount left no runtime to resolve the sidecar Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a decode when an unmount left no runtime to resolve the sidecar Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
80f32bc074 |
weed/command: skip remote bucket delete when bucket is unmounted (#12418) (#12425)
* weed/command: skip remote bucket delete when bucket is unmounted * remote_storage: count deleting an already-missing bucket as done filer.remote.gateway can see the same bucket delete more than once, for example once per filer when several filers peer their metadata. A copy that still finds the mount mapping calls DeleteBucket again, and S3 (NoSuchBucket), Azure (ContainerNotFound, ContainerBeingDeleted) and GCS (404) all failed it, which held the sync offset on the delete. A missing bucket now counts as deleted, the way deleting a missing object already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: keep the mapping table writable after mount.mapping is deleted Deleting mount.mapping replaced the gateway's mappings with a table whose map was nil, so the next bucket create panicked writing its mount into it. Reset to an empty map instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: count a delete of an absent object as done filer.remote.sync already treats ErrRemoteObjectNotFound from DeleteFile as deleted, since GCS reports a missing object that way where S3 and Azure succeed. The gateway returned it, so a delete event seen twice, as it is when several filers peer their metadata, held the sync offset on GCS. Use the same deleteRemoteFile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: drop cached remote clients after the new gateway tests GetRemoteStorage caches clients by configuration, so a repeated run (-count=2) got the first run's recorder back and saw no calls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: keep the default storage across mount.mapping changes The default storage for new buckets is derived at startup and kept only in the gateway's copy of the mappings; mount.mapping never stores it. Replacing or deleting that file dropped it, so a later bucket create with no -createBucketAt skipped the remote bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
c971252871 |
admin: make the cluster lock's fairness window configurable (#12426)
* admin: make the cluster lock's fairness window configurable Scheduled jobs hold the cluster admin lock for their whole run. Once the lock had been held for a minute, every new job waited until all running jobs had released it, so with hour-long EC jobs dispatch ran in waves and most workers sat idle between them. -adminLock.fairnessWindow keeps the one-minute default; 0 lets new jobs join the shared hold, and shell clients get the lock when the scheduler next drains. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * admin: wait for the blocked acquire before disabling the fairness window The wake-up test slept 50 ms before SetFairnessWindow(0), so a late acquire that started after the change joined without exercising the Broadcast. Wait until it is registered as a waiter instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8c8d0f1cdf |
weed/shell: resolve collection from topology when ec.decode is given only volumeId (#12424)
* weed/shell: resolve collection from topology when ec.decode is given only volumeId
When ec.decode is invoked with -volumeId for a volume that belongs to a named collection without explicitly specifying -collection, DoEcDecode was called with an empty collection string. Target volume servers then rejected VolumeEcShardsCopy with 'CopyFile not found ec volume id <vid>' because they looked under the default collection.
This commit introduces FindEcVolumeCollection to discover the collection name directly from topology metadata (checking EC shard info and regular volume info), ensuring DoEcDecode receives the accurate collection.
* ec.decode: resolve every batch volume's collection from the topology
The batch passed -collection through unless it looked like a regex, so a
comma list (a,b), a {n} quantifier, ALL or _default_ reached the volume
servers as a collection name and the copy found no shards. The selector
only picks the volumes; each one is decoded in its own collection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: take the collection only from the volume's ec shards
The lookup returned the first record per disk, so a regular volume on an
earlier node answered before the ec shards on a later one. A decode needs
the shards anyway, so their records are the only ones to read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: leave the -volumeId lookup to DoEcDecode
DoEcDecode already resolves an empty collection from the topology.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
120d17a092 |
filer.remote.sync: stop a touched file from being restamped forever (#12419)
* filer.remote.sync: stop a touched file from being restamped forever A metadata-only update that moves the mtime past RemoteMtime (touch, or an O_TRUNC open of an empty lock file) goes through UpdateFileMetadata, which changes nothing on the remote, and the entry is stamped with the same RemoteEntry and a new LastLocalSyncTsNs. That stamp is an update event of its own. shouldSendToRemote only looks at LastLocalSyncTsNs when RemoteMtime and the mtime share a second, so it still reads the entry as dirty, and the event is stamped again. Nothing is uploaded, but the daemon keeps rewriting the entry as fast as it reads its own events back. Skip the restamp when the backend left the object alone and the event's entry already carries a stamp taken after its mtime. The touch itself is still stamped once. * filer.remote.sync: log the skipped restamp The other skips in processUpdateEvent log at V(2); this one did not, so with -v=2 a touched file showed an "update meta" line with nothing after it. * filer.remote.sync: skip the restamp of a stamp's own event, whatever the clocks The skip compared the entry's stamp with its mtime, so an mtime ahead of the sync's clock (a skewed writer, touch -d) stayed past every stamp and kept the loop going until the clock caught up. An event that changed only the RemoteEntry is the sync's own stamp, which needs no other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: ignore the filer's commit timestamp when spotting a stamp The filer gives every stored update a new CommitTsNs, so a stamp's event never compared equal outside the RemoteEntry and a touched file kept being restamped against a real filer. The test now assigns one per update. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
23598afb31 |
volume: bound each replica forward so a silent peer cannot hold a write (#12417)
* volume: bound replica forwards so a silent peer cannot hold a write The Rust volume server forwards each replicated write to its peers with the shared outgoing reqwest client, which had no connect or request timeout. When a peer is silently partitioned (packets dropped, no reset) the forward waits until TCP gives up, minutes later, and the write waits with it. A chaos test (replication 001, one volume server partitioned) saw every write on a volume with a copy there block. - Replica forwards go through replica_request, bounded at REPLICA_REQUEST_TIMEOUT = 10 s from connect to the response. - Both outgoing clients start from outgoing_client_builder, which adds a 5 s connect timeout, so new connections to a peer that never answers fail fast. The shared client gets no overall request timeout, since it also serves lookups and other transfers. Tests: a replica that receives the forward and never answers fails the whole replicated write at the bound, naming that replica; a silent peer fails the request builder at the bound on a paused clock; a connect that stalls in the resolver fails at the 5 s connect bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: give a replica forward more time for a larger body The 10 s replica forward bound runs from connect to the end of the response, so it also covers sending the body. A write near the 256 MB file size limit needs about 26 MB/s to reach its replica in 10 s, so on a link that is slower or busy the forward failed and the write returned 500 even though the peer was answering. A forward now gets REPLICA_REQUEST_TIMEOUT plus one second per MiB of body. A bodiless forward keeps the 10 s bound (a 1 MiB write gets 11 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: bound each Go replica forward like the Rust one The Go volume server forwards a replicated write with a 10 s dial bound and the transport's 30 s response header timeout. A replica that takes the forward and never answers held the write for 30 s, and a body larger than the socket buffers to a silently partitioned peer was held until TCP gave up, since nothing bounds sending the body. Each forward now runs under replicaRequestBound: 10 s plus one second per MiB of body, the same bound the Rust volume server puts on its forwards. Test: a replica that receives the forward and never answers fails the write at the bound (300 ms in the test); without it the write fails at 30 s on the response header timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: keep proxy variables out of the stalled-connect test reqwest reads HTTP_PROXY and ALL_PROXY by default, so with a proxy set the test's request went to the proxy instead of the stalled resolver. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: count a replica forward that hit its bound as a timeout DistributedOperationResult joined the replica errors into a plain string, and the uploader reports a cancelled request without wrapping the context error, so a forward cut off at replicaRequestBound was counted under server_error in the replication failure metric, as was a response header timeout before it. The joined error now unwraps to each replica's error, and a forward that ran out of its bound reports context.DeadlineExceeded. Test: the silent replica write raises the timeout failure count by one; with the flattened join it stays put. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: test that a large Go replica forward gets its body's bound A replica answers after the bodiless bound but within the bound for a 1 MiB body, so the write fails if replicateTo drops the body scaling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: run the Rust whole-path replica test with a 1 MiB body The forward then has to wait the bound for its body, so the test fails if do_replicated_request stops passing the body length to the bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: pingqiu <pingqiu@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
aab947b65e |
s3api: answer success when a failed completion already committed (#12416)
* s3api: name the check that a completion already committed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: answer success when a failed completion already committed A finalize can fail after its commit: the filer stored the object and the call still reported an error. The client then got a 500 with an unknown outcome. When the failure may have left this attempt's object behind, check the key the way a retry would and answer success with the committed result when it holds this upload's object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: keep part data when the commit found is another attempt's The object carrying this upload ID can be a concurrent attempt's commit of the same upload, and its part selection can include entries this attempt counts as unused. Free those parts' data only when the committed chunks are this attempt's own; otherwise drop their metadata only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test the committed-completion read on versioned buckets Cover the latest-version read (version id and checksum in the answer), a suspended bucket whose delete marker is still current, and that the upload directory is removed without its data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test versioned recovery through the completion handler Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
fe1cf5263f |
volume: bound each FetchAndWriteNeedle replica write (#12415)
* seaweed-volume: bound each FetchAndWriteNeedle replica write A replica that accepted the connection and then went silent held the request until TCP gave up, because the shared outgoing client has no request timeout. Each replica write now fails after 30 s, from connect to the response status, matching the Go volume server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: bound each FetchAndWriteNeedle replica write The default replica path uploads through the guarded peer client, which bounds only the dial and has no response header timeout, and the filer calls FetchAndWriteNeedle with no deadline. A replica that accepted the connection and went silent held the request for minutes. Each replica write, retries included, now has 30 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: require the silent-peer test to fail at the deadline A connection error that came back early also passed, since the test only checked an upper bound. It now also requires the paused clock to have reached FETCH_AND_WRITE_REPLICA_TIMEOUT. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: grow the FetchAndWriteNeedle replica bound with the chunk The reqwest timeout includes sending the body, and a remote-cache chunk is a thousandth of the object once it passes the 1000-chunk cap, so a fixed 30 s would fail a large chunk on a link under ~35 MB/s. The bound is now 30 s plus 1 s per MiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: grow the FetchAndWriteNeedle replica bound with the chunk The context deadline includes sending the body, so a fixed 30 s would fail a large remote-cache chunk on a slow link. The bound is now 30 s plus 1 s per MiB, matching the Rust volume server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
247e010865 |
volume: redial a master that just dropped a registered stream after a second (#12420)
* volume: redial a master that just dropped a registered stream after a second A registered volume server that loses its master with no peer naming a leader waited a full pulse before redialing. A restarted master is usually back within a second, so its volumes stayed unregistered, and uncached lookups kept failing, for most of a pulse. For one pulse after a stream a leader had answered ends, redial on the one-second cap. A master still down after that is redialed at the pulse pace, so redials never come later than before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: redial a master that just dropped a registered stream after a second Mirror of the Go heartbeat change. The loop keeps when the last stream a leader had answered ended, which also stands in for the confirmed-leader-once flag (None until first registration), and redials on the one-second cap for a pulse after that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: test that a dropped registered stream records the loss A fake master confirms the stream and then ends it. The server starts as registered long ago, so only a recorded loss makes the redial come after a second instead of a pulse. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: record a lost stream only after its receiver has exited doHeartbeatWithRetry can return on a send error while the receiver is still handling the leader's first reply, so the deferred recorder could read the registered flag too early and keep an old loss time. Cancel the stream and wait for the receiver before reading it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: require the prompt heartbeat redial to wait out its second Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: fail the down master's dials at once in the redial test Windows takes about two seconds to refuse a connection to a closed loopback port, so the stream dial and the leader lookup each held the prompt case past its two-second bound. A listener that closes every connection fails both dials immediately on every platform. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
8cc3a8ae21 |
ec.rebuild: repair shards that fail a checksum scrub in complete volumes (#12410)
* ec.rebuild: rebuild shards that fail a checksum scrub, also in complete volumes ec.rebuild skipped every volume with all shards present, so a shard whose bytes no longer match the .ecsum sidecar stayed in place until some other shard went missing. With -checksum, ec.rebuild first scrubs each node's shards in checksum mode and takes the failing copies out of the plan; a shard left without a good copy counts as missing and is regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: remove the copies that failed the checksum once a good copy serves A shard regenerated on the rebuilder left the failing copy mounted on its old holder, so the volume kept a corrupt duplicate that a later dedup could keep instead of the good one. Remove each failing copy once the rebuilt shard is mounted, and right away when another holder already serves a copy that passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: test the checksum scrub against fake volume servers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: check the lock before removing a corrupt copy Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: print the checksum scrub details A malformed sidecar or a wholesale mismatch comes back only in the scrub details, with no broken shard, so the volume would look verified. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: fail the run when a corrupt duplicate cannot be removed For a volume whose only problem is a corrupt duplicate, the removal is the whole repair, so its failure has to reach the caller. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: test duplicate removal failures and a lost lock Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: fail the run when a node's checksum scrub fails The node's copies stay in the plan unverified, so a corrupt shard there is left in place; the rest of the cluster is still repaired. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: preview the corrupt-copy removals in dry-run Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: test a failed checksum scrub Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
fcf62db8d0 |
seaweed-volume: stage optional EC sidecar copies before replacing (#12403)
* seaweed-volume: stage optional EC sidecar copies before replacing VolumeEcShardsCopy opened the destination .vif/.ecsum with File::create and then streamed into it. A source without the file left a 0-byte .vif, wiped an existing .ecsum (turning bitrot protection off on that holder), and a failed stream removed the destination's existing copy. Stream optional copies into <file>.copying and rename into place only once the source sent bytes, as the Go volume server's writeToFile does for ignore_source_file_not_found. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * seaweed-volume: keep the drain doc comment on drain_copy_stream_to_file Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: stage the .vif copied during missing-index recovery fetch_ec_index_from_one_peer drained the optional .vif straight into the local file, so a peer without one left a 0-byte .vif (losing a local EC ratio) and a failed stream left a partial one. Stage it and rename only once the peer sent bytes, as Go's recovery does through writeToFile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: test optional copies against a stream that fails mid-transfer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: remove a staged optional copy when the RPC is cancelled A cancelled CopyFile handler drops its future at the stream await, so the cleanup after it never ran and the .copying file stayed behind. Go's writeToFile runs to completion and removes it. A guard now owns the staging cleanup and is disarmed only once the rename commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: test optional copies dropped mid-stream Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: cancel the staged-copy test only once the staging file exists tokio's File::create runs on the blocking pool, so a fixed timeout could drop the .ecsum copy before .copying existed and pass without exercising the cleanup guard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
85232f7414 |
seaweed-volume: run the preallocating max-volume test on a fake disk (#12423)
* seaweed-volume: run the preallocating max-volume test on a fake disk The test sized the volume limit from the real free space but the store subtracts the 1% floor first, so on a disk more than 95% full the second slot no longer fit and the count came out 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: start the preallocating max-volume test above the expected count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b74111ea2f |
s3vectors test: skip a query lease tag released between list and HEAD (#12409)
A query's lease tag is deleted by a background task after the query answers, so the encryption helper could list it and then get a 404 on HEAD. Skip only that case; any other missing object still fails. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Fable 5.1
|
48df16ea0d |
weed-worker: keep a vector index covering as its table grows (ANN phase 3) (#12414)
* seaweed-vector: build and extend the vector index weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the table was created with, sizes partitions from the row count, and extends an index with a new segment or a retrain; the core crate re-exports it and adds the probes a query spends. Tests cover coverage after build, append, retrain and compaction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector-cli: index build and index optimize Logs go to stderr so stdout stays the JSON a caller parses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_INDEX builds the index before the timed queries The cluster fixture hands out its catalog URL, the Makefile builds weed-vector-cli, and the perf suite builds the index after ingest and reports the build time and segment count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: probe a tenth of the index's partitions, more under a filter A query that names no probe count gets one from the index's partition count. Lance searches at least that many partitions, and under a prefilter keeps going until it has k rows; without one it stops there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: --nprobes overrides the probe policy Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that a build takes the table's metric Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make clean removes weed-vector-cli Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: measure recall with vectors the table does not hold Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep the read cache's doc comment on with_read_cache Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold Detection keeps the tables a vector bucket created, with at least min_rows rows (50,000 by default) and no index on their vector column. Execution builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: optimize only the indices on the vector column Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: resolve the query metric with index::distance_metric Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test the L2 metric under the name CreateIndex stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: build the index through the admin server and the Lance worker VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new test sets the job's threshold, runs it through the plugin API, checks coverage and recall, and that a row written after the build is found. CI gains the worker leg for both volume servers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: read an index's partitions, and when a table has outgrown them Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: probes from the shared partition reader, partitions in coverage Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index reindexes at a tenth of its rows and retrains once the table outgrows it lance_optimize_indices proposes a vector table once its uncovered rows pass min(the absolute budget, a tenth of the indexed rows), and trains the index again instead of extending it when sqrt(rows) has reached twice its partitions. The end-to-end test covers both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: open a table inside a caller's Lance session Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep opened indices between queries in one session Every open used a fresh Lance session, so the index and metadata caches died with the request and each query loaded the IVF index again. --index-cache-mb sizes the shared cache, 512 MiB by default. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the index test draws embedding-like vectors Independent Gaussian vectors have no locality for an IVF index to use, so recall through one only tracks the fraction of partitions probed; points around shared centres behave like embeddings. The test must also run before the one that stops the engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the optimize and retrain steps draw embedding-like vectors Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that compaction drops the deleted key from tied hits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test that an append-only optimize serves the new row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: record PERF_NOISE in the perf report config Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: a filtered query widens to four times the probes, not without bound With no upper bound Lance prepares every partition for the late search even when the minimum probes already hold k rows, and a filtered query through the index cost as much as the exact scan. A query that names its own probe count gets exactly that count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: report recall through an index without asserting it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a hit tying the k-th best score toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: lance_vector_index runs in the Lance lane Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: check a table's index before counting its rows Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make build and clean cover the worker leg's binaries Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a retrain rebuilds the index with partitions for the row count Lance's OptimizeOptions::retrain does not size the partitions: it left a 6,000-row index with one. The retrain goes through the builder with sqrt(rows) partitions, for the job and for weed-vector-cli index optimize --retrain alike. The end-to-end test no longer expects a merge to keep the partition count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: allow float32 rounding at the recall cutoff Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a repeated hit key once toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: credit cutoff ties only up to the slots the top-k leaves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: assert every filtered hit passes the filter, indexed or not Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: coverage over an index's names, one name extended, a merge at the delta limit A sharded build leaves one name per shard, each missing the others' fragments, so coverage is the union of every index on the column and an optimize gives new rows to the name with the fewest deltas only. At MAX_DELTAS the policy passes OptimizeOptions::merge(deltas); Lance's default options may append instead of merging. A retrain keeps an index's own metric and rebuilds only IVF_FLAT indices. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: the optimize job measures and extends a vector table's indices together Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: tests for bounded deltas, disjoint shards and a retrain's metric Nine append-and-optimize cycles give delta counts 2 to 8 and then 1; two names on disjoint fragments stay disjoint through an optimize; a cosine index on a table without metric metadata is still cosine after a retrain. The end-to-end test expects the appended delta after the first optimize. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: forget the shared session when a table was dropped and created again The session keys index lists and transactions by dataset version, and a table created again at the same location restarts its versions. An open that finds an older version than the newest seen at that URI, or the same version committed later, replaces the session and opens again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: test an open after another engine created the table again Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: retrain only a single IVF_FLAT vector index name The job compared each shard name's partitions with the whole table's rows, so a sharded index was always due, and a non-IVF_FLAT index made optimize bail on every run before the new rows were indexed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: refuse to retrain a vector index sharded over several names A rebuild per name over the whole table left every shard covering every fragment, so a query searched each row once per shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test which vector indices a retrain is due for Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: replace the session in the same lock that records a recreated table A second open of the recreated table could see the new version recorded and the old session still shared, and keep the dropped table's index list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: skip a vector table whose rows cannot be counted Detection returned on the first table whose count_rows failed, so one unreadable table kept every other table from being proposed. It now skips that table like an open or index failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: panel for vector indexes the Lance worker built Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: size default probes from the largest index on the column A sharded build gives the column one index name per shard. The first name alone could be a small shard, and its probe count would then be spent on a larger one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: skip the probe lookup when a query is given no segments Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: open every table in its own Lance session again One session shared by every open keeps a table's index loaded between queries, but Lance keys what it caches by location and version, so a table dropped and created again at the same location could be served the old table's index list. Telling the two apart from version and commit time needs a map that only grows, a retry loop and clocks that agree across engines. A private session per open, as before, has no such state. Measured at 50k x 384 through an index, eight clients: 164 queries/s with the shared session, 137 without; one client is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: reject --nprobes 0 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: read the partition count from the opened index Lance's index statistics took 3.6 ms a query on a 200k-row table with 447 partitions. Opening the index and asking it takes 0.4 ms, and the scan that follows finds the index already open. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * ci: give the S3 Vectors worker legs the vendored protoc The worker workspace has no protoc of its own for lance's build scripts, so the worker legs failed at their first build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: give a vector index build an hour before the admin times it out The job set no execution timeout, so the admin's default of 90 seconds applied, and a build that trains on a large table would be reported as failed and retried while it was still running. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_vector_index proposes at most the admin's max_results Every qualifying table went into one message however many there were. The detector stops at the limit the request carries and sets has_more, as the Go detectors do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index job re-checks the table it was proposed for A table emptied since detection completes without a build, and one without a vector column fails instead of reporting it already indexed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_optimize_indices gets a build's execution timeout A retrain rebuilds the index over the whole table, which the admin's default of 90 seconds would cut off on a large one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index run window as long as one build An attempt is capped at the end of the job type's run window plus the drain grace, and the default window is 30 minutes, so a build could be cut off before its own hour was up. The window is an hour too, as the Go iceberg handler sets both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a lance_optimize_indices run window as long as one retrain Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: vector index proposals declare a build's runtime A run started by hand while the job type's schedule is off takes no timeout from the descriptor, only the admin's 90 seconds. The admin gives a job at least the runtime its proposal declares, so the build and the optimize proposals of a vector table declare the hour. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: coverage reports no partitions for an index of unknown coverage Reading the partition count goes through Lance's index statistics, which fail on an index that records no fragment bitmap, and the whole coverage report failed with them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: an index that records no fragments is not read as uncovered Its rows counted as uncovered, so detection proposed the table and the execution then failed in Lance's statistics. Coverage fails for such an index instead, and detection skips the table with a warning. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: coverage reads partition counts from the opened index Lance's index statistics rewrite the manifest of a table whose fragments carry no row counts, which a coverage report must not do and cannot do with read-only credentials. The report asks each opened segment instead, as the probe lookup does, and the statistics-based reader has no caller left. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Fable 5.1
|
e465880ee8 |
weed-worker: build the vector index of S3 Vectors tables past a row threshold (ANN phase 2) (#12413)
* seaweed-vector: build and extend the vector index weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the table was created with, sizes partitions from the row count, and extends an index with a new segment or a retrain; the core crate re-exports it and adds the probes a query spends. Tests cover coverage after build, append, retrain and compaction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector-cli: index build and index optimize Logs go to stderr so stdout stays the JSON a caller parses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_INDEX builds the index before the timed queries The cluster fixture hands out its catalog URL, the Makefile builds weed-vector-cli, and the perf suite builds the index after ingest and reports the build time and segment count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: probe a tenth of the index's partitions, more under a filter A query that names no probe count gets one from the index's partition count. Lance searches at least that many partitions, and under a prefilter keeps going until it has k rows; without one it stops there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: --nprobes overrides the probe policy Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that a build takes the table's metric Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make clean removes weed-vector-cli Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: measure recall with vectors the table does not hold Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep the read cache's doc comment on with_read_cache Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold Detection keeps the tables a vector bucket created, with at least min_rows rows (50,000 by default) and no index on their vector column. Execution builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: optimize only the indices on the vector column Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: resolve the query metric with index::distance_metric Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test the L2 metric under the name CreateIndex stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: build the index through the admin server and the Lance worker VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new test sets the job's threshold, runs it through the plugin API, checks coverage and recall, and that a row written after the build is found. CI gains the worker leg for both volume servers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: open a table inside a caller's Lance session Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep opened indices between queries in one session Every open used a fresh Lance session, so the index and metadata caches died with the request and each query loaded the IVF index again. --index-cache-mb sizes the shared cache, 512 MiB by default. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the index test draws embedding-like vectors Independent Gaussian vectors have no locality for an IVF index to use, so recall through one only tracks the fraction of partitions probed; points around shared centres behave like embeddings. The test must also run before the one that stops the engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that compaction drops the deleted key from tied hits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test that an append-only optimize serves the new row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: record PERF_NOISE in the perf report config Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: a filtered query widens to four times the probes, not without bound With no upper bound Lance prepares every partition for the late search even when the minimum probes already hold k rows, and a filtered query through the index cost as much as the exact scan. A query that names its own probe count gets exactly that count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: report recall through an index without asserting it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a hit tying the k-th best score toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: lance_vector_index runs in the Lance lane Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: check a table's index before counting its rows Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make build and clean cover the worker leg's binaries Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: allow float32 rounding at the recall cutoff Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a repeated hit key once toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: credit cutoff ties only up to the slots the top-k leaves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: assert every filtered hit passes the filter, indexed or not Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: forget the shared session when a table was dropped and created again The session keys index lists and transactions by dataset version, and a table created again at the same location restarts its versions. An open that finds an older version than the newest seen at that URI, or the same version committed later, replaces the session and opens again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: test an open after another engine created the table again Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: replace the session in the same lock that records a recreated table A second open of the recreated table could see the new version recorded and the old session still shared, and keep the dropped table's index list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: skip a vector table whose rows cannot be counted Detection returned on the first table whose count_rows failed, so one unreadable table kept every other table from being proposed. It now skips that table like an open or index failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: panel for vector indexes the Lance worker built Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: size default probes from the largest index on the column A sharded build gives the column one index name per shard. The first name alone could be a small shard, and its probe count would then be spent on a larger one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: skip the probe lookup when a query is given no segments Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: open every table in its own Lance session again One session shared by every open keeps a table's index loaded between queries, but Lance keys what it caches by location and version, so a table dropped and created again at the same location could be served the old table's index list. Telling the two apart from version and commit time needs a map that only grows, a retry loop and clocks that agree across engines. A private session per open, as before, has no such state. Measured at 50k x 384 through an index, eight clients: 164 queries/s with the shared session, 137 without; one client is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: reject --nprobes 0 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: read the partition count from the opened index Lance's index statistics took 3.6 ms a query on a 200k-row table with 447 partitions. Opening the index and asking it takes 0.4 ms, and the scan that follows finds the index already open. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * ci: give the S3 Vectors worker legs the vendored protoc The worker workspace has no protoc of its own for lance's build scripts, so the worker legs failed at their first build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: give a vector index build an hour before the admin times it out The job set no execution timeout, so the admin's default of 90 seconds applied, and a build that trains on a large table would be reported as failed and retried while it was still running. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_vector_index proposes at most the admin's max_results Every qualifying table went into one message however many there were. The detector stops at the limit the request carries and sets has_more, as the Go detectors do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index job re-checks the table it was proposed for A table emptied since detection completes without a build, and one without a vector column fails instead of reporting it already indexed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index run window as long as one build An attempt is capped at the end of the job type's run window plus the drain grace, and the default window is 30 minutes, so a build could be cut off before its own hour was up. The window is an hour too, as the Go iceberg handler sets both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Fable 5.1
|
b5951c5c7c |
seaweed-vector: probe policy for indexed queries (ANN phase 1) (#12412)
* seaweed-vector: build and extend the vector index weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the table was created with, sizes partitions from the row count, and extends an index with a new segment or a retrain; the core crate re-exports it and adds the probes a query spends. Tests cover coverage after build, append, retrain and compaction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector-cli: index build and index optimize Logs go to stderr so stdout stays the JSON a caller parses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_INDEX builds the index before the timed queries The cluster fixture hands out its catalog URL, the Makefile builds weed-vector-cli, and the perf suite builds the index after ingest and reports the build time and segment count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: probe a tenth of the index's partitions, more under a filter A query that names no probe count gets one from the index's partition count. Lance searches at least that many partitions, and under a prefilter keeps going until it has k rows; without one it stops there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: --nprobes overrides the probe policy Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that a build takes the table's metric Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make clean removes weed-vector-cli Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: measure recall with vectors the table does not hold Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep the read cache's doc comment on with_read_cache Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: optimize only the indices on the vector column Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: resolve the query metric with index::distance_metric Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test the L2 metric under the name CreateIndex stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: open a table inside a caller's Lance session Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep opened indices between queries in one session Every open used a fresh Lance session, so the index and metadata caches died with the request and each query loaded the IVF index again. --index-cache-mb sizes the shared cache, 512 MiB by default. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that compaction drops the deleted key from tied hits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test that an append-only optimize serves the new row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: record PERF_NOISE in the perf report config Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: a filtered query widens to four times the probes, not without bound With no upper bound Lance prepares every partition for the late search even when the minimum probes already hold k rows, and a filtered query through the index cost as much as the exact scan. A query that names its own probe count gets exactly that count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: report recall through an index without asserting it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a hit tying the k-th best score toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: allow float32 rounding at the recall cutoff Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a repeated hit key once toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: credit cutoff ties only up to the slots the top-k leaves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: assert every filtered hit passes the filter, indexed or not Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: forget the shared session when a table was dropped and created again The session keys index lists and transactions by dataset version, and a table created again at the same location restarts its versions. An open that finds an older version than the newest seen at that URI, or the same version committed later, replaces the session and opens again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: test an open after another engine created the table again Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: replace the session in the same lock that records a recreated table A second open of the recreated table could see the new version recorded and the old session still shared, and keep the dropped table's index list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: size default probes from the largest index on the column A sharded build gives the column one index name per shard. The first name alone could be a small shard, and its probe count would then be spent on a larger one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: skip the probe lookup when a query is given no segments Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: open every table in its own Lance session again One session shared by every open keeps a table's index loaded between queries, but Lance keys what it caches by location and version, so a table dropped and created again at the same location could be served the old table's index list. Telling the two apart from version and commit time needs a map that only grows, a retry loop and clocks that agree across engines. A private session per open, as before, has no such state. Measured at 50k x 384 through an index, eight clients: 164 queries/s with the shared session, 137 without; one client is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: reject --nprobes 0 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: read the partition count from the opened index Lance's index statistics took 3.6 ms a query on a 200k-row table with 447 partitions. Opening the index and asking it takes 0.4 ms, and the scan that follows finds the index already open. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Fable 5.1
|
f0f0ba85ae |
seaweed-vector: build a vector index from the CLI and measure it in the perf suite (ANN phase 0) (#12411)
* seaweed-vector: build and extend the vector index weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the table was created with, sizes partitions from the row count, and extends an index with a new segment or a retrain; the core crate re-exports it and adds the probes a query spends. Tests cover coverage after build, append, retrain and compaction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector-cli: index build and index optimize Logs go to stderr so stdout stays the JSON a caller parses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_INDEX builds the index before the timed queries The cluster fixture hands out its catalog URL, the Makefile builds weed-vector-cli, and the perf suite builds the index after ingest and reports the build time and segment count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that a build takes the table's metric Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make clean removes weed-vector-cli Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: optimize only the indices on the vector column Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: resolve the query metric with index::distance_metric Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test the L2 metric under the name CreateIndex stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that compaction drops the deleted key from tied hits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test that an append-only optimize serves the new row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: record PERF_NOISE in the perf report config Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: report recall through an index without asserting it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a hit tying the k-th best score toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: allow float32 rounding at the recall cutoff Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a repeated hit key once toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: credit cutoff ties only up to the slots the top-k leaves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: assert every filtered hit passes the filter, indexed or not Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
8350f297af |
seaweed-vector: dev plan for ANN search over S3 Vectors indexes (#12408)
* seaweed-vector: dev plan for ANN search over S3 Vectors indexes Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: one probe count per query, a budget that can shrink Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: smaller deltas are searched in full only up to the probe count Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: drop the private issue reference Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: phase 1 probes use the Design count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: status after phases 0 to 3, and what the measurements changed Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: name the Enterprise repository, not the private one Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: the Enterprise repository by that name Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: a filtered query widens to four times the probes, not without bound Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: throughput through the index at 50k and 200k Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: union coverage over shard names, and an explicit merge at the delta limit Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: status table names branches, not pull requests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: maximum_nprobes is always four times the count Lance reads maximum_nprobes only under a prefilter, and the phase 1 policy sets it the same way for every query. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: the retrain follows the partition count, not a recorded row count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: phase 3 is open like the other phases Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: the retrain stops past 2,048 partitions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: the merge at the delta limit takes the name's delta count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |