mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-07 20:47:46 +01:00
b650303aee7ce4e41fa9a0e5346fb38de09a4127
16086
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
b650303aee |
S3 Vectors: weed-vector reads tables from the filers and volume servers (#12469)
* sw-client: the filer and volume client library The client the Enterprise repository's VFS and vector engine use, with the same API: filer gRPC for entries, listings, volume locations, assignments and remote caching; volume HTTP, plain or TLS, for needle bytes; chunk resolution for sparse, overlapping and manifest chunks; a location cache; and the write side. Enterprise-only protocol fields (license, conditional mutations, hard links, KV listing) are left out. Compiled against weed/pb/filer.proto like the other worker crates. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-lance-worker: reads straight from the filers and volume servers direct_read::DirectReads wraps the object store Lance opens so that entries, listings and bytes come from the filers (gRPC) and the volume servers (HTTP, signed with the read JWT), whatever the entry holds: inline content, chunk manifests, sparse or overlapping chunks, the filer's volume encryption, the gateway's SSE-S3 (sse.rs, with the KEK from security.toml or the filer) and objects tiered to remote storage. Writes still go to the inner store. The code and its tests are those of the Enterprise repository's engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: read tables from the filers and volume servers The engine asks the masters for the filers (and follows the list), takes jwt.signing.read.key, grpc.client, https.client and the SSE-S3 keys from security.toml, and opens every table with DirectReads innermost and the read cache outside it. --masters is required: the gateway finds the engine through them and the engine finds the filers. Writes still go to the gateway. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sse: chunks out of a manifest keep their own SSE-S3 keys A chunk expanded from a manifest carries its id as fid, so keying the per-chunk key cache by file_id gave every such chunk the first one's key and IV. Keyed by the chunk's id however it is carried. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: a silent filer is passed over like an unreachable one Each filer attempt is bounded, so a filer that accepts the connection and never answers does not hold the read or the startup lookup. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: carry the assignment's fsync to the volume upload The filer's storage rule may want a needle on disk before the volume answers; the assignment says so and the upload now asks for it, as the Go uploader does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: argument tests pass the required masters; connect timeout for filers Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: a failed read is tried once more against fresh locations A fetch that fails forgets the volume locations it was made against; the request now looks them up again and retries instead of failing on a volume that moved. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: the filer deadline bounds a listing page, not the listing Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: security.toml booleans read as viper reads them Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a filer the master lists badly does not hide the others Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: register with the masters only once ready to serve Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: the HTTP transport asks the volume to fsync when the assignment does write_synced fell back to the trait default, which refuses fsync, so every upload under a fsync storage rule failed before it was sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: https.client.ca alone vouches for volume servers With a CA configured, reqwest still trusted the system roots next to it; weed processes trust only the configured pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an empty volume range is read without a request A zero length underflowed the Range end, and a range past u64::MAX overflowed it; the first now returns no bytes, the second an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: a whole-needle answer to a ranged read is cut to the range A server or proxy that ignores Range answers 200 with the whole needle; the read returned its first bytes in place of the ones asked for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: a read size past u64::MAX reads to the end of the file offset + size overflowed before the clamp to the file size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an SSE-encrypted manifest is refused without a decoder Only filer-encrypted manifests were; an SSE one went to the protobuf decoder as ciphertext. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an empty write leaves the staged file size alone A zero-byte write past the end grew the size to its offset, as a zero-byte pwrite does not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: the chunk cache keeps no empty chunks They add no bytes, so nothing evicted them and the map grew with every distinct empty needle, even at capacity zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: flushing with a zero chunk size is an error, not a panic Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: an empty master list fails at startup "" split into one empty address that passed the check, and the engine then waited two minutes for a filer before a generic error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Fable 5.1
|
2fe480a6e2 |
S3 Vectors: the gateway resolves the table it sends to weed-vector (#12468)
* lance catalog: in-process DescribeTable for trusted callers DescribeTable and DescribeTableInternal resolve a table's location and storage options without the HTTP layer; the internal form runs the lookup through a trusted manager so the caller's catalog grants do not gate a resolution its own permission already authorized. Same code as the Enterprise repository. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector.proto: Table carries its location and storage options The gateway resolves the table it addresses, so the engine can open it without a describe round trip of its own. Both fields are optional; an empty location means the engine looks the table up as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: test the in-process describe Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: resolve the table before calling the engine The gateway describes the table in process, through the trusted catalog manager, and sends its location and storage options with every engine call. Storage options are vended for the caller's principal, as the Enterprise router does for queries. A catalog failure answers with the code the engine would have given. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: open a table at the location the gateway sent A request whose table carries a location is opened there, with the storage options the gateway vended over the engine's fallback; the catalog is described only for a table that comes without one, as from an older gateway. dataset::open_at and dataset::drop_at take a description the caller already holds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: run the engine without a catalog VECTOR_ENGINE_NO_CATALOG=1 gives the engine an unreachable catalog URL; every CI leg sets it, so the suite proves the gateway resolves each table it sends. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: an internal describe may write a vector index With credential vending on, the options vended to the engine followed the caller's s3tables grants, which never allow writing the vector namespace, so every write the gateway sent would have run with read-only credentials and reads would have lost their version lease. The engine is that namespace's writer and the caller's vector permission authorized the operation, so an internal describe of a vector table vends write-capable options; other namespaces keep the UpdateTable check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: resolve tables for the engine only while the catalog runs With -port.lance=0 the gateway has no catalog to resolve through; the engine then keeps looking tables up itself, as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: a deregistered table describes as NoSuchTable So the gateway answers NotFoundException for it, not InternalFailure. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: trust a sent location only for a colocated vector index A caller that reaches the engine could otherwise name any storage the engine's credentials cover. A location is taken from the request only when the id names a vector index and the location is its colocated path, which the engine writes already; anything else is described through the catalog as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: scope the read cache by storage endpoint The cache was keyed by bucket and path alone, so the same path reached through another endpoint would have shared entries. Each open wraps the store with the cache scoped to the endpoint its options name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep a configured endpoint over one a request names A caller that reaches the engine could otherwise send its fallback credentials to an endpoint of the caller's choosing. When the engine's --storage-option names an aws_endpoint, a table whose request names another one is described through the catalog instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: use request options only where credentials cannot leave A request's storage options are taken only when they hold the keys a gateway's describe produces, so a proxy or an endpoint alias cannot route the engine's credentials elsewhere, and an endpoint only when it is the configured one or the engine holds no credentials to send. Anything else sends the table to the catalog. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: give the engine its endpoint with its credentials Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a drop checks the request's options like an open Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: an unpinned endpoint needs the request to sign for itself Without configured credentials the S3 store still signs with the AWS default chain, so an endpoint from a request is taken only when the request skips signing or brings its own credentials. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a trailing slash does not unpin the endpoint, and say so A configured aws_endpoint written with a trailing slash matched nothing the gateway sends, so every table fell back to the catalog in silence. The comparison ignores the slash, and a fallback for a resolved table is logged at debug. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: resolved by the gateway, or refused The engine no longer has a catalog: --namespace and --filer-signing-key are gone, a request whose table names no location is InvalidArgument, and one whose location is not the index's own or whose storage options the engine's credentials must not be used with is PermissionDenied. A drop removes the colocated path it was given. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: the catalog resolves vector tables whether or not its port is served newLanceServer builds the Lance catalog for the S3 Vectors API; the listener on -port.lance is optional. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors tests: start the engine without a catalog Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: the aux listener test builds the catalog before serving it Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: an unpinned endpoint takes only credentials the request signs with alone Lance fills an option a request leaves out from the engine's environment, under any of object_store's names for it (TOKEN, SESSION_TOKEN, ...), and asks the AWS default chain when the key pair is incomplete. A request that named its own endpoint with only an access key id, or a key pair while the engine's environment held a session token, would have sent the engine's credentials there. Such a request now needs a full key pair, and its own session token whenever the environment would supply one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: accept --namespace and --filer-signing-key, ignored Both flags shipped in 4.52 and are documented, so dropping them made an existing command line exit at startup with an unexpected-argument error. They are kept hidden and ignored, with a warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors design: the engine is started without a catalog Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: the trust check takes the environment it reads The options test read the process environment through signs_alone, so a host exporting AWS_SESSION_TOKEN or TOKEN failed it. The caller now passes the variable names in; describe passes the engine's own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: log both endpoints when a request's options are refused The gateway turns the PermissionDenied into a bare AccessDenied and logs nothing, so a pinned aws_endpoint written in another form than the one the gateway sends (host case, a default port, an IP for a name) failed every call with no trace of which value differed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: the Lance catalog sets the credential role whether or not Iceberg runs -iceberg.credentialRole reached the S3 server only while the Iceberg listener started, so with -port.iceberg=0 the Lance catalog installed its vendor but vending stayed off, and the vector engine got no credentials for the tables the gateway resolves. Both catalogs now set the role through one helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: set the credential role once, before either catalog serves Both catalogs wrote the role as they were built; the second write landed while the first was already serving requests that read it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
7beac0e8ea |
admin: show disk usage on the dashboard instead of the data size (#12466)
* admin: show disk usage on the dashboard instead of the data size The dashboard's Total Size card summed the volume and EC shard sizes the volume servers report. That leaves out deleted needles waiting for vacuum, EC generations kept after a vacuum, and anything else on the disks, so on a cluster with a lot of EC vacuum activity it showed far less than the disks held. The card now shows the bytes in use on the volume servers' disks, the same statfs numbers the volume server list uses, with the disks' capacity, and its trend line follows that number. Servers that report no statfs numbers count their data size, as on the volume server list. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * admin: compute the disk usage percentage in float64 current*100 overflows int64 past about 92 PB of used space, so the dashboard card showed a negative percentage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: size the volume server disk bar with calculatePercent64 int(host.DiskUsage) truncates byte counts on 32-bit builds, so the bar width was wrong for any disk over 2 GB there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: test the cluster disk totals across statfs and fallback servers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
fd81a50c57 |
mount: keep retrying writeback chunk uploads through a volume outage (#12465)
* mount: keep retrying writeback chunk uploads through a volume outage With -writebackCache, close() returns before the data is uploaded. When the volume servers stay unreachable past the upload's own retry (~20 s), the chunk was dropped and the file never appeared, with no error to the application. Retry transient upload errors with the same bytes until they succeed, unless the dirty pages were discarded. Permanent errors and the synchronous mode keep failing as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test writeback upload retries and the cases that still fail Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * operation: keep the filer's read-only verdict on assign failures UploadWithRetry flattened AssignVolumeResponse errors into a string, so callers could not tell a read-only rule from an outage. Wrap the filer's coded error so errors.Is(err, filer_pb.ErrReadOnly) works; the message is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: retry writeback uploads the upload layer reassigns, never read-only A filer proxy that cannot reach the volume answers 500 with an empty body, which util.IsTransientError does not match, so that outage still dropped the chunk. Also retry what operation.ShouldReassignUpload takes. A read-only rule stays final even when its path or message contains a transient word such as "unavailable". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: stop a writeback upload once its dirty pages are discarded Destroy() was only seen after a failed attempt, so a truncate during the retry wait still started another upload, and a chunk that finished after its pages were replaced was attached to the new content. Check before each attempt, and drop the chunk instead of attaching it; volume.fsck reclaims it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep Destroy from slipping between the discard check and the append truncateEntry destroys the pages and then installs the new entry, so a Destroy landing after the check but before AddChunks put the old chunk into the replacing entry. Hold one lock across the check and the append, and take it in Destroy while setting the flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: leave the Windows interrupt drain to the capped filesystem hook Interrupt hooks run in reverse registration order, so the Windows hook ran before the filesystem hook and waited on pending writeback flushes without a bound. With writeback uploads now retrying through an outage, Ctrl-C could wait until the volume servers came back. Unmount only, as on Linux, and let the filesystem hook drain for at most 30 s and keep the write cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test a writeback upload whose assign keeps timing out With every volume server gone the assign RPC times out until the filer client gives up; that error must be retried like a refused upload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
github-actions[bot]
|
7d6680c7a4 | docs: regenerate star history chart | ||
|
Chris LuandClaude Opus 5.5
|
99539484b6 |
grafana: chart the remote-deletion tombstone gauges (#12444)
* grafana: chart the remote-deletion tombstone gauges Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: describe the fetch hold that outlasts a full tombstone table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: word the tombstone panel description like the gauge help Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
1c4fe61fcd |
master: stop the resumed single-master leader from checking quorum every 1 ms (#12463)
* deps: bump seaweedfs/raft so the leader quorum check follows the election timeout A single master resuming raft state self-elects on a 1 ms election timeout and restores the configured one afterwards, but the leader kept running its quorum check every 1 ms for the rest of the process. The raft change re-arms that check from the current timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * deps: bump seaweedfs/raft to the raft PR head with the extended test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * deps: bump seaweedfs/raft for the non-positive election timeout guard Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * deps: bump seaweedfs/raft so a timeout change re-arms the leader quorum check Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * deps: pin seaweedfs/raft v1.2.4 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
cdea7e79e0 |
filer: fold small chunks into manifests 256 at a time (#12462)
* filer: fold small chunks into manifests 256 at a time Every metadata event carries the file's whole chunk list, so a file grown by small appends re-sent each chunk record on every append until the list reached ManifestBatch (10000). Chunks under 1 MiB now fold once 256 of them build up; full-size chunks still fold at ManifestBatch, so large files keep their flat lists and read without a manifest fetch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: merge appended small chunks at the small-chunk fold count The append path merges small chunks into full-size ones, but only at 1000 of them; the fold now hides small chunks in a manifest at 256, so the merge would never see enough of them. Merging at the same count keeps rewriting them into full chunks first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
728b476ba2 |
filer.remote.gateway: keep the remote object while a version under a delete marker is locked (#12459)
* filer.remote.gateway: keep the remote object while a version under a delete marker is locked Every version of a versioned bucket uploads to the object's key, so on a remote without versioning that object is the only remote copy of the latest version. A delete marker deleted it even when a version still held a legal hold or unexpired retention. Keep it while any version of the key is locked, the same rule the filer uses before pruning a remote entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway, filer.remote.sync: say that earlier versions need remote versioning Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: delete a kept delete marker's object once the version lock is released A delete marker kept for a locked version stays unstamped. When an update or delete leaves a version that carried a legal hold or retention without an active lock, look up the key's latest version and, if it is still that unstamped marker, run the delete again. Only changes to versions that carried a lock trigger it, so the noncurrent mark and tag updates of unlocked versions do not send repeat remote deletes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: test the kept delete marker retry through the bucket mount The retry reads the key's versions through a filer client parameter, as the rename helpers do, so a test can check the mapped object key and the marker stamp, and that lock-releasing updates and deletes reach it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: check the lock release lookup goes to the gateway's filer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
8c22336944 |
filer.meta.backup: stream from the start of an unfinished walk (#12458)
* filer.meta.backup: stream from the start of an unfinished walk A walk only inserts entries, so when the first walk is interrupted the next start walks again and streams from the second walk's start: deletes and renames between the two walks never reach the backup store. Keep the first walk's start time in the store until a walk completes and stream from it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.meta.backup: start fresh once the stream has passed an earlier walk's start A walk start left in the store by a crash between saving the offset and clearing it, or by an interrupted -restart followed by a plain restart, stops mattering once the stream saves an offset past it. Ignore it then instead of replaying from it on the next walk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.meta.backup: fail the walk on an unreadable offset Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
jsasandjames
|
5ba97b4432 |
filer: a full remote-deletion tombstone table keeps the lazy fetch off (#12440)
* filer: a full remote-deletion tombstone table keeps the lazy fetch off At the cap a new delete under a remote mount was dropped with a log line and isRemoteDeletionPending answered false for it, so the lazy remote fetch resurrected deleted paths. While the table is full every path may be a pending delete: isRemoteDeletionPending answers true until the table has room again. The size and the full state are gauges (filer_remote_deletion_tombstones, filer_remote_deletion_tombstones_full). (cherry picked from commit f3be0ab965cde3bfd3053d3ac720fbbe3f68e98b) * filer: a full tombstone table drains on reads and lifts after the unrecorded deletes age out While the table is full, a lookup miss evicts expired tombstones and those the write-back offset has passed, so the table drains without a mutation. The full state lifts only once the table has room and the newest unrecorded delete is past the TTL: an unrecorded delete has no event to confirm it, so only the TTL makes it safe, as for a pending tombstone. (cherry picked from commit 19394fe5d67798af7b0d755ac18c83c75e20a12b) * filer: a full tombstone table drains per mount, only when its offset advances Offset-based release applies to the tombstones under the mount whose offset was read; another mount's tombstones answer to their own offset. A read sweeps the table only once that offset advanced since the last drain for the mount or the unrecorded deletes aged past the TTL, so a listing of many missing children does not rescan the table per child. (cherry picked from commit 9cf00e2e54d70e6ba530e6ec1b1ff4b502873847) * filer: the tombstones-full gauge help names when it clears --------- Co-authored-by: james <james@medable.com> |
||
|
|
7fc33e7fe3 |
filer: no lazy remote fetch inside a store transaction (#12439)
* filer: no lazy remote fetch inside a store transaction A rename onto an existing target deletes the target inside the store transaction, then CreateEntry's FindEntry misses the row and runs the lazy remote fetch, which persists the archive copy on a second connection. The store wrapper drops that call's deadline, so the UPDATE waits without bound on the row lock of the transaction's own uncommitted DELETE while the rename goroutine holds the path's exclusive locks: every later lookup, rename and delete of the path on that filer waits forever. A lookup whose context carries the store transaction skips the fetch. (cherry picked from commit b69e9535c2597d0c8e856f031d3778fddfd4159c) * filer: materialize remote-only rename paths before the transaction With the lazy fetch off inside a store transaction, a source or target the filer holds only on the remote would read as absent to AtomicRenameEntry and StreamRenameEntry: a rename with FailIfTargetExists could overwrite it, and a remote-only source would skip the deferred copy. PrefetchRemoteEntries runs the fetch for both paths before BeginTransaction, under the path locks. (cherry picked from commit e61f9718018669ef59768cd5eda25b6466d69597) * filer: skip the lazy remote fetch only in CreateEntry's lookup inside a transaction The guard on every lookup inside a store transaction hid a source or target the filer holds only on the remote from the rename, and the prefetch before the transaction did not reach the children of a directory rename. The deadlock needs a write: CreateEntry's lookup after the transaction deleted the path. That lookup alone skips the fetch; reads inside the transaction keep it, and their persist on a second connection holds no lock the transaction waits for. (cherry picked from commit f555092c324780a8fe5314b238694551e3ef6be5) * filer: mark a store transaction independently of the store's own key ArangoDB and FoundationDB keep their transaction under other context keys, so the CreateEntry guard keyed on "tx" missed them. The filer now marks every context returned by BeginTransaction and the guard checks that marker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit |
||
|
Chris LuandClaude Opus 5.5
|
86db60b669 |
s3api: mint a versioned PUT's id at commit, after the lock wait and the upload (#12454)
* s3api: mint a versioned PUT's id after the object write lock putVersionedObject minted the version id before the upload and the lock wait. A DELETE that committed while the PUT waited got a newer id, so a conditional PUT whose If-None-Match passed against the delete marker was stored behind it: accepted, but GET returned NoSuchKey. The finalize now re-mints the id once the lock is held and the precondition passed, so the version orders after everything the check saw. An ambiguous routed attempt keeps its id, since recovery looks for that entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: mint a routed versioned PUT's id after the upload On the routed path the version id was still the one minted before the upload, and RECOMPUTE_LATEST picks the newest name. A DELETE that committed during a long upload left its marker latest over the accepted PUT. Re-mint right before the routed transaction, as the lock path does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1176f975b3 |
filer: release metadata log chunks a failed append leaves unreferenced (#12455)
* filer: stop unbounded metadata-log growth on append-only files Two independent leaks make a filer metadata log grow far faster than the data it describes. On a cluster whose live logs weigh a few hundred megabytes, they produced tens of gigabytes of metadata per day. 1. Metadata events embed the whole old and new entry, chunk lists included. An append-only file — a log or a journal written in small pieces — therefore re-serializes every chunk it ever had on every append, so the cost is quadratic in the number of chunks. A 2,700-chunk file emits a couple of megabytes per append. Cap the carried chunk list at 256 entries and set metadata_event_chunks_truncated so subscribers that need the content read the entry from the filer instead of trusting the event. Subscribers that only track names, attributes, or chunk deltas keep working unchanged. 2. appendToFile uploads the chunk before committing the filer entry. When the commit fails — store error, cancelled context, filer shutting down — the chunk is stranded: no entry references it, so no later delete can mark it, and neither vacuum nor fsck can see it because both work from the entry index. The volume server keeps counting those bytes as live data forever. Release the chunk when the commit fails. * filer: keep chunk lists in metadata events Mount meta caches, peer filers replaying into their own store, and the filer.sync, filer.backup and filer.meta.backup sinks all store the event's entry as the file. Dropping the chunk list past a count left those copies of a large file without content. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: release a metadata log chunk the lookup or commit leaves unreferenced The flush retries a failed append with a new upload, so a store outage left one stranded chunk per retry. A failed lookup strands the chunk too. Use DeleteUncommittedChunks, as the other write paths do: the chunk was never part of the file, so path chunk-deletion rules do not apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: keep a metadata log chunk the failed create stored CreateEntry inserts a new entry before its parent folders, so a parent failure returns an error with the entry already referencing the chunk. Release it only when the store shows no entry, or an entry without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: count a metadata log piece the failed create stored as appended The flush retry would append the same bytes after it, and a piece cut inside a record then breaks replay of that segment. The retry takes the update path, which does not create parent folders either, so returning the error only duplicated the piece. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Martin Morez <martin@morez.org> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
2be4498c04 |
s3 lifecycle: walker deletes carry the listed identity and must target the latest version (#12452)
* s3 lifecycle: send the listed entry's identity on walker deletes The daily walker dispatched LifecycleDelete with no ExpectedIdentity, so the server skipped its compare-and-swap and an object rewritten between listing and dispatch was deleted. The list adapter now records each listed entry's identity, computed the same way as on the event path, and the walker sends it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: current-version expiration must target the latest version The walker addresses the latest version of a versioned key by id. A version file never changes, so its identity still matches after a newer PUT, and the delete went ahead: a delete marker over the new version with versioning enabled, or the new null version deleted while suspended. LifecycleDelete now resolves the key's latest version for EXPIRATION_DAYS/EXPIRATION_DATE requests that carry a version id and answers NOOP_RESOLVED (VERSION_NOT_LATEST) when it is a different one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: check the latest version for unaddressed suspended expirations With versioning suspended, an expiration without a version id reads the plain entry, which is the null version but not necessarily the latest: a newer real version can sit in .versions. The latest-version check now runs in the expiration branch, by versioning state: for every suspended request (an unaddressed one addresses the null version) and for enabled requests that carry a version id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test that each version sibling carries its own identity Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
9af736da62 |
volume: stop serving EC reads from quarantined shards (#12450)
* storage: skip quarantined EC shards on local reads A needle read tried the local shard first even when its disk was quarantined after repeated IO errors, so every read paid the failing disk's EIO latency before falling back. Treat such a shard as not local so the read goes to peers or reconstruction, and read it only as a last resort when no healthy copy answers. Scrub still reads it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: refuse EC shard reads from a quarantined disk VolumeEcShardRead served peers from a shard whose disk was quarantined, and it ignored the .ecx lookup error: a failed index read returns a tombstone size, so the peer was told a live needle was deleted. Refuse quarantined shards and fail the request on an index read error; a needle missing from the index still reads, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: report quarantined EC shards deleted right away Quarantining an EC volume only dropped it from later full EC heartbeats, so the master kept routing reads to the faulty disk for up to 17 heartbeat intervals. The heartbeat that quarantines the volume now carries its shards as deleted, once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: do not announce EC shards mounted onto a quarantined volume MountEcShards and the shard load notifier reused a quarantined EcVolume and announced the new shard, pointing the master back at the faulty disk until the next full EC heartbeat. Skip the announcement, and fail the mount so a caller moving a shard here keeps its source copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: count IO errors on local EC shard reads Go's readLocalEcShardInterval feeds every local shard read into the EC volume's IO error streak; the Rust needle read and recovery gather did not, so a failing disk under local reads never moved toward quarantine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: skip quarantined EC shards on local reads Mirror the Go read path: a needle read and the recovery gather treat a shard on a quarantined disk as not local, and read it only as a last resort once peers and reconstruction fail. Scrub still reads it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse EC shard reads from a quarantined disk Mirror Go's VolumeEcShardRead: answer Unavailable for a shard whose disk is quarantined, so the peer tries the next holder or reconstructs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: report quarantined EC shards deleted right away Mirror Go: the volume heartbeat that quarantines an EC volume carries its shards as deleted, once, instead of leaving the master to drop them at the next full EC heartbeat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: fail EC shard mounts onto a quarantined volume Mirror Go's MountEcShards: the heartbeat already leaves such a shard unannounced, so fail the mount and let a caller moving the shard here keep its source copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: keep reading EC needles when the shard lookup fails A failed or short master lookup aborted the read before any shard was tried, so the quarantined-disk last resort never ran once the master had dropped those shards, and local shards went unread whenever the master was unreachable. Read on with the cached locations, as the Rust server already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: log a failed last-resort read of a quarantined shard When reconstruction and the quarantined read both fail, say why the local read failed too. Go and Rust. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: drop a shard whose mount was refused on a quarantined disk The refused mount left the new shard loaded on the quarantined volume, where it was neither served nor announced, and the in-memory claim drew later copies of that shard back to the same disk. Unload it again, keeping a shard that was already mounted there. Go and Rust. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: finish a mount batch before reporting a failed shard A shard that failed to mount, now including one refused on a quarantined disk, returned before the sidecar reload, and in Rust before the heartbeat notify, so shards mounted earlier in the same batch kept checksum protection off and reached the master only at the next full EC heartbeat. Stop at the failed shard but finish the rest first. Go and Rust. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: keep a peer's deleted answer when its stream fails after it A shard read recorded the holder's deleted answer, and a stream error after it failed the read: Rust discarded the flag, and Go let the next source or the reconstruction result overwrite it. With the last-resort read of a quarantined shard, that could serve a needle a peer had reported deleted. Once the deleted answer is in, end the read on a later stream error instead of failing it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: use the platform media error in the quarantine heartbeat test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: rebuild EC intervals from quarantined shards when healthy ones fall short Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: rebuild EC intervals from quarantined shards when healthy ones fall short Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: serve a peer's EC shard read from a healthy copy on another disk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: serve a peer's EC shard read from a healthy copy on another disk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a quarantined EC shard read instead of serving a sibling copy A sibling disk's copy can have its own deletion journal, and an EC blob delete is journaled on one local copy only, so the sibling could serve a deleted needle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: refuse a quarantined EC shard read instead of serving a sibling copy A sibling disk's copy can have its own deletion journal, and VolumeEcBlobDelete journals a delete on one local copy only, so the sibling could serve a deleted needle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: check the disk an EC shard mounted on for quarantine mount_ec_shard now returns the disk it mounted on, and the mount RPC checks only that disk, like Go's MountEcShards. Before, a quarantined sibling holding the same shard failed the mount while the healthy disk's copy stayed mounted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: test an EC mount beside a quarantined sibling Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: check the quarantined-sibling mount test by its result A server that remounts a shard on the disk already claiming it refuses the mount there instead. The test now only requires that the result matches whether the healthy disk ended up holding the shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: check the quarantined-sibling mount test by its result Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: expect the quarantined-sibling mount result from the disk the store picks The test accepted a refused mount whenever the healthy disk stayed empty, so a handler that refused before mounting passed too. Ask the store which disk it picks first, then require success exactly when that is the healthy disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: expect the quarantined-sibling mount result from the disk the store picks Mirrors the Go test. add_location's cross-disk reconcile mounted shard 1 on the healthy disk too, so the test now unmounts it there first, leaving the quarantined sibling as the only claimant, as in Go. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2df7d435f2 |
filer.remote.sync: bound the stamp and offset writes, startup offset fixes, stamp a copy (#12447)
* filer.remote.sync: bounded filer lookups, one chunk cache per process The filer lookups on the sync path (currentEntry, isSuperseded, uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no deadline. A lookup that never answers keeps its job alive, the job keeps its path in the MetadataProcessor conflict index, AddSyncJob waits on the condition and the daemon stops consuming events with no log line. Each lookup now carries a 30 s deadline, so the job fails and the processor's stop-and-resubscribe path runs. remoteUploadReader built a chunk cache per upload; the cache's worker goroutine is never stopped, so each upload leaked one goroutine and its cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory limit). One chunk cache per process. (cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78) * filer.remote.sync: a failed mount lookup at startup never resolves to now collectLastSyncOffset returned time.Now() when the mounted directory lookup failed, which skips every event since the saved offset. The saved offset is used instead; without one the daemon exits so a restart retries. (cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae) * filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read retriedWriteFile returns the supersession lookup's error instead of treating an unanswered lookup as "not superseded": in a replay an older snapshot could otherwise overwrite a newer remote object. The processor retries the event. isSuperseded keeps its boolean form for the other call sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls back to the saved offset cannot hang on it. TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed lookup; it now asserts the withheld upload and the lookup error. (cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa) * filer.remote.sync test: keep the slices import the other gateway tests use * filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo After a failed upload attempt the supersession lookup's error ended in isSuperseded's boolean and the loop could upload the snapshot again unverified. The lookup error now ends the loop (errLookupUnanswered, not retried) and the processor retries the event. With -timeAgo set, a failed read of the saved offset resolved to now minus timeAgo, skipping everything older; the daemon exits instead so a restart retries. (cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566) * filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting resolveLastSyncOffset returns the lookup or offset-read error; both callers (directory and buckets) hand it to util.RetryUntil, which retries with backoff instead of the process exiting. * filer.remote.sync: keep doc comments on the functions they describe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: isSuperseded returns the lookup error itself Its only production callers needed the error, so the boolean wrapper was left to the tests. The tests now assert the error on a failed lookup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: bound the stamp written after each upload updateLocalEntry's UpdateEntry was the last filer call on the sync path without a deadline. A filer that never answers the stamp keeps the job and its path in the conflict index the same way an unanswered lookup did. The stamp carries IF_ENTRY_EQUAL, so a retried event after a timeout either stamps again or skips a stamp that already landed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: an unread offset never resolves to the mount's creation time With the offset read now bounded, a timed-out read fell through to the directory's creation time and replayed every event already synced. The error goes back to the startup retry loop, as with -timeAgo; a missing offset still reads as zero without an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: stamp a copy of the event entry updateLocalEntry set the RemoteEntry and dropped the rename marker on the caller's entry before the RPC. A transient stamp failure retries the same event in place, which then read as already synced and skipped both the upload and the stamp. The stub filer now applies a successful update the way the filer does, since tests had relied on that shared object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: a directory not created yet starts now, as before A gateway started before the first bucket looked up a missing buckets directory, had no saved offset, and returned an error to the retry loop on every attempt until a bucket appeared. A missing directory with no saved offset has nothing to replay, so it starts now as it did before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: take the start time before the mount lookup A bucket created between a not-found lookup and the end of the offset read had its creation before a start taken afterwards, so the gateway never created its remote bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: save the resume position when a subscription ends A failed event ends the subscription with ErrResubscribe, and the retry loop runs startup again, recomputing -timeAgo from a later now. The periodic checkpoint only runs as events arrive, so a failure before the first one could start the retry after the event it failed on. The resume position is now saved whenever the subscription returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * remote_storage: bound the sync offset write like the read The offset write runs in the subscription's checkpoint callback and, now, when a subscription ends; a filer that never answers it stalled either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: say that any saved offset wins over the creation time The creation time in seconds was compared with the offset in milliseconds, which made any saved offset win. That is the behaviour to keep: remote.unmount recreates the directory and then saves the offset, usually in the same second, so a seconds comparison would start at the creation second and replay the purge. The comparison now states it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: keep the resume position for the retry when its write fails The retry in the same process now resumes from the position the ended subscription returned, as filer.backup does, so a failed offset write followed by an empty read cannot recompute -timeAgo past a failed event. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync test: give the same-second saved offset a fractional part With whole seconds, returning the creation second resolved to the same time and passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: james <james@medable.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
206d7b1418 |
filer.remote.sync: bounded filer lookups, one chunk cache per process (#12441)
* filer.remote.sync: bounded filer lookups, one chunk cache per process The filer lookups on the sync path (currentEntry, isSuperseded, uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no deadline. A lookup that never answers keeps its job alive, the job keeps its path in the MetadataProcessor conflict index, AddSyncJob waits on the condition and the daemon stops consuming events with no log line. Each lookup now carries a 30 s deadline, so the job fails and the processor's stop-and-resubscribe path runs. remoteUploadReader built a chunk cache per upload; the cache's worker goroutine is never stopped, so each upload leaked one goroutine and its cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory limit). One chunk cache per process. (cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78) * filer.remote.sync: a failed mount lookup at startup never resolves to now collectLastSyncOffset returned time.Now() when the mounted directory lookup failed, which skips every event since the saved offset. The saved offset is used instead; without one the daemon exits so a restart retries. (cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae) * filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read retriedWriteFile returns the supersession lookup's error instead of treating an unanswered lookup as "not superseded": in a replay an older snapshot could otherwise overwrite a newer remote object. The processor retries the event. isSuperseded keeps its boolean form for the other call sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls back to the saved offset cannot hang on it. TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed lookup; it now asserts the withheld upload and the lookup error. (cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa) * filer.remote.sync test: keep the slices import the other gateway tests use * filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo After a failed upload attempt the supersession lookup's error ended in isSuperseded's boolean and the loop could upload the snapshot again unverified. The lookup error now ends the loop (errLookupUnanswered, not retried) and the processor retries the event. With -timeAgo set, a failed read of the saved offset resolved to now minus timeAgo, skipping everything older; the daemon exits instead so a restart retries. (cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566) * filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting resolveLastSyncOffset returns the lookup or offset-read error; both callers (directory and buckets) hand it to util.RetryUntil, which retries with backoff instead of the process exiting. * filer.remote.sync: keep doc comments on the functions they describe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit |
||
|
Chris LuandClaude Opus 5.5
|
1406335301 |
volume: a delete cancels writes of its needle still in flight (#12445)
* volume: a delete cancels the writes of its needle still in flight A write queued behind the volume locks, for example behind another write's fsync on a busy disk, is not in the needle map yet. A delete of that needle in the meantime finds nothing, and the write lands afterwards, so the needle comes back on this copy only while the other replicas have it deleted. Each volume now tracks its writes from entry until they return. A delete cancels the ones for its needle, and a cancelled write that landed is tombstoned before it returns, unless the needle was deleted or rewritten since. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: a replicate delete that finds nothing cancels the pending write Rolling back a failed chunk upload sends DELETE ?type=replicate to every holder, including the slow one whose write timed out on the client. That holder has not indexed the write yet, so the delete answered 404 and the write landed live afterwards. Cancel the needle's pending writes, then look again in case one finished in between. A plain delete that finds nothing still answers 404 without cancelling: it is not sent on to the replicas. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: a delete cancels the writes of its needle still in flight Mirrors the Go volume. Volume::track_pending_write registers a write until its guard drops, delete_needle cancels the needle's pending writes, and settle_pending_write tombstones a cancelled write that landed unless the needle was deleted or rewritten since. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: track uploads and cancel them on a replicate delete An upload is tracked from the handler, and a durable one from submission to the write queue, where the guard travels with the request: a handler dropped while the write waits in the queue does not untrack it, and the worker settles it after the batch. A replicate delete that finds nothing cancels the needle's pending writes and looks again, like the Go handler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: take a cancelled write back in the lock span that publishes it The write tombstoned itself only after it returned, with the volume locks released. In that gap a compaction could move the record so the offset check skipped it, a close could drop it, and an identical write sent after the delete could match the live record and report success before the tombstone removed it. The check now runs right after the publish, under the same dataFileSyncLock and dataFileAccessLock hold, in the plain, fsync and batch write paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: a delete cancels only the writes carrying its cookie A delete with the right needle id but the wrong cookie could cancel an in-flight write and get it tombstoned. Each pending write now records its cookie, and a cancel only marks the matching ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: take a cancelled write back in the lock span that publishes it Mirror the Go change: settle_pending_write ran after write_needles_grouped returned, so a compaction, a close or an identical write later in the same batch could slip in before the tombstone. drop_cancelled_write now runs right after the publish, under the same sync and access lock hold, in the synced run, the unsynced run and write_needle_unsynced_or_refused. write_pending_needles carries each write's PendingWrite down to that point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: a delete cancels only the writes carrying its cookie Mirror the Go change: a delete with the right needle id and the wrong cookie no longer cancels an in-flight write. Each tracked write keeps its cookie next to its cancel flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a queued write whose volume was remounted The write queue tracks a write on the volume found at submit, but the worker looks the volume up again by id. After a remount, a delete reaches the new volume and cannot cancel the write registered on the old one, so the worker now refuses such a write instead of landing it on the new volume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: decode the delete's type parameter like Go's FormValue The delete handler matched the raw query text for type=replicate, so a percent-encoded name or value was not treated as a replica delete and did not cancel the write it undoes. Parse the query the way the upload handler does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: test a replica delete landing in a write's fsync window Drive DeleteHandler while a durable write is parked in its .dat fsync, appended but not indexed: the replica delete answers 404 and the write is taken back when it publishes, while a plain delete leaves it live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: flush a take-back tombstone before the index row that points at it A durable write taken back during its fsync appends the tombstone after the .dat sync, and the .idx sync that follows made its row durable while the record itself could still be lost, which reloads the volume read only. The fsync and batch paths now sync the .dat again first when a take-back appended a tombstone; a failure stops the volume taking writes like a failed index sync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: flush a take-back tombstone before the index row that points at it A durable write taken back in write_synced_run appends its tombstone after the .dat sync, so the .idx flush that follows could make the row durable while the tombstone record itself was still in the page cache. The run now syncs the .dat again first when a take-back appended a tombstone; a failure stops the volume taking writes like a failed index flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: a skip-cookie batch delete cancels the writes of the file id it names BatchDelete with SkipCookieCheck left the needle's cookie at zero, so the delete's cancel matched none of the in-flight writes and a retry queued behind it could land afterwards. The non-EC delete now carries the file id's cookie, as the Rust volume server already does; the EC path keeps cookie 0, which DeleteEcShardNeedle reads as "skip the check". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: release a held test write before the store closes A failed wait ended the test before release ran, so the writer stayed parked in its fsync and the store cleanup blocked on its lock until the package timed out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f71c0fe523 |
fix(s3tables): route REST RenameTable and refuse unrouted S3 Tables requests (#12446)
* s3tables: route REST RenameTable and refuse unrouted S3 Tables requests
PUT /tables/<arn>/<ns>/<name>/rename had no route, so it fell through to the
/{bucket} routers and ran as a PutObject into a bucket named "tables",
answering 2xx while the table kept its name.
- Answer any s3tables-signed request no S3 Tables route takes with 404
UnknownOperationException, before the health, domain and bucket routers.
- Route the AWS RenameTable REST call (newNamespaceName, newName, both
optional; versionToken).
- Honor versionToken on RenameTable, checked against the snapshot the claim
asserts so a commit between the two reads cannot slip a stale token through.
* s3tables: match the credential scope's service, not a region named s3tables
The unrouted-request fallback widens what a false match costs: an S3
client whose region is named s3tables had every request answered as an
unknown S3 Tables operation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: refuse an empty newName or newNamespaceName on REST rename
An empty field was read as omitted, so {"newNamespaceName":"other",
"newName":""} moved the table under its old name instead of failing
validation the way AWS does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: check the rename version token after the snapshot comparison
Once the separately read metadata is known to equal the claimed snapshot,
the token compares against the parsed metadata; no second decode of the
snapshot bytes is needed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables test: cover policy and tag changes in the rename snapshot compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables test: give each REST rename subtest its own source table
The token subtests renamed the table the first subtest produced, so run
alone they failed on a missing table instead of testing the token.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Chris LuandClaude Opus 5.5
|
c84c43f3da |
master: export Hashicorp raft metrics through the go-metrics global raft uses (#12448)
* master: keep the hostname out of raft metric names go-metrics prefixes gauge names with the hostname by default, which gives every master its own metric family, e.g. SeaweedFS_<host>_raft_peers. Prometheus already tells masters apart by the scrape target. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: send Hashicorp raft metrics through the go-metrics global raft uses hashicorp/raft and raft-boltdb report through github.com/hashicorp/go-metrics, but the master set up its prometheus sink on the armon/go-metrics global, so a -raftHashicorp master exported no raft metric families. The sink is now registered on the hashicorp/go-metrics global, and armon/go-metrics is no longer a dependency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: chart raft commit time and leader contact Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: keep the cluster in the raft panels Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: average raft commit time and leader contact over the rate window The go-metrics summaries keep quantiles for 10s, so a p99 scraped every 30s misses most observations. The _sum/_count rates count every sample. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: leave raft panels empty when nothing was observed Clamping the count rate plotted 0 ms for windows without commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: let the raft metrics test use its own registry Registering on stats.Gather made go test -count=2 fail with a duplicate collector registration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: register raft metrics before opening the raft stores raft.NewRaft starts its run loop before returning, so registering the sink afterwards could drop the first state and boltdb samples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
7b3983f154 |
s3api: speed up recursive listing of branched directory trees (#12438)
* s3api: look up a trailing-slash prefix's directory instead of listing its parent A prefix such as "account-0001/" names one directory, but the walk found it by listing the parent with the name as a prefix. SQL stores answer that with a LIKE that bounds the scan from below only, so in a wide bucket root the query reads every sibling after the name: seconds for a prefix holding a few keys. Look the directory up by name and serve it as the level's only entry. A miss, an error or a non-directory falls back to the prefix listing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: track failover receives atomically The failover wrapper flags a received reply through a plain bool, which a callback reading from the filer on several goroutines would race on. Use an atomic.Bool so the callback may share its filer client. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: read directories ahead of a recursive listing walk A recursive ListObjects walk enters directories one at a time, so a bucket made of many directories holding one or two keys each costs one filer round trip per directory, in series: seconds per 1000-key page on a store whose reads wait on disk. Read the first window of the directories the walk will enter on a few workers, nearest first in walk order, and hand each window over when the walk gets there. Every window the walk reads itself is drained first so its subdirectories are queued before the walk descends into the first one. The number of windows held ahead grows while directories end inside their first window and falls back when one does not, so a page ending inside a large directory leaves few reads wasted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: drop the prefetcher's record of every directory it was handed A page can cross any number of empty directories without spending its key budget, so a set of every announced directory grows with the tree. A directory is already handled when it is at or behind the walk, held or in flight, or queued; check those, which the queue and window caps bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: end a trailing-slash prefix listing at the file its lookup finds A file named by the prefix lists nothing under it, the same as the sibling listing it fell back to, which threw the lookup's answer away and paid the parent scan anyway. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: read a walk's own listing ahead instead of draining it Draining the window before the walk took its first entry made a page wait for the end of the listing, which a SQL store answering a name prefix reaches only after scanning the rest of the directory; a page that fills before then used to return at once. A background reader now hands the window's directories to the prefetcher as they arrive while the walk takes each entry, holding at most 1 MiB of entries ahead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: bound the bytes a prefetched listing window holds Entries with long chunk lists or inline content made the up to 64 held windows of 32 entries an unbounded amount of memory per listing. A window now stops at 128 KiB of entries and the walk lists the rest of that directory from the window's last entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test recursive listings whose streams break partway Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3e64326f2b |
s3: require their own actions to change a bucket's CORS or versioning (#12433)
* s3: require their own actions to change a bucket's CORS or versioning PUT/DELETE ?cors and PUT ?versioning were authorized as Write, so an identity granted only object writes on a bucket could replace its CORS rules or suspend versioning. They now need PutBucketCors, DeleteBucketCors and PutBucketVersioning (or Admin), and inline policies granting s3:PutBucketCors, s3:DeleteBucketCors or s3:PutBucketVersioning map to those actions instead of Write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test that object Write cannot change bucket CORS or versioning Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: offer the bucket CORS and versioning permissions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test each bucket config action with its own identity Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3a579ece3b |
s3: commit the delete marker in the owner's finalize transaction (#12432)
* s3: commit the delete marker in the owner's finalize transaction createDeleteMarker wrote the marker with mkFile to the gateway's current filer and then sent RECOMPUTE_LATEST to the object's ring owner. When the two differ, the owner scans .versions before peer metadata sync delivers the marker, re-derives the pointer to the previous version, and the marker stays non-current on every filer. routedVersionedFinalize now takes an optional version entry and PUTs it in the same ObjectTransaction ahead of the recompute, the way PutObject and CompleteMultipartUpload already do. The copy destination keeps passing nil: its version file is already written on the owner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test a versioned delete when the gateway's filer is not the owner Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: assert the delete marker and the pointer flip share one transaction Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a8766f5177 |
s3api: apply copy bucket-default SSE after the ACL re-auth (#12436)
* s3api: apply copy bucket-default SSE after the ACL re-auth CopyObject without its own x-amz-server-side-encryption header takes the destination bucket's default encryption, which CopyObjectHandler writes into the request headers. It did so before preparePutObjectACL, which re-authenticates a request that carries an ACL (x-amz-acl or a grant header) for s3:PutObjectAcl. verifySignedHeadersCoverage then found the synthesized x-amz-server-side-encryption* headers outside SignedHeaders and rejected the copy with SignatureDoesNotMatch. Move the bucket-default injection after the ACL check, so the re-authentication sees the request as the client signed it. The destination object still gets the bucket default encryption. * s3api: assert the copy's KMS key and stored ciphertext Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Stefan Scharlott <stefans@Stefans-MacBook-Pro.local> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d127be610f |
admin: re-send the master vacuum disable every poll so it survives leader changes (#12430)
* admin: re-send the master vacuum disable every poll so it survives leader changes
The admin disabled the master's scheduled vacuum only when a vacuum plugin
worker connected or disconnected. The master keeps that flag in memory and it
is not replicated through raft, so after a leader change or a leader restart
the new leader ran its own vacuum next to the plugin worker. When both picked
the same volume the master's commit failed ("compact swap aborted: missing
.cpd/.cpx") and it dropped that replica, leaving the volume under-replicated.
Send the desired state on every poll instead, and log on the master only
when the flag actually changes.
* admin: shorten the vacuum sync comments
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Kevin <kevin@leetify.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
github-actions[bot]
|
0dd33f5f0f | docs: regenerate star history chart | ||
|
Chris LuandClaude Opus 5.5
|
878979af70 |
filer: hide write controls in the UI on the read-only port (#12429)
* filer: hide write controls in the UI on the read-only port The read-only port accepts only GET and HEAD, but its directory page offered New Folder, Upload, drag-and-drop, rename and delete, each of which failed with 405. Requests that come through the read-only handler now render the page without those controls or the upload script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test the directory page controls on each port Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: match the rendered row buttons in the port test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e44c16802d |
weed/server: verify data shards against bitrot sidecar before ec decode (#12422) (#12427)
* weed/server: verify data shards against bitrot sidecar before ec decode (#12422) * ec.decode: verify data shards against the mounted volume's bitrot protection The decode looked up only the generation-0 .ecsum beside the data base and each location's data directory, so it missed a sidecar kept with the index and ignored the generation the volume actually serves. Take the protection the volume's per-disk runtimes resolved instead (On, else Invalid, else Off), and test a corrupt block past the first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.decode: check for live entries before verifying the shards A volume whose needles are all deleted gets its shards purged on the no-live-entries status; a failing checksum must not keep it around. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: verify data shards against the bitrot protection before a decode Mirrors the Go volume server: the decode takes the protection the volume's per-disk runtimes resolved and refuses a data shard that fails its .ecsum checksum, after the no-live-entries check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: a decode refuses a data shard that fails its .ecsum checksum Runs against both volume server implementations: the refused decode keeps every shard and the sidecar, and the needle stays readable through EC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.decode: reload the bitrot sidecar before verifying the shards A rebuild's backfill writes the sidecar without reloading the mounted runtimes, which then still report no protection; resolve it again across the EC metadata directories, as a shard mount does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: reload the bitrot sidecar before verifying the shards Mirrors the Go volume server: the decode job resolves the sidecar again on every runtime of the volume before choosing its protection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: a decode finds a sidecar written after the shards were mounted Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: read through the EC shards after the refused decode The source volume stayed mounted, so the final read never touched the shards it is meant to check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.decode: refuse when a sidecar reload reports another layout ReloadBitrotSidecar logged and dropped the layout-mismatch error, so the runtime stayed Off and the decode skipped the data-shard check. It now returns that error and the decode refuses on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a decode when a sidecar reload reports another layout Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: a decode refuses a late sidecar that records another layout Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.decode: refuse when an unmount left no runtime to resolve the sidecar Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a decode when an unmount left no runtime to resolve the sidecar Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
80f32bc074 |
weed/command: skip remote bucket delete when bucket is unmounted (#12418) (#12425)
* weed/command: skip remote bucket delete when bucket is unmounted * remote_storage: count deleting an already-missing bucket as done filer.remote.gateway can see the same bucket delete more than once, for example once per filer when several filers peer their metadata. A copy that still finds the mount mapping calls DeleteBucket again, and S3 (NoSuchBucket), Azure (ContainerNotFound, ContainerBeingDeleted) and GCS (404) all failed it, which held the sync offset on the delete. A missing bucket now counts as deleted, the way deleting a missing object already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: keep the mapping table writable after mount.mapping is deleted Deleting mount.mapping replaced the gateway's mappings with a table whose map was nil, so the next bucket create panicked writing its mount into it. Reset to an empty map instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: count a delete of an absent object as done filer.remote.sync already treats ErrRemoteObjectNotFound from DeleteFile as deleted, since GCS reports a missing object that way where S3 and Azure succeed. The gateway returned it, so a delete event seen twice, as it is when several filers peer their metadata, held the sync offset on GCS. Use the same deleteRemoteFile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: drop cached remote clients after the new gateway tests GetRemoteStorage caches clients by configuration, so a repeated run (-count=2) got the first run's recorder back and saw no calls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: keep the default storage across mount.mapping changes The default storage for new buckets is derived at startup and kept only in the gateway's copy of the mappings; mount.mapping never stores it. Replacing or deleting that file dropped it, so a later bucket create with no -createBucketAt skipped the remote bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
c971252871 |
admin: make the cluster lock's fairness window configurable (#12426)
* admin: make the cluster lock's fairness window configurable Scheduled jobs hold the cluster admin lock for their whole run. Once the lock had been held for a minute, every new job waited until all running jobs had released it, so with hour-long EC jobs dispatch ran in waves and most workers sat idle between them. -adminLock.fairnessWindow keeps the one-minute default; 0 lets new jobs join the shared hold, and shell clients get the lock when the scheduler next drains. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * admin: wait for the blocked acquire before disabling the fairness window The wake-up test slept 50 ms before SetFairnessWindow(0), so a late acquire that started after the change joined without exercising the Broadcast. Wait until it is registered as a waiter instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8c8d0f1cdf |
weed/shell: resolve collection from topology when ec.decode is given only volumeId (#12424)
* weed/shell: resolve collection from topology when ec.decode is given only volumeId
When ec.decode is invoked with -volumeId for a volume that belongs to a named collection without explicitly specifying -collection, DoEcDecode was called with an empty collection string. Target volume servers then rejected VolumeEcShardsCopy with 'CopyFile not found ec volume id <vid>' because they looked under the default collection.
This commit introduces FindEcVolumeCollection to discover the collection name directly from topology metadata (checking EC shard info and regular volume info), ensuring DoEcDecode receives the accurate collection.
* ec.decode: resolve every batch volume's collection from the topology
The batch passed -collection through unless it looked like a regex, so a
comma list (a,b), a {n} quantifier, ALL or _default_ reached the volume
servers as a collection name and the copy found no shards. The selector
only picks the volumes; each one is decoded in its own collection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: take the collection only from the volume's ec shards
The lookup returned the first record per disk, so a regular volume on an
earlier node answered before the ec shards on a later one. A decode needs
the shards anyway, so their records are the only ones to read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: leave the -volumeId lookup to DoEcDecode
DoEcDecode already resolves an empty collection from the topology.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
120d17a092 |
filer.remote.sync: stop a touched file from being restamped forever (#12419)
* filer.remote.sync: stop a touched file from being restamped forever A metadata-only update that moves the mtime past RemoteMtime (touch, or an O_TRUNC open of an empty lock file) goes through UpdateFileMetadata, which changes nothing on the remote, and the entry is stamped with the same RemoteEntry and a new LastLocalSyncTsNs. That stamp is an update event of its own. shouldSendToRemote only looks at LastLocalSyncTsNs when RemoteMtime and the mtime share a second, so it still reads the entry as dirty, and the event is stamped again. Nothing is uploaded, but the daemon keeps rewriting the entry as fast as it reads its own events back. Skip the restamp when the backend left the object alone and the event's entry already carries a stamp taken after its mtime. The touch itself is still stamped once. * filer.remote.sync: log the skipped restamp The other skips in processUpdateEvent log at V(2); this one did not, so with -v=2 a touched file showed an "update meta" line with nothing after it. * filer.remote.sync: skip the restamp of a stamp's own event, whatever the clocks The skip compared the entry's stamp with its mtime, so an mtime ahead of the sync's clock (a skewed writer, touch -d) stayed past every stamp and kept the loop going until the clock caught up. An event that changed only the RemoteEntry is the sync's own stamp, which needs no other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: ignore the filer's commit timestamp when spotting a stamp The filer gives every stored update a new CommitTsNs, so a stamp's event never compared equal outside the RemoteEntry and a touched file kept being restamped against a real filer. The test now assigns one per update. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
23598afb31 |
volume: bound each replica forward so a silent peer cannot hold a write (#12417)
* volume: bound replica forwards so a silent peer cannot hold a write The Rust volume server forwards each replicated write to its peers with the shared outgoing reqwest client, which had no connect or request timeout. When a peer is silently partitioned (packets dropped, no reset) the forward waits until TCP gives up, minutes later, and the write waits with it. A chaos test (replication 001, one volume server partitioned) saw every write on a volume with a copy there block. - Replica forwards go through replica_request, bounded at REPLICA_REQUEST_TIMEOUT = 10 s from connect to the response. - Both outgoing clients start from outgoing_client_builder, which adds a 5 s connect timeout, so new connections to a peer that never answers fail fast. The shared client gets no overall request timeout, since it also serves lookups and other transfers. Tests: a replica that receives the forward and never answers fails the whole replicated write at the bound, naming that replica; a silent peer fails the request builder at the bound on a paused clock; a connect that stalls in the resolver fails at the 5 s connect bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: give a replica forward more time for a larger body The 10 s replica forward bound runs from connect to the end of the response, so it also covers sending the body. A write near the 256 MB file size limit needs about 26 MB/s to reach its replica in 10 s, so on a link that is slower or busy the forward failed and the write returned 500 even though the peer was answering. A forward now gets REPLICA_REQUEST_TIMEOUT plus one second per MiB of body. A bodiless forward keeps the 10 s bound (a 1 MiB write gets 11 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: bound each Go replica forward like the Rust one The Go volume server forwards a replicated write with a 10 s dial bound and the transport's 30 s response header timeout. A replica that takes the forward and never answers held the write for 30 s, and a body larger than the socket buffers to a silently partitioned peer was held until TCP gave up, since nothing bounds sending the body. Each forward now runs under replicaRequestBound: 10 s plus one second per MiB of body, the same bound the Rust volume server puts on its forwards. Test: a replica that receives the forward and never answers fails the write at the bound (300 ms in the test); without it the write fails at 30 s on the response header timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: keep proxy variables out of the stalled-connect test reqwest reads HTTP_PROXY and ALL_PROXY by default, so with a proxy set the test's request went to the proxy instead of the stalled resolver. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: count a replica forward that hit its bound as a timeout DistributedOperationResult joined the replica errors into a plain string, and the uploader reports a cancelled request without wrapping the context error, so a forward cut off at replicaRequestBound was counted under server_error in the replication failure metric, as was a response header timeout before it. The joined error now unwraps to each replica's error, and a forward that ran out of its bound reports context.DeadlineExceeded. Test: the silent replica write raises the timeout failure count by one; with the flattened join it stays put. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: test that a large Go replica forward gets its body's bound A replica answers after the bodiless bound but within the bound for a 1 MiB body, so the write fails if replicateTo drops the body scaling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: run the Rust whole-path replica test with a 1 MiB body The forward then has to wait the bound for its body, so the test fails if do_replicated_request stops passing the body length to the bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: pingqiu <pingqiu@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
aab947b65e |
s3api: answer success when a failed completion already committed (#12416)
* s3api: name the check that a completion already committed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: answer success when a failed completion already committed A finalize can fail after its commit: the filer stored the object and the call still reported an error. The client then got a 500 with an unknown outcome. When the failure may have left this attempt's object behind, check the key the way a retry would and answer success with the committed result when it holds this upload's object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: keep part data when the commit found is another attempt's The object carrying this upload ID can be a concurrent attempt's commit of the same upload, and its part selection can include entries this attempt counts as unused. Free those parts' data only when the committed chunks are this attempt's own; otherwise drop their metadata only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test the committed-completion read on versioned buckets Cover the latest-version read (version id and checksum in the answer), a suspended bucket whose delete marker is still current, and that the upload directory is removed without its data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test versioned recovery through the completion handler Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
fe1cf5263f |
volume: bound each FetchAndWriteNeedle replica write (#12415)
* seaweed-volume: bound each FetchAndWriteNeedle replica write A replica that accepted the connection and then went silent held the request until TCP gave up, because the shared outgoing client has no request timeout. Each replica write now fails after 30 s, from connect to the response status, matching the Go volume server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: bound each FetchAndWriteNeedle replica write The default replica path uploads through the guarded peer client, which bounds only the dial and has no response header timeout, and the filer calls FetchAndWriteNeedle with no deadline. A replica that accepted the connection and went silent held the request for minutes. Each replica write, retries included, now has 30 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: require the silent-peer test to fail at the deadline A connection error that came back early also passed, since the test only checked an upper bound. It now also requires the paused clock to have reached FETCH_AND_WRITE_REPLICA_TIMEOUT. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: grow the FetchAndWriteNeedle replica bound with the chunk The reqwest timeout includes sending the body, and a remote-cache chunk is a thousandth of the object once it passes the 1000-chunk cap, so a fixed 30 s would fail a large chunk on a link under ~35 MB/s. The bound is now 30 s plus 1 s per MiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: grow the FetchAndWriteNeedle replica bound with the chunk The context deadline includes sending the body, so a fixed 30 s would fail a large remote-cache chunk on a slow link. The bound is now 30 s plus 1 s per MiB, matching the Rust volume server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
247e010865 |
volume: redial a master that just dropped a registered stream after a second (#12420)
* volume: redial a master that just dropped a registered stream after a second A registered volume server that loses its master with no peer naming a leader waited a full pulse before redialing. A restarted master is usually back within a second, so its volumes stayed unregistered, and uncached lookups kept failing, for most of a pulse. For one pulse after a stream a leader had answered ends, redial on the one-second cap. A master still down after that is redialed at the pulse pace, so redials never come later than before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: redial a master that just dropped a registered stream after a second Mirror of the Go heartbeat change. The loop keeps when the last stream a leader had answered ended, which also stands in for the confirmed-leader-once flag (None until first registration), and redials on the one-second cap for a pulse after that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: test that a dropped registered stream records the loss A fake master confirms the stream and then ends it. The server starts as registered long ago, so only a recorded loss makes the redial come after a second instead of a pulse. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: record a lost stream only after its receiver has exited doHeartbeatWithRetry can return on a send error while the receiver is still handling the leader's first reply, so the deferred recorder could read the registered flag too early and keep an old loss time. Cancel the stream and wait for the receiver before reading it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: require the prompt heartbeat redial to wait out its second Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: fail the down master's dials at once in the redial test Windows takes about two seconds to refuse a connection to a closed loopback port, so the stream dial and the leader lookup each held the prompt case past its two-second bound. A listener that closes every connection fails both dials immediately on every platform. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
8cc3a8ae21 |
ec.rebuild: repair shards that fail a checksum scrub in complete volumes (#12410)
* ec.rebuild: rebuild shards that fail a checksum scrub, also in complete volumes ec.rebuild skipped every volume with all shards present, so a shard whose bytes no longer match the .ecsum sidecar stayed in place until some other shard went missing. With -checksum, ec.rebuild first scrubs each node's shards in checksum mode and takes the failing copies out of the plan; a shard left without a good copy counts as missing and is regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: remove the copies that failed the checksum once a good copy serves A shard regenerated on the rebuilder left the failing copy mounted on its old holder, so the volume kept a corrupt duplicate that a later dedup could keep instead of the good one. Remove each failing copy once the rebuilt shard is mounted, and right away when another holder already serves a copy that passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: test the checksum scrub against fake volume servers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: check the lock before removing a corrupt copy Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: print the checksum scrub details A malformed sidecar or a wholesale mismatch comes back only in the scrub details, with no broken shard, so the volume would look verified. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: fail the run when a corrupt duplicate cannot be removed For a volume whose only problem is a corrupt duplicate, the removal is the whole repair, so its failure has to reach the caller. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: test duplicate removal failures and a lost lock Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: fail the run when a node's checksum scrub fails The node's copies stay in the plan unverified, so a corrupt shard there is left in place; the rest of the cluster is still repaired. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: preview the corrupt-copy removals in dry-run Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: test a failed checksum scrub Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
fcf62db8d0 |
seaweed-volume: stage optional EC sidecar copies before replacing (#12403)
* seaweed-volume: stage optional EC sidecar copies before replacing VolumeEcShardsCopy opened the destination .vif/.ecsum with File::create and then streamed into it. A source without the file left a 0-byte .vif, wiped an existing .ecsum (turning bitrot protection off on that holder), and a failed stream removed the destination's existing copy. Stream optional copies into <file>.copying and rename into place only once the source sent bytes, as the Go volume server's writeToFile does for ignore_source_file_not_found. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * seaweed-volume: keep the drain doc comment on drain_copy_stream_to_file Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: stage the .vif copied during missing-index recovery fetch_ec_index_from_one_peer drained the optional .vif straight into the local file, so a peer without one left a 0-byte .vif (losing a local EC ratio) and a failed stream left a partial one. Stage it and rename only once the peer sent bytes, as Go's recovery does through writeToFile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: test optional copies against a stream that fails mid-transfer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: remove a staged optional copy when the RPC is cancelled A cancelled CopyFile handler drops its future at the stream await, so the cleanup after it never ran and the .copying file stayed behind. Go's writeToFile runs to completion and removes it. A guard now owns the staging cleanup and is disarmed only once the rename commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: test optional copies dropped mid-stream Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: cancel the staged-copy test only once the staging file exists tokio's File::create runs on the blocking pool, so a fixed timeout could drop the .ecsum copy before .copying existed and pass without exercising the cleanup guard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
85232f7414 |
seaweed-volume: run the preallocating max-volume test on a fake disk (#12423)
* seaweed-volume: run the preallocating max-volume test on a fake disk The test sized the volume limit from the real free space but the store subtracts the 1% floor first, so on a disk more than 95% full the second slot no longer fit and the count came out 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: start the preallocating max-volume test above the expected count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b74111ea2f |
s3vectors test: skip a query lease tag released between list and HEAD (#12409)
A query's lease tag is deleted by a background task after the query answers, so the encryption helper could list it and then get a 404 on HEAD. Skip only that case; any other missing object still fails. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Fable 5.1
|
48df16ea0d |
weed-worker: keep a vector index covering as its table grows (ANN phase 3) (#12414)
* seaweed-vector: build and extend the vector index weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the table was created with, sizes partitions from the row count, and extends an index with a new segment or a retrain; the core crate re-exports it and adds the probes a query spends. Tests cover coverage after build, append, retrain and compaction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector-cli: index build and index optimize Logs go to stderr so stdout stays the JSON a caller parses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_INDEX builds the index before the timed queries The cluster fixture hands out its catalog URL, the Makefile builds weed-vector-cli, and the perf suite builds the index after ingest and reports the build time and segment count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: probe a tenth of the index's partitions, more under a filter A query that names no probe count gets one from the index's partition count. Lance searches at least that many partitions, and under a prefilter keeps going until it has k rows; without one it stops there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: --nprobes overrides the probe policy Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that a build takes the table's metric Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make clean removes weed-vector-cli Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: measure recall with vectors the table does not hold Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep the read cache's doc comment on with_read_cache Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold Detection keeps the tables a vector bucket created, with at least min_rows rows (50,000 by default) and no index on their vector column. Execution builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: optimize only the indices on the vector column Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: resolve the query metric with index::distance_metric Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test the L2 metric under the name CreateIndex stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: build the index through the admin server and the Lance worker VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new test sets the job's threshold, runs it through the plugin API, checks coverage and recall, and that a row written after the build is found. CI gains the worker leg for both volume servers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: read an index's partitions, and when a table has outgrown them Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: probes from the shared partition reader, partitions in coverage Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index reindexes at a tenth of its rows and retrains once the table outgrows it lance_optimize_indices proposes a vector table once its uncovered rows pass min(the absolute budget, a tenth of the indexed rows), and trains the index again instead of extending it when sqrt(rows) has reached twice its partitions. The end-to-end test covers both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: open a table inside a caller's Lance session Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep opened indices between queries in one session Every open used a fresh Lance session, so the index and metadata caches died with the request and each query loaded the IVF index again. --index-cache-mb sizes the shared cache, 512 MiB by default. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the index test draws embedding-like vectors Independent Gaussian vectors have no locality for an IVF index to use, so recall through one only tracks the fraction of partitions probed; points around shared centres behave like embeddings. The test must also run before the one that stops the engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the optimize and retrain steps draw embedding-like vectors Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that compaction drops the deleted key from tied hits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test that an append-only optimize serves the new row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: record PERF_NOISE in the perf report config Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: a filtered query widens to four times the probes, not without bound With no upper bound Lance prepares every partition for the late search even when the minimum probes already hold k rows, and a filtered query through the index cost as much as the exact scan. A query that names its own probe count gets exactly that count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: report recall through an index without asserting it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a hit tying the k-th best score toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: lance_vector_index runs in the Lance lane Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: check a table's index before counting its rows Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make build and clean cover the worker leg's binaries Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a retrain rebuilds the index with partitions for the row count Lance's OptimizeOptions::retrain does not size the partitions: it left a 6,000-row index with one. The retrain goes through the builder with sqrt(rows) partitions, for the job and for weed-vector-cli index optimize --retrain alike. The end-to-end test no longer expects a merge to keep the partition count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: allow float32 rounding at the recall cutoff Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a repeated hit key once toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: credit cutoff ties only up to the slots the top-k leaves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: assert every filtered hit passes the filter, indexed or not Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: coverage over an index's names, one name extended, a merge at the delta limit A sharded build leaves one name per shard, each missing the others' fragments, so coverage is the union of every index on the column and an optimize gives new rows to the name with the fewest deltas only. At MAX_DELTAS the policy passes OptimizeOptions::merge(deltas); Lance's default options may append instead of merging. A retrain keeps an index's own metric and rebuilds only IVF_FLAT indices. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: the optimize job measures and extends a vector table's indices together Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: tests for bounded deltas, disjoint shards and a retrain's metric Nine append-and-optimize cycles give delta counts 2 to 8 and then 1; two names on disjoint fragments stay disjoint through an optimize; a cosine index on a table without metric metadata is still cosine after a retrain. The end-to-end test expects the appended delta after the first optimize. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: forget the shared session when a table was dropped and created again The session keys index lists and transactions by dataset version, and a table created again at the same location restarts its versions. An open that finds an older version than the newest seen at that URI, or the same version committed later, replaces the session and opens again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: test an open after another engine created the table again Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: retrain only a single IVF_FLAT vector index name The job compared each shard name's partitions with the whole table's rows, so a sharded index was always due, and a non-IVF_FLAT index made optimize bail on every run before the new rows were indexed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: refuse to retrain a vector index sharded over several names A rebuild per name over the whole table left every shard covering every fragment, so a query searched each row once per shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test which vector indices a retrain is due for Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: replace the session in the same lock that records a recreated table A second open of the recreated table could see the new version recorded and the old session still shared, and keep the dropped table's index list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: skip a vector table whose rows cannot be counted Detection returned on the first table whose count_rows failed, so one unreadable table kept every other table from being proposed. It now skips that table like an open or index failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: panel for vector indexes the Lance worker built Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: size default probes from the largest index on the column A sharded build gives the column one index name per shard. The first name alone could be a small shard, and its probe count would then be spent on a larger one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: skip the probe lookup when a query is given no segments Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: open every table in its own Lance session again One session shared by every open keeps a table's index loaded between queries, but Lance keys what it caches by location and version, so a table dropped and created again at the same location could be served the old table's index list. Telling the two apart from version and commit time needs a map that only grows, a retry loop and clocks that agree across engines. A private session per open, as before, has no such state. Measured at 50k x 384 through an index, eight clients: 164 queries/s with the shared session, 137 without; one client is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: reject --nprobes 0 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: read the partition count from the opened index Lance's index statistics took 3.6 ms a query on a 200k-row table with 447 partitions. Opening the index and asking it takes 0.4 ms, and the scan that follows finds the index already open. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * ci: give the S3 Vectors worker legs the vendored protoc The worker workspace has no protoc of its own for lance's build scripts, so the worker legs failed at their first build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: give a vector index build an hour before the admin times it out The job set no execution timeout, so the admin's default of 90 seconds applied, and a build that trains on a large table would be reported as failed and retried while it was still running. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_vector_index proposes at most the admin's max_results Every qualifying table went into one message however many there were. The detector stops at the limit the request carries and sets has_more, as the Go detectors do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index job re-checks the table it was proposed for A table emptied since detection completes without a build, and one without a vector column fails instead of reporting it already indexed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_optimize_indices gets a build's execution timeout A retrain rebuilds the index over the whole table, which the admin's default of 90 seconds would cut off on a large one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index run window as long as one build An attempt is capped at the end of the job type's run window plus the drain grace, and the default window is 30 minutes, so a build could be cut off before its own hour was up. The window is an hour too, as the Go iceberg handler sets both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a lance_optimize_indices run window as long as one retrain Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: vector index proposals declare a build's runtime A run started by hand while the job type's schedule is off takes no timeout from the descriptor, only the admin's 90 seconds. The admin gives a job at least the runtime its proposal declares, so the build and the optimize proposals of a vector table declare the hour. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: coverage reports no partitions for an index of unknown coverage Reading the partition count goes through Lance's index statistics, which fail on an index that records no fragment bitmap, and the whole coverage report failed with them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: an index that records no fragments is not read as uncovered Its rows counted as uncovered, so detection proposed the table and the execution then failed in Lance's statistics. Coverage fails for such an index instead, and detection skips the table with a warning. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: coverage reads partition counts from the opened index Lance's index statistics rewrite the manifest of a table whose fragments carry no row counts, which a coverage report must not do and cannot do with read-only credentials. The report asks each opened segment instead, as the probe lookup does, and the statistics-based reader has no caller left. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Fable 5.1
|
e465880ee8 |
weed-worker: build the vector index of S3 Vectors tables past a row threshold (ANN phase 2) (#12413)
* seaweed-vector: build and extend the vector index weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the table was created with, sizes partitions from the row count, and extends an index with a new segment or a retrain; the core crate re-exports it and adds the probes a query spends. Tests cover coverage after build, append, retrain and compaction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector-cli: index build and index optimize Logs go to stderr so stdout stays the JSON a caller parses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_INDEX builds the index before the timed queries The cluster fixture hands out its catalog URL, the Makefile builds weed-vector-cli, and the perf suite builds the index after ingest and reports the build time and segment count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: probe a tenth of the index's partitions, more under a filter A query that names no probe count gets one from the index's partition count. Lance searches at least that many partitions, and under a prefilter keeps going until it has k rows; without one it stops there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: --nprobes overrides the probe policy Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that a build takes the table's metric Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make clean removes weed-vector-cli Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: measure recall with vectors the table does not hold Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep the read cache's doc comment on with_read_cache Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold Detection keeps the tables a vector bucket created, with at least min_rows rows (50,000 by default) and no index on their vector column. Execution builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: optimize only the indices on the vector column Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: resolve the query metric with index::distance_metric Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test the L2 metric under the name CreateIndex stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: build the index through the admin server and the Lance worker VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new test sets the job's threshold, runs it through the plugin API, checks coverage and recall, and that a row written after the build is found. CI gains the worker leg for both volume servers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: open a table inside a caller's Lance session Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep opened indices between queries in one session Every open used a fresh Lance session, so the index and metadata caches died with the request and each query loaded the IVF index again. --index-cache-mb sizes the shared cache, 512 MiB by default. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the index test draws embedding-like vectors Independent Gaussian vectors have no locality for an IVF index to use, so recall through one only tracks the fraction of partitions probed; points around shared centres behave like embeddings. The test must also run before the one that stops the engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that compaction drops the deleted key from tied hits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test that an append-only optimize serves the new row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: record PERF_NOISE in the perf report config Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: a filtered query widens to four times the probes, not without bound With no upper bound Lance prepares every partition for the late search even when the minimum probes already hold k rows, and a filtered query through the index cost as much as the exact scan. A query that names its own probe count gets exactly that count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: report recall through an index without asserting it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a hit tying the k-th best score toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: lance_vector_index runs in the Lance lane Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: check a table's index before counting its rows Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make build and clean cover the worker leg's binaries Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: allow float32 rounding at the recall cutoff Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a repeated hit key once toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: credit cutoff ties only up to the slots the top-k leaves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: assert every filtered hit passes the filter, indexed or not Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: forget the shared session when a table was dropped and created again The session keys index lists and transactions by dataset version, and a table created again at the same location restarts its versions. An open that finds an older version than the newest seen at that URI, or the same version committed later, replaces the session and opens again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: test an open after another engine created the table again Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: replace the session in the same lock that records a recreated table A second open of the recreated table could see the new version recorded and the old session still shared, and keep the dropped table's index list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: skip a vector table whose rows cannot be counted Detection returned on the first table whose count_rows failed, so one unreadable table kept every other table from being proposed. It now skips that table like an open or index failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: panel for vector indexes the Lance worker built Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: size default probes from the largest index on the column A sharded build gives the column one index name per shard. The first name alone could be a small shard, and its probe count would then be spent on a larger one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: skip the probe lookup when a query is given no segments Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: open every table in its own Lance session again One session shared by every open keeps a table's index loaded between queries, but Lance keys what it caches by location and version, so a table dropped and created again at the same location could be served the old table's index list. Telling the two apart from version and commit time needs a map that only grows, a retry loop and clocks that agree across engines. A private session per open, as before, has no such state. Measured at 50k x 384 through an index, eight clients: 164 queries/s with the shared session, 137 without; one client is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: reject --nprobes 0 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: read the partition count from the opened index Lance's index statistics took 3.6 ms a query on a 200k-row table with 447 partitions. Opening the index and asking it takes 0.4 ms, and the scan that follows finds the index already open. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * ci: give the S3 Vectors worker legs the vendored protoc The worker workspace has no protoc of its own for lance's build scripts, so the worker legs failed at their first build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: give a vector index build an hour before the admin times it out The job set no execution timeout, so the admin's default of 90 seconds applied, and a build that trains on a large table would be reported as failed and retried while it was still running. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_vector_index proposes at most the admin's max_results Every qualifying table went into one message however many there were. The detector stops at the limit the request carries and sets has_more, as the Go detectors do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index job re-checks the table it was proposed for A table emptied since detection completes without a build, and one without a vector column fails instead of reporting it already indexed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index run window as long as one build An attempt is capped at the end of the job type's run window plus the drain grace, and the default window is 30 minutes, so a build could be cut off before its own hour was up. The window is an hour too, as the Go iceberg handler sets both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Fable 5.1
|
b5951c5c7c |
seaweed-vector: probe policy for indexed queries (ANN phase 1) (#12412)
* seaweed-vector: build and extend the vector index weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the table was created with, sizes partitions from the row count, and extends an index with a new segment or a retrain; the core crate re-exports it and adds the probes a query spends. Tests cover coverage after build, append, retrain and compaction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector-cli: index build and index optimize Logs go to stderr so stdout stays the JSON a caller parses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_INDEX builds the index before the timed queries The cluster fixture hands out its catalog URL, the Makefile builds weed-vector-cli, and the perf suite builds the index after ingest and reports the build time and segment count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: probe a tenth of the index's partitions, more under a filter A query that names no probe count gets one from the index's partition count. Lance searches at least that many partitions, and under a prefilter keeps going until it has k rows; without one it stops there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: --nprobes overrides the probe policy Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that a build takes the table's metric Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make clean removes weed-vector-cli Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: measure recall with vectors the table does not hold Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep the read cache's doc comment on with_read_cache Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: optimize only the indices on the vector column Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: resolve the query metric with index::distance_metric Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test the L2 metric under the name CreateIndex stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: open a table inside a caller's Lance session Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep opened indices between queries in one session Every open used a fresh Lance session, so the index and metadata caches died with the request and each query loaded the IVF index again. --index-cache-mb sizes the shared cache, 512 MiB by default. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that compaction drops the deleted key from tied hits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test that an append-only optimize serves the new row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: record PERF_NOISE in the perf report config Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: a filtered query widens to four times the probes, not without bound With no upper bound Lance prepares every partition for the late search even when the minimum probes already hold k rows, and a filtered query through the index cost as much as the exact scan. A query that names its own probe count gets exactly that count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: report recall through an index without asserting it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a hit tying the k-th best score toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: allow float32 rounding at the recall cutoff Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a repeated hit key once toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: credit cutoff ties only up to the slots the top-k leaves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: assert every filtered hit passes the filter, indexed or not Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: forget the shared session when a table was dropped and created again The session keys index lists and transactions by dataset version, and a table created again at the same location restarts its versions. An open that finds an older version than the newest seen at that URI, or the same version committed later, replaces the session and opens again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: test an open after another engine created the table again Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: replace the session in the same lock that records a recreated table A second open of the recreated table could see the new version recorded and the old session still shared, and keep the dropped table's index list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: size default probes from the largest index on the column A sharded build gives the column one index name per shard. The first name alone could be a small shard, and its probe count would then be spent on a larger one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: skip the probe lookup when a query is given no segments Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: open every table in its own Lance session again One session shared by every open keeps a table's index loaded between queries, but Lance keys what it caches by location and version, so a table dropped and created again at the same location could be served the old table's index list. Telling the two apart from version and commit time needs a map that only grows, a retry loop and clocks that agree across engines. A private session per open, as before, has no such state. Measured at 50k x 384 through an index, eight clients: 164 queries/s with the shared session, 137 without; one client is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: reject --nprobes 0 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: read the partition count from the opened index Lance's index statistics took 3.6 ms a query on a 200k-row table with 447 partitions. Opening the index and asking it takes 0.4 ms, and the scan that follows finds the index already open. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Fable 5.1
|
f0f0ba85ae |
seaweed-vector: build a vector index from the CLI and measure it in the perf suite (ANN phase 0) (#12411)
* seaweed-vector: build and extend the vector index weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the table was created with, sizes partitions from the row count, and extends an index with a new segment or a retrain; the core crate re-exports it and adds the probes a query spends. Tests cover coverage after build, append, retrain and compaction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector-cli: index build and index optimize Logs go to stderr so stdout stays the JSON a caller parses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_INDEX builds the index before the timed queries The cluster fixture hands out its catalog URL, the Makefile builds weed-vector-cli, and the perf suite builds the index after ingest and reports the build time and segment count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that a build takes the table's metric Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make clean removes weed-vector-cli Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: optimize only the indices on the vector column Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: resolve the query metric with index::distance_metric Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test the L2 metric under the name CreateIndex stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that compaction drops the deleted key from tied hits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test that an append-only optimize serves the new row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: record PERF_NOISE in the perf report config Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: report recall through an index without asserting it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a hit tying the k-th best score toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: allow float32 rounding at the recall cutoff Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a repeated hit key once toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: credit cutoff ties only up to the slots the top-k leaves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: assert every filtered hit passes the filter, indexed or not Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
8350f297af |
seaweed-vector: dev plan for ANN search over S3 Vectors indexes (#12408)
* seaweed-vector: dev plan for ANN search over S3 Vectors indexes Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: one probe count per query, a budget that can shrink Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: smaller deltas are searched in full only up to the probe count Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: drop the private issue reference Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: phase 1 probes use the Design count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: status after phases 0 to 3, and what the measurements changed Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: name the Enterprise repository, not the private one Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: the Enterprise repository by that name Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: a filtered query widens to four times the probes, not without bound Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: throughput through the index at 50k and 200k Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: union coverage over shard names, and an explicit merge at the delta limit Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: status table names branches, not pull requests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: maximum_nprobes is always four times the count Lance reads maximum_nprobes only under a prefilter, and the phase 1 policy sets it the same way for every query. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: the retrain follows the partition count, not a recorded row count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: phase 3 is open like the other phases Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: the retrain stops past 2,048 partitions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: the merge at the delta limit takes the name's delta count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
github-actions[bot]
|
cd10b4a5c1 | docs: regenerate star history chart | ||
|
Chris Lu
|
683c5ad4d3 |
admin: regenerate the plugin pages' templ output from weed/admin (#12405)
Both were generated at the repo root, so their embedded source paths read weed/admin/view/... where every other admin template, generated by weed/admin's Makefile, reads view/.... No markup changes. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi |
||
|
Chris LuandClaude Opus 5.5
|
7047d3947a |
admin/plugin: fail a dispatch its restarted worker no longer lists (#12402)
* admin/plugin: fail a dispatch its restarted worker no longer lists A job dispatched to a worker that then restarts dies with the old process. The worker re-registers under the same id on a new stream and its heartbeats stop listing the job, but nothing ended the pending execution: the dispatcher kept waiting and held the executor slot until the attempt's execution timeout. Each pending execution now records the stream it went out on. An itemized heartbeat from a newer stream of that worker that leaves the job out twice in a row fails it through the expire path, so the dispatch returns, the slot is released and the retry machinery takes over. Jobs on the heartbeat's own stream, jobs the new stream still lists, and heartbeats that count slots without itemizing them are left alone. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * admin/plugin: let a partly itemized listing clear a dispatch's misses A heartbeat that counted more slots than it itemized returned before the listed jobs had their misses reset, so an omission, a partial heartbeat listing the job, and another omission failed a job the worker was still running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: look up the worker's stream before tracking the attempt A dispatch to a worker with no stream returned after the job was already recorded as assigned, and that record kept new proposals with the same dedupe key out until the stale-job sweep expired it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: fail the attempt whose request could not be sent The dispatch now sends on the stream it looked up before tracking the attempt, so a worker that reconnects in between makes the send fail. That returned with the job still tracked as assigned, and with the worker back nothing closed the record for the 24h stale timeout while its dedupe key kept new proposals out. The attempt is now completed as failed, after it is unregistered so the completion does not wait on a channel nobody reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: leave a send failure to the expiry that already claimed it The send-error path and an orphan expiry could both complete the same attempt and record two failed runs for it. The send-error path now completes the attempt only if no expiry has marked it failed, decided under the same lock that unregisters it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: record a send failure whose entry Shutdown already removed Shutdown closes shutdownCh before it unregisters pending dispatches, so a blocked send returns while its map entry is gone. Read the expiry claim from the dispatch's own state instead of the map lookup, so the attempt is still failed and does not stay assigned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: match a padded job id against its trimmed listing Heartbeat work ids are trimmed before the orphan check, so compare the dispatch's job id trimmed too; otherwise a job id with surrounding spaces never matches its own listing and the running job is failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: skip orphan misses once a completion was handed over The buffer length only shows a completion the dispatcher has not taken yet; after it takes one, the attempt stays registered until the dispatch returns. Mark the hand-off under the pending lock and check that flag instead, so heartbeats in that gap cannot fail a finished attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3d411c6a6c |
volume: report EC file and delete counts in incremental shard messages (#12404)
* seaweed-volume: report EC file and delete counts in heartbeat deltas The full EC heartbeat already carries file_count and delete_count, but the incremental new/deleted shard messages built in collect_ec_shard_delta_messages left both at zero. A volume the master first learns of through a delta then reads 0 files until the next full EC heartbeat. Fill both from file_and_delete_count, the same source the full heartbeat uses, computed once per volume rather than once per shard. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * volume: report EC file and delete counts in mount notifications Go parity with the Rust delta change: the NewEcShardsChan messages sent on MountEcShards and on startup shard load now carry FileCount and DeleteCount from EcVolume.FileAndDeleteCount, the same values the full EC heartbeat reports, instead of leaving them at zero. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * seaweed-volume: leave count-only EC changes out of the shard delta The delta messages now carry file_count and delete_count, so the whole-message comparison re-sent every shard of a volume after any delete, and the master broadcast each one as a new EC vid. Compare the shard fields only; the full EC heartbeat already carries the counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * topology: mount messages now carry EC counts, only older servers omit them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: guard the captured EC mount messages in the disk-type tests The drain goroutine appended to the slice that waitForShardMsg polled with no lock, which go test -race reports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: test EC counts on the startup notification and repeated .ecj ids Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: move file_count too in the count-only delta test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |