mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-07 04:27:49 +01:00
4bb74382a6a9b5fcb84f6b2fd3eb121de484b44e
16078
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
4bb74382a6 |
filer.remote.sync test: give the same-second saved offset a fractional part
With whole seconds, returning the creation second resolved to the same time and passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
39ef551bcf |
filer.remote.sync: keep the resume position for the retry when its write fails
The retry in the same process now resumes from the position the ended subscription returned, as filer.backup does, so a failed offset write followed by an empty read cannot recompute -timeAgo past a failed event. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a0d60ee00f |
filer.remote.sync: say that any saved offset wins over the creation time
The creation time in seconds was compared with the offset in milliseconds, which made any saved offset win. That is the behaviour to keep: remote.unmount recreates the directory and then saves the offset, usually in the same second, so a seconds comparison would start at the creation second and replay the purge. The comparison now states it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
f888ac6b8f |
remote_storage: bound the sync offset write like the read
The offset write runs in the subscription's checkpoint callback and, now, when a subscription ends; a filer that never answers it stalled either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
7be919ded1 |
filer.remote.sync: save the resume position when a subscription ends
A failed event ends the subscription with ErrResubscribe, and the retry loop runs startup again, recomputing -timeAgo from a later now. The periodic checkpoint only runs as events arrive, so a failure before the first one could start the retry after the event it failed on. The resume position is now saved whenever the subscription returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
f535fca8f6 |
filer.remote.sync: take the start time before the mount lookup
A bucket created between a not-found lookup and the end of the offset read had its creation before a start taken afterwards, so the gateway never created its remote bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
c31497a9a3 |
filer.remote.sync: a directory not created yet starts now, as before
A gateway started before the first bucket looked up a missing buckets directory, had no saved offset, and returned an error to the retry loop on every attempt until a bucket appeared. A missing directory with no saved offset has nothing to replay, so it starts now as it did before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
1f332e5866 |
filer.remote.sync: stamp a copy of the event entry
updateLocalEntry set the RemoteEntry and dropped the rename marker on the caller's entry before the RPC. A transient stamp failure retries the same event in place, which then read as already synced and skipped both the upload and the stamp. The stub filer now applies a successful update the way the filer does, since tests had relied on that shared object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
0a1b6c2944 |
filer.remote.sync: an unread offset never resolves to the mount's creation time
With the offset read now bounded, a timed-out read fell through to the directory's creation time and replayed every event already synced. The error goes back to the startup retry loop, as with -timeAgo; a missing offset still reads as zero without an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
c7338fb68f |
filer.remote.sync: bound the stamp written after each upload
updateLocalEntry's UpdateEntry was the last filer call on the sync path without a deadline. A filer that never answers the stamp keeps the job and its path in the conflict index the same way an unanswered lookup did. The stamp carries IF_ENTRY_EQUAL, so a retried event after a timeout either stamps again or skips a stamp that already landed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
51f603b5ec |
filer.remote.sync: isSuperseded returns the lookup error itself
Its only production callers needed the error, so the boolean wrapper was left to the tests. The tests now assert the error on a failed lookup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
952d80c4d3 |
filer.remote.sync: keep doc comments on the functions they describe
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
james
|
da377157ef |
filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting
resolveLastSyncOffset returns the lookup or offset-read error; both callers (directory and buckets) hand it to util.RetryUntil, which retries with backoff instead of the process exiting. |
||
|
james
|
2bcb748a67 |
filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo
After a failed upload attempt the supersession lookup's error ended in isSuperseded's boolean and the loop could upload the snapshot again unverified. The lookup error now ends the loop (errLookupUnanswered, not retried) and the processor retries the event. With -timeAgo set, a failed read of the saved offset resolved to now minus timeAgo, skipping everything older; the daemon exits instead so a restart retries. (cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566) |
||
|
james
|
db4ea51fd2 | filer.remote.sync test: keep the slices import the other gateway tests use | ||
|
james
|
e4c0acfa61 |
filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read
retriedWriteFile returns the supersession lookup's error instead of treating an unanswered lookup as "not superseded": in a replay an older snapshot could otherwise overwrite a newer remote object. The processor retries the event. isSuperseded keeps its boolean form for the other call sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls back to the saved offset cannot hang on it. TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed lookup; it now asserts the withheld upload and the lookup error. (cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa) |
||
|
james
|
4f8f817461 |
filer.remote.sync: a failed mount lookup at startup never resolves to now
collectLastSyncOffset returned time.Now() when the mounted directory lookup failed, which skips every event since the saved offset. The saved offset is used instead; without one the daemon exits so a restart retries. (cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae) |
||
|
james
|
3b8c4702bd |
filer.remote.sync: bounded filer lookups, one chunk cache per process
The filer lookups on the sync path (currentEntry, isSuperseded, uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no deadline. A lookup that never answers keeps its job alive, the job keeps its path in the MetadataProcessor conflict index, AddSyncJob waits on the condition and the daemon stops consuming events with no log line. Each lookup now carries a 30 s deadline, so the job fails and the processor's stop-and-resubscribe path runs. remoteUploadReader built a chunk cache per upload; the cache's worker goroutine is never stopped, so each upload leaked one goroutine and its cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory limit). One chunk cache per process. (cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78) |
||
|
github-actions[bot]
|
0dd33f5f0f | docs: regenerate star history chart | ||
|
Chris LuandClaude Opus 5.5
|
878979af70 |
filer: hide write controls in the UI on the read-only port (#12429)
* filer: hide write controls in the UI on the read-only port The read-only port accepts only GET and HEAD, but its directory page offered New Folder, Upload, drag-and-drop, rename and delete, each of which failed with 405. Requests that come through the read-only handler now render the page without those controls or the upload script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test the directory page controls on each port Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: match the rendered row buttons in the port test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e44c16802d |
weed/server: verify data shards against bitrot sidecar before ec decode (#12422) (#12427)
* weed/server: verify data shards against bitrot sidecar before ec decode (#12422) * ec.decode: verify data shards against the mounted volume's bitrot protection The decode looked up only the generation-0 .ecsum beside the data base and each location's data directory, so it missed a sidecar kept with the index and ignored the generation the volume actually serves. Take the protection the volume's per-disk runtimes resolved instead (On, else Invalid, else Off), and test a corrupt block past the first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.decode: check for live entries before verifying the shards A volume whose needles are all deleted gets its shards purged on the no-live-entries status; a failing checksum must not keep it around. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: verify data shards against the bitrot protection before a decode Mirrors the Go volume server: the decode takes the protection the volume's per-disk runtimes resolved and refuses a data shard that fails its .ecsum checksum, after the no-live-entries check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: a decode refuses a data shard that fails its .ecsum checksum Runs against both volume server implementations: the refused decode keeps every shard and the sidecar, and the needle stays readable through EC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.decode: reload the bitrot sidecar before verifying the shards A rebuild's backfill writes the sidecar without reloading the mounted runtimes, which then still report no protection; resolve it again across the EC metadata directories, as a shard mount does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: reload the bitrot sidecar before verifying the shards Mirrors the Go volume server: the decode job resolves the sidecar again on every runtime of the volume before choosing its protection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: a decode finds a sidecar written after the shards were mounted Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: read through the EC shards after the refused decode The source volume stayed mounted, so the final read never touched the shards it is meant to check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.decode: refuse when a sidecar reload reports another layout ReloadBitrotSidecar logged and dropped the layout-mismatch error, so the runtime stayed Off and the decode skipped the data-shard check. It now returns that error and the decode refuses on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a decode when a sidecar reload reports another layout Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: a decode refuses a late sidecar that records another layout Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.decode: refuse when an unmount left no runtime to resolve the sidecar Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a decode when an unmount left no runtime to resolve the sidecar Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
80f32bc074 |
weed/command: skip remote bucket delete when bucket is unmounted (#12418) (#12425)
* weed/command: skip remote bucket delete when bucket is unmounted * remote_storage: count deleting an already-missing bucket as done filer.remote.gateway can see the same bucket delete more than once, for example once per filer when several filers peer their metadata. A copy that still finds the mount mapping calls DeleteBucket again, and S3 (NoSuchBucket), Azure (ContainerNotFound, ContainerBeingDeleted) and GCS (404) all failed it, which held the sync offset on the delete. A missing bucket now counts as deleted, the way deleting a missing object already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: keep the mapping table writable after mount.mapping is deleted Deleting mount.mapping replaced the gateway's mappings with a table whose map was nil, so the next bucket create panicked writing its mount into it. Reset to an empty map instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: count a delete of an absent object as done filer.remote.sync already treats ErrRemoteObjectNotFound from DeleteFile as deleted, since GCS reports a missing object that way where S3 and Azure succeed. The gateway returned it, so a delete event seen twice, as it is when several filers peer their metadata, held the sync offset on GCS. Use the same deleteRemoteFile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: drop cached remote clients after the new gateway tests GetRemoteStorage caches clients by configuration, so a repeated run (-count=2) got the first run's recorder back and saw no calls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: keep the default storage across mount.mapping changes The default storage for new buckets is derived at startup and kept only in the gateway's copy of the mappings; mount.mapping never stores it. Replacing or deleting that file dropped it, so a later bucket create with no -createBucketAt skipped the remote bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
c971252871 |
admin: make the cluster lock's fairness window configurable (#12426)
* admin: make the cluster lock's fairness window configurable Scheduled jobs hold the cluster admin lock for their whole run. Once the lock had been held for a minute, every new job waited until all running jobs had released it, so with hour-long EC jobs dispatch ran in waves and most workers sat idle between them. -adminLock.fairnessWindow keeps the one-minute default; 0 lets new jobs join the shared hold, and shell clients get the lock when the scheduler next drains. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * admin: wait for the blocked acquire before disabling the fairness window The wake-up test slept 50 ms before SetFairnessWindow(0), so a late acquire that started after the change joined without exercising the Broadcast. Wait until it is registered as a waiter instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8c8d0f1cdf |
weed/shell: resolve collection from topology when ec.decode is given only volumeId (#12424)
* weed/shell: resolve collection from topology when ec.decode is given only volumeId
When ec.decode is invoked with -volumeId for a volume that belongs to a named collection without explicitly specifying -collection, DoEcDecode was called with an empty collection string. Target volume servers then rejected VolumeEcShardsCopy with 'CopyFile not found ec volume id <vid>' because they looked under the default collection.
This commit introduces FindEcVolumeCollection to discover the collection name directly from topology metadata (checking EC shard info and regular volume info), ensuring DoEcDecode receives the accurate collection.
* ec.decode: resolve every batch volume's collection from the topology
The batch passed -collection through unless it looked like a regex, so a
comma list (a,b), a {n} quantifier, ALL or _default_ reached the volume
servers as a collection name and the copy found no shards. The selector
only picks the volumes; each one is decoded in its own collection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: take the collection only from the volume's ec shards
The lookup returned the first record per disk, so a regular volume on an
earlier node answered before the ec shards on a later one. A decode needs
the shards anyway, so their records are the only ones to read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: leave the -volumeId lookup to DoEcDecode
DoEcDecode already resolves an empty collection from the topology.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
120d17a092 |
filer.remote.sync: stop a touched file from being restamped forever (#12419)
* filer.remote.sync: stop a touched file from being restamped forever A metadata-only update that moves the mtime past RemoteMtime (touch, or an O_TRUNC open of an empty lock file) goes through UpdateFileMetadata, which changes nothing on the remote, and the entry is stamped with the same RemoteEntry and a new LastLocalSyncTsNs. That stamp is an update event of its own. shouldSendToRemote only looks at LastLocalSyncTsNs when RemoteMtime and the mtime share a second, so it still reads the entry as dirty, and the event is stamped again. Nothing is uploaded, but the daemon keeps rewriting the entry as fast as it reads its own events back. Skip the restamp when the backend left the object alone and the event's entry already carries a stamp taken after its mtime. The touch itself is still stamped once. * filer.remote.sync: log the skipped restamp The other skips in processUpdateEvent log at V(2); this one did not, so with -v=2 a touched file showed an "update meta" line with nothing after it. * filer.remote.sync: skip the restamp of a stamp's own event, whatever the clocks The skip compared the entry's stamp with its mtime, so an mtime ahead of the sync's clock (a skewed writer, touch -d) stayed past every stamp and kept the loop going until the clock caught up. An event that changed only the RemoteEntry is the sync's own stamp, which needs no other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: ignore the filer's commit timestamp when spotting a stamp The filer gives every stored update a new CommitTsNs, so a stamp's event never compared equal outside the RemoteEntry and a touched file kept being restamped against a real filer. The test now assigns one per update. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
23598afb31 |
volume: bound each replica forward so a silent peer cannot hold a write (#12417)
* volume: bound replica forwards so a silent peer cannot hold a write The Rust volume server forwards each replicated write to its peers with the shared outgoing reqwest client, which had no connect or request timeout. When a peer is silently partitioned (packets dropped, no reset) the forward waits until TCP gives up, minutes later, and the write waits with it. A chaos test (replication 001, one volume server partitioned) saw every write on a volume with a copy there block. - Replica forwards go through replica_request, bounded at REPLICA_REQUEST_TIMEOUT = 10 s from connect to the response. - Both outgoing clients start from outgoing_client_builder, which adds a 5 s connect timeout, so new connections to a peer that never answers fail fast. The shared client gets no overall request timeout, since it also serves lookups and other transfers. Tests: a replica that receives the forward and never answers fails the whole replicated write at the bound, naming that replica; a silent peer fails the request builder at the bound on a paused clock; a connect that stalls in the resolver fails at the 5 s connect bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: give a replica forward more time for a larger body The 10 s replica forward bound runs from connect to the end of the response, so it also covers sending the body. A write near the 256 MB file size limit needs about 26 MB/s to reach its replica in 10 s, so on a link that is slower or busy the forward failed and the write returned 500 even though the peer was answering. A forward now gets REPLICA_REQUEST_TIMEOUT plus one second per MiB of body. A bodiless forward keeps the 10 s bound (a 1 MiB write gets 11 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: bound each Go replica forward like the Rust one The Go volume server forwards a replicated write with a 10 s dial bound and the transport's 30 s response header timeout. A replica that takes the forward and never answers held the write for 30 s, and a body larger than the socket buffers to a silently partitioned peer was held until TCP gave up, since nothing bounds sending the body. Each forward now runs under replicaRequestBound: 10 s plus one second per MiB of body, the same bound the Rust volume server puts on its forwards. Test: a replica that receives the forward and never answers fails the write at the bound (300 ms in the test); without it the write fails at 30 s on the response header timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: keep proxy variables out of the stalled-connect test reqwest reads HTTP_PROXY and ALL_PROXY by default, so with a proxy set the test's request went to the proxy instead of the stalled resolver. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: count a replica forward that hit its bound as a timeout DistributedOperationResult joined the replica errors into a plain string, and the uploader reports a cancelled request without wrapping the context error, so a forward cut off at replicaRequestBound was counted under server_error in the replication failure metric, as was a response header timeout before it. The joined error now unwraps to each replica's error, and a forward that ran out of its bound reports context.DeadlineExceeded. Test: the silent replica write raises the timeout failure count by one; with the flattened join it stays put. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: test that a large Go replica forward gets its body's bound A replica answers after the bodiless bound but within the bound for a 1 MiB body, so the write fails if replicateTo drops the body scaling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: run the Rust whole-path replica test with a 1 MiB body The forward then has to wait the bound for its body, so the test fails if do_replicated_request stops passing the body length to the bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: pingqiu <pingqiu@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
aab947b65e |
s3api: answer success when a failed completion already committed (#12416)
* s3api: name the check that a completion already committed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: answer success when a failed completion already committed A finalize can fail after its commit: the filer stored the object and the call still reported an error. The client then got a 500 with an unknown outcome. When the failure may have left this attempt's object behind, check the key the way a retry would and answer success with the committed result when it holds this upload's object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: keep part data when the commit found is another attempt's The object carrying this upload ID can be a concurrent attempt's commit of the same upload, and its part selection can include entries this attempt counts as unused. Free those parts' data only when the committed chunks are this attempt's own; otherwise drop their metadata only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test the committed-completion read on versioned buckets Cover the latest-version read (version id and checksum in the answer), a suspended bucket whose delete marker is still current, and that the upload directory is removed without its data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test versioned recovery through the completion handler Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
fe1cf5263f |
volume: bound each FetchAndWriteNeedle replica write (#12415)
* seaweed-volume: bound each FetchAndWriteNeedle replica write A replica that accepted the connection and then went silent held the request until TCP gave up, because the shared outgoing client has no request timeout. Each replica write now fails after 30 s, from connect to the response status, matching the Go volume server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: bound each FetchAndWriteNeedle replica write The default replica path uploads through the guarded peer client, which bounds only the dial and has no response header timeout, and the filer calls FetchAndWriteNeedle with no deadline. A replica that accepted the connection and went silent held the request for minutes. Each replica write, retries included, now has 30 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: require the silent-peer test to fail at the deadline A connection error that came back early also passed, since the test only checked an upper bound. It now also requires the paused clock to have reached FETCH_AND_WRITE_REPLICA_TIMEOUT. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: grow the FetchAndWriteNeedle replica bound with the chunk The reqwest timeout includes sending the body, and a remote-cache chunk is a thousandth of the object once it passes the 1000-chunk cap, so a fixed 30 s would fail a large chunk on a link under ~35 MB/s. The bound is now 30 s plus 1 s per MiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: grow the FetchAndWriteNeedle replica bound with the chunk The context deadline includes sending the body, so a fixed 30 s would fail a large remote-cache chunk on a slow link. The bound is now 30 s plus 1 s per MiB, matching the Rust volume server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
247e010865 |
volume: redial a master that just dropped a registered stream after a second (#12420)
* volume: redial a master that just dropped a registered stream after a second A registered volume server that loses its master with no peer naming a leader waited a full pulse before redialing. A restarted master is usually back within a second, so its volumes stayed unregistered, and uncached lookups kept failing, for most of a pulse. For one pulse after a stream a leader had answered ends, redial on the one-second cap. A master still down after that is redialed at the pulse pace, so redials never come later than before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: redial a master that just dropped a registered stream after a second Mirror of the Go heartbeat change. The loop keeps when the last stream a leader had answered ended, which also stands in for the confirmed-leader-once flag (None until first registration), and redials on the one-second cap for a pulse after that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: test that a dropped registered stream records the loss A fake master confirms the stream and then ends it. The server starts as registered long ago, so only a recorded loss makes the redial come after a second instead of a pulse. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: record a lost stream only after its receiver has exited doHeartbeatWithRetry can return on a send error while the receiver is still handling the leader's first reply, so the deferred recorder could read the registered flag too early and keep an old loss time. Cancel the stream and wait for the receiver before reading it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: require the prompt heartbeat redial to wait out its second Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: fail the down master's dials at once in the redial test Windows takes about two seconds to refuse a connection to a closed loopback port, so the stream dial and the leader lookup each held the prompt case past its two-second bound. A listener that closes every connection fails both dials immediately on every platform. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
8cc3a8ae21 |
ec.rebuild: repair shards that fail a checksum scrub in complete volumes (#12410)
* ec.rebuild: rebuild shards that fail a checksum scrub, also in complete volumes ec.rebuild skipped every volume with all shards present, so a shard whose bytes no longer match the .ecsum sidecar stayed in place until some other shard went missing. With -checksum, ec.rebuild first scrubs each node's shards in checksum mode and takes the failing copies out of the plan; a shard left without a good copy counts as missing and is regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: remove the copies that failed the checksum once a good copy serves A shard regenerated on the rebuilder left the failing copy mounted on its old holder, so the volume kept a corrupt duplicate that a later dedup could keep instead of the good one. Remove each failing copy once the rebuilt shard is mounted, and right away when another holder already serves a copy that passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: test the checksum scrub against fake volume servers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: check the lock before removing a corrupt copy Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: print the checksum scrub details A malformed sidecar or a wholesale mismatch comes back only in the scrub details, with no broken shard, so the volume would look verified. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: fail the run when a corrupt duplicate cannot be removed For a volume whose only problem is a corrupt duplicate, the removal is the whole repair, so its failure has to reach the caller. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: test duplicate removal failures and a lost lock Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: fail the run when a node's checksum scrub fails The node's copies stay in the plan unverified, so a corrupt shard there is left in place; the rest of the cluster is still repaired. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: preview the corrupt-copy removals in dry-run Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: test a failed checksum scrub Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
fcf62db8d0 |
seaweed-volume: stage optional EC sidecar copies before replacing (#12403)
* seaweed-volume: stage optional EC sidecar copies before replacing VolumeEcShardsCopy opened the destination .vif/.ecsum with File::create and then streamed into it. A source without the file left a 0-byte .vif, wiped an existing .ecsum (turning bitrot protection off on that holder), and a failed stream removed the destination's existing copy. Stream optional copies into <file>.copying and rename into place only once the source sent bytes, as the Go volume server's writeToFile does for ignore_source_file_not_found. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * seaweed-volume: keep the drain doc comment on drain_copy_stream_to_file Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: stage the .vif copied during missing-index recovery fetch_ec_index_from_one_peer drained the optional .vif straight into the local file, so a peer without one left a 0-byte .vif (losing a local EC ratio) and a failed stream left a partial one. Stage it and rename only once the peer sent bytes, as Go's recovery does through writeToFile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: test optional copies against a stream that fails mid-transfer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: remove a staged optional copy when the RPC is cancelled A cancelled CopyFile handler drops its future at the stream await, so the cleanup after it never ran and the .copying file stayed behind. Go's writeToFile runs to completion and removes it. A guard now owns the staging cleanup and is disarmed only once the rename commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: test optional copies dropped mid-stream Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: cancel the staged-copy test only once the staging file exists tokio's File::create runs on the blocking pool, so a fixed timeout could drop the .ecsum copy before .copying existed and pass without exercising the cleanup guard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
85232f7414 |
seaweed-volume: run the preallocating max-volume test on a fake disk (#12423)
* seaweed-volume: run the preallocating max-volume test on a fake disk The test sized the volume limit from the real free space but the store subtracts the 1% floor first, so on a disk more than 95% full the second slot no longer fit and the count came out 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: start the preallocating max-volume test above the expected count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b74111ea2f |
s3vectors test: skip a query lease tag released between list and HEAD (#12409)
A query's lease tag is deleted by a background task after the query answers, so the encryption helper could list it and then get a 404 on HEAD. Skip only that case; any other missing object still fails. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Fable 5.1
|
48df16ea0d |
weed-worker: keep a vector index covering as its table grows (ANN phase 3) (#12414)
* seaweed-vector: build and extend the vector index weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the table was created with, sizes partitions from the row count, and extends an index with a new segment or a retrain; the core crate re-exports it and adds the probes a query spends. Tests cover coverage after build, append, retrain and compaction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector-cli: index build and index optimize Logs go to stderr so stdout stays the JSON a caller parses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_INDEX builds the index before the timed queries The cluster fixture hands out its catalog URL, the Makefile builds weed-vector-cli, and the perf suite builds the index after ingest and reports the build time and segment count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: probe a tenth of the index's partitions, more under a filter A query that names no probe count gets one from the index's partition count. Lance searches at least that many partitions, and under a prefilter keeps going until it has k rows; without one it stops there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: --nprobes overrides the probe policy Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that a build takes the table's metric Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make clean removes weed-vector-cli Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: measure recall with vectors the table does not hold Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep the read cache's doc comment on with_read_cache Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold Detection keeps the tables a vector bucket created, with at least min_rows rows (50,000 by default) and no index on their vector column. Execution builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: optimize only the indices on the vector column Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: resolve the query metric with index::distance_metric Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test the L2 metric under the name CreateIndex stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: build the index through the admin server and the Lance worker VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new test sets the job's threshold, runs it through the plugin API, checks coverage and recall, and that a row written after the build is found. CI gains the worker leg for both volume servers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: read an index's partitions, and when a table has outgrown them Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: probes from the shared partition reader, partitions in coverage Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index reindexes at a tenth of its rows and retrains once the table outgrows it lance_optimize_indices proposes a vector table once its uncovered rows pass min(the absolute budget, a tenth of the indexed rows), and trains the index again instead of extending it when sqrt(rows) has reached twice its partitions. The end-to-end test covers both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: open a table inside a caller's Lance session Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep opened indices between queries in one session Every open used a fresh Lance session, so the index and metadata caches died with the request and each query loaded the IVF index again. --index-cache-mb sizes the shared cache, 512 MiB by default. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the index test draws embedding-like vectors Independent Gaussian vectors have no locality for an IVF index to use, so recall through one only tracks the fraction of partitions probed; points around shared centres behave like embeddings. The test must also run before the one that stops the engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the optimize and retrain steps draw embedding-like vectors Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that compaction drops the deleted key from tied hits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test that an append-only optimize serves the new row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: record PERF_NOISE in the perf report config Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: a filtered query widens to four times the probes, not without bound With no upper bound Lance prepares every partition for the late search even when the minimum probes already hold k rows, and a filtered query through the index cost as much as the exact scan. A query that names its own probe count gets exactly that count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: report recall through an index without asserting it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a hit tying the k-th best score toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: lance_vector_index runs in the Lance lane Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: check a table's index before counting its rows Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make build and clean cover the worker leg's binaries Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a retrain rebuilds the index with partitions for the row count Lance's OptimizeOptions::retrain does not size the partitions: it left a 6,000-row index with one. The retrain goes through the builder with sqrt(rows) partitions, for the job and for weed-vector-cli index optimize --retrain alike. The end-to-end test no longer expects a merge to keep the partition count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: allow float32 rounding at the recall cutoff Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a repeated hit key once toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: credit cutoff ties only up to the slots the top-k leaves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: assert every filtered hit passes the filter, indexed or not Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: coverage over an index's names, one name extended, a merge at the delta limit A sharded build leaves one name per shard, each missing the others' fragments, so coverage is the union of every index on the column and an optimize gives new rows to the name with the fewest deltas only. At MAX_DELTAS the policy passes OptimizeOptions::merge(deltas); Lance's default options may append instead of merging. A retrain keeps an index's own metric and rebuilds only IVF_FLAT indices. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: the optimize job measures and extends a vector table's indices together Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: tests for bounded deltas, disjoint shards and a retrain's metric Nine append-and-optimize cycles give delta counts 2 to 8 and then 1; two names on disjoint fragments stay disjoint through an optimize; a cosine index on a table without metric metadata is still cosine after a retrain. The end-to-end test expects the appended delta after the first optimize. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: forget the shared session when a table was dropped and created again The session keys index lists and transactions by dataset version, and a table created again at the same location restarts its versions. An open that finds an older version than the newest seen at that URI, or the same version committed later, replaces the session and opens again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: test an open after another engine created the table again Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: retrain only a single IVF_FLAT vector index name The job compared each shard name's partitions with the whole table's rows, so a sharded index was always due, and a non-IVF_FLAT index made optimize bail on every run before the new rows were indexed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: refuse to retrain a vector index sharded over several names A rebuild per name over the whole table left every shard covering every fragment, so a query searched each row once per shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test which vector indices a retrain is due for Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: replace the session in the same lock that records a recreated table A second open of the recreated table could see the new version recorded and the old session still shared, and keep the dropped table's index list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: skip a vector table whose rows cannot be counted Detection returned on the first table whose count_rows failed, so one unreadable table kept every other table from being proposed. It now skips that table like an open or index failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: panel for vector indexes the Lance worker built Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: size default probes from the largest index on the column A sharded build gives the column one index name per shard. The first name alone could be a small shard, and its probe count would then be spent on a larger one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: skip the probe lookup when a query is given no segments Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: open every table in its own Lance session again One session shared by every open keeps a table's index loaded between queries, but Lance keys what it caches by location and version, so a table dropped and created again at the same location could be served the old table's index list. Telling the two apart from version and commit time needs a map that only grows, a retry loop and clocks that agree across engines. A private session per open, as before, has no such state. Measured at 50k x 384 through an index, eight clients: 164 queries/s with the shared session, 137 without; one client is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: reject --nprobes 0 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: read the partition count from the opened index Lance's index statistics took 3.6 ms a query on a 200k-row table with 447 partitions. Opening the index and asking it takes 0.4 ms, and the scan that follows finds the index already open. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * ci: give the S3 Vectors worker legs the vendored protoc The worker workspace has no protoc of its own for lance's build scripts, so the worker legs failed at their first build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: give a vector index build an hour before the admin times it out The job set no execution timeout, so the admin's default of 90 seconds applied, and a build that trains on a large table would be reported as failed and retried while it was still running. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_vector_index proposes at most the admin's max_results Every qualifying table went into one message however many there were. The detector stops at the limit the request carries and sets has_more, as the Go detectors do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index job re-checks the table it was proposed for A table emptied since detection completes without a build, and one without a vector column fails instead of reporting it already indexed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_optimize_indices gets a build's execution timeout A retrain rebuilds the index over the whole table, which the admin's default of 90 seconds would cut off on a large one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index run window as long as one build An attempt is capped at the end of the job type's run window plus the drain grace, and the default window is 30 minutes, so a build could be cut off before its own hour was up. The window is an hour too, as the Go iceberg handler sets both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a lance_optimize_indices run window as long as one retrain Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: vector index proposals declare a build's runtime A run started by hand while the job type's schedule is off takes no timeout from the descriptor, only the admin's 90 seconds. The admin gives a job at least the runtime its proposal declares, so the build and the optimize proposals of a vector table declare the hour. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: coverage reports no partitions for an index of unknown coverage Reading the partition count goes through Lance's index statistics, which fail on an index that records no fragment bitmap, and the whole coverage report failed with them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: an index that records no fragments is not read as uncovered Its rows counted as uncovered, so detection proposed the table and the execution then failed in Lance's statistics. Coverage fails for such an index instead, and detection skips the table with a warning. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: coverage reads partition counts from the opened index Lance's index statistics rewrite the manifest of a table whose fragments carry no row counts, which a coverage report must not do and cannot do with read-only credentials. The report asks each opened segment instead, as the probe lookup does, and the statistics-based reader has no caller left. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Fable 5.1
|
e465880ee8 |
weed-worker: build the vector index of S3 Vectors tables past a row threshold (ANN phase 2) (#12413)
* seaweed-vector: build and extend the vector index weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the table was created with, sizes partitions from the row count, and extends an index with a new segment or a retrain; the core crate re-exports it and adds the probes a query spends. Tests cover coverage after build, append, retrain and compaction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector-cli: index build and index optimize Logs go to stderr so stdout stays the JSON a caller parses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_INDEX builds the index before the timed queries The cluster fixture hands out its catalog URL, the Makefile builds weed-vector-cli, and the perf suite builds the index after ingest and reports the build time and segment count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: probe a tenth of the index's partitions, more under a filter A query that names no probe count gets one from the index's partition count. Lance searches at least that many partitions, and under a prefilter keeps going until it has k rows; without one it stops there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: --nprobes overrides the probe policy Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that a build takes the table's metric Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make clean removes weed-vector-cli Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: measure recall with vectors the table does not hold Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep the read cache's doc comment on with_read_cache Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold Detection keeps the tables a vector bucket created, with at least min_rows rows (50,000 by default) and no index on their vector column. Execution builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: optimize only the indices on the vector column Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: resolve the query metric with index::distance_metric Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test the L2 metric under the name CreateIndex stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: build the index through the admin server and the Lance worker VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new test sets the job's threshold, runs it through the plugin API, checks coverage and recall, and that a row written after the build is found. CI gains the worker leg for both volume servers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: open a table inside a caller's Lance session Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep opened indices between queries in one session Every open used a fresh Lance session, so the index and metadata caches died with the request and each query loaded the IVF index again. --index-cache-mb sizes the shared cache, 512 MiB by default. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the index test draws embedding-like vectors Independent Gaussian vectors have no locality for an IVF index to use, so recall through one only tracks the fraction of partitions probed; points around shared centres behave like embeddings. The test must also run before the one that stops the engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that compaction drops the deleted key from tied hits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test that an append-only optimize serves the new row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: record PERF_NOISE in the perf report config Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: a filtered query widens to four times the probes, not without bound With no upper bound Lance prepares every partition for the late search even when the minimum probes already hold k rows, and a filtered query through the index cost as much as the exact scan. A query that names its own probe count gets exactly that count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: report recall through an index without asserting it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a hit tying the k-th best score toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: lance_vector_index runs in the Lance lane Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: check a table's index before counting its rows Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make build and clean cover the worker leg's binaries Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: allow float32 rounding at the recall cutoff Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a repeated hit key once toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: credit cutoff ties only up to the slots the top-k leaves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: assert every filtered hit passes the filter, indexed or not Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: forget the shared session when a table was dropped and created again The session keys index lists and transactions by dataset version, and a table created again at the same location restarts its versions. An open that finds an older version than the newest seen at that URI, or the same version committed later, replaces the session and opens again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: test an open after another engine created the table again Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: replace the session in the same lock that records a recreated table A second open of the recreated table could see the new version recorded and the old session still shared, and keep the dropped table's index list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: skip a vector table whose rows cannot be counted Detection returned on the first table whose count_rows failed, so one unreadable table kept every other table from being proposed. It now skips that table like an open or index failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: panel for vector indexes the Lance worker built Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: size default probes from the largest index on the column A sharded build gives the column one index name per shard. The first name alone could be a small shard, and its probe count would then be spent on a larger one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: skip the probe lookup when a query is given no segments Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: open every table in its own Lance session again One session shared by every open keeps a table's index loaded between queries, but Lance keys what it caches by location and version, so a table dropped and created again at the same location could be served the old table's index list. Telling the two apart from version and commit time needs a map that only grows, a retry loop and clocks that agree across engines. A private session per open, as before, has no such state. Measured at 50k x 384 through an index, eight clients: 164 queries/s with the shared session, 137 without; one client is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: reject --nprobes 0 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: read the partition count from the opened index Lance's index statistics took 3.6 ms a query on a 200k-row table with 447 partitions. Opening the index and asking it takes 0.4 ms, and the scan that follows finds the index already open. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * ci: give the S3 Vectors worker legs the vendored protoc The worker workspace has no protoc of its own for lance's build scripts, so the worker legs failed at their first build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: give a vector index build an hour before the admin times it out The job set no execution timeout, so the admin's default of 90 seconds applied, and a build that trains on a large table would be reported as failed and retried while it was still running. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_vector_index proposes at most the admin's max_results Every qualifying table went into one message however many there were. The detector stops at the limit the request carries and sets has_more, as the Go detectors do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index job re-checks the table it was proposed for A table emptied since detection completes without a build, and one without a vector column fails instead of reporting it already indexed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index run window as long as one build An attempt is capped at the end of the job type's run window plus the drain grace, and the default window is 30 minutes, so a build could be cut off before its own hour was up. The window is an hour too, as the Go iceberg handler sets both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Fable 5.1
|
b5951c5c7c |
seaweed-vector: probe policy for indexed queries (ANN phase 1) (#12412)
* seaweed-vector: build and extend the vector index weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the table was created with, sizes partitions from the row count, and extends an index with a new segment or a retrain; the core crate re-exports it and adds the probes a query spends. Tests cover coverage after build, append, retrain and compaction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector-cli: index build and index optimize Logs go to stderr so stdout stays the JSON a caller parses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_INDEX builds the index before the timed queries The cluster fixture hands out its catalog URL, the Makefile builds weed-vector-cli, and the perf suite builds the index after ingest and reports the build time and segment count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: probe a tenth of the index's partitions, more under a filter A query that names no probe count gets one from the index's partition count. Lance searches at least that many partitions, and under a prefilter keeps going until it has k rows; without one it stops there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: --nprobes overrides the probe policy Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that a build takes the table's metric Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make clean removes weed-vector-cli Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: measure recall with vectors the table does not hold Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep the read cache's doc comment on with_read_cache Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: optimize only the indices on the vector column Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: resolve the query metric with index::distance_metric Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test the L2 metric under the name CreateIndex stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: open a table inside a caller's Lance session Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep opened indices between queries in one session Every open used a fresh Lance session, so the index and metadata caches died with the request and each query loaded the IVF index again. --index-cache-mb sizes the shared cache, 512 MiB by default. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that compaction drops the deleted key from tied hits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test that an append-only optimize serves the new row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: record PERF_NOISE in the perf report config Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: a filtered query widens to four times the probes, not without bound With no upper bound Lance prepares every partition for the late search even when the minimum probes already hold k rows, and a filtered query through the index cost as much as the exact scan. A query that names its own probe count gets exactly that count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: report recall through an index without asserting it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a hit tying the k-th best score toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: allow float32 rounding at the recall cutoff Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a repeated hit key once toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: credit cutoff ties only up to the slots the top-k leaves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: assert every filtered hit passes the filter, indexed or not Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: forget the shared session when a table was dropped and created again The session keys index lists and transactions by dataset version, and a table created again at the same location restarts its versions. An open that finds an older version than the newest seen at that URI, or the same version committed later, replaces the session and opens again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: test an open after another engine created the table again Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: replace the session in the same lock that records a recreated table A second open of the recreated table could see the new version recorded and the old session still shared, and keep the dropped table's index list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: size default probes from the largest index on the column A sharded build gives the column one index name per shard. The first name alone could be a small shard, and its probe count would then be spent on a larger one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: skip the probe lookup when a query is given no segments Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: open every table in its own Lance session again One session shared by every open keeps a table's index loaded between queries, but Lance keys what it caches by location and version, so a table dropped and created again at the same location could be served the old table's index list. Telling the two apart from version and commit time needs a map that only grows, a retry loop and clocks that agree across engines. A private session per open, as before, has no such state. Measured at 50k x 384 through an index, eight clients: 164 queries/s with the shared session, 137 without; one client is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: reject --nprobes 0 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: read the partition count from the opened index Lance's index statistics took 3.6 ms a query on a 200k-row table with 447 partitions. Opening the index and asking it takes 0.4 ms, and the scan that follows finds the index already open. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Fable 5.1
|
f0f0ba85ae |
seaweed-vector: build a vector index from the CLI and measure it in the perf suite (ANN phase 0) (#12411)
* seaweed-vector: build and extend the vector index weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the table was created with, sizes partitions from the row count, and extends an index with a new segment or a retrain; the core crate re-exports it and adds the probes a query spends. Tests cover coverage after build, append, retrain and compaction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector-cli: index build and index optimize Logs go to stderr so stdout stays the JSON a caller parses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_INDEX builds the index before the timed queries The cluster fixture hands out its catalog URL, the Makefile builds weed-vector-cli, and the perf suite builds the index after ingest and reports the build time and segment count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that a build takes the table's metric Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make clean removes weed-vector-cli Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: optimize only the indices on the vector column Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: resolve the query metric with index::distance_metric Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test the L2 metric under the name CreateIndex stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that compaction drops the deleted key from tied hits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test that an append-only optimize serves the new row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: record PERF_NOISE in the perf report config Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: report recall through an index without asserting it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a hit tying the k-th best score toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: allow float32 rounding at the recall cutoff Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a repeated hit key once toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: credit cutoff ties only up to the slots the top-k leaves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: assert every filtered hit passes the filter, indexed or not Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
8350f297af |
seaweed-vector: dev plan for ANN search over S3 Vectors indexes (#12408)
* seaweed-vector: dev plan for ANN search over S3 Vectors indexes Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: one probe count per query, a budget that can shrink Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: smaller deltas are searched in full only up to the probe count Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: drop the private issue reference Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: phase 1 probes use the Design count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: status after phases 0 to 3, and what the measurements changed Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: name the Enterprise repository, not the private one Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: the Enterprise repository by that name Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: a filtered query widens to four times the probes, not without bound Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: throughput through the index at 50k and 200k Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: union coverage over shard names, and an explicit merge at the delta limit Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: status table names branches, not pull requests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: maximum_nprobes is always four times the count Lance reads maximum_nprobes only under a prefilter, and the phase 1 policy sets it the same way for every query. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: the retrain follows the partition count, not a recorded row count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: phase 3 is open like the other phases Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: the retrain stops past 2,048 partitions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: dev plan: the merge at the delta limit takes the name's delta count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
github-actions[bot]
|
cd10b4a5c1 | docs: regenerate star history chart | ||
|
Chris Lu
|
683c5ad4d3 |
admin: regenerate the plugin pages' templ output from weed/admin (#12405)
Both were generated at the repo root, so their embedded source paths read weed/admin/view/... where every other admin template, generated by weed/admin's Makefile, reads view/.... No markup changes. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi |
||
|
Chris LuandClaude Opus 5.5
|
7047d3947a |
admin/plugin: fail a dispatch its restarted worker no longer lists (#12402)
* admin/plugin: fail a dispatch its restarted worker no longer lists A job dispatched to a worker that then restarts dies with the old process. The worker re-registers under the same id on a new stream and its heartbeats stop listing the job, but nothing ended the pending execution: the dispatcher kept waiting and held the executor slot until the attempt's execution timeout. Each pending execution now records the stream it went out on. An itemized heartbeat from a newer stream of that worker that leaves the job out twice in a row fails it through the expire path, so the dispatch returns, the slot is released and the retry machinery takes over. Jobs on the heartbeat's own stream, jobs the new stream still lists, and heartbeats that count slots without itemizing them are left alone. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * admin/plugin: let a partly itemized listing clear a dispatch's misses A heartbeat that counted more slots than it itemized returned before the listed jobs had their misses reset, so an omission, a partial heartbeat listing the job, and another omission failed a job the worker was still running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: look up the worker's stream before tracking the attempt A dispatch to a worker with no stream returned after the job was already recorded as assigned, and that record kept new proposals with the same dedupe key out until the stale-job sweep expired it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: fail the attempt whose request could not be sent The dispatch now sends on the stream it looked up before tracking the attempt, so a worker that reconnects in between makes the send fail. That returned with the job still tracked as assigned, and with the worker back nothing closed the record for the 24h stale timeout while its dedupe key kept new proposals out. The attempt is now completed as failed, after it is unregistered so the completion does not wait on a channel nobody reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: leave a send failure to the expiry that already claimed it The send-error path and an orphan expiry could both complete the same attempt and record two failed runs for it. The send-error path now completes the attempt only if no expiry has marked it failed, decided under the same lock that unregisters it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: record a send failure whose entry Shutdown already removed Shutdown closes shutdownCh before it unregisters pending dispatches, so a blocked send returns while its map entry is gone. Read the expiry claim from the dispatch's own state instead of the map lookup, so the attempt is still failed and does not stay assigned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: match a padded job id against its trimmed listing Heartbeat work ids are trimmed before the orphan check, so compare the dispatch's job id trimmed too; otherwise a job id with surrounding spaces never matches its own listing and the running job is failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: skip orphan misses once a completion was handed over The buffer length only shows a completion the dispatcher has not taken yet; after it takes one, the attempt stays registered until the dispatch returns. Mark the hand-off under the pending lock and check that flag instead, so heartbeats in that gap cannot fail a finished attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3d411c6a6c |
volume: report EC file and delete counts in incremental shard messages (#12404)
* seaweed-volume: report EC file and delete counts in heartbeat deltas The full EC heartbeat already carries file_count and delete_count, but the incremental new/deleted shard messages built in collect_ec_shard_delta_messages left both at zero. A volume the master first learns of through a delta then reads 0 files until the next full EC heartbeat. Fill both from file_and_delete_count, the same source the full heartbeat uses, computed once per volume rather than once per shard. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * volume: report EC file and delete counts in mount notifications Go parity with the Rust delta change: the NewEcShardsChan messages sent on MountEcShards and on startup shard load now carry FileCount and DeleteCount from EcVolume.FileAndDeleteCount, the same values the full EC heartbeat reports, instead of leaving them at zero. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * seaweed-volume: leave count-only EC changes out of the shard delta The delta messages now carry file_count and delete_count, so the whole-message comparison re-sent every shard of a volume after any delete, and the master broadcast each one as a new EC vid. Compare the shard fields only; the full EC heartbeat already carries the counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * topology: mount messages now carry EC counts, only older servers omit them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: guard the captured EC mount messages in the disk-type tests The drain goroutine appended to the slice that waitForShardMsg polled with no lock, which go test -race reports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: test EC counts on the startup notification and repeated .ecj ids Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: move file_count too in the count-only delta test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3cbace2471 |
s3: expose ReadAcp/WriteAcp in shell and admin UI, log ACL-upload denials (#12396)
* shell: accept ReadAcp and WriteAcp in per-bucket and service account actions s3.bucket.access, s3.anonymous.set and s3.serviceaccount.create rejected ReadAcp and WriteAcp, so the ACL actions the S3 gateway enforces could not be granted through them. Share one canonical action table across the three commands and list the ACL actions in their help. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: list ReadAcp and WriteAcp in s3.configure and s3.circuitBreaker help The -actions usage of both commands stopped at Admin, so the ACL actions the gateway enforces could only be found in the source. s3.circuitBreaker now prints the actions it actually accepts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: offer ReadAcp and WriteAcp in the object store user permissions The create and edit user forms only offered Read, Write, List, Tagging and Admin among the bucket permissions, so an identity could not be given the ACL actions from the UI even though uploads with x-amz-acl need WriteAcp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: log the WriteAcp denial of an upload that sets an ACL An upload or copy that carries x-amz-acl or an x-amz-grant-* header also needs WriteAcp. When that check denies, the client only sees a bare AccessDenied and nothing reached the log at the default verbosity, so the header was never suspected. Log the identity and the missing action. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: quote the key in the ACL upload denial log An object key can carry a decoded newline, which let a caller add forged lines to the default log. Quote the bucket and key, and word the message so it also fits a policy deny, not only a missing grant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: say ACL uploads need both Write and WriteAcp The s3.configure example granted WriteAcp alone, which still fails the upload's Write check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: parse S3 action lists with one shared helper s3.bucket.access, s3.anonymous.set and s3.serviceaccount.create each had their own validation loop, so a test of the table did not cover the parsing the commands run. They now call canonicalActionList, which skips empty entries and repeats. s3.anonymous.set validates before contacting IAM, and s3.bucket.access still rejects a list with no actions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep bucket-scoped actions the user form cannot show When a user has both global and bucket-scoped actions, the edit form shows only the global ones, and saving replaced the identity's actions with what the form rebuilt, deleting the scoped grants. Carry those actions through the save. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: list hidden bucket grants in the user edit form Restoring the bucket-scoped grants the form cannot show kept them active even when an admin narrowed the user's permissions, with no way to see or drop them. Show them as checked boxes under Other Bucket Permissions; unchecking one removes it on save. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: reject an s3.anonymous.set -access list with no actions A blank list such as ",," parsed to nothing and cleared the bucket's anonymous grants. Use "none" for that, as s3.bucket.access does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: skip the ACL upload denial log for anonymous requests Anonymous uploads with ACL headers can be sent by anyone, so logging each denial at the default level lets unauthenticated clients flood the log. Keep the line for named identities. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: count kept bucket grants when checking an edited user for permissions The empty check ran before the checked Other Bucket Permissions were merged, so a user left with only those grants could not be saved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: list bucket grants of Admin users in the user edit form parseBucketPermissions returned before reading scoped actions when the user had Admin, so removing Admin and saving dropped grants such as WriteAcp:bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep unpaired bucket grants as separate checkboxes in the user edit form The form saves every selected permission on every selected bucket, so grants such as ReadAcp:a with WriteAcp:b came back as all four pairs. Show the grants through the picker only when they cover that full product; otherwise list each grant on its own. Opening the form also resets the bucket scope left over from the previous user. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep grants the edit pickers have no option for A permission x bucket product was shown through the pickers even when a bucket was not in the bucket list (deleted, prefix or wildcard grants) or a permission had no picker option (DeleteBucket, PutBucketQuota). Saving then dropped those grants. Such grants now go to the checkbox list, global ones included, and are submitted unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: name the s3.bucket.access anonymous example by what it grants Read,Write,List leaves out ReadAcp and WriteAcp, so it is not full access. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep unlisted global grants of Admin users in the user edit form parseBucketPermissions returned for Admin before splitting global grants into picker options and the rest, and only raw option values were re-selected. Saving an Admin user dropped s3tables:* and grants with no picker option such as DeleteBucket. Admin users now use the same split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep the /* suffix of bucket grants in the user edit form parseBucketPermissions dropped a trailing /* before matching bucket options, so saving rewrote ReadAcp:bucket/* as ReadAcp:bucket. The bare form also matches bucket-level requests such as GET ?acl, so the save widened the grant. A /* bucket is not a picker option, so the grant now stays in the checkbox list as written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: test that s3.bucket.access saves ReadAcp and WriteAcp grants Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
c437791991 |
s3: reject POST Object header fields with control characters (#12400)
A Content-Type form field such as "\n" was copied verbatim into the multipart part header sent to the volume server, which broke the framing and surfaced as a Content-MD5 mismatch and a 500. Form fields promoted to request headers must be valid header values, as on a PUT; otherwise answer 400 MalformedPOSTRequest. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0742d316aa |
fix(s3): accept fresh admin tokens when context carries stale authorization (#12390) (#12397)
* fix(s3): accept fresh admin tokens when context carries stale authorization (#12390) Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com> * security: share the admin Bearer check between S3 and admin-plane peers Move the authorization scan into security.CheckGrpcAdminToken, built on IsFilerAdminToken, and call it from CheckGrpcAdminPeer and the S3 IAM cache check. Both now accept a valid admin token in any authorization value instead of S3 re-implementing the decode and claim checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: accept a fresh admin token after a stale one on IAM gRPC The S3 IAM cache check is meant to mirror the filer's, so route the filer's checkAdminAuth through security.CheckGrpcAdminToken too. IamGrpcStore appends its token to the caller's context, so a forwarded older token would otherwise still shadow it here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * security: parse at most four authorization values per admin call Scanning every value let one request repeat a junk Bearer token (cheap over HPACK) tens of thousands of times and burn about half a second of JWT parsing on an unauthenticated port. Real callers send their own token plus at most one forwarded from their context. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * credential: match the other WithS3InternalAdminAuth replace fix The plugin worker fix makes the same set-instead-of-append change to WithS3InternalAdminAuth with different code and a different test at the same spot, so whichever lands second would conflict. Use the identical code and test so the two merge cleanly in either order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * security: test an expired admin token ahead of a fresh one Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * security: test that a same-key session token is not an admin token Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test IAM gRPC with only invalid authorization values Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
51ee165279 |
ec: rebuild a needle when other shards are corrupt over its range (#12401)
* storage: search corrupt shard sets when an EC needle fails to rebuild A needle whose reassembled bytes fail to parse is rebuilt with its own shards excluded. When other shards are corrupt over the same range, that decode reads them as inputs, and the retry dropped only one input at a time, so two or more extra corrupt shards left the needle unreadable even within parity. After the single pass fails, try every set of up to ParityShards shards that holds a suspect as the corrupt set, smallest first. Suspect ranges inside the set are rebuilt from the shards outside it and the rest keep their bytes, so a clean suspect can still feed the decode. Sets whose decodes read the same inputs are tried once, and the search is bounded by set count and by rebuilt bytes past the single-shard sets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: search corrupt shard sets when an EC needle fails to rebuild Mirror the Go read path: after the single pass over the suspects fails, gather each interval from every other shard once and try each set of up to parity_shards shards holding a suspect as the corrupt set, smallest first, with the same dedup and bounds. The search runs on the blocking pool, holding the recovery permit, since it can take many decodes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: charge only multi-shard sets against the rebuild byte budget The budget check already let single-shard sets through, but each one was still charged, so a needle spanning many shards could spend the budget before the first multi-shard set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: bound the corrupt-set gather to the recovery budget The search gathered every other shard per interval, so a large needle spanning several shards held up to Total-1 copies of itself under a permit capped at the budget. Gather only as many as the budget covers, but never fewer than the suspects-excluded gather the fallback always held, so single-suspect recovery keeps every other shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: gather at least one decode's worth for the corrupt-set search A needle over more shards than parity skips the single pass, and its suspects-excluded width can fall below DataShards, so the bounded gather left even one corrupt shard unrecoverable. Floor the width at DataShards, which the per-suspect rebuild this search replaced read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: rebuild one suspect at a time when the search gather is narrow Flooring the search gather at DataShards made a large needle over more shards than parity hold ten copies of itself, where the per-suspect rebuild it replaced held one interval's fan-out. Keep that rebuild after the search, and skip the search when the budget leaves fewer inputs than one decode. Parity slack still widens the gather to every shard outside the suspects, as the dropped-input retry did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: keep the dropped-input retry and run the set search within budget A narrowed search gather skipped only each interval's own shard, so it could read corrupt suspects in place of healthy shards the old retry kept, and its byte bound could stop before that retry's hypotheses. Restore the dropped-input retry as it was, move the search into its own function after it, and run the search only when every interval's full gather fits the recovery budget. Past the budget a read takes the same paths, with the same footprint, as before the search. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
8f29e38837 |
grafana: chart EC reconstruction and other unplotted metrics (#12406)
* grafana: chart EC intervals served by reconstruction Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: chart volume server storage IO errors and quarantine Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: chart frozen volume growth and duplicate volume ids Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: chart filer metadata log gaps and replay failures Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: add failed bytes to filer sync throughput Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: chart remote mount cache reads and evictions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: chart plugin worker detection duration Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: show duplicate volume ids whose first detection was the only one Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: keep cluster apart in the master growth and duplicate id panels Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d6c65cd924 |
volume: count EC intervals served by reconstruction (#12392)
* volume: count EC reads that fall back to reconstruction
When a needle read can't read an EC shard interval directly (the shard isn't
mounted here and no holder answers), both volume servers rebuild it from the
other shards and the read still succeeds, with nothing to show for it.
Count those fallbacks in SeaweedFS_volumeServer_ec_degraded_read_total{result}
(success, failure), one per interval, at the one place each server falls back:
Go readOneEcShardInterval, Rust fetch_one_interval. Corruption repair and scrub
aren't counted. A line binding each one to the needle and the volume's encode
generation is logged at V(3) in Go and debug in Rust.
* volume: name the EC counter for what it counts
Reed-Solomon reconstruction can't tell corrupt inputs from good ones, and the
needle is validated later, so "success" only ever meant the reconstruction
produced data. Rename the counter to
SeaweedFS_volumeServer_ec_reconstructed_interval_total and the log line to
"ec reconstructed interval", say so in the help text, and pin the behaviour
with a test: an interval rebuilt from corrupt parity counts success while the
read itself fails.
* storage: restore stderr when the log-line test stops early
A fixture failure in readEcNeedle ends the test through t.Fatalf, which
skipped restoring os.Stderr and closing the pipe, so later tests wrote
into a pipe nobody read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
15ff5c113a |
fix(s3tables): validate policy conditions and fail safe on unusable ones (#12389)
* s3tables: validate policy conditions and fail safe on unusable ones * s3vectors: reject a condition operator with no keys The shared evaluator now treats a keyless operator as unusable, so a stored keyless Allow would never grant while the write succeeded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: reject an Allow whose Condition is null A null Condition decodes as no Condition, so the Allow granted unconditionally. A null Condition on a Deny only widens the veto and stays accepted, as in the S3 Tables validator. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: treat a NaN numeric condition value as unreadable strconv.ParseFloat accepts "NaN", but no comparison with it holds, so a Deny on NumericLessThan never applied and an Allow on NumericNotEquals matched every request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: reject condition keys the evaluator never supplies A condition on a key such as aws:SourceIp reads no request value, so a Deny on it never applied and an Allow never granted. Accept only the keys getConditionContextValues resolves, as the vector validator does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: require Statement and Effect spelled as written The evaluator decodes field names regardless of case, so an "effect" or "statement" key slipped past the null Condition check and the other raw-key checks while the stored Allow still granted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: treat an infinite numeric condition value as unreadable Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: a stored Allow with a null Condition never grants The typed decode reads "Condition": null as no Condition, so an Allow stored before validation refused it still granted unconditionally, for table and vector policies alike. Record the literal null while decoding a Statement and skip such an Allow during evaluation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: validate a null Condition from the decoded Statement validatePolicyDocument re-parsed the document to find a literal null Condition; the Statement decode now records it, so the validator reads the same flag the evaluator does and nullConditionStatements goes away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: keep Condition objects merged across key spellings Decoding Condition through a json.RawMessage kept only the last of "Condition" and "condition", so an empty object under one spelling erased the other's restriction. Decode the statement as before and probe for the literal null separately. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: refuse a Condition split across key spellings Condition objects under differently cased keys decode into one map, and an operator they share keeps only the last spelling's keys, so a restriction under the other spelling was dropped. Validation now refuses a statement whose Condition is under more than one key, and one stored before that never grants on an Allow and vetoes on a Deny. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: refuse a Condition key repeated under the same spelling PolicyDocument re-marshaled each statement through interface{}, and Statement.UnmarshalJSON counted keys from a map, so both kept only the last of a repeated Condition key. "Condition":null,"Condition":{} or two Condition objects sharing an operator passed validation with a restriction dropped. Statements now decode from their original bytes and the keys are counted while walking the object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: refuse a repeated Condition key in a vector bucket policy The validator read Condition from a map, which keeps only the last of a repeated key, so it accepted a statement the evaluator never grants on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: keep the re-marshaled statement decode, count Condition keys apart Decoding statements from their original bytes let document order pick between spellings of every field, where the re-marshal sorted the keys first, so a stored "effect":"Deny","Effect":"Allow" became an Allow. Statements decode as before again; only the Condition keys are counted from the original bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: test that validation refuses a split Condition on a Deny Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: test that validation refuses a repeated Condition on a Deny Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d727c6970f |
plugin worker: stop forwarding the stream's admin token to job RPCs (#12391)
* plugin worker: keep the stream's admin token out of job contexts The bearer token minted when the worker opens its control stream was attached to connCtx, which every detection and execution context derives from. Job RPCs therefore forwarded that connect-time token, and when a job added its own fresh token (s3_lifecycle's LifecycleDelete) the S3 server validated the stale one first and rejected the call once it expired. Attach the token only to the WorkerStream call, as the Rust worker does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * credential: replace an inherited admin token instead of appending one WithS3InternalAdminAuth appended its fresh Bearer token, so a context that already carried an authorization value sent both, and the S3 gateway's checkAdminAuth validates only the first. Set the header on a copy of the outgoing metadata so the fresh token is the only one sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * plugin worker: read the captured job metadata after the stream ends Both result channels could be ready at once, letting select fail a successful run. The handler reports before JobCompleted reaches admin, so wait for the stream to finish and then read the captured metadata. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * plugin worker: test that dropping the connection cancels running jobs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |