mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-07 12:37:41 +01:00
7d6680c7a4d5b1d28168ea8290be5b85a5e3b4ca
10509
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
cdea7e79e0 |
filer: fold small chunks into manifests 256 at a time (#12462)
* filer: fold small chunks into manifests 256 at a time Every metadata event carries the file's whole chunk list, so a file grown by small appends re-sent each chunk record on every append until the list reached ManifestBatch (10000). Chunks under 1 MiB now fold once 256 of them build up; full-size chunks still fold at ManifestBatch, so large files keep their flat lists and read without a manifest fetch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: merge appended small chunks at the small-chunk fold count The append path merges small chunks into full-size ones, but only at 1000 of them; the fold now hides small chunks in a manifest at 256, so the merge would never see enough of them. Merging at the same count keeps rewriting them into full chunks first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
728b476ba2 |
filer.remote.gateway: keep the remote object while a version under a delete marker is locked (#12459)
* filer.remote.gateway: keep the remote object while a version under a delete marker is locked Every version of a versioned bucket uploads to the object's key, so on a remote without versioning that object is the only remote copy of the latest version. A delete marker deleted it even when a version still held a legal hold or unexpired retention. Keep it while any version of the key is locked, the same rule the filer uses before pruning a remote entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway, filer.remote.sync: say that earlier versions need remote versioning Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: delete a kept delete marker's object once the version lock is released A delete marker kept for a locked version stays unstamped. When an update or delete leaves a version that carried a legal hold or retention without an active lock, look up the key's latest version and, if it is still that unstamped marker, run the delete again. Only changes to versions that carried a lock trigger it, so the noncurrent mark and tag updates of unlocked versions do not send repeat remote deletes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: test the kept delete marker retry through the bucket mount The retry reads the key's versions through a filer client parameter, as the rename helpers do, so a test can check the mapped object key and the marker stamp, and that lock-releasing updates and deletes reach it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: check the lock release lookup goes to the gateway's filer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
8c22336944 |
filer.meta.backup: stream from the start of an unfinished walk (#12458)
* filer.meta.backup: stream from the start of an unfinished walk A walk only inserts entries, so when the first walk is interrupted the next start walks again and streams from the second walk's start: deletes and renames between the two walks never reach the backup store. Keep the first walk's start time in the store until a walk completes and stream from it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.meta.backup: start fresh once the stream has passed an earlier walk's start A walk start left in the store by a crash between saving the offset and clearing it, or by an interrupted -restart followed by a plain restart, stops mattering once the stream saves an offset past it. Ignore it then instead of replaying from it on the next walk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.meta.backup: fail the walk on an unreadable offset Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
jsasandjames
|
5ba97b4432 |
filer: a full remote-deletion tombstone table keeps the lazy fetch off (#12440)
* filer: a full remote-deletion tombstone table keeps the lazy fetch off At the cap a new delete under a remote mount was dropped with a log line and isRemoteDeletionPending answered false for it, so the lazy remote fetch resurrected deleted paths. While the table is full every path may be a pending delete: isRemoteDeletionPending answers true until the table has room again. The size and the full state are gauges (filer_remote_deletion_tombstones, filer_remote_deletion_tombstones_full). (cherry picked from commit f3be0ab965cde3bfd3053d3ac720fbbe3f68e98b) * filer: a full tombstone table drains on reads and lifts after the unrecorded deletes age out While the table is full, a lookup miss evicts expired tombstones and those the write-back offset has passed, so the table drains without a mutation. The full state lifts only once the table has room and the newest unrecorded delete is past the TTL: an unrecorded delete has no event to confirm it, so only the TTL makes it safe, as for a pending tombstone. (cherry picked from commit 19394fe5d67798af7b0d755ac18c83c75e20a12b) * filer: a full tombstone table drains per mount, only when its offset advances Offset-based release applies to the tombstones under the mount whose offset was read; another mount's tombstones answer to their own offset. A read sweeps the table only once that offset advanced since the last drain for the mount or the unrecorded deletes aged past the TTL, so a listing of many missing children does not rescan the table per child. (cherry picked from commit 9cf00e2e54d70e6ba530e6ec1b1ff4b502873847) * filer: the tombstones-full gauge help names when it clears --------- Co-authored-by: james <james@medable.com> |
||
|
|
7fc33e7fe3 |
filer: no lazy remote fetch inside a store transaction (#12439)
* filer: no lazy remote fetch inside a store transaction A rename onto an existing target deletes the target inside the store transaction, then CreateEntry's FindEntry misses the row and runs the lazy remote fetch, which persists the archive copy on a second connection. The store wrapper drops that call's deadline, so the UPDATE waits without bound on the row lock of the transaction's own uncommitted DELETE while the rename goroutine holds the path's exclusive locks: every later lookup, rename and delete of the path on that filer waits forever. A lookup whose context carries the store transaction skips the fetch. (cherry picked from commit b69e9535c2597d0c8e856f031d3778fddfd4159c) * filer: materialize remote-only rename paths before the transaction With the lazy fetch off inside a store transaction, a source or target the filer holds only on the remote would read as absent to AtomicRenameEntry and StreamRenameEntry: a rename with FailIfTargetExists could overwrite it, and a remote-only source would skip the deferred copy. PrefetchRemoteEntries runs the fetch for both paths before BeginTransaction, under the path locks. (cherry picked from commit e61f9718018669ef59768cd5eda25b6466d69597) * filer: skip the lazy remote fetch only in CreateEntry's lookup inside a transaction The guard on every lookup inside a store transaction hid a source or target the filer holds only on the remote from the rename, and the prefetch before the transaction did not reach the children of a directory rename. The deadlock needs a write: CreateEntry's lookup after the transaction deleted the path. That lookup alone skips the fetch; reads inside the transaction keep it, and their persist on a second connection holds no lock the transaction waits for. (cherry picked from commit f555092c324780a8fe5314b238694551e3ef6be5) * filer: mark a store transaction independently of the store's own key ArangoDB and FoundationDB keep their transaction under other context keys, so the CreateEntry guard keyed on "tx" missed them. The filer now marks every context returned by BeginTransaction and the guard checks that marker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit |
||
|
Chris LuandClaude Opus 5.5
|
86db60b669 |
s3api: mint a versioned PUT's id at commit, after the lock wait and the upload (#12454)
* s3api: mint a versioned PUT's id after the object write lock putVersionedObject minted the version id before the upload and the lock wait. A DELETE that committed while the PUT waited got a newer id, so a conditional PUT whose If-None-Match passed against the delete marker was stored behind it: accepted, but GET returned NoSuchKey. The finalize now re-mints the id once the lock is held and the precondition passed, so the version orders after everything the check saw. An ambiguous routed attempt keeps its id, since recovery looks for that entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: mint a routed versioned PUT's id after the upload On the routed path the version id was still the one minted before the upload, and RECOMPUTE_LATEST picks the newest name. A DELETE that committed during a long upload left its marker latest over the accepted PUT. Re-mint right before the routed transaction, as the lock path does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1176f975b3 |
filer: release metadata log chunks a failed append leaves unreferenced (#12455)
* filer: stop unbounded metadata-log growth on append-only files Two independent leaks make a filer metadata log grow far faster than the data it describes. On a cluster whose live logs weigh a few hundred megabytes, they produced tens of gigabytes of metadata per day. 1. Metadata events embed the whole old and new entry, chunk lists included. An append-only file — a log or a journal written in small pieces — therefore re-serializes every chunk it ever had on every append, so the cost is quadratic in the number of chunks. A 2,700-chunk file emits a couple of megabytes per append. Cap the carried chunk list at 256 entries and set metadata_event_chunks_truncated so subscribers that need the content read the entry from the filer instead of trusting the event. Subscribers that only track names, attributes, or chunk deltas keep working unchanged. 2. appendToFile uploads the chunk before committing the filer entry. When the commit fails — store error, cancelled context, filer shutting down — the chunk is stranded: no entry references it, so no later delete can mark it, and neither vacuum nor fsck can see it because both work from the entry index. The volume server keeps counting those bytes as live data forever. Release the chunk when the commit fails. * filer: keep chunk lists in metadata events Mount meta caches, peer filers replaying into their own store, and the filer.sync, filer.backup and filer.meta.backup sinks all store the event's entry as the file. Dropping the chunk list past a count left those copies of a large file without content. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: release a metadata log chunk the lookup or commit leaves unreferenced The flush retries a failed append with a new upload, so a store outage left one stranded chunk per retry. A failed lookup strands the chunk too. Use DeleteUncommittedChunks, as the other write paths do: the chunk was never part of the file, so path chunk-deletion rules do not apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: keep a metadata log chunk the failed create stored CreateEntry inserts a new entry before its parent folders, so a parent failure returns an error with the entry already referencing the chunk. Release it only when the store shows no entry, or an entry without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: count a metadata log piece the failed create stored as appended The flush retry would append the same bytes after it, and a piece cut inside a record then breaks replay of that segment. The retry takes the update path, which does not create parent folders either, so returning the error only duplicated the piece. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Martin Morez <martin@morez.org> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
2be4498c04 |
s3 lifecycle: walker deletes carry the listed identity and must target the latest version (#12452)
* s3 lifecycle: send the listed entry's identity on walker deletes The daily walker dispatched LifecycleDelete with no ExpectedIdentity, so the server skipped its compare-and-swap and an object rewritten between listing and dispatch was deleted. The list adapter now records each listed entry's identity, computed the same way as on the event path, and the walker sends it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: current-version expiration must target the latest version The walker addresses the latest version of a versioned key by id. A version file never changes, so its identity still matches after a newer PUT, and the delete went ahead: a delete marker over the new version with versioning enabled, or the new null version deleted while suspended. LifecycleDelete now resolves the key's latest version for EXPIRATION_DAYS/EXPIRATION_DATE requests that carry a version id and answers NOOP_RESOLVED (VERSION_NOT_LATEST) when it is a different one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: check the latest version for unaddressed suspended expirations With versioning suspended, an expiration without a version id reads the plain entry, which is the null version but not necessarily the latest: a newer real version can sit in .versions. The latest-version check now runs in the expiration branch, by versioning state: for every suspended request (an unaddressed one addresses the null version) and for enabled requests that carry a version id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test that each version sibling carries its own identity Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
9af736da62 |
volume: stop serving EC reads from quarantined shards (#12450)
* storage: skip quarantined EC shards on local reads A needle read tried the local shard first even when its disk was quarantined after repeated IO errors, so every read paid the failing disk's EIO latency before falling back. Treat such a shard as not local so the read goes to peers or reconstruction, and read it only as a last resort when no healthy copy answers. Scrub still reads it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: refuse EC shard reads from a quarantined disk VolumeEcShardRead served peers from a shard whose disk was quarantined, and it ignored the .ecx lookup error: a failed index read returns a tombstone size, so the peer was told a live needle was deleted. Refuse quarantined shards and fail the request on an index read error; a needle missing from the index still reads, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: report quarantined EC shards deleted right away Quarantining an EC volume only dropped it from later full EC heartbeats, so the master kept routing reads to the faulty disk for up to 17 heartbeat intervals. The heartbeat that quarantines the volume now carries its shards as deleted, once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: do not announce EC shards mounted onto a quarantined volume MountEcShards and the shard load notifier reused a quarantined EcVolume and announced the new shard, pointing the master back at the faulty disk until the next full EC heartbeat. Skip the announcement, and fail the mount so a caller moving a shard here keeps its source copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: count IO errors on local EC shard reads Go's readLocalEcShardInterval feeds every local shard read into the EC volume's IO error streak; the Rust needle read and recovery gather did not, so a failing disk under local reads never moved toward quarantine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: skip quarantined EC shards on local reads Mirror the Go read path: a needle read and the recovery gather treat a shard on a quarantined disk as not local, and read it only as a last resort once peers and reconstruction fail. Scrub still reads it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse EC shard reads from a quarantined disk Mirror Go's VolumeEcShardRead: answer Unavailable for a shard whose disk is quarantined, so the peer tries the next holder or reconstructs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: report quarantined EC shards deleted right away Mirror Go: the volume heartbeat that quarantines an EC volume carries its shards as deleted, once, instead of leaving the master to drop them at the next full EC heartbeat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: fail EC shard mounts onto a quarantined volume Mirror Go's MountEcShards: the heartbeat already leaves such a shard unannounced, so fail the mount and let a caller moving the shard here keep its source copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: keep reading EC needles when the shard lookup fails A failed or short master lookup aborted the read before any shard was tried, so the quarantined-disk last resort never ran once the master had dropped those shards, and local shards went unread whenever the master was unreachable. Read on with the cached locations, as the Rust server already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: log a failed last-resort read of a quarantined shard When reconstruction and the quarantined read both fail, say why the local read failed too. Go and Rust. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: drop a shard whose mount was refused on a quarantined disk The refused mount left the new shard loaded on the quarantined volume, where it was neither served nor announced, and the in-memory claim drew later copies of that shard back to the same disk. Unload it again, keeping a shard that was already mounted there. Go and Rust. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: finish a mount batch before reporting a failed shard A shard that failed to mount, now including one refused on a quarantined disk, returned before the sidecar reload, and in Rust before the heartbeat notify, so shards mounted earlier in the same batch kept checksum protection off and reached the master only at the next full EC heartbeat. Stop at the failed shard but finish the rest first. Go and Rust. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: keep a peer's deleted answer when its stream fails after it A shard read recorded the holder's deleted answer, and a stream error after it failed the read: Rust discarded the flag, and Go let the next source or the reconstruction result overwrite it. With the last-resort read of a quarantined shard, that could serve a needle a peer had reported deleted. Once the deleted answer is in, end the read on a later stream error instead of failing it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: use the platform media error in the quarantine heartbeat test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: rebuild EC intervals from quarantined shards when healthy ones fall short Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: rebuild EC intervals from quarantined shards when healthy ones fall short Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: serve a peer's EC shard read from a healthy copy on another disk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: serve a peer's EC shard read from a healthy copy on another disk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a quarantined EC shard read instead of serving a sibling copy A sibling disk's copy can have its own deletion journal, and an EC blob delete is journaled on one local copy only, so the sibling could serve a deleted needle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: refuse a quarantined EC shard read instead of serving a sibling copy A sibling disk's copy can have its own deletion journal, and VolumeEcBlobDelete journals a delete on one local copy only, so the sibling could serve a deleted needle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: check the disk an EC shard mounted on for quarantine mount_ec_shard now returns the disk it mounted on, and the mount RPC checks only that disk, like Go's MountEcShards. Before, a quarantined sibling holding the same shard failed the mount while the healthy disk's copy stayed mounted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: test an EC mount beside a quarantined sibling Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: check the quarantined-sibling mount test by its result A server that remounts a shard on the disk already claiming it refuses the mount there instead. The test now only requires that the result matches whether the healthy disk ended up holding the shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: check the quarantined-sibling mount test by its result Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: expect the quarantined-sibling mount result from the disk the store picks The test accepted a refused mount whenever the healthy disk stayed empty, so a handler that refused before mounting passed too. Ask the store which disk it picks first, then require success exactly when that is the healthy disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: expect the quarantined-sibling mount result from the disk the store picks Mirrors the Go test. add_location's cross-disk reconcile mounted shard 1 on the healthy disk too, so the test now unmounts it there first, leaving the quarantined sibling as the only claimant, as in Go. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
206d7b1418 |
filer.remote.sync: bounded filer lookups, one chunk cache per process (#12441)
* filer.remote.sync: bounded filer lookups, one chunk cache per process The filer lookups on the sync path (currentEntry, isSuperseded, uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no deadline. A lookup that never answers keeps its job alive, the job keeps its path in the MetadataProcessor conflict index, AddSyncJob waits on the condition and the daemon stops consuming events with no log line. Each lookup now carries a 30 s deadline, so the job fails and the processor's stop-and-resubscribe path runs. remoteUploadReader built a chunk cache per upload; the cache's worker goroutine is never stopped, so each upload leaked one goroutine and its cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory limit). One chunk cache per process. (cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78) * filer.remote.sync: a failed mount lookup at startup never resolves to now collectLastSyncOffset returned time.Now() when the mounted directory lookup failed, which skips every event since the saved offset. The saved offset is used instead; without one the daemon exits so a restart retries. (cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae) * filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read retriedWriteFile returns the supersession lookup's error instead of treating an unanswered lookup as "not superseded": in a replay an older snapshot could otherwise overwrite a newer remote object. The processor retries the event. isSuperseded keeps its boolean form for the other call sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls back to the saved offset cannot hang on it. TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed lookup; it now asserts the withheld upload and the lookup error. (cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa) * filer.remote.sync test: keep the slices import the other gateway tests use * filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo After a failed upload attempt the supersession lookup's error ended in isSuperseded's boolean and the loop could upload the snapshot again unverified. The lookup error now ends the loop (errLookupUnanswered, not retried) and the processor retries the event. With -timeAgo set, a failed read of the saved offset resolved to now minus timeAgo, skipping everything older; the daemon exits instead so a restart retries. (cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566) * filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting resolveLastSyncOffset returns the lookup or offset-read error; both callers (directory and buckets) hand it to util.RetryUntil, which retries with backoff instead of the process exiting. * filer.remote.sync: keep doc comments on the functions they describe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit |
||
|
Chris LuandClaude Opus 5.5
|
1406335301 |
volume: a delete cancels writes of its needle still in flight (#12445)
* volume: a delete cancels the writes of its needle still in flight A write queued behind the volume locks, for example behind another write's fsync on a busy disk, is not in the needle map yet. A delete of that needle in the meantime finds nothing, and the write lands afterwards, so the needle comes back on this copy only while the other replicas have it deleted. Each volume now tracks its writes from entry until they return. A delete cancels the ones for its needle, and a cancelled write that landed is tombstoned before it returns, unless the needle was deleted or rewritten since. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: a replicate delete that finds nothing cancels the pending write Rolling back a failed chunk upload sends DELETE ?type=replicate to every holder, including the slow one whose write timed out on the client. That holder has not indexed the write yet, so the delete answered 404 and the write landed live afterwards. Cancel the needle's pending writes, then look again in case one finished in between. A plain delete that finds nothing still answers 404 without cancelling: it is not sent on to the replicas. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: a delete cancels the writes of its needle still in flight Mirrors the Go volume. Volume::track_pending_write registers a write until its guard drops, delete_needle cancels the needle's pending writes, and settle_pending_write tombstones a cancelled write that landed unless the needle was deleted or rewritten since. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: track uploads and cancel them on a replicate delete An upload is tracked from the handler, and a durable one from submission to the write queue, where the guard travels with the request: a handler dropped while the write waits in the queue does not untrack it, and the worker settles it after the batch. A replicate delete that finds nothing cancels the needle's pending writes and looks again, like the Go handler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: take a cancelled write back in the lock span that publishes it The write tombstoned itself only after it returned, with the volume locks released. In that gap a compaction could move the record so the offset check skipped it, a close could drop it, and an identical write sent after the delete could match the live record and report success before the tombstone removed it. The check now runs right after the publish, under the same dataFileSyncLock and dataFileAccessLock hold, in the plain, fsync and batch write paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: a delete cancels only the writes carrying its cookie A delete with the right needle id but the wrong cookie could cancel an in-flight write and get it tombstoned. Each pending write now records its cookie, and a cancel only marks the matching ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: take a cancelled write back in the lock span that publishes it Mirror the Go change: settle_pending_write ran after write_needles_grouped returned, so a compaction, a close or an identical write later in the same batch could slip in before the tombstone. drop_cancelled_write now runs right after the publish, under the same sync and access lock hold, in the synced run, the unsynced run and write_needle_unsynced_or_refused. write_pending_needles carries each write's PendingWrite down to that point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: a delete cancels only the writes carrying its cookie Mirror the Go change: a delete with the right needle id and the wrong cookie no longer cancels an in-flight write. Each tracked write keeps its cookie next to its cancel flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a queued write whose volume was remounted The write queue tracks a write on the volume found at submit, but the worker looks the volume up again by id. After a remount, a delete reaches the new volume and cannot cancel the write registered on the old one, so the worker now refuses such a write instead of landing it on the new volume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: decode the delete's type parameter like Go's FormValue The delete handler matched the raw query text for type=replicate, so a percent-encoded name or value was not treated as a replica delete and did not cancel the write it undoes. Parse the query the way the upload handler does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: test a replica delete landing in a write's fsync window Drive DeleteHandler while a durable write is parked in its .dat fsync, appended but not indexed: the replica delete answers 404 and the write is taken back when it publishes, while a plain delete leaves it live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: flush a take-back tombstone before the index row that points at it A durable write taken back during its fsync appends the tombstone after the .dat sync, and the .idx sync that follows made its row durable while the record itself could still be lost, which reloads the volume read only. The fsync and batch paths now sync the .dat again first when a take-back appended a tombstone; a failure stops the volume taking writes like a failed index sync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: flush a take-back tombstone before the index row that points at it A durable write taken back in write_synced_run appends its tombstone after the .dat sync, so the .idx flush that follows could make the row durable while the tombstone record itself was still in the page cache. The run now syncs the .dat again first when a take-back appended a tombstone; a failure stops the volume taking writes like a failed index flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: a skip-cookie batch delete cancels the writes of the file id it names BatchDelete with SkipCookieCheck left the needle's cookie at zero, so the delete's cancel matched none of the in-flight writes and a retry queued behind it could land afterwards. The non-EC delete now carries the file id's cookie, as the Rust volume server already does; the EC path keeps cookie 0, which DeleteEcShardNeedle reads as "skip the check". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: release a held test write before the store closes A failed wait ended the test before release ran, so the writer stayed parked in its fsync and the store cleanup blocked on its lock until the package timed out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f71c0fe523 |
fix(s3tables): route REST RenameTable and refuse unrouted S3 Tables requests (#12446)
* s3tables: route REST RenameTable and refuse unrouted S3 Tables requests
PUT /tables/<arn>/<ns>/<name>/rename had no route, so it fell through to the
/{bucket} routers and ran as a PutObject into a bucket named "tables",
answering 2xx while the table kept its name.
- Answer any s3tables-signed request no S3 Tables route takes with 404
UnknownOperationException, before the health, domain and bucket routers.
- Route the AWS RenameTable REST call (newNamespaceName, newName, both
optional; versionToken).
- Honor versionToken on RenameTable, checked against the snapshot the claim
asserts so a commit between the two reads cannot slip a stale token through.
* s3tables: match the credential scope's service, not a region named s3tables
The unrouted-request fallback widens what a false match costs: an S3
client whose region is named s3tables had every request answered as an
unknown S3 Tables operation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: refuse an empty newName or newNamespaceName on REST rename
An empty field was read as omitted, so {"newNamespaceName":"other",
"newName":""} moved the table under its old name instead of failing
validation the way AWS does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: check the rename version token after the snapshot comparison
Once the separately read metadata is known to equal the claimed snapshot,
the token compares against the parsed metadata; no second decode of the
snapshot bytes is needed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables test: cover policy and tag changes in the rename snapshot compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables test: give each REST rename subtest its own source table
The token subtests renamed the table the first subtest produced, so run
alone they failed on a missing table instead of testing the token.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Chris LuandClaude Opus 5.5
|
c84c43f3da |
master: export Hashicorp raft metrics through the go-metrics global raft uses (#12448)
* master: keep the hostname out of raft metric names go-metrics prefixes gauge names with the hostname by default, which gives every master its own metric family, e.g. SeaweedFS_<host>_raft_peers. Prometheus already tells masters apart by the scrape target. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: send Hashicorp raft metrics through the go-metrics global raft uses hashicorp/raft and raft-boltdb report through github.com/hashicorp/go-metrics, but the master set up its prometheus sink on the armon/go-metrics global, so a -raftHashicorp master exported no raft metric families. The sink is now registered on the hashicorp/go-metrics global, and armon/go-metrics is no longer a dependency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: chart raft commit time and leader contact Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: keep the cluster in the raft panels Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: average raft commit time and leader contact over the rate window The go-metrics summaries keep quantiles for 10s, so a p99 scraped every 30s misses most observations. The _sum/_count rates count every sample. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: leave raft panels empty when nothing was observed Clamping the count rate plotted 0 ms for windows without commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: let the raft metrics test use its own registry Registering on stats.Gather made go test -count=2 fail with a duplicate collector registration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: register raft metrics before opening the raft stores raft.NewRaft starts its run loop before returning, so registering the sink afterwards could drop the first state and boltdb samples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
7b3983f154 |
s3api: speed up recursive listing of branched directory trees (#12438)
* s3api: look up a trailing-slash prefix's directory instead of listing its parent A prefix such as "account-0001/" names one directory, but the walk found it by listing the parent with the name as a prefix. SQL stores answer that with a LIKE that bounds the scan from below only, so in a wide bucket root the query reads every sibling after the name: seconds for a prefix holding a few keys. Look the directory up by name and serve it as the level's only entry. A miss, an error or a non-directory falls back to the prefix listing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: track failover receives atomically The failover wrapper flags a received reply through a plain bool, which a callback reading from the filer on several goroutines would race on. Use an atomic.Bool so the callback may share its filer client. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: read directories ahead of a recursive listing walk A recursive ListObjects walk enters directories one at a time, so a bucket made of many directories holding one or two keys each costs one filer round trip per directory, in series: seconds per 1000-key page on a store whose reads wait on disk. Read the first window of the directories the walk will enter on a few workers, nearest first in walk order, and hand each window over when the walk gets there. Every window the walk reads itself is drained first so its subdirectories are queued before the walk descends into the first one. The number of windows held ahead grows while directories end inside their first window and falls back when one does not, so a page ending inside a large directory leaves few reads wasted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: drop the prefetcher's record of every directory it was handed A page can cross any number of empty directories without spending its key budget, so a set of every announced directory grows with the tree. A directory is already handled when it is at or behind the walk, held or in flight, or queued; check those, which the queue and window caps bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: end a trailing-slash prefix listing at the file its lookup finds A file named by the prefix lists nothing under it, the same as the sibling listing it fell back to, which threw the lookup's answer away and paid the parent scan anyway. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: read a walk's own listing ahead instead of draining it Draining the window before the walk took its first entry made a page wait for the end of the listing, which a SQL store answering a name prefix reaches only after scanning the rest of the directory; a page that fills before then used to return at once. A background reader now hands the window's directories to the prefetcher as they arrive while the walk takes each entry, holding at most 1 MiB of entries ahead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: bound the bytes a prefetched listing window holds Entries with long chunk lists or inline content made the up to 64 held windows of 32 entries an unbounded amount of memory per listing. A window now stops at 128 KiB of entries and the walk lists the rest of that directory from the window's last entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test recursive listings whose streams break partway Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3e64326f2b |
s3: require their own actions to change a bucket's CORS or versioning (#12433)
* s3: require their own actions to change a bucket's CORS or versioning PUT/DELETE ?cors and PUT ?versioning were authorized as Write, so an identity granted only object writes on a bucket could replace its CORS rules or suspend versioning. They now need PutBucketCors, DeleteBucketCors and PutBucketVersioning (or Admin), and inline policies granting s3:PutBucketCors, s3:DeleteBucketCors or s3:PutBucketVersioning map to those actions instead of Write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test that object Write cannot change bucket CORS or versioning Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: offer the bucket CORS and versioning permissions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test each bucket config action with its own identity Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3a579ece3b |
s3: commit the delete marker in the owner's finalize transaction (#12432)
* s3: commit the delete marker in the owner's finalize transaction createDeleteMarker wrote the marker with mkFile to the gateway's current filer and then sent RECOMPUTE_LATEST to the object's ring owner. When the two differ, the owner scans .versions before peer metadata sync delivers the marker, re-derives the pointer to the previous version, and the marker stays non-current on every filer. routedVersionedFinalize now takes an optional version entry and PUTs it in the same ObjectTransaction ahead of the recompute, the way PutObject and CompleteMultipartUpload already do. The copy destination keeps passing nil: its version file is already written on the owner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test a versioned delete when the gateway's filer is not the owner Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: assert the delete marker and the pointer flip share one transaction Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a8766f5177 |
s3api: apply copy bucket-default SSE after the ACL re-auth (#12436)
* s3api: apply copy bucket-default SSE after the ACL re-auth CopyObject without its own x-amz-server-side-encryption header takes the destination bucket's default encryption, which CopyObjectHandler writes into the request headers. It did so before preparePutObjectACL, which re-authenticates a request that carries an ACL (x-amz-acl or a grant header) for s3:PutObjectAcl. verifySignedHeadersCoverage then found the synthesized x-amz-server-side-encryption* headers outside SignedHeaders and rejected the copy with SignatureDoesNotMatch. Move the bucket-default injection after the ACL check, so the re-authentication sees the request as the client signed it. The destination object still gets the bucket default encryption. * s3api: assert the copy's KMS key and stored ciphertext Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Stefan Scharlott <stefans@Stefans-MacBook-Pro.local> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d127be610f |
admin: re-send the master vacuum disable every poll so it survives leader changes (#12430)
* admin: re-send the master vacuum disable every poll so it survives leader changes
The admin disabled the master's scheduled vacuum only when a vacuum plugin
worker connected or disconnected. The master keeps that flag in memory and it
is not replicated through raft, so after a leader change or a leader restart
the new leader ran its own vacuum next to the plugin worker. When both picked
the same volume the master's commit failed ("compact swap aborted: missing
.cpd/.cpx") and it dropped that replica, leaving the volume under-replicated.
Send the desired state on every poll instead, and log on the master only
when the flag actually changes.
* admin: shorten the vacuum sync comments
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Kevin <kevin@leetify.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Chris LuandClaude Opus 5.5
|
878979af70 |
filer: hide write controls in the UI on the read-only port (#12429)
* filer: hide write controls in the UI on the read-only port The read-only port accepts only GET and HEAD, but its directory page offered New Folder, Upload, drag-and-drop, rename and delete, each of which failed with 405. Requests that come through the read-only handler now render the page without those controls or the upload script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test the directory page controls on each port Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: match the rendered row buttons in the port test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e44c16802d |
weed/server: verify data shards against bitrot sidecar before ec decode (#12422) (#12427)
* weed/server: verify data shards against bitrot sidecar before ec decode (#12422) * ec.decode: verify data shards against the mounted volume's bitrot protection The decode looked up only the generation-0 .ecsum beside the data base and each location's data directory, so it missed a sidecar kept with the index and ignored the generation the volume actually serves. Take the protection the volume's per-disk runtimes resolved instead (On, else Invalid, else Off), and test a corrupt block past the first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.decode: check for live entries before verifying the shards A volume whose needles are all deleted gets its shards purged on the no-live-entries status; a failing checksum must not keep it around. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: verify data shards against the bitrot protection before a decode Mirrors the Go volume server: the decode takes the protection the volume's per-disk runtimes resolved and refuses a data shard that fails its .ecsum checksum, after the no-live-entries check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: a decode refuses a data shard that fails its .ecsum checksum Runs against both volume server implementations: the refused decode keeps every shard and the sidecar, and the needle stays readable through EC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.decode: reload the bitrot sidecar before verifying the shards A rebuild's backfill writes the sidecar without reloading the mounted runtimes, which then still report no protection; resolve it again across the EC metadata directories, as a shard mount does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: reload the bitrot sidecar before verifying the shards Mirrors the Go volume server: the decode job resolves the sidecar again on every runtime of the volume before choosing its protection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: a decode finds a sidecar written after the shards were mounted Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: read through the EC shards after the refused decode The source volume stayed mounted, so the final read never touched the shards it is meant to check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.decode: refuse when a sidecar reload reports another layout ReloadBitrotSidecar logged and dropped the layout-mismatch error, so the runtime stayed Off and the decode skipped the data-shard check. It now returns that error and the decode refuses on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a decode when a sidecar reload reports another layout Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: a decode refuses a late sidecar that records another layout Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.decode: refuse when an unmount left no runtime to resolve the sidecar Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a decode when an unmount left no runtime to resolve the sidecar Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
80f32bc074 |
weed/command: skip remote bucket delete when bucket is unmounted (#12418) (#12425)
* weed/command: skip remote bucket delete when bucket is unmounted * remote_storage: count deleting an already-missing bucket as done filer.remote.gateway can see the same bucket delete more than once, for example once per filer when several filers peer their metadata. A copy that still finds the mount mapping calls DeleteBucket again, and S3 (NoSuchBucket), Azure (ContainerNotFound, ContainerBeingDeleted) and GCS (404) all failed it, which held the sync offset on the delete. A missing bucket now counts as deleted, the way deleting a missing object already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: keep the mapping table writable after mount.mapping is deleted Deleting mount.mapping replaced the gateway's mappings with a table whose map was nil, so the next bucket create panicked writing its mount into it. Reset to an empty map instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: count a delete of an absent object as done filer.remote.sync already treats ErrRemoteObjectNotFound from DeleteFile as deleted, since GCS reports a missing object that way where S3 and Azure succeed. The gateway returned it, so a delete event seen twice, as it is when several filers peer their metadata, held the sync offset on GCS. Use the same deleteRemoteFile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: drop cached remote clients after the new gateway tests GetRemoteStorage caches clients by configuration, so a repeated run (-count=2) got the first run's recorder back and saw no calls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: keep the default storage across mount.mapping changes The default storage for new buckets is derived at startup and kept only in the gateway's copy of the mappings; mount.mapping never stores it. Replacing or deleting that file dropped it, so a later bucket create with no -createBucketAt skipped the remote bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
c971252871 |
admin: make the cluster lock's fairness window configurable (#12426)
* admin: make the cluster lock's fairness window configurable Scheduled jobs hold the cluster admin lock for their whole run. Once the lock had been held for a minute, every new job waited until all running jobs had released it, so with hour-long EC jobs dispatch ran in waves and most workers sat idle between them. -adminLock.fairnessWindow keeps the one-minute default; 0 lets new jobs join the shared hold, and shell clients get the lock when the scheduler next drains. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * admin: wait for the blocked acquire before disabling the fairness window The wake-up test slept 50 ms before SetFairnessWindow(0), so a late acquire that started after the change joined without exercising the Broadcast. Wait until it is registered as a waiter instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8c8d0f1cdf |
weed/shell: resolve collection from topology when ec.decode is given only volumeId (#12424)
* weed/shell: resolve collection from topology when ec.decode is given only volumeId
When ec.decode is invoked with -volumeId for a volume that belongs to a named collection without explicitly specifying -collection, DoEcDecode was called with an empty collection string. Target volume servers then rejected VolumeEcShardsCopy with 'CopyFile not found ec volume id <vid>' because they looked under the default collection.
This commit introduces FindEcVolumeCollection to discover the collection name directly from topology metadata (checking EC shard info and regular volume info), ensuring DoEcDecode receives the accurate collection.
* ec.decode: resolve every batch volume's collection from the topology
The batch passed -collection through unless it looked like a regex, so a
comma list (a,b), a {n} quantifier, ALL or _default_ reached the volume
servers as a collection name and the copy found no shards. The selector
only picks the volumes; each one is decoded in its own collection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: take the collection only from the volume's ec shards
The lookup returned the first record per disk, so a regular volume on an
earlier node answered before the ec shards on a later one. A decode needs
the shards anyway, so their records are the only ones to read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: leave the -volumeId lookup to DoEcDecode
DoEcDecode already resolves an empty collection from the topology.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
120d17a092 |
filer.remote.sync: stop a touched file from being restamped forever (#12419)
* filer.remote.sync: stop a touched file from being restamped forever A metadata-only update that moves the mtime past RemoteMtime (touch, or an O_TRUNC open of an empty lock file) goes through UpdateFileMetadata, which changes nothing on the remote, and the entry is stamped with the same RemoteEntry and a new LastLocalSyncTsNs. That stamp is an update event of its own. shouldSendToRemote only looks at LastLocalSyncTsNs when RemoteMtime and the mtime share a second, so it still reads the entry as dirty, and the event is stamped again. Nothing is uploaded, but the daemon keeps rewriting the entry as fast as it reads its own events back. Skip the restamp when the backend left the object alone and the event's entry already carries a stamp taken after its mtime. The touch itself is still stamped once. * filer.remote.sync: log the skipped restamp The other skips in processUpdateEvent log at V(2); this one did not, so with -v=2 a touched file showed an "update meta" line with nothing after it. * filer.remote.sync: skip the restamp of a stamp's own event, whatever the clocks The skip compared the entry's stamp with its mtime, so an mtime ahead of the sync's clock (a skewed writer, touch -d) stayed past every stamp and kept the loop going until the clock caught up. An event that changed only the RemoteEntry is the sync's own stamp, which needs no other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: ignore the filer's commit timestamp when spotting a stamp The filer gives every stored update a new CommitTsNs, so a stamp's event never compared equal outside the RemoteEntry and a touched file kept being restamped against a real filer. The test now assigns one per update. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
23598afb31 |
volume: bound each replica forward so a silent peer cannot hold a write (#12417)
* volume: bound replica forwards so a silent peer cannot hold a write The Rust volume server forwards each replicated write to its peers with the shared outgoing reqwest client, which had no connect or request timeout. When a peer is silently partitioned (packets dropped, no reset) the forward waits until TCP gives up, minutes later, and the write waits with it. A chaos test (replication 001, one volume server partitioned) saw every write on a volume with a copy there block. - Replica forwards go through replica_request, bounded at REPLICA_REQUEST_TIMEOUT = 10 s from connect to the response. - Both outgoing clients start from outgoing_client_builder, which adds a 5 s connect timeout, so new connections to a peer that never answers fail fast. The shared client gets no overall request timeout, since it also serves lookups and other transfers. Tests: a replica that receives the forward and never answers fails the whole replicated write at the bound, naming that replica; a silent peer fails the request builder at the bound on a paused clock; a connect that stalls in the resolver fails at the 5 s connect bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: give a replica forward more time for a larger body The 10 s replica forward bound runs from connect to the end of the response, so it also covers sending the body. A write near the 256 MB file size limit needs about 26 MB/s to reach its replica in 10 s, so on a link that is slower or busy the forward failed and the write returned 500 even though the peer was answering. A forward now gets REPLICA_REQUEST_TIMEOUT plus one second per MiB of body. A bodiless forward keeps the 10 s bound (a 1 MiB write gets 11 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: bound each Go replica forward like the Rust one The Go volume server forwards a replicated write with a 10 s dial bound and the transport's 30 s response header timeout. A replica that takes the forward and never answers held the write for 30 s, and a body larger than the socket buffers to a silently partitioned peer was held until TCP gave up, since nothing bounds sending the body. Each forward now runs under replicaRequestBound: 10 s plus one second per MiB of body, the same bound the Rust volume server puts on its forwards. Test: a replica that receives the forward and never answers fails the write at the bound (300 ms in the test); without it the write fails at 30 s on the response header timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: keep proxy variables out of the stalled-connect test reqwest reads HTTP_PROXY and ALL_PROXY by default, so with a proxy set the test's request went to the proxy instead of the stalled resolver. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: count a replica forward that hit its bound as a timeout DistributedOperationResult joined the replica errors into a plain string, and the uploader reports a cancelled request without wrapping the context error, so a forward cut off at replicaRequestBound was counted under server_error in the replication failure metric, as was a response header timeout before it. The joined error now unwraps to each replica's error, and a forward that ran out of its bound reports context.DeadlineExceeded. Test: the silent replica write raises the timeout failure count by one; with the flattened join it stays put. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: test that a large Go replica forward gets its body's bound A replica answers after the bodiless bound but within the bound for a 1 MiB body, so the write fails if replicateTo drops the body scaling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: run the Rust whole-path replica test with a 1 MiB body The forward then has to wait the bound for its body, so the test fails if do_replicated_request stops passing the body length to the bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: pingqiu <pingqiu@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
aab947b65e |
s3api: answer success when a failed completion already committed (#12416)
* s3api: name the check that a completion already committed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: answer success when a failed completion already committed A finalize can fail after its commit: the filer stored the object and the call still reported an error. The client then got a 500 with an unknown outcome. When the failure may have left this attempt's object behind, check the key the way a retry would and answer success with the committed result when it holds this upload's object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: keep part data when the commit found is another attempt's The object carrying this upload ID can be a concurrent attempt's commit of the same upload, and its part selection can include entries this attempt counts as unused. Free those parts' data only when the committed chunks are this attempt's own; otherwise drop their metadata only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test the committed-completion read on versioned buckets Cover the latest-version read (version id and checksum in the answer), a suspended bucket whose delete marker is still current, and that the upload directory is removed without its data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test versioned recovery through the completion handler Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
fe1cf5263f |
volume: bound each FetchAndWriteNeedle replica write (#12415)
* seaweed-volume: bound each FetchAndWriteNeedle replica write A replica that accepted the connection and then went silent held the request until TCP gave up, because the shared outgoing client has no request timeout. Each replica write now fails after 30 s, from connect to the response status, matching the Go volume server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: bound each FetchAndWriteNeedle replica write The default replica path uploads through the guarded peer client, which bounds only the dial and has no response header timeout, and the filer calls FetchAndWriteNeedle with no deadline. A replica that accepted the connection and went silent held the request for minutes. Each replica write, retries included, now has 30 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: require the silent-peer test to fail at the deadline A connection error that came back early also passed, since the test only checked an upper bound. It now also requires the paused clock to have reached FETCH_AND_WRITE_REPLICA_TIMEOUT. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: grow the FetchAndWriteNeedle replica bound with the chunk The reqwest timeout includes sending the body, and a remote-cache chunk is a thousandth of the object once it passes the 1000-chunk cap, so a fixed 30 s would fail a large chunk on a link under ~35 MB/s. The bound is now 30 s plus 1 s per MiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: grow the FetchAndWriteNeedle replica bound with the chunk The context deadline includes sending the body, so a fixed 30 s would fail a large remote-cache chunk on a slow link. The bound is now 30 s plus 1 s per MiB, matching the Rust volume server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
247e010865 |
volume: redial a master that just dropped a registered stream after a second (#12420)
* volume: redial a master that just dropped a registered stream after a second A registered volume server that loses its master with no peer naming a leader waited a full pulse before redialing. A restarted master is usually back within a second, so its volumes stayed unregistered, and uncached lookups kept failing, for most of a pulse. For one pulse after a stream a leader had answered ends, redial on the one-second cap. A master still down after that is redialed at the pulse pace, so redials never come later than before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: redial a master that just dropped a registered stream after a second Mirror of the Go heartbeat change. The loop keeps when the last stream a leader had answered ended, which also stands in for the confirmed-leader-once flag (None until first registration), and redials on the one-second cap for a pulse after that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: test that a dropped registered stream records the loss A fake master confirms the stream and then ends it. The server starts as registered long ago, so only a recorded loss makes the redial come after a second instead of a pulse. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: record a lost stream only after its receiver has exited doHeartbeatWithRetry can return on a send error while the receiver is still handling the leader's first reply, so the deferred recorder could read the registered flag too early and keep an old loss time. Cancel the stream and wait for the receiver before reading it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: require the prompt heartbeat redial to wait out its second Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: fail the down master's dials at once in the redial test Windows takes about two seconds to refuse a connection to a closed loopback port, so the stream dial and the leader lookup each held the prompt case past its two-second bound. A listener that closes every connection fails both dials immediately on every platform. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
8cc3a8ae21 |
ec.rebuild: repair shards that fail a checksum scrub in complete volumes (#12410)
* ec.rebuild: rebuild shards that fail a checksum scrub, also in complete volumes ec.rebuild skipped every volume with all shards present, so a shard whose bytes no longer match the .ecsum sidecar stayed in place until some other shard went missing. With -checksum, ec.rebuild first scrubs each node's shards in checksum mode and takes the failing copies out of the plan; a shard left without a good copy counts as missing and is regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: remove the copies that failed the checksum once a good copy serves A shard regenerated on the rebuilder left the failing copy mounted on its old holder, so the volume kept a corrupt duplicate that a later dedup could keep instead of the good one. Remove each failing copy once the rebuilt shard is mounted, and right away when another holder already serves a copy that passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: test the checksum scrub against fake volume servers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: check the lock before removing a corrupt copy Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: print the checksum scrub details A malformed sidecar or a wholesale mismatch comes back only in the scrub details, with no broken shard, so the volume would look verified. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: fail the run when a corrupt duplicate cannot be removed For a volume whose only problem is a corrupt duplicate, the removal is the whole repair, so its failure has to reach the caller. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: test duplicate removal failures and a lost lock Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: fail the run when a node's checksum scrub fails The node's copies stay in the plan unverified, so a corrupt shard there is left in place; the rest of the cluster is still repaired. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: preview the corrupt-copy removals in dry-run Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec.rebuild: test a failed checksum scrub Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Fable 5.1
|
e465880ee8 |
weed-worker: build the vector index of S3 Vectors tables past a row threshold (ANN phase 2) (#12413)
* seaweed-vector: build and extend the vector index weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the table was created with, sizes partitions from the row count, and extends an index with a new segment or a retrain; the core crate re-exports it and adds the probes a query spends. Tests cover coverage after build, append, retrain and compaction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector-cli: index build and index optimize Logs go to stderr so stdout stays the JSON a caller parses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_INDEX builds the index before the timed queries The cluster fixture hands out its catalog URL, the Makefile builds weed-vector-cli, and the perf suite builds the index after ingest and reports the build time and segment count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: probe a tenth of the index's partitions, more under a filter A query that names no probe count gets one from the index's partition count. Lance searches at least that many partitions, and under a prefilter keeps going until it has k rows; without one it stops there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: --nprobes overrides the probe policy Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that a build takes the table's metric Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make clean removes weed-vector-cli Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: measure recall with vectors the table does not hold Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep the read cache's doc comment on with_read_cache Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold Detection keeps the tables a vector bucket created, with at least min_rows rows (50,000 by default) and no index on their vector column. Execution builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: optimize only the indices on the vector column Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: resolve the query metric with index::distance_metric Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test the L2 metric under the name CreateIndex stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: build the index through the admin server and the Lance worker VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new test sets the job's threshold, runs it through the plugin API, checks coverage and recall, and that a row written after the build is found. CI gains the worker leg for both volume servers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: open a table inside a caller's Lance session Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep opened indices between queries in one session Every open used a fresh Lance session, so the index and metadata caches died with the request and each query loaded the IVF index again. --index-cache-mb sizes the shared cache, 512 MiB by default. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the index test draws embedding-like vectors Independent Gaussian vectors have no locality for an IVF index to use, so recall through one only tracks the fraction of partitions probed; points around shared centres behave like embeddings. The test must also run before the one that stops the engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: test that compaction drops the deleted key from tied hits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: test that an append-only optimize serves the new row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: record PERF_NOISE in the perf report config Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: a filtered query widens to four times the probes, not without bound With no upper bound Lance prepares every partition for the late search even when the minimum probes already hold k rows, and a filtered query through the index cost as much as the exact scan. A query that names its own probe count gets exactly that count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: report recall through an index without asserting it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a hit tying the k-th best score toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: lance_vector_index runs in the Lance lane Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: check a table's index before counting its rows Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: make build and clean cover the worker leg's binaries Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: allow float32 rounding at the recall cutoff Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: count a repeated hit key once toward recall Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: credit cutoff ties only up to the slots the top-k leaves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: assert every filtered hit passes the filter, indexed or not Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: forget the shared session when a table was dropped and created again The session keys index lists and transactions by dataset version, and a table created again at the same location restarts its versions. An open that finds an older version than the newest seen at that URI, or the same version committed later, replaces the session and opens again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: test an open after another engine created the table again Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: replace the session in the same lock that records a recreated table A second open of the recreated table could see the new version recorded and the old session still shared, and keep the dropped table's index list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-worker: skip a vector table whose rows cannot be counted Detection returned on the first table whose count_rows failed, so one unreadable table kept every other table from being proposed. It now skips that table like an open or index failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: panel for vector indexes the Lance worker built Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-vector: size default probes from the largest index on the column A sharded build gives the column one index name per shard. The first name alone could be a small shard, and its probe count would then be spent on a larger one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: skip the probe lookup when a query is given no segments Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: open every table in its own Lance session again One session shared by every open keeps a table's index loaded between queries, but Lance keys what it caches by location and version, so a table dropped and created again at the same location could be served the old table's index list. Telling the two apart from version and commit time needs a map that only grows, a retry loop and clocks that agree across engines. A private session per open, as before, has no such state. Measured at 50k x 384 through an index, eight clients: 164 queries/s with the shared session, 137 without; one client is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: reject --nprobes 0 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: read the partition count from the opened index Lance's index statistics took 3.6 ms a query on a 200k-row table with 447 partitions. Opening the index and asking it takes 0.4 ms, and the scan that follows finds the index already open. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * ci: give the S3 Vectors worker legs the vendored protoc The worker workspace has no protoc of its own for lance's build scripts, so the worker legs failed at their first build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: give a vector index build an hour before the admin times it out The job set no execution timeout, so the admin's default of 90 seconds applied, and a build that trains on a large table would be reported as failed and retried while it was still running. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: lance_vector_index proposes at most the admin's max_results Every qualifying table went into one message however many there were. The detector stops at the limit the request carries and sets has_more, as the Go detectors do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index job re-checks the table it was proposed for A table emptied since detection completes without a build, and one without a vector column fails instead of reporting it already indexed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: a vector index run window as long as one build An attempt is capped at the end of the job type's run window plus the drain grace, and the default window is 30 minutes, so a build could be cut off before its own hour was up. The window is an hour too, as the Go iceberg handler sets both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris Lu
|
683c5ad4d3 |
admin: regenerate the plugin pages' templ output from weed/admin (#12405)
Both were generated at the repo root, so their embedded source paths read weed/admin/view/... where every other admin template, generated by weed/admin's Makefile, reads view/.... No markup changes. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi |
||
|
Chris LuandClaude Opus 5.5
|
7047d3947a |
admin/plugin: fail a dispatch its restarted worker no longer lists (#12402)
* admin/plugin: fail a dispatch its restarted worker no longer lists A job dispatched to a worker that then restarts dies with the old process. The worker re-registers under the same id on a new stream and its heartbeats stop listing the job, but nothing ended the pending execution: the dispatcher kept waiting and held the executor slot until the attempt's execution timeout. Each pending execution now records the stream it went out on. An itemized heartbeat from a newer stream of that worker that leaves the job out twice in a row fails it through the expire path, so the dispatch returns, the slot is released and the retry machinery takes over. Jobs on the heartbeat's own stream, jobs the new stream still lists, and heartbeats that count slots without itemizing them are left alone. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * admin/plugin: let a partly itemized listing clear a dispatch's misses A heartbeat that counted more slots than it itemized returned before the listed jobs had their misses reset, so an omission, a partial heartbeat listing the job, and another omission failed a job the worker was still running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: look up the worker's stream before tracking the attempt A dispatch to a worker with no stream returned after the job was already recorded as assigned, and that record kept new proposals with the same dedupe key out until the stale-job sweep expired it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: fail the attempt whose request could not be sent The dispatch now sends on the stream it looked up before tracking the attempt, so a worker that reconnects in between makes the send fail. That returned with the job still tracked as assigned, and with the worker back nothing closed the record for the 24h stale timeout while its dedupe key kept new proposals out. The attempt is now completed as failed, after it is unregistered so the completion does not wait on a channel nobody reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: leave a send failure to the expiry that already claimed it The send-error path and an orphan expiry could both complete the same attempt and record two failed runs for it. The send-error path now completes the attempt only if no expiry has marked it failed, decided under the same lock that unregisters it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: record a send failure whose entry Shutdown already removed Shutdown closes shutdownCh before it unregisters pending dispatches, so a blocked send returns while its map entry is gone. Read the expiry claim from the dispatch's own state instead of the map lookup, so the attempt is still failed and does not stay assigned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: match a padded job id against its trimmed listing Heartbeat work ids are trimmed before the orphan check, so compare the dispatch's job id trimmed too; otherwise a job id with surrounding spaces never matches its own listing and the running job is failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin/plugin: skip orphan misses once a completion was handed over The buffer length only shows a completion the dispatcher has not taken yet; after it takes one, the attempt stays registered until the dispatch returns. Mark the hand-off under the pending lock and check that flag instead, so heartbeats in that gap cannot fail a finished attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3d411c6a6c |
volume: report EC file and delete counts in incremental shard messages (#12404)
* seaweed-volume: report EC file and delete counts in heartbeat deltas The full EC heartbeat already carries file_count and delete_count, but the incremental new/deleted shard messages built in collect_ec_shard_delta_messages left both at zero. A volume the master first learns of through a delta then reads 0 files until the next full EC heartbeat. Fill both from file_and_delete_count, the same source the full heartbeat uses, computed once per volume rather than once per shard. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * volume: report EC file and delete counts in mount notifications Go parity with the Rust delta change: the NewEcShardsChan messages sent on MountEcShards and on startup shard load now carry FileCount and DeleteCount from EcVolume.FileAndDeleteCount, the same values the full EC heartbeat reports, instead of leaving them at zero. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * seaweed-volume: leave count-only EC changes out of the shard delta The delta messages now carry file_count and delete_count, so the whole-message comparison re-sent every shard of a volume after any delete, and the master broadcast each one as a new EC vid. Compare the shard fields only; the full EC heartbeat already carries the counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * topology: mount messages now carry EC counts, only older servers omit them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: guard the captured EC mount messages in the disk-type tests The drain goroutine appended to the slice that waitForShardMsg polled with no lock, which go test -race reports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: test EC counts on the startup notification and repeated .ecj ids Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: move file_count too in the count-only delta test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3cbace2471 |
s3: expose ReadAcp/WriteAcp in shell and admin UI, log ACL-upload denials (#12396)
* shell: accept ReadAcp and WriteAcp in per-bucket and service account actions s3.bucket.access, s3.anonymous.set and s3.serviceaccount.create rejected ReadAcp and WriteAcp, so the ACL actions the S3 gateway enforces could not be granted through them. Share one canonical action table across the three commands and list the ACL actions in their help. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: list ReadAcp and WriteAcp in s3.configure and s3.circuitBreaker help The -actions usage of both commands stopped at Admin, so the ACL actions the gateway enforces could only be found in the source. s3.circuitBreaker now prints the actions it actually accepts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: offer ReadAcp and WriteAcp in the object store user permissions The create and edit user forms only offered Read, Write, List, Tagging and Admin among the bucket permissions, so an identity could not be given the ACL actions from the UI even though uploads with x-amz-acl need WriteAcp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: log the WriteAcp denial of an upload that sets an ACL An upload or copy that carries x-amz-acl or an x-amz-grant-* header also needs WriteAcp. When that check denies, the client only sees a bare AccessDenied and nothing reached the log at the default verbosity, so the header was never suspected. Log the identity and the missing action. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: quote the key in the ACL upload denial log An object key can carry a decoded newline, which let a caller add forged lines to the default log. Quote the bucket and key, and word the message so it also fits a policy deny, not only a missing grant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: say ACL uploads need both Write and WriteAcp The s3.configure example granted WriteAcp alone, which still fails the upload's Write check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: parse S3 action lists with one shared helper s3.bucket.access, s3.anonymous.set and s3.serviceaccount.create each had their own validation loop, so a test of the table did not cover the parsing the commands run. They now call canonicalActionList, which skips empty entries and repeats. s3.anonymous.set validates before contacting IAM, and s3.bucket.access still rejects a list with no actions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep bucket-scoped actions the user form cannot show When a user has both global and bucket-scoped actions, the edit form shows only the global ones, and saving replaced the identity's actions with what the form rebuilt, deleting the scoped grants. Carry those actions through the save. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: list hidden bucket grants in the user edit form Restoring the bucket-scoped grants the form cannot show kept them active even when an admin narrowed the user's permissions, with no way to see or drop them. Show them as checked boxes under Other Bucket Permissions; unchecking one removes it on save. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: reject an s3.anonymous.set -access list with no actions A blank list such as ",," parsed to nothing and cleared the bucket's anonymous grants. Use "none" for that, as s3.bucket.access does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: skip the ACL upload denial log for anonymous requests Anonymous uploads with ACL headers can be sent by anyone, so logging each denial at the default level lets unauthenticated clients flood the log. Keep the line for named identities. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: count kept bucket grants when checking an edited user for permissions The empty check ran before the checked Other Bucket Permissions were merged, so a user left with only those grants could not be saved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: list bucket grants of Admin users in the user edit form parseBucketPermissions returned before reading scoped actions when the user had Admin, so removing Admin and saving dropped grants such as WriteAcp:bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep unpaired bucket grants as separate checkboxes in the user edit form The form saves every selected permission on every selected bucket, so grants such as ReadAcp:a with WriteAcp:b came back as all four pairs. Show the grants through the picker only when they cover that full product; otherwise list each grant on its own. Opening the form also resets the bucket scope left over from the previous user. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep grants the edit pickers have no option for A permission x bucket product was shown through the pickers even when a bucket was not in the bucket list (deleted, prefix or wildcard grants) or a permission had no picker option (DeleteBucket, PutBucketQuota). Saving then dropped those grants. Such grants now go to the checkbox list, global ones included, and are submitted unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: name the s3.bucket.access anonymous example by what it grants Read,Write,List leaves out ReadAcp and WriteAcp, so it is not full access. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep unlisted global grants of Admin users in the user edit form parseBucketPermissions returned for Admin before splitting global grants into picker options and the rest, and only raw option values were re-selected. Saving an Admin user dropped s3tables:* and grants with no picker option such as DeleteBucket. Admin users now use the same split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep the /* suffix of bucket grants in the user edit form parseBucketPermissions dropped a trailing /* before matching bucket options, so saving rewrote ReadAcp:bucket/* as ReadAcp:bucket. The bare form also matches bucket-level requests such as GET ?acl, so the save widened the grant. A /* bucket is not a picker option, so the grant now stays in the checkbox list as written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: test that s3.bucket.access saves ReadAcp and WriteAcp grants Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
c437791991 |
s3: reject POST Object header fields with control characters (#12400)
A Content-Type form field such as "\n" was copied verbatim into the multipart part header sent to the volume server, which broke the framing and surfaced as a Content-MD5 mismatch and a 500. Form fields promoted to request headers must be valid header values, as on a PUT; otherwise answer 400 MalformedPOSTRequest. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0742d316aa |
fix(s3): accept fresh admin tokens when context carries stale authorization (#12390) (#12397)
* fix(s3): accept fresh admin tokens when context carries stale authorization (#12390) Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com> * security: share the admin Bearer check between S3 and admin-plane peers Move the authorization scan into security.CheckGrpcAdminToken, built on IsFilerAdminToken, and call it from CheckGrpcAdminPeer and the S3 IAM cache check. Both now accept a valid admin token in any authorization value instead of S3 re-implementing the decode and claim checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: accept a fresh admin token after a stale one on IAM gRPC The S3 IAM cache check is meant to mirror the filer's, so route the filer's checkAdminAuth through security.CheckGrpcAdminToken too. IamGrpcStore appends its token to the caller's context, so a forwarded older token would otherwise still shadow it here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * security: parse at most four authorization values per admin call Scanning every value let one request repeat a junk Bearer token (cheap over HPACK) tens of thousands of times and burn about half a second of JWT parsing on an unauthenticated port. Real callers send their own token plus at most one forwarded from their context. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * credential: match the other WithS3InternalAdminAuth replace fix The plugin worker fix makes the same set-instead-of-append change to WithS3InternalAdminAuth with different code and a different test at the same spot, so whichever lands second would conflict. Use the identical code and test so the two merge cleanly in either order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * security: test an expired admin token ahead of a fresh one Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * security: test that a same-key session token is not an admin token Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test IAM gRPC with only invalid authorization values Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
51ee165279 |
ec: rebuild a needle when other shards are corrupt over its range (#12401)
* storage: search corrupt shard sets when an EC needle fails to rebuild A needle whose reassembled bytes fail to parse is rebuilt with its own shards excluded. When other shards are corrupt over the same range, that decode reads them as inputs, and the retry dropped only one input at a time, so two or more extra corrupt shards left the needle unreadable even within parity. After the single pass fails, try every set of up to ParityShards shards that holds a suspect as the corrupt set, smallest first. Suspect ranges inside the set are rebuilt from the shards outside it and the rest keep their bytes, so a clean suspect can still feed the decode. Sets whose decodes read the same inputs are tried once, and the search is bounded by set count and by rebuilt bytes past the single-shard sets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: search corrupt shard sets when an EC needle fails to rebuild Mirror the Go read path: after the single pass over the suspects fails, gather each interval from every other shard once and try each set of up to parity_shards shards holding a suspect as the corrupt set, smallest first, with the same dedup and bounds. The search runs on the blocking pool, holding the recovery permit, since it can take many decodes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: charge only multi-shard sets against the rebuild byte budget The budget check already let single-shard sets through, but each one was still charged, so a needle spanning many shards could spend the budget before the first multi-shard set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: bound the corrupt-set gather to the recovery budget The search gathered every other shard per interval, so a large needle spanning several shards held up to Total-1 copies of itself under a permit capped at the budget. Gather only as many as the budget covers, but never fewer than the suspects-excluded gather the fallback always held, so single-suspect recovery keeps every other shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: gather at least one decode's worth for the corrupt-set search A needle over more shards than parity skips the single pass, and its suspects-excluded width can fall below DataShards, so the bounded gather left even one corrupt shard unrecoverable. Floor the width at DataShards, which the per-suspect rebuild this search replaced read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: rebuild one suspect at a time when the search gather is narrow Flooring the search gather at DataShards made a large needle over more shards than parity hold ten copies of itself, where the per-suspect rebuild it replaced held one interval's fan-out. Keep that rebuild after the search, and skip the search when the budget leaves fewer inputs than one decode. Parity slack still widens the gather to every shard outside the suspects, as the dropped-input retry did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: keep the dropped-input retry and run the set search within budget A narrowed search gather skipped only each interval's own shard, so it could read corrupt suspects in place of healthy shards the old retry kept, and its byte bound could stop before that retry's hypotheses. Restore the dropped-input retry as it was, move the search into its own function after it, and run the search only when every interval's full gather fits the recovery budget. Past the budget a read takes the same paths, with the same footprint, as before the search. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d6c65cd924 |
volume: count EC intervals served by reconstruction (#12392)
* volume: count EC reads that fall back to reconstruction
When a needle read can't read an EC shard interval directly (the shard isn't
mounted here and no holder answers), both volume servers rebuild it from the
other shards and the read still succeeds, with nothing to show for it.
Count those fallbacks in SeaweedFS_volumeServer_ec_degraded_read_total{result}
(success, failure), one per interval, at the one place each server falls back:
Go readOneEcShardInterval, Rust fetch_one_interval. Corruption repair and scrub
aren't counted. A line binding each one to the needle and the volume's encode
generation is logged at V(3) in Go and debug in Rust.
* volume: name the EC counter for what it counts
Reed-Solomon reconstruction can't tell corrupt inputs from good ones, and the
needle is validated later, so "success" only ever meant the reconstruction
produced data. Rename the counter to
SeaweedFS_volumeServer_ec_reconstructed_interval_total and the log line to
"ec reconstructed interval", say so in the help text, and pin the behaviour
with a test: an interval rebuilt from corrupt parity counts success while the
read itself fails.
* storage: restore stderr when the log-line test stops early
A fixture failure in readEcNeedle ends the test through t.Fatalf, which
skipped restoring os.Stderr and closing the pipe, so later tests wrote
into a pipe nobody read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
15ff5c113a |
fix(s3tables): validate policy conditions and fail safe on unusable ones (#12389)
* s3tables: validate policy conditions and fail safe on unusable ones * s3vectors: reject a condition operator with no keys The shared evaluator now treats a keyless operator as unusable, so a stored keyless Allow would never grant while the write succeeded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: reject an Allow whose Condition is null A null Condition decodes as no Condition, so the Allow granted unconditionally. A null Condition on a Deny only widens the veto and stays accepted, as in the S3 Tables validator. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: treat a NaN numeric condition value as unreadable strconv.ParseFloat accepts "NaN", but no comparison with it holds, so a Deny on NumericLessThan never applied and an Allow on NumericNotEquals matched every request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: reject condition keys the evaluator never supplies A condition on a key such as aws:SourceIp reads no request value, so a Deny on it never applied and an Allow never granted. Accept only the keys getConditionContextValues resolves, as the vector validator does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: require Statement and Effect spelled as written The evaluator decodes field names regardless of case, so an "effect" or "statement" key slipped past the null Condition check and the other raw-key checks while the stored Allow still granted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: treat an infinite numeric condition value as unreadable Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: a stored Allow with a null Condition never grants The typed decode reads "Condition": null as no Condition, so an Allow stored before validation refused it still granted unconditionally, for table and vector policies alike. Record the literal null while decoding a Statement and skip such an Allow during evaluation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: validate a null Condition from the decoded Statement validatePolicyDocument re-parsed the document to find a literal null Condition; the Statement decode now records it, so the validator reads the same flag the evaluator does and nullConditionStatements goes away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: keep Condition objects merged across key spellings Decoding Condition through a json.RawMessage kept only the last of "Condition" and "condition", so an empty object under one spelling erased the other's restriction. Decode the statement as before and probe for the literal null separately. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: refuse a Condition split across key spellings Condition objects under differently cased keys decode into one map, and an operator they share keeps only the last spelling's keys, so a restriction under the other spelling was dropped. Validation now refuses a statement whose Condition is under more than one key, and one stored before that never grants on an Allow and vetoes on a Deny. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: refuse a Condition key repeated under the same spelling PolicyDocument re-marshaled each statement through interface{}, and Statement.UnmarshalJSON counted keys from a map, so both kept only the last of a repeated Condition key. "Condition":null,"Condition":{} or two Condition objects sharing an operator passed validation with a restriction dropped. Statements now decode from their original bytes and the keys are counted while walking the object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: refuse a repeated Condition key in a vector bucket policy The validator read Condition from a map, which keeps only the last of a repeated key, so it accepted a statement the evaluator never grants on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: keep the re-marshaled statement decode, count Condition keys apart Decoding statements from their original bytes let document order pick between spellings of every field, where the re-marshal sorted the keys first, so a stored "effect":"Deny","Effect":"Allow" became an Allow. Statements decode as before again; only the Condition keys are counted from the original bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: test that validation refuses a split Condition on a Deny Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: test that validation refuses a repeated Condition on a Deny Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d727c6970f |
plugin worker: stop forwarding the stream's admin token to job RPCs (#12391)
* plugin worker: keep the stream's admin token out of job contexts The bearer token minted when the worker opens its control stream was attached to connCtx, which every detection and execution context derives from. Job RPCs therefore forwarded that connect-time token, and when a job added its own fresh token (s3_lifecycle's LifecycleDelete) the S3 server validated the stale one first and rejected the call once it expired. Attach the token only to the WorkerStream call, as the Rust worker does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * credential: replace an inherited admin token instead of appending one WithS3InternalAdminAuth appended its fresh Bearer token, so a context that already carried an authorization value sent both, and the S3 gateway's checkAdminAuth validates only the first. Set the header on a copy of the outgoing metadata so the fresh token is the only one sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * plugin worker: read the captured job metadata after the stream ends Both result channels could be ready at once, letting select fail a successful run. The handler reports before JobCompleted reaches admin, so wait for the stream to finish and then read the captured metadata. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * plugin worker: test that dropping the connection cancels running jobs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a0f30afe5f |
s3: keep service account keys when a pushed identity replaces its parent (#12394)
* s3: keep service account keys when a pushed identity replaces its parent Service account changes push the parent user, whose proto carries only the user's own keys. The merge replaced the cached parent with that proto and dropped every service account key until a restart. Read the parent's service accounts from the credential store so the push rebuilds them, deleted and disabled ones included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test that a parent identity push keeps its service account keys Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test a sibling service account survives a deletion push Also check the restored secret and status, not just the key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
ba3fd257a3 |
filer: clear a dead test replica's reachability record before closing it (#12387)
* filer: clear a dead test replica's reachability record before closing it The read path tries hosts that recently failed last. A dead test server can land on a port an earlier test left marked as failed, so the live replica answered first, nothing failed and the location refresh under test never ran. Have the dead replica answer once through util_http so its record is cleared before it closes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: close the dead test replica when reaching it fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: clear the live test replica's reachability record too Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
329ccb8bc0 |
master: tell a warming-up Assign or LookupVolume client when warm-up ends (#12386)
* master: tell a warming-up Assign client when warm-up ends During warm-up the master refuses Assign with Unavailable and now attaches the remaining warm-up as a gRPC RetryInfo detail. RetryWithBackoff may shorten a backoff step to that hint plus 50 ms, never lengthen it, and only for a valid, positive hint, so a writer retries right after warm-up ends instead of up to one backoff step later. The warm-up itself is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: jitter the retry-hint slack and hint LookupVolume too Per review: the slack added to a server hint is now random in [50ms, 300ms], so clients refused at the same moment do not all retry in one instant. The warm-up refusal of LookupVolume carries the same RetryInfo hint as Assign; the masterclient's lookup retry already goes through RetryWithBackoff, so it lands at warm-up end too. Tests: the slow guard tests are replaced by one sleep-free table test of the step logic (zero, negative, huge, malformed, valid and wrapped hints, a hint longer than the step, none). One timing test of the shortened step stays, plus a jitter-bounds test and a LookupVolume hint test. The no-hint control test, which was timing-sensitive, is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: name the shared warm-up test and file the lookup test by topic Rename the no-remaining test to TestWarmingUpErrorWithoutRemainingHasNoHint, since the helper now serves Assign and LookupVolume, and move the LookupVolume hint test next to the other LookupVolume warm-up tests. Tests only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * util: read the retry hint through ServerStatus Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: bound the warm-up hint in tests by the remaining warm-up around the RPC Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: leave more scheduler margin in the retry-hint timing tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * util, master: trim retry-hint comments Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: widen the retry-hint timing bounds to just under the unhinted wait Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: require a refused warm-up call before the hinted assign retry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: check the hinted retry wait through a test timer, not the wall clock Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: move the assign warm-up end to 2.6s for more hinted-retry headroom Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: require the hinted retry to wait out its timer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandClaude Opus 5.5
|
cd82e663ad |
admin: credit an unitemized slot to a pending job only after a later heartbeat (#12385)
* admin: credit an unitemized slot to a pending job only after a later heartbeat A pending reservation the worker does not list took one of the slots the worker counts without itemizing, even when that heartbeat came before the reservation and could not have counted the job. The slot belonged to unknown work, so the admin undercounted it and could admit one job too many, past a job type's per-worker limit, until the next heartbeat. A reservation now records when it was made, and claims an unitemized slot only from a heartbeat received at or after that time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: time unitemized-slot credit by the heartbeat, not the last hello A reconnect's hello moves LastSeenAt but keeps the old heartbeat, so a pending reservation could claim a slot counted before it existed, and a completed job's echo could end without a new heartbeat. Record when the heartbeat itself arrived and compare reservations against that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: a completed job claims no slot from a heartbeat before its reservation A job that completes before any heartbeat counts it has no echo, so the slot the last heartbeat counted is other work. Crediting it to the job's type let another job type pass its per-worker limit until the next heartbeat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
5dfda3645e |
admin: count executor slots per job type and stop lanes waiting on capacity they cannot free (#12384)
* admin: count executor slots per job type, deduped against the heartbeat per_worker_execution_concurrency is a per-job-type setting, but the scheduler compared it against the worker's execution_slots_used, which counts every job type. One s3_lifecycle job a worker reported running after a reconnect made admin_script (per-worker 1) undispatchable on a worker with three free slots. A reservation and the heartbeat entry for the same job were also charged as two slots, so a per-worker limit of N let N-1 jobs run. Reservations now carry the job id and job type. The per-type limit is checked against that type's work alone, the worker's slot total against work of every type, and a heartbeat entry covered by a reservation is counted once. Per-lane reservation pools go away: the per-type count is what kept lanes from starving each other. * admin: stop a lane waiting on executor capacity it cannot free A queued job no executor could take was retried every backoff until job_type_max_runtime ran out, up to 30 minutes, while the default lane sat in executing and every other job type in it idled. The dispatch loop now asks what holds the capacity. Slots held by the lane's own jobs release when they complete, so it keeps waiting. Slots held by other lanes or by work this admin never dispatched clear on a schedule the lane has no view of, so the queued jobs are canceled at once with a skipped_no_executor activity naming the worker, its slot count and the job types holding it, and the next detection proposes them again. A missing worker is waited for only the orphan grace. * admin: cancel a job on its worker when the admin expires it Expiring a job, by the stale sweep or by the operator's expire endpoint, only failed the tracked record. The worker never heard about it and kept running the job with its slot taken, so a job stuck in "starting" held a slot for over a day and, with the old slot accounting, blocked a whole lane. Expiring a job now also sends the worker a cancel for it, so the slot comes back on the next heartbeat. * admin: count a completed job's heartbeat echo as its lane's own slot The worker sends JobCompleted before the heartbeat that frees the slot, so right after a dispatch goroutine releases its reservation the job still shows in the worker's running work. That entry was counted as foreign, and a lane with per-worker 1 gave up and canceled the rest of its queue after the first job. A released reservation now stays for completedJobEchoGrace and claims that heartbeat entry as the lane's own load. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: let the admin's own jobs claim slots a worker does not itemize The Rust worker reports only a slot count, every 10s, and the Go worker counts a slot one heartbeat before it lists the job. Those slots were charged on top of the reservations behind them, and once a job completed its stale count read as foreign work, so the lane gave up its queue. Unitemized slots now go first to this admin's unlisted reservations and recent completions; the rest are of unknown type and count against every type's per-worker limit, not only the slot total. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: give up pinned jobs whose busy home a free worker cannot relieve A job homed on a replica-host worker stays on that home while its queue is shorter than LocalQueuePin, so a free remote worker cannot take it. The outlook treated that free worker as a reason to wait, and a home held by foreign work parked the lane for its whole window. A free worker now only holds the wait for the orphan grace, after which the held homes decide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: let a redispatched job's reservation replace its released one A job the worker turns away for capacity is released, kept as an echo, and queued again under the same id. Once the new attempt ran, both the echo and the new reservation matched its heartbeat entry and it counted as two slots. Reserving a job now drops what an earlier reservation of it left. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: credit a count-only slot to a completed job only until the next heartbeat A released reservation claimed an unitemized slot for its full 30s grace, so on a count-only heartbeat other work of another type was charged to the completed job's type. A heartbeat that arrives after the release already reflects the freed slot; from then on the slot is of unknown type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: name unitemized slots in the no-capacity reason holders() listed only the job types the worker itemized, so a worker with named and count-only work read as "2/2 slots used by <type>". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep the blocked clock running until a reservation succeeds A popped job whose reservation then failed reset the clock before the failure pushed it back, so pop-then-fail cycles never aged the block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: resend an expiry cancel the worker did not receive Expiring a job sends its worker one cancel. When the worker is disconnected, or its outgoing buffer stays full past the send timeout, that cancel is dropped. The record is already failed, so later sweeps and expire calls skip it, and a worker that kept the job running (a detached job across a reconnect, or a jammed stream) holds the slot until the job ends on its own. The admin now holds an undelivered expiry cancel and sends it again on the worker's next heartbeat that still lists the job. It is forgotten when that heartbeat no longer lists the job, and skipped when the job is active again, since a new attempt reuses the job id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: cancel an expired attempt by its request id The expiry cancel named the job id, and a retry reuses it. The worker keys its cancel funcs by both ids, so a cancel arriving after a retry started on the same worker stopped the retry. The held resend also gave up on any job that was active again, and the expired attempt's own late progress marks the job running, so a worker that buffered progress while disconnected kept the expired job. The cancel now names the attempt's request id, as the other execution cancels do, and the held resend no longer depends on the tracked state. Only a cancel that can name nothing but the job id still skips a job that is active again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: ignore late reports from an attempt the admin already ended Expiring a job now cancels it on its worker, and the worker answers with its own JobCompleted for that attempt. That overwrote the expiry's failure with the cancel error and logged a second run record under the same run id. Late progress from the expired attempt likewise reopened the record as running. A progress update or completion whose request id names an attempt the record already closed is now dropped after the dispatcher's waiter is told. A retry runs under a new request id, so its reports still apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
ca15fdbff6 |
remote.uncache, remote.meta.sync, remote.cache: keep local writes that land while they run (#12382)
* remote.uncache: skip a file overwritten since the walk read it The uncache cleared the chunks of the entry its directory walk returned and wrote it back unconditionally. A file overwritten after the walk read it was reset to the remote pointer, and its new chunks were dropped, so an acknowledged write was lost. Write back only while the filer still holds the entry as read, and report the file as skipped otherwise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * remote.meta.sync: keep files written locally after the directory listing meta.sync lists each local directory once and works from that listing. A changed remote file was written back over the listed entry unconditionally, and a name missing from the listing was created without O_EXCL, so a file written locally in between was replaced by the remote pointer either way. Update only while the filer still holds the listed entry, create exclusively, and report such files as skipped. The create now also surfaces errors the filer returns in its response. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * remote.cache: keep files written locally after the lookup The metadata step looks each file up and then either rewrites it from the remote unconditionally or creates it without O_EXCL, so a local write landing between the lookup and the write was replaced by the remote pointer. Reuse doSaveRemoteEntry, which updates only while the filer still holds the looked-up entry, create exclusively, and report such files as skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * remote.meta.sync: do not descend into a name a local file took When the exclusive create of a remote directory finds the name taken, the sync still descended into it, and creating the remote children under a local file would turn that file into a directory. Descend only when the name now holds a directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * remote.meta.sync: delete only the file version it listed The type-change and -deleteStale paths removed a remote-backed file unconditionally, so a local write landing after the directory listing was deleted with it. Condition the delete on the listed mtime, as fs.verify and volume.fsck do; the filer then keeps a file written since. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * remote.meta.sync: condition directory and mtime-less deletes too The listed-mtime cutoff was only sent for files, so a remote-backed directory replaced locally by a file between the emptiness check and the delete was removed unconditionally, and so was a file listed without an mtime. Send the cutoff for directories as well, and for an entry listed without an mtime keep it once a write has set one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test that DeleteEntry applies the request's mtime cutoff Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * remote.meta.sync: keep the listed nanoseconds as the cutoff of an epoch mtime An entry listed at the epoch with a sub-second mtime was sent a 1 ns cutoff, so the filer kept it unchanged although the old code deleted it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * remote.meta.sync: report an entry the filer kept instead of deleting it The filer answers a delete it skipped under the mtime cutoff with success, so the sync printed (delete) for an entry a local write kept. deleteLocalEntry now looks the name up after the delete and prints the outcome, and a replaced file that was kept is not recreated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b0c6297041 |
s3api: per-bucket Block Public Access (BlockPublicAcls, IgnorePublicAcls) (#12381)
* s3api: store and serve a bucket's PublicAccessBlock configuration PutPublicAccessBlock, GetPublicAccessBlock and DeletePublicAccessBlock answered NotImplemented. Store the configuration in the bucket entry's extended attributes next to the ownership controls, carry it in the cached BucketConfig, and answer Get on an unset bucket with NoSuchPublicAccessBlockConfiguration like S3 does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: refuse public ACLs when a bucket blocks them With BlockPublicAcls set, a PutObject, CopyObject, CreateMultipartUpload, POST Object, PutObjectAcl or PutBucketAcl whose canned ACL or grants name the AllUsers or AuthenticatedUsers group is refused with AccessDenied. Private and account grants keep working, so clients that send x-amz-acl: private on every upload are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: ignore public ACL grants for anonymous reads under IgnorePublicAcls A bucket with IgnorePublicAcls no longer lets an AllUsers grant on the bucket or on an object answer an anonymous read or listing. Bucket policies and identity permissions are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: reject PublicAccessBlock bodies with unknown elements A misspelled element such as BlockPublicAcl was ignored, so the request succeeded and stored the flag as false. Answer MalformedXML instead, and store the configuration without the request's namespace. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: fail closed when a bucket's public access block cannot be read A stored PublicAccessBlock that does not parse now blocks and ignores public ACLs instead of acting as if none were set. PutBucketAcl and POST Object return the bucket lookup error rather than skipping the BlockPublicAcls check when the bucket config cannot be loaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test BlockPublicAcls on copy, multipart initiate and a PUT-then-GET Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: keep stored public access block bytes on unrelated bucket updates An unreadable stored configuration was replaced with the in-memory all-true fallback whenever another bucket setting was updated, losing the original value. Write the key only when the update changes the public access block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris Lu
|
ab6aa9b645 |
admin: let a buffered detection completion win over the stream closing (#12379)
The detection waiter selects on the request's completion channel and on the stream's done channel. A worker that finishes a scan and then drops its stream (a restart right after a scan is the usual shape) buffers the completion into the request before the receive loop sees the EOF, and the stream's cleanup then closes done. A waiter that reaches its select after both are ready can take the disconnect branch, and a completed detection, proposals included, is failed as "detection worker disconnected" and retried or recorded as an error. When the stream closes or the liveness tick finds the worker gone, the waiter now takes a completion that is already queued and returns it as the success it is; the detector lease still goes, since the worker that answered is unreachable. A ctx that ended keeps its meaning: the caller stopped wanting the answer. The wait loop moves into awaitDetectionOutcome so a test can hand it both channels already ready; before this change that test fails on about half its runs. A completion that lands after the waiter gave up stays unmatched, since the request's pending entry is gone and the next request has a new id. |
||
|
Chris Lu
|
836c60b2c9 |
topology: refresh EC file and delete counts from heartbeats (#12378)
* topology: refresh EC file and delete counts from heartbeats UpdateEcShards only re-stored a heartbeated EC record when shard membership changed, so FileCount and DeleteCount stayed at whatever the first heartbeat after registration carried, while the volume server reports fresh tombstone counts on every full EC heartbeat. Per-collection object counts and EC vacuum candidate detection read the stored values. Compare the counts per (volume, disk) against the record already in hand and copy them onto the stored record in place, so the refresh raises no new/deleted shard events and rebuilds no maps. A message reporting 0/0 did not count (older volume servers, and the incremental mount message omits the fields), so it keeps the stored values, also across a membership-triggered rewrite. The incremental path's existing branch picks counts up the same way. * topology: swap EC shard records instead of mutating them in place GetEcShards hands out the stored *EcVolumeInfo pointers and releases ecShardsLock; CollectionStatistics, CollectionEcVolumeStats, ToDiskInfo and the master's shard broadcast read fields through them afterwards. The heartbeat count refresh wrote FileCount/DeleteCount into those records under the lock, which the readers never take, so it was a data race and a reader could pair a FileCount from one heartbeat with a DeleteCount from the next. Disk.AddOrUpdateEcShard had the same write on the incremental path, and its shard-bits/ratio merge was already in place. Stored records are now never written after they are published: a count refresh, an incremental merge and a shard delete each build a copy beside the stored record and swap the map entry under ecShardsLock, so a reader holding the old pointer keeps one heartbeat's snapshot. The refresh compares against whatever record is current under the lock, since the incremental path may have replaced it since UpdateEcShards looked. The 0/0-means-not-reported rule is unchanged. The count tests now assert that the stored pointer changes while a previously obtained one still reads its old values, and a -race test runs full and incremental count refreshes against readers; it failed on the previous code and passes now. |
||
|
Chris Lu
|
30f0364256 |
admin: fail an in-flight detection when its worker disconnects (#12376)
* admin: keep a reconnected plugin worker registered when its old stream unwinds cleanupSession removed whatever session was registered under the worker ID, so a worker that reconnected while its previous stream was still unwinding lost the new session and its registry entry to the old stream's deferred cleanup. Tie cleanup to the stream that is ending and publish the new stream before the registry upsert so a lingering predecessor sees itself superseded. * admin: fail an in-flight detection when its worker disconnects RunDetectionWithReport waited only on the caller's context and the DetectionComplete channel, so a detection whose worker stream ended mid-request could only finish by timeout. On a customer cluster a planned restart of ec-vacuum-1 during an ec_vacuum detection left the default lane in 'detecting' with every execution slot idle until the 3600s detection timeout, blocking every job type in that lane. Bind the request to the stream it was sent on and fail the detection as soon as that stream closes, whether by disconnect or by a reconnect that replaced the session. A liveness tick also fails it once the registry no longer considers the worker available (heartbeat stale), sending a cancel in that case because the stream may still be open. The detector lease is dropped so the next pick starts fresh, and a 'disconnected' detector activity is recorded. WorkerDisconnectForTest now goes through the real session cleanup. * admin: retry a scheduled detection once when its detector drops When a detection fails because its worker disconnected, pick a detector again and retry once within the same iteration while the job type's runtime budget allows, instead of giving the whole cycle up. Any other failure, or a failed retry, ends the run as before and the lane moves on; the next cycle re-detects anyway. * admin: register a plugin worker's hello and stream as one step WorkerStream published the stream under sessionsMu but ran the registry upsert after releasing it, so two concurrent streams for one worker ID could publish their hellos in the opposite order from their sessions: the registry then described the older hello's capabilities while only the newer stream was sendable, and detector picks routed on them. registerSession upserts the hello and installs the stream under one sessionsMu critical section, and a hello or heartbeat arriving on a stream that is no longer the registered one is dropped instead of rewriting the live stream's entry. * admin: steer a detection retry off the stream that dropped The retry picked while the lost worker could still be in the registry: cleanupSession closed done before taking the registry entry down, and a stream closed by a reconnect keeps its entry on purpose. The lost worker ranks first on free slots, so the single retry went back to it and failed on send while a healthy worker sat idle. The disconnect error now carries the stream it came from and the retry picks around it; pickDetector only returns workers whose current stream is open, excluding registry entries whose stream is closed; and cleanupSession removes the registry entry before closing done, so a waiter woken by the closure never sees the dead worker as a candidate. |