mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-15 08:18:42 +01:00
c819e3fbea269f8f33b8d219baa71f91052ced4e
16222
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
c819e3fbea |
s3 lifecycle: per-pass run results and last full-pass status (#12643)
* s3 lifecycle: collect per-pass bucket results and work counters WalkBuckets kept only the first bucket error and logged the rest. A RunStats shared by the walker, the walker dispatcher and the replay drain now records every bucket's walk result (walked, failed, skipped on cancellation), entry and object visits, replay events, and LifecycleDelete outcomes under the dispatch_total outcome labels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: persist the last fully successful pass Shard cursors and walk timestamps advance on partial passes as well, so neither says when lifecycle last covered every policy. Keep that in its own record next to the cursors, plus a worker gauge for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: report each pass's result in the job record The job ended with a fixed success summary, or with only the first error, and malformed lifecycle policies were V(1) logs. The completion now carries the pass status (complete, partial, failed, canceled), bucket totals and failures, invalid policies, entry, object and event counts, LifecycleDelete outcomes, the last phase and bucket, the elapsed time and the last full pass, on failed passes as well. Only a complete pass advances the full-pass record; a malformed policy makes the pass partial. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep a failed job's result in its record Completions with Success=false dropped Result.Summary and Result.OutputValues, so a handler could not report what a failed run had done, which buckets failed, or how far it got. Keep them; the error message still drives the job's message and run record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: chart the time since the last full lifecycle pass Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a replay halted on an unresolved delete is a partial pass BLOCKED, RETRY_LATER and exhausted transport retries stop a shard's replay without an error, leaving its due events for a later pass. Count those shards and report the pass as partial, so it no longer advances the full-pass record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a pass with throttled walks is not a full pass WalkerInterval can skip the walk for walk-only rules (ExpirationDate, ExpiredDeleteMarker, NewerNoncurrent) with no error. Count those shards and keep the full-pass record for passes that evaluated every rule; the job reports full_pass and walks_deferred. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: keep the saved full-pass time when the save fails Publishing a timestamp that never reached the filer let the gauge move back on the next job's reload. Keep the previous value and report the save error in the job result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: drop a failed attempt's summary when the retry reports none Retries reuse the job record, and failed completions now keep their result summary, so a retry that completes without a Result showed the failed attempt's summary, and a successful one used it as its message. Clear the summary when a completion carries no Result, as the output values already are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
6ad4b07598 |
mount: scope and order the final-release writeback fence (#12651)
* mount: fence a dropped handle only when it holds unpublished content dirtyMetadata is also set by chmod, xattr and fallocate, so a failed metadata-only close refused every later write of the inode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: fence the inode before its last handle leaves the handle map A reopen between the map removal and the fence could publish at the stale kernel EOF. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: fence a failed metadata flush only when the handle holds content A refused synchronous flush or an abandoned writeback metadata flush of a handle with only chmod, xattr or fallocate changes fenced the inode, so later writes were refused until FORGET. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: count a size change on an open handle as unpublished content A truncate or extend through an open handle changes the entry's size and chunks, but it set only dirtyMetadata. A failed close then skipped the final-release fence, so a reopen before FORGET could publish against the untruncated entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: mark copied content dirty before each page of a streaming copy A copy that failed or was cancelled after earlier pages were accepted left dirtyContent false, so the final-release fence skipped that content. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: mark copied content dirty only after a page is accepted A copy whose first page is rejected leaves the destination unchanged and the kernel EOF where it was, so fencing the inode on release only blocked later writes until FORGET. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: skip the server-side copy while the destination holds unpublished content A whole-file server copy clears dirtyContent when it commits, so a page a cancelled streaming copy left behind would lose its release fence. Fall back to the streaming copy instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: skip the server-side copy while the source holds unpublished content The filer copies the stored source, so pages still pending on the source handle would be missing from the destination. The streaming copy reads them through the dirty pages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: check the publication fence before a server-side whole-file copy The filer copy publishes the destination without going through the flush-time guard, so a fenced writeback inode could still take new content before FORGET. Fall back to the streaming copy, whose flush is guarded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Ping Qiu
|
65ecb6aa12 |
fix(mount): prevent negative activity-score shifts (#12631)
fix(mount): bound activity decay shifts across time regressions |
||
|
Chris LuandClaude Opus 5.5
|
0a38738076 |
filer: keep the listing snapshot below later and unfinished event stamps (#12642)
filer: take the listing snapshot as a claim below later and unfinished stamps A ListEntries snapshot tells the client that every event at or below it is already in the listing, so the mount drops those events from a directory build and fences them with the directory floor. The snapshot was a raw time.Now(), which an event stamp can tie on a coarse clock: on Windows a mkdir right after a listing of its parent was stamped with the snapshot's own timestamp, the mount took it as listed, and the new directory stayed missing from a parent marked cached. A create stamped before the snapshot but not yet in the store was dropped the same way. Take the snapshot through the same in-flight claim the idle heartbeat uses: it stays below every unfinished stamp, and every later stamp lands above it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
10f8b2d5b1 |
mount: guard peer-truncate gaps, stale append EOF, and terminal writeback refusal (#12630)
* mount: guard append gaps over concurrent content merges * mount: fence a writeback inode when its async publication is abandoned A drain that gives up on a non-precondition error left the token clear, so a reopened writeback handle could publish past the stored EOF. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: fence a writeback inode when its async data flush fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: fence the inode when its last handle drops uncommitted content A synchronous release whose final flush fails destroys the handle's dirty pages, while under writeback the kernel EOF already counts them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b50d4a9cab |
mount: don't spend the close()/fsync commit deadline on uploads (#12641)
* mount: start the close()/fsync commit deadline after the data upload Flush, Fsync and the truncate path of SetAttr create the 30 s metadataFlushTimeout context on entry and pass it to doFlush, so the clock also runs through FlushData, the upload of the dirty pages. That upload has its own timeouts and retries. When an assign through a stalled filer or a slow volume server takes most of the 30 s, CreateEntry then fails at once with "context deadline exceeded" and close() returns EIO for data that was already stored. The deadline is meant for the commit alone (#10102: "a healthy CreateEntry completes in well under a second"). doFlush now takes the commit timeout and starts the deadline once the data is uploaded. close() still cannot hang on a stuck filer: the commit keeps its 30 s bound. Release and the flush before a rename pass 0, as before (no overall bound, only the per-call filer timeouts). Seen in production on a training cluster: a 30 s stall of one replica's disk, and a filer that stalled for about 30 s, each failed close() with EIO although the upload had succeeded. Tests: close(), fsync() and a truncate below dirty data, each after an assign slower than the deadline (all three fail before this change), and a filer that never answers CreateEntry still fails close() at the deadline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: charge the flush commit deadline only for its filer calls Starting the 30 s deadline after the data upload is not enough. flushMetadataToFiler does more than call the filer before it commits: it waits for the handle's locks, reads the file back and uploads it again when its chunks are bloated (maybeMergeChunks), and uploads a manifest once enough chunks have built up (MaybeManifestize). None of that takes the context. When it outlasts the deadline, CreateEntry again fails at once with "context deadline exceeded", and close() or fsync() returns an error for data that was stored. The deadline becomes a budget, commitBudget, charged only while a filer call of the commit runs: CreateEntry, and the read of a changed hard link before it. The attempts of one flush share it, and a flush is not retried once it is spent, so a stuck filer still fails close() after 30 s. Release, the flush before a rename and the async flush pass no budget, as before. Seen in production: the merge of a 1.8 MiB sqlite file with 183 chunks took 32 s because one chunk's read first tried a volume server that was down, and the commit failed 32 microseconds after the merge finished. Tests: a flush whose manifest upload is slower than the deadline commits (it fails without this change), and the filer calls of one flush share the budget. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * mount: give the stalled-lookup refusal test the new doFlush argument TestContentFlushRefusalWithStalledLookup came in from master after this branch changed doFlush's signature, so the merged package did not build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandClaude Opus 5.5
|
e0890391ac |
mount: keep a refused content create refused when the stored-entry read stalls (#12629)
The read of the stored entry ran in the create's filer callback, so a read that hit the call deadline turned the refusal into a filer timeout, and both retry layers resent the refused create. Read it in its own call, bounded to one filer call. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
6eb2fb4842 |
filer: stop full-chunk reads at the chunk size after a mount truncate (#12632)
* util/http: stop a full-chunk read at the chunk's recorded size weed mount shrinks a chunk's Size in place on ftruncate but leaves its needle untouched. A view covering the whole shortened chunk is fetched without a Range header, so the reader returned every stored byte. The filer GET then wrote past the declared Content-Length and sent an empty body, and the stream reader served the stale tail. ReadUrlAsStream and the cipher path now deliver at most size bytes; size 0 still reads the whole needle for manifest and log chunks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: limit a plain chunk read to the requested size A full-chunk read sends no Range header, so a chunk shortened by a mount truncate streamed its whole needle into the S3 response. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * util/http: still report a failure past the chunk size The volume server aborts a full-needle response when the needle fails its checksum, and only at the end. Stopping at the chunk size missed that, so a damaged copy read as healthy. Read the rest of the body and return its error so the caller tries another copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
63c99183d1 |
s3vectors: name weed-vector processes servers (#12627)
* s3vectors: call the registered weed-vector processes servers The gateway's discovery named them workers, the name of the admin plugin workers. Also drops two comments left without a function. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: build the Enterprise engine as weed-vector In an Enterprise checkout, make ./weed-vector builds the distributed engine, which also serves VectorService, instead of a separate weed-vector-worker. An open-source checkout builds weed-vector-server as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: distributed mode starts two more weed-vector engines It started weed-vector-worker, which no target builds now, and waited for a gateway log line that no longer exists. The Enterprise weed-vector's engines all join the placement controller. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a8436a621a |
admin: vector servers on the dashboard (#12606)
* admin: vector servers on the dashboard A Vector servers count in the Cluster card and a Vector Servers table after S3 Servers, from the engines the dashboard already discovers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * admin: drop the dashboard's vector server kind Every vector server is a weed-vector engine, so the column said the same thing on every row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
58d25998d0 |
admin: Cluster → Vector Servers page (#12603)
Lists the weed-vector engines registered with the master, with version and health, next to the other server lists. Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
github-actions[bot]
|
d91d58ce47 | docs: regenerate star history chart | ||
|
Chris LuandClaude Opus 5.5
|
4fc41872e3 |
mount: keep an open file publishing when a peer unlinks one of its hard link names (#12613)
* mount: record a hard link's looked-up names on its inode Lookup mapped a second name of a hard-linked file to the existing inode without adding it to the inode's names, so GetAllPaths missed it and the final forget left it mapped to a released inode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep an open file publishing when a peer unlinks one of its names A delete event from another client marked the inode's open handle deleted even when other names still held the hard-linked file, so every later fsync reported success without saving metadata. The delete event's entry carries the link count the filer decided on, so an alias delete now drops only that name, keeps the handle on a name this mount knows, and has the next flush read the survivor's stored entry before publishing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test peer unlinks of a hard link name held open Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: leave an unlinked file out of the periodic metadata flush flushFileMetadata did not check the handle's delete mark. A file unlinked while open before its deferred create reached the filer was created again by the next periodic flush, which expects no entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: read a pending link change before a periodic metadata flush Only flushMetadataToFiler applied linkRefresh. After a peer unlinked one name, the periodic flush kept publishing over the old link count, so the filer refused every one until a regular flush ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test the periodic flush after a peer unlinks a hard link name Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
08f9dcb6de |
admin: vector buckets link maintenance jobs to the Lance lane (#12633)
The Vector Engines and Maintenance card linked to the general plugin monitoring page; the vector index jobs run in the Lance lane. Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
8a756af348 |
vector: key shared leases by manifest commit time too (#12626)
On Windows Lance commits local tables by rename and leaves the manifest ETag empty, so a table dropped and written again at its location reused the old table's lease for the same version number. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
68b6a2ed08 |
mount: model the refused async flush as a replaced file in recovery tests (#12628)
A peer's chunk append now merges into the flush, so these tests no longer saw the refusal they exercise. A replaced file is still refused. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
anantandanant
|
fdd3a692d3 |
shell: show -collection, not -c, in ec.balance and ec.rebuild help (#12617)
Both usage lines showed [-c EACH_COLLECTION|<collection_name>], but the flag is -collection; there is no -c. ec.rebuild's own Options section already lists -collection. Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com> Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com> |
||
|
Chris LuandClaude Opus 5.5
|
45cd6033a0 |
s3: make conditional DeleteObject atomic on versioned buckets (#12624)
* s3: re-check a versioned DeleteObject If-Match under the object lock A versioned or suspended DeleteObject without a versionId routed to the owner filer after checking If-Match outside any lock, so concurrent conditional deletes could all pass the check and each add a delete marker. Keep conditional deletes on the lock path, which re-checks the precondition under the object write lock, as conditional versioned PUTs already do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: guard a conditional version delete on the owner filer DeleteObject with a versionId checked If-Match before routing the delete to the owner filer, so concurrent conditional deletes of one version could all report success. Carry the If-Match clause into the owner's transaction as a guard on the version (the null entry for versionId=null), evaluated under the object's path lock before anything applies, and map that guard failing to 412 instead of the object-lock 403. An If-Match the filer cannot evaluate stays on the lock path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: carry an If-Match ETag list into the routed delete A DeleteObject whose If-Match listed several ETags fell back to the gateway's object write lock, while a single-ETag delete of the same key ran under the owner filer's path lock. The two locks do not exclude each other, so a list delete and a single-ETag delete could both report success. Send every candidate to the filer, whose strong comparison matches checkDeleteIfMatch, so all conditional deletes with an owner are decided under the owner's lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: fail a conditional null delete on an owner transport error The owner may have committed the delete before the error, and the gateway lock path it fell back to does not exclude routed deletes, so a second If-Match delete could pass there too. Answer 500 as the specific-version path does; unconditional deletes still fall back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: fail a conditional null delete on any owner error but the nested refusal Only the owner's non-empty-folder refusal is known to apply nothing. After any other owner error the delete may have been partly applied, and a lock-path re-check does not exclude a concurrent routed delete of the same version, so a conditional null-version delete answers 500 instead of falling back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: strip a nested conditional null delete on the owner A conditional versionId=null delete that the owner refuses because other keys are nested under the entry used to fall back to the gateway lock and re-check If-Match there. That lock does not exclude routed deletes: once the last nested key is removed, a concurrent routed delete can pass the same check on the owner, and the fallback then treats the missing entry as deleted, so both requests answer 204. Demote the entry on the owner instead, in a transaction guarded by the If-Match clauses and by the entry it was read as, with the object-lock guards on the request. A missing or already demoted entry answers 412. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: add an IF_OBJECT_EXISTS write condition IF_EXISTS also holds for a plain directory, which is what is left of a key once the object on it is stripped while other keys stay nested under it. IF_OBJECT_EXISTS holds only for an entry that carries an object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: require an object for a conditional null delete with If-Match: * Stripping the null version off an entry other keys are nested under leaves a plain directory. Once the last nested key is gone, a routed delete with If-Match: * still found an entry there and removed the empty directory, so a second conditional delete could also answer 204. The owner now checks that the entry carries an object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: settle a wildcard null delete on the entry when the filer predates IF_OBJECT_EXISTS A filer older than IF_OBJECT_EXISTS fails it closed, so with the gateway upgraded first every If-Match: * delete of a null version answered 412. When the narrowed guard fails, the gateway now reads the entry on the owner: absent or a plain directory is a 412, otherwise the file is deleted, or the directory stripped, under IF_EXISTS and IF_ENTRY_EQUAL on the entry it read. The demote keeps the caller's If-Match clauses, since the entry guard already requires an object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d695d101b5 |
Fix/s3 abandoned chunk download (#12616)
* filer: cancel a chunk download that no reader waits for * filer: stop waiting for reader cache budget once cancelled * filer: keep a cancelled download's re-lookups off its context The refresh and retry lookups took the download's context. FilerClient coalesces concurrent lookups of a volume, so a cancelled download leading one failed the other readers waiting on it. Fetches still stop on cancel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test that cancelling a download leaves its re-lookup running Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: cancel unheld downloads when the reader cache is destroyed ReaderCache.destroy waited for every downloader, so closing a reader waited for a stalled prefetch nobody read. Downloads with no reader or stream on them are now cancelled first; ones still in use keep running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test that destroying the reader cache cancels an unread prefetch Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: cancel a destroyed download once its last holder leaves destroy skipped downloads a reader or stream still held, and once they were out of the map nothing cancelled them, so destroy waited for a fetch nobody used. removeConsumedLocked now cancels any consumable download, whether or not it is still in the map. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test a destroyed download cancelled after its last stream leaves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: release the stream only after destroy dropped the download Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test a cancelled stream leaving a waiting reader's download running Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
813ac74d79 |
s3tables: keep legacy-subject owners and policy principals for federated STS sessions (#12615)
* s3tables: re-anchor legacy owners for federated STS sessions A web-identity role session was keyed on its bare OIDC subject until owners became issuer-scoped. Its sessions now carry the scoped aws:userid but no asserted issuers, so the legacy subject check never applied and the session lost the table buckets and namespaces it had created. Read the configured issuer set for such sessions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: keep token claims from asserting trusted issuers on STS sessions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: give the catalog's table handler the local accounts and issuer set Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: test a federated session on a bucket its legacy subject owns Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sts: drop a token-supplied aws:userid when a session has no parent user Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: take a session's aws:userid from its signed parent user Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test that session claims keep only the server-asserted aws:userid Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: match legacy subject patterns in table bucket policy principals A policy written before issuer scoping can name role sessions by a pattern over their bare OIDC subjects. Match such a pattern against the caller's subject claim once that subject resolves to the session's scoped principal under the single configured issuer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: test a federated session against a legacy subject pattern principal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Minh Vu
|
4bc7598a43 |
admin: escape dynamic text in S3 bucket modals (#12625)
admin: escape dynamic text in bucket modals |
||
|
Chris LuandClaude Opus 5.5
|
d685709832 |
s3api: refuse writes to a stored version's physical path (#12622)
* s3api: refuse a PUT to a stored version's physical path In a Suspended bucket a PUT to <key>.versions/v_<id> replaced the stored version's body in place. Look the entry up when the key has the version file shape and deny the write when it is a version file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: refuse copy, multipart, POST and rename writes to a version file Apply the same version file check to the CopyObject, CreateMultipartUpload, CompleteMultipartUpload, POST Object and RenameObject destinations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: read a version file write target on its logical owner The guard resolved the key as an object, which on a ring reads the physical path's owner and, in a versioned bucket, may return a nested version under <key>.versions/v_<id>.versions instead of the stored version. Look up the physical entry directly, routed by the logical key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: confirm a version file miss on a skipped owner When the owner was recently unreachable the routed lookup answers from a replica, whose miss does not show the version is absent. Ask the owner directly in that case and refuse the write when it cannot answer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: confirm every routed version file miss on the owner Whether the routed lookup skipped the owner can change between the lookup and a second unreachable check, letting a replica's miss through. Ask the owner after any miss instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d1a97f4c3d |
fix(mount): retain refused async metadata before releasing handle (#12598)
* fix(mount): retain refused async metadata before releasing handle * mount: sync the failed-metadata directories with util.FsyncDir Directory fsync fails on Windows, where the mount also builds, and returns EINVAL on some filesystems. Either kept the retention loop retrying a record that could never be saved, so the drain never finished. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep failed-metadata records in a per-user directory The write cache root defaults to the shared temp directory. The first mount to save a record created failed-metadata there as 0700, so another user's mount could never save one and its drain retried forever. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: drop the async conflict probe test TestContentAsyncConflictKeepsRecoveryRecord drives the same refused drain with the retry backoff stubbed out; the probe slept through the real backoff (7 s) and its kept-handle branch could not run. Its peer-entry check moves into the recovery record test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5a54009439 |
filer/etcd: client keepalive and per-request deadlines (#12621)
* filer/etcd: client keepalive and per-request deadlines An etcd member that goes silent with its connections open (a dead or hung node, a network partition, a frozen process) keeps the store's requests balanced onto it waiting forever, because the client has no keepalive and the calls have no deadline. Writes through the filer stop although the other members keep quorum. keepalive_timeout (default 3s, "0s" turns keepalive off) enables gRPC keepalive with the ping interval fixed at gRPC's 10s minimum. gRPC also sets TCP_USER_TIMEOUT to it on Linux, so a connection whose sent data goes unacknowledged that long is closed by the kernel. request_timeout (default 3s) bounds every Get, Put and Delete unless the caller set a deadline; timeout keeps its meaning as the dial timeout. DeleteFolderChildren gets no request deadline: its DeleteRange grows with the directory, and keepalive still ends it if the member goes silent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Update weed/filer/etcd/etcd_store.go Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> * Update weed/filer/etcd/etcd_store_test.go Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> * Update weed/command/scaffold/filer.toml Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> * filer/etcd: no request deadline on entry writes A Put whose client deadline expires may still have been committed. The filer's upload handler treats the error as a failed write and deletes the uploaded chunks, which the saved entry may then point at. InsertEntry and UpdateEntry therefore keep the caller's context: etcd's own request timeout ends a Put a live member cannot commit, and keepalive ends one on a member that goes silent. Reads and deletes keep request_timeout; a timed-out store delete returns before the filer deletes any chunks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/etcd: no request deadline on KvPut The filer wrapper stores a hard-linked file's attributes and chunk list through KvPut. An HTTP append to a hard-linked file whose KvPut times out after etcd saved it returns an error, and the upload handler deletes the appended chunks the saved hard link now points at. KvPut keeps the caller's context like InsertEntry and UpdateEntry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Yip LeoandClaude Opus 5.5
|
911bfe4014 |
mount: stop overwriting the mount root's filer entry (#12623)
When a mount is rooted at an existing directory, two code paths replace that directory's filer entry with one the mount builds itself: - every mount start re-creates the root with a non-exclusive mkdir, which the filer handles as an update of the whole entry; - creating or removing an entry directly in the root saves the root's entry to touch its mtime, and maybeLoadEntry synthesizes that entry from the mount options instead of reading the filer. The filer keeps only crtime and inode on such an update, so the root loses its mode, owner and extended attributes. When the root is a bucket it also loses the bucket quota, which silently drops to 0 (unlimited), and the settings S3 keeps on the bucket entry. The explicit allowEmptyFolders=false that #11370 meant to keep is replaced by true at every mount start, because the mkdir has wiped it before ensureBucketAllowEmptyFolders looks. Create the root only when a lookup says it is missing, and then exclusively. Leave the root out of the filer-side mtime touch: its attributes come from the mount options (setRootAttr), so the filer copy of its mtime was never visible through the mount anyway. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
737ef68a11 |
mount: keep publishing an open file through its surviving hard link (#12609)
* mount: keep flushing an open file whose other hard link survives an unlink Unlink marks the inode's open handle deleted so no flush recreates the removed name. When the entry was one of several hard links the inode lives on under another name, yet the mark stayed and every later fsync and close returned OK without publishing metadata. Lift the unlink's own mark once the name is gone and the inode still has a path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: carry a link's stored entry into the file's open handle Link rewrites the source entry with the new hard link id, count and times, but an open handle kept its old base and entry. Its next conditional content flush was refused (EIO), and an unconditional one would have dropped the hard link id. Move the handle onto the stored entry when only the link fields and times changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: carry an alias unlink's stored entry into the open handle Removing one of several hard links lowers the stored link count and stamps a new ctime, and the handle may still hold the removed name. Look up the entry at the name the handle flushes through and adopt it, so the next content flush matches the stored entry, writes the right count, and attribute saves address the surviving name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test an open handle across link and alias unlink Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: read an alias unlink's lost entry again at the next flush When the read of the surviving entry right after an alias unlink fails, the handle kept a base with the old link count and ctime, and every later flush was refused even after the filer recovered. Leave the hard link id on the handle and have the next flush read the stored entry and adopt it before publishing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: publish a queued close before unlinking a name another link keeps With writeback caching, close only queues the flush. Unlink marked the handle deleted before waiting for that flush, so the flush skipped its metadata and the surviving link kept the content from before the close. When another link keeps the file, wait for the queued flush first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: let a queued close reach the filer before linking the file With writeback caching a new file is created on the filer by its queued close, and a listed directory serves the source from the cache, so a link right after close updated a name the filer did not have yet and failed with EIO. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: skip a link change no newer than the handle's entry A link change carried into the handle could replace a base already moved past it by a later acknowledgment. Use the same clock-domain check as installAckedEntry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep a link change's ctime in the dirty entry The flush does not restamp times, so a dirty entry carrying an earlier write's ctime moved the filer's ctime back from the link or unlink. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: flush through the file's current name when the resolved one is gone doFlush resolves the name before uploading the data. An unlink of that alias, or a rename, in the meantime sent the publish to a name the file no longer has. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep the pending link read when its name is already gone A flush could read the removed alias name, find nothing, and drop the pending read of the surviving link, leaving the stale base in place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test link and alias unlink across queued closes, flush races, versions and ctime Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: pick a flush's name again once its locks are held An unlink or rename that lands between the name pick and the locks would otherwise send the flush to the removed name, where the create guard fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test a flush waiting on its lock through an alias unlink Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: check the flush target after every lock acquisition A flush whose name kept changing published to the name chosen before its last lock wait. Check after each acquisition and return a retryable error once the attempts run out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test a flush whose name changes during every lock wait Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bac2196d65 |
shell: return flag parse errors from every command (#12618)
* shell: return flag parse errors from every command 48 commands returned nil when their flags failed to parse, so a mistyped flag printed the flag error but the command still counted as a success: a piped weed shell run exited 0, the master maintenance loop logged no error, and admin script jobs reported the line as done. Return the parse error as s3.bucket.create (#12544) and the other shell commands already do. TestCommandsReturnFlagParseErrors runs every registered command with an unknown flag and fails for the 48 on master. Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com> * shell: don't report -h as a failure With the parse errors returned, -h on those commands now returns flag.ErrHelp, so the shell printed "error: flag: help requested" and a piped run exited 1. The flag package has already printed the usage, so processEachCmd now treats flag.ErrHelp as success for every command. Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com> * shell: keep the -h test beside the flag parse test TestProcessEachCmdTreatsHelpAsSuccess covers the same change as TestCommandsReturnFlagParseErrors, so it lives in the same new file and shell_liner_test.go stays as it was. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com> Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
c248ab7865 |
credential: keep IAM changes made during the legacy-file migration (#12614)
* credential: migrate the legacy IAM file without overwriting newer entries The migration wrote every identity, service account and group from the snapshot the load read, over whatever the per-entry files held by then, so a revoke or delete landing in between was undone for good. It now migrates only the legacy file's own entries, creates each per-entry file only when none exists, and waits for the next load when the legacy file changed after the snapshot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * credential: test changes landing between the IAM load and its migration Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * credential: keep a per-entry file deleted after the IAM load deleted An entry the load read from its own file is already migrated. When that file is removed before the migration runs, the legacy copy must not recreate it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * credential: test a deleted per-entry file the legacy file also holds Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
12cb707e58 |
seaweed-volume: copy master.proto with metrics_port (#12605)
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
aa901b7ac7 |
s3tables, iceberg: create missing directories exclusively so a concurrent create keeps its record (#12600)
* s3tables: create a missing directory exclusively in ensureDirectory ensureDirectory looked the path up and, when it was missing, created it with a plain CreateEntry. A table create that lost the lookup race then replaced the directory another create had made, claimed and confirmed since, erasing that table's record, so a third create could claim the name and win as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: create missing metadata parent directories exclusively saveMetadataBlob created each missing directory on the way to the metadata directory with a plain CreateEntry after a lookup. A table directory another create had made and written its record onto since the lookup was replaced by a bare directory, erasing that record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: create missing stage-create marker directories exclusively The marker path starts at the table bucket directory, which carries the bucket record; a plain create after a missed lookup replaces a directory another writer made in between, with whatever it carries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
e0beb59ef0 |
topology: treat a nil ttl as the empty ttl when looking for duplicate volume ids (#12601)
Volume layouts are keyed by ttl.String(), so a nil ttl and the empty ttl share one layout. ttlEqual told them apart, so a replica reported with a nil ttl into a layout created with the empty ttl matched its own layout as a duplicate, and every copy was quarantined. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
fe14880f4d |
shell: tier download replicates the downloaded volume before deleting the remote object (#12599)
* shell: replicate the downloaded volume before deleting its remote object volume.tier.download downloaded every replica from the remote tier and let the last one in the master's location list delete the shared object, so a replica missing from that list kept reading a deleted object. Download onto one replica, copy it over every other remote-backed replica, re-read the topology so replicas that show up meanwhile are replaced too, and only then delete the last remote-backed replica together with the object before copying the local volume onto it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: test tier download replicates before deleting the remote object Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: copy the downloaded volume onto each replica's own disk type Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: test tier download keeps each replica's disk type Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: name the hdd disk type when copying a downloaded volume VolumeCopy treats an empty disk type as the source's, so an HDD replica copied from an SSD source was placed on SSD or failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: block deletes on the tier download copy source A readonly-can-delete source took deletes after VolumeCopy read its index, so the replaced replica lost them and served the files again. Hard-mark the source readonly while replicas are replaced, then restore the writable mark or reload the persisted state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep the remote object while replicas are unlisted Tier download deleted the shared object once the listed replicas were local, even when the replica placement expected more copies than the topology listed. Keep it in that case so a rerun can finish later. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: fail tier download when no replica is listed Also point the final-replica delete error at volume.fix.replication. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep the remote object when a server lists the volume twice A stale twin on another disk of a replaced server still references the object but is filtered out with its server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: restore the tier download source when its readonly mark fails VolumeMarkReadonly blocks local writes and deletes before its second master notification, so a failed notification returns an error with the source already frozen. Restore it whenever the status was read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: report a failed tier download source restore A restore that leaves the source unmounted or frozen was only printed, so the command still returned success. Join the restore error into the result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: wrap the tier download restore error Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
94c3869057 |
volume_move: keep the live-move source frozen across volume reloads (#12597)
* volume_move: persist the move's source freeze across reloads A live move froze the source in memory only, so an unmount/mount or a volume server restart mid-move reopened it for writes that the move then deletes with the source. Persist the freeze when the move froze a writable source, and reopen the target right after the copy since the copied .vif carries the freeze. An already-readonly source keeps an in-memory mark, and a failed move still restores the source writable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: live move keeps the source frozen across a reload, on Go and Rust volume servers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: require 409 from the reloaded frozen source Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume_move: reopen an interrupted move's source only without a target copy Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
652648d9c4 |
mount: merge concurrent content changes from another mount instead of failing or dropping them (#12596)
* mount: merge a refused content flush onto the stored entry A content flush conditioned on the handle's base failed with EIO whenever a peer had changed the file since, so two mounts appending to one file could not both close it cleanly. On a refused condition, reread the entry and replay this handle's changes since its base onto it: the chunks it added or dropped, and the attributes, extended keys and inline content it changed. A local truncation keeps its size and clips the peer's chunks to it; otherwise the larger size wins. The merged entry is published against the stored one, up to three times, and installed in the handle so its next flush keeps the peer's chunks. A replaced or deleted file is still refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: merge a peer's content change into a dirty handle A subscription event carrying a peer's content change replaced a dirty handle's entry and discarded its unflushed pages, so the handle's writes were lost and its close still returned OK. Fold the peer's entry into the handle with the same merge a refused flush uses, keep the dirty pages, and take the peer's entry as the new base. A replaced file still invalidates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: stop a flush merge once a filer accepts the write withFilerClient can call the callback on several filers. A refusal from one left its stored entry behind, so a later filer's success went on to another merge round. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: refuse a merge that puts inline content next to chunks A read inside the inline bytes returns them without looking at chunks, so one side's inline content merged with the other side's chunks hid data after a successful flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: refuse to merge a whole-file rewrite over a peer's chunk change maybeMergeChunks re-uploads the handle's whole view as unstamped chunks. Merged after a peer's change, they tie with a peer's own rewrite or bring back a range the peer truncated, so that flush fails as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: clip peer chunks at the lowest truncate, not the final size A handle that truncated and then wrote past the old end ends up larger than its base, so the merge did not see the truncate and kept the peer's chunks inside the range the truncate cleared. SetAttr now records the lowest size truncated to over the current base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep a peer's truncate when the handle left the size alone The merge took the larger of the handle's and the stored size, so a peer truncate came back as a zero-filled tail when the handle only overwrote bytes inside it. The stored size now stands unless the handle changed its size or wrote past it, and rebasing keeps only the growth writes made after the flush was taken. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: retry a read view whose merged manifests did not resolve A failed SetChunks leaves the chunk group answering every read with the resolution error, and nothing re-ran it while the entry stayed the same. The handle now marks such a view (on rebase and on SetEntry) and the next read or lseek resolves the entry's chunks again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep a dirty handle's writes when a peer change cannot merge The new merge refusals sent a peer event on the same file down the old path, which discarded the dirty pages and adopted the peer's entry, so close returned OK without the writes. The handle now keeps its writes and base, and the flush meets the conflict and fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep an unpublished truncate across a peer's base change The truncate mark was tied to the base pointer, so a peer event that moved a dirty handle's base (a merged content change or a metadata-only change) dropped it, and a later merge on a regrown file kept peer chunks inside the truncated range. Those base moves now carry the mark; a flush ack or a discarded handle state still clears it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: wait for the flush before merging a shrink over buffered writes A write inside the file leaves its bytes in dirty pages without moving the size. Merging a peer's truncate then adopted the smaller size, and the chunk uploaded later sat past it, unreadable through the handle. The event path and the periodic metadata flush now refuse a merge that shrinks the handle; the close flush uploads the pages first and merges then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: tell a shortened base chunk from a whole-file rewrite A truncate shortens a base chunk in place, so it gets a new reference key, and an old chunk can carry no timestamp. The rewrite refusal now applies only to unstamped chunks whose file id is new to the base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: check the creation time when the base has no inode An entry without an inode or hard link id matched any replacement at the same path, so a stale handle could merge into a different file. Without an inode the merge now needs the same creation time. The filer keeps it across updates; only a synthesized base has neither and still merges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep a dirty handle's writes when a peer replaces the file A peer entry with another inode at the handle's path went down the fallback that destroys the dirty pages and installs the replacement, so close returned OK without the writes. The handle now keeps its writes and base like it does for an unmergeable change, and its flush refuses to update the replacement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: build the content flush tests on freebsd The invalidation tests, which every platform compiles, now use the content flush helpers, so the freebsd cross-compile vet failed with undefined: flushConditionEntry. Nothing in the file is freebsd-specific. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep a dirty handle's base when a same-content file replaces it A replacement with the base's chunks but another inode counted as a metadata-only change, so the dirty handle adopted it as its base and the next conditional flush overwrote the new file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: bring a peer's metadata change into a dirty handle A metadata-only event moved a dirty handle's base but left its live entry, so the next flush read the peer's chmod as a local edit and published the old mode back. Replay the handle's own changes onto the peer's entry instead, as for a content change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: wait for the flush before merging peer inline content over buffered writes A buffered write is not a chunk yet, so a merge could install a peer's inline content that later reads return ahead of the write's chunk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: track a whole-file chunk rewrite on the handle The merge told a rewrite by its unstamped chunks, which a manifest fold hides: manifests are unstamped either way. Record the rewrite against the base it was made from, carried across peer base changes like a pending truncate, and refuse the merge from that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: refuse a merge that shortens a manifest at the truncate Clipping a manifest reference leaves its members whole, so peer bytes past the handle's truncate could reappear once the file grew again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: flush a dirty handle after a peer replaces its file The replacement test checked only the retained handle state. It now flushes against a filer holding the replacement, including one with the base's content, and checks the flush fails and leaves it unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test an append fsync after a peer's attribute-only change A second mount's acknowledged chmod, chown, utimes or setxattr leaves the content alone, so the writer's conditional content flush must still publish its append and keep that change. On master the refused flush surfaces as EIO from Fsync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a576398662 |
weed-vector: advertise its metrics port when registering (#12604)
* master.proto: KeepConnectedRequest carries the node's metrics port Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: advertise --metricsPort when registering with the master Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * master.proto: number metrics_port 9 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
baaa60b24a |
admin: an arrow icon for vector buckets (#12602)
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
917cf534db |
s3api: sign every operation subresource in signature V2 (#12575)
* s3api: sign every operation subresource in signature V2 The V2 canonical resource only kept the subresources inherited from the original allowlist, so tagging, retention, legal-hold, cors, encryption, object-lock and the other keys the router dispatches on were left out of the signature. A V2 signature issued for one request therefore also verified that request with one of those keys appended, running a different operation under the signer's identity. Add every operation-selecting subresource to the signed list. Clients that already sign tagging, cors, object-lock or replication in V2 now verify as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: sign every value of a repeated signature V2 subresource canonicalizedResourceV2 kept only the last value of a repeated query key, while handlers read the first. A request signed for versionId=A could be sent as versionId=B&versionId=A and act on version B. Sign every value in order, as V2 clients do, so a second value no longer verifies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
github-actions[bot]
|
5ba77af65f | docs: regenerate star history chart | ||
|
|
c2b2ed769b |
fix(mount): preserve appended chunks during stale attribute updates (#12593)
* fix(mount): preserve acknowledged chunks during attribute updates * mount: keep the WFS field block aligned Adding the publication-lock fields inside the aligned block made gofmt re-indent 25 unrelated lines. Put them in their own group after dirIdleEvict, matching the enterprise layout, so syncs do not conflict there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: finish attribute saves through a FUSE interrupt saveEntryAttributes derived its RPC context from the syscall's cancel channel, so an interrupt while the conditional UpdateEntry was at the owner returned EIO for an update the owner then applied, and skipped the local install and retention. Keep the metadata deadline but drop the cancellation, like the other metadata mutations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: map attribute CAS ids in place instead of cloning the entry saveEntryAttributes cloned the whole entry three times per attempt only to rewrite uid/gid for the filer, and each clone is as large as the chunk list. Switch the ids in place and restore the exact local values after the call, keeping one clone. The test now checks the reply carries the local uid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: compare attribute updates against the filer's own uid/gid With -map.uid/-map.gid, a filer id that is also a mapped local id reads back unchanged and maps forward to another filer id. The IF_ENTRY_EQUAL expected entry then never matched, so chmod, utimes and xattr changes on such files failed with EAGAIN after three attempts, and an update would have rewritten the owner. Keep the filer ids read with the entry for the ids the syscall leaves alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(mount): align attribute retention with read lock order * mount: take the handle lock before entryLock in the attribute snapshot attributeEntrySnapshot held entryLock while it queued for the shared handle lock. A Read holding that lock waits on entryLock, the snapshot waits behind a queued exclusive (an invalidation or a Write), and the exclusive waits on the Read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(mount): complete handle lock order and preserve explicit ID intent * mount: re-read a refused attribute update from the file's owner filer The filer checks the IF_ENTRY_EQUAL condition on the owner of the path's write route key. The retry read any filer, so behind one that had not applied the owner's latest update, every attempt sent the same stale base and the syscall failed with EAGAIN. Read from the owner once the lock ring is known, like lookupFilerEntry, but keep the filer's raw ids. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d7ae08ecad |
fix(mount): refuse stale whole-entry content flushes (#12594)
* fix(mount): condition content flushes on the observed entry * mount: stop retrying a flush the filer refused on its condition The filer reports a failed CreateEntry condition as FilerError_PRECONDITION_FAILED in the response, which surfaces as filer_pb.ErrPreconditionFailed rather than a FailedPrecondition status. The flush retry loop treated it as transient and replayed the refused flush three more times over 7 s before returning EIO. The test filer now answers like the real one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: take the handle base from the entry the filer stored After an acknowledged flush the handle base was the requested entry, but the filer stores it with a fresh commit_ts_ns (and fills crtime, inode, atime), so the next IF_ENTRY_EQUAL flush on the same handle never matched: write, fsync, write, fsync returned EIO, as did close after a periodic metadata flush and the first flush after an eager create. Use the stored entry from the acknowledgement's metadata event. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: condition the periodic metadata flush like the close flush The periodic metadata flush of an open, dirty handle sent the same whole entry through an unconditional CreateEntry, so a file held open past the flush interval still overwrote a peer's acknowledged append, and the close flush then matched the overwritten state. Both paths now publish through one helper that carries the handle's write condition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: republish a refused flush once when the stored entry is unchanged A handle's base can differ from the stored entry without any peer write: a rename while the file is open restamps the entry under its new name, and with -map.uid a filer id that is also a mapped local id does not map back to itself. Each made every flush of the handle fail with EIO. On a miss, read the stored entry and, if it still holds the observed state apart from name, commit stamp, id encoding and derived size, retry once with that entry as the condition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: take the filer's entry once a write open holds the DLM lock The lock orders this writer after the previous holder, but the holder's flush may not have reached the local cache yet, so the handle's base named the entry from before the wait and the conditional flush refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: share the open-existing path of Create Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: open the file a previous DLM holder created during a deferred create A deferred create never asks the filer, so a create that waited for the lock went on to publish over a file the previous holder had just made, and its flush expecting no entry was refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: send the lock ring snapshot to mounts A mount running -dlm is a lock client like the S3 gateway, and needs the ring on connect to reach a file's owner filer before the next membership change is broadcast. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: follow the masters' lock ring under -dlm The lock client only knew the seed filer, so every lock took the forward hop and the mount had no way to tell which filer owns a file. Subscribe to the masters named by the filer, like the S3 gateway, and wait briefly for the first ring before serving. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: read the DLM-ordered entry from the file's owner filer The filer checks a flush's condition on the owner of the path's write route key, and another filer may not have applied the previous holder's flush yet. Look the entry up on that owner once the ring is known. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a2b9b37ee5 |
volume(rust): untrack queued writes before replying (#12595)
* volume(rust): untrack queued writes before replying
The durable write worker sent each write's result and only then dropped
the batch's pending-write trackers, so a caller could see its write
finish while the write was still tracked as pending. Go untracks in the
caller before writeNeedle2 returns. Drop the trackers before sending the
results so the Rust queue path matches.
test_replicate_delete_cancels_a_queued_write asserted the count was zero
right after the fsync write returned and failed about 1 run in 100 under
load (Rust Unit Tests on master
|
||
|
Chris LuandClaude Opus 5.5
|
a2ffdce2ec |
volume: skip VolumeDelete when its RPC was cancelled before the delete (#12586)
* volume: skip VolumeDelete when its RPC was cancelled before the delete A caller that cancels (a lost tiering lease, an expired deadline) still had its admitted delete run in the Go server, including one that waited behind the copy guard. The Rust server drops a cancelled handler, so return the context error before the delete starts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: wait until the queued delete blocks on the guard before cancelling Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
6388dfca9b |
s3api: verify signed request bodies against the signed payload hash (#12577)
* s3api: verify signed request bodies against the signed payload hash A SigV4 signature covers the declared x-amz-content-sha256, but only getRequestDataReader (PutObject, UploadPart) compared the body with it. Every other handler parsed the body as received, so a signed request, DeleteObjects for example, could be replayed with a different body. After a V4 signature verifies, wrap the body so it is handed out only once all of it has been read and matched against the hash the signature covered; a decoder that stops reading early never sees unverified bytes. getRequestDataReader unwraps it and keeps verifying uploads as they stream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: answer XAmzContentSHA256Mismatch for refused DeleteObjects bodies DeleteObjects answered a body that failed its payload-hash check with InternalError and CompleteMultipartUpload with MalformedXML. Name the check that failed instead, as PutObject does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: check Content-MD5 on signed request bodies Signature V2 signs Content-MD5 rather than a payload hash, and a V4 request sent with UNSIGNED-PAYLOAD may still sign it, but bodies read outside the upload path never compared it with the body. Check it with the payload hash once a V2 or V4 signature verifies: BadDigest on a mismatch, InvalidDigest when it is malformed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: verify presigned uploads against a signed payload hash A presigned V4 URL can sign X-Amz-Content-Sha256 as a query parameter, but getRequestDataReader compared upload bodies only with the header of that name, so the signed hash went unchecked. Verify uploads against the hash the signature covered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: check Content-MD5 on bodies buffered for their payload hash IAM and other non-S3 requests signed without x-amz-content-sha256 have their body buffered and hashed during signature verification. That buffer counted as already bound, so their Content-MD5 was never checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: check Content-MD5 on directory marker bodies A PUT of a key ending in "/" stores its body on the directory entry without going through putToFiler, so its Content-MD5 was never compared with the body. Answer BadDigest on a mismatch, as putToFiler does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: wrap and unwrap signed bodies through helpers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a3b68aa37c |
shell: confirm trim survivors still serve the volume before deleting (#12590)
* shell: confirm trim survivors still serve the volume before deleting volume.fix.replication compared the doomed replica's index with every survivor and then deleted it unconditionally. A survivor that went away while the indexes were read still let the delete through, leaving fewer copies than the check had counted. Re-read each survivor's VolumeStatus after the index checks and skip the trim if any no longer has the volume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: test a trim whose survivor goes away after its index read Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
4cba4eb7bf |
s3: ignore unsigned object headers on POST Object (#12585)
* s3: ignore unsigned object headers on POST Object The POST policy signs form fields only, but the upload read tagging, user metadata, storage class, object lock and the stored content headers from the request headers as well. Drop them before the form fields are applied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test POST Object request headers against signed form fields Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
249f04e63a |
s3api: NoSuchKey for object lock calls on missing or deleted keys (#12592)
* s3api: answer PutObjectLegalHold on a missing key with NoSuchKey In a versioned bucket a key with no latest version resolves to ErrLatestVersionNotFound, which the legal hold handler did not map, so the request failed with 500 InternalError. PutObjectRetention already maps it to NoSuchKey. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: keep failed reads distinct when resolving an object lock target getObjectEntryForUpdate turned every lookup error into a not-found sentinel, so a failed filer read made PutObjectRetention and PutObjectLegalHold answer NoSuchKey for an object that exists. Only a genuine miss becomes the sentinel now, matching getObjectEntry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: answer object lock calls on a delete marker with NoSuchKey PutObjectLegalHold and PutObjectRetention on a key whose current version is a delete marker returned 200 and wrote the lock onto the marker, and the matching GETs read it back. A key whose current version is a delete marker does not exist, and delete markers take no lock, so these calls now answer NoSuchKey like the tagging and ACL handlers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: let a lock recorded on a delete marker not block its removal Delete markers are not WORM-protected, but a legal hold or retention written onto one (possible before the previous change) made both the gateway check and the routed delete's filer-side WORM guard refuse to remove the marker, so the object could not be restored and lifecycle could not clean the expired marker up. Lock enforcement now ignores a lock on a delete marker, and the routed version deletes in DeleteObject and lifecycle drop the WORM guard when the version they read is a marker. A version id never changes kind, so the marker read is the one deleted; the null version keeps its guard because a PUT can replace it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
4ee82977af |
docker: pass entrypoint arguments to weed without splitting them (#12589)
The master, volume, volume-rust, server, mini, filer and s3 cases and the fallback forwarded the user arguments as an unquoted $@, so the shell split each one at whitespace and globbed it against /data. An -admin.password with a space reached weed as two words, and weed ignored every flag after the stray word. Quote "$@" as the other cases already do; $ARGS stays unquoted since the script builds it from its own words. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
bef1b13b3d |
replication: keep a local sink path the source recreated after a delete (#12581)
* replication: let the local sink check a delete against the live source LocalSink removed the destination path for any delete it was handed. A delete that reaches the sink after the source recreated the path (a peer filer's event delivered late, or a historical event replayed) removed the newer file, and no later event brings it back. DeleteEntryVersion receives the deleted entry. Before removing an existing destination, the local sink looks up the source path and keeps the destination when the source holds an entry of the same kind committed after the deleted version (commit_ts_ns, or mtime for unstamped entries). A live entry that is the deleted version itself or older means the source store lags the event, so the delete proceeds. A failed lookup returns the error so the event is retried instead of deleting a file that may be live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.sync, filer.replicate: hand the deleted version to the sink Pure deletes, renames without a native move and moves out of the watched directory now go through sink.DeleteEntryVersion with the old entry, so a sink that checks the live source (the local sink) can keep a path the source recreated after the deleted version. Other sinks fall back to DeleteEntry unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * replication: order the local sink check by the delete's mutation time A source filer can hold a version written concurrently with the delete: newer than the deleted entry but older than the delete itself. The source's own replay applies the delete over it, so keeping the backup copy because it outranks the deleted entry left a file the source no longer has. The sink now gets the delete's mutation time (MutationTsNs, else TsNs) and keeps the destination only when the live entry was committed after it, the order Filer.Replay keeps. That also covers a stamped replacement of an unstamped entry whose mtime did not move. filer.replicate events carry no time, so they keep the deleted entry's stamp as the bound, and mtime only when nothing is stamped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * replication: count an unstamped live entry from its mtime A source filer that predates commit stamps stores entries with no CommitTsNs, so comparing the live stamp with the delete's time never kept a replacement there. An unstamped live entry now counts from its mtime against the delete's time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * replication: keep a stamped live entry when the deleted one is unstamped and the delete has no time filer.replicate gets no delete time from the notification queue. When the deleted entry has no commit stamp either, a stamped live entry was written after it, so its mtime no longer decides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
c2625ddf00 |
seaweed-volume: read security.toml values as TOML strings (#12582)
The line parser only trimmed quotes, so `key = "abc\n"` gave the Rust volume server the bytes `abc\n` with a literal backslash while the Go servers sign with a newline, and every JWT failed. Literal strings kept their quotes and a trailing comment stayed part of the value. Decode each value with the TOML parser; text it rejects keeps the old quote trimming. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b7d6b48921 |
operation: probe octet-stream uploads for gzip like untyped ones (#12584)
* operation: probe octet-stream uploads for gzip like untyped ones The 128-byte gzip probe ran only when the caller passed no MimeType, so an upload labelled application/octet-stream or binary/octet-stream was never compressed. S3 PutObject and UploadPart pass the request Content-Type (or the octet-stream that mimeDetect fills in), so every binary S3 upload skipped compression that the same bytes get through the filer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * operation: check the probed upload keeps the caller's content type Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |