mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-25 06:20:42 +01:00
8841eb6c7044ccfc334dbcae0eee1f5601b95dad
10260
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
8841eb6c70 |
volume: check If-None-Match before If-Modified-Since and match ETag lists (#12027)
* filer: match If-Match and If-None-Match against "*" and ETag lists checkPreconditions compared the whole header as one ETag, so "*", a list of ETags and a weak W/ tag in If-None-Match never matched (RFC 9110 13.1.1, 13.1.2). * volume: check If-None-Match before If-Modified-Since and match ETag lists A volume GET answered 304 from If-Modified-Since even when If-None-Match named a different ETag, and compared If-None-Match as one exact string, so "*", lists and weak tags never matched (RFC 9110 13.1.2, 13.2.2). * filer: split etag lists only outside quoted tags A comma inside a quoted opaque tag is a legal etagc, not a list separator; strings.Split cut such tags into fragments that CanonicalizeETag could turn into a false match. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: match repeated If-Match and If-None-Match header lines Header.Get reads only the first field line, so a match on a later repeated header was missed; join all field values like any other comma-separated list. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: match repeated If-None-Match header lines Header.Get reads only the first field line; join all field values so a match on a later repeated header is seen. Also covers a comma inside a quoted opaque tag. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: send the etag on a 304 like the filer does RFC 9110 15.4.5 lists ETag among the fields a 304 must carry when a 200 would send it; SetEtag now runs before the notModified check. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: mirror the conditional-GET fixes in the rust server not_modified_response now checks If-None-Match before If-Modified-Since, matches "*", weak tags, quoted commas and repeated field lines through the same list rules, and sends the etag on a 304. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: treat an empty If-None-Match like the Go server does * filer: ignore empty elements in etag precondition lists * volume: ignore empty elements in etag lists like the filer * filer: let a present-but-empty precondition line suppress the date check * volume: gate If-Modified-Since on a present If-None-Match line * volume: gate If-Modified-Since on header presence like the Go server --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
59cccc788c |
master: read the writable-volumes cap from master.toml (#12009)
* master: read the writable-volumes cap from master.toml -maxWritableVolumes becomes the master.max_writable_volumes config key, so the new master limit is set the same way as other master options instead of a flag wired into every command that embeds a master. * master: name the writable cap master.max_writable_volumes_per_layout The cap applies per volume layout (collection + replica placement + ttl + disk type), so the key names its scope instead of reading as a global cap. * master: keep an embedder's MaxWritableVolumes when the key is unset NewMasterServer overwrote option.MaxWritableVolumes with the viper value, which reads as 0 whenever master.toml omits the key, silently dropping a cap a Go caller supplied through MasterOption. Seed the default from the option instead so the field survives an absent key; an explicit config entry still wins. * master: drop stray nested-checkout gitlinks Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
9e7fd2eb90 |
filer: match If-Match and If-None-Match against "*" and ETag lists (#12026)
* filer: match If-Match and If-None-Match against "*" and ETag lists checkPreconditions compared the whole header as one ETag, so "*", a list of ETags and a weak W/ tag in If-None-Match never matched (RFC 9110 13.1.1, 13.1.2). * filer: split etag lists only outside quoted tags A comma inside a quoted opaque tag is a legal etagc, not a list separator; strings.Split cut such tags into fragments that CanonicalizeETag could turn into a false match. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: match repeated If-Match and If-None-Match header lines Header.Get reads only the first field line, so a match on a later repeated header was missed; join all field values like any other comma-separated list. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: ignore empty elements in etag precondition lists * filer: let a present-but-empty precondition line suppress the date check --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
d6a68468ad |
volume: keep a below-minimum fsync batch open for stragglers (#12016)
* storage: hold a below-minimum fsync batch open for stragglers When the durable-write queue drained, the worker flushed whatever it held, so low-concurrency fsync writes collapsed to one sync per request -- a seek plus a rotation on rotational media no matter the payload. Once the queue empties with the batch still below the configured minimum bytes and requests, the batch now stays open up to SyncBatchOption.Wait, armed once per batch so the added latency stays bounded. DiskLocation carries the option and defaults it; Store exposes a setter the server calls once at startup. * volume server: expose the fsync batch window as -fsync* flags -fsyncMinBatchBytes, -fsyncMinBatchRequests and -fsyncBatchWait carry the operator's batching floor into every disk's SyncBatchOption. -fsyncBatchWait=0 restores the previous flush-on-drain behavior. * server/mini: register the fsync batch window flags weed server and weed mini build their own VolumeServerOptions and left the three new fsync flag pointers nil, so startVolumeServer panicked dereferencing them at startup. * storage: bound the fsync batch window by Wait while the queue refills Once the straggler timer was armed, a steady trickle of queued writes kept taking the len(requests)>0 continue path, so the first write could wait past Wait until the size or count cap flushed the batch. Track the armed deadline and flush once it passes even with requests buffered. * storage: sync the straggler test on the worker taking the first write The fixed sleep let both writes queue together (passing under the old flush-on-drain behavior) or the straggler arrive after the window closed (flaky failure). Poll the drained queue instead so the second write can only land inside the armed window or a later batch. * volume server: default the fsync batch window when its options are unset weed server and weed mini build VolumeServerOptions without the fsync flag pointers, and outside embedders can too. Resolve the batch window from the defaults so startVolumeServer never dereferences nil options. * storage: drive the batch-window test through an injectable timer The window is armed by the worker now, so the straggler provably joins an open batch and expiry is fired by hand instead of racing a sleep. * seaweed-volume: accept -fsyncMinBatch/-fsyncBatchWait flags Same flags and defaults as weed volume so the Rust server's durable write batching can be tuned identically. * seaweed-volume: hold a below-minimum fsync batch open for stragglers Mirrors the Go worker's batchWait: once the queue drains under the minimum batch, the window stays open for stragglers and an expired window still bounds the batch while the queue refills. Paused-time coverage replaces any sleep-based assertion. * seaweed-volume: sync the straggler test on the armed batch window The test fired both writes together, so the straggler could already be queued when the worker drained — a regression that drops mid-window writes from the open batch would still pass. Wait until the worker arms the window before sending the last write (Go's test polls the drained channel for the same guarantee). Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * seaweed-volume: reject fsyncMinBatchBytes that does not fit usize --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
b0e6f7a096 |
s3: allow up to 50 tags in a bucket tag set (#12030)
* s3: allow up to 50 tags in a bucket tag set PutBucketTagging validated the tag set with the object tag rules, which allow 10 tags. AWS allows a bucket up to 50 tags, so tag sets of 11 to 50 tags were rejected with InvalidTag. Validate bucket tags against 50 and keep the limit of 10 for objects and S3 Tables. A tag set that repeats a key is rejected with InvalidTag, as AWS does, instead of keeping one of the values, so 51 tags with a repeated key no longer pass as 50. * s3: reject duplicate tag keys in PutObjectTagging The tag set collapses to a map, so a repeated key was silently kept once.\nMatch the bucket tagging check and answer InvalidTag instead. * s3: reject empty tag keys AWS requires a tag key to be at least one character; the shared\nvalidator only checked the 128-char upper bound. * s3: reject duplicate tag keys in the tagging header parser parseTagsHeader (the x-amz-tagging path used by CopyObject with TaggingDirective=REPLACE) silently kept the last value when a key repeated, while the PutObject and multipart header paths and the new XML checks return InvalidTag for the same input. A repeated key after URL decoding now fails the same way. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: validate x-amz-tagging tags on PutObject before storing the entry The header path only rejected duplicate keys, so an upload could store more than ten tags or keys and values that fail S3 tag rules. Run the parsed set through ValidateTags like the XML tagging handlers do. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: validate x-amz-tagging tags on multipart initiate ParseS3Metadata stored header tags with no tag-set validation, the same gap putToFiler had. Multipart uploads now reject oversize or malformed tag sets with InvalidTag. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: count tag length in characters, not bytes * s3: validate object tags before uploading chunks * s3: reuse parseTagsHeader and drop raw header from warnings --------- Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
de02158212 |
filer: let NewChunkStreamReaderFromFiler fall back to the master on a cache miss (#11964)
refreshUrls invalidates every cached location of a volume after a failed read and re-looks it up through the reader's lookup function. With the cache-only masterClient.LookupFileId, one failed read to one replica left the volume missing from the filer's cache until restart, so metadata-log replay skipped it. Use GetLookupFileIdFunction like the other read paths. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
8844d9823f |
iam/oidc: wrap the jti replay cache in its own type (#12017)
* iam/oidc: wrap the jti replay cache in its own type The replay-protection state lived as ten fields and nine methods spread across OIDCProvider. jtiReplayCache now owns the store, count, bound, sweep scheduling, and cleanup worker; the provider keeps a single jti field and delegates configure, consume, stop, adopt, and merge. * iam/oidc: lazily allocate the jti replay cache A zero-value OIDCProvider left jti nil, so Initialize, single-use validation, Shutdown, and the replay-state handoffs panicked on a nil cache. Allocate the cache through a sync.Once-guarded getter instead. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
3e5aa8c3b3 |
topology: mark replica quarantined before recording the pair (#11983)
* topology: serialize quarantine records with their layout marks A quarantine publishes a (node, vid) record and marks the replica read-only at the layout level; release and heartbeat flag updates must observe that pair as one state. Store-then-mark let a concurrent release delete the record before the mark landed, stranding the surviving replica read-only; mark-then-store let a heartbeat clear the mark before the record was visible, returning a conflicting replica to the writable list. quarantineLock now covers every record<->mark pair: flagDuplicateVolumeId's quarantine, releaseResolvedQuarantine's undo, the isQuarantined-gated marks on registration, and the quarantine-folded read-only flag writes on heartbeat paths. Remote server pushes stay outside the lock; a push that lands after the record was released is undone, and a failed release push re-adds the record for a later retry. The release test now waits for in-flight async flags to settle instead of asserting on a single read. * topology: fold quarantine into RegisterVolume's flag rebuild RegisterVolume rewrites every replica's location read-only flag from what its node last reported, and reported flags deliberately exclude quarantine. Any registration of a shared vid -- a reconnected replica, a new copy, the self-heal and delta-heartbeat re-register paths -- therefore cleared the quarantine mark on every replica in the location, while the gate afterwards only re-marked the registering node. Two call sites had no gate at all. RegisterVolume now takes an isQuarantined predicate and evaluates it under accessLock for every replica it touches, so the mark is never dropped in the first place: a release unmarks after the rebuild, and a quarantine mark re-applies on the next rebuild, with no window between. * topology: keep quarantine when a vacuum commit re-adds a replica SetVolumeAvailable re-adds the replica into the location list; a re-added slot starts with a cleared read-only flag, so a quarantined replica came back flag-free and the next ensureCorrectWritables could return a conflicting volume to the writable list. Pass the quarantine state in and set the flag back under the same accessLock, with the caller holding quarantineLock so release cannot interleave between the check and the mark. * topology: merge quarantine records instead of overwriting them enforceQuarantine wrote back the state copy it loaded before its remote push, so a concurrent writer's enforced=true was lost: the server stayed read-only while the record said unenforced, and a later release skipped the writable push it owed. The write-back now reloads the record under quarantineLock and merges -- enforced wins from either side, reported is taken from the newer pre-enforcement observation only while no push has landed yet. All three compensating writable pushes had the same hole in reverse: a failed un-quarantine left the server read-only with no record to retry from, and release's restore blindly overwrote any record a fresh flag pass had created in between. They now go through restoreQuarantineRecord, which adds the record only when absent and otherwise sets enforced on the current one, preserving its reported flag. * topology: hold quarantineLock over registration's quarantine check-and-mark RegisterVolume reads the quarantine record and marks the replica read-only under accessLock, but the pair ran without quarantineLock. A release could push writable, watch the record get re-read, and then delete it: the replica stayed read-only with no record left to clear, and unchanged writable heartbeats skip SetReplicaReadOnlyFlag so the mark never repaired. Holding quarantineLock around the check-and-mark in all three production callers serializes it against the release clear-and-delete. * topology: serialize remote quarantine pushes behind a record recheck releaseResolvedQuarantine deletes the record under quarantineLock and sends the writable request after unlock. A conflict flagged while that request is in flight creates a fresh record and marks the replica read-only again, and the older writable request can still land last, leaving the server writable under an active quarantine. The reverse flap strands a stale read-only mark the same way. Route every QuarantineVolumeFn call through pushQuarantine, which holds quarantinePushLock across the remote call and re-checks the record under quarantineLock right before sending: a read-only push requires the pair to exist, a writable push requires it to be gone. A skipped push means the decision already moved on, so the final server state always follows the newest record. * topology: serialize quarantine pushes per replica, not globally A single quarantinePushLock held across the remote mark call let one slow volume server (up to the 10s client timeout) delay pushes for unrelated replicas. Keying the lock by (node, vid) keeps pushes to one replica ordered in record-decision order while pushes to others run in parallel. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: restore the layout read-only mark with a released record A failed writable push re-armed the quarantine record but left the replica marked writable in the layout, so PickForWrite could route to a replica the server still holds read-only. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: wait out async duplicate checks in the release-retry test TestFailedReleasePushRestoresQuarantineMark asserts layout marks around a release pass, but the two RegisterVolumeLayout calls each start a background flagDuplicateVolumeId that can still be mid-flight. A late mark could land after the final release and fail the writable assertion even though the code behaved correctly. Track the async passes in a WaitGroup and drain them before the explicit flag call so the rest of the test is deterministic. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: keep quarantine when a replica's layout key is unreadable * topology: keep a failed writable push's restore ordered under the push lock * topology: retry the quarantine release on every duplicate check * ci: isolate Java and S3 test servers on persistent runners * ci: drop the duplicated -ip flags in the lifecycle test server The bind flags were already on the command; the isolation pass added a second pair. * topology: retry a failed quarantine writable push on the next report * topology: retry pending quarantine releases off the leader pulse A failed writable push re-arms the quarantine record with a retry that only enforceQuarantine fires, and that runs only for volumes a heartbeat carries. Once the server has reported the replica read-only it drops out of the delta reports, and restoreQuarantineRecord keeps the stored flag matching the server's so the digests agree and no full list is requested - the pending release then never retries and the idle volume stays read-only forever. Run the same release pass from the leader's pulse loop so a pending retry no longer waits on a report. * topology: exercise the pulse-driven quarantine release retry --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
5700aa5e91 |
s3: evaluate bucket CORS for path-style OPTIONS on the bucket root (#12028)
* s3: evaluate bucket CORS for path-style OPTIONS on the bucket root
A path-style preflight for /bucket (no trailing slash) missed the bucket
OPTIONS route, whose template is /{bucket}/, and was answered by the global
OPTIONS handler from the global AllowedOrigins instead of the bucket CORS
configuration. Register the bucket OPTIONS route without a path matcher, and
keep OPTIONS for the health probe paths on the service-level handler ahead of
each path-style router.
Fixes #12025
* test: drop the issue reference from the bucket-root preflight test comment
---------
Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Chris LuandDevin
|
9281ed8007 |
master: never auto-admit raft peers on a single-node configuration (#12021)
* master: never auto-admit raft peers on a single-node configuration A master started without cluster peers (peers=none, or no peers so the list holds only itself) declares a one-node cluster, yet any foreign master that registered over KeepConnected was admitted as a second voter. The resulting two-voter cluster cannot reach quorum when the foreign peer is unreachable, wedging a previously healthy single master. Gate raft admission on having been configured with cluster peers; masters started with a peer list keep the announce-to-join flow used to grow the cluster. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: source SingleMaster from -peers=none, not the peer count A master started with an empty peer list ends up with a one-peer map containing itself, and NewMasterServer inferred declared single-master mode from that count. The master still mints a raft cluster through the bootstrap path, but AdmitRaftPeer then ignores every later joiner, so a deployment that grows from one master to several can never form a quorum. isSingleMasterMode already spells out the declared case (-peers=none); populate MasterOption.SingleMaster from it instead. * master: cover clustered raft admission with a real in-memory leader The previous clustered test had no raft leader and asserted nothing, so an AdmitRaftPeer that always returned early still passed. Bootstrap a single-node hashicorp raft over the in-memory transport, wait for leadership, and assert the leader reports and admits a missing peer. * master: return the raft admission result and assert committed join AdmitRaftPeer ran AddVoter on a detached goroutine, so the only observable state was an in-flight marker deleted on success and failure alike, and the clustered test could never tell whether the join worked. Admission now hands back a result channel; the test runs a connected second in-memory node and asserts the peer sees itself a committed voter. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
fd3b437e04 |
s3api: write the null version's object-lock metadata to the plain entry (#12033)
* s3api: write the null version's lock metadata to the plain entry PutObjectRetention and PutObjectLegalHold with versionId=null loaded the pre-versioning entry but wrote the update under <key>.versions/v_null, so the acknowledged lock never bound the object it was read from. Resolve the entry and its write path in one helper that applies the same null-version rule as getSpecificObjectVersion. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: cover the null-version lock write path Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: persist object-lock metadata with an in-place entry update mkFile rebuilds the target entry from chunks alone, so a lock write on a plain (null-version) entry discarded attributes, inline content, and the remote reference. UpdateEntry keeps the whole fetched entry, matching how the tagging and ACL writers persist metadata. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
be081b49b7 |
notification: named webhook targets for per-path event routing (#11925)
* notification: honor the config prefix in the multi-queue guard validateOneEnabledQueue checked "<name>.enabled" without the notification prefix, so the fatal-on-two-queues guard never saw the real keys and could never fire. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: support named webhook targets under [notification.webhook.<name>] The webhook queue only accepted one endpoint, so events could not be routed to different services by path prefix. Each enabled [notification.webhook.<name>] subsection now spins up its own target — own filter, http client, retry middleware and worker budget — published on a per-target topic so every matching endpoint gets each event exactly once. The flat [notification.webhook] section keeps working as the unnamed target, and named subsections alone now activate the queue. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: require enabled for the flat webhook target buildTargets included the flat section whenever an endpoint was set, so a disabled [notification.webhook] block — which the scaffold ships with a placeholder endpoint — would start receiving events as soon as any named target activated the queue. * notification: resolve named target enabled flags through config namedTargetPrefixes read the raw map value, which never sees environment variable overrides; GetBool resolves WEED_* env bindings, so a named target disabled in the file can now be switched on by WEED_NOTIFICATION_WEBHOOK_<NAME>_ENABLED. * notification: cancel the webhook queue only after all workers stop Each target's router goroutine deferred the shared cancel, so the first worker to exit tore down the queue context for every other target and the dead letter logger. * notification: isolate webhook targets behind per-target queues SendMessage published to each target's topic inline, so a subscriber channel that filled up blocked every later target and the filer's notification path with it. Each target now drains its own inbox through a forwarder goroutine; SendMessage fails fast on a full inbox, matching the "throw message" contract the filer already applies to errors. * notification: name the webhook target in failure and dead letter logs Every target dead-letters to the same topic, so without the name there is no way to tell which endpoint failed or needs a replay. * notification: assert delivery in the named targets test The test only exercised SendMessage; it now records requests on both endpoints and checks matching, non-matching, and unrouted keys reach the right servers exactly once. * notification: wait for the webhook router before accepting events * notification: apply backpressure instead of dropping a full webhook inbox * notification: bound queued and in-flight webhook deliveries per target * notification: release the pending token when shutdown races the enqueue A SendMessage that wins the pending select as the queue is canceling could block forever on a full, undrained inbox. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: stop waiting for a router that failed to start A router that exits before reaching Running() would leave initialize blocked on a context nothing had canceled yet. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: route webhook events past saturated targets A full pending reservation made SendMessage wait on the slow target before it ever reached the healthy ones after it, coupling every endpoint's latency to the most congested one. Targets that can take a reservation now get the event immediately; saturated targets defer to a second pass that still blocks for a slot, so backpressure is kept and no event is dropped. * notification: retry webhook deliveries before dead-lettering PoisonQueue must wrap the retry middleware: added after it, the poison middleware intercepts the first handler error, republishes to the dead letter topic, and returns nil — the retry chain never sees a failure, so max_retries/backoff settings were dead code and every delivery failed straight to dead letter on the first attempt. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
fb3fb9dcb8 |
s3api: support s3:RequestObjectTagKeys in bucket policies (#11993)
* s3api: support s3:RequestObjectTagKeys in bucket policies Condition operators accept the ForAllValues: and ForAnyValue: qualifiers, and s3:RequestObjectTagKeys holds the keys of the tags taken from X-Amz-Tagging wherever s3:RequestObjectTag/<key> is set. CopyObject no longer stores an empty-key tag for an empty segment in X-Amz-Tagging, so it keeps the key set a policy has checked. Fixes #7679 * s3: parse tagging header like the tag handlers, not url.ParseQuery url.ParseQuery stops at 10,000 parameters, so an X-Amz-Tagging header padded with empty & segments made requestObjectTags return nil: the copy handler still stored the denied tag while every s3:RequestObjectTag* condition saw nothing. Scan non-empty segments the way parseTagsHeader does so the policy sees the same tags the object gets. * test: gofmt s3tables test files after #11997 The format check fails on these files on master as merged, so every PR's S3 Tables Format Check goes red until they are re-formatted. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
ecd778a4a0 |
iceberg maintenance: a merged file with null row ids names no first_row_id (#11973)
* iceberg maintenance: a merged file with null row ids names no first_row_id
Compaction stamps a merged v3 data file with the _row_id of its first row as
first_row_id (mergedFileFirstRowID). When the bin mixed rows that carry a
materialized _row_id with rows that carry none (a data file with no
first_row_id, as a v2 table's files are after an upgrade to v3, gives null
ids), the null rows then inherit first_row_id + their position in the file:
ids that other rows already hold, so two rows answer to the same row id.
A merged file now names a first_row_id only when its footer shows no null
_row_id in any row group. Otherwise the commit assigns the file a fresh range
from the table's next-row-id: the null rows get new, unused ids and the
materialized ones keep theirs (a materialized value overrides inheritance).
Test: TestMergedFileFirstRowIDOnlyWithoutNullIDs ("materialized first, a null
later" fails without the fix).
* iceberg maintenance: name no first_row_id when the merged file has a null _row_id
mergedFileFirstRowID stamped the merged file with its first row's id
without checking later rows. A null _row_id in a merged file inherits
first_row_id+position, an id another row already holds, so the commit
produced two rows answering to the same row id.
Check the footer's null counts for the _row_id column across row groups
first: any null means the file is committed without a first_row_id, so its
nulls take fresh ids from the table's next-row-id while materialized
values still override inheritance.
* shell: retry filer discovery until master knows a filer
A shell started against a freshly booted cluster (weed mini, the
s3tables/lance test harnesses) can list cluster nodes before the filer's
first heartbeat lands on the master. The empty list left FilerAddress
unset and the next command died with "dial tcp: missing address". Poll
ListClusterNodes until a filer appears or the connect timeout expires.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: discover the filer on first use instead of at startup
The startup retry loop made every weed shell invocation wait out the
connect timeout on clusters that have no filer, even for commands that
only talk to the master. Discovery now runs once, on the first command
that actually needs a filer, keeping the retry-until-registered fix for
the fresh-cluster race without stalling master-only commands.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: stop filer discovery retries on non-transient errors
A master that keeps answering ListClusterNodes with a permanent error
(permission denied, unimplemented) cannot gain a filer by retrying, so
discovery returns instead of polling until the deadline.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
|
||
|
Chris LuandDevin
|
6a34d82f05 |
shell: retry filer discovery until master knows a filer (#11989)
* shell: discover the filer on first use instead of at startup The startup retry loop made every weed shell invocation wait out the connect timeout on clusters that have no filer, even for commands that only talk to the master. Discovery now runs once, on the first command that actually needs a filer, keeping the retry-until-registered fix for the fresh-cluster race without stalling master-only commands. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: stop filer discovery retries on non-transient errors A master that keeps answering ListClusterNodes with a permanent error (permission denied, unimplemented) cannot gain a filer by retrying, so discovery returns instead of polling until the deadline. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: explain an empty filer list when discovery times out * shell: retry filer discovery while the address is still empty Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
57a122fbcf |
filer.remote.sync: dispatch an update event by the side of the mount it touches instead of slicing the mount prefix off a path outside it (#11958)
* filer.remote.sync: dispatch an update event by the side of the mount it touches instead of slicing the mount prefix off a path outside it The daemon subscribes to its mount prefix and to /etc/remote. An update event whose path lies outside the mount still reaches processUpdateEvent: an update of the /etc/remote directory entry itself (which fs.meta.load of a mapping dump re-creates) through the /etc/remote prefix, or a rename across the mount boundary through the path on the inside. processUpdateEvent then maps both paths to remote keys by slicing the mount prefix off, which panics with "slice bounds out of range [20:11]" on a path shorter than the mount and names a key in the wrong tree on a longer one. The event sits in the replay window, so every restart dies on it and the persisted offset never moves. Before dispatch, an update event is now mapped onto the side the mount covers: a rename into the mount is the creation of the new entry, a rename out of it the deletion of the old entry's object (kept when the entry is remote-only and the object holds its only copy), an event with neither path inside is not the mount's and is dropped, and an update with both paths inside is unchanged. * filer.remote.sync: extract processCreateEvent Moves the create branch of eachEntryFunc into a function, matching processDeleteEvent and processUpdateEvent, so the create path can be exercised directly in tests. No behavior change. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.remote.sync: upload a create reshaped from a rename into the mount A file synced under the mount keeps its RemoteEntry across renames. The rename out deletes the remote object, and the rename back in reshapes to a create whose inherited stamp still matches the entry, so shouldSendToRemote skipped the upload and the new key stayed empty. The reshaped create now sends unconditionally: the stamp describes the old key, not this one. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.remote.sync: a create reshaped from a rename into the mount writes an empty entry as a zero-byte object * filer.remote.sync: a create reshaped from a rename sends a stamped entry truncated since its sync A stamped entry still reports remote-only once truncated to zero (the stamp's RemoteSize stays positive), so the remote-only exemption skipped it and the new key got no zero-byte object. Gate the exemption on isDroppedLocalCopy: only an entry whose size still matches its stamp has its content already at the new key via the rename's remote copy. * filer.remote.sync: recover a superseded create reshaped from a rename-in A create reshaped from a rename into the mount whose write is superseded was skipped like a plain create, but no later event covers this destination: remote.uncache accepts the rename's inherited RemoteEntry stamp and drops the local chunks, and the update it logs skips the upload through isDroppedLocalCopy, while the source mount's sync still deletes the source object. The destination object is then missing and the content exists nowhere. recoverSupersededCreate finishes the event from the entry the filer holds now: a rewritten entry is left to its own event's upload or uploaded here on the stamp edge, and a remote-only entry is completed by copying the object its stamp describes -- resolved through a pending-rename marker under this mount, or through the filer's mount mappings to the source mount's remote location. --------- Co-authored-by: james <james@medable.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
5e01caa522 |
master: cap parallel vacuums per rack via master.vacuum.max_parallel_per_rack (#11935)
* master: cap parallel vacuums per rack via -maxParallelVacuumPerRack Hosts that run one volume server per disk under a single rack label could not bound host-level vacuum I/O: maxParallelVacuumPerServer limits each process, so every disk on the machine compacted at once. A second quota map keyed on rack id debits alongside the per-server one, so at most N volume copies compact in parallel within a rack. Defaults to 0 (no limit); the flag is wired on master, server, mini and master.follower. * master: resolve a data node's rack key once per vacuum sweep A volume server that unregisters mid-sweep loses its parent node, so the quota credit path calling dn.GetRack() panicked and crashed the master. Rack names are also local to their data center, so same-named racks in different centers shared one quota bucket. Each node's rack key is now captured when the limiters are built and carries the data center id, so checks, debits and credits never touch the live parent pointer. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: reserve vacuum quota per needed copy count, not per copy Each copy of a volume checked the shared server/rack quota independently, so several same-rack replicas all passed against one remaining slot and the debits drove the bucket negative. Needs are now counted per server and per rack first; a volume whose own copies exceed a cap reserves the whole bucket once it is free instead of pending forever. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master.follower: register -maxParallelVacuumPerRack as a flag It was only a fixed default, so passing the flag on the command line failed flag parsing. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: key the rack vacuum quota by rack node identity Joining data center and rack names with "/" still let two different racks share one bucket when the names themselves contained slashes. Key the limiter by the rack node instead, which is unique per rack regardless of naming. * master: keep the tiered-replica vacuum test through the rebase * master: document 0 as no limit for -maxParallelVacuumPerServer The clamped per-server need makes a non-positive cap mean "no limit", matching the new rack flag; say so in the flag help. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: key the rack-key cache by data node identity Node ids are only unique within a rack, so two servers that share an id in different racks would have reused the first server's cached rack and escaped the per-rack vacuum cap. * master: read the per-rack vacuum cap from master.toml -maxParallelVacuumPerRack becomes the master.max_parallel_vacuum_per_rack config key, so the new master limit is set the same way as other master options instead of a flag wired into every command that embeds a master. * master: let master.max_parallel_vacuum_per_server override the flag The released -maxParallelVacuumPerServer flag keeps working, but a value in master.toml now wins when set, matching how master.mdir already overrides -mdir. * master: group the vacuum caps under master.vacuum in master.toml Rename master.max_parallel_vacuum_per_rack to master.vacuum.max_parallel_per_rack and master.max_parallel_vacuum_per_server to master.vacuum.max_parallel_per_server, so the keys sit in one subsystem table like master.volume_growth and master.maintenance. * master: deprecate -maxParallelVacuumPerServer for master.toml The flag still works - and is overridden by master.vacuum.max_parallel_per_server when set - but the help text now points operators at the config key, matching the codebase style of help-text-only deprecation. * topology: read node parent atomically UnlinkChildNode clears a data node's parent under the rack's lock while readers like the vacuum sweep's rack capture reach it with no shared lock, so a torn interface read could crash the master. Store the parent in an atomic.Pointer and nil-guard the detached walk in DataNode.GetTopology. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
0d00e92178 |
iceberg maintenance: match percent-decoded keys when checking file references (#11978)
* iceberg maintenance: match percent-decoded keys when checking file references Writers may percent-escape partition directory names in the file paths they record in manifests (e.g. "part%3D1"), while the S3 layer decodes the key before storing the object. remove_orphans compared the escaped reference to the decoded filer path, counted the live file as an orphan, and deleted it once it passed the age window — readers then got NoSuchKey on the next scan. Match both spellings of every referenced path in remove_orphans, apply the same coverage to the kept-file set in expire_snapshots, and delete expired files at their decoded key so the object the S3 layer stored is the one removed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: retry filer lookups at the percent-decoded key A manifest reference carrying an escaped segment resolves to a filer name that does not exist, because the S3 layer decoded the key on write. When a lookup misses and the reference contains escapes, retry once at the decoded spelling so maintenance can still read the file. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: resolve referenced files verbatim-first, decoded fallback filer_pb.LookupEntry reports misses as ErrNotFound rather than a gRPC NotFound, so the escaped-reference retry never fired against a real filer. Both maintenance reads and expired-snapshot deletes now resolve through one verbatim-first/decoded-fallback lookup, and expiry refuses to delete a resolved path outside the table's data directory. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: test expiry of a file stored at a literal percent key * iceberg maintenance: resolve referenced files in one filer lookup lookupFileByIcebergPath re-fetched the entry resolveFilerPath had just looked up, doubling the RPC per referenced file during compaction and expiration. The resolver now returns the entry alongside the path so a hit costs one lookup and a decoded fallback two. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: keep every decoded spelling of a referenced path The kept-reference sets recorded a path plus a single decode of it, but lookupResolvedEntry decodes recursively until the spelling is stable. A doubly-escaped reference could resolve to a spelling absent from the set and be deleted while still referenced. addFilerPathSpellings now walks the same decode chain the resolver does. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: keep every decoded spelling of a referenced path * iceberg maintenance: test expiry keeps a doubly-escaped kept reference * iceberg maintenance: clean decoded spellings in the referenced-path set * iceberg maintenance: test expiry keeps a kept reference resolved via decoded parent segment * iceberg maintenance: decode only the reference in the spelling set Decoding the composed filer path also decoded the table directory, so a table at a literal percent-escaped path produced a kept set missing the spelling the resolver returns, and expiry could delete a live file. * test: gofmt the s3tables cluster helpers Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
8541f959a9 |
filer.conf: merge fsync on presence like worm (#11953)
* filer.conf: merge fsync on presence like worm fsync is a pure preference, so OR-merging it meant a nested rule could never turn a parent's fsync off. Make PathConf.fsync optional and merge it most-specific-wins, matching worm: an unset rule inherits, an explicit value overrides in either direction. readOnly and disableChunkDeletion stay OR'ed — they are fail-closed locks quota enforcement depends on. Pre-version-2 configs were written with EmitUnpopulated, so every rule carries an explicit fsync=false that meant nothing; it is dropped to unset on load so it does not read as an opt-out. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.conf: test fsync override survives a save/reload round trip --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
673d1a02e0 |
operation: euthanize pooled upload conns that stall far below a fast host (#12003)
* operation: euthanize pooled upload conns that stall far below a fast host A pooled keep-alive conn can wedge and drain the request body at a trickle while fresh conns to the same volume run at full rate; every later upload routed onto it stalls for tens of seconds, indefinitely. Track per-host healthy-rate uploads (>=1MiB/s on a >=256KiB body) and, on a host that was fast within the last minute, cancel an upload still in flight past max(10s, size/256KiB/s): the transport discards the wedged conn and the existing retry dials a fresh one. A watchdog kill also clears the fast mark so the retried attempt runs unbounded, and hosts that never prove fast keep today's unbounded behavior - a chronically slow link is never penalized. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * operation: keep the stall watchdog armed through the response read uploadDone was set as soon as Do() returned, so a conn that accepted the request body then wedged on the reply escaped the watchdog and could park the upload for hours — the same failure this watchdog exists to kill. Deferring the done-store keeps the timer live until the function returns, so a stalled io.ReadAll is cancelled and retried like a stalled write. The redundant post_err recheck goes with it. * operation: pin watchdog coverage of a stalled response read * operation: disarm the stall watchdog for an upload's remaining attempts Once the watchdog has cancelled an attempt, a concurrent healthy upload to the same host can restore the fast mark during the retry backoff. The re-armed watchdog would then cancel the retry too, and a large upload that merely exceeds the floor could burn every attempt. Track the watchdog kill on uploadStatusError and carry a stallWatchdogDisarmed flag through retriedUploadData so the rest of that upload's attempts run unbounded; other uploads to the host keep their watchdog. * operation: pin watchdog disarm on retry, fresh-dial, and slow-upload cases * operation: drop the send-stall watchdog once the request body drains A wedged pooled conn is only blameworthy while it is still trickling the request body; once the body is on the wire the wait is server-side commit, which the transport's response header timeout already bounds. Cancelling then would fail a healthy write whose replication or fsync is simply slow. Track body EOF through a wrapping reader, gate the send-side timer on it, and re-arm the watchdog only around the response-body read, which the header timeout does not cover. * operation: prove a host fast only on a verified upload A quick rejection moves bytes no faster than the wire did, so mark the host from the body-send duration and only after the response checks out. * operation: count an abandoned request body as sent The transport closes a request body it never finished draining when the server answers early, so EOF alone under-reports the send boundary and the stall watchdog could still blame the conn during the reply wait. * operation: scope the body-sent mark to the current attempt A lagging Close or EOF from the first attempt's body could stamp the shared mark after the inner retry reset it, disarming the watchdog for a send that never happened; each attempt now points the watchdog at its own sent timestamp. * operation: drop only the fast mark the stall timer observed A concurrent upload can re-prove the host while a stalled attempt is in flight; deleting the key outright would erase that newer mark, so the watchdog now removes only the value it armed against. * operation: keep the stall disarm on a per-call upload option retriedUploadData marked the caller's UploadOption, so a caller reusing it across UploadData calls would start the next upload with the watchdog off and lose the cancellation this adds. * test: stall watchdog retry must fail if the conn gets cancelled The fake retry slept through any cancellation, so a re-armed watchdog firing mid-retry still let the upload succeed — the guard was untested. Park on the request context like a real wedged conn so a re-armed watchdog cancels the attempt and fails the test. * operation: prove the fast-host mark with elapsed-to-response Body-drain time only measures when the transport consumed the reader: kernel socket buffers can swallow a small body instantly on a slow link, inflating the rate and marking the host fast. The next upload then gets its conn cancelled mid-send despite making progress. Elapsed-to-response bounds remote speed from below — the server cannot answer before the body lands — so the mark can only under-mark, never over-mark. * operation: scale the stall budget off the host's measured rate Fixed bps limits fit no link well: 256KiB/s never fires on a slow link and waits far too long on a fast one. Record the wire rate of each sizeable upload and let a later upload run for slowFactor times the body's span at that rate — a wedged conn shows as an order-of-magnitude drop on any link speed. The recorded rate uses the wire body size so a pre-compressed payload no longer inflates the mark. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
6e8a2cdb14 |
filer.backup: process metadata events through the concurrent processor (#11951)
* filer.backup: process metadata events through the concurrent processor Each event was handled inline on the subscription loop, so one fetch+write at a time capped the stream at a few entries per second and a busy source kept building lag. Route events through the same MetadataProcessor filer.sync uses: it fans jobs out behind -concurrency while serializing same-path and subtree-barrier conflicts, and the checkpoint now advances on the processor's contiguous watermark via GetResumeTsNs, with Resubscribe replaying anything a failure pins. -disableErrorRetry keeps its log-and-skip meaning by returning nil after logging, since the processor would otherwise pin the checkpoint forever. * filer.backup: drain in-flight jobs before the stream retry loop reconnects FollowMetadata returns as soon as the stream ends while admitted jobs keep writing; the next doFilerBackup builds a fresh processor that cannot order its replayed writes against them, so an older upload could finish last and clobber a newer file. * filer.backup: checkpoint the completed watermark on a ticker and at stream end The offset callback only fires while events arrive, so work draining on a quiet stream stayed uncheckpointed until the next event and a restart recopied it. Persist the processed watermark every 3s when it advances, and once more after the post-stream drain. * filer.backup: bound the checkpoint wait before reconnecting setOffset has no context, so a stalled KvPut would hold persistDone and block the retry loop forever. The persist goroutine now performs the final write itself on stopPersist, and the caller waits at most 30s. * filer.backup: carry the processed position across retry attempts A follow attempt that ended with a pinned failure recomputed startFrom on retry, so a -timeAgo window could move past the failed event and skip it. doFilerBackup now returns the failure pin (or processed watermark) and the retry loop clamps startFrom to it. * filer.backup: resume retries at the previous attempt's pinned position The retry loop threaded resumeTsNs through but never consulted it, so a -timeAgo run recomputed startFrom as now-minus-the-window on every attempt — a failed event older than the window was skipped on reconnect. Use the pinned position when one exists; -timeAgo and the persisted checkpoint only choose the first starting point. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: resume below the failure pin so the failed event replays The metadata reader treats the resume position as exclusive — events at or below it are skipped — so resuming at the failed event's own timestamp would skip it. Back one tick off the pin. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: label the progress line as the received-event rate The counter increments when a job is admitted, not when its copy finishes, so the old wording showed backup throughput during a burst of slow copies while the processed watermark had not moved. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: bound the offset checkpoint RPC setOffset called KvPut on the background context, so a stalled write could pin a caller's drain or persistence loop indefinitely. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: keep the resume pin when filer-source setup fails A retried attempt carries resumeTsNs so the failed event replays from the same position, but filer-source and SSE initialization failures returned 0, so the next attempt recomputed -timeAgo and could move the window past the pinned event. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: serialize checkpoint writes per key A checkpoint write can outlive its caller's wait and land beside the next subscription attempt's writes, moving saved progress backward. Serialize writes per key inside setOffset and drop any value older than one already written from this process. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: gofmt * replication: serialize checkpoint writes per filer and key A single process-wide lock made a stalled checkpoint RPC block every stream's checkpoint, including the opposite direction of a bidirectional filer.sync. Keeping the last-written watermark per persisted key also let one stream's offset shadow another stream checkpointing the same key to a different target filer. Lock and track progress per (filer, key). * replication: keep the resume position behind a cleared failure pin A redelivery that clears failedTs lets the processed watermark advance past the earliest failure, but events dropped while the processor was stopped — possibly sharing the failure's timestamp — have to replay. Latched earliestFailedTsNs floors the resume position one tick below that failure so reconnects still cover them. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: persist floored checkpoints through the write ordering setOffset drops a value older than the last written, which also swallows a deliberate regression: the resume floor, an operator-seeded offset, or a snapshot high-water mark. allowRegress marks those callers so the checkpoint can move backward when a failure forces replay. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: order checkpoint writes by issuance, not value The value-based stale guard could never fire — every caller passed allowRegress because floors and seeds are deliberate regressions — and ordering by value cannot tell a stale high watermark apart from real progress: a write that outlives its attempt could still land after a floored resume and skip events owed a replay. Order by issuance instead: a write issued before the last one to land under the key is dropped, so late writes from a superseded attempt can no longer move the checkpoint in either direction while deliberate regressions still apply. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: keep the resume position one tick below the frontier * remote.sync: resume and persist through the floored watermark * test: concurrent metadata jobs fan out up to the limit Distinct-path events must all be in flight before any finishes — a serial loop could only ever have one running — while the limit still bounds the pool. Pins the property behind the backup-throughput fix. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
0d4e11a6e5 |
filer: bootstrap a timestamp-less peer entry onto a missing local path (#11975)
A zero local commit time conflated 'no local state' with 'state at the epoch', so a missing entry was skipped whenever the sender reported no commit time and the entry mtime was at or before the epoch. Only hold the copy off when a local entry actually exists. |
||
|
Chris LuandDevin
|
679835e4fc |
volume.fsck: run all filer-entry purges through the shared worker pool (#11960)
* volume.fsck: queue per-volume orphan deletes through the shared purge pool The worker pool added for -reallyDeleteFilerEntries only covered entries whose volume was missing entirely; the per-volume check that finds filer entries pointing at absent needles still issued one synchronous HTTP DELETE per orphan, which is the bulk of the work the report measured. One pool now serves both purge phases and is drained before the empty- directory sweep. * volume.fsck: keep hard link ids in the filer fid file Orphans queued by the per-volume check must hash into the pool by their hard link id just like the missing-volume path, so two paths sharing an id still serialize on one worker. The fid file now stores the id beside each path and readFilerFileIdFile hands it to the caller. * volume.fsck: cover orphan purge queuing and hard link id round-trip * volume.fsck: fail on a truncated hard link id in the fid file A .fid record cut short inside the hard link id length was emitted with a nil id, so its purge could hash to a different worker than the entries it shares the link with. Read the length strictly like the header and path. * volume.fsck: drain the save consumer before returning a root genFn error The early return left the save goroutine blocked on outputChan while the caller ran finishPurge, so a late queuePurge could send on a closed channel and panic. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * storage: skip EC shard-size encoding tests when disk space is low TestCalculateExpectedShardSizeWithRealEncoding writes a .dat file plus 14 EC shards per subtest (~1.3GB peak for the 512MB case) into the temp dir. Shared CI runners can run low on space while other test packages write to /tmp in parallel, and the encode then dies with ENOSPC mid-write, surfacing as a test failure. Check free space up front and skip the subtest instead. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
3810fb1123 |
s3: url-encode ListMultipartUploads results for encoding-type=url (#11998)
* s3: url-encode ListMultipartUploads results for encoding-type=url The response echoed EncodingType=url but returned raw upload keys, and XML 1.0 cannot carry control characters, so the encoder replaced them with U+FFFD. The listed key then matched no upload: AbortMultipartUpload with it answered 404 NoSuchUpload, leaving such uploads uncleanable. Encode the key-named elements AWS encodes for this API (Key, Prefix, Delimiter, KeyMarker, NextKeyMarker) the way ListObjects encodes its keys when encoding-type=url is requested. Upload ids stay opaque, and responses without encoding-type=url are unchanged. Fixes #11991 * s3: echo EncodingType only when encoding-type=url is honored A request with an unrecognized encoding-type echoed the value back in <EncodingType> while the keys stayed raw, matching neither AWS (which omits the element unless url encoding applies) nor the ListObjects handlers, which gate the echo on the parsed url flag. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
baf0dd680e |
ec.encode: require shard placement to meet -shardReplicaPlacement (#11947)
* ec: measure shard placement against replica-placement caps ecShardPlacementViolations reports where a volume's newest-generation shard placement exceeds the per-node and per-rack caps a -shardReplicaPlacement requests, using the same digit mapping the balance planner enforces (SameRackCount = shards/node, DiffRackCount = shards/rack, dc-qualified rack keys). * ec.encode: -strictPlacement keeps originals when placement breaches caps -shardReplicaPlacement tells the balancer the requested per-node and per-rack shard limits, but nothing verifies the outcome. With -strictPlacement the pre-delete shard verification also measures the final placement against those caps: a volume still over the node or rack limit aborts before its source is deleted, so the encode only completes when the requested failure-domain spread is actually met. * ec.encode: cover the strict-placement delete gate with a test The gate in verifyEcShardsBeforeDelete now runs against a stubbed topology: a compliant spread passes while an over-cap placement is refused before source deletion. The retry knobs become package vars so the test does not wait out the heartbeat-propagation backoff. * ec.encode: enforce -shardReplicaPlacement without a flag Drop the -strictPlacement option: the requested per-node and per-rack caps are always requirements at the pre-delete verification, so the encode only completes when the placement actually honors them. |
||
|
|
f5e319ae8e |
iam: GetUserPolicy returns NoSuchEntity for a policy name the user does not have (#11944)
* iam: GetUserPolicy returns NoSuchEntity for a policy name the user does not have When the user has stored inline policies, an unknown name fell through to the policy rebuilt from the user's actions, so a deleted policy still read back. The standalone IAM server already returns NoSuchEntity here. * iam: return service failure when listing inline policies fails A ListUserInlinePolicies error in GetUserPolicy fell through to action-based reconstruction, masking a store failure as a policy document. Propagate it as ServiceFailureException instead, matching the standalone iam server and the other callers in this file. * iam: return service failure when the inline policy store read fails --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
f8579bf728 |
s3: gate POST Object acl fields behind PutObjectAcl (#11939)
* s3: gate POST Object acl fields behind PutObjectAcl The POST Object handler promotes acl and x-amz-grant-* form fields onto the upload's ACL headers, and resolveFileMode honors the promoted canned ACL. The PUT path only applies such headers after an s3:PutObjectAcl grant; mirror that on POST by running the shared per-key authorization with the WriteAcp action whenever the request carries ACL headers. * s3: cover the POST Object acl gate Signed POST requests carrying an acl form field are denied for a writer without the WriteAcp action and for a writer a bucket policy denies s3:PutObjectAcl on the target key; the authorization boundary is exercised directly as well. * s3: cover x-amz-grant-* POST fields and the no-ACL case * s3: encode POST policy conditions as JSON in the acl-gate test |
||
|
Chris LuandDevin
|
168789d5f3 |
master: cap writable volumes per collection via -maxWritableVolumes (#11923)
* master: cap writable volumes per collection via -maxWritableVolumes Some deployments serialize disk writes (e.g. an Unraid array where concurrent writes to multiple disks collapse throughput). With more than one writable volume in a collection, the master's weighted pick spreads assignments over all of them, so several disks get written at once. -maxWritableVolumes=N narrows the pick candidates to the N fullest writable volumes, so a collection's writes concentrate on one (or few) volumes at a time. 0 keeps the current spread-everywhere behavior. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: give master.follower a maxWritableVolumes default * master: cap writable volumes inside the requested location * master: pick the capped volumes without sorting the writable list * master: assert the capped candidate set instead of random pick outcomes * master: describe -maxWritableVolumes as a per-layout cap --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
cd97c93deb |
command: keep filer, s3, and sftp up when the local unix socket cannot bind (#11999)
* pb: share the stale-socket tolerant unix listener ServeGrpcOnLocalSocket already treats the local socket as optional; pull the remove-and-listen block out so the filer, s3, and sftp http sockets can use it too. * command: keep filer, s3, and sftp up when the local socket cannot bind A stale socket file owned by another uid (e.g. left in /tmp by a containerized job) made os.Remove fail with EPERM and the Fatalf took the whole server down. The gRPC local socket already treats the unix listener as optional; do the same for the http/sftp sockets. * filer: clear shared localSocket flag when the unix socket cannot bind s3.localFilerSocket and (in weed server) the sftp local socket share the filer's -localSocket flag pointer. If the filer cannot bind the socket it now keeps running without it, so consumers still holding the configured path would dial a dead or foreign-owned socket. * command: keep sftp on its own localSocket flag under weed server sftpOptions.localSocket was overwritten with the filer's -localSocket flag, so -sftp.localSocket was ignored and an explicit filer socket path made the sftp server try to bind the same socket the filer already held. * s3: fall back to TCP when the filer local socket cannot be dialed The embedded S3 server copies the filer's -localSocket flag before the filer tries to bind it, so a socket that never comes up left every filer HTTP request dialing a dead path. Try the unix socket first and fall back to the request's TCP address on dial failure. * filer: keep the shared localSocket flag read-only after startup Clearing *fo.localSocket raced with the embedded s3 goroutine reading the same flag pointer and could not help a reader that had already copied it. The s3 filer client's TCP fallback covers the unbound-socket case without mutating shared flag state. |
||
|
Chris Lu
|
6c87fb70f9 |
server: skip local unix sockets that cannot be bound (#12000)
server: skip default local unix sockets that cannot be bound weed server fatally exited during S3, filer, or SFTP startup when the default /tmp/seaweedfs-<svc>-<port>.sock path could not be removed or bound. On shared hosts (e.g. CI runners where another user's weed left a socket in the sticky /tmp directory) a stale socket file turns into 'operation not permitted' and kills the whole server, even though the TCP listeners are unaffected. Warn and skip the unix socket listener instead, matching how the gRPC local socket already degrades. An explicitly configured -localSocket path still fails fast, since a co-located S3 server may be configured to dial the filer through it. |
||
|
Chris Lu
|
29b81a827d |
volume: also listen on localhost like other components (#11995)
* volume: also listen on localhost like other components Master, filer and s3 bind both the detected host ip and localhost via NewIpAndLocalListeners, but the volume server used a plain NewListener on bindIp, so a volume bound to the host ip never answers on loopback. Any tooling that polls localhost (e.g. the java test workflow's readiness curl on localhost:8080/status) then waits out its full timeout even though the volume is up and registered with the master. Use NewIpAndLocalListeners for the volume grpc, public and cluster http listeners, and add a small multiHttpdownServer composite so graceful shutdown still covers both listeners. * volume: stop and wait dual listeners concurrently A serial multiHttpdownServer.Stop let the localhost listener keep accepting requests while the bound-ip listener drained, and a serial Wait hid a localhost serving failure until the bound-ip listener ended. Run both waits and both stops in parallel: Wait reports the first error promptly, Stop closes every listener when shutdown begins. |
||
|
Chris Lu
|
c925869dcc |
shell: retry filer discovery until a filer registers (#11994)
* shell: retry filer discovery until a filer registers When -filer is not given, the shell discovers a filer with a single ListClusterNodes call right after connecting to the master. A filer can take several seconds to register after its port reports ready, so the list routinely comes back empty and the shell proceeds with an empty FilerAddress. Commands then fail with "dial tcp: missing address", which is how the s3tables Dremio integration test flaked in CI. Treat an empty list as retryable and keep polling ListClusterNodes until a filer appears or -connectTimeout expires. On timeout, print a clear "no filer registered" notice instead of failing later inside a command; master-only shell commands keep working either way. * shell: report the last filer discovery attempt failure A successful empty ListClusterNodes followed by a master outage left the diagnostic claiming no filer registered, hiding the real RPC failure. Track the outcome of the latest attempt instead: if the final list call failed, report that error; only when the last response was genuinely empty report that no filer registered within the timeout. * shell: test filer discovery retries against a fake master Extract the discovery loop into listFilers and cover the retry behavior with a fake master_pb.SeaweedServer: an empty cluster turning populated, an empty cluster exhausting the window without error, the last attempt's RPC failure being reported, and master-only commands still running after the discovery timeout. |
||
|
|
ad89f003a5 |
s3: end a catalog-vended session with the access key it was vended for (#11945)
* s3: end a catalog-vended session with the access key it was vended for A table credential the Iceberg catalog vends is a self-contained STS session: it stays valid until it expires (an hour by default) even after the access key that loaded the table is deleted or deactivated, or its identity is removed or disabled. An operator revoking a leaked key cannot cut off the table credentials that key already obtained. Sessions are stateless JWTs, so there is no revocation list to consult. This change follows the same pattern as role sessions, which are already refused once their role's ID no longer matches (checkSessionRoleBinding): a vended session carries the access key it was vended for (src_ak, recorded by the previous change), and validateSTSSessionToken checks it against the identity table held in memory. A session of provider iceberg-catalog that carries a source key is refused (InvalidAccessKeyId, as for an unknown key) once its identity is gone or disabled, or the key is no longer one of that identity's active, unexpired credentials. Scope: - Only catalog-vended sessions are checked; web-identity, AssumeRole and every other STS session are validated as before. - The vendor binds a session to the caller's key only when the caller is a stored identity holding that key. A caller that signed with another STS session's key, an OIDC user, or a request that recorded no key gets an unbound session, as today; binding it would refuse every request made with it. A session minted before this change carries no key and is unbound. - The check is one map lookup under the read lock and a scan of one identity's keys (about 11 ns, 0 allocations, with 1,000 identities). Tests (s3api_iceberg_vended_session_test.go) mint real sessions through the catalog vendor and validate them the way a data-path request is validated: deactivating, expiring or deleting key A ends A's session while the session vended through key B on the same identity keeps working; disabling or deleting the identity ends it; callers that are not a stored identity holding the key get unbound sessions that keep working; other STS sessions are not checked. The first two tests and the third fail without this change. * s3: keep the vended-session key binding when the key dies before vend vendedSessionKeyLive gated the binding, so a key revoked between the request's authentication and the vend skipped it and minted an unbound session that stayed usable until it expired. Every catalog auth path pairs the identity name in context with the key that authenticated, so a stored-identity principal always means the request signed with that identity's own key. Bind whenever the principal is a stored identity: a session bound to a now-dead key is refused on its first request, which is what the binding is for. * s3: gate the vended-session key binding on the authenticated caller type lookupByIdentityName(principal) named only that some identity exists, not that the request signed with a stored key: an STS session or OIDC principal that collides with an identity name got a session bound to its temporary key, which vendedSessionKeyLive then refused on every request. Verifiers now record the verified key as a stored credential, and the vendor binds only when that marker is present — keeping the binding even when the key was revoked between authentication and the vend. * s3: enforce the vended key binding under a dedicated claim Sessions minted before the binding existed carry the caller's key in the audit-only SourceAccessKeyId — including temporary STS keys that can never resolve in the identity table. Enforcing on that field would end live table credentials at upgrade; the binding now travels under BoundAccessKeyId, leaving legacy sessions unbound as before. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: bind vended sessions to a credential fingerprint The key id alone cannot identify a credential: an operator can delete a key and re-create it under the same id with a different secret, reviving sessions minted for the original. Verifiers now record a fingerprint of the secret at authentication; the session dies unless it still matches. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: keep recording the caller's key on every catalog session SourceAccessKeyId was unconditional audit metadata before the binding existed; gating it on the stored-credential marker dropped the record for callers that authenticated another way. With the binding under its own claim there is no ambiguity left, so the field records the caller's key again regardless of how it authenticated. * s3: key the vended-session credential fingerprint with the signing key The bound-credential fingerprint travels inside the vended session token. As a plain SHA-256 of the secret it gave any token holder an offline guessing oracle for weak secrets, and CodeQL flags it twice. Store the verified secret in the request marker instead and HMAC it with the STS signing key at vend and at validation. The claim can no longer be mined, and the per-request auth path stops hashing secrets. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
bf3081e23a |
s3 lifecycle: route version-file writes to current-version expiration (#11956)
* s3lifecycle: schedule ExpirationDays from version-file writes On a versioned bucket a PUT lands in the meta-log as the version file plus the .versions/ directory update — there is no bare-key event, so the version-folder skip dropped every version-file event for all kinds but ExpiredDeleteMarker. ExpirationDays therefore only fired via the recovery walker's cold-start / rule-change pass: once a shard had a saved cursor, no daily run ever scheduled another delete marker. A version-file write is the latest for its logical key at write time, so route it through the same event-driven evaluation the regular path runs. The match carries the file's identity with an empty VersionID — the dispatcher resolves the current latest at due time, and a fresher PUT meanwhile resolves to STALE_IDENTITY there. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lifecycle test: versioned Expiration fires on the second daily run Establishes the shard cursor with a first pass, then PUTs and backdates a new versioned object and runs again. The delete marker must appear without a rule change forcing a recovery walk. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: run TestLifecycleVersionedBucketExpiresOnSecondRun in the lifecycle matrix --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
2e7e691bed |
volume: drop stale index entries that resolve to a different needle (#11948)
* volume: drop index entries that resolve to a different needle An index row whose offset lands on a needle with a different id and a different size cannot have been produced by any write path: every appender indexes a key at the offset and size of the record it wrote, and WriteNeedleBlob keeps its indexed size consistent with the blob header. Such rows are corruption (for example an offset replayed with stale high bytes) and until now made the volume reject every write for the key with "mismatching cookie" while reads resolved to the wrong needle. When the duplicate check finds the indexed record belongs to a different key and size, drop the stale entry and let the append re-index the key. Same-key conflicts still fail the cookie check, and entries staying consistent with a re-keyed blob keep their existing semantics. * volume: verify the indexed needle header before streaming reads The buffered read path re-parses the record and rejects a needle whose stored size disagrees with the index, but the streamed path serves data bytes without ever looking at the header, so a stale index row would stream another needle's payload. Probe the header once up front and report a miss when the record is neither the requested key nor the indexed size, the same staleness test the write path applies. * rust volume: drop index entries that resolve to a different needle Mirror of the Go volume change: when the duplicate check finds the indexed record belongs to a different key and size, delete the stale entry and let the append re-index the key. * volume: overwrite a stale index entry on commit, not up front Deleting the stale row during the duplicate check published a tombstone before the replacement append and, for durable writes, before its fsync. A failed append or sync then rolled back the data but left the key deleted. The publish step already overwrites any older offset, so leave the entry in place and let it be replaced once the record commits. * volume: report streaming header-probe failures to the io error tracker A failed header read returned before the tracked body-read loop, so repeated EIO on the probe never reached checkReadWriteError. Skip io.EOF the way the loop does: it is a truncated record, not a disk fault. * rust volume: overwrite a stale index entry on commit, not up front Mirror of the Go volume change: the staged delete published a tombstone before the replacement append and its .dat flush, so a failed append or sync left the key deleted. publish_write already overwrites any older offset once the record commits. * rust volume: report needle meta read failures to the io error tracker read_meta is the header probe ahead of a streamed body: an EIO there escaped check_read_write_error entirely, so a failing disk could not trip the tolerance. Report the outcome the way read_full already does. * volume: retry the wrapped offset before judging an index entry stale A 4-byte index value can index past the 32 GiB wrap: the low offset may hold another needle while the real record sits at +MaxPossibleVolumeSize, which is where reads already retry on a size mismatch. Skipping the cookie check on the low-offset record alone would accept a foreign-cookie write over a live key. Retry the wrapped header read first, the same condition the read path uses; a wrapped read that fails for reasons other than EOF stops the write like the low-offset read does. * volume: probe the streamed needle header under the access lock With HasSlowRead the access lock is dropped after the index lookup, so a compaction commit could swap the backend between the lookup and the header probe and fail a valid download on the old offset. Take the read lock for the probe and resolve the compaction revision first, the way the streaming loop re-resolves it on every page. * rust volume: retry the wrapped offset before judging an index entry stale Mirror of the Go volume change: probe offset + MAX_POSSIBLE_VOLUME_SIZE under the 4-byte-offset build before treating a mismatched record as stale, so a live wrapped entry still gets its cookie checked. * rust volume: do not clear the io error streak on a successful meta read read_meta runs before the payload read in read_needle_for_get, so recording success there reset the streak before every failing download and the volume could never reach quarantine. Report failures only; read_full still records success after a complete read. * volume: record a meta-only read's success against the io error streak A meta read no longer clears the streak because a payload read still follows on GETs, but requests the meta read ends outright — HEAD answers and cookie mismatches — never reached the body path, so their success left earlier failures counted. Note the success on those terminal paths; mirrored in the Rust volume server. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: report streamed reads to the io error streak Streamed GETs never touched the tracker: a failing chunk read went uncounted and a completed download never cleared the streak, unlike readNeedleDataInto which does both. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: reset the io error streak only at terminal meta reads A streamed GET's metadata read runs before its payload read, so clearing the streak there erased the count before every failing download and the volume could never reach quarantine. Record the meta-read success only where the request actually ends: must-meta-only reads, HEAD responses, and cookie mismatches — matching the Rust volume server's placement. * rust volume: record the streamed read's success at the final frame Hyper can complete a Content-Length response without polling the body again, so a success recorded only when the stream is next polled could be skipped, leaving an earlier io error streak in place after a successful download. * volume: record a meta-only read's success on conditional 304 replies * rust volume: record a meta-only read's success on conditional 304 replies * volume: reset the io error streak only after the checksum passes A streamed read that ends in a CRC mismatch still cleared the EIO streak, so corruption could mask progress toward quarantine. The Rust side records success only after the final frame's checksum verifies; move the reset below the CRC check to match. * volume: satisfy codespell in the meta-read comment * rust volume: report range reads to the io error streak * rust volume: cover the range read's io error reporting --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
27fa285d48 |
filer: do not stall startup waiting out the leader-settle window (#11972)
* filer: do not stall startup waiting out the leader-settle window ListExistingPeerUpdates re-polled the leader for up to 10s when it reported zero filers. A fresh single-filer cluster can never gain peers, so every filer start paid the full settle timeout before listening — past the 6s health window in weed mini and short probe budgets. Take the leader's first answer instead; the fallback fullest-answer scan below still covers a leader that is mid-convergence. * filer: bound leader-settle re-poll below the startup health window * filer: update ListExistingPeerUpdates doc for empty-leader fallback * filer: re-check empty membership at the announce gate before registering A fresh filer that scans empty membership during a master restart or re-election skipped bootstrap and announced immediately, serving missing files until a peer happened to reconnect — and a late peer only got an incremental subscription, so pre-existing metadata was never backfilled. The announce gate now re-polls ListExistingPeerUpdates while the scan can still be disproved (10s), backfills from any late peers before their streams start, and then waits for their catch-up like startup peers. A genuinely empty cluster pays a delayed master listing rather than a stalled startup, since this runs after the ports already serve. * filer: overlap the late-peer copy cutoff and stream start A peer event stamped just behind this filer's clock could land after the traversal read but carry a timestamp below the stream start — the startup scan margins both cutoffs by LogFlushInterval for exactly this skew; mirror it for peers discovered at the announce gate. * filer: bound the late-peer bootstrap copy by the shared announce deadline MaybeBootstrapFromOnePeer now takes the caller's context so a stalled peer metadata copy cannot out-wait the announce cap, and the bootstrap re-check shares one announceCatchUpCap deadline with the catch-up wait instead of each spending a full minute. * filer: merge bootstrap entries by commit time instead of blind insert The late-peer copy runs while gRPC already serves, so a blind InsertEntry could replace a write the local filer just accepted. Routing each entry through applyTraversedEntry orders it against the local commit time under the replay locks — a fresher local write survives the backfill. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
b65832bdc9 |
iam/oidc: reject reused jti claims with a bounded replay cache (#11936)
* iam/oidc: reject reused jti claims with a bounded replay cache * iam/sts: pass jti replay settings through convertToOIDCConfig * iam/oidc: keep consumed jtis and stop the cleanup worker across re-init Re-initializing a provider reset the JTI store, so an already-used token could be presented again after a config refresh; the store is now allocated once and survives Initialize. The cleanup worker is stopped on every re-init (previously only when protection stayed enabled), is started only after the rest of Initialize has succeeded, and Shutdown no longer consumes a permanent sync.Once, so a re-initialized worker can be stopped again. * iam/oidc: enforce the jti replay cache bound for every token Tokens without a usable jti claim skipped the cache entirely and stayed reusable; they are now keyed by the token's own fingerprint so a repeat presentation is still a replay. Tokens without exp previously failed as expired on first use; their entry now gets a bounded retention. The replay test-and-set now runs before any capacity work so replayed tokens cannot each trigger a full-cache scan, the entry count is incremented as part of the insert so concurrent validations cannot overshoot the bound, and a guaranteed serialized sweep runs before a capacity rejection while opportunistic sweeps are rate limited and confined to one at a time. Raw token identifiers are no longer written to logs or errors, a stable fingerprint is logged instead. * iam/sts: shut down providers dropped by the managed-provider swap Each managed OIDC provider runs a background JTI cleanup worker, but SetIAMManagedOIDCProviders replaced the provider map without stopping the old providers, so every refresh leaked goroutines. Providers absent from the new map are now shut down; providers that stay keep running. * iam/integration: reuse providers across refreshes for unchanged records Every refresh rebuilt providers from their records, so a refreshed provider forgot the JTIs it had already consumed and an already-used token could be presented again. Records are now fingerprinted and an unchanged record reuses its provider, keeping the consumed-token cache; changed records are rebuilt, removed records are dropped, and the swap in STS releases the workers of whatever is no longer referenced. * iam/oidc: consume replay entries only on the single-use exchange path Bearer authentication validates the same token on every request, so consuming it in shared ValidateToken/Authenticate made every token single-use and rejected all but the first request. The replay check now runs only for the STS exchange via AuthenticateSingleUse; bearer paths validate without consuming. A token without exp also no longer gets a bounded replay entry that ages out while the token stays valid: its entry is kept until the capacity bound evicts it, so it can never become usable again. * iam/oidc: keep the mock provider on the mock path for single-use calls * iam/oidc: require exp on single-use tokens A cache slot frees only by expiry, so an expiry-less entry would hold it forever; a burst of distinct tokens could exhaust the bound permanently and block every later exchange. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: give each web-identity test token a unique jti Subtests minted tokens with identical claims inside one second, so the exchange saw the same token fingerprint twice and the second call read as a replay. Each exchange now carries its own jti. * iam/oidc: carry the consumed-token replay store across provider rebuilds A record change rebuilds the managed provider with a fresh JTI store, so a token consumed before the rebuild could be presented again until it expired. The new provider adopts the replaced provider's replay store before it is published, and the replaced provider is shut down so only one cleanup worker sweeps the shared store. * iam/oidc: stop the cleanup worker of providers dropped from the store A refresh removed the provider object from the managed cache but left its JTI cleanup goroutine running for the life of the process. * iam/oidc: merge a provider's consumed replay entries into another's store MergeReplayStateFrom folds one provider's consumed-token entries into another's, skipping already-expired ones. Unlike AdoptReplayStateFrom it swaps no pointers, so it is safe on a provider still serving validators. * iam/integration: share one provider and replay store across identical issuer records Records for the same issuer URL with identical validation settings now reuse a single provider, so a token consumed through one account's record is rejected through the other's — previously each record owned a separate jti store and a spent token could be replayed cross-account. A superseded provider's consumed entries merge into the shared store so no rejection is lost, and a removed or rebuilt provider is only shut down once no remaining record references it. * iam/oidc: hand the superseded provider's live replay store to the shared one * iam/oidc: carry merged replay entries without charging the capacity bound * iam/oidc: key the jti-less replay fingerprint on the signed input Hashing the whole token lets ECDSA signature malleability (s -> N-s) mint a second cache key for identical claims, so a consumed token could be replayed for another exchange. Key the fallback on header.payload instead. * iam/oidc: publish the jti cache bound before the enabled flag Initialize stored jtiEnabled ahead of jtiMaxSize, so a provider flipped from disabled to enabled exposed enabled=true with a zero bound for a window; a single-use validation landing there would compute newCount > maxSize and reject with "JTI cache capacity exceeded". Store the bound first so the flag is the last thing published. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
186bc5cf7f |
s3: bound ListMultipartUploads memory to the page size (#11971)
* s3: bound ListMultipartUploads memory to the page size listMultipartUploads loaded every open upload in the bucket into a slice and selectMultipartUploads filtered and sorted a second full copy, so one request cost O(total open uploads) heap regardless of prefix, key-marker, or max-uploads; concurrent calls multiplied it and could OOM the gateway. The response is ordered by the object key kept in extended attributes, not the upload id the filer sorts by, so every entry still has to be seen. Streaming the listing into a bounded top-K collector keeps at most maxUploads (capped at maxUploadsList) candidates, bounding heap by the page instead of the folder. A caller-supplied max-uploads above the server cap now also paginates rather than accumulating unboundedly. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: cover bounded ListMultipartUploads accumulation Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: report the effective max-uploads cap in ListMultipartUploads * s3: cover ListMultipartUploads retry through the collector --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
7154732b84 |
worker/ec: retry the pinned volume copy after a mid-copy compaction (#11929)
* worker/ec: retry the pinned volume copy after a mid-copy compaction The EC task pins .idx/.dat copies to the CompactionRevision read from ReadVolumeFileStatus so the pair comes from one volume generation. When vacuum compaction lands between that read and either CopyFile, the server rejects the copy with "volume N is compacted" and the whole task fails — on clusters with queued maintenance this makes most EC tasks die to a race they cannot avoid. On a compaction mismatch the copy now restarts under a freshly read revision, up to a bounded number of attempts; other errors still fail immediately. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * worker/ec: read the retried copy's expiry from its own file status A mid-copy compaction makes the source reload the volume, and the fresh ReadVolumeFileStatus can carry a different expireAtSec than the pre-compaction read. Keeping the attempt-0 expiry could let the EC volume outlive its source TTL. * worker/ec: cover the compaction retry with a stub volume server * worker/ec: also observe revised .idx bytes in the compaction retry test * worker/ec: read the fake source under its mutex in the retry test --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
f2d5c27f65 |
operation: renew the write JWT when a fan-out rollback delete is rejected (#11970)
* operation: renew the cleanup JWT when a fan-out rollback delete is rejected A stalled holder can out-wait the assignment JWT's 10s expiry, so the type=replicate delete that rolls back landed copies comes back 401 and the needle leaks. Retry the delete once with a fresh token minted through the new LookupJwtFunc hook when a holder answers unauthorized. * s3api: wire a cleanup-JWT renewer into chunked uploads Keep the master client on the server and use it to mint fresh volume JWTs for rollback deletes that run past the assignment token's expiry. * admin: wire a cleanup-JWT renewer into file-browser uploads * operation: bound the cleanup-JWT renewal with a per-attempt deadline A rollback delete that hits 401 renewed the token through a background context, so a master outage could park the wait for a connection and the lookup RPC inside the failed upload's cleanup path. Thread a context through LookupJwtFunc and LookupJwt, and bound each renewal with a 10s deadline covering both the master wait and the lookup call. * operation: let every holder's 401 retry with a renewed cleanup JWT A single renewed flag meant only the first rejecting holder refreshed the token; a holder that stalled past the renewed JWT's lifetime left later copies stranded with a 401 they could not resolve. The retry limit is now per holder — each delete is attempted once with the current token and once more after a bounded renewal — so a slow middle holder cannot strand the rest. * operation: log the volume server's error body on a failed cleanup delete Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * operation: count test upload attempts atomically The chunk upload loop fans AssignFunc out to parallel goroutines, so the plain int counter in the partial-results test raced under -race. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
b97ed0f101 | 4.49 | ||
|
Mohd Quamar Tyagi
|
94dd96a322 | weed filer.remote.sync: ignore .system directory under buckets (#11966) (#11969) | ||
|
Chris Lu
|
00e64441c6 |
fix: clone protobuf snapshots without copying internal locks (#11968)
* fix: clone protobuf snapshots without copying internal locks * fix: reuse cloned snapshot fields when filtering |
||
|
Chris Lu
|
8f6121694a |
test: fix HEAD bucket assertions and FUSE failover startup timing (#11967)
test: fix HEAD bucket assertions and warm up failover clusters |
||
|
Chris Lu
|
137d29633b | s3: fix prefix filtering and delimiter pagination (#11962) | ||
|
Chris LuandJavier de la Torre
|
82f0a82564 |
filer+volume: make acknowledged writes durable under the fsync rule (#11943)
* volume: never compact a volume tiered to remote storage A tiered volume's data backend is the remote object, but compaction reads the local .dat path. When tiering removed the local .dat, every vacuum of a tiered volume with garbage fails with "no such file or directory", and the master's sweep and the worker's vacuum task keep selecting it. When the local .dat was kept, the compaction succeeds against the stale copy and the commit reloads the volume to read the remote object (old layout) through the compacted index, so live needles stop reading. Mirror the Rust volume server (#11545): - CompactByIndex and CompactByVolumeData refuse a volume with a remote file before creating .cpd/.cpx ("volume N is tiered to remote storage, cannot compact"). - CommitCompact refuses under the data-file locks when the volume became tiered during the copy, and removes .cpd/.cpx. - Store.CheckCompactVolume reports no garbage for a tiered volume, so the master's vacuum and volume.vacuum do not select it. - The worker's vacuum detection skips a volume with a remote copy, as balance and EC detection already do. Tests (each fails on master): TestCompactRefusesTieredVolume, TestCommitCompactRefusesVolumeTieredDuringCopy (on master the commit succeeds and a surviving needle no longer reads), TestPickPrimaryReplicaSkipsTieredVolumes. * volume: skip tiered replicas in vacuum selection, keep their real ratio Reporting a zero garbage ratio for a tiered volume hid it from volume.tier.compact, whose preflight reads that ratio while the volume is still remote, and did not exclude it from a vacuum with a zero threshold. Report the real ratio again, as the Rust volume server does, and have the master's vacuum check skip a replica whose volume info names remote storage, whatever the threshold or request. When CommitCompact refuses a volume tiered during the copy, also close the temporary needle map and remove .cpldb: nothing else would, since no later vacuum selects the volume. The tiered-compaction test no longer depends on random needle sizes reaching a 0.3 garbage ratio. * volume tier: do not attach a copy of a .dat a compaction replaced The upload copies the .dat, then records the remote file in the .vif and switches the backend. A compaction commit in between swaps in a compacted .dat and index; the upload then served the old-layout remote object through the new index, so live needles stopped reading. CommitCompact's tiered check cannot see this: the volume is not tiered until the switch. AttachRemoteDatFile records the remote file and switches the backend under the data-file locks, and refuses when the backend is no longer the file that was copied. The upload then removes the unused remote copy. * volume tier: keep a committed .vif entry when only the dir fsync fails AttachRemoteDatFile rolled back the in-memory remote entry on any SaveVolumeInfo error. A NotCrashDurableError comes after the rename, so the .vif on disk already names the copy; with the entry gone, the upload handler deleted that object, and a reload would serve from a missing remote file. Keep the entry and switch to the copy as the .vif says, and still return the error so the caller keeps the local .dat. * vacuum worker: filter tiered replicas instead of skipping the volume A partially tiered volume is the leftover of an interrupted volume.tier.upload; skipping the whole volume left garbage on its local replicas unreclaimed. Per-replica filtering matches the master's batchVacuumVolumeCheck and is safe: a tiered replica refuses both compaction and commit. * filer: sync leveldb entry writes when the path rule asks for fsync The fsync storage rule only reached the volume upload; the metadata writes behind an acknowledged create, update or delete still went to the leveldb journal with Sync:false, so a power loss could roll back an acknowledged bucket or object entry. A write-side flag on the entry context carries the request: the filer sets it from the path's storage rule in CreateEntry, UpdateEntry, EnsureDirectoryEntry and DeleteEntryMetaAndData, and the HTTP write path sets it for an explicit ?fsync=true. The leveldb stores write with Sync:true when the flag is present. * volume: flush the .idx before acknowledging a durable write A write with fsync synced only the .dat: the index row written on publish stayed in the page cache, so a power loss could leave the needle's bytes durable with nothing indexing them. The durable paths now run one .idx sync after publish, before the requests complete — one per syncWrite and one per batch run, paired with the .dat sync, the same split the Rust volume server makes between flush_dat and flush_idx. A failed .idx sync cannot be rolled back, so it stops the volume taking writes and fails the requests instead of acknowledging an index entry that may not survive. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
|
7603528aff |
s3api: support s3:RequestObjectTag/<key> in bucket policies (#11910)
* s3api: support s3:RequestObjectTag/<key> in bucket policies The tags come from X-Amz-Tagging only where the handler stores them from there: not on PutObjectTagging, which reads the body, and not on a copy that keeps the source's tags. For a repeated key the last value counts, since that is what CopyObject stores. * s3api: extract RequestObjectTag only where the handler stores it The header was honored on any request carrying it, so a tag attached to a GET, DELETE, part upload, multipart completion, directory marker or a PUT subresource satisfied a tag-conditioned policy the object never receives. Extraction now follows the handlers: PutObject, a copy under X-Amz-Tagging-Directive: REPLACE, and CreateMultipartUpload store the header; a plain object PUT is identified by carrying only X-Amz-* signing parameters. * s3api: exclude multipart part PUTs before the copy-source check A PUT carrying X-Amz-Copy-Source plus a numeric partNumber and uploadId routes to CopyObjectPartHandler, which stores a part and never reads X-Amz-Tagging. Checking the copy source first let such a request satisfy an s3:RequestObjectTag condition under REPLACE with tags that are never stored. Mirror the router: the part handlers match only when both partNumber (digits only) and uploadId are present, so a lone or non-numeric partNumber still falls through to the storing handlers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: gate tag extraction on the routed subresource, not query key shape Rejecting every PUT whose query held a key outside X-Amz- dropped X-Amz-Tagging from ordinary object PUTs that carry arbitrary parameters — Signature V2 presigns (AWSAccessKeyId/Signature/Expires) among them — so a tag-conditioned Deny could be bypassed by appending an unrelated query parameter while putToFiler still stored the tags. Exclude the object PUT subresources the router actually dispatches (tagging, renameObject, acl, retention, legal-hold) ahead of the copy-source check instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: cover the routed-subresource and part-copy tag extraction cases Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
cec96c02ea |
s3: reject requests carrying more than one routed subresource (#11942)
* s3: reject requests carrying more than one routed subresource Routing picks the first matching subresource route while authorization resolves a single action from the query, and the two can disagree. A bucket-level PUT ?tagging&policy is authorized as s3:PutBucketTagging but served by PutBucketPolicyHandler, letting a tagger rewrite the bucket policy; the same shape lets a list grant read the policy via ?list-type=2&policy or ?versions&policy, and lets an object ACL writer retag via PUT ?acl&tagging. Declining ambiguous requests keeps the authorized action and the served handler identical. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: reuse hasAmbiguousSubresource in the routed-subresource guard The standalone key list dropped versioning, so ?versioning&policy still reached route matching and auth with two operations. The bucket-level check already keys off operationSubresources plus bucketQueryActions; running the same predicate keeps the lists from drifting apart. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Junker der Provinzandzouyixiong
|
ee2718b14c |
filer: reduce forwarded lock log noise (#11933)
Forwarding a lock to its owner filer is routine in a multi-filer setup, so the forward and acquire messages move from V(0) to V(3). The acquire message is only logged when the forwarded response holds a renew token and no application error. Transport failures still log at V(0). Co-authored-by: zouyixiong <3390062+zouyixiong@users.noreply.github.com> |
||
|
Chris LuandDevin
|
7ff77a4f3b |
s3api: skip error response body when http forbids it (#11931)
WriteResponse wrote the payload even for HEAD requests and statuses that disallow a body (1xx, 204, 304), which made net/http return ErrBodyNotAllowed and logged 'write err: http: request method or response status code does not allow body' for ordinary requests. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |