15757 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
3 people	515380c349	s3: answer GET and HEAD of a delete marker as S3 does (#12051) ... * s3: answer GET and HEAD of a delete marker as S3 does A delete marker as the current version now returns 404 NoSuchKey with x-amz-delete-marker: true and the marker's x-amz-version-id. Asking for the marker by versionId returns 405 MethodNotAllowed with the same headers and Last-Modified, instead of 404 NoSuchKey. * s3: check the tag-conditioned read grant before answering a delete marker The marker branch returned before the entry-based policy recheck the rest of the read path performs, so an s3:ExistingObjectTag-conditioned Deny did not apply to the delete-marker response and its version headers. * s3: send Allow: DELETE with the delete-marker 405 RFC 9110 15.5.6 requires a 405 answer to list the methods the resource supports; S3 sends allow: DELETE for a delete marker named by versionId. * s3: keep delete-marker headers on conditional-read misses If-Match or If-Unmodified-Since against a current delete marker short- circuited to a bare NoSuchKey. Carry the resolved marker through the conditional result so the 404 keeps x-amz-delete-marker and x-amz-version-id, matching the unconditional read. * s3: exercise delete-marker responses through GET and HEAD handlers * s3: read marker version id from the entry before the caller's guess A delete marker written outside the normal PUT path can miss the version-id attribute while the v_<id> file name still names it; the new x-amz-version-id header would then report "null". Reuse versionIdFromEntry so both the handler and conditional-miss paths emit the real id. * s3: answer 405 for versionId=null on a suspended-versioning marker The null marker lives in .versions while getSpecificObjectVersion only checks the plain path, so GET/HEAD ?versionId=null answered NoSuchVersion. On that miss, look up the null marker and answer it like any other marker named by versionId. * s3: propagate null-marker lookup failures instead of answering 404 A transient filer error reading .versions/v_null was swallowed as "no marker", so GET/HEAD answered NoSuchVersion for a live read failure. NotFound and absent entries still mean no marker; anything else is InternalError now. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>	2026-10-25 12:02:19 -07:00
Chris LuandDevin	9620e7ebfe	filer: auto-enable per-path commit records when a peer has its own store (#12062) ... * filer: enable per-path commit records only when a peer has its own store Since seaweedfs#11698 every metadata change writes a per-path commit record into the filer store (an MpTs key, plus an MpDl key on deletes), and the records stay after the path is deleted. They order the events a filer replays from peer filers that keep their own store, since replay only runs when peerSignature != f.Signature. Filers that all share one store (one Redis, MySQL, TiKV, ...) never replay each other's events, so the records are never read, yet each change still costs one or two extra store writes and the key count grows with every path ever changed — unbounded growth for in-memory stores. This turns the records off by default and latches them on the first time a peer reports a different store signature — no option needed. The latch is persisted in the store (MpRep) so it survives restarts and converges filers sharing a store even before they discover the peer themselves. While off, a filer keeps an MpOff marker so enabling later bumps the records' generation (MpGen, folded into each record key) past the unrecorded window: a record older than a change made while off cannot let an older replayed peer event win. Ordering for unrecorded changes falls back to the entry's mtime, the same as entries written before the records existed. Supersedes the manual option approach in seaweedfs#12060. * filer: keep delete tombstones while records off, harden enable path Review on the first commit: - A delete committed while the records were off left no entry and no usable tombstone: after enabling, an older replayed create could resurrect the path. Deletes and rename sources are now always recorded under the generation-0 MpDl key — they are never fenced by a generation bump, and they cannot go stale because every delete is recorded. Creates/updates still write nothing while off. - bumpPathCommitGen could regress MpGen under concurrent bumpers on a shared store. It now loops: each pass writes only a generation higher than any it has seen, so concurrent bumps converge upward. - doSubscribeToOneFiler proceeded into replay when enablePathCommitRecords failed, running the whole session without ordering. It now aborts the subscribe so the reconnect retries with the records on. * filer: bind pending commit-record repairs to their generation A failed MpTs write kept in metaTsPending could land after a shared-store generation bump, writing a pre-bump timestamp as a current-generation record and overriding newer unrecorded state. Each pending change record now carries the generation it failed under: repairs write it only while that generation is current, and ordering reads ignore it otherwise. Pending delete records need no binding — MpDl keys are never fenced. * filer: never move the local records' generation backward The converge loop in bumpPathCommitGen seeded from zero and both it and syncPathCommitGen stored unconditionally: a stale writer could leave the stored generation below what this filer already adopted, and the next sync or bump then walked the local value back — orphaning records it wrote under the higher generation. The bump now seeds from the local generation and every store only moves it forward. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: name a records generation after the off marker it consumes Two filers consuming the same off marker each counted one past the stored generation, so the second could store N+2 after the first stored N+1 and write records under a key the other could not read until the next check. The bump now takes the marker's own timestamp — filers racing on one marker land on one value — a marker older than the stored generation is cleaned as a leftover, and a stored value that already passed the bump is adopted instead of counted past. A newly enabling filer whose marker was already consumed names its own off window with the current time, so its unrecorded period is still fenced. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: tell a fresh off marker from one the generation already consumed A marker whose timestamp reads at or below the stored generation was dropped as leftover, but a new off period can also read that old — the off filer's clock lagging, or a host clock moving back — and dropping it kept records from before that period live, so stale replay could win. The stored generation now names the marker it was bumped for: a marker the store still shows is consumed unless it is that one, however old it reads, and only the consumed marker itself is cleared without a bump. The generation trails in the value, so a whole-slice read still yields it on stores written before the pair was stored. Enabling fences the filer's own off window with the current time instead of a marker a sharer may already have consumed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: cover racing off-marker consumers settling on one generation * filer: adopt the stored generation, not the consumed marker it names A filer syncing with no off marker present read the value's first field — the consumed marker's timestamp — so a restart after a lagging-clock off period under-counted the generation and kept records the bump dropped. * filer: re-check the off marker before consuming it A sync whose marker read ran behind a later off period's consume cycle still bumped for its stale read: the stored generation already covered a different marker, so the bump overwrote the consumed name, counted the generation past records a newer consumption meant to keep, and cleared whatever marker was live by then. Each pass now re-reads the marker and only bumps or clears while it is still the one the pass saw. * filer: keep a failed generation read from resetting the records' generation The final generation read ignored errors and then stored whatever gen held — possibly zero — so a transient store failure could hide records the current generation wrote. Unexpected read errors now return before the local generation changes. Also cover a sync whose marker read runs behind another consumer's delete: it must adopt the stored generation, not bump for the stale read. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: fail a bump that cannot verify the stored generation A generation read that failed mid-bump was treated as absent: the write could land below a generation a racing bumper had stored, or re-store a marker that bumper had already consumed. Read failures now abort the pass like every other store error on the path. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: cover generation reads that fail instead of returning a value Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: carry the commit timestamp on the entry itself The per-path MpTs record only exists because a commit timestamp had nowhere else to live. Stamping commit_ts_ns on the entry before its store write - and reusing that stamp as the emitted event's TsNs - makes the entry its own commit record: replay and bootstrap traversal order on it with zero extra writes for creates and updates. Deletes still need a durable key since a removed entry cannot carry a timestamp, so MpDl tombstones stay (one write per delete, growth bounded by delete history). Entries predating the field fall back to mtime, matching what peers and old records already assume. This replaces the MpTs/MpOff/MpGen/MpRep machinery entirely: shared- store filers write nothing extra, and dedicated-store replication keeps last-writer-wins ordering plus tombstone coverage for deletes and ancestor drops. * filer: keep a directory update replay from tombstoning its subtree A same-path update wrote a delete record before the delete-and-reinsert, so a directory metadata change read as a subtree delete: a later peer event for a child older than the update was then skipped even though nobody deleted the children, leaving files missing on this filer. Only file updates take the tombstone now — the re-inserted directory entry's own commit stamp orders the path itself. * filer: order unstamped entries by their recorded commit before mtime Entries written before stamping existed have no CommitTsNs, so the ordering check fell back to the client-settable mtime and dropped the MpTs record those versions wrote: an entry committed at event time 300 with mtime 100 lost to a delayed peer event at 200 after upgrade. Read the legacy record for unstamped entries in localCommitTsNs, CommitTsNs, and the traverse stamp (which also stopped falling back to mtime, regressing entry_ts_ns to 0). * filer: release queued rename event stamps when the rename fails Each moved entry carried an in-flight stamp its queued event would adopt at emit time. When the move succeeded but a later child move or the commit failed, the handler rolled back without emitting the queued events or releasing their stamps, pinning the delivered and flushed watermarks below the aborted rename forever. Release every queued stamp on both failure exits in both rename handlers. * filer: apply a directory update replay with an in-place upsert Deleting the stored entry before inserting its replacement left a window where the directory had no entry and no ordering record — a tombstone cannot stand in for it without reading as a subtree delete that suppresses the children's own records. A stale peer event accepted in that gap could replay an older update or delete and sweep the directory's children. Upsert the replacement instead: the entry keeps its recorded commit order across the whole update, and a directory has no chunks or hard links for a delete to release. * filer: update an existing directory entry on replay instead of upserting InsertEntry is not an upsert on SQL stores configured without upsert support — PostgreSQL with enableUpsert=false issues a plain INSERT and its duplicate-key error does not match the MySQL-only fallback, so a replay updating a present directory failed on every retry. Pick UpdateEntry when the entry exists and InsertEntry when it is absent; UpdateEntry touches no row that is not there. * filer: detect a shared-store delete racing a replayed directory update UpdateEntry reports success even when the row was deleted between the find and the update — the UPDATE changes zero rows and the replayed change is dropped while the subscription offset advances. Re-read the entry after the update and retry the write, re-checking the recorded commit order each pass so a newer peer change wins instead of being resurrected over. * filer: declare commit_ts_ns in field-number order Field 6 sits between extended and hard_link_id; the declaration now reads in the same order. Wire format unchanged. Also syncs the Java client's checked-in copy of filer.proto, which the pb Makefile copies on gen. * filer: option to skip delete tombstones entirely filer.options.deleteTombstones=false stops writing MpDl records on deletes - a single filer or shared-store cluster never replays peer events, so the tombstones are dead weight. The tradeoff is explicit: if a dedicated-store peer does replicate, a stale event can resurrect a deleted path. * filer: drop delete tombstones entirely Deletes no longer write MpDl records, and the pending-repair machinery they needed is gone with them: per-mutation metadata overhead is now zero for every store topology. Ordering information lives only on the entry itself via commit_ts_ns. The accepted tradeoff: a dedicated-store peer replaying an event older than a delete can resurrect the path, and a kept directory no longer fences stale child creates. Recorded tombstones and MpTs records left by older builds are still read, so upgrades keep the protections they already have. * filer: stop reading legacy per-path records too Drops the MpTs/MpDl read path along with the writes: meta_path_ts.go keeps only the entry-field readers (CommitTsNs, EntryCommitTsNs, localCommitTsNs). Entries written before stamping existed fall back to mtime, and tombstones or commit records older builds left behind no longer fence replayed events. * filer: trim commit timestamp comments --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-25 11:37:06 -07:00
Chris Lu	53834117a6	mini: lift the post-mint allocation hold on first inventory report (#12061) ... * mini: lift the post-mint allocation hold on first inventory report DeferAllocation armed a fixed warmup window whenever the leader minted a fresh topology id, so the first write on a brand-new weed mini blocked for the whole window even though no prior cluster state could exist. The window still has to cover a rebuilt master's veteran servers, which report on their first post-reconnect heartbeat. Mini has exactly one possible reporter — the embedded volume server, whose first heartbeat carries the full inventory — so hold allocation only until that report is applied. The bound raise and topology-id freeze checks run while the heartbeat is processed, so a wiped master state still raises the bound or trips the freeze before the hold lifts. Standalone weed master and weed server keep the fixed window: volume servers outside the process can still be on their way. * master: apply queued topology-id reports before opening the allocation gate SetTopologyInitPending(false) cleared the pending flag before replaying topology ids stashed while the local id was still being minted. A NextVolumeId landing in the drain window saw no gate and no freeze yet. The post-mint allocation hold used to cover the gap by delaying the winner past the freeze; once the hold can release on first inventory, the gate must stay closed until the queued checks have run. * mini: gate the first-report release on single-master mode releaseAllocOnFirstReport assumes the embedded volume server is the only possible reporter of prior cluster state. A mini started with -master.peers joins other masters, whose own volume servers can still be on their way when the first report lands, so keep the regular quiet-pulse hold there. * master: check reported topology id on empty full reports A HasNoVolumes heartbeat carries the same LastSeenTopologyId a populated list does, but the mismatch freeze only ran for non-empty reports. An EC-only veteran could therefore release the post-mint allocation hold through NoteInventoryApplied while its stale topology id never reached CheckReportedTopologyId. * master: drain the topology-id stash inside the init-gate lock CheckReportedTopologyId only held reportedTopologyIdsLock while stashing, so a report in flight during SetTopologyInitPending(false) could land its freeze after topologyInitPending cleared, letting a NextVolumeId slip through the gap. The whole check now runs under the lock, and the gate is released inside the same critical section as the drain. * mini, master: pin and enforce the single volume-server reporter isSingleMasterMode only checks master peers; SendHeartbeat still accepted any volume server, so a late foreign report could carry prior cluster state after the first-report release dropped the hold. Mini now pins the embedded volume server's address and the master refuses registration from any other reporter while ReleaseAllocOnFirstReport is set. * master: admit the volume-server reporter before applying heartbeat state * test: cover SendHeartbeat admission ordering A refused reporter must not freeze growth or raise the id bound, and the pinned reporter's stale topology id still must. * mini: re-derive the single-reporter release on every run The package-level master options survive in-process reruns, so a multi-master mini following a single-master one must not inherit the first-report release or a stale reporter pin. * master: admit addressless first heartbeats under the reporter pin An unregistered heartbeat with no Ip skipped the single-reporter admission check but still reached the seen-bound raise and the topology-id freeze before the delta-heartbeat skip. Admission now runs for every unregistered heartbeat; an empty address can never equal the pin.	2026-10-25 09:25:39 -07:00
Chris LuandDevin	88b91cccba	command: no extra localhost listener when -ip.bind is specified (#12055) ... * util: add listenOnLocalhost parameter to NewIpAndLocalListeners Nil keeps the current behavior of also binding localhost when the host is a specific address; all callers pass nil for now. * command: add -ip.bind.localhost flag to opt out of the extra localhost listener Binding a command to a specific -ip.bind address also opened a second listener on localhost for every port, with no way to turn it off. Each command that calls NewIpAndLocalListeners (master, master.follower, filer, s3, iam, volume, sftp, mq.broker, mq.agent) now accepts -ip.bind.localhost=false to bind only the requested address. Embedded services get prefixed flags (s3.ip.bind.localhost, sftp.ip.bind.localhost, iam.ip.bind.localhost), and the command-level flag also disables them so a single switch turns off every extra listener the command opens. * mini: only require all-interface port availability while the loopback listener is on With -ip.bind.localhost=false each service binds just -ip.bind, so a port held on another address no longer collides; the all-interface precheck would reject or relocate usable ports. WebDAV, the Admin UI, and the Admin worker gRPC (which binds -admin.worker.ip, not -ip.bind) never open a loopback companion listener, so their checks now use the address they actually bind. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mq.broker: bind the gRPC listener to -ip.bind instead of all interfaces The listener used an empty host so -ip was ignored for binding and the localhost companion was never created, leaving -ip.bind.localhost with nothing to control. Give the broker the same -ip/-ip.bind pair the other servers use; embedded mode binds to -ip.bind like its peers. * server: add -sftp.ip.bind.localhost and -iam.ip.bind.localhost switches weed filer already exposes the prefixed opt-outs for its embedded sftp and iam; give weed server the same pair so each extra listener can be disabled individually while -ip.bind.localhost=false still turns them all off. * mq.broker: keep the default all-interface bind unless localhost is off An unset -ip.bind used to mean the standalone broker listened on every interface; defaulting it to -ip silently cut off multi-homed clients. Only fall back to -ip when -ip.bind.localhost=false, where a specific bind is needed for the opt-out to take effect. * mini: validate the Admin HTTP port on the address it will bind The port check always used -ip.bind while startMiniAdminWithWorker falls back to loopback when Admin has no credential, so a loopback-held port passed validation and killed the admin at startup. * mq.broker: keep the all-interface default bind when -ip.bind is unset -ip only becomes the fallback once -ip.bind.localhost=false needs a specific address for the listener. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mini: validate the Admin HTTP port on its effective bind IP When the Admin listener falls back to loopback (no credential or mTLS configured), the pre-flight port check now probes that same address instead of -ip.bind. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * util: keep NewIpAndLocalListeners source-compatible for 3-arg callers * command: drop the -ip.bind.localhost flag; an explicit -ip.bind drops the extra listener Instead of an opt-out flag, the loopback companion listener now simply follows whether the bind address was chosen: it is created only while -ip.bind is unset and the bind address falls back to -ip. Passing -ip.bind=<addr> binds each port to exactly that address. Each service options struct carries a listenOnLocalhost *bool that embedders (weed server, weed mini, embedded s3/sftp/iam in weed filer) set from the parent's -ip.bind state, captured before the bind flag is repointed or filled in by the -ip fallback; standalone commands derive it from their own flag before the fallback runs. iam and mq.agent gain a real -ip.bind flag so they have a way to pin a single address. mq.broker keeps its new -ip.bind flag but drops the -ip.bind.localhost fallback hack: empty still listens on all interfaces. * server: default embedded mq.agent brokers to the broker's bind address With an explicit -ip.bind the embedded broker no longer listens on localhost, so an unset -mq.agent.brokers defaulting to localhost:17777 could not reach it. Default to <serverBindIp>:<mq.broker.port> (the address the embedded broker binds), falling back to localhost when the bind address is a wildcard. An explicit -mq.agent.brokers still wins. Also fix the NewIpAndLocalListeners skip message to say localhost. * server: bind embedded mq.agent to serverBindIp The embedded agent's ipBind was never set, so startQueueAgent fell back to the advertised -ip address and an explicit -ip.bind did not pin the agent listener. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-25 06:52:16 -07:00
3 people	fefac028ec	feat: send EC shards in parallel per destination disk (#12050) ... * feat: send EC shards in parallel per destination disk One stream per destination disk. max_shard_transfers unset means no global cap, 0 sends shards sequentially, and N is a global cap that still keeps one stream per disk. The value is persisted through TaskPolicy. The maintenance form is a checkbox plus Max transfers because unset and 0 differ. The key is intentionally omitted from GetConfigSpec. * ec shard distribution: treat negative max_shard_transfers as sequential The config layer (normalizedMaxShardTransfers) clamps negative values to 0, which selects sequential distribution. shardTransferCap diverged: a negative cap that reached ShardDistributionOptions directly would have selected uncapped parallel mode, the most aggressive setting. Align the options layer with the same normalization so invalid input degrades to the conservative mode. * ec: serialize shard transfers to a server when a target leaves disk choice to the receiver disk_id 0 is auto-placement on the receiving volume server, so an auto-placed stream can land on a disk another queued stream is pinned to. Grouping by (destNode, disk_id) alone let both run at once; collapse all of that server's transfers under the destNode key instead. * worker: clamp max_shard_transfers to the int32 wire range ReadIntConfig returned the 0 fallback for values above MaxInt32, turning a large positive cap into sequential mode. Read int64 and clamp into range instead. * worker: drop unused shard-transfer form helper The maintenance form renders and collects max_shard_transfers in plugin.templ JavaScript; the Go copy had no callers. * worker: bound max_shard_transfers conversion under an in-range guard --------- Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2026-10-25 06:37:26 -07:00
Jay HemnaniandChris Lu	0c9872f842	filer, volume: ignore Range when If-Range no longer matches (#12052) ... * filer, volume: ignore Range when If-Range no longer matches A GET with Range and a stale If-Range validator got 206 with the requested bytes of the current file. RFC 9110 13.1.5: send the whole content instead, so a resumed download restarts rather than mixing two versions. * volume(rs): drop Range when If-Range no longer matches Mirror the Go fix for the Rust volume server: evaluate If-Range against the response ETag (strong comparison) or Last-Modified once the needle's validators are known, then drop the range on a stale validator so the whole current representation is sent. A dropped range on the RangeFromSource strategy falls back to a full stream of the data-file region; buffered and chunk-manifest paths already take Option<&str>. * volume(rs): fail If-Range when the header value is not readable * volume(rs): accept all HTTP-date formats in If-Range like Go * server: cover alternate HTTP-date formats in If-Range test --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2026-10-25 06:36:40 -07:00
Chris LuandDevin	da1d7444b8	master: shorten the first-write hold after a fresh topology id (#12059) ... * master: release the post-mint allocation hold after a quiet pulse DeferAllocation armed a fixed warmup window whenever the leader minted a fresh topology id, so the first write on a brand-new cluster blocked for the whole window even though no prior state could exist. The window still has to cover a rebuilt master's veteran servers, which report on their first post-reconnect heartbeat. Track the last heartbeat that could carry prior cluster state — a server joining, held volumes or shards, a nonzero id bound, or a foreign topology id — and lift the hold one pulse after it goes quiet, capped by the original window. * master: make the post-mint allocation hold configurable master.volume_growth.allocation_deferral overrides the warmup window a leader holds volume id allocation for after minting a fresh topology id. Unset keeps the existing window; "0s" disables the hold for setups that know the cluster is fresh (test harnesses, single-node labs); any other duration replaces the cap while the quiet-pulse early release still applies. * master: re-check the allocation deadline while it is held NextVolumeId slept for the whole computed delay, so a heartbeat landing mid-sleep that pushed the quiet deadline out was ignored — the pending allocation could still run ahead of a straggler's report. Wait in short slices, re-evaluating the deadline and the growth freeze each round. * master: count only ids above the issued bound as prior-state evidence Heartbeat echoes of volumes and shards this master allocated during the hold refreshed the quiet window, so each growth allocation re-extended the deferral until the hard cap. Evidence now requires an id beyond the bound this leader has issued, a higher reported bound, or a foreign topology id; a joining server (dn == nil) still counts once. * master: reject an unparseable allocation_deferral instead of disabling the hold Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: do not count a skipped heartbeat as a joining server Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: assert the allocation hold extends past the original quiet deadline Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: release the post-mint hold after two quiet pulses One pulse proved short for a veteran that reconnects on a staggered restart: with only five seconds of quiet, allocation could resume and reuse ids before a slower server landed its first heartbeat. Two pulses still halves the fresh-cluster first-write stall versus the full warmup cap, and known-fresh setups keep the "0s" override. * master: pick once more when assign times out as growth concludes The pick loop's final attempt can predate the growth it triggered concluding: growth registers its volumes in the layout before clearing the pending flag, so a timeout landing on that boundary used to shed a hard 406 over volumes that had just arrived. Pick once more after the loop in both the http and grpc assign paths before deciding how to shed. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-25 06:17:42 -07:00
Chris LuandDevin	d15396fc4d	s3tables: scope the anonymous identity's grant on the table catalogs (#12056) ... * s3tables: scope the resolved anonymous identity instead of defaulting to allow Catalog authorization fell back to the open default whenever the request resolved to the configured anonymous identity, ignoring its grant. The fallback now applies only when no principal was resolved (zero-config), so anonymous requests are checked against the identity's configured actions like any other principal. The S3 Tables API and the Iceberg and Lance catalogs share this handler, so all three are covered. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: authenticate the catalog clients in the s3tables suites Anonymous calls into the Lance and Iceberg catalogs are now scoped by the anonymous identity's grant, so the suites that used to rely on the anonymous pass authenticate like their real deployments do: x-api-key for the Lance probes and REST drivers, client-credentials OAuth for pyiceberg. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * worker(lance): authenticate namespace calls with the storage credential Anonymous catalog calls are now scoped by the anonymous grant, so a worker that holds an S3 credential sends it as x-api-key on the namespace requests instead of relying on the anonymous pass. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: authenticate the pushdown catalog connection in catalog_lancedb Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: follow catalog redirects only within the same host reqwest forwards custom headers on a follow, so a redirect to another host would carry the x-api-key credential to a server it was never meant for. Keep same-host hops for proxies that normalize the path or upgrade the scheme. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: warn when the catalog api key travels in cleartext The x-api-key pair is a real storage credential: over plain http to a non-loopback host the network can read it. Warn at client build so the operator sees it; https and loopback stay quiet, and the key still sends since an http catalog is a valid deployment. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: require the same origin on catalog redirects A same-host hop can still downgrade https to http and put the api key on the wire; matching scheme, host, and port keeps the credential on the listener it was issued for. * lance: bound catalog redirect hops so loops fail fast The custom redirect policy drops reqwest's default 10-hop limit, so a catalog that redirects back to itself would hold each call open until the request timeout. Keep the bound alongside the origin check. * lance: gofmt the redirect helpers --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-25 06:13:22 -07:00
Chris LuandDevin	39bdbc18db	admin: keep read-only sessions out of the filer /etc tree (#12057) ... * admin: keep read-only sessions out of the filer /etc tree The file browser read routes are open to any authenticated session, so a read-only login could download or view arbitrary filer paths. The filer stores the credential and cluster configuration under /etc (identity JSON with plaintext S3 secret keys, SSE-S3 keys, remote storage config), which let a view-only session recover live secrets that the user API redacts. Gate the read paths on the session role: read-only sessions get 403 for /api/files/download, /view, /properties, /metadata on /etc paths, the browser page redirects away, and recursive metadata exports skip the /etc subtree for read-only sessions. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * admin: only reject real .. path components in file browser paths path.Clean already resolves interior parent references, so any '..' surviving in the cleaned path can only be a leading escape from a relative input or part of a valid name. Matching on the substring rejected legitimate entries like /buckets/reports/report..csv. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * admin: cover the read-only /etc export exclusion with a test Pull the excluded-prefix selection into exportExcludedPrefixes so the role-dependent /etc skip is exercised directly — the cleanReadPath tests alone would still pass if the exclusion were dropped. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-25 06:06:21 -07:00
Chris Lu	403d51631b	s3api: stop echoing stored tags and internal attributes on GET/HEAD (#12053) ... * s3api: stop echoing stored tags and internal attributes on GET/HEAD setResponseHeaders copied every entry.Extended key into the response, only filtering xattr- and x-seaweedfs-. Stored object tags (X-Amz-Tagging-*) and the gateway's own attributes (Seaweed-X-Amz-Acl, -Owner, -Version-Id, -Object-Lock-Mode, ...) therefore leaked to any caller holding s3:GetObject, and to anonymous readers on public-read objects, without s3:GetObjectTagging or READ_ACP. Tag count, user metadata, storage class and the real S3 headers (x-amz-version-id, x-amz-object-lock-*) are unaffected. * s3api: match stored tag and internal attribute keys case-insensitively	2026-10-25 06:05:06 -07:00
Chris Lu	52d91008ac	ci: bump hosted runners to ubuntu-24.04 ... GitHub is retiring ubuntu-22.04; move all 94 runs-on pins across 40 workflows to ubuntu-24.04. rust_binaries_release: the arm64 cross step was jammy-era — noble keeps apt sources in deb822 ubuntu.sources (the sources.list edit was a silent no-op that left amd64 repos open to arm64 and 404'd) and ports suites must say noble. Pin deb822 stanzas to amd64 and add noble ports entries.	2026-10-24 23:07:28 -07:00
github-actions[bot]	ec8f979350	4.50 4.50	2026-10-25 03:09:42 +00:00
Chris LuandDevin	1a0485a43d	test: bound trino container memory + dump logs when it exits before ready (#12044) ... * test: dump trino container state and logs when it exits before ready When the Trino container dies during startup, waitForTrino only reported the docker exec error ("container is not running"), which gives no hint whether the JVM was OOM-killed, the image broke, or the config is bad. Capture docker inspect exit state and the container logs in the failure message so runner-side flakes are diagnosable from the test output alone. * test: cap trino container memory at 4g in catalog tests Trino's jvm.config sizes the heap at 80% of host RAM. On a loaded self-hosted runner the JVM attempts to claim most of system memory at boot and the container is killed ~15s after docker run — the flake seen in CI today. Bounding the container to 4g gives the JVM a fixed ~3.2g ceiling (observed usage is ~750mb), immune to host memory pressure. * test: keep docker errors in trino container diagnostics Report docker inspect/logs failures instead of dropping them, and include the last docker exec output in the container-dead fatal so a missing container still explains what happened. * test: detect trino container exit in readiness fallback When the polling loop times out while the container still appears alive, the trino --version fallback may itself fail because the container has since exited. Re-check its output for the exited-container signature and route through the diagnostic dump. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: dump trino container diagnostics on readiness timeout too The alive-but-never-ready timeout path reported only the last docker exec error, losing the container state and logs that the dead-container path already captures. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 20:00:06 -07:00
Chris LuandDevin	8c1a0a0cc2	filer: fix prefixed listing when the resume point sorts below the prefix (#12049) ... * test: prefixed directory listing with start below the prefix A listing constrained to prefix "item-b" and resumed from "item" must return item-b.versions. The seek-based stores return an empty page because the iterator lands before the prefix range and breaks on the first non-matching name. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: clamp prefixed-listing seek to the prefix range Seek-based stores (leveldb, leveldb2, leveldb3, rocksdb, tikv, etcd) started the scan at startFileName and stopped at the first key outside the prefix. A startFileName below the prefix landed the iterator on a non-matching sibling first, so the listing returned an empty page even though matching entries followed. etcd had the mirror problem: a range beginning below the prefix end leaked non-matching entries. Begin the seek at max(startFileName, prefix), which is what foundationdb and the sql stores already do. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer(hbase): honor startFileName lower bound in prefixed listings The scan starts at the prefix but only skipped the exact startFileName, so entries sorting before it were returned again on resume. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: keep paging a prefixed listing while matches can still follow The fallback for stores without native prefix support stopped paging as soon as the last scanned name reached the prefix, dropping prefix matches that continue on the next page. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: return the last delivered name from the prefixed-listing fallback The fallback for stores without native prefix support pages raw entries and filters client-side. Keep the scan cursor and the returned cursor separate: resuming must continue after the last delivered match, not the last scanned name, or matches still inside the current raw page are skipped on the next page. Also keep paging while the scan cursor is inside or below the prefix range instead of stopping at the first name that reaches the prefix. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer(hbase): start prefixed-listing scans at the resume key Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 19:59:19 -07:00
github-actions[bot]	0c32eab48e	docs: regenerate star history chart	2026-10-25 00:48:29 +00:00
Chris Lu	85470416cb	volume: overlap the local needle write with the replica fan-out (#10074) ... * volume: overlap the local needle write with the replica fan-out ReplicatedWrite wrote the needle to local disk and then fanned it out to the replicas serially, so a replicated write cost local + replicas. The needle is already fully buffered, so the two can run at once. Snapshot the fields the fan-out reads first (the local write stamps the volume TTL and updates needle flags, so the replica goroutine must not touch the shared needle), then run the local write and the replica fan-out concurrently and ack only after both. Where local disk and replica network are separate resources the write latency drops to max(local, replicas) instead of their sum; co-located on one disk the two writes contend, so this helps distributed placements, not single-host tests. * volume: return the local isUnchanged from the write goroutine * volume: trim overlap-write comments * volume-rs: overlap the replica fan-out with the local needle write * volume: test the overlapped local write and replica fan-out * volume: decode replica pairs only when replicas exist A single-copy or incoming replicate write never uses the decoded pair map, so a malformed Pairs blob only burned an error counter and a log line. Gate the decode on replicaCount. * volume: release the stalled replica on test failure paths An early t.Fatal left the httptest handler blocked on releaseReplica, so the deferred Close hung instead of reporting the failure. Release idempotently ahead of the server close. * test: mismatched If-None-Match serves the object over If-Modified-Since The precedence case asserted the old If-Modified-Since-first ordering and expected 304. The volume servers now evaluate If-None-Match first, so a non-matching tag must return 200 with the body even when If-Modified-Since would allow 304 (RFC 9110 13.1.3).	2026-10-24 15:16:16 -07:00
Chris LuandDevin	e442eda88e	s3: authorize POST policy uploads through the shared object check (#12047) ... * s3: authorize POST policy uploads through the shared object check doesPolicySignatureMatch gated a verified POST Object signature on the identity's legacy Write action, which cannot represent attached IAM policies, STS session policies, or key-scoped grants. Users authorized only by an attached policy (e.g. s3:PutObject on bucket/*) were denied with AccessDenied even though the equivalent PUT succeeds. The handler already runs AuthorizeObjectWrite against the real object key after the form's x-amz-* fields are promoted to headers, so the coarse check only shadowed the full evaluation. Signature verification now just returns the identity; authorization stays in one place. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: POST Object authorizes attached-policy identities, still denies others Cover the signature check and AuthorizeObjectWrite for an identity whose only write grant comes from an attached IAM policy, plus the deny leg for an identity whose attached policy omits s3:PutObject. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 15:09:34 -07:00
Chris Lu	216c1032e6	test: mismatched If-None-Match serves the object over If-Modified-Since (#12045) ... The precedence case asserted the old If-Modified-Since-first ordering and expected 304. The volume servers now evaluate If-None-Match first, so a non-matching tag must return 200 with the body even when If-Modified-Since would allow 304 (RFC 9110 13.1.3).	2026-10-24 15:06:19 -07:00
3 people	03f773f087	redis2: pipeline entry reads when listing a directory (#12042) ... ListDirectoryEntries fetched every member with its own GET, so a page of 1024 entries cost 1024 round trips. Fetch the values in pipelined batches of 128 as the loop reaches them and walk the replies in order. Each reply keeps its own error, so missing and expired members are cleaned up and skipped exactly as before, and an unreadable member still ends the listing at that name. Batching keeps a caller that lists with a huge limit and stops after a few entries from reading the whole directory. Co-authored-by: Patrick Schmidt <1310072+Woellchen@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com>	2026-10-24 15:04:37 -07:00
Junker der Provinz	f9dac64374	helm: remove filer.redirectOnRead, which weed filer does not accept (#12040) ... weed filer lost -redirectOnRead in 9b3109a5d (1.59), which also removed the chart value, but 2bed15f1d added it back. Setting it makes the filer exit with "flag provided but not defined".	2026-10-24 10:45:57 -07:00
4 people	7b43dec665	shell: run volume.configure.replication in parallel per volume server (#12041) ... * shell: run volume.configure.replication in parallel per volume server Volume servers were configured one after another, and each server's result overwrote the previous error, so a failure on one server was lost when a later server succeeded. Run one job per volume server, bounded by -maxParallelization like the other shell commands, and return the errors from all servers. VolumeConfigure unmounts and remounts the volume, so replicas of the same volume on different servers take turns instead of all going offline at once. Co-authored-by: NyaMisty <5344431+NyaMisty@users.noreply.github.com> * shell: test that volume.configure.replication overlaps servers up to the limit * shell: assert every server ran in the parallel configure-replication test Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: NyaMisty <5344431+NyaMisty@users.noreply.github.com> Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 10:29:38 -07:00
Chris LuandDevin	95d8a709e8	ci: run s3-go-tests on remote_storage changes (#12043) ... The S3 Remote Cache tests exercise the remote-storage S3 client, but the path filter did not include weed/remote_storage, so a tagging change there can regress the suite without running it on the PR. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 09:42:17 -07:00
Chris LuandDevin	e432b75822	remote_storage/s3: send only real object tags to the remote (#12039) ... * remote_storage/s3: send only real object tags to the remote WriteFile and toTagging copied every entry.Extended attribute into the remote object's tag set, including Seaweed-internal keys such as Seaweed-X-Amz-Acl whose values are not legal S3 tag values. Remotes that validate x-amz-tagging now reject these uploads with InvalidTag. Only X-Amz-Tagging-* attributes are S3 tags: filter to that prefix, strip it, and encode through s3TaggingHeader so keys and values are properly escaped. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * remote_storage/s3: skip remote tagging call when the tag set is unchanged UpdateFileMetadata fired PutObjectTagging on any Extended diff, but most attribute changes are not tag changes. Compare the effective tag sets and return early when they match. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * remote_storage/s3: assert WriteFile errors in tag-header tests Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 09:41:01 -07:00
3 people	81c90cb446	filer.backup: retry a failed checkpoint read instead of replaying from the epoch (#12036) ... * filer.backup: retry a failed checkpoint read instead of replaying from the epoch Fixes #12035 * test: reset sink.local.directory in backup offset test cleanup The new test sets sink.local.directory on the shared viper but only reset the enabled flag, leaking a key that points at a deleted t.TempDir() to later tests in the package. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover the retried attempt re-reading the checkpoint The fake filer now fails only its first KvGet, and the test runs a second doFilerBackup attempt asserting the checkpoint is re-read and the attempt proceeds past the offset stage. LookupDirectoryEntry is stubbed so the SSE-S3 KEK probe returns a miss instead of retrying 10x2s. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 08:22:29 -07:00
Chris LuandDevin	4c2c0ff21c	s3: fix wildcard matcher skipping '*' targets in policy patterns (#12034) ... * s3: check pattern '*' before literal equality in matchWildcardString When the target string carried a '*' at the offset of the pattern's '*', the literal-equality case consumed it as a one-character match and no backtrack point was recorded, so the comparison returned false. A Deny Resource carrying a policy variable, or an Allow carrying NotResource, could then be sidestepped by an object key containing '*' at the aligned offset. * test: cover literal asterisks in wildcard match targets --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 08:20:22 -07:00
Chris Lu	1d2d9dfed2	test(fuse): raise mini/filer and mount readiness waits to 90s (#12037) ... Same startup-timing flake class as the Makefile waits: volume/filer readiness at 30s is marginal on loaded runners — observed failing on a github-hosted runner too (TestPOSIXCompliance + TestExternalPOSIXSuites). The process-death early return still reports real startup failures quickly.	2026-10-24 08:13:36 -07:00
4 people	c4d743a69f	s3: make ListObjectsV2 continuation tokens round-trip XML-unsafe keys (#11996) ... * s3: keep ListObjectsV2 continuation tokens round-tripping XML-unsafe keys A page ending on a key XML 1.0 cannot carry (for example one containing \x01) produced a NextContinuationToken from that raw key. The XML encoder replaced the control characters with U+FFFD, so a client resuming at the token restarted after the mangled name and skipped every key sorting between it and the real one. encoding-type=url encoded the keys but not the token, so it hit the same skip. Encode the token as base64url behind a prefix when the key is not valid XML text; keys that survive XML unchanged keep the bare key as the token, so their tokens are byte-identical to before. The handler decodes the prefix back to the raw key before listing. Fixes #11990 * s3: keep undecodable base64: list tokens resuming at the raw key * s3: round-trip ListObjectsV1 markers over XML-unsafe keys * s3: round-trip ListObjectVersions key markers over XML-unsafe keys * s3: round-trip ListMultipartUploads key markers over XML-unsafe keys * s3: decode only tokens whose key could not have stayed bare * s3: document and pin the encoded-wins rule for ambiguous list tokens A token like base64:AQ reads as both a legacy bare key and the encoded form of an XML-unsafe key. The encoded reading already won in code; state why and cover it so the choice is not reverted as a false simplification. * s3: keep request-marker echoes in the response's wire encoding The decoded marker/key-marker is a raw key; echoing it writes XML-unsafe bytes straight into the response, where the encoder turns them into U+FFFD. Re-encode each echo through encodeListContinuationToken so the client reads back the token it sent, and apply urlPathEscape for V1 when encoding-type=url is active, matching the listed keys. * s3: wire-encode the V1 NextMarker echo like the request marker * s3: pin resuming an XML-unsafe key past a sibling directory * s3: wire-encode ListMultipartUploads markers and keys under encoding-type=url Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: unescape url-encoded marker echoes on list resume Under encoding-type=url the V1 and ListMultipartUploads responses emit marker echoes through urlPathEscape, and clients pass them back verbatim. The incoming marker was used as the resume position without undoing that escaping, so resuming after a url-unsafe-but-XML-safe key like "a b" (marker "a%20b") restarted the listing at the literal escaped text. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: assert the full decoded V1 sequence in the marker round-trip test The test only checked the page count, so a regression that skipped one key while repeating another still passed. Requesting encoding-type=url keeps the XML-unsafe keys legible for the decoded comparison and exercises resuming with the escaped marker echo. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: only unescape markers in the canonical urlPathEscape form A url-encoded request marker like a%41 cannot be encoder output, since urlPathEscape never emits %41 for 'A'. Unescaping it anyway resumed at aA and skipped every key sorting between a%41 and aA, including a%42. Require the unescape to re-escape to the received marker so only verbatim echoes decode; anything else stays the raw key the client sent. * s3: pin a raw-key marker that looks like an escape sequence * s3: cover the marker-echo decode guard with unit cases * s3: wire-encode the ListObjectsV2 StartAfter echo under encoding-type=url AWS encodes Delimiter, Prefix, Key, and StartAfter in V2 responses when url encoding is requested, but StartAfter was echoed raw. The echo now takes urlPathEscape and the incoming start-after param runs through decodeMarkerEcho, so a client resuming from the emitted form lands on the raw key. * Merge master and reconcile ListMultipartUploads marker encoding Resolve the overlap with the merged encoding-type=url pass: keys and markers now go through the single post-pass so nothing is url-escaped twice. Markers are still token-encoded when the response is not url-encoded, and the key-marker decode checks the query-escaped wire form the endpoint actually emits. * s3: echo the decoded start-after under encoding-type=url The escaped echo of the raw input is not a fixpoint: a client resubmitting the echoed StartAfter decoded it to the literal escape text and resumed at the wrong position. * s3: keep url-mode multipart key-marker decodes off the token path Under encoding-type=url the emitted KeyMarker/NextKeyMarker are the url-escaped raw key, never a base64-prefixed token, so the request decode must unescape only. Running the token decode afterward misread a literal "base64:..." key as an encoded token and resumed at the decoded bytes, skipping the keys in between. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 07:18:10 -07:00
4 people	fcd28031e7	mq/kafka: wait for a filer to register instead of panicking on an empty list (#12019) ... * mq/kafka: wait for a filer to register instead of panicking on an empty list NewSeaweedMQBrokerHandler queried the masters for filers once, ~2 s after start, and handed an empty list to NewFilerClientAccessor, which panics. A fresh filer registers only after its empty-membership confirm window (10 s), so a gateway started with a fresh cluster crashed (Kafka Quick Test, kafka-gateway exit 2). Retry while the list is empty for up to 60 s, then return an error. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * mq/kafka: wait out a transient-empty broker listing too A fresh SMQ broker registers with the masters on the same announce delay as a filer, so the one-shot broker lookup could fail startup the same way. Generalize waitForFilers into waitForClusterNodes and bound both discoveries by the same window; the empty-broker check becomes unreachable and drops away. * mq/kafka: wait out the broker health check like node discovery A broker announces to the masters before it has acquired its balancer lock, so the gateway's three-attempt probe could still fail fast on a fresh cluster. Retry FindBrokerLeader for the same window the caller waits out empty listings. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 07:09:43 -07:00
Chris LuandDevin	34c7bd03ad	ci: clean up stale FUSE mounts before running tests (#12020) ... * ci(s3tables): tolerate stale FUSE mounts in disk diagnostics * ci(s3tables): bound df diagnostics with timeout, explain once A stale FUSE mount can also hang statfs, not just fail it, so wrap df in timeout 15; one comment at the first site covers all sixteen. * ci: tolerate stale FUSE mounts in df diagnostics elsewhere The same self-hosted runners serve the other workflow files; a bare df -h there can stall or fail a job the same way. * ci: unmount disconnected FUSE test mounts before jobs start * ci: skip FUSE cleanup when the variant checkout is skipped * ci: reclaim the crashed run's test dir once its stale FUSE mount is detached * ci: stop tracking python bytecode for the FUSE cleanup action The helper's __pycache__ slipped into the tree; ignore it so local test runs cannot re-add it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: run the FUSE cleanup action's unit tests on action changes test_cleanup.py was never wired into CI, so a regression in cleanup.py would ship silently. A path-filtered workflow runs it on every change under .github/actions/. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: match every FUSE test mount layout in the cleanup action The candidates list only matched seaweedfs_fuse_test_*/mount, so stale mounts left by fuse_failover and fuse_p2p runs (mountN leaves under seaweedfs_fuse_* dirs) and the posix-compliance workflow's fixed /tmp/seaweedfs/mount were never reclaimed and could still wedge df. * ci: run the FUSE mount cleanup in the fuse_failover and fuse_p2p workflows --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 07:07:55 -07:00
3 people	18f9d9a0b6	feat: Free Volume Deadband (M) for EC shard placement (#12029) ... * feat: Free Volume Deadband (M) for EC Place nodes and racks |ΔfreeSlots| ≤ M is a tie; remainder is liveShards. M=0 stays strict freeSlots. M is free volume slots (× dataShards). pickNodeInRackEligible and pickRackEligible share the rule. Worker plugin config free_slots_deadband, default 0. Not on the write path. * fix: anchor EC freeSlots deadband on class maxFree Pairwise |Δfree|≤M vs the current winner is not transitive: 120→110→100 at M=10 slots picked 100. After evenness, rank by maxFree, keep |free-maxFree|≤M, then min liveShards. Shared by node and rack. M=0 is still only maxFree then live. * worker: add free_slots_deadband to ErasureCodingTaskConfig * erasure_coding: persist free_slots_deadband through TaskPolicy ApplyMaintenanceConfigFromToml accepted the key but the admin save path round-trips config through ToTaskPolicy, so the value was dropped on reload; serialize it like the other EC settings. * erasure_coding: bound free_slots_deadband at its declared maximum The TOML loader and the plugin worker config only clamped negatives, so a value above the spec's 100000 could reach the int32 TaskPolicy field and wrap; cap at 100000 on both entry paths and declare MaxValue on the plugin field. * erasure_coding: document free_slots_deadband in scaffold admin.toml --------- Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com>	2026-10-24 07:06:46 -07:00
3 people	52e05ccda4	s3: accept a self-copy that changes the storage class (#12031) ... * s3: accept a self-copy that changes the storage class or encryption AWS changes an object's storage class or encryption by copying the object onto itself with the new x-amz-storage-class or SSE headers, and only rejects a self-copy that changes none of metadata, storage class, website redirect location or encryption. CopyObject rejected every self-copy without REPLACE metadata or tagging directives as InvalidRequest, so these requests failed. * s3: reject an invalid x-amz-storage-class on CopyObject A self-copy with a bad storage class used to fail as InvalidRequest; once such copies are admitted the value must be validated like PutObject does, or it would be stored as opaque metadata. * test: cover the invalid storage class rejection on self-copy * s3: only a storage class change admits a self-copy Admitting encryption changes on a self-copy is unsafe: the metadata-only path compares only KMS key IDs, so a new encryption context or bucket-key setting is advertised over the old data key, and SSE-KMS key rotation onto the same entry keeps the old data key. Count only a requested storage class that differs from the object's (STANDARD when none is stored) as a change, so a self-copy to the current class or with only encryption headers is rejected as before. Such a self-copy updates the object in place and has to keep its encryption; one that would re-encrypt it or uses SSE-KMS is rejected too. --------- Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com>	2026-10-24 07:06:20 -07:00
Chris LuandDevin	514c0c1e99	topology: stop a stale duplicate-vid report from re-quarantining the survivor (#12032) ... * topology: drop a duplicate-vid report whose own copy is already gone flagDuplicateVolumeId runs on a goroutine after registration, so by the time it executes the reporter's copy may have been unregistered (a finished move drops the losing copy). The surviving copies are then legitimate, yet the stale report still quarantined them and stored a phantom (node, vid) key for the departed reporter. Skip the marks when the reporter no longer serves the volume under its own layout key. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(topology): cover a stale duplicate report against a lone survivor The existing tests only flag the conflict before the losing copy is unregistered. Run the delayed duplicate check after the reporter's copy is already gone and assert nothing is quarantined and the survivor stays writable, so removing the stale-report guard fails the test outright instead of needing a lucky schedule. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(topology): cover the enforceQuarantine-driven release retry A failed writable push re-arms the record, and the next volume report drains that retry through enforceQuarantine itself - a different entry point than the pulse-driven pass the other tests exercise. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 07:05:46 -07:00
4 people	8841eb6c70	volume: check If-None-Match before If-Modified-Since and match ETag lists (#12027) ... * filer: match If-Match and If-None-Match against "*" and ETag lists checkPreconditions compared the whole header as one ETag, so "*", a list of ETags and a weak W/ tag in If-None-Match never matched (RFC 9110 13.1.1, 13.1.2). * volume: check If-None-Match before If-Modified-Since and match ETag lists A volume GET answered 304 from If-Modified-Since even when If-None-Match named a different ETag, and compared If-None-Match as one exact string, so "*", lists and weak tags never matched (RFC 9110 13.1.2, 13.2.2). * filer: split etag lists only outside quoted tags A comma inside a quoted opaque tag is a legal etagc, not a list separator; strings.Split cut such tags into fragments that CanonicalizeETag could turn into a false match. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: match repeated If-Match and If-None-Match header lines Header.Get reads only the first field line, so a match on a later repeated header was missed; join all field values like any other comma-separated list. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: match repeated If-None-Match header lines Header.Get reads only the first field line; join all field values so a match on a later repeated header is seen. Also covers a comma inside a quoted opaque tag. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: send the etag on a 304 like the filer does RFC 9110 15.4.5 lists ETag among the fields a 304 must carry when a 200 would send it; SetEtag now runs before the notModified check. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: mirror the conditional-GET fixes in the rust server not_modified_response now checks If-None-Match before If-Modified-Since, matches "*", weak tags, quoted commas and repeated field lines through the same list rules, and sends the etag on a 304. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: treat an empty If-None-Match like the Go server does * filer: ignore empty elements in etag precondition lists * volume: ignore empty elements in etag lists like the filer * filer: let a present-but-empty precondition line suppress the date check * volume: gate If-Modified-Since on a present If-None-Match line * volume: gate If-Modified-Since on header presence like the Go server --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chris.lu@gmail.com>	2026-10-24 07:05:29 -07:00
Chris LuandDevin	59cccc788c	master: read the writable-volumes cap from master.toml (#12009) ... * master: read the writable-volumes cap from master.toml -maxWritableVolumes becomes the master.max_writable_volumes config key, so the new master limit is set the same way as other master options instead of a flag wired into every command that embeds a master. * master: name the writable cap master.max_writable_volumes_per_layout The cap applies per volume layout (collection + replica placement + ttl + disk type), so the key names its scope instead of reading as a global cap. * master: keep an embedder's MaxWritableVolumes when the key is unset NewMasterServer overwrote option.MaxWritableVolumes with the viper value, which reads as 0 whenever master.toml omits the key, silently dropping a cap a Go caller supplied through MasterOption. Seed the default from the option instead so the field survives an absent key; an explicit config entry still wins. * master: drop stray nested-checkout gitlinks Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 07:04:45 -07:00
4 people	9e7fd2eb90	filer: match If-Match and If-None-Match against "*" and ETag lists (#12026) ... * filer: match If-Match and If-None-Match against "*" and ETag lists checkPreconditions compared the whole header as one ETag, so "*", a list of ETags and a weak W/ tag in If-None-Match never matched (RFC 9110 13.1.1, 13.1.2). * filer: split etag lists only outside quoted tags A comma inside a quoted opaque tag is a legal etagc, not a list separator; strings.Split cut such tags into fragments that CanonicalizeETag could turn into a false match. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: match repeated If-Match and If-None-Match header lines Header.Get reads only the first field line, so a match on a later repeated header was missed; join all field values like any other comma-separated list. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: ignore empty elements in etag precondition lists * filer: let a present-but-empty precondition line suppress the date check --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chris.lu@gmail.com>	2026-10-24 07:04:41 -07:00
Chris LuandDevin	d6a68468ad	volume: keep a below-minimum fsync batch open for stragglers (#12016) ... * storage: hold a below-minimum fsync batch open for stragglers When the durable-write queue drained, the worker flushed whatever it held, so low-concurrency fsync writes collapsed to one sync per request -- a seek plus a rotation on rotational media no matter the payload. Once the queue empties with the batch still below the configured minimum bytes and requests, the batch now stays open up to SyncBatchOption.Wait, armed once per batch so the added latency stays bounded. DiskLocation carries the option and defaults it; Store exposes a setter the server calls once at startup. * volume server: expose the fsync batch window as -fsync* flags -fsyncMinBatchBytes, -fsyncMinBatchRequests and -fsyncBatchWait carry the operator's batching floor into every disk's SyncBatchOption. -fsyncBatchWait=0 restores the previous flush-on-drain behavior. * server/mini: register the fsync batch window flags weed server and weed mini build their own VolumeServerOptions and left the three new fsync flag pointers nil, so startVolumeServer panicked dereferencing them at startup. * storage: bound the fsync batch window by Wait while the queue refills Once the straggler timer was armed, a steady trickle of queued writes kept taking the len(requests)>0 continue path, so the first write could wait past Wait until the size or count cap flushed the batch. Track the armed deadline and flush once it passes even with requests buffered. * storage: sync the straggler test on the worker taking the first write The fixed sleep let both writes queue together (passing under the old flush-on-drain behavior) or the straggler arrive after the window closed (flaky failure). Poll the drained queue instead so the second write can only land inside the armed window or a later batch. * volume server: default the fsync batch window when its options are unset weed server and weed mini build VolumeServerOptions without the fsync flag pointers, and outside embedders can too. Resolve the batch window from the defaults so startVolumeServer never dereferences nil options. * storage: drive the batch-window test through an injectable timer The window is armed by the worker now, so the straggler provably joins an open batch and expiry is fired by hand instead of racing a sleep. * seaweed-volume: accept -fsyncMinBatch/-fsyncBatchWait flags Same flags and defaults as weed volume so the Rust server's durable write batching can be tuned identically. * seaweed-volume: hold a below-minimum fsync batch open for stragglers Mirrors the Go worker's batchWait: once the queue drains under the minimum batch, the window stays open for stragglers and an expired window still bounds the batch while the queue refills. Paused-time coverage replaces any sleep-based assertion. * seaweed-volume: sync the straggler test on the armed batch window The test fired both writes together, so the straggler could already be queued when the worker drained — a regression that drops mid-window writes from the open batch would still pass. Wait until the worker arms the window before sending the last write (Go's test polls the drained channel for the same guarantee). Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * seaweed-volume: reject fsyncMinBatchBytes that does not fit usize --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 07:04:05 -07:00
4 people	b0e6f7a096	s3: allow up to 50 tags in a bucket tag set (#12030) ... * s3: allow up to 50 tags in a bucket tag set PutBucketTagging validated the tag set with the object tag rules, which allow 10 tags. AWS allows a bucket up to 50 tags, so tag sets of 11 to 50 tags were rejected with InvalidTag. Validate bucket tags against 50 and keep the limit of 10 for objects and S3 Tables. A tag set that repeats a key is rejected with InvalidTag, as AWS does, instead of keeping one of the values, so 51 tags with a repeated key no longer pass as 50. * s3: reject duplicate tag keys in PutObjectTagging The tag set collapses to a map, so a repeated key was silently kept once.\nMatch the bucket tagging check and answer InvalidTag instead. * s3: reject empty tag keys AWS requires a tag key to be at least one character; the shared\nvalidator only checked the 128-char upper bound. * s3: reject duplicate tag keys in the tagging header parser parseTagsHeader (the x-amz-tagging path used by CopyObject with TaggingDirective=REPLACE) silently kept the last value when a key repeated, while the PutObject and multipart header paths and the new XML checks return InvalidTag for the same input. A repeated key after URL decoding now fails the same way. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: validate x-amz-tagging tags on PutObject before storing the entry The header path only rejected duplicate keys, so an upload could store more than ten tags or keys and values that fail S3 tag rules. Run the parsed set through ValidateTags like the XML tagging handlers do. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: validate x-amz-tagging tags on multipart initiate ParseS3Metadata stored header tags with no tag-set validation, the same gap putToFiler had. Multipart uploads now reject oversize or malformed tag sets with InvalidTag. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: count tag length in characters, not bytes * s3: validate object tags before uploading chunks * s3: reuse parseTagsHeader and drop raw header from warnings --------- Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 07:04:01 -07:00
3 people	de02158212	filer: let NewChunkStreamReaderFromFiler fall back to the master on a cache miss (#11964) ... refreshUrls invalidates every cached location of a volume after a failed read and re-looks it up through the reader's lookup function. With the cache-only masterClient.LookupFileId, one failed read to one replica left the volume missing from the filer's cache until restart, so metadata-log replay skipped it. Use GetLookupFileIdFunction like the other read paths. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2026-10-24 07:03:47 -07:00
Chris LuandDevin	8844d9823f	iam/oidc: wrap the jti replay cache in its own type (#12017) ... * iam/oidc: wrap the jti replay cache in its own type The replay-protection state lived as ten fields and nine methods spread across OIDCProvider. jtiReplayCache now owns the store, count, bound, sweep scheduling, and cleanup worker; the provider keeps a single jti field and delegates configure, consume, stop, adopt, and merge. * iam/oidc: lazily allocate the jti replay cache A zero-value OIDCProvider left jti nil, so Initialize, single-use validation, Shutdown, and the replay-state handoffs panicked on a nil cache. Allocate the cache through a sync.Once-guarded getter instead. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 07:03:42 -07:00
Chris LuandDevin	3e5aa8c3b3	topology: mark replica quarantined before recording the pair (#11983) ... * topology: serialize quarantine records with their layout marks A quarantine publishes a (node, vid) record and marks the replica read-only at the layout level; release and heartbeat flag updates must observe that pair as one state. Store-then-mark let a concurrent release delete the record before the mark landed, stranding the surviving replica read-only; mark-then-store let a heartbeat clear the mark before the record was visible, returning a conflicting replica to the writable list. quarantineLock now covers every record<->mark pair: flagDuplicateVolumeId's quarantine, releaseResolvedQuarantine's undo, the isQuarantined-gated marks on registration, and the quarantine-folded read-only flag writes on heartbeat paths. Remote server pushes stay outside the lock; a push that lands after the record was released is undone, and a failed release push re-adds the record for a later retry. The release test now waits for in-flight async flags to settle instead of asserting on a single read. * topology: fold quarantine into RegisterVolume's flag rebuild RegisterVolume rewrites every replica's location read-only flag from what its node last reported, and reported flags deliberately exclude quarantine. Any registration of a shared vid -- a reconnected replica, a new copy, the self-heal and delta-heartbeat re-register paths -- therefore cleared the quarantine mark on every replica in the location, while the gate afterwards only re-marked the registering node. Two call sites had no gate at all. RegisterVolume now takes an isQuarantined predicate and evaluates it under accessLock for every replica it touches, so the mark is never dropped in the first place: a release unmarks after the rebuild, and a quarantine mark re-applies on the next rebuild, with no window between. * topology: keep quarantine when a vacuum commit re-adds a replica SetVolumeAvailable re-adds the replica into the location list; a re-added slot starts with a cleared read-only flag, so a quarantined replica came back flag-free and the next ensureCorrectWritables could return a conflicting volume to the writable list. Pass the quarantine state in and set the flag back under the same accessLock, with the caller holding quarantineLock so release cannot interleave between the check and the mark. * topology: merge quarantine records instead of overwriting them enforceQuarantine wrote back the state copy it loaded before its remote push, so a concurrent writer's enforced=true was lost: the server stayed read-only while the record said unenforced, and a later release skipped the writable push it owed. The write-back now reloads the record under quarantineLock and merges -- enforced wins from either side, reported is taken from the newer pre-enforcement observation only while no push has landed yet. All three compensating writable pushes had the same hole in reverse: a failed un-quarantine left the server read-only with no record to retry from, and release's restore blindly overwrote any record a fresh flag pass had created in between. They now go through restoreQuarantineRecord, which adds the record only when absent and otherwise sets enforced on the current one, preserving its reported flag. * topology: hold quarantineLock over registration's quarantine check-and-mark RegisterVolume reads the quarantine record and marks the replica read-only under accessLock, but the pair ran without quarantineLock. A release could push writable, watch the record get re-read, and then delete it: the replica stayed read-only with no record left to clear, and unchanged writable heartbeats skip SetReplicaReadOnlyFlag so the mark never repaired. Holding quarantineLock around the check-and-mark in all three production callers serializes it against the release clear-and-delete. * topology: serialize remote quarantine pushes behind a record recheck releaseResolvedQuarantine deletes the record under quarantineLock and sends the writable request after unlock. A conflict flagged while that request is in flight creates a fresh record and marks the replica read-only again, and the older writable request can still land last, leaving the server writable under an active quarantine. The reverse flap strands a stale read-only mark the same way. Route every QuarantineVolumeFn call through pushQuarantine, which holds quarantinePushLock across the remote call and re-checks the record under quarantineLock right before sending: a read-only push requires the pair to exist, a writable push requires it to be gone. A skipped push means the decision already moved on, so the final server state always follows the newest record. * topology: serialize quarantine pushes per replica, not globally A single quarantinePushLock held across the remote mark call let one slow volume server (up to the 10s client timeout) delay pushes for unrelated replicas. Keying the lock by (node, vid) keeps pushes to one replica ordered in record-decision order while pushes to others run in parallel. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: restore the layout read-only mark with a released record A failed writable push re-armed the quarantine record but left the replica marked writable in the layout, so PickForWrite could route to a replica the server still holds read-only. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: wait out async duplicate checks in the release-retry test TestFailedReleasePushRestoresQuarantineMark asserts layout marks around a release pass, but the two RegisterVolumeLayout calls each start a background flagDuplicateVolumeId that can still be mid-flight. A late mark could land after the final release and fail the writable assertion even though the code behaved correctly. Track the async passes in a WaitGroup and drain them before the explicit flag call so the rest of the test is deterministic. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: keep quarantine when a replica's layout key is unreadable * topology: keep a failed writable push's restore ordered under the push lock * topology: retry the quarantine release on every duplicate check * ci: isolate Java and S3 test servers on persistent runners * ci: drop the duplicated -ip flags in the lifecycle test server The bind flags were already on the command; the isolation pass added a second pair. * topology: retry a failed quarantine writable push on the next report * topology: retry pending quarantine releases off the leader pulse A failed writable push re-arms the quarantine record with a retry that only enforceQuarantine fires, and that runs only for volumes a heartbeat carries. Once the server has reported the replica read-only it drops out of the delta reports, and restoreQuarantineRecord keeps the stored flag matching the server's so the digests agree and no full list is requested - the pending release then never retries and the idle volume stays read-only forever. Run the same release pass from the leader's pulse loop so a pending retry no longer waits on a report. * topology: exercise the pulse-driven quarantine release retry --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 07:00:45 -07:00
3 people	5700aa5e91	s3: evaluate bucket CORS for path-style OPTIONS on the bucket root (#12028) ... * s3: evaluate bucket CORS for path-style OPTIONS on the bucket root A path-style preflight for /bucket (no trailing slash) missed the bucket OPTIONS route, whose template is /{bucket}/, and was answered by the global OPTIONS handler from the global AllowedOrigins instead of the bucket CORS configuration. Register the bucket OPTIONS route without a path matcher, and keep OPTIONS for the health probe paths on the service-level handler ahead of each path-style router. Fixes #12025 * test: drop the issue reference from the bucket-root preflight test comment --------- Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 06:58:59 -07:00
Chris LuandDevin	9281ed8007	master: never auto-admit raft peers on a single-node configuration (#12021) ... * master: never auto-admit raft peers on a single-node configuration A master started without cluster peers (peers=none, or no peers so the list holds only itself) declares a one-node cluster, yet any foreign master that registered over KeepConnected was admitted as a second voter. The resulting two-voter cluster cannot reach quorum when the foreign peer is unreachable, wedging a previously healthy single master. Gate raft admission on having been configured with cluster peers; masters started with a peer list keep the announce-to-join flow used to grow the cluster. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: source SingleMaster from -peers=none, not the peer count A master started with an empty peer list ends up with a one-peer map containing itself, and NewMasterServer inferred declared single-master mode from that count. The master still mints a raft cluster through the bootstrap path, but AdmitRaftPeer then ignores every later joiner, so a deployment that grows from one master to several can never form a quorum. isSingleMasterMode already spells out the declared case (-peers=none); populate MasterOption.SingleMaster from it instead. * master: cover clustered raft admission with a real in-memory leader The previous clustered test had no raft leader and asserted nothing, so an AdmitRaftPeer that always returned early still passed. Bootstrap a single-node hashicorp raft over the in-memory transport, wait for leadership, and assert the leader reports and admits a missing peer. * master: return the raft admission result and assert committed join AdmitRaftPeer ran AddVoter on a detached goroutine, so the only observable state was an in-flight marker deleted on success and failure alike, and the clustered test could never tell whether the join worked. Admission now hands back a result channel; the test runs a connected second in-memory node and asserts the peer sees itself a committed voter. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 06:58:16 -07:00
Chris LuandDevin	fd3b437e04	s3api: write the null version's object-lock metadata to the plain entry (#12033) ... * s3api: write the null version's lock metadata to the plain entry PutObjectRetention and PutObjectLegalHold with versionId=null loaded the pre-versioning entry but wrote the update under <key>.versions/v_null, so the acknowledged lock never bound the object it was read from. Resolve the entry and its write path in one helper that applies the same null-version rule as getSpecificObjectVersion. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: cover the null-version lock write path Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: persist object-lock metadata with an in-place entry update mkFile rebuilds the target entry from chunks alone, so a lock write on a plain (null-version) entry discarded attributes, inline content, and the remote reference. UpdateEntry keeps the whole fetched entry, matching how the tagging and ACL writers persist metadata. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 06:57:02 -07:00
Chris LuandDevin	689ea6ea3e	release: digest-bound provenance and SBOM attestations for container images (#12024) ... * release: emit digest-bound provenance and SBOM attestations Per-platform builds now record SLSA provenance at mode=max and a BuildKit SBOM, so the builder identity and resolved dependencies are captured instead of the empty predicate the 4.48 build record carried. After the multi-arch index is assembled and signed, the merge job publishes GitHub artifact attestations bound to the merged digest and a syft SPDX SBOM of the image, giving digest-pinning consumers a verifiable record on the exact index they pull. * release: keep unfixed findings in the trivy SARIF upload ignore-unfixed dropped every finding without an available patch, so the uploaded report understated the image's exposure. The scan stays report-only; only the completeness of what is uploaded changes. * release: generate and attest SBOMs per platform digest An index-wide syft scan only describes its default platform, so the SBOM attested to the merged multi-arch digest misrepresented every other manifest. Scan and attest each platform digest instead, which also drops the sbom-action upload defaults that raced release-asset permissions and artifact names across the merge matrix. * release: resolve per-platform index to its image manifest before syft/cosign push-by-digest records point at OCI indexes (image + attestation manifests), so scanning or attesting the index digest picks the runner's platform and fails outright for the other arches under set -e. Resolve the non-attestation manifest inside each index and bind the SBOM attestation to the image digest a same-platform pull of the merged index actually returns. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 06:47:24 -07:00
Chris LuandDevin	be081b49b7	notification: named webhook targets for per-path event routing (#11925) ... * notification: honor the config prefix in the multi-queue guard validateOneEnabledQueue checked "<name>.enabled" without the notification prefix, so the fatal-on-two-queues guard never saw the real keys and could never fire. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: support named webhook targets under [notification.webhook.<name>] The webhook queue only accepted one endpoint, so events could not be routed to different services by path prefix. Each enabled [notification.webhook.<name>] subsection now spins up its own target — own filter, http client, retry middleware and worker budget — published on a per-target topic so every matching endpoint gets each event exactly once. The flat [notification.webhook] section keeps working as the unnamed target, and named subsections alone now activate the queue. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: require enabled for the flat webhook target buildTargets included the flat section whenever an endpoint was set, so a disabled [notification.webhook] block — which the scaffold ships with a placeholder endpoint — would start receiving events as soon as any named target activated the queue. * notification: resolve named target enabled flags through config namedTargetPrefixes read the raw map value, which never sees environment variable overrides; GetBool resolves WEED_* env bindings, so a named target disabled in the file can now be switched on by WEED_NOTIFICATION_WEBHOOK_<NAME>_ENABLED. * notification: cancel the webhook queue only after all workers stop Each target's router goroutine deferred the shared cancel, so the first worker to exit tore down the queue context for every other target and the dead letter logger. * notification: isolate webhook targets behind per-target queues SendMessage published to each target's topic inline, so a subscriber channel that filled up blocked every later target and the filer's notification path with it. Each target now drains its own inbox through a forwarder goroutine; SendMessage fails fast on a full inbox, matching the "throw message" contract the filer already applies to errors. * notification: name the webhook target in failure and dead letter logs Every target dead-letters to the same topic, so without the name there is no way to tell which endpoint failed or needs a replay. * notification: assert delivery in the named targets test The test only exercised SendMessage; it now records requests on both endpoints and checks matching, non-matching, and unrouted keys reach the right servers exactly once. * notification: wait for the webhook router before accepting events * notification: apply backpressure instead of dropping a full webhook inbox * notification: bound queued and in-flight webhook deliveries per target * notification: release the pending token when shutdown races the enqueue A SendMessage that wins the pending select as the queue is canceling could block forever on a full, undrained inbox. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: stop waiting for a router that failed to start A router that exits before reaching Running() would leave initialize blocked on a context nothing had canceled yet. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: route webhook events past saturated targets A full pending reservation made SendMessage wait on the slow target before it ever reached the healthy ones after it, coupling every endpoint's latency to the most congested one. Targets that can take a reservation now get the event immediately; saturated targets defer to a second pass that still blocks for a slot, so backpressure is kept and no event is dropped. * notification: retry webhook deliveries before dead-lettering PoisonQueue must wrap the retry middleware: added after it, the poison middleware intercepts the first handler error, republishes to the dead letter topic, and returns nil — the retry chain never sees a failure, so max_retries/backoff settings were dead code and every delivery failed straight to dead letter on the first attempt. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 02:12:17 -07:00
3 people	fb3fb9dcb8	s3api: support s3:RequestObjectTagKeys in bucket policies (#11993) ... * s3api: support s3:RequestObjectTagKeys in bucket policies Condition operators accept the ForAllValues: and ForAnyValue: qualifiers, and s3:RequestObjectTagKeys holds the keys of the tags taken from X-Amz-Tagging wherever s3:RequestObjectTag/<key> is set. CopyObject no longer stores an empty-key tag for an empty segment in X-Amz-Tagging, so it keeps the key set a policy has checked. Fixes #7679 * s3: parse tagging header like the tag handlers, not url.ParseQuery url.ParseQuery stops at 10,000 parameters, so an X-Amz-Tagging header padded with empty & segments made requestObjectTags return nil: the copy handler still stored the denied tag while every s3:RequestObjectTag* condition saw nothing. Scan non-empty segments the way parseTagsHeader does so the policy sees the same tags the object gets. * test: gofmt s3tables test files after #11997 The format check fails on these files on master as merged, so every PR's S3 Tables Format Check goes red until they are re-formatted. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 02:04:39 -07:00
Chris LuandDevin	7a9a9f10af	ci: invoke pip via python -m in s3tests clone steps (#12010) ... * ci: invoke pip via python -m in s3tests clone steps The self-hosted runner reaches the Clone s3-tests step with the setup-python toolcache interpreter on PATH but no pip shim, so the bare 'pip install' exits 127. python -m pip resolves through the interpreter that is already on PATH, matching the helm_ci workflow. * ci: invoke pip via python3 -m in s3tests clone steps The adjacent fix_s3_tests_bucket_conflicts step already runs python3, so python3 is the interpreter guaranteed on the self-hosted PATH; a missing python alias would fail the same way the missing pip shim did. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-23 23:05:56 -07:00
5 people	ecd778a4a0	iceberg maintenance: a merged file with null row ids names no first_row_id (#11973) ... * iceberg maintenance: a merged file with null row ids names no first_row_id Compaction stamps a merged v3 data file with the _row_id of its first row as first_row_id (mergedFileFirstRowID). When the bin mixed rows that carry a materialized _row_id with rows that carry none (a data file with no first_row_id, as a v2 table's files are after an upgrade to v3, gives null ids), the null rows then inherit first_row_id + their position in the file: ids that other rows already hold, so two rows answer to the same row id. A merged file now names a first_row_id only when its footer shows no null _row_id in any row group. Otherwise the commit assigns the file a fresh range from the table's next-row-id: the null rows get new, unused ids and the materialized ones keep theirs (a materialized value overrides inheritance). Test: TestMergedFileFirstRowIDOnlyWithoutNullIDs ("materialized first, a null later" fails without the fix). * iceberg maintenance: name no first_row_id when the merged file has a null _row_id mergedFileFirstRowID stamped the merged file with its first row's id without checking later rows. A null _row_id in a merged file inherits first_row_id+position, an id another row already holds, so the commit produced two rows answering to the same row id. Check the footer's null counts for the _row_id column across row groups first: any null means the file is committed without a first_row_id, so its nulls take fresh ids from the table's next-row-id while materialized values still override inheritance. * shell: retry filer discovery until master knows a filer A shell started against a freshly booted cluster (weed mini, the s3tables/lance test harnesses) can list cluster nodes before the filer's first heartbeat lands on the master. The empty list left FilerAddress unset and the next command died with "dial tcp: missing address". Poll ListClusterNodes until a filer appears or the connect timeout expires. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: discover the filer on first use instead of at startup The startup retry loop made every weed shell invocation wait out the connect timeout on clusters that have no filer, even for commands that only talk to the master. Discovery now runs once, on the first command that actually needs a filer, keeping the retry-until-registered fix for the fresh-cluster race without stalling master-only commands. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: stop filer discovery retries on non-transient errors A master that keeps answering ListClusterNodes with a permanent error (permission denied, unimplemented) cannot gain a filer by retrying, so discovery returns instead of polling until the deadline. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-10-23 22:53:43 -07:00
Chris LuandDevin	6a34d82f05	shell: retry filer discovery until master knows a filer (#11989) ... * shell: discover the filer on first use instead of at startup The startup retry loop made every weed shell invocation wait out the connect timeout on clusters that have no filer, even for commands that only talk to the master. Discovery now runs once, on the first command that actually needs a filer, keeping the retry-until-registered fix for the fresh-cluster race without stalling master-only commands. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: stop filer discovery retries on non-transient errors A master that keeps answering ListClusterNodes with a permanent error (permission denied, unimplemented) cannot gain a filer by retrying, so discovery returns instead of polling until the deadline. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: explain an empty filer list when discovery times out * shell: retry filer discovery while the address is still empty Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-23 22:52:51 -07:00
dependabot[bot]anduser.email	b86b2f3cee	build(deps): bump github.com/go-chi/chi/v5 from 5.3.1 to 5.4.0 (#12004) ... Bumps [github.com/go-chi/chi/v5](https://github.com/go-chi/chi) from 5.3.1 to 5.4.0. - [Release notes](https://github.com/go-chi/chi/releases) - [Changelog](https://github.com/go-chi/chi/blob/master/CHANGELOG.md) - [Commits](https://github.com/go-chi/chi/compare/v5.3.1...v5.4.0) --- updated-dependencies: - dependency-name: github.com/go-chi/chi/v5 dependency-version: 5.4.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com>	2026-10-23 22:47:23 -07:00