mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-19 20:44:28 +02:00
d04d838ebdfe4571e5464be5a82a4b2e33cb80b4
15558
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
d04d838ebd |
ec: compare live idx NeedleId sets before encode (#11812)
* ec: compare live idx NeedleId sets before encode Refuse encode when replicas' live .idx NeedleId sets differ. Walk only live rows (skip deleted and Size<=0) and compare NeedleId, ignoring Size so MIME/flags drift does not abort a matching id set. Mark the volume read-only for encode and reject writes after that. * fix(ec): compare all replicas then pick capacity source Log matching live idx sets with "replicas are consistent" for the shell CI check. After readonly, compare every replica's live NeedleId set, then require the encode source to have FreeVolumeCount>=2. Empty live-idx vs non-empty refuses encode instead of skipping superblock stubs. Drop the FileCount rebind after the idx gate. * volume: mark readonly under the write lock so checked appends finish first * ec: fix readonly deadlock and restore worker-host source preference MarkVolumeReadonly held dataFileAccessLock while calling reopenIdxForWrite, which re-acquires the same mutex and deadlocks TestMarkVolumeReadonlyCanDelete. Split out reopenIdxForWriteLocked for callers already holding the lock, matching the existing Locked pattern. The worker task lost its worker-host source preference when the idx gate replaced the replica union. Late-bind the source to a replica on the worker host using only the checked location set, with no FileCount rebinding. Drop the now-unused helpers, trim comments added by the earlier commits, and rename tests whose names still claimed convergence while asserting refusal. --------- Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com> Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
10ce19cb17 |
test: poll for TopologyId convergence in multi-master failover tests
A restarted goraft master wipes its raft state and only learns the TopologyId after the leader re-admits it and replays the entry. That trails HTTP readiness by several seconds, so the one-shot check in assertTopologyIdConsistent raced the propagation (CI run 39199033687: TopologyId applied on the follower ~2s after the test gave up). |
||
|
Mohd Quamar TyagiandMohd Quamar Tyagi
|
e20640acb4 |
fix(admin): improve Total Volumes count contrast (#11817)
fix(admin): improve Total Volumes count contrast on dashboard Co-authored-by: Mohd Quamar Tyagi <Tyagiquamar@users.noreply.github.com> |
||
|
Chris Lu
|
8ac5133abe |
admin: default to port 12535, keep 23646 as a best-effort compat listener (#11816)
* admin: default to port 12535, keep 23646 as a best-effort compat listener The default admin HTTP port 23646 derives a worker gRPC port of 33646 (port + GrpcPortOffset), inside the Linux default ephemeral range (32768+). One of our own startup dials can claim 33646 before the admin binds it, killing startup. This is the same failure #10209 fixed for the test port allocator, still reachable via the mini default (#10838) and via standalone `weed admin` defaults. Move the default admin port to 12535 (gRPC 22535) everywhere, keeping the http+10000 convention so worker -admin derivation, the Rust worker and host:http.grpc addressing keep working: - weed admin -port, weed mini -admin.port - weed worker -admin, the plugin worker fallback, Rust worker and weed-lance-worker defaults - help text, README, helm values, docker compose, tests So old workers, bookmarks and scripts keep working, admin also serves the same handler and the worker gRPC server on the previous default (-port.legacy / -admin.port.legacy, default 23646, 0 disables). The legacy gRPC port binds best-effort: it lives in the ephemeral range, so a failed bind only warns instead of reintroducing the startup failure. weed mini reserves it early like the primary port and keeps it out of its own port relocation. grpc.Server tracks every listener handed to Serve, so the compat listener stops with the primary. Also warn when an explicitly configured admin port puts the derived gRPC port at or above the ephemeral floor, the other fix suggested in the issue. Supersedes the pinned-grpc approach in #11814: pinning -admin.port.grpc off the +10000 convention breaks external workers that derive the admin gRPC port from the HTTP address. ClickHouse/ClickHouse#115512 pins -admin.port on their side; this makes that pin unnecessary. Fixes #10838 * admin: let services win the legacy ports, close compat listener on failure Review findings on the compat listeners: - Do not put the legacy ports in mini's reservedPorts: an explicit -s3.port=23646 or -master.port.grpc=33646 must be honored, not rejected or stolen by the early reservation. Mini now checks the finalized port set and skips only the conflicting compat listener, so HTTP and gRPC compat degrade independently and the service keeps its port. - Also skip the legacy HTTP listener when a mini service owns the port: a wildcard-bound service can still leave loopback bindable, which would let the compat listener shadow it for loopback clients. - Close legacyServer's raw listener in the deferred cleanup: Serve may not have registered it yet when startAdminServer returns early. - Helm: add admin.legacyPort (default 23646) and expose the legacy http/grpc ports through the admin Service, container ports and NetworkPolicy so older workers resolved via the Service keep working. * helm: only pass -port.legacy when it differs from the binary default ct install deploys the released image with the new chart, and 4.48 does not know -port.legacy, so the admin pod crash-loops. Omitting the flag at the default value is behavior-identical on new binaries (their default is also 23646) and keeps older releases installable; legacyPort: 0 or a custom port still renders the flag. |
||
|
github-actions[bot]
|
d3ee817b2e | docs: regenerate star history chart | ||
|
|
863efb3d35 |
webdav: delete a folder with its contents (#11811)
* webdav: delete a folder with its contents * webdav: guard the recursive delete against stale and confined paths - propagate stat lookup errors instead of returning an error-bearing FileInfo: OpenFile and DELETE would otherwise take the recursive removeAll on a path whose type was never confirmed, wiping a directory on a transient filer error - refuse wrappedFs.Rename from or to the served root, matching the RemoveAll guard and webdav.Dir - normalize trailing slashes in removeAll so the delete names the entry itself, and refuse the root for every caller * webdav: keep file-only deletes shallow and refuse ancestor destinations - removeAll used by OpenFile and failed-upload cleanup now deletes non-recursively, so a folder swapped in after the stat check is not emptied by a pending file delete - MOVE and COPY whose destination contains the source are refused before the handler removes the destination, which would otherwise take the source down with it * webdav: also refuse a destination inside the source a MOVE or COPY into its own subtree recurses into the destination it just created; refuse it with the same guard instead of burning the recursion budget for a 500 * webdav: allow a shallow copy into the source subtree a COPY with Depth: 0 creates only the destination collection and never reads children, so it is safe inside the source; only deep copies and moves are refused there * webdav: refuse a copy/move whose source is unreadable * webdav: cover the missing-source refusal in the guard test * webdav: 500 on a source lookup failure, 404 only when absent --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
8cb42cd69f |
build(deps): bump github.com/seaweedfs/raft from 1.2.1 to 1.2.2 (#11815)
Brings in the raft snapshot-recovery fixes: a recovered follower returns to Follower so it can campaign when the leader goes away (masters on ephemeral -mdir no longer wedge leaderless), and currentTerm stays monotonic and persisted through recovery. |
||
|
Chris Lu
|
6a594da739 |
filer: free orphaned hard-link chunks and remote objects in delete sweeps (#11809)
* filer: free shared chunks when a folder delete drops the last hard link doBatchDeleteFolderMetaAndData skipped chunk collection for every hard-linked child, relying on maybeDeleteHardLinks to retire the shared inode record. When the delete removes the last link the record is gone and nobody else will ever free those chunks. After the link callback runs, check whether the record survived; if it did not, queue the entry's chunks for deletion. * filer: free orphaned hard-link chunks in the replayed delete sweep The subtree sweep on a replayed directory delete skipped every hard-linked child, so a link that was the inode's last owner leaked its shared chunks. Check the inode record after the store delete; when it is gone, queue the child's chunks like any other swept entry. * filer: mirror remote-object cleanup in the replayed delete sweep Children a lagging peer alone still holds under a replayed directory delete were dropped with plain store deletes: their remote objects leaked and lazy remote fetches could resurrect them. The sweep now records event-stamped tombstones and deletes swept non-directory children through maybeDeleteFromRemoteIfUnchanged, which carries the same stat guard the write-back daemon's processDeleteEvent applies plus a modification-time check — a delayed replay cannot erase an object another writer replaced, even a same-size one. A failed remote delete fails the replay so the event is retried instead of orphaning the object. The directory's own tombstone comes from the event pipeline at the event timestamp and covers the subtree. Metadata-only deletes and cross-cluster events keep the origin's semantics. * filerstore: return real errors from KvGet on MongoDB, ArangoDB, Elastic These stores mapped every lookup failure to ErrKvNotFound, so a database error looked identical to a missing key: hard-link records could appear gone while links still owned their chunks, and path timestamp reads could treat committed state as absent. Return the underlying error and keep ErrKvNotFound for genuinely missing keys. * filer: bind remote-object deletes to the observed object version A guarded stat-then-delete still lets a replacement landing between the two requests go down with the deleted object. Add an optional conditional-delete operation to remote storage clients — S3 If-Match on the live ETag, GCS generation match, Azure IfMatch — and use it everywhere a delete is guarded by a stored stamp: the write-back daemon's delete, rename-source, and pending-rename paths, and the replayed-delete sweep. Backends without it keep the guarded stat-then-delete. * filer: restamp the remote entry after a metadata-only remote update Changing Content-Encoding rewrites the object in place on S3 and advances its LastModified, while the recorded stamp keeps the old mtime. A version-checked delete would then mistake the object for a replacement and keep it forever. Re-stat after UpdateFileMetadata so the saved stamp describes the live object. * filer: mirror the remote-delete ownership checks in the replayed sweep The sweep deleted swept children's remote objects on the listing's snapshot alone. Two gaps: a remote-only entry that took the path since — one a commit record may not cover, such as a lazy fetch materialized from the remote — reads the object as its only copy, and an entry removed while its rename copy was pending leaves the object at the source key orphaned. The per-child delete now runs under the child's entry lock with the commit-time check repeated, keeps a remote-only entry that holds the path, and releases the pending rename's source under the same ownership and stamp conditions the write-back daemon's releasePendingSource applies. * remote.sync: restamp only the object the metadata update touched The live stat after UpdateFileMetadata was saved unconditionally, so a replacement written between the update and the stat stamped the entry with an object it never described — and a later delete would then pass the freshness guard on the replacement's data. The restamp now requires the live object to still match the stored stamp on size and ETag; a different object keeps the old stamp, which the delete guard reads as replaced and keeps. * filer: skip unbudgeted reader-cache test on 32-bit TestReaderCacheUnboundedWithoutBudget proves the reader cache has no implicit memory budget by keeping 96 concurrent 4 MiB downloads in flight (>256 MiB). On linux/386 the weed/filer test binary already runs near the 4 GiB address-space limit, so the burst reliably crashes it with "out of memory" mid-suite. * filer: bail out of a swept child lock instead of waiting on it Replay already holds the event's path locks when the sweep reaches a descendant. A local rename can hold that descendant while waiting on one of the replayed paths, so blocking on the child lock deadlocks both. The sweep now tries the lock and reports ErrLockContended — transient, so the replication retry replays the event once the local write settles. * remote: report the stamp a metadata update produced A metadata write can rewrite the object it touches — S3 changes Content-Encoding through an in-place copy, which can mint a new ETag — so restamping from a fresh stat can reject the very object the update produced, or describe a replacement that landed in between. UpdateFileMetadata now returns the RemoteEntry of the object it wrote (S3 from the copy result, GCS from the update attributes, Azure from a post-update stat); nil means the object was untouched and callers keep the stored stamp. * remote: restamp Azure from the metadata write's own etag |
||
|
|
eea3f232da |
iam: scope CreateServiceAccount's ParentUser by policy resource (#11736)
* iam: scope CreateServiceAccount's ParentUser by policy resource AuthorizeIamAction evaluated the iam:CreateServiceAccount grant against a hardcoded arn:aws:iam:::* resource, so the policy's Resource never had a say; #11218 then bound the target to the caller outright. Evaluate self-target actions against the target identity's ARN instead: a wildcard resource keeps the deliberate any-target grant, a scoped resource confines issuance to the named parent identity, anything else is denied. buildResourceARN gains the ARN pass-through buildS3ResourceArn already has, so iam: ARNs reach the policy engine intact on the attached-policies route. Closes #11735 * ci: retrigger after unrelated weed/storage metrics flake * iam: fold IAM-ARN matching into awsIAMMatch, drop debug test - remove zz_debug_test.go (local debugging leftover) - fold awsIAMMatchResource into awsIAMMatch: the arn:aws:iam:: value prefix selects case-sensitive matching, so no separate name is needed - trim comments --------- Co-authored-by: batonogov <batonogov@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
4a57432488 |
s3: list children of the directory a ListObjects marker names (#11805)
* s3: resume into the directory a listing marker names * s3: keep delimited marker resumes exclusive * s3: descend a marker-named directory without repeating its key * s3: emit marker-named dir objects but keep internal folders hidden |
||
|
Jay HemnaniandChris Lu
|
006d4150f0 |
webdav: create a folder under its own name (#11810)
* webdav: create a folder under its own name * webdav: reuse NormalizePath for the MKCOL entry split --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
ab39e3a51b |
Rust volume: stamp ExpireAtSec into the live volume_info on .vif persist (#11813)
The worker fix reads the source's TTL destroy time through ReadVolumeFileStatus, which returns the in-memory volume_info. Go's SaveVolumeInfo stamps volumeInfo.ExpireAtSec before writing; Rust's save_vif wrote it only to the serialized file, so a Rust source kept reporting the stale creation-time value and the EC .vif would carry a wrong expiry. Stamp the live volume_info first, mirroring Go and save_volume_info. |
||
|
Chris Lu
|
8c55425d55 |
s3tables: tighten legacy OIDC subject re-anchoring (#11808)
* sts: recognize issuer-scoped parent user ids
ComputeParentUser output has a fixed shape (base64url SHA-256); let
callers distinguish it from bare subject strings.
* s3tables: keep legacy owner re-anchoring off local and scoped account ids
Re-anchoring a stored bare subject under the single trusted issuer must
only apply to strings that could actually be a legacy OIDC subject. A
stored owner naming the reserved or a configured local account, or an
already issuer-scoped id, can otherwise be claimed by any caller whose
token asserts that string as sub.
* s3tables: re-anchor legacy policy principals in map and array forms
A stored bare subject can appear as {"AWS": sub} or inside a principal
list, not only as a top-level string; re-anchoring only the string form
left legacy Deny statements unable to match the caller.
* s3tables: keep built-in legacy exclusions independent of the account checker
A missing local-account predicate must not disable re-anchoring entirely:
the reserved accounts, ARN-shaped values, and scoped-id shapes are never
OIDC subjects regardless, while legacy Deny statements for plain subjects
still need to resolve so a denied caller cannot fall through to an
identity grant.
* s3tables: exercise legacy Deny against an owner grant
The previous case denied the caller regardless of whether the statement
principal matched; make the caller the owner so a missed match shows.
Cover the array Deny form alongside the AWS map.
|
||
|
Chris Lu
|
88d9926b35 |
s3api: recheck suspended-version condition under the lock when the filer ignores condition_alt_key (#11806)
* s3api: recheck condition under lock when routed filer ignores condition_alt_key A filer that predates condition_alt_key silently drops the field and evaluates a suspended-versioning condition against the empty null slot alone, so a matching If-Match still gets a false 412. Recheck under the object lock instead of trusting that response: the lock path resolves the same entry the read path serves. Genuine precondition failures still return 412 from the recheck. * s3api: cover suspended If-Match against a filer ignoring condition_alt_key * s3api: resend against the resolved entry instead of the lock-path recheck Rechecking under the gateway object lock does not serialize against routed writers, which commit under the filer's path lock; a concurrent routed PUT could land between the recheck and the create and be overwritten with a stale precondition. Resend the same transaction with condition_key set to the resolved version file so the condition and the write stay under the same filer lock. When the resolved entry is the null slot, a delete marker, or unresolvable, the resend targets the object path and fails closed. |
||
|
Chris Lu
|
e596be10f0 |
s3api: evaluate object-tag policy conditions on delete, ACL and lock operations (#11807)
* s3api: resolve the target version in tag-condition policy lookups * s3api: apply tag-conditioned policy to object deletes * s3api: apply tag-conditioned policy to object ACL reads and writes * s3api: apply tag-conditioned policy to retention and legal hold * s3api: cover tag-conditioned delete, ACL and object-lock rechecks * s3api: let tag-conditioned grants redecide denied batch deletes * s3api: resolve the tagged version in delete policy tests |
||
|
Chris Lu
|
2ff7888546 |
filer: honor a storage rule's rack/data-center constraint instead of falling back (#11804)
filer: stop widening a storage rule's location on assign failure |
||
|
Nabil RagabandClaude Opus 5.5
|
6ab6926c94 |
s3: read chunks from HTTPS volume servers with the global HTTP client (#11800)
readChunkRange sent the volume lookup's http:// URL through a bare http.Client, so with [https.client]/[https.volume] configured every read of an SSE-KMS object larger than one chunk (and every ranged read through this path) failed with "client sent an HTTP request to an HTTPS server" and the GET returned 500. Normalise the URL's scheme and use util_http's global client, which carries the [https.client] TLS configuration, as the other volume reads already do. Drop the now-unused volumeServerHTTPClient. Fixes #11799 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris Lu
|
e6a0534e3b |
s3api: make ambiguous routed-PUT recovery atomic and stop transaction replay (#11787)
* filer_pb: map PRECONDITION_FAILED to a sentinel error A conditional CreateEntry that the owner's path lock refuses comes back as FilerError_PRECONDITION_FAILED; mapping it to ErrPreconditionFailed lets the S3 gateway tell a rejected precondition apart from a transport failure and report 412 instead of retrying the write. * s3api: make the ambiguous-PUT recovery create conditional under the lock createAfterAmbiguousRoute verified state and then issued an unconditional create, so a write landing between the lookup and the create was still clobbered by the stale re-commit (issue #11631). The recovery create now carries a WriteCondition — IF_NOT_EXISTS for a proven-absent entry, IF_ENTRY_EQUAL for a directory it may need to write onto — which the owner filer evaluates under its per-path lock, making the check and the write atomic. A refused precondition loops back to re-read the state instead of overwriting whatever won the race. * s3api: do not fail over a routed transaction past a possibly-sent request withFilerClientFailover retried the callback on the next filer on any transport error, so an ObjectTransaction whose response was lost could be re-applied over a write that committed in between (issue #11631). A gRPC error cannot distinguish "never sent" from "committed", but channel readiness can: a non-replayable callback now runs only on a connection that just reached READY, and failover continues only over channels that never came up — the one failure mode that provably sent nothing. * s3api: run ambiguous-route recovery for empty PUTs too The recovery was gated on a nonempty chunk list, so a zero-byte PUT whose routed transaction lost its response still fell back to an unconditional create that could overwrite a newer write and delete its chunks. Chunk-count mismatch on an empty upload list is enough to distinguish a superseding entry from this PUT's — there is nothing to save by skipping the check. * s3api: keep conditional recovery creates off the replay path The conditional create in the ambiguous-PUT fallback rode the replayable WithFilerClient failover, so a committed create whose response was lost could be re-sent to another filer — where a delete in between would let IF_NOT_EXISTS pass again and resurrect an entry pointing at reclaimed chunks. Conditional creates now run through the non-replayable failover; an ambiguous result falls through to confirmCreateLanded, which resolves the committed entry rather than resending. * s3api: do not mark a filer unhealthy for an answered request A reply that reached the callback — including an application-level failure like a refused write precondition — proves the filer is alive. Checking received before the health bookkeeping keeps definitive application errors from tripping the failure circuit or flagging a live routed owner as unreachable. * s3api: do not claim an empty stored entry as an ambiguous PUT's own Two chunkless entries compare equal under sameFileChunks, so an older empty object satisfies "the route landed" even though it predates this write — recovery would return success while leaving the request's Content-Type and metadata unapplied. A landed PUT stores the entry exactly as built, so require mime/size/md5/content and the extended attributes to match before claiming it; anything else stays uncertain and the client retries. * s3api: prove ambiguous empty PUTs by write id and pin recovery to the owner - Conditional recovery creates are sent to the routed owner alone instead of failing over: the condition only serializes with routed writes under the owner's per-path lock, and another filer's forwarding is only as correct as its ring view. An unreachable owner now yields an uncertain result rather than a create under a different serialization boundary. - A proven-absent path no longer satisfies an empty, chunkless PUT: the routed commit may have landed and been deleted, and there are no chunks to verify, so IF_NOT_EXISTS could resurrect a removed object. Recovery returns uncertain for the client to retry. - Chunkless identity moves from metadata equality to a per-write id (x-seaweedfs-write-id) stamped into Extended: an identical older object can match mime/md5/content but not the id. Entries without this write's id are not claimed, and confirmCreateLanded can now positively confirm a committed empty object instead of always reporting failure. |
||
|
Chris Lu
|
e6633146d8 |
filer: shrink zip spool threshold in tests to fix 32-bit OOM (#11801)
* filer: shrink zip spool threshold in tests to fix 32-bit OOM TestSpoolZipReaderTempFile and TestSpoolZipReaderLimit each allocate ~33-70 MiB to cross the 32 MiB in-memory spool threshold. On linux/386 the weed/filer test binary already runs near the 4 GiB address-space limit, so these allocations reliably crash it with "out of memory" mid-suite. Make zipSpoolMemoryLimit a var and drop it to 1 KiB in those tests, which exercises the same memory-vs-temp-file spool paths with KiB-sized buffers. * s3 lifecycle: update stale runShard and WalkerFunc call sites in tests Recent master changes added a walks param to runShard (#11797) and made WalkerFunc take []int, but replay_view_test.go and run_bounded_test.go still use the old signatures, breaking go vet and test builds on merge. |
||
|
Chris Lu
|
fe8221c453 |
fs.meta.load: wait for in-flight entry creation before returning (#11776)
* fs.meta.load: wait for in-flight entry creation before returning On io.EOF and error paths the command returned while CreateEntry goroutines were still running, so trailing entries could be silently dropped once the shell exited. Wait for them on every exit path and guard the shared error with a mutex. * fs.meta.load: surface worker errors on read/unmarshal failure paths The ReadFull and proto.Unmarshal error paths waited for in-flight CreateEntry workers but returned only the local error, hiding a failed create from the caller. Check loadErr after wg.Wait() like the other exit paths so the worker error is reported. Extract the load loop into loadEntries so the concurrency and error propagation can be exercised by unit tests, and add regression tests covering delayed creates at EOF and on corrupt input plus worker error reporting. |
||
|
github-actions[bot]
|
e32d8ca2a6 | docs: regenerate star history chart | ||
|
Chris Lu
|
602ddb31f2 |
s3: count stored versions as bucket contents on delete (#11775)
* s3: count stored versions as bucket contents A <key>.versions directory holding version entries is user data; skipping it let bucketHasUserObjects report an empty bucket while the recursive delete dropped every stored version. * test: rename stale reserved-folders case in bucket delete test .versions directories are now descended into, so the case only holds because the folder is empty. |
||
|
Chris Lu
|
44bb697495 |
s3api: hydrate runtime policies on propagated IAM updates (#11784)
* s3api: hydrate runtime policies on pushed identity upserts PutIdentity propagation (used on every IAM write when the credential store has a master client) carries the persisted identity proto. That proto records inline policies only as flattened legacy Actions; their runtime __inline_policy__ names live in the policy store and were dropped by the merge. For the interval until the next full reload, an identity whose policy denied e.g. DeleteObject routed to legacy CanDo, where "Write:bucket" authorizes deletes. Hydrate the pushed identity the same way a full load does, so the merged state always carries its runtime policy names and documents. * s3api: hydrate runtime policies on pushed group upserts PutGroup propagation has the same gap as identity pushes: the wire proto does not carry the group's runtime __inline_group_policy__ names, so members' attached-policy check lost them until the next reload. * s3api: attach only runtime inline policy names on pushes Listing the pushed identity's or group's own inline policies is enough to keep it on the attached-policy route; a whole-store hydration per pushed proto is not. Pushes now never write iam.policies, so a stale read can no longer overwrite a newer committed policy document. * s3api: carry inline policy documents on pushed upserts Attaching only runtime policy names left a window where a pushed identity or group referenced a document the local cache had never loaded, denying allowed requests until the next reload. Fetch the pushed entity own inline documents and publish them with the update; reads stay scoped to the pushed entity rather than the whole store, and inline runtime names are only ever populated from this store so a read cannot install a document older than the committed one. Resync the IAM manager only when a push actually installs documents. * s3api: order pushed inline policy reads against their installs A pushed UpsertIdentity or PutGroup read inline policy documents from the credential store and installed them without holding the ordering guard that full reloads already take, so a delayed push could overwrite a newer committed document with its stale read and restore a revoked grant until the next update or reload. Pushes now hold reloadMu across the store read and the install, the same snapshot-through-commit serialization the credential-manager load uses. MergeS3ApiConfiguration also applies config.Policies to the live iam.policies at commit instead of swapping in its earlier snapshot, so a policy installed while a merge prepared (a pushed inline document or a PutPolicy update) is no longer discarded. |
||
|
|
74ed12d6b3 |
s3 lifecycle: replay only the rules that fit the retention window (#11789)
* s3 lifecycle: replay only the rules that fit the retention window runShard handed drainShardEvents the full snapshot, so rules the retention window assigns to the walk were also matched during meta-log replay. Their matches are not due yet, and drainShardEvents freezes the cursor at the first event with a not-yet-due match. With a 3-day and a 30-day expiration rule on one bucket and RetentionWindow of 8 days, every shard's cursor stopped at the first 30-day object's PUT and stayed there until that object came due. Each pass then re-read the log from that point and re-dispatched every 3-day expiration already handled, which surfaces as NOOP_RESOLVED dispatches, a growing cursor_lag_max, and passes that still report status=ok. Snapshot.ReplayView drops the replay-kind actions promoted to the walk and keeps every other action as compiled, and the drain uses it. Promoted rules are served by the walk as before. RulesForShard, PromotedHash and ReplayView share one isPromoted predicate. With RetentionWindow unset no active rule is promoted, so behaviour without a configured window is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: scale the replay-view test with DaysToDuration The test took its retention window from DaysToDuration but dated its PUTs in hours, so under the s3tests build tag (one lifecycle day = 10s) every event fell before the seeded cursor and replay never ran. Express the event ages in DaysToDuration too, so the test exercises replay at both scales. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: rehabilitate scan_only mode in the replay drain view ReplayView shared the base CompiledAction pointers, so a replay-kind rule locked into ModeScanOnly by a prior compile stayed invisible to router.Route's event_driven gate even when the runtime retention window kept it in the replay partition - and it is absent from the walk view too, so the rule would be served by neither. Clone non-promoted replay-kind actions the way RulesForShard does (Mode forced to event_driven), reactivating the scan_only case whose bootstrap rendezvous already completed, while preserving the base active bit for genuinely pre-bootstrap actions. Disabled actions are excluded for parity with the other views, and DESIGN.md now documents the view. * s3 lifecycle: keep drain activation observable, require walker with retention ReplayView now shares pointers for event-driven actions so a mid-pass MarkActive flip remains visible to the drain; only a scan_only-locked action is cloned to event_driven, and reactivated only when its bootstrap rendezvous completed (tracked via a new bootstrapComplete field on CompiledAction). validate() rejects RetentionWindow without Walker: promoted rules would lose their only dispatch path while the drain cursor still advanced past their events. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
443ae1cd83 |
s3 lifecycle: share daily bucket walk and prune listings (#11797)
* s3 lifecycle: share the daily bucket walk across shards The daily pass invoked the walker once per shard, and every shard's walk listed the full bucket tree while keeping only its own 1/16 of the entries — 16 full listings per pass by default (#11791). Run now groups the shards by the walk branch they'll take (recovery vs steady, using the same cursor data and hash checks runShard applies) and starts one shared walk per group, so a pass lists each bucket at most twice. WalkerFunc and WalkBuckets take a shard set instead of a single shard. runShard keeps its own decision and waits on its group's walk; a shard whose freshly-loaded cursor disagrees with the plan falls back to a per-shard walk, so mid-pass cursor rewrites can't under-cover a shard. * s3 lifecycle: skip the .versions/ pre-pass on non-versioned buckets walkBucketTree listed every directory twice — once looking for .versions/ dirs, once for regular entries. On a bucket that never had versioning enabled the first pass can only find coincidentally-named user folders, so the extra listing is pure cost (#11791). FilerListFunc now takes the walk view and consults Snapshot.BucketVersioned; non-versioned buckets skip pass 1 entirely. A *-suffixed .versions user folder on such a bucket is recursed into as a regular directory so real objects inside still surface; a nil view keeps the two-pass behavior. * s3 lifecycle: prune walk directories to active rule prefixes The daily walk listed every directory under a bucket even when no rule prefix could reach it. Collect the active rule prefixes of each bucket from the walk view (new Snapshot.BucketPrefixes) and skip recursing into directories whose keys can never match; a bucket with no active actions is skipped entirely, saving the listing. .uploads stays in scope regardless of prefixes because AbortMPU rules match DestKey, and .versions expansion is pruned only conservatively (the logical key is a string-prefix of the dir key). * s3 lifecycle: join planned walks before Run returns planWalks starts a shared walk goroutine per group, but a shard whose authoritative cursor load then fails exits runShard without waiting on it. Run returned while the walk still listed or dispatched deletes against connections the caller closes on return. Join every planned walk after the shard waitgroup. * s3 lifecycle: treat object-locked buckets as versioned in the walk Object Lock requires versioning and every writer sets both extended attributes, but IsBucketVersioned only read ExtVersioningKey. A locked bucket missing that attribute would have its .versions dirs walked as ordinary folders, hiding version siblings from lifecycle. Classify ExtObjectLockEnabledKey as versioned too. |
||
|
Chris Lu
|
bc29ffbfbd |
helm: apply placement values to the chart's hook Jobs (#11795)
* helm: honor placement values in the bucket creation hook The post-install/post-upgrade bucket Job ignored nodeSelector, tolerations and affinity, so it could land on nodes the operator kept clear, or fail to schedule at all on a fully tainted cluster. The hook now accepts nodeSelector, tolerations and affinity under each createBucketsHook (filer.s3, s3, allInOne.s3), read from the same values section that supplies createBuckets. Unset nodeSelector and tolerations fall back to the workload the hook already borrows its security context from; affinity is explicit-only since the anti-affinity rules of a long-lived server do not suit a one-off Job. * helm: honor placement values in the volume resize hook The pre-install/pre-upgrade resize Job inherited nothing from the volume server, so on a cluster whose storage nodes are all tainted the Job could not be scheduled and the upgrade timed out. The hook now falls back to the volume server's nodeSelector and tolerations, and accepts explicit nodeSelector, tolerations and affinity under volume.resizeHook. Affinity stays explicit-only: the default podAntiAffinity against component=volume would keep the Job off every node already running a volume pod, which is most of them during an upgrade. Placement strings are rendered through tpl with $volumeName bound, same as the StatefulSet. |
||
|
Chris Lu
|
01d8160323 |
mini/server/filer: let embedded s3/sftp gateways use their own -dataCenter (#11798)
* mini: stop forcing -dataCenter onto the embedded S3 gateway -dataCenter names the embedded volume server's data center, but it was also copied into the S3 gateway options, overwriting an explicit -s3.dataCenter and making every S3 write carry an explicit dataCenter pin in its assign request. That pin outranks fs.configure dataCenter rules, so a mini node in one DC could never steer S3 writes to volume servers in another. With the copy removed, assign requests carry no DC by default: the filer's own -dataCenter still provides the write fallback, and fs.configure rules now apply to S3 traffic. -s3.dataCenter still pins reads and writes when set explicitly. * server: give embedded s3/sftp gateways their own -dataCenter flags -dataCenter names the embedded volume server's data center, but it was also copied into the S3 and SFTP gateway options, and neither gateway had a flag of its own to override it. The injected DC lands in every S3 assign request as an explicit pin, outranking fs.configure dataCenter rules. Add -s3.dataCenter and -sftp.dataCenter (same names as weed filer) and stop copying -dataCenter into them, so gateway writes go through the filer's rules and defaults unless a gateway DC is set explicitly. * filer: stop defaulting embedded gateway dataCenter to -dataCenter The embedded S3 and SFTP gateways inherited -dataCenter whenever their own flag was unset. That made every S3 assign request carry the DC as an explicit pin, outranking fs.configure dataCenter rules, and left no way to let rules steer writes while -dataCenter was set. Let -s3.dataCenter and -sftp.dataCenter stay empty unless set, matching standalone weed s3 and weed sftp. The filer's own -dataCenter still provides the write fallback. |
||
|
Chris LuandDevin
|
8d9c7e76a5 |
sftp: enforce file ACLs on rename targets (#11770)
* sftp: check delete permission on an existing rename target renameEntry authorized only the source path and the target directory, so renaming over a file with a more restrictive ACL destroyed it even though Remove on the same file was denied. * sftp: deny a rename that would strand a carved-out ACL A rename moves data without moving its path-based ACL: renaming an ancestor directory left a more specific read-only entry behind, so the relocated file fell under a permissive ancestor ACL and became writable. Deny the rename when a configured ACL under the source would grant less write/delete access than the relocated path would get. * sftp: cover inherited and destination-side ACLs on rename The rename ACL check compared only configured paths under the source, so it missed restrictions inherited from ancestors above the source and deeper grants configured under the destination. It now compares the moved root plus every configured path under either tree mapped to its source counterpart, and skips paths whose entries do not actually move. The target delete check no longer depends on an existence lookup, so a protected target created during a rename cannot be overwritten; a no-op rename skips it. Root bypasses the comparison like every other check. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * sftp: fail closed on rename ACL lookups and restore new-name renames - checkRenameKeepsAcl returns lookup errors instead of treating them as a missing entry, so a transient filer failure cannot skip the ACL comparison; it also compares read and list gains, not only write and delete, and evaluates each pair with the moved entry's own IsDirectory. - The target delete check now distinguishes an existing target (still requires delete) from a fresh name (no delete needed) while still enforcing a configured ACL on the target name itself, so a concurrent create cannot slip a protected target under the rename. * filer: add fail_if_target_exists option to AtomicRenameEntry Callers that authorize a rename while the target is absent cannot see a target created in the gap before the move runs. The flag makes the filer reject the rename when the target exists, checked under the rename path locks that CreateEntry on the same filer also takes. * sftp: guard rename against concurrent targets and file-child ACL gains A rename onto a name that inherits its directory's ACL still needs no delete access, but now asks the filer to fail if a target appears before the move, so a concurrently created file cannot be overwritten without delete permission. Moved directories are also compared in the file ACL shape, where list no longer counts as read, so their file children cannot gain read access through the relocation. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
887b1aac76 |
ec: carry the source volume's TTL expiry into the worker-written .vif (#11777)
* ec: carry the source volume's TTL expiry into the worker-written .vif ReadVolumeFileStatus now reports the volume's ExpireAtSec (last write + TTL), and the admin worker stamps it into the .vif it distributes with the shards. Previously every worker-encoded .vif had expireAtSec 0, so EC volumes of TTL collections never expired. * ec: report expiry from the Rust volume server and cover the .vif carry The seaweed-volume drop-in server omitted expire_at_sec from its ReadVolumeFileStatus response, so worker-encoded volumes sourced from it lost the TTL. Fall back to the nested VolumeInfo expiry for servers that predate the top-level field, and pin the nonzero/zero carry with a test. |
||
|
Chris Lu
|
ac996c0ad1 |
filer: shrink zip spool threshold in tests to fix 32-bit OOM (#11796)
TestSpoolZipReaderTempFile and TestSpoolZipReaderLimit each allocate ~33-70 MiB to cross the 32 MiB in-memory spool threshold. On linux/386 the weed/filer test binary already runs near the 4 GiB address-space limit, so these allocations reliably crash it with "out of memory" mid-suite. Make zipSpoolMemoryLimit a var and drop it to 1 KiB in those tests, which exercises the same memory-vs-temp-file spool paths with KiB-sized buffers. |
||
|
Chris Lu
|
1155e65fc5 |
admin: regenerate the gzipped static mirror
admin.js changed in
|
||
|
joma1022
|
161310d834 |
admin: fix Create Bucket sending POST /api/s3/buckets twice (#11790)
admin: fix Create Bucket submitting twice admin.js and the Buckets page template both bound a submit listener to #createBucketForm, so one click sent POST /api/s3/buckets twice. Keep only the page handler. |
||
|
Peter Dodd
|
42ff535511 |
vacuum: bound concurrent commits per volume server (#11658)
CommitCompact holds the volume's write lock for the whole commit, and reads of that volume wait on it. Volumes compacted together tend to finish together, so their commits land at once and their fsyncs contend: each commit's lock hold grows with how many run alongside it, even when the disk is far from its throughput limit. Add -compactionMaxParallelCommits (volume.compactionMaxParallelCommits for server/mini). A commit waits for a slot before CommitCompactVolume, so a queued commit holds no volume lock and reads keep flowing. The wait counts against the caller's deadline and returns the context error if it expires; a slot granted to an already-expired context is handed back, so a request the caller gave up on never starts a commit, and maintenance mode is checked again once the slot is held. The slot is released whether the commit succeeds or fails. Compaction itself is not gated, so -maxParallelVacuumPerServer still sets reclaim throughput. 0, the default, keeps today's behaviour. The limit sits in the volume server so it covers every commit caller: the master's vacuum sweep, the vacuum worker task and volume.tier.compact. The Rust volume server mirrors the flag and behaviour: queued commits wait on a semaphore keyed to the grpc-timeout deadline, hold no store lock, recheck maintenance mode once a permit is held, and release it on every exit path. |
||
|
Chris Lu
|
42ff7f2b8e |
s3: report an empty probed directory as a common prefix, not a key (#11779)
The "<dir>/" existence probe emitted the empty directory as a FolderMimeType Contents entry, so delimited listings returned a fake readable key that breaks clients treating every key as an object. Delimited probes now report the directory as a CommonPrefix -- which the probe accepts for existence -- while the recursive (no delimiter) probe keeps the directory marker. |
||
|
Chris Lu
|
3e55971e0e |
ec.shard.unmount: treat same-node shard copies as one target (#11778)
A shard held on two disks of one server produced two identical live ShardRefs, so resolution reported "ambiguous" even for a fully node-qualified -shardId. Ambiguity now means the shard lives on more than one node; duplicate refs on the same node resolve to it. |
||
|
|
3c6700c310 |
volume balance: stop planning moves onto disk types a server does not have (#11762)
* fix(topology): capacity queries no longer create empty disk-type entries AvailableSpaceFor and CapacityFor called getOrCreateDisk, so asking a node whether it had room for a disk type it lacks created a zeroed DiskUsageCounts entry for that type. Volume growth walks every data center, rack and data node, and ToDataNodeInfo publishes diskUsages, so every node ended up advertising an empty disk (MaxVolumeCount 0, DiskId 0) of every type ever grown: HDD-only servers showed an "ssd" disk and SSD-only servers an "hdd" disk. Read the counters through a non-creating getDisk; a missing entry answers 0, the same numbers as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * fix(admin): don't let an empty disk-type entry shadow a node's real disk UpdateTopology keys disks by node and disk id. The master published an empty entry (no capacity, nothing on it, DiskId 0) for every disk type a capacity check had asked about, and it collided with the node's real disk 0; map iteration order decided which survived. An SSD-only node intermittently showed an empty HDD disk, and an HDD node lost its disk -- and with it its volumes from the volume index, hence "Could not find volume N on source server" in balance detection. IsShadowedEmptyDiskEntry matches an empty entry whose disk id another DiskInfo of the node, one with capacity or data, already uses. UpdateTopology and CountTopologyResources skip it. A lone empty entry is still an empty disk of unknown capacity and is kept, as GetDisksWithEffectiveCapacity expects. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * fix(balance): don't seed balance candidates from shadowed empty disk entries detectForDiskType registered every node with a DiskInfos entry of the type as a zero-volume candidate. An empty entry the master made up for a type the node lacks made that node the least-loaded destination, and its zero max switched the whole disk type to raw-count balancing. The volume server then refused every move: "no space left hdd" / "no space left ssd". Skip entries IsShadowedEmptyDiskEntry matches when seeding candidates. A server that holds volumes of the type but reports no max still enters through the metrics and still triggers the raw-count fallback. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * fix(admin,balance): ignore made-up disk-type entries at any disk id The made-up entry is published with DiskId 0, but it need not collide with a real disk: when location 0 is unavailable (not reported) the node's usable disk is location 1. IsShadowedEmptyDiskEntry required a collision, so such an entry survived both UpdateTopology -- where the unknown-capacity fallback offers it to planners -- and balance detection, which planned moves onto it again. Its disk id cannot tell it apart: a DiskInfo with no volumes, no EC shards and no per-disk max is published with DiskId 0 whichever disk it stands for. IsPhantomDiskTypeEntry (renamed, since nothing needs to be shadowed now) matches an empty entry on any node that has a disk of another type with capacity or data. A real disk looks the same only while it offers no volume slot, when it can take no volume either. A node whose entries are all empty keeps them as disks of unknown capacity, as before. Reported by review on #11762. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * test(balance): assert where the real disks land in the empty-entry test Check through GetNodeDisks that each server's real disk sits at the subtest's disk id before planning. The id-1 case already landed on physical disk 1 (SplitByPhysicalDisk folds all-zero record ids onto the entry's DiskId), so this only makes the setup explicit; on the first revision of the fix it shows the real disk at 1 beside the made-up entry at 0. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * fix(topology,balance,ec): don't publish or trust usage-only disk-type entries ToDataNodeInfo started from the raw disk-usage counters, so a node that was asked for capacity of a disk type it lacks kept advertising an empty DiskInfo for that type even though no such disk was ever reported. Skip usage entries that have no disk child and hold all-zero counters. The same stale entries still misled planners that read DiskInfos directly: the balance plugin's move precondition treated a phantom entry as proof a target has that disk type, and the EC balancer counted it as a matching disk. Filter them with IsPhantomDiskTypeEntry. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
5a0be275cc |
s3: keep the gateway up when an auxiliary listener fails to bind (#11768)
* s3: keep the gateway up when the Iceberg or Lance listener fails to bind The optional catalog servers ran glog.Fatalf on a bind error, so an occupied port (e.g. the default Lance port 9101 taken by another process) killed the entire S3 server. Log the error and run without the auxiliary listener instead; the S3 API itself still fails hard on bind errors. * mini: only probe and advertise catalogs whose listeners bound With aux listener failures no longer fatal, the configured catalog port may answer as an unrelated process; mini's readiness probe accepts any HTTP response and would announce that foreign service as the catalog. Record the bind outcome on S3Options (final by the time the S3 port serves, since the aux binds happen first) and gate mini's readiness wait and admin UI advertisement on it. * s3: test that a failed aux listener bind is non-fatal Hold the Iceberg and Lance ports open and assert both starters return no server and leave the bound flags false, covering the new nonfatal path. * command: range the endpoint test table by index S3Options now holds atomic.Bool flags, so copying a test case trips go vet's lock-copy check. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
cabaf9c38d |
iceberg: collapse doubled /v1 prefix in catalog requests (#11767)
* s3: verify SigV4 against the request path the client sent
Internal routers that rewrite URL.Path (the Iceberg catalog doubled-/v1
collapse) broke signature verification: verifyV4Signature re-derived the
canonical URI from the rewritten path while the client had signed the
original one, so valid signed catalog requests were rejected. Stash the
client-sent path in the request context at rewrite time and prefer it when
computing pathForSignature.
* iceberg: collapse repeated /v1 prefixes before route matching
Move the doubled-prefix collapse from the catch-all into a middleware so it
also covers paths that match the {prefix} routes (/v1/v1/namespaces would
otherwise resolve "v1" as the table bucket; the name is unreachable anyway
since bucket names require 3-63 chars). Collapse all leading repetitions in
a single pass and re-dispatch once instead of recursing per segment, and
record the original path in context so signed requests still verify.
* iceberg: slice off repeated /v1 prefixes instead of concatenating
Stripping path[len("/v1"):] per iteration reuses the original string
instead of allocating a fresh copy of the remaining path each pass.
|
||
|
|
ad2486919c |
iam: accept AWS action names for listing multipart uploads and parts (#11765)
* iam: accept AWS action names for listing multipart uploads and parts PutUserPolicy rejected s3:ListBucketMultipartUploads and s3:ListMultipartUploadParts as invalid actions, so no inline policy could grant these operations by their AWS names. Map both to Read, which is the action their routes are registered under. Fixes #11738 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm * test: check each multipart listing action separately Granting both actions in one policy let a correct mapping for one hide a wrong mapping for the other. Test each action on its own and assert it grants only Read. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm * iam: scope multipart listing grants to their resolved action A policy naming s3:ListBucketMultipartUploads or s3:ListMultipartUploadParts projected to a Read:<bucket> grant, which the legacy route check honored for every object in the bucket. Map the AWS names to dedicated action constants and let VerifyActionPermission fall back to the resolved request action, so the grant covers only the listing operation it names. * iam: narrow legacy ListParts and ListMultipartUploads mappings Both policy names projected to Write:<bucket>, granting full object writes on any bucket where listing was intended. They now map to the same scoped listing actions as their AWS names. * iam: bind the multipart listing fallback to the listing routes The legacy-action fallback in VerifyActionPermission resolved the operation with ResolveS3Action, which reads the promoted authorization object. For GET /bucket?uploads&prefix=foo/ the prefix becomes the object, so the request resolved to GetObject and a ListBucketMultipartUploads grant was denied. Resolving from query priority also did not match how the router picks a handler. Only fall back for the two listing routes, identified from the route vars the same way the router dispatches them, and add the scoped actions to MapToIdentitiesAction so GetUserPolicy in the standalone IAM API rebuilds valid policy actions. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
2521c202aa |
s3: accept signed streaming uploads when IAM is disabled (#11766)
* s3: skip seed-signature verification in chunked reader when IAM is off When no credentials are configured there is nothing to verify streaming chunk signatures against. Leave the credential unset so the reader decodes the aws-chunked body and skips per-chunk verification, matching the unsigned-streaming path. This also fixes SigV4-signed unsigned-streaming uploads on open gateways, which previously failed in calculateSeedSignature. setupIam() now sets isAuthEnabled to match the credentials it installs, so the signature-verification tests still exercise the enabled path. * s3: decode signed streaming uploads when IAM is disabled AWS SDK v2 clients (Trino, Spark, modern aws-cli) send PutObject bodies as STREAMING-AWS4-HMAC-SHA256-PAYLOAD. On a gateway without S3 auth this was rejected with ErrAuthNotSetup, breaking writes to table buckets and regular buckets alike. The auth check already short-circuits when IAM is off, so route both streaming payload types through the chunked reader. * s3: cover signed trailer streaming uploads without IAM Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
b0ab4c4e1e |
shell: make every -collection flag parse the same way (#11794)
* shell: make every -collection flag parse the same way The filter vocabulary (comma-separated names, wildcards, regexes, "_default" for the empty-named collection) already existed in wildcard.CompileCollectionMatcher, but several commands still did their own exact-match comparison or sent the raw flag value to servers. fs.mergeVolumes used a literal string compare, so -collection=_default silently produced an empty merge plan; it now uses the shared matcher while keeping "*" as the match-all default. volume.fsck and volume.mark filter through the matcher too, and volume.vacuum resolves the filter client-side into per-collection and per-volume requests, since no Collection value scopes the VacuumVolume RPC to the unnamed collection alone. Commands that put one concrete collection name on the wire normalize "_default" to "" (volume.grow, fs.configure, mq.topic.compact, collection.delete, ec.rebuild, ec.balance), and the tier -volumeId commands resolve the volume's actual collection from topology first so patterns and named-collection requests pass the volume server's collection check. * shell: spell the default-collection keyword via CollectionDefault * shell: address review findings on collection filter handling volume.fsck keeps a set of every volume the master knows before any filtering, so a filer entry whose volume was excluded by -collection or -volumeId is skipped instead of reported "volume not found" and deleted by -reallyDeleteFilerEntries. The collection filter also no longer scopes the filer walk to /buckets/<collection>: fs.configure can bind a collection to any path prefix, and a narrower walk could mark live chunks as orphans under -reallyDeleteFromVolume. ec.rebuild and ec.balance resolve -collection through the new wildcard.ExpandCollectionFilter, which passes literal names through even when the master does not list them — ec.rebuild must be able to recover a collection whose shards all lost their index and never registered. fs.configure and mq.topic.compact now reject _default outright: an empty Collection means "unset" in both the path-rule merge and the filer's storage-option fallback chain, so the keyword would have been silently ignored. The unnamed-collection vacuum sweep keeps the master's low-disk-space exception: a volume whose every replica is read-only is probed with VacuumVolumeCheck and stays eligible when a replica reports DiskSpaceLow, instead of being dropped outright. |
||
|
Chris Lu
|
268424a5e1 |
s3: extract zip members on PutObject (#11786)
* filer: add zip archive extraction helpers SpoolZipReader makes an upload body seekable for archive/zip, SanitizeZipMemberName rejects zip-slip and non-regular members, and CheckZipLimits caps member count and total uncompressed size. For seaweedfs/seaweedfs#11722 * filer: extract zip members on ?extract=zip upload POST/PUT to a directory path with ?extract=zip (or X-Seaweedfs-Extract: zip) writes each archive member as its own entry, so a bundle of small files lands in one request instead of one roundtrip each. Members keep their mtime, mode and mime-by-extension; directories are created; unsafe names are skipped and reported. extract.keep=true also stores the archive itself. For seaweedfs/seaweedfs#11722 * filer: bound zip spool memory by actual bytes A chunked upload has no Content-Length, so the old size hint let an unbounded body go through io.ReadAll. Read up to the memory threshold plus one byte first, then spill the prefix and the rest to a temp file, capped at ZipExtractMaxBytes total. * filer: guard zip member size total against overflow Declared member sizes summing past math.MaxUint64 wrapped the running total and slipped past the 1 TiB cap. * server: only take the inline fast path at a short first read uploadReaderToChunks read at most chunkSize bytes, then treated any read below SaveToFilerLimit as the whole file. When SaveToFilerLimit exceeded chunkSize, longer uploads were stored truncated. * server: resolve storage options per extracted zip member A member can land under a storage rule more specific than the request path. Re-running detectStorageOption0 per destination applies that rule's collection, replication, TTL, filename limit, and read-only guard to files and directories alike, while explicit query overrides still win. * server: accept raw request body for POST extract uploads Every POST was forced through MultipartReader, so a raw zip body always got a 400. Only parse multipart when the Content-Type is actually multipart. * server: extract zip directories before files and stop on cancel Processing directory members first keeps their stored mtime/mode from being overwritten by parents auto-created for an earlier file member. Checking ctx.Err() between members keeps a disconnected client from driving thousands more writes. * server: assert stored mtime and mode for extracted members * s3: add zip extraction header constants * s3: extract zip members on PutObject with extract header x-amz-meta-snowball-auto-extract or x-seaweedfs-extract: zip on a PutObject fans out the archive to one object per member under the object's directory, reusing putToFiler so chunking, collections, storage rules, versioning, and bucket-default encryption apply to each member. The archive itself is not stored. Explicit SSE request headers are rejected since one encryption config cannot represent independently created member objects. * s3: bound member fan-out by request cancellation Stop starting new member writes once the request context is done, matching the filer extract loop. * filer: bound compressed zip spool and reject non-local member names The 1 TiB uncompressed limit was also capping the compressed bytes spooled to the host's temp directory, so a handful of concurrent extractions could fill the filer's disk before any ZIP validation ran. SpoolZipReader now takes a separate compressed-size limit (the filer passes its TUS max upload size) with a 5 GiB default. SanitizeZipMemberName gains a filepath.IsLocal check: the segment loop already rejects "..", but CodeQL only treats member names as sanitized through a recognized guard, and the extra check also covers reserved Windows names. * server: honor destination rules for zip members and clamp maxMB An x-seaweedfs-destination header picks the storage rules for a logical destination while members still land under the request path, matching PostHandler. Directory members now also enforce the actual path's read-only rule instead of relying on option resolution to fail. maxMB up to 2048 overflowed the int32 chunk size, so an oversized query value made extraction (and ordinary autoChunk uploads) store members empty yet report success. The parsing is shared and clamped at 2047 MB. * s3: scope zip member writes through the per-key checks PutObject applies The extraction loop bypassed most of what a standalone PutObject would face: members were written without IAM authorization for their key, directory members replaced markers without entry lock checks, and lock headers persisted retention without the s3:PutObjectRetention or s3:PutObjectLegalHold permissions. authorizeMember now runs IAM, table-bucket path rules, bucket policy, lock-header validation, and non-versioned lock enforcement per member key, and directory members take the same guarded mkdir path as an explicit dir marker. The archive's own Content-Md5 and x-amz-checksum-* values are verified while the body spools instead of being silently dropped, members stop inheriting the archive's Content-Encoding, and non-versioned members resolve their lifecycle TTL like normal uploads. * s3: fail malformed lock headers before spooling the archive Per-member re-validation turned a bad object-lock header into one error entry per member inside a 200 response. Validate once up front like PutObjectHandler does, and keep only the dedicated retention/legal-hold permission checks scoped to each member key. * filer: return 413 for oversized archives and honor saveInside in extract Map filer.ErrZipTooLarge to 413 instead of 500 for client-side archive size violations. Extracted members written under saveInside now store as inline entry content instead of unreachable volume chunks. * s3: write extracted members under their own request context Point member writes at the member key via mux.SetURLVars so SSE-KMS builds the encryption context from the member ARN, not the archive's. Map the spool limit error to ErrEntityTooLarge. |
||
|
Chris Lu
|
a8affc851d |
filer: extract zip members on ?extract=zip upload (#11782)
* filer: add zip archive extraction helpers SpoolZipReader makes an upload body seekable for archive/zip, SanitizeZipMemberName rejects zip-slip and non-regular members, and CheckZipLimits caps member count and total uncompressed size. For seaweedfs/seaweedfs#11722 * filer: extract zip members on ?extract=zip upload POST/PUT to a directory path with ?extract=zip (or X-Seaweedfs-Extract: zip) writes each archive member as its own entry, so a bundle of small files lands in one request instead of one roundtrip each. Members keep their mtime, mode and mime-by-extension; directories are created; unsafe names are skipped and reported. extract.keep=true also stores the archive itself. For seaweedfs/seaweedfs#11722 * filer: bound zip spool memory by actual bytes A chunked upload has no Content-Length, so the old size hint let an unbounded body go through io.ReadAll. Read up to the memory threshold plus one byte first, then spill the prefix and the rest to a temp file, capped at ZipExtractMaxBytes total. * filer: guard zip member size total against overflow Declared member sizes summing past math.MaxUint64 wrapped the running total and slipped past the 1 TiB cap. * server: only take the inline fast path at a short first read uploadReaderToChunks read at most chunkSize bytes, then treated any read below SaveToFilerLimit as the whole file. When SaveToFilerLimit exceeded chunkSize, longer uploads were stored truncated. * server: resolve storage options per extracted zip member A member can land under a storage rule more specific than the request path. Re-running detectStorageOption0 per destination applies that rule's collection, replication, TTL, filename limit, and read-only guard to files and directories alike, while explicit query overrides still win. * server: accept raw request body for POST extract uploads Every POST was forced through MultipartReader, so a raw zip body always got a 400. Only parse multipart when the Content-Type is actually multipart. * server: extract zip directories before files and stop on cancel Processing directory members first keeps their stored mtime/mode from being overwritten by parents auto-created for an earlier file member. Checking ctx.Err() between members keeps a disconnected client from driving thousands more writes. * server: assert stored mtime and mode for extracted members * filer: bound compressed zip spool and reject non-local member names The 1 TiB uncompressed limit was also capping the compressed bytes spooled to the host's temp directory, so a handful of concurrent extractions could fill the filer's disk before any ZIP validation ran. SpoolZipReader now takes a separate compressed-size limit (the filer passes its TUS max upload size) with a 5 GiB default. SanitizeZipMemberName gains a filepath.IsLocal check: the segment loop already rejects "..", but CodeQL only treats member names as sanitized through a recognized guard, and the extra check also covers reserved Windows names. * server: honor destination rules for zip members and clamp maxMB An x-seaweedfs-destination header picks the storage rules for a logical destination while members still land under the request path, matching PostHandler. Directory members now also enforce the actual path's read-only rule instead of relying on option resolution to fail. maxMB up to 2048 overflowed the int32 chunk size, so an oversized query value made extraction (and ordinary autoChunk uploads) store members empty yet report success. The parsing is shared and clamped at 2047 MB. * filer: return 413 for oversized archives and honor saveInside in extract Map filer.ErrZipTooLarge to 413 instead of 500 for client-side archive size violations. Extracted members written under saveInside now store as inline entry content instead of unreachable volume chunks. |
||
|
Chris LuandDevin
|
6da432b8bf |
filer: make a retried ObjectTransaction replay its recorded outcome (#11788)
* filer_pb: add request_id to ObjectTransactionRequest Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: replay recorded outcome for a retried ObjectTransaction A failover replay carries the same request_id; under the same path lock the owner replays the recorded outcome instead of re-applying the mutations over whatever superseded them. Only application-level outcomes are recorded — a lookup error leaves the request free to re-run. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: stamp routed object transactions with a request id One id per logical transaction so failover retries replay the recorded outcome instead of re-applying the mutations. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: scope transaction replay tokens per path and keep them for their TTL The dedup key includes the lock path so a token reused on a different object cannot replay this outcome. Records are never evicted early — an early eviction would let a replay re-apply — so the expiry sweep is amortized and the map is bounded by the insert rate over the TTL. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: bound the transaction replay cache at capacity * filer: do not record indeterminate transaction outcomes A mutation error can be transient or follow a partial apply; recording it replays that failure to a retry for the whole TTL instead of letting the same idempotent mutations converge. Only successful transactions and precondition failures are definitive outcomes worth replaying. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: replay recorded mutation failures instead of re-applying --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
da3b7feeae |
admin: move dashboard progress labels beside the bar (#11781)
* s3: keep suspended-versioning conditional writes off the routed condition A suspended PUT writes the null-version slot at the regular path while the entry the conditional headers evaluate lives under .versions/. The routed WriteCondition checked the request's etag against that empty slot, so a correct If-Match failed with 412. Passing uniqueWritePath sends conditional requests through the lock path, whose precondition resolves the same entry the read path serves; unconditional writes still route. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * admin: keep the volumes progress label readable at low fill The dashboard's Volumes and tier-usage cells rendered their text inside the progress bar, so a low fill left white label text on the light track and the count was unreadable. Render the label beside the bar instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * admin: keep a minimum width on dashboard progress tracks A nonwrapping label beside a flex track could squeeze it to zero in a narrow table, hiding the bar entirely. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
00804b32fa |
s3tables: let an explicit policy Deny veto fallback and owner allows (#11772)
* s3tables: let an explicit policy Deny veto fallback and owner allows Table operations evaluated the table or namespace policy and the bucket policy as an OR, so an explicit Deny in one policy was overridden by an Allow in the other. Owner and identity grants also short-circuited before the resource policy was read. Deny statements are now evaluated first at both levels, matching AWS explicit-deny semantics. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: fail closed when a resource side has no owner --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
44657988ea |
s3: apply destination retention on copies instead of inheriting it (#11773)
* s3: apply destination retention on copies instead of inheriting it CopyObject carried the source object's retention mode, retain-until date and legal hold into the destination entry, so a copy onto a bucket with default COMPLIANCE retention produced an unlocked object that could be deleted early. Copied objects now take their retention from the copy request's lock headers or the destination bucket's default, matching PUT semantics; a versioned self-copy gets the same treatment while an unversioned metadata-only self-copy keeps the object's existing lock. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: validate lock headers on metadata-only self-copies --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
2d06e2d6c1 |
s3: evaluate tag conditions on copy-source and tagging subresources (#11771)
* s3: evaluate tag conditions on copy-source and tagging subresources Policy evaluation ran without the object entry on several paths, so a Deny conditioned on s3:ExistingObjectTag never matched: - CopyObject, UploadPartCopy and RenameObject checked the source key before its entry was fetched; repeat the check on a synthesized GET once the entry resolves. - GetObjectAttributes and Get/Put/DeleteObjectTagging never supplied the entry; recheck with the resolved entry. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: fail closed on tag-policy lookups and honor tag-conditioned grants - recheckTagConditionedPolicyWithLookup returned success when the entry lookup errored, letting a tag-gated Deny slip past a transient miss; only ErrNotFound still falls through to the handler's own NoSuchKey. - A read or delete denied in the tagless first phase now gets one more evaluation with the resolved entry's tags in hand, so an s3:ExistingObjectTag-conditioned Allow can still grant a copy-source read or rename. The deferred recheck requires an explicit Allow; indeterminate results keep the original verdict, and the existence of the key never leaks past a deny. - RenameObject now applies the same tag-aware recheck to the source removal, so a tag-gated Deny on s3:DeleteObject blocks a rename the way it blocks a direct DELETE. - GetObjectAttributes evaluates the object-entry policy recheck before conditional headers, so a 304/412 response can no longer bypass a tag-based Deny. * s3: honor identity denies in the tag-aware source recheck The deferred recheck re-evaluates only the bucket policy, so a bucket Allow could lift an explicit Deny in the caller's attached or session policies on a copy/rename source. Apply the same applicable-policy Deny check the first phase uses. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
446e6f3444 |
s3tables: scope OIDC-derived account IDs to the token issuer (#11774)
* s3tables: scope OIDC-derived account IDs to the token issuer Table ownership keyed off the bare sub claim, so two identity providers issuing the same sub collided on one account - a caller could act on tables owned by a different provider's user. OIDC bearer tokens now carry the same issuer-namespaced (sub, iss) hash STS sessions already expose as aws:userid, and getAccountID prefers it over the raw claims. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: keep legacy owners working and scrub client-subject claims - OIDC bearer tokens without a subject are rejected, and a token-supplied aws:userid claim is always overwritten with the verified (sub, iss) hash, so a forged claim can no longer stand in for the principal. - Owners written before issuer scoping hold a bare subject; permission checks now re-anchor that subject under the caller's issuer, so the same (sub, iss) pair keeps its ownership while scoped owners remain issuer-isolated. The token issuer is asserted into claims alongside sub and carried through PolicyContext. * s3tables: anchor legacy subjects to the configured issuer set * s3: enumerate trusted issuers from registered providers too * s3: fail closed when the OIDC provider store cannot be listed trustedOIDCIssuers kept the registered issuers when ListOIDCProviders failed, presenting a partial set as complete. A deployment trusting one registered issuer A plus a stored issuer B would, on a listing error, report only A — the single-issuer shape under which a legacy bare subject re-anchors — letting a caller from A claim a stored subject that belongs to B and gain owner access. With no provider store the registered providers are provably complete, so keep returning them; with a store, a failed listing means the real set is unknown, so return none. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
348116b2c4 |
filer: sweep the local subtree on a replayed directory delete (#11785)
* filer: sweep the local subtree on a replayed directory delete A recursive delete emits only the directory's own key, so peers applied it with a single DeleteEntry. Children the peer held that the origin had not listed stayed behind unlisted but still readable, and their needles were never deleted. Sweep the local children on a replayed directory delete and drop their chunks when the origin deleted chunks. * filer: walk the subtree on a replayed directory delete DeleteFolderChildren removes only direct children in stores keyed by immediate parent, so the sweep now lists and deletes each descendant entry itself. Entries the local filer committed after the event survive — with the directories that hold them — and hard-linked files keep their chunks for the origin's own events to settle. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: key replayed subtree chunk deletes to each entry's path The sweep queued every descendant's chunks under the deleted directory's storage rule and skipped directory entries entirely. Chunks now collect keyed by their owning path, so per-path rules apply to each file — and a directory carrying its own chunks (a file promoted by an S3 child) frees them too. * filer: fence kept replayed dirs and flush collected chunks on error A directory kept because a newer descendant survived still needs its delete timestamp recorded, or a stale replayed create can resurrect another child under it. And chunks collected for deleted children must be flushed even when the sweep fails partway — the entries are gone and a retry cannot list them again. Collection now also runs only after the entry delete succeeds. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: flush swept chunk references in batches --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |