* iam: take the implicit user name from the verified identity
For self-service actions sent without UserName, the implicit user was
resolved by re-parsing the raw Authorization header for a SigV4
credential and looking that access key up. The header is not what
authentication verified: a request can be authenticated by one
mechanism while carrying a header that names another user's access
key, and the action then ran against that other user.
The auth middleware already records the verified identity in the
request context, so use it, on both the standalone iam server and the
embedded IAM surface. When no verified identity is present the name is
left unset rather than guessed.
* iam: test implicit user name resolution
Cover a request authenticated by presigned credentials that also
carries a junk SigV4 header naming another user: the self-service
actions must run as the verified signer. Update the unit tests for the
context-based lookup.
* iam: resolve the implicit user for standalone GetUser too
The embedded surface fills UserName for GetUser; the standalone server
skipped it, so an implicit GetUser on `weed iam` answered NoSuchEntity.
Same verified-identity source as the other self-service actions.
* iam: cover all implicit-username actions in the regression test
Table and partition statistics are Puffin files that the table metadata lists
in its statistics entries; orphan cleanup keeps them (#11651), and its test
places one at metadata/stats-1.puffin. The table bucket layout rule accepted
only *.stats among them, so an engine that names a statistics file *.puffin
had the upload refused with 403 and could not write statistics at all.
The metadata patterns now admit a *.puffin file directly under metadata/,
with the same safe-character rule as the other catch-all patterns. Nothing
else widens: a Puffin file in a subdirectory of metadata/, with a trailing
extension, or outside <namespace>/<table>/ is still refused.
Test: the new metadata/*.puffin cases in
TestIcebergLayoutValidator_AcceptsRealWorldManifestNames and
TestTableBucketFileValidator_PuffinStatisticsOnlyInTableMetadata fail on
master ("invalid metadata file format") and pass with the change; the
rejected cases pass on both.
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
* filer_pb: add MkdirExclusive
A taken name leaves the callback as a result, not an error, so filer
failover does not count a lost race as a filer failure.
* s3tables: create the entry with its attributes in one exclusive create
CreateTableBucket and CreateNamespace checked that the name was free and
then wrote the entry in several calls, so a second create in the window
replaced the first caller's entry including its owner. The create now
carries the marker, metadata and tags in one O_EXCL CreateEntry that
answers ErrEntryAlreadyExists with the same 409.
* s3: PutBucket uses an exclusive mkdir
The same window let a plain PutBucket replace a table bucket created
concurrently. The already-exists branch now answers from the entry itself,
since the registry just cached the name as absent, and the admin
auto-create path is exclusive too.
* s3: read the winning entry when a create loses the race
The concurrent-create branch called existingBucketError on a name the
registry had just cached as absent, answering NoSuchBucket, and the
auto-create fallback could assign an S3 owner to a table bucket it met
there. The negative cache is cleared before the existing-bucket error is
built, and a table bucket keeps its owner-less shape.
* s3: fail an auto-create that met a table bucket
A table bucket carries no S3 owner and must not gain one, and the upload
that auto-create ran for must not continue into it either.
* s3: fail an auto-create when the winning entry cannot be read
An unreadable existing entry fell through to success, so the write
continued into an unvalidated entry, possibly a table bucket.
* s3: resolve versionId tagging requests to the version-tagging actions
A request for an object's tags with versionId acted on a historical
version but authorized against the current-object tagging action, so an
explicit Deny on s3:*ObjectVersionTagging never applied.
* iam: map PutObjectVersionTagging and DeleteObjectVersionTagging to the tagging action
* s3: only resolve tagging when the query value is empty
The tagging routes match ?tagging with an empty value, so a non-empty one
is not a tagging request and would resolve a tagging grant onto the plain
read or version read that actually serves it.
* s3: keep resolving tagging for any query value
The routes match ?tagging whenever the key is present, so tagging=x is a
tagging request served by the tagging handlers and must resolve to the
tagging actions.
* s3tables: only the admin repairs a table bucket that has no owner metadata
CreateNamespace wrote bucket metadata naming the caller as owner before any
permission check ran, so an ordinary caller that found the marker without
metadata was written in as the bucket's owner and passed every check after.
A bucket in that state is answered with the operation's own denial; the
admin or a trusted internal handler may still repair it.
* s3tables: repair the marker-only bucket under the entry lock
A creator finishing its own create between the missing-metadata check and
the repair write would have its owner overwritten. The mutation now adopts
whatever metadata is there and writes only while it is still absent.
* s3tables: gate the repair on the Admin action, not an operation grant
hasAdminAction also answers s3tables:*, so a non-admin identity holding
the wildcard operation grant could pass the repair check and write itself
in as the bucket's owner.
* s3api: persist single-identity IAM changes through the credential manager
CreateAccessKey, DeleteAccessKey, UpdateAccessKey, PutUserPolicy, DeleteUserPolicy, SetUserStatus, UpdateUser and DeleteUser used to load the whole configuration, change one identity and then SaveConfiguration, rewriting and re-propagating every identity. They now write only the identity (and, for rename/delete, the groups and service accounts that referenced it) through the credential manager, which propagates that identity alone. Static identities and skipPersist keep the previous full-save/no-write contract.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iamapi: persist single-identity IAM changes through the credential manager
Same targeted-write change as the embedded IAM API for the standalone weed iam dispatcher: UpdateUser, DeleteUser, CreateAccessKey, DeleteAccessKey, UpdateAccessKey, PutUserPolicy and DeleteUserPolicy write one identity through the credential manager instead of saving the whole configuration. CreateAccessKey keeps the full save when the handler created a user the store does not know.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* credential/memory: re-key identity on UpdateUser rename
UpdateUser stored the identity under the old username key even when identity.Name changed. Re-key it the way filer_etc does (delete old, save new) so the targeted rename path sees the moved record.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover targeted single-identity IAM writes
Recording-store tests assert each rerouted action uses the single-identity credential store methods and never SaveConfiguration, in both the embedded S3 IAM dispatcher and the standalone iamapi dispatcher, plus the skipPersist and static-identity contracts and a memory-store rename case.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* credential: keep not-found and rename-collision semantics in stores
IamGrpcStore.DeleteAccessKey translates gRPC NotFound to ErrAccessKeyNotFound so a repeat delete stays the no-op the IAM handlers advertise. MemoryStore.UpdateUser rejects a rename onto an existing user instead of overwriting it, matching RenameUser.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: mask anonymous misses behind AccessDenied unless the bucket is listable
Deferred anonymous GET/HEAD authorization only runs once an entry is
selected, so misses returned 404 while existing keys returned 403 — an
unauthenticated existence oracle for bucket names and object keys.
Route not-found responses in Get/HeadObjectHandler through a check that
only discloses 404 when anonymous listing is allowed (policy, identity or
a public-read bucket); otherwise the miss is indistinguishable from a
denied read, matching AWS and 4.48 behavior.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: mask missing-directory misses for anonymous reads too
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: check a fixed request shape when masking anonymous misses
The miss check passed the caller's request through, so query parameters
could retarget the resolved action (?policy) or satisfy an s3:prefix
condition on an unrelated subtree. Each check now runs on a request with a
fixed shape: a ListBucket scoped to the probed key and a plain read of the
key, so the miss only shows when the caller could learn the same fact
directly.
* s3: mask conditional-header misses for anonymous reads
An If-Match conditional on a missing key wrote NoSuchKey directly from
processConditionalHeaders, before the handlers reached the masked miss
path, so anonymous callers could still distinguish a missing key from a
denied read.
* s3: keep the versionId when masking anonymous misses
The read leg must evaluate the action the caller's request resolves to:
a versioned miss is s3:GetObjectVersion, and dropping versionId would let a
plain GetObject grant disclose versions the caller may not read.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: share the UpdateTable authorization gate
Extract the validation, load, and permission check from handleUpdateTable
into authorizeUpdateTable so the table an update was authorized against is
the same snapshot the mutation writes to. Manager gains
AuthorizeUpdateTable, mirroring AuthorizeCreateTable, so callers can ask the
question without updating anything.
* s3tables: share the UpdateView authorization gate
Extract the lookup, policy load, and permission check from handleUpdateView
into authorizeUpdateView, mirroring authorizeUpdateTable, and expose it as
Manager.AuthorizeUpdateView for callers that need the answer without the
update.
* s3api: scope vended table credentials to read-only on request
VendTableCredentials takes a readOnly flag and tablePrefixSessionPolicy
emits only s3:GetObject for the table prefix when it is set, keeping the
conditioned ListBucket and unconditioned GetBucketLocation statements that
reads still need.
* iceberg: vend read-only credentials when the caller may not write
LoadTable and friends vend storage credentials only for the authority the
caller holds: the vend now checks UpdateTable/UpdateView permission and a
caller that may only read the catalog entry gets a read-only session. Paths
that just passed a write gate (stage-create, view commit) keep the full
session.
* lance: vend read-only credentials when the caller may not write
storage_options vends the same read-only session as the Iceberg catalog
when the caller lacks UpdateTable on the table.
* lance: keep write credentials for callers authorized to declare
A Lance commit is a direct S3 write that never revisits the catalog, so a
caller that just passed CreateTable still gets a writable session; the
UpdateTable gate applies to describe, the read path.
* iceberg, lance: log a failed write check instead of only reading it as denied
* s3: list keys in byte order around directory markers
* s3: order deferred dir blocks by slash name and hold quota
* s3: fix deferred dir blocks at page and marker boundaries
Review follow-ups on the marker byte-order change:
- A marker inside an emitted "<name>/" block re-descended and re-listed its
folded contents under delimiter "/" instead of skipping the block.
- A prefix object's "<name>/" common prefix was emitted eagerly at the name
position, outranking siblings like "a.txt" and repeating across pages; the
plain key now emits alone at the name position and the block emits the
common prefix when children exist.
- A recursion starved on quota before listing a deferred subtree returned
without truncation, dropping keys silently; the block now reports a
resumable truncation when the subtree still holds entries.
- Clone prefix-object entries with proto.Clone rather than copying the
protobuf message by value.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: close the hasEntries probe stream on return
* s3: resume delimited listings at the dir block boundary
A deferred directory block emits at its "<name>/" position, but emitDir
left nextMarker at the bare name, which sorts inside the (name, name/)
band and reads as "block pending". Every pass of the listing loop then
re-emitted the same common prefix until maxKeys ran out (999 prefixes,
IsTruncated), and a "<name>/" continuation token trimmed to "<name>" by
adjustMarkerForDelimiter reopened the block and paged forever.
Advance nextMarker to the block boundary the same way the recursive
branch does, and keep a "/" -ending marker intact — it already names the
bound past the block.
---------
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: strip internal object-lock and version headers on auth entry
putToFiler persists Seaweed-X-Amz-Object-Lock-Mode, -Retention-Until-Date,
-Legal-Hold and -Version-Id request headers into authoritative entry
metadata without a permission check, so an s3:PutObject-only writer could
pin COMPLIANCE retention and legal hold on its own uploads, bypassing the
dedicated-permission gate on the standard x-amz-object-lock-* headers.
The server only sets these headers from validated inputs, so drop them in
authenticateRequestInternal alongside the identity headers already
scrubbed there.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: strip internal SSE headers on auth entry
X-SeaweedFS-SSE-S3-Key-Data, -SSE-S3-Base-IV and -SSE-KMS-Base-IV are read
from the request to drive multipart encryption, so a client could inject
its own key material and IVs. The server sets them itself between handler
stages of a request; drop inbound values in authenticateRequestInternal.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: scrub internal headers before the auth-enabled early return
Auth and AuthPostPolicy skip authenticateRequestInternal when IAM is
disabled, so client-supplied internal headers still reached putToFiler.
Hoist the scrub into scrubInternalHeaders and run it unconditionally in
both wrappers as well as the shared auth entry point.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: test internal header scrub through the auth-disabled wrapper
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: sort ListMultipartUploads by key and honor key-marker
* s3: skip upload folders without an object key in ListMultipartUploads
* s3: order same-key multipart uploads by initiation time
The upload id suffix is now a time-ordered UUIDv7, so sorting by upload id lists a key's uploads oldest first, like S3. Pagination is unchanged: ordering and the upload-id-marker comparison share the same key.
selectMultipartUploads now builds the whole result, including NextKeyMarker and NextUploadIdMarker, and the pagination test pages with the response markers.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3tables: accept .puffin data files for Iceberg v3 deletion vectors
Iceberg v3 writes deletion vectors as Puffin files under a table's data/
directory, so merge-on-read DELETE and UPDATE on v3 tables were refused
with AccessDenied by the table-bucket upload validator.
Fixes#11842
* s3tables: name all accepted data formats in rejection error
The message stopped at .avro even though .lance and now .puffin are
accepted.
* s3tables: cover deletion-vector filenames in data layout test
Uses the names real engines emit: Spark/Iceberg Java
{seq}-{task}-{uuid}-{counter}-deletes.puffin, UUID-suffixed names, and
the same under a partition directory.
* s3api: check the direct peer against the trusted-proxy allowlist
* s3: verify X-Forwarded-Prefix only from trusted proxies
* s3: note that trusted proxies must sanitize forwarded headers
* s3tables: honor a table policy Deny for maintenance operations
authorizeMaintenanceTable evaluated only the bucket policy, so a Deny in the table policy never vetoed PutTableMaintenanceConfiguration, GetTableMaintenanceConfiguration or GetTableMaintenanceJobStatus even though they act on the table ARN. The stored table policy is now checked for a matching Deny first via deniedByPolicy, a veto helper that keeps the admin exemption, and resource and bucket tags are loaded into the policy context so tag-conditioned Denies evaluate. readBucketPolicy is renamed to readPolicy and reused for the table path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: honor the stored policy's Deny on policy management operations
PutTablePolicy, GetTablePolicy and DeleteTablePolicy only evaluated the bucket policy, so a Deny in the table's own policy was ignored and a bucket-level Allow could strip or replace the policy guarding a table. The existing table policy is now loaded and a matching Deny vetoes the call. PutTableBucketPolicy likewise consults the existing bucket policy for a Deny before replacing it. Resource and bucket tags populate the policy context so tag-conditioned Denies evaluate, and policy writes assert the policy authorized against is still in force, returning a conflict when it changed mid-request.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: apply a table policy Deny to tag operations
TagResource, ListTagsForResource and UntagResource evaluated only the bucket policy; a Deny in the table's own policy is now honored when the target ARN is a table. ListTagsForResource and UntagResource also returned early when the resource carried no tags attribute, skipping the permission check entirely; the check now always runs. Policy evaluation uses the canonical resource ARN rebuilt from stored metadata, so a request ARN with a forged account or region cannot dodge the Deny.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: regression tests for resource policy denies
Cover the case where a bucket policy grants an identity s3tables:* while the table policy denies it: the policy, tag and maintenance operations must stay denied so the protection cannot be stripped. Also cover a bucket policy that denies PutTableBucketPolicy under the open fallback, and a tag-conditioned Deny, which vetoes only while the stored tag matches.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iam: scope CreateServiceAccount's ParentUser by policy resource
AuthorizeIamAction evaluated the iam:CreateServiceAccount grant against a
hardcoded arn:aws:iam:::* resource, so the policy's Resource never had a
say; #11218 then bound the target to the caller outright. Evaluate
self-target actions against the target identity's ARN instead: a wildcard
resource keeps the deliberate any-target grant, a scoped resource confines
issuance to the named parent identity, anything else is denied.
buildResourceARN gains the ARN pass-through buildS3ResourceArn already has,
so iam: ARNs reach the policy engine intact on the attached-policies route.
Closes#11735
* ci: retrigger after unrelated weed/storage metrics flake
* iam: fold IAM-ARN matching into awsIAMMatch, drop debug test
- remove zz_debug_test.go (local debugging leftover)
- fold awsIAMMatchResource into awsIAMMatch: the arn:aws:iam:: value
prefix selects case-sensitive matching, so no separate name is needed
- trim comments
---------
Co-authored-by: batonogov <batonogov@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: resume into the directory a listing marker names
* s3: keep delimited marker resumes exclusive
* s3: descend a marker-named directory without repeating its key
* s3: emit marker-named dir objects but keep internal folders hidden
* sts: recognize issuer-scoped parent user ids
ComputeParentUser output has a fixed shape (base64url SHA-256); let
callers distinguish it from bare subject strings.
* s3tables: keep legacy owner re-anchoring off local and scoped account ids
Re-anchoring a stored bare subject under the single trusted issuer must
only apply to strings that could actually be a legacy OIDC subject. A
stored owner naming the reserved or a configured local account, or an
already issuer-scoped id, can otherwise be claimed by any caller whose
token asserts that string as sub.
* s3tables: re-anchor legacy policy principals in map and array forms
A stored bare subject can appear as {"AWS": sub} or inside a principal
list, not only as a top-level string; re-anchoring only the string form
left legacy Deny statements unable to match the caller.
* s3tables: keep built-in legacy exclusions independent of the account checker
A missing local-account predicate must not disable re-anchoring entirely:
the reserved accounts, ARN-shaped values, and scoped-id shapes are never
OIDC subjects regardless, while legacy Deny statements for plain subjects
still need to resolve so a denied caller cannot fall through to an
identity grant.
* s3tables: exercise legacy Deny against an owner grant
The previous case denied the caller regardless of whether the statement
principal matched; make the caller the owner so a missed match shows.
Cover the array Deny form alongside the AWS map.
* s3api: recheck condition under lock when routed filer ignores condition_alt_key
A filer that predates condition_alt_key silently drops the field and
evaluates a suspended-versioning condition against the empty null slot
alone, so a matching If-Match still gets a false 412. Recheck under the
object lock instead of trusting that response: the lock path resolves the
same entry the read path serves. Genuine precondition failures still
return 412 from the recheck.
* s3api: cover suspended If-Match against a filer ignoring condition_alt_key
* s3api: resend against the resolved entry instead of the lock-path recheck
Rechecking under the gateway object lock does not serialize against
routed writers, which commit under the filer's path lock; a concurrent
routed PUT could land between the recheck and the create and be
overwritten with a stale precondition. Resend the same transaction with
condition_key set to the resolved version file so the condition and the
write stay under the same filer lock. When the resolved entry is the
null slot, a delete marker, or unresolvable, the resend targets the
object path and fails closed.
* s3api: resolve the target version in tag-condition policy lookups
* s3api: apply tag-conditioned policy to object deletes
* s3api: apply tag-conditioned policy to object ACL reads and writes
* s3api: apply tag-conditioned policy to retention and legal hold
* s3api: cover tag-conditioned delete, ACL and object-lock rechecks
* s3api: let tag-conditioned grants redecide denied batch deletes
* s3api: resolve the tagged version in delete policy tests
readChunkRange sent the volume lookup's http:// URL through a bare
http.Client, so with [https.client]/[https.volume] configured every read of
an SSE-KMS object larger than one chunk (and every ranged read through this
path) failed with "client sent an HTTP request to an HTTPS server" and the
GET returned 500. Normalise the URL's scheme and use util_http's global
client, which carries the [https.client] TLS configuration, as the other
volume reads already do. Drop the now-unused volumeServerHTTPClient.
Fixes#11799
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer_pb: map PRECONDITION_FAILED to a sentinel error
A conditional CreateEntry that the owner's path lock refuses comes back as
FilerError_PRECONDITION_FAILED; mapping it to ErrPreconditionFailed lets the
S3 gateway tell a rejected precondition apart from a transport failure and
report 412 instead of retrying the write.
* s3api: make the ambiguous-PUT recovery create conditional under the lock
createAfterAmbiguousRoute verified state and then issued an unconditional
create, so a write landing between the lookup and the create was still
clobbered by the stale re-commit (issue #11631). The recovery create now
carries a WriteCondition — IF_NOT_EXISTS for a proven-absent entry,
IF_ENTRY_EQUAL for a directory it may need to write onto — which the owner
filer evaluates under its per-path lock, making the check and the write
atomic. A refused precondition loops back to re-read the state instead of
overwriting whatever won the race.
* s3api: do not fail over a routed transaction past a possibly-sent request
withFilerClientFailover retried the callback on the next filer on any
transport error, so an ObjectTransaction whose response was lost could be
re-applied over a write that committed in between (issue #11631). A gRPC
error cannot distinguish "never sent" from "committed", but channel
readiness can: a non-replayable callback now runs only on a connection that
just reached READY, and failover continues only over channels that never
came up — the one failure mode that provably sent nothing.
* s3api: run ambiguous-route recovery for empty PUTs too
The recovery was gated on a nonempty chunk list, so a zero-byte PUT whose
routed transaction lost its response still fell back to an unconditional
create that could overwrite a newer write and delete its chunks. Chunk-count
mismatch on an empty upload list is enough to distinguish a superseding
entry from this PUT's — there is nothing to save by skipping the check.
* s3api: keep conditional recovery creates off the replay path
The conditional create in the ambiguous-PUT fallback rode the replayable
WithFilerClient failover, so a committed create whose response was lost
could be re-sent to another filer — where a delete in between would let
IF_NOT_EXISTS pass again and resurrect an entry pointing at reclaimed
chunks. Conditional creates now run through the non-replayable failover;
an ambiguous result falls through to confirmCreateLanded, which resolves
the committed entry rather than resending.
* s3api: do not mark a filer unhealthy for an answered request
A reply that reached the callback — including an application-level failure
like a refused write precondition — proves the filer is alive. Checking
received before the health bookkeeping keeps definitive application errors
from tripping the failure circuit or flagging a live routed owner as
unreachable.
* s3api: do not claim an empty stored entry as an ambiguous PUT's own
Two chunkless entries compare equal under sameFileChunks, so an older
empty object satisfies "the route landed" even though it predates this
write — recovery would return success while leaving the request's
Content-Type and metadata unapplied. A landed PUT stores the entry
exactly as built, so require mime/size/md5/content and the extended
attributes to match before claiming it; anything else stays uncertain
and the client retries.
* s3api: prove ambiguous empty PUTs by write id and pin recovery to the owner
- Conditional recovery creates are sent to the routed owner alone instead of failing over: the condition only serializes with routed writes under the owner's per-path lock, and another filer's forwarding is only as correct as its ring view. An unreachable owner now yields an uncertain result rather than a create under a different serialization boundary.
- A proven-absent path no longer satisfies an empty, chunkless PUT: the routed commit may have landed and been deleted, and there are no chunks to verify, so IF_NOT_EXISTS could resurrect a removed object. Recovery returns uncertain for the client to retry.
- Chunkless identity moves from metadata equality to a per-write id (x-seaweedfs-write-id) stamped into Extended: an identical older object can match mime/md5/content but not the id. Entries without this write's id are not claimed, and confirmCreateLanded can now positively confirm a committed empty object instead of always reporting failure.
* filer: shrink zip spool threshold in tests to fix 32-bit OOM
TestSpoolZipReaderTempFile and TestSpoolZipReaderLimit each allocate
~33-70 MiB to cross the 32 MiB in-memory spool threshold. On linux/386 the
weed/filer test binary already runs near the 4 GiB address-space limit, so
these allocations reliably crash it with "out of memory" mid-suite.
Make zipSpoolMemoryLimit a var and drop it to 1 KiB in those tests, which
exercises the same memory-vs-temp-file spool paths with KiB-sized buffers.
* s3 lifecycle: update stale runShard and WalkerFunc call sites in tests
Recent master changes added a walks param to runShard (#11797) and made WalkerFunc take []int, but replay_view_test.go and run_bounded_test.go still use the old signatures, breaking go vet and test builds on merge.
* s3: count stored versions as bucket contents
A <key>.versions directory holding version entries is user data; skipping
it let bucketHasUserObjects report an empty bucket while the recursive
delete dropped every stored version.
* test: rename stale reserved-folders case in bucket delete test
.versions directories are now descended into, so the case only holds
because the folder is empty.
* s3api: hydrate runtime policies on pushed identity upserts
PutIdentity propagation (used on every IAM write when the credential
store has a master client) carries the persisted identity proto. That
proto records inline policies only as flattened legacy Actions; their
runtime __inline_policy__ names live in the policy store and were
dropped by the merge. For the interval until the next full reload, an
identity whose policy denied e.g. DeleteObject routed to legacy
CanDo, where "Write:bucket" authorizes deletes.
Hydrate the pushed identity the same way a full load does, so the
merged state always carries its runtime policy names and documents.
* s3api: hydrate runtime policies on pushed group upserts
PutGroup propagation has the same gap as identity pushes: the wire
proto does not carry the group's runtime __inline_group_policy__ names,
so members' attached-policy check lost them until the next reload.
* s3api: attach only runtime inline policy names on pushes
Listing the pushed identity's or group's own inline policies is enough
to keep it on the attached-policy route; a whole-store hydration per
pushed proto is not. Pushes now never write iam.policies, so a stale
read can no longer overwrite a newer committed policy document.
* s3api: carry inline policy documents on pushed upserts
Attaching only runtime policy names left a window where a pushed identity or group referenced a document the local cache had never loaded, denying allowed requests until the next reload. Fetch the pushed entity own inline documents and publish them with the update; reads stay scoped to the pushed entity rather than the whole store, and inline runtime names are only ever populated from this store so a read cannot install a document older than the committed one. Resync the IAM manager only when a push actually installs documents.
* s3api: order pushed inline policy reads against their installs
A pushed UpsertIdentity or PutGroup read inline policy documents from
the credential store and installed them without holding the ordering
guard that full reloads already take, so a delayed push could overwrite
a newer committed document with its stale read and restore a revoked
grant until the next update or reload.
Pushes now hold reloadMu across the store read and the install, the same
snapshot-through-commit serialization the credential-manager load uses.
MergeS3ApiConfiguration also applies config.Policies to the live
iam.policies at commit instead of swapping in its earlier snapshot, so a
policy installed while a merge prepared (a pushed inline document or a
PutPolicy update) is no longer discarded.
* s3 lifecycle: replay only the rules that fit the retention window
runShard handed drainShardEvents the full snapshot, so rules the retention
window assigns to the walk were also matched during meta-log replay. Their
matches are not due yet, and drainShardEvents freezes the cursor at the first
event with a not-yet-due match. With a 3-day and a 30-day expiration rule on one
bucket and RetentionWindow of 8 days, every shard's cursor stopped at the first
30-day object's PUT and stayed there until that object came due. Each pass
then re-read the log from that point and re-dispatched every 3-day expiration
already handled, which surfaces as NOOP_RESOLVED dispatches, a growing
cursor_lag_max, and passes that still report status=ok.
Snapshot.ReplayView drops the replay-kind actions promoted to the walk and
keeps every other action as compiled, and the drain uses it. Promoted rules
are served by the walk as before. RulesForShard, PromotedHash and ReplayView
share one isPromoted predicate. With RetentionWindow unset no active rule is
promoted, so behaviour without a configured window is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: scale the replay-view test with DaysToDuration
The test took its retention window from DaysToDuration but dated its PUTs
in hours, so under the s3tests build tag (one lifecycle day = 10s) every
event fell before the seeded cursor and replay never ran. Express the
event ages in DaysToDuration too, so the test exercises replay at both
scales.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: rehabilitate scan_only mode in the replay drain view
ReplayView shared the base CompiledAction pointers, so a replay-kind
rule locked into ModeScanOnly by a prior compile stayed invisible to
router.Route's event_driven gate even when the runtime retention window
kept it in the replay partition - and it is absent from the walk view
too, so the rule would be served by neither.
Clone non-promoted replay-kind actions the way RulesForShard does
(Mode forced to event_driven), reactivating the scan_only case whose
bootstrap rendezvous already completed, while preserving the base
active bit for genuinely pre-bootstrap actions. Disabled actions are
excluded for parity with the other views, and DESIGN.md now documents
the view.
* s3 lifecycle: keep drain activation observable, require walker with retention
ReplayView now shares pointers for event-driven actions so a mid-pass
MarkActive flip remains visible to the drain; only a scan_only-locked
action is cloned to event_driven, and reactivated only when its
bootstrap rendezvous completed (tracked via a new bootstrapComplete
field on CompiledAction).
validate() rejects RetentionWindow without Walker: promoted rules
would lose their only dispatch path while the drain cursor still
advanced past their events.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3 lifecycle: share the daily bucket walk across shards
The daily pass invoked the walker once per shard, and every shard's
walk listed the full bucket tree while keeping only its own 1/16 of
the entries — 16 full listings per pass by default (#11791).
Run now groups the shards by the walk branch they'll take (recovery
vs steady, using the same cursor data and hash checks runShard applies)
and starts one shared walk per group, so a pass lists each bucket at
most twice. WalkerFunc and WalkBuckets take a shard set instead of a
single shard. runShard keeps its own decision and waits on its group's
walk; a shard whose freshly-loaded cursor disagrees with the plan falls
back to a per-shard walk, so mid-pass cursor rewrites can't under-cover
a shard.
* s3 lifecycle: skip the .versions/ pre-pass on non-versioned buckets
walkBucketTree listed every directory twice — once looking for
.versions/ dirs, once for regular entries. On a bucket that never had
versioning enabled the first pass can only find coincidentally-named
user folders, so the extra listing is pure cost (#11791).
FilerListFunc now takes the walk view and consults
Snapshot.BucketVersioned; non-versioned buckets skip pass 1 entirely.
A *-suffixed .versions user folder on such a bucket is recursed into
as a regular directory so real objects inside still surface; a nil
view keeps the two-pass behavior.
* s3 lifecycle: prune walk directories to active rule prefixes
The daily walk listed every directory under a bucket even when no
rule prefix could reach it. Collect the active rule prefixes of each
bucket from the walk view (new Snapshot.BucketPrefixes) and skip
recursing into directories whose keys can never match; a bucket
with no active actions is skipped entirely, saving the listing.
.uploads stays in scope regardless of prefixes because AbortMPU
rules match DestKey, and .versions expansion is pruned only
conservatively (the logical key is a string-prefix of the dir key).
* s3 lifecycle: join planned walks before Run returns
planWalks starts a shared walk goroutine per group, but a shard whose
authoritative cursor load then fails exits runShard without waiting on
it. Run returned while the walk still listed or dispatched deletes
against connections the caller closes on return. Join every planned
walk after the shard waitgroup.
* s3 lifecycle: treat object-locked buckets as versioned in the walk
Object Lock requires versioning and every writer sets both extended
attributes, but IsBucketVersioned only read ExtVersioningKey. A locked
bucket missing that attribute would have its .versions dirs walked as
ordinary folders, hiding version siblings from lifecycle. Classify
ExtObjectLockEnabledKey as versioned too.
The "<dir>/" existence probe emitted the empty directory as a
FolderMimeType Contents entry, so delimited listings returned a fake
readable key that breaks clients treating every key as an object.
Delimited probes now report the directory as a CommonPrefix -- which
the probe accepts for existence -- while the recursive (no delimiter)
probe keeps the directory marker.
* s3: verify SigV4 against the request path the client sent
Internal routers that rewrite URL.Path (the Iceberg catalog doubled-/v1
collapse) broke signature verification: verifyV4Signature re-derived the
canonical URI from the rewritten path while the client had signed the
original one, so valid signed catalog requests were rejected. Stash the
client-sent path in the request context at rewrite time and prefer it when
computing pathForSignature.
* iceberg: collapse repeated /v1 prefixes before route matching
Move the doubled-prefix collapse from the catch-all into a middleware so it
also covers paths that match the {prefix} routes (/v1/v1/namespaces would
otherwise resolve "v1" as the table bucket; the name is unreachable anyway
since bucket names require 3-63 chars). Collapse all leading repetitions in
a single pass and re-dispatch once instead of recursing per segment, and
record the original path in context so signed requests still verify.
* iceberg: slice off repeated /v1 prefixes instead of concatenating
Stripping path[len("/v1"):] per iteration reuses the original string
instead of allocating a fresh copy of the remaining path each pass.
* iam: accept AWS action names for listing multipart uploads and parts
PutUserPolicy rejected s3:ListBucketMultipartUploads and
s3:ListMultipartUploadParts as invalid actions, so no inline policy could
grant these operations by their AWS names. Map both to Read, which is the
action their routes are registered under.
Fixes#11738
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm
* test: check each multipart listing action separately
Granting both actions in one policy let a correct mapping for one hide a
wrong mapping for the other. Test each action on its own and assert it
grants only Read.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm
* iam: scope multipart listing grants to their resolved action
A policy naming s3:ListBucketMultipartUploads or s3:ListMultipartUploadParts
projected to a Read:<bucket> grant, which the legacy route check honored for
every object in the bucket. Map the AWS names to dedicated action constants
and let VerifyActionPermission fall back to the resolved request action, so
the grant covers only the listing operation it names.
* iam: narrow legacy ListParts and ListMultipartUploads mappings
Both policy names projected to Write:<bucket>, granting full object writes
on any bucket where listing was intended. They now map to the same scoped
listing actions as their AWS names.
* iam: bind the multipart listing fallback to the listing routes
The legacy-action fallback in VerifyActionPermission resolved the
operation with ResolveS3Action, which reads the promoted authorization
object. For GET /bucket?uploads&prefix=foo/ the prefix becomes the
object, so the request resolved to GetObject and a
ListBucketMultipartUploads grant was denied. Resolving from query
priority also did not match how the router picks a handler.
Only fall back for the two listing routes, identified from the route
vars the same way the router dispatches them, and add the scoped
actions to MapToIdentitiesAction so GetUserPolicy in the standalone
IAM API rebuilds valid policy actions.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* s3: skip seed-signature verification in chunked reader when IAM is off
When no credentials are configured there is nothing to verify streaming
chunk signatures against. Leave the credential unset so the reader decodes
the aws-chunked body and skips per-chunk verification, matching the
unsigned-streaming path. This also fixes SigV4-signed unsigned-streaming
uploads on open gateways, which previously failed in calculateSeedSignature.
setupIam() now sets isAuthEnabled to match the credentials it installs, so
the signature-verification tests still exercise the enabled path.
* s3: decode signed streaming uploads when IAM is disabled
AWS SDK v2 clients (Trino, Spark, modern aws-cli) send PutObject bodies as
STREAMING-AWS4-HMAC-SHA256-PAYLOAD. On a gateway without S3 auth this was
rejected with ErrAuthNotSetup, breaking writes to table buckets and regular
buckets alike. The auth check already short-circuits when IAM is off, so
route both streaming payload types through the chunked reader.
* s3: cover signed trailer streaming uploads without IAM
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: add zip archive extraction helpers
SpoolZipReader makes an upload body seekable for archive/zip,
SanitizeZipMemberName rejects zip-slip and non-regular members, and
CheckZipLimits caps member count and total uncompressed size.
For seaweedfs/seaweedfs#11722
* filer: extract zip members on ?extract=zip upload
POST/PUT to a directory path with ?extract=zip (or X-Seaweedfs-Extract:
zip) writes each archive member as its own entry, so a bundle of small
files lands in one request instead of one roundtrip each. Members keep
their mtime, mode and mime-by-extension; directories are created;
unsafe names are skipped and reported. extract.keep=true also stores
the archive itself.
For seaweedfs/seaweedfs#11722
* filer: bound zip spool memory by actual bytes
A chunked upload has no Content-Length, so the old size hint let an unbounded body go through io.ReadAll. Read up to the memory threshold plus one byte first, then spill the prefix and the rest to a temp file, capped at ZipExtractMaxBytes total.
* filer: guard zip member size total against overflow
Declared member sizes summing past math.MaxUint64 wrapped the running total and slipped past the 1 TiB cap.
* server: only take the inline fast path at a short first read
uploadReaderToChunks read at most chunkSize bytes, then treated any read below SaveToFilerLimit as the whole file. When SaveToFilerLimit exceeded chunkSize, longer uploads were stored truncated.
* server: resolve storage options per extracted zip member
A member can land under a storage rule more specific than the request path. Re-running detectStorageOption0 per destination applies that rule's collection, replication, TTL, filename limit, and read-only guard to files and directories alike, while explicit query overrides still win.
* server: accept raw request body for POST extract uploads
Every POST was forced through MultipartReader, so a raw zip body always got a 400. Only parse multipart when the Content-Type is actually multipart.
* server: extract zip directories before files and stop on cancel
Processing directory members first keeps their stored mtime/mode from being overwritten by parents auto-created for an earlier file member. Checking ctx.Err() between members keeps a disconnected client from driving thousands more writes.
* server: assert stored mtime and mode for extracted members
* s3: add zip extraction header constants
* s3: extract zip members on PutObject with extract header
x-amz-meta-snowball-auto-extract or x-seaweedfs-extract: zip on a PutObject fans out the archive to one object per member under the object's directory, reusing putToFiler so chunking, collections, storage rules, versioning, and bucket-default encryption apply to each member. The archive itself is not stored. Explicit SSE request headers are rejected since one encryption config cannot represent independently created member objects.
* s3: bound member fan-out by request cancellation
Stop starting new member writes once the request context is done, matching the filer extract loop.
* filer: bound compressed zip spool and reject non-local member names
The 1 TiB uncompressed limit was also capping the compressed bytes
spooled to the host's temp directory, so a handful of concurrent
extractions could fill the filer's disk before any ZIP validation ran.
SpoolZipReader now takes a separate compressed-size limit (the filer
passes its TUS max upload size) with a 5 GiB default.
SanitizeZipMemberName gains a filepath.IsLocal check: the segment loop
already rejects "..", but CodeQL only treats member names as sanitized
through a recognized guard, and the extra check also covers reserved
Windows names.
* server: honor destination rules for zip members and clamp maxMB
An x-seaweedfs-destination header picks the storage rules for a logical
destination while members still land under the request path, matching
PostHandler. Directory members now also enforce the actual path's
read-only rule instead of relying on option resolution to fail.
maxMB up to 2048 overflowed the int32 chunk size, so an oversized query
value made extraction (and ordinary autoChunk uploads) store members
empty yet report success. The parsing is shared and clamped at 2047 MB.
* s3: scope zip member writes through the per-key checks PutObject applies
The extraction loop bypassed most of what a standalone PutObject would
face: members were written without IAM authorization for their key,
directory members replaced markers without entry lock checks, and lock
headers persisted retention without the s3:PutObjectRetention or
s3:PutObjectLegalHold permissions. authorizeMember now runs IAM,
table-bucket path rules, bucket policy, lock-header validation, and
non-versioned lock enforcement per member key, and directory members
take the same guarded mkdir path as an explicit dir marker.
The archive's own Content-Md5 and x-amz-checksum-* values are verified
while the body spools instead of being silently dropped, members stop
inheriting the archive's Content-Encoding, and non-versioned members
resolve their lifecycle TTL like normal uploads.
* s3: fail malformed lock headers before spooling the archive
Per-member re-validation turned a bad object-lock header into one error
entry per member inside a 200 response. Validate once up front like
PutObjectHandler does, and keep only the dedicated retention/legal-hold
permission checks scoped to each member key.
* filer: return 413 for oversized archives and honor saveInside in extract
Map filer.ErrZipTooLarge to 413 instead of 500 for client-side archive
size violations. Extracted members written under saveInside now store
as inline entry content instead of unreachable volume chunks.
* s3: write extracted members under their own request context
Point member writes at the member key via mux.SetURLVars so SSE-KMS
builds the encryption context from the member ARN, not the archive's.
Map the spool limit error to ErrEntityTooLarge.
* filer: add zip archive extraction helpers
SpoolZipReader makes an upload body seekable for archive/zip,
SanitizeZipMemberName rejects zip-slip and non-regular members, and
CheckZipLimits caps member count and total uncompressed size.
For seaweedfs/seaweedfs#11722
* filer: extract zip members on ?extract=zip upload
POST/PUT to a directory path with ?extract=zip (or X-Seaweedfs-Extract:
zip) writes each archive member as its own entry, so a bundle of small
files lands in one request instead of one roundtrip each. Members keep
their mtime, mode and mime-by-extension; directories are created;
unsafe names are skipped and reported. extract.keep=true also stores
the archive itself.
For seaweedfs/seaweedfs#11722
* filer: bound zip spool memory by actual bytes
A chunked upload has no Content-Length, so the old size hint let an unbounded body go through io.ReadAll. Read up to the memory threshold plus one byte first, then spill the prefix and the rest to a temp file, capped at ZipExtractMaxBytes total.
* filer: guard zip member size total against overflow
Declared member sizes summing past math.MaxUint64 wrapped the running total and slipped past the 1 TiB cap.
* server: only take the inline fast path at a short first read
uploadReaderToChunks read at most chunkSize bytes, then treated any read below SaveToFilerLimit as the whole file. When SaveToFilerLimit exceeded chunkSize, longer uploads were stored truncated.
* server: resolve storage options per extracted zip member
A member can land under a storage rule more specific than the request path. Re-running detectStorageOption0 per destination applies that rule's collection, replication, TTL, filename limit, and read-only guard to files and directories alike, while explicit query overrides still win.
* server: accept raw request body for POST extract uploads
Every POST was forced through MultipartReader, so a raw zip body always got a 400. Only parse multipart when the Content-Type is actually multipart.
* server: extract zip directories before files and stop on cancel
Processing directory members first keeps their stored mtime/mode from being overwritten by parents auto-created for an earlier file member. Checking ctx.Err() between members keeps a disconnected client from driving thousands more writes.
* server: assert stored mtime and mode for extracted members
* filer: bound compressed zip spool and reject non-local member names
The 1 TiB uncompressed limit was also capping the compressed bytes
spooled to the host's temp directory, so a handful of concurrent
extractions could fill the filer's disk before any ZIP validation ran.
SpoolZipReader now takes a separate compressed-size limit (the filer
passes its TUS max upload size) with a 5 GiB default.
SanitizeZipMemberName gains a filepath.IsLocal check: the segment loop
already rejects "..", but CodeQL only treats member names as sanitized
through a recognized guard, and the extra check also covers reserved
Windows names.
* server: honor destination rules for zip members and clamp maxMB
An x-seaweedfs-destination header picks the storage rules for a logical
destination while members still land under the request path, matching
PostHandler. Directory members now also enforce the actual path's
read-only rule instead of relying on option resolution to fail.
maxMB up to 2048 overflowed the int32 chunk size, so an oversized query
value made extraction (and ordinary autoChunk uploads) store members
empty yet report success. The parsing is shared and clamped at 2047 MB.
* filer: return 413 for oversized archives and honor saveInside in extract
Map filer.ErrZipTooLarge to 413 instead of 500 for client-side archive
size violations. Extracted members written under saveInside now store
as inline entry content instead of unreachable volume chunks.
* filer_pb: add request_id to ObjectTransactionRequest
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: replay recorded outcome for a retried ObjectTransaction
A failover replay carries the same request_id; under the same path lock
the owner replays the recorded outcome instead of re-applying the
mutations over whatever superseded them. Only application-level
outcomes are recorded — a lookup error leaves the request free to
re-run.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: stamp routed object transactions with a request id
One id per logical transaction so failover retries replay the recorded
outcome instead of re-applying the mutations.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: scope transaction replay tokens per path and keep them for their TTL
The dedup key includes the lock path so a token reused on a different
object cannot replay this outcome. Records are never evicted early —
an early eviction would let a replay re-apply — so the expiry sweep is
amortized and the map is bounded by the insert rate over the TTL.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: bound the transaction replay cache at capacity
* filer: do not record indeterminate transaction outcomes
A mutation error can be transient or follow a partial apply; recording
it replays that failure to a retry for the whole TTL instead of letting
the same idempotent mutations converge. Only successful transactions
and precondition failures are definitive outcomes worth replaying.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: replay recorded mutation failures instead of re-applying
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: let an explicit policy Deny veto fallback and owner allows
Table operations evaluated the table or namespace policy and the bucket
policy as an OR, so an explicit Deny in one policy was overridden by an
Allow in the other. Owner and identity grants also short-circuited
before the resource policy was read. Deny statements are now evaluated
first at both levels, matching AWS explicit-deny semantics.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: fail closed when a resource side has no owner
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: apply destination retention on copies instead of inheriting it
CopyObject carried the source object's retention mode, retain-until date
and legal hold into the destination entry, so a copy onto a bucket with
default COMPLIANCE retention produced an unlocked object that could be
deleted early. Copied objects now take their retention from the copy
request's lock headers or the destination bucket's default, matching
PUT semantics; a versioned self-copy gets the same treatment while an
unversioned metadata-only self-copy keeps the object's existing lock.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: validate lock headers on metadata-only self-copies
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: evaluate tag conditions on copy-source and tagging subresources
Policy evaluation ran without the object entry on several paths, so a
Deny conditioned on s3:ExistingObjectTag never matched:
- CopyObject, UploadPartCopy and RenameObject checked the source key
before its entry was fetched; repeat the check on a synthesized GET
once the entry resolves.
- GetObjectAttributes and Get/Put/DeleteObjectTagging never supplied
the entry; recheck with the resolved entry.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: fail closed on tag-policy lookups and honor tag-conditioned grants
- recheckTagConditionedPolicyWithLookup returned success when the entry
lookup errored, letting a tag-gated Deny slip past a transient miss;
only ErrNotFound still falls through to the handler's own NoSuchKey.
- A read or delete denied in the tagless first phase now gets one more
evaluation with the resolved entry's tags in hand, so an
s3:ExistingObjectTag-conditioned Allow can still grant a copy-source
read or rename. The deferred recheck requires an explicit Allow;
indeterminate results keep the original verdict, and the existence of
the key never leaks past a deny.
- RenameObject now applies the same tag-aware recheck to the source
removal, so a tag-gated Deny on s3:DeleteObject blocks a rename the
way it blocks a direct DELETE.
- GetObjectAttributes evaluates the object-entry policy recheck before
conditional headers, so a 304/412 response can no longer bypass a
tag-based Deny.
* s3: honor identity denies in the tag-aware source recheck
The deferred recheck re-evaluates only the bucket policy, so a bucket
Allow could lift an explicit Deny in the caller's attached or session
policies on a copy/rename source. Apply the same applicable-policy Deny
check the first phase uses.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: scope OIDC-derived account IDs to the token issuer
Table ownership keyed off the bare sub claim, so two identity providers
issuing the same sub collided on one account - a caller could act on
tables owned by a different provider's user. OIDC bearer tokens now
carry the same issuer-namespaced (sub, iss) hash STS sessions already
expose as aws:userid, and getAccountID prefers it over the raw claims.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: keep legacy owners working and scrub client-subject claims
- OIDC bearer tokens without a subject are rejected, and a token-supplied
aws:userid claim is always overwritten with the verified (sub, iss)
hash, so a forged claim can no longer stand in for the principal.
- Owners written before issuer scoping hold a bare subject; permission
checks now re-anchor that subject under the caller's issuer, so the
same (sub, iss) pair keeps its ownership while scoped owners remain
issuer-isolated. The token issuer is asserted into claims alongside
sub and carried through PolicyContext.
* s3tables: anchor legacy subjects to the configured issuer set
* s3: enumerate trusted issuers from registered providers too
* s3: fail closed when the OIDC provider store cannot be listed
trustedOIDCIssuers kept the registered issuers when ListOIDCProviders
failed, presenting a partial set as complete. A deployment trusting one
registered issuer A plus a stored issuer B would, on a listing error,
report only A — the single-issuer shape under which a legacy bare
subject re-anchors — letting a caller from A claim a stored subject
that belongs to B and gain owner access.
With no provider store the registered providers are provably complete,
so keep returning them; with a store, a failed listing means the real
set is unknown, so return none.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: keep suspended-versioning conditional writes off the routed condition
A suspended PUT writes the null-version slot at the regular path while the
entry the conditional headers evaluate lives under .versions/. The routed
WriteCondition checked the request's etag against that empty slot, so a
correct If-Match failed with 412. Passing uniqueWritePath sends conditional
requests through the lock path, whose precondition resolves the same entry
the read path serves; unconditional writes still route.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer_pb: add condition_alt_key to ObjectTransactionRequest
* filer: evaluate a transaction condition against a fallback entry
condition_alt_key names the entry consulted when the primary is absent,
so a suspended-versioning write can check 'the null slot if present,
else the latest version' atomically under the lock.
* s3: route suspended-versioning conditional writes with a fallback key
The gateway names the latest version file as condition_alt_key, so the
transaction checks the entry the read path serves — null slot first,
latest version when it is absent — under the same lock that applies
the write. Conditional suspended PUTs route again instead of splitting
check and write across a gateway lock that routed writes skip.
* filer: resolve the condition fallback under the transaction lock
The gateway named the pointed-at version file as condition_alt_key,
resolved before the transaction ran: a concurrent version delete could
move the latest pointer between the gateway read and the locked check,
and a delete-marker latest read as an existing object. The alt key is
now the .versions directory and the filer resolves the latest-version
pointer under the same lock that serializes the write — a delete marker
resolves to absent, and unresolvable pointer state fails closed.
* filer: fail closed on alt-key lookups and skip directory primaries
A transient FindEntry error returning a nil entry read as "versions dir
absent" and let If-None-Match writes pass. Check the error before the
entry. And a bare directory at the object path satisfied the primary
lookup, so the .versions fallback never ran — directories are not the
object, treat them as an absent slot.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: scope directory-as-absent conditions to alt-key callers
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.8.0 to 1.8.2.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.8.0...sdk/azidentity/v1.8.2)
---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
dependency-version: 1.8.2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* test: restore loadBucketMetadataFromFiler stub after TestGetBucketMetadata
TestGetBucketMetadata replaced the package-level loadBucketMetadataFromFiler
with a stub that sleeps one second per load and never restored it. Every
test running after it in the same binary paid that sleep on each bucket
metadata cache miss: TestPutObjectUploadACL (~245s) and
TestCopyObjectDestinationACL (~46s) each created a fresh BucketRegistry per
subtest, so the stub fired hundreds of times. In CI this stretched the
Test job past 11 minutes and left it exposed to runner cancellation.
Restore the real loader via t.Cleanup so the slow stub is scoped to the
test that needs it.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
The local-peer fallback and bearer-token validation used by the filer IAM
gRPC path move into weed/security so every admin-plane gRPC service can
enforce the same contract. A client certificate verified by the server's
TLS config now also counts as authenticated.
CreateMultipartUpload accepts canned and custom ACLs on initiation;
run them through the same authorization path as PutObject so an
explicit ACL requires s3:PutObjectAcl, and persist the validated
grants on the upload entry for completion to apply.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Multipart initiation stored client-supplied Seaweed-X-Amz-Acl and
Seaweed-X-Amz-Owner request headers onto the upload entry, and
CompleteMultipartUpload copied them to the finished object, letting a
caller forge ownership or a public ACL without s3:PutObjectAcl.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The decoupled-delete guard only inspects the resolved data path itself,
but the catalog marker lives on a table's name path, so a location that
resolves inside another table's directory (its marker-less data
subdirectory) slips through and the recursive purge destroys the other
table's data. Walk the data path's ancestors within the bucket and
refuse when any carries a table or view catalog entry.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
CreateTable/RegisterTable/UpdateTable (and the view equivalents) validate
the location only for bucket and traversal, so a tenant can register a
table whose location resolves into another namespace's directory and
DeleteTable will purge it. Require the location to live under the
caller's namespace subtree, the boundary per-namespace delegation uses.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: preserve remote object ETags across cache fills
* s3api: align remote copy validators and reject stale ETags
* s3api: keep remote ETag through metadata touches, drop it after resizes
The remote-ETag check required Mtime to match RemoteMtime, so a utimens
on a cold remote object returned an empty ETag until a later cache fill
restored the remote value. It also trusted LastLocalSyncTsNs alone, so a
setattr resize that leaves chunks and the stamp untouched kept serving
the old remote ETag for different local bytes.
Require FileSize to match RemoteSize for both the synced and remote-only
paths and stop comparing modification times: content changes already
surface through cleared sync stamps, chunks, or inline content, while a
metadata-only touch leaves the remote bytes authoritative. Regression
tests cover touched cold and cached entries and resized cached entries.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* s3api: register the azurekms KMS provider under its build tag
weed/kms/azure registers itself with weed/kms from an init(), but the
blank import that triggers it was commented out in auth_credentials.go
(TODO: Fix Azure SDK compatibility issues). Nothing else imports the
package, so building with -tags azurekms still ends with
"KMS provider 'azure' not registered" at startup for any config using
"type": "azure".
Move the import into weed/s3api/kms_azure.go behind //go:build azurekms so
default builds stay free of the Azure SDK, and drop the dead comment.
Closes#11686
* ci: compile and test azurekms-gated code
weed/kms/azure is excluded from every default build, so nothing in CI
compiled it; that is how the provider shipped unregistered. Build the
tree and run the kms tests with -tags azurekms on every Go change.
---------
Co-authored-by: Yi-111-a <>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Chris Lu