* s3: evaluate bucket policy before ACL public-read for anonymous requests
AuthWithPublicRead granted anonymous access on a public-read ACL before
consulting the bucket policy, so an explicit Deny (e.g. s3:ListBucket)
was skipped for anonymous callers while still enforced for authenticated
ones. Run the policy engine first: a matching Deny or Allow is honored,
otherwise fall through to the ACL grant as before.
* s3: defer object-level anonymous requests to the handler's policy recheck
Evaluating the bucket policy with a nil entry at middleware time makes
tag conditions like s3:ExistingObjectTag/<key> resolve against missing
values, so a conditional Deny could wrongly block anonymous Get/Head on
a public bucket whose handler recheck would permit it. Object requests
now take the ACL grant and let Get/HeadObjectHandler re-evaluate with
the fetched entry; only bucket-level requests (List, HeadBucket), which
have no such recheck, are decided by the middleware policy verdict.
Reading the bucket config first also refreshes the compiled policy on a
cache miss, so a remotely deleted policy cannot leave a stale verdict
in the engine for nonresident buckets.
* s3: recheck bucket policy before serving directory objects
handleDirectoryObjectRequest runs before the object handlers' policy
recheck, so directory content on a public-read bucket was served to
anonymous callers without any policy evaluation. Evaluate the policy
with the directory entry, matching the recheck the file path performs.
* s3api: add Snowflake s3compat API integration tests
Run the upstream snowflakedb/snowflake-s3compat-api-test-suite against a
local SeaweedFS server in CI. test/s3/snowflake/run.sh starts weed server
with S3 (-s3.autoCreateBucket=false so missing-bucket PUTs return
NoSuchBucket), prepares the fixtures the suite needs (versioned bucket,
deny-all-policy bucket, >1000-object prefix), clones the suite, patches
it to path-style addressing, and runs mvn -Dtest=S3CompatApiTest.
The suite also exposed that GetBucketLocation returned 404 NoSuchBucket
for a malformed bucket name; validate the name first and return
400 InvalidBucketName like AWS.
* test: harden snowflake s3compat runner per review
- Pin the upstream suite to a tested commit (SUITE_REV) instead of the
moving default branch
- Bind the test server to loopback only
- Require the AccessDenied error code when verifying the denied bucket
- Fix README so go install runs in a subshell
- checkout with persist-credentials: false
- Make the concurrency group unique per PR, and widen path filters to
the storage/operation/wdclient/cluster/pb packages the S3 stack uses
* test: advertise loopback ip for snowflake test server
-ip.bind 127.0.0.1 alone left the volume server advertising the host's
primary address, so chunk uploads were refused. Also set -ip 127.0.0.1
and disable the Iceberg/Lance listeners so the harness is loopback-only
and does not collide with other local services.
* s3api: fail closed when S3 Tables signature verification fails
* s3api: avoid nil Account dereference in S3 Tables auth log
* iceberg: return auth error instead of falling back to DefaultAllow
* lance: return auth error instead of falling back to DefaultAllow
* s3api: stop trusting client-supplied s3-account-id
The header is set by the server after successful authentication; scrub
inbound values alongside the other internal headers, and apply the same
admin guard to the header fallback branch of getAccountID that the
identity branch already has.
* test: cover table-catalog auth wrappers and principal resolution
* test: configure anonymous identity where catalog clients do not sign
* s3api: scrub s3-account-id after signature verification
* s3: add option to disable bucket auto-creation on upload
* command: expose -autoCreateBucket in s3, filer, server, and mini
* s3: apply the bucket auto-create policy to directory marker uploads
* s3: validate the bucket name before the auto-create disabled check
* s3: cover the disabled auto-create gate at all three upload entry points
* s3: report the effective ownership when a bucket has none stored
GetBucketOwnershipControls read Seaweed-X-Amz-Ownership straight out of the
bucket entry, so a bucket that never had one written reported an empty
ObjectOwnership. The object write path defaults the same missing attribute to
BucketOwnerEnforced, so the API contradicted the behavior it describes.
Resolve the stored value through one helper both readers share, and let
PutBucketOwnershipControls persist unconditionally so setting the default
value still gives DeleteBucketOwnershipControls something to remove.
* test: cover the bucket ownership controls round trip
Pins the behaviors the ownership default fix depends on: a bucket that never
had ownership controls written reports BucketOwnerEnforced, and putting that
same value on such a bucket still persists it, so the delete that follows has
something to remove. The put-then-delete case gets its own bucket -- run after
an ObjectWriter put, it would pass against an implementation that skips only
the initial write.
The acl workflow already runs this package against a live weed mini, so it
needs no wiring.
* s3: stop warming every bucket's config at startup
Listing all buckets in BucketRegistry.init() made S3 gateway startup
O(buckets) and pinned every bucket's metadata and config resident, which
does not scale past a few hundred thousand buckets. Both caches already
have lazy miss paths, so load on first access instead and let the
metadata subscription refresh only entries already resident; cold
buckets cost one filer round-trip on their first request.
* s3: bound the per-bucket caches with LRU eviction
The bucket config cache, bucket registry, and their negative caches were
plain maps that only ever grew: the config cache TTL made Get miss but
never evicted the entry, and the not-found sets grew on every probe of a
nonexistent bucket name. Cap all four at 65536 entries with LRU eviction
so a gateway keeps its hot working set and evicted buckets reload from
the filer on next access.
* s3: cache parsed bucket config instead of the full filer entry
Each cached BucketConfig retained the whole bucket entry (extended
attribute map plus raw content bytes) alongside the fields parsed from
it, roughly doubling per-bucket cache cost and keeping data the read
path never looks at. Parse everything up front in
newBucketConfigFromEntry - now also the creator identity, tags,
encryption config, and stored lifecycle XML - and drop the entry.
updateBucketConfig now reads the entry fresh from the filer and diffs
the mapped extended attributes against it, so the patch is computed
against current state instead of a cached copy; the config clone
helpers that existed for that path go away.
* s3: dedup cold bucket-registry loads per bucket
The registry's notFound lock doubled as the load serializer, holding one
global mutex across the filer round-trip so first-touch requests for
different buckets queued behind each other; the cache fill also happened
after the lock was released, so two concurrent misses for the same
bucket could both reach the filer. Replace it with a singleflight per
bucket that fills the cache inside the flight: different buckets load
concurrently, the same bucket loads once.
* s3: return BucketAlreadyOwnedByYou when recreating your own bucket
PutBucket returned BucketAlreadyExists for every existing bucket, even
when the caller already owns it, so idempotent re-creation (e.g. a
container that creates its bucket on startup) couldn't tell "someone
else took the name" from "it's already mine".
Recreating a bucket you own now returns BucketAlreadyOwnedByYou, unless
the request conflicts with the existing bucket: a different Object Lock
setting, or an ACL on the request or the existing bucket. To detect the
latter, a requested non-default canned/grant ACL is now persisted on
creation instead of being dropped.
* s3: fail PutBucket when the existing bucket's config can't be read
When a bucket already exists, an unreadable config left the recreate
defaulting to BucketAlreadyOwnedByYou, masking the backend error and
possibly accepting a conflicting recreate (Object Lock / ACL unknown).
Surface the read error instead.
* s3: return the stored bucket ACL from GetBucketAcl
GetBucketAcl always returned the owner's default full-control grant and
ignored any stored ACL, so a bucket created with a canned ACL or one set
via PutBucketAcl never read back correctly. Decode the stored grants
instead, sharing one grants-to-XML helper with the object ACL handler.
The shared helper also emits each grantee's real xsi:type (e.g. Group for
public-read) instead of a hardcoded CanonicalUser, so group grants read
back correctly for both bucket and object ACLs.
* s3: resolve the right already-exists error on the concurrent-create race
When two requests create the same bucket at once, the loser's mkdir
fails and the handler fell back to a flat BucketAlreadyExists, bypassing
the same-owner idempotency check. Route both the pre-check and the race
fallback through one existingBucketError helper so a same-owner recreate
still gets BucketAlreadyOwnedByYou.
* s3: record the bucket owner's account id at creation
setBucketOwner only stored the creating identity name, so the canonical
account id wasn't available later. Persist it under ExtAmzOwnerKey too,
the same field PutBucketAcl writes, so the bucket owner can be reported
independently of whoever reads it.
* s3: report the bucket owner from GetBucketAcl, not the caller
GetBucketAcl built the ACL Owner from the caller's account header, so an
admin or cross-account read returned the wrong owner. Use the owner
persisted on the bucket, falling back to the caller only when none is
recorded.
* feat(s3): add four bucket configuration handlers
- GetBucketPolicyStatus: computes IsPublic from the existing bucket policy
- PutBucketRequestPayment: companion writer to the existing GET; accepts
only BucketOwner
- GetBucketAccelerateConfiguration: returns <Status>Suspended</Status>
- GetBucketLogging: returns an empty BucketLoggingStatus
Lets AWS SDK probes succeed instead of returning MethodNotAllowed.
* review: route GetBucketPolicyStatus through checkBucket
Mirrors the existence/auth gating used by other bucket handlers and
drops the bespoke filer_pb lookup so NoSuchBucket precedence is
consistent across the API surface.
* review: cap PutBucketRequestPayment body with MaxBytesReader
The body is unmarshalled as RequestPaymentConfiguration, which is a
handful of bytes; reject excessively large payloads up front and
defer Close immediately after wrapping.
* review: gate static getters on checkBucket
GetBucketAccelerateConfiguration and GetBucketLogging now run the
standard bucket existence check before returning the static
Suspended / empty-status response so a missing bucket cannot appear
to have valid configuration.
* review: share cache helper across misc tests; check io.ReadAll error
Accelerate and Logging tests now run through newMiscTestServer like
the others so the checkBucket guard sees a cached bucket; the
ReadAll error is explicitly checked.
Chris Lu