mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-03 10:37:35 +01:00
ec968aa0165ce0df0959fd4482b61d50e2eb006e
31
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
ec968aa016 |
s3vectors: support TagResource, UntagResource and ListTagsForResource (#12335)
* s3vectors: tag vector buckets and indexes
TagResource, UntagResource and ListTagsForResource answer on
/tags/{resourceArn} (POST, DELETE, GET) and by name. Each authorizes its
own s3vectors action on the bucket or index ARN, then reads or writes the
catalog record's tags. A resource holds at most 50 tags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: apply the tags passed to CreateVectorBucket and CreateIndex
The tags field was accepted and dropped. It is now stored on the new
bucket or index, and creating with tags needs s3vectors:TagResource in
addition to the create action.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: cover resource tagging end to end
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: keep tag operations to a bound table bucket
TagResource, UntagResource and ListTagsForResource run through Execute
with WithTableBucketCreatedAt now answer not found when the bucket name
holds another bucket, as the bound creates and deletes already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bind tagging to the authorized bucket and to vector indexes
The catalog tag calls carry the creation time of the bucket authorizeVector
approved, so a bucket created again under the name is not touched. An index
ARN must name a Lance table: another table in the vector namespace of a
format-less bucket is refused, as DeleteIndex does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: match a TagResource grant scoped to the bucket being created
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: merge tag writes into the tags the record holds at the write
TagResource and UntagResource merged into the tags they read before the
compare-and-swap and wrote that map on every attempt, so a tag write that
landed in between was undone. The merge now runs inside the mutation.
WithMaxTags lets an in-process caller cap the merged set there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: check the 50-tag limit inside the catalog's tag write
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: check a tag operation's bound bucket after its reads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: validate each tag key UntagResource removes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: validate untag keys one at a time
UntagResource sized a map by the number of submitted tag keys before
validating them, so a large body of repeated keys allocated far more than
the request itself, before authorization ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: authorize UntagResource before validating its keys
Validating a near-32 MiB body of tag keys costs about 13x its JSON
decode, so an unauthorized caller could make the gateway spend that
work on a request it then refuses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Chris LuandClaude Opus 5.5
|
66a6598fdd |
fix(s3tables,s3vectors): bind table and vector bucket writes to the bucket they were authorized against (#12329)
* s3tables: bind table bucket policy writes to the authorized bucket PutTableBucketPolicy compared only the policy bytes, and DeleteTableBucketPolicy removed the policy unconditionally. A bucket deleted and created again under the same name between authorization and the write took a policy change nobody authorized for it. Both writes now require the bucket's CreatedAt and the policy they were authorized against, answer 409 otherwise and 404 for a bucket that is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: bind bucket tag writes to the authorized bucket TagResource and UntagResource checked the creation time only for a table ARN. On a bucket ARN the merged tags landed on a bucket deleted and created again since authorization. extractResourceOwnerAndBucket now returns the resource's CreatedAt for both kinds, and the write requires it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: bind bucket maintenance writes to the authorized bucket PutTableBucketMaintenanceConfiguration wrote onto whatever bucket held the name at write time. authorizeMaintenanceBucket now returns the bucket's CreatedAt like authorizeMaintenanceTable, and putMaintenanceConfiguration takes the check and creation time from both callers instead of an optional closure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: bind vector bucket policy writes to the authorized bucket Put/DeleteVectorBucketPolicy authorized inside the compare-and-swap with the owner from an earlier bucket read, so a bucket deleted and created again by another account in between had its policy rewritten by the old owner. The mutation now requires the entry's createdAt to match the record it authorized against. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: bind DeleteVectorBucket to the authorized bucket DeleteVectorBucket authorized one bucket record, then ran the trusted catalog DeleteTableBucket on whatever bucket held the name by then, so an empty bucket deleted and created again by another account in between was deleted by the old owner. The catalog delete now accepts the authorized bucket's createdAt through an in-process context value and treats any other bucket as gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: bind CreateIndex and DeleteIndex to the authorized bucket Both authorized the bucket, then ran trusted CreateNamespace, CreateTable or DeleteTable on whatever bucket held the name by then, so an index could be created in, or deleted from, a bucket another account created again in between. Those catalog operations now honor the bucket binding DeleteTableBucket already takes, and the vector handlers pass the authorized bucket's createdAt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4850def831 |
fix(s3tables): bind table and view writes to the record they were authorized against (#12328)
* s3tables: bind table and view writes to the record they were authorized against * s3tables: bind UpdateNamespace to the namespace it was authorized against UpdateNamespace writes back the whole record it read, owner and creation time included. A namespace deleted and created again before the write took the first one's owner and CreatedAt, which also undoes the CreatedAt binding creates now rely on. Refuse the write with 409 when the namespace at the path has another CreatedAt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nguyễn Lộc Phúcanduser.email
|
7d8b72c2b4 |
fix(s3tables): answer 404 for a table name that was renamed away (#12258)
* s3tables: answer 404 for a table name that was renamed away A rename leaves the source directory in place to hold the data under it, with its catalog attributes cleared. UpdateTable, DeleteTable, the table policy calls and the table maintenance calls read the metadata attribute of that directory and reported its absence as an internal error, so they answered 500, and GetTablePolicy answered NoSuchPolicy. An absent metadata attribute now reads as a missing entry. tableMetadata reports it as both filer_pb.ErrNotFound and the attribute error, so callers that test for either keep working. The policy and maintenance handlers read the attribute through the new readTableMetadataAttr, which reports filer_pb.ErrNotFound, so each call answers NoSuchTable. * s3tables: a view record without its metadata reads absent too * s3tables: refuse a catalog write that lands on a metadata-less record --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Nguyễn Lộc Phúcandchrislusf
|
eba2cb926d |
fix(s3tables): refuse a create that a namespace delete can lose (#12239)
* s3tables: refuse a create that a namespace delete can lose DeleteNamespace checks the namespace for children and then deletes its directory recursively in a second filer call. The filer locks each path on its own, so a table created under the namespace between the two is not seen by the check and is removed by the delete, while both requests answer 200. The mirror case loses the table too: a create that passed its namespace check before the delete finished re-created the namespace directory through the filer's parent creation and recorded a table in a namespace that no longer exists. Each side now writes before it reads. DeleteNamespace claims the namespace record with a fresh token before it scans, deletes only while the record still holds that claim, and lifts its own claim when it stops. CreateTable, RegisterTable and CreateView re-read the namespace after writing their record and back out when it is claimed or gone: they remove what they wrote and answer 409 or 404 instead of reporting an entry that does not survive, or 500 when the record could not be removed and may remain. An entry the scan missed was written after the claim, so its create sees the claim; a create and a delete can both fail, but no longer both succeed. A create is also refused up front while a claim is held, but only after its authorization check, so a caller that may not create there does not learn of the delete. A create writes its record marked "create" and lifts the mark once the namespace is confirmed open. Until then GetTable, GetView, UpdateTable and a second create treat the record as absent, so no request is told about a table or view that the first create may still back out. * s3tables: keep a create's unfinished record out of reads and listings * s3tables: refuse catalog writes on a record an operation claimed * s3tables: answer 404 when a create's write path loses the namespace * s3tables: confirm the rename destination's namespace like a create * s3tables: cover unfinished-record reads, claimed-record writes, rename claim * s3tables: lift a leftover claim when a no-token delete clears the record * s3tables: answer 409 when a claimed record turns away a write UpdateView, TagResource, and UntagResource surfaced ErrConcurrentUpdate as a 500, so an expected conflict looked like a server fault to the caller. Map it to Conflict the way UpdateTable and the policy handlers already do. * s3tables: name the lost source when a rename finds it deleted * s3tables: report an entry deleted before its update as not found The filer answers an UpdateEntry on an entry removed since the lookup with codes.NotFound. mutateEntryExtended returned that error unchanged instead of filer_pb.ErrNotFound, so a rename whose source was deleted mid-claim answered 500 instead of NoSuchTable or NoSuchView. MemFiler now answers a missing entry the same way, so the rename tests for tables and views take the real filer's path. * s3tables: unpublish the destination when a rename's source clear fails * s3tables: refuse to delete a record another operation claimed * s3tables: release catalog claims on an uncanceled cleanup context * s3tables: answer 404 when a racing delete already removed the table * s3tables: keep the rename destination marked until the source clears * s3tables: guard the rename rollback against a recreated source and an unfinished destination * s3tables: drop the unfinished destination only once the source is restored * s3tables: re-resolve the purge target on the record a delete claims * s3tables: cover a delete that meets a moved metadata location * s3tables: answer 409 when a create's record changed under it * s3tables: adopt a record still marked by its unfinished create * s3tables: skip unfinished creates in the namespace empty check * s3tables: delete only the record the caller was authorized on DeleteView and DeleteTable check permissions on the record they read first, then read the name again to claim it. A record that took the name in between, such as a table or another owner's view created after a concurrent delete, was claimed and removed on the first record's authorization, and DeleteView never compared the caller's version token with it. The second read now repeats the first one's checks: a record of the other kind or an unfinished create answers 404, and a token that no longer matches answers 409. A delete without a token also compares the record's version token with the authorized one and answers 409 when they differ. * s3tables: re-check the parent after a create lifts its mark A create that passed checkParentOpen and paused before lifting its mark could still confirm while a namespace delete held the claim: the delete scans the still-marked record, skips it, and removes it under a create that already answered success. confirmParent now re-reads the same parent record after the mark clears and backs the confirmed record out when the namespace moved on. * s3tables: mark the rename destination with a claim of its own claimCatalogEntry adopted every create-marked record, including a rename's destination: a create landing after the source cleared could replace the destination before the rename published it, and with the source name reused the restore could leave the table on neither name. The destination now marks with the rename's own claim, which readers still treat as unpublished but a create must refuse, and the commit re-checks the destination namespace after the record goes live. The source keeps its rename claim until the commit lands so a namespace delete scanning the cleared record still refuses. * s3tables: compare the parent's createdAt, not its editable metadata A namespace update rewrites ExtendedKeyMetadata in place, so a create or rename whose second parent read landed across an UpdateNamespace saw a changed blob and reported the still-live namespace as gone: creates backed their record out and answered 404, renames undid their destination. createdAt survives the edit but changes on a delete+recreate, so it still identifies the record. * s3tables: retry lifting the rename's claim off the cleared source The release's only failure mode is a transient filer error — nothing else can write the cleared record — yet a single loss left the claim behind: the old name stayed refused to creates and the namespace counted the record as a child. Retry the release before warning. * s3tables: page the namespace scan before deleting it A page of skippable entries — bare directories or unfinished creates — ended the scan with hasChildren false, so a table on a later page was removed with the namespace. Keep listing with StartFromFileName until a child shows or the listing ends. --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> |
||
|
|
eecc461b39 |
s3vectors: put, get and delete vector bucket policies (#12201)
* util: do not bind outbound dials to a loopback -ip.bind `weed server -ip.bind=127.0.0.1` keeps every listener local, but the same address became the source of outbound connections, so the filer dialled its remote mount from 127.0.0.1 and failed with "connect: invalid argument". A loopback source cannot reach off the host; leave the source to the OS, as for a wildcard bind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: put, get and delete vector bucket policies The AWS S3 Vectors policy operations had no handlers; the policy document is stored on the shared table bucket, so the handlers delegate to the catalog's PutTableBucketPolicy, GetTableBucketPolicy and DeleteTableBucketPolicy and keep its guard. * s3vectors: treat ICEBERG buckets as absent in policy handlers GetVectorBucketPolicy/PutVectorBucketPolicy/DeleteVectorBucketPolicy delegated straight to the table bucket, so a bucket created for ICEBERG tables — invisible to every other vectors operation — could still have its policy read and written through the vectors API. Gate all three on lanceBucket, the same format check DeleteVectorBucket and CreateIndex already apply. * test/vectors: cover vector bucket policy put/get/delete Round-trips a policy document through the new ops and asserts a get on the deleted policy surfaces NotFoundException, so the boto3 conformance run exercises the added handlers end to end. * s3vectors: validate the bucket ARN and policy document Bucket-level operations now reject an index ARN and an ARN minted for another service, and PutVectorBucketPolicy refuses a document carrying s3vectors actions — the evaluator only honors s3tables actions, so such a rule would be a dead grant. * s3vectors: validate the bucket ARN on the remaining bucket-level operations * s3vectors: cover bucket ARN and policy document validation TestBucketARNs pins the accepted forms and the index/foreign-service rejections; TestValidatePolicyDocument covers a valid document, a malformed one, and the dead s3vectors action namespace. * s3vectors test: use an enforceable action in the bucket policy test The catalog refuses s3vectors actions in a stored policy, so the put call rejected the document before the get and delete assertions ran. --------- Co-authored-by: Ólafur Bjarki Bogason <olafur@genkiinstruments.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandDevin
|
8c3d8fcea0 |
s3tables: keep absent-target answers identical to denied ones (#11905)
* s3tables: authorize before revealing a list or tag target exists ListTables, ListViews and the tag operations read the target's catalog metadata before any permission check, so a denied caller could tell an existing name (403 or the resource's own error) from an absent one (an empty list, or 404). Evaluate the operation's permission with whatever bucket context could be read and answer a denied caller the same way for absent, damaged and denied targets. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: answer bucket reads uniformly when the entry is not a table bucket A plain S3 bucket directory answers GetTableBucket and ListNamespaces with 500 attribute-not-found while a missing name gets 404, and a denied CreateNamespace is 403 while a missing bucket is 404. Run the permission check before the absence answer and read not-a-table-bucket as NoSuchTableBucket. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: cover absent-target answers matching denied ones Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
e7923c14d1 |
s3tables: honor resource policy Deny on remaining operations (#11825)
* s3tables: honor a table policy Deny for maintenance operations authorizeMaintenanceTable evaluated only the bucket policy, so a Deny in the table policy never vetoed PutTableMaintenanceConfiguration, GetTableMaintenanceConfiguration or GetTableMaintenanceJobStatus even though they act on the table ARN. The stored table policy is now checked for a matching Deny first via deniedByPolicy, a veto helper that keeps the admin exemption, and resource and bucket tags are loaded into the policy context so tag-conditioned Denies evaluate. readBucketPolicy is renamed to readPolicy and reused for the table path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: honor the stored policy's Deny on policy management operations PutTablePolicy, GetTablePolicy and DeleteTablePolicy only evaluated the bucket policy, so a Deny in the table's own policy was ignored and a bucket-level Allow could strip or replace the policy guarding a table. The existing table policy is now loaded and a matching Deny vetoes the call. PutTableBucketPolicy likewise consults the existing bucket policy for a Deny before replacing it. Resource and bucket tags populate the policy context so tag-conditioned Denies evaluate, and policy writes assert the policy authorized against is still in force, returning a conflict when it changed mid-request. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: apply a table policy Deny to tag operations TagResource, ListTagsForResource and UntagResource evaluated only the bucket policy; a Deny in the table's own policy is now honored when the target ARN is a table. ListTagsForResource and UntagResource also returned early when the resource carried no tags attribute, skipping the permission check entirely; the check now always runs. Policy evaluation uses the canonical resource ARN rebuilt from stored metadata, so a request ARN with a forged account or region cannot dodge the Deny. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: regression tests for resource policy denies Cover the case where a bucket policy grants an identity s3tables:* while the table policy denies it: the policy, tag and maintenance operations must stay denied so the protection cannot be stripped. Also cover a bucket policy that denies PutTableBucketPolicy under the open fallback, and a tag-conditioned Deny, which vetoes only while the stored tag matches. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
8c55425d55 |
s3tables: tighten legacy OIDC subject re-anchoring (#11808)
* sts: recognize issuer-scoped parent user ids
ComputeParentUser output has a fixed shape (base64url SHA-256); let
callers distinguish it from bare subject strings.
* s3tables: keep legacy owner re-anchoring off local and scoped account ids
Re-anchoring a stored bare subject under the single trusted issuer must
only apply to strings that could actually be a legacy OIDC subject. A
stored owner naming the reserved or a configured local account, or an
already issuer-scoped id, can otherwise be claimed by any caller whose
token asserts that string as sub.
* s3tables: re-anchor legacy policy principals in map and array forms
A stored bare subject can appear as {"AWS": sub} or inside a principal
list, not only as a top-level string; re-anchoring only the string form
left legacy Deny statements unable to match the caller.
* s3tables: keep built-in legacy exclusions independent of the account checker
A missing local-account predicate must not disable re-anchoring entirely:
the reserved accounts, ARN-shaped values, and scoped-id shapes are never
OIDC subjects regardless, while legacy Deny statements for plain subjects
still need to resolve so a denied caller cannot fall through to an
identity grant.
* s3tables: exercise legacy Deny against an owner grant
The previous case denied the caller regardless of whether the statement
principal matched; make the caller the owner so a missed match shows.
Cover the array Deny form alongside the AWS map.
|
||
|
Chris LuandDevin
|
446e6f3444 |
s3tables: scope OIDC-derived account IDs to the token issuer (#11774)
* s3tables: scope OIDC-derived account IDs to the token issuer Table ownership keyed off the bare sub claim, so two identity providers issuing the same sub collided on one account - a caller could act on tables owned by a different provider's user. OIDC bearer tokens now carry the same issuer-namespaced (sub, iss) hash STS sessions already expose as aws:userid, and getAccountID prefers it over the raw claims. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: keep legacy owners working and scrub client-subject claims - OIDC bearer tokens without a subject are rejected, and a token-supplied aws:userid claim is always overwritten with the verified (sub, iss) hash, so a forged claim can no longer stand in for the principal. - Owners written before issuer scoping hold a bare subject; permission checks now re-anchor that subject under the caller's issuer, so the same (sub, iss) pair keeps its ownership while scoped owners remain issuer-isolated. The token issuer is asserted into claims alongside sub and carried through PolicyContext. * s3tables: anchor legacy subjects to the configured issuer set * s3: enumerate trusted issuers from registered providers too * s3: fail closed when the OIDC provider store cannot be listed trustedOIDCIssuers kept the registered issuers when ListOIDCProviders failed, presenting a partial set as complete. A deployment trusting one registered issuer A plus a stored issuer B would, on a listing error, report only A — the single-issuer shape under which a legacy bare subject re-anchors — letting a caller from A claim a stored subject that belongs to B and gain owner access. With no provider store the registered providers are provably complete, so keep returning them; with a store, a failed listing means the real set is unknown, so return none. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
b13463880c |
s3tables: scope management authorization to the caller's identity (#9961)
* s3tables: resolve account-less identities to a distinct principal Static identities with no account block default to the shared admin account, so getAccountID returned "admin" for every such user and the permission checks treated them all as the admin principal. Only keep the admin account when the identity actually carries an admin action; otherwise fall back to the unique identity name. * s3tables: limit the open-by-default fallback to anonymous access The legacy permission path allowed any request that no policy explicitly denied whenever default-allow was on, which is the zero-config default. That let an authenticated identity without table permissions reach table resources owned by others. Restrict the fallback to requests with no identity or the anonymous identity; authenticated callers must pass an explicit action or policy check. Zero-config and anonymous access are unchanged. * s3tables: drop the no-op ListTableBuckets account gate The top-level check passed the principal as its own owner, so it always allowed. Per-bucket filtering in the loop is the real authority; remove the dead gate and the now-unused locals. * s3tables: derive the Iceberg catalog's default-allow from auth state The Iceberg catalog reuses the S3 Tables Manager, which hardcoded default-allow on. Authenticated callers were enforced only because the identity struct happens to propagate into the handler; if it were ever dropped, a secured catalog would fall open. Mirror the S3 port and set the Manager's default-allow from the authenticator, so an authenticated caller is enforced regardless. Shell and admin keep their own trusted Manager. Regression test covers the struct, name-only, and admin paths. * s3tables: drop redundant ACTION_ADMIN string conversion ACTION_ADMIN is an untyped string constant, so the conversion is a no-op. * s3tables: enforce name-only authenticated callers, add trusted bypass defaultAllowFor treated a request with no identity object as anonymous, but the Manager path forwards only the identity name (not the struct). A name-only authenticated caller could therefore be misclassified as anonymous and allowed under the open default. Treat a server-set identity name as authenticated too, and add an explicit trusted flag for the local shell/admin tooling that legitimately bypasses authorization. * s3tables: trim verbose comments |
||
|
Chris Lu
|
0d8588e3ae |
S3: Implement IAM defaults and STS signing key fallback (#8348)
* S3: Implement IAM defaults and STS signing key fallback logic * S3: Refactor startup order to init SSE-S3 key manager before IAM * S3: Derive STS signing key from KEK using HKDF for security isolation * S3: Document STS signing key fallback in security.toml * fix(s3api): refine anonymous access logic and secure-by-default behavior - Initialize anonymous identity by default in `NewIdentityAccessManagement` to prevent nil pointer exceptions. - Ensure `ReplaceS3ApiConfiguration` preserves the anonymous identity if not present in the new configuration. - Update `NewIdentityAccessManagement` signature to accept `filerClient`. - In legacy mode (no policy engine), anonymous defaults to Deny (no actions), preserving secure-by-default behavior. - Use specific `LookupAnonymous` method instead of generic map lookup. - Update tests to accommodate signature changes and verify improved anonymous handling. * feat(s3api): make IAM configuration optional - Start S3 API server without a configuration file if `EnableIam` option is set. - Default to `Allow` effect for policy engine when no configuration is provided (Zero-Config mode). - Handle empty configuration path gracefully in `loadIAMManagerFromConfig`. - Add integration test `iam_optional_test.go` to verify empty config behavior. * fix(iamapi): fix signature mismatch in NewIdentityAccessManagementWithStore * fix(iamapi): properly initialize FilerClient instead of passing nil * fix(iamapi): properly initialize filer client for IAM management - Instead of passing `nil`, construct a `wdclient.FilerClient` using the provided `Filers` addresses. - Ensure `NewIdentityAccessManagementWithStore` receives a valid `filerClient` to avoid potential nil pointer dereferences or limited functionality. * clean: remove dead code in s3api_server.go * refactor(s3api): improve IAM initialization, safety and anonymous access security * fix(s3api): ensure IAM config loads from filer after client init * fix(s3): resolve test failures in integration, CORS, and tagging tests - Fix CORS tests by providing explicit anonymous permissions config - Fix S3 integration tests by setting admin credentials in init - Align tagging test credentials in CI with IAM defaults - Added goroutine to retry IAM config load in iamapi server * fix(s3): allow anonymous access to health targets and S3 Tables when identities are present * fix(ci): use /healthz for Caddy health check in awscli tests * iam, s3api: expose DefaultAllow from IAM and Policy Engine This allows checking the global "Open by Default" configuration from other components like S3 Tables. * s3api/s3tables: support DefaultAllow in permission logic and handler Updated CheckPermissionWithContext to respect the DefaultAllow flag in PolicyContext. This enables "Open by Default" behavior for unauthenticated access in zero-config environments. Added a targeted unit test to verify the logic. * s3api/s3tables: propagate DefaultAllow through handlers Propagated the DefaultAllow flag to individual handlers for namespaces, buckets, tables, policies, and tagging. This ensures consistent "Open by Default" behavior across all S3 Tables API endpoints. * s3api: wire up DefaultAllow for S3 Tables API initialization Updated registerS3TablesRoutes to query the global IAM configuration and set the DefaultAllow flag on the S3 Tables API server. This completes the end-to-end propagation required for anonymous access in zero-config environments. Added a SetDefaultAllow method to S3TablesApiServer to facilitate this. * s3api: fix tests by adding DefaultAllow to mock IAM integrations The IAMIntegration interface was updated to include DefaultAllow(), breaking several mock implementations in tests. This commit fixes the build errors by adding the missing method to the mocks. * env * ensure ports * env * env * fix default allow * add one more test using non-anonymous user * debug * add more debug * less logs |
||
|
Chris Luandgemini-code-assist[bot]
|
e6ee293c17 |
Add table operations test (#8241)
* Add Trino blog operations test * Update test/s3tables/catalog_trino/trino_blog_operations_test.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * feat: add table bucket path helpers and filer operations - Add table object root and table location mapping directories - Implement ensureDirectory, upsertFile, deleteEntryIfExists helpers - Support table location bucket mapping for S3 access * feat: manage table bucket object roots on creation/deletion - Create .objects directory for table buckets on creation - Clean up table object bucket paths on deletion - Enable S3 operations on table bucket object roots * feat: add table location mapping for Iceberg REST - Track table location bucket mappings when tables are created/updated/deleted - Enable location-based routing for S3 operations on table data * feat: route S3 operations to table bucket object roots - Route table-s3 bucket names to mapped table paths - Route table buckets to object root directories - Support table location bucket mapping lookup * feat: emit table-s3 locations from Iceberg REST - Generate unique table-s3 bucket names with UUID suffix - Store table metadata under table bucket paths - Return table-s3 locations for Trino compatibility * fix: handle missing directories in S3 list operations - Propagate ErrNotFound from ListEntries for non-existent directories - Treat missing directories as empty results for list operations - Fixes Trino non-empty location checks on table creation * test: improve Trino CSV parsing for single-value results - Sanitize Trino output to skip jline warnings - Handle single-value CSV results without header rows - Strip quotes from numeric values in tests * refactor: use bucket path helpers throughout S3 API - Replace direct bucket path operations with helper functions - Leverage centralized table bucket routing logic - Improve maintainability with consistent path resolution * fix: add table bucket cache and improve filer error handling - Cache table bucket lookups to reduce filer overhead on repeated checks - Use filer_pb.CreateEntry and filer_pb.UpdateEntry helpers to check resp.Error - Fix delete order in handler_bucket_get_list_delete: delete table object before directory - Make location mapping errors best-effort: log and continue, don't fail API - Update table location mappings to delete stale prior bucket mappings on update - Add 1-second sleep before timestamp time travel query to ensure timestamps are in past - Fix CSV parsing: examine all lines, not skip first; handle single-value rows * fix: properly handle stale metadata location mapping cleanup - Capture oldMetadataLocation before mutation in handleUpdateTable - Update updateTableLocationMapping to accept both old and new locations - Use passed-in oldMetadataLocation to detect location changes - Delete stale mapping only when location actually changes - Pass empty string for oldLocation in handleCreateTable (new tables have no prior mapping) - Improve logging to show old -> new location transitions * refactor: cleanup imports and cache design - Remove unused 'sync' import from bucket_paths.go - Use filer_pb.UpdateEntry helper in setExtendedAttribute and deleteExtendedAttribute for consistent error handling - Add dedicated tableBucketCache map[string]bool to BucketRegistry instead of mixing concerns with metadataCache - Improve cache separation: table buckets cache is now separate from bucket metadata cache * fix: improve cache invalidation and add transient error handling Cache invalidation (critical fix): - Add tableLocationCache to BucketRegistry for location mapping lookups - Clear tableBucketCache and tableLocationCache in RemoveBucketMetadata - Prevents stale cache entries when buckets are deleted/recreated Transient error handling: - Only cache table bucket lookups when conclusive (found or ErrNotFound) - Skip caching on transient errors (network, permission, etc) - Prevents marking real table buckets as non-table due to transient failures Performance optimization: - Cache tableLocationDir results to avoid repeated filer RPCs on hot paths - tableLocationDir now checks cache before making expensive filer lookups - Cache stores empty string for 'not found' to avoid redundant lookups Code clarity: - Add comment to deleteDirectory explaining DeleteEntry response lacks Error field * go fmt * fix: mirror transient error handling in tableLocationDir and optimize bucketDir Transient error handling: - tableLocationDir now only caches definitive results - Mirrors isTableBucket behavior to prevent treating transient errors as permanent misses - Improves reliability on flaky systems or during recovery Performance optimization: - bucketDir avoids redundant isTableBucket call via bucketRoot - Directly use s3a.option.BucketsPath for regular buckets - Saves one cache lookup for every non-table bucket operation * fix: revert bucketDir optimization to preserve bucketRoot logic The optimization to directly use BucketsPath bypassed bucketRoot's logic and caused issues with S3 list operations on delimiter+prefix cases. Revert to using path.Join(s3a.bucketRoot(bucket), bucket) which properly handles all bucket types and ensures consistent path resolution across the codebase. The slight performance cost of an extra cache lookup is worth the correctness and consistency benefits. * feat: move table buckets under /buckets Add a table-bucket marker attribute, reuse bucket metadata cache for table bucket detection, and update list/validation/UI/test paths to treat table buckets as /buckets entries. * Fix S3 Tables code review issues - handler_bucket_create.go: Fix bucket existence check to properly validate entryResp.Entry before setting s3BucketExists flag (nil Entry should not indicate existing bucket) - bucket_paths.go: Add clarifying comment to bucketRoot() explaining unified buckets root path for all bucket types - file_browser_data.go: Optimize by extracting table bucket check early to avoid redundant WithFilerClient call * Fix list prefix delimiter handling * Handle list errors conservatively * Fix Trino FOR TIMESTAMP query - use past timestamp Iceberg requires the timestamp to be strictly in the past. Use current_timestamp - interval '1' second instead of current_timestamp. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
b244bb58aa |
s3tables: redesign Iceberg REST Catalog using iceberg-go and automate integration tests (#8197)
* full integration with iceberg-go * Table Commit Operations (handleUpdateTable) * s3tables: fix Iceberg v2 compliance and namespace properties This commit ensures SeaweedFS Iceberg REST Catalog is compliant with Iceberg Format Version 2 by: - Using iceberg-go's table.NewMetadataWithUUID for strict v2 compliance. - Explicitly initializing namespace properties to empty maps. - Removing omitempty from required Iceberg response fields. - Fixing CommitTableRequest unmarshaling using table.Requirements and table.Updates. * s3tables: automate Iceberg integration tests - Added Makefile for local test execution and cluster management. - Added docker-compose for PyIceberg compatibility kit. - Added Go integration test harness for PyIceberg. - Updated GitHub CI to run Iceberg catalog tests automatically. * s3tables: update PyIceberg test suite for compatibility - Updated test_rest_catalog.py to use latest PyIceberg transaction APIs. - Updated Dockerfile to include pyarrow and pandas dependencies. - Improved namespace and table handling in integration tests. * s3tables: address review feedback on Iceberg Catalog - Implemented robust metadata version parsing and incrementing. - Ensured table metadata changes are persisted during commit (handleUpdateTable). - Standardized namespace property initialization for consistency. - Fixed unused variable and incorrect struct field build errors. * s3tables: finalize Iceberg REST Catalog and optimize tests - Implemented robust metadata versioning and persistence. - Standardized namespace property initialization. - Optimized integration tests using pre-built Docker image. - Added strict property persistence validation to test suite. - Fixed build errors from previous partial updates. * Address PR review: fix Table UUID stability, implement S3Tables UpdateTable, and support full metadata persistence individually * fix: Iceberg catalog stable UUIDs, metadata persistence, and file writing - Ensure table UUIDs are stable (do not regenerate on load). - Persist full table metadata (Iceberg JSON) in s3tables extended attributes. - Add `MetadataVersion` to explicitly track version numbers, replacing regex parsing. - Implement `saveMetadataFile` to persist metadata JSON files to the Filer on commit. - Update `CreateTable` and `UpdateTable` handlers to use the new logic. * test: bind weed mini to 0.0.0.0 in integration tests to fix Docker connectivity * Iceberg: fix metadata handling in REST catalog - Add nil guard in createTable - Fix updateTable to correctly load existing metadata from storage - Ensure full metadata persistence on updates - Populate loadTable result with parsed metadata * S3Tables: add auth checks and fix response fields in UpdateTable - Add CheckPermissionWithContext to UpdateTable handler - Include TableARN and MetadataLocation in UpdateTable response - Use ErrCodeConflict (409) for version token mismatches * Tests: improve Iceberg catalog test infrastructure and cleanup - Makefile: use PID file for precise process killing - test_rest_catalog.py: remove unused variables and fix f-strings * Iceberg: fix variable shadowing in UpdateTable - Rename inner loop variable `req` to `requirement` to avoid shadowing outer request variable * S3Tables: simplify MetadataVersion initialization - Use `max(req.MetadataVersion, 1)` instead of anonymous function * Tests: remove unicode characters from S3 tables integration test logs - Remove unicode checkmarks from test output for cleaner logs * Iceberg: improve metadata persistence robustness - Fix MetadataLocation in LoadTableResult to fallback to generated location - Improve saveMetadataFile to ensure directory hierarchy existence and robust error handling |
||
|
Chris Lu
|
79722bcf30 |
Add s3tables shell and admin UI (#8172)
* Add shared s3tables manager * Add s3tables shell commands * Add s3tables admin API * Add s3tables admin UI * Fix admin s3tables namespace create * Rename table buckets menu * Centralize s3tables tag validation * Reuse s3tables manager in admin * Extract s3tables list limit * Add s3tables bucket ARN helper * Remove write middleware from s3tables APIs * Fix bucket link and policy hint * Fix table tag parsing and nav link * Disable namespace table link on invalid ARN * Improve s3tables error decode * Return flag parse errors for s3tables tag * Accept query params for namespace create * Bind namespace create form data * Read s3tables JS data from DOM * s3tables: allow empty region ARN * shell: pass s3tables account id * shell: require account for table buckets * shell: use bucket name for namespaces * shell: use bucket name for tables * shell: use bucket name for tags * admin: add table buckets links in file browser * s3api: reuse s3tables tag validation * admin: harden s3tables UI handlers * fix admin list table buckets * allow admin s3tables access * validate s3tables bucket tags * log s3tables bucket metadata errors * rollback table bucket on owner failure * show s3tables bucket owner * add s3tables iam conditions * Add s3tables user permissions UI * Authorize s3tables using identity actions * Add s3tables permissions to user modal * Disambiguate bucket scope in user permissions * Block table bucket names that match S3 buckets * Pretty-print IAM identity JSON * Include tags in s3tables permission context * admin: refactor S3 Tables inline JavaScript into a separate file * s3tables: extend IAM policy condition operators support * shell: use LookupEntry wrapper for s3tables bucket conflict check * admin: handle buildBucketPermissions validation in create/update flows |
||
|
Chris Lu
|
88c27615c4 | /table-buckets | ||
|
Chris Lu
|
c99e8d4152 |
s3tables: Remove duplicate bucket extraction logic in helper
Move bucket name extraction outside the if/else block in
extractResourceOwnerAndBucket since the logic is identical for both
ResourceTypeTable and ResourceTypeBucket cases. This reduces code
duplication and improves maintainability.
The extraction pattern (parts[1] from /tables/{bucket}/...) works for
both resource types, so it's now performed once before the type-specific
metadata unmarshaling.
|
||
|
Chris Lu
|
8eee6b2a0e |
s3tables: Fix bucket policy error handling in permission checks
Replace error-swallowing pattern where all errors from getExtendedAttribute were ignored for bucket policy reads. Now properly distinguish between: - ErrAttributeNotFound: Policy not found is expected; continue with empty policy - Other errors: Return internal server error and stop processing Applied fix to all bucket policy reads in: - handleDeleteTableBucketPolicy (line 220) - handleTagResource (line 313) - handleUntagResource (line 405) - handleListTagsForResource (line 488) - And additional occurrences in closures This prevents silent failures and ensures policy-related errors are surfaced to callers rather than being silently ignored. |
||
|
Chris Lu
|
a27f6527ab |
s3tables: Extract resource owner and bucket extraction into helper method
Create extractResourceOwnerAndBucket() helper to consolidate the repeated pattern of unmarshaling metadata and extracting bucket name from resource path. This pattern was duplicated in handleTagResource, handleListTagsForResource, and handleUntagResource. Update all three handlers to use the helper. Also update remaining uses of getPrincipalFromRequest() (in handler_bucket_create, handler_bucket_get_list_delete, handler_namespace) to use getAccountID() after consolidating the two identical methods. |
||
|
Chris Lu
|
0b41ade726 |
s3tables: Fetch bucket policy in handleListTagsForResource for permission evaluation
Update handleListTagsForResource to fetch and pass bucket policy to CheckPermission, matching the behavior of handleTagResource/handleUntagResource. This enables bucket-policy-based permission grants to be evaluated for ListTagsForResource, not just ownership-based checks. |
||
|
Chris Lu
|
2d556ac2a5 |
S3 Tables API now properly enforces resource policies
addressing the critical security gap where policies were created but never evaluated. |
||
|
Chris Lu
|
1697ec862f | ownerAccountID | ||
|
Chris Lu
|
1fdd9c3372 | s3tables: refactor permission checks to use resource owner in policy and tagging handlers | ||
|
Chris Lu
|
2b2ff008cd |
s3tables: improve resource resolution and error mapping for policies and tagging
Refactored resolveResourcePath to return resource type, enabling accurate NoSuchBucket vs NoSuchTable error codes. Added existence checks before deleting policies. |
||
|
Chris Lu
|
f83cef1259 | s3tables: add table name validation and 404 propagation to policy handlers | ||
|
Chris Lu
|
dc4c62e742 |
s3tables: harden auth and error handling
- Add authorization checks to all S3 Tables handlers (policy, table ops) to enforce security - Improve error handling to distinguish between NotFound (404) and InternalError (500) - Fix directory FileMode usage in filer_ops - Improve test randomness for version tokens - Update permissions comments to acknowledge IAM gaps |
||
|
Chris Lu
|
62a1178a0b |
s3tables: improve robustness, security, and error propagation in handlers
- Implement strict table name validation (prevention of path traversal and character enforcement) - Add nil checks for entry.Entry in all listing loops to prevent panics - Propagate backend errors instead of swallowing them or assuming 404 - Correctly map filer_pb.ErrNotFound to appropriate S3 error codes - Standardize existence checks across bucket, namespace, and table handlers |
||
|
Chris Lu
|
33da87452b |
Refine S3 Tables implementation to address code review feedback
- Standardize namespace representation to []string - Improve listing logic with pagination and StartFromFileName - Enhance error handling with sentinel errors and robust checks - Add JSON encoding error logging - Fix CI workflow to use gofmt -l - Standardize timestamps in directory creation - Validate single-level namespaces |
||
|
Chris Lu
|
b30631c3b5 | s3tables: propagate request context to filer operations | ||
|
Chris Lu
|
ef3873b616 |
s3tables: add error handling for json.Marshal calls
- Add error handling in handler_namespace.go (metadata marshaling) - Add error handling in handler_table.go (metadata and tags marshaling) - Add error handling in handler_policy.go (tag marshaling in TagResource and UntagResource) - Return proper errors with context instead of silently ignoring failures |
||
|
Chris Lu
|
3b1920cf43 |
s3tables: add handler_ prefix to operation handler files
- Rename bucket_create.go → handler_bucket_create.go - Rename bucket_get_list_delete.go → handler_bucket_get_list_delete.go - Rename namespace.go → handler_namespace.go - Rename table.go → handler_table.go - Rename policy.go → handler_policy.go Improves file organization by clearly identifying handler implementations. No code changes, refactoring only. |