Commit Graph
100 Commits
Author SHA1 Message Date
Chris Lu bfb0771a5b feat(plugin): Add plugins overview page template 2026-02-17 02:07:16 -08:00
Chris Lu 01f310ac9b feat(plugin): Add UI template helpers 2026-02-17 02:06:57 -08:00
Chris Lu f266a25a32 feat(plugin): Add HTTP API handlers for plugin management 2026-02-17 02:06:39 -08:00
Chris Lu 2be97d7190 feat(plugin): Add history management to plugin manager 2026-02-17 02:06:13 -08:00
Chris Lu 75644c7e3d feat(plugin): Add history tracking to plugin types 2026-02-17 02:06:05 -08:00
Chris Lu 3320911984 fix(plugin): Fix testing framework and plugin compilation issues 2026-02-17 02:03:38 -08:00
Chris Lu dafa8d79f5 feat(plugin): Add balance plugin implementation 2026-02-17 02:00:39 -08:00
Chris Lu e7efe201e3 feat(plugin): Add vacuum plugin implementation 2026-02-17 02:00:36 -08:00
Chris Lu 9b2fd24e52 feat(plugin): Add balance plugin implementation 2026-02-17 01:57:51 -08:00
Chris Lu 574f72906a feat(plugin): Add vacuum plugin implementation 2026-02-17 01:55:11 -08:00
Chris Lu babc4751db test(plugin): Add EC plugin comprehensive tests 2026-02-17 01:46:34 -08:00
Chris Lu efb69400ff feat(plugin): Add EC plugin worker client 2026-02-17 01:46:12 -08:00
Chris Lu 7dc40d3747 feat(plugin): Add EC plugin execution logic 2026-02-17 01:45:49 -08:00
Chris Lu fb7c7b3744 feat(plugin): Add EC plugin detection logic 2026-02-17 01:45:30 -08:00
Chris Lu 7ff0026e46 feat(plugin): Add EC plugin configuration schema 2026-02-17 01:45:11 -08:00
Chris Lu f57992b81e feat(plugin): Add test harness for plugin testing 2026-02-17 01:44:52 -08:00
Chris Lu 471b0cf8f3 feat(plugin): Add mock plugin for testing 2026-02-17 01:44:24 -08:00
Chris Lu a2d2d77e09 feat(plugin): Add mock admin server for testing 2026-02-17 01:43:57 -08:00
Chris Lu eb13f9ce82 fix(plugin): Fix protobuf enum naming and build issues 2026-02-17 01:41:19 -08:00
Chris Lu 8c6e627af2 build(plugin): Add plugin.proto to build system 2026-02-17 01:39:41 -08:00
Chris Lu 67e37e180e feat(plugin): Add plugin manager orchestrator 2026-02-17 01:39:28 -08:00
Chris Lu 2fcd6a3056 feat(plugin): Add gRPC server implementation 2026-02-17 01:39:06 -08:00
Chris Lu 13f39d4ad2 feat(plugin): Add configuration manager for persistence 2026-02-17 01:38:39 -08:00
Chris Lu eead8b2337 feat(plugin): Add job dispatcher for orchestration 2026-02-17 01:38:20 -08:00
Chris Lu b728dc00ed feat(plugin): Add job queue with priority and deduplication 2026-02-17 01:37:58 -08:00
Chris Lu 5ac3e80d2b feat(plugin): Add plugin registry with lifecycle management 2026-02-17 01:37:39 -08:00
Chris Lu 42c52f9371 feat(plugin): Add plugin system core types and domain models 2026-02-17 01:37:22 -08:00
Chris Lu 2b4600cb08 feat(plugin): Add plugin system gRPC protocol definition 2026-02-17 01:37:06 -08:00
Chris Lu 63f641a6c9 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2026-02-16 17:01:26 -08:00
Chris Lu 3c3a78d08e 4.13 2026-02-16 17:01:19 -08:00
Chris Lu 564fc56698 Update docker-compose.yml 2026-02-16 15:01:09 -08:00
Chris Lu 35ad7d08a5 remove debug 2026-02-16 14:03:02 -08:00
Chris Lu e863767ac7 cleanup(iam): final removal of temporary debug logging from STS and S3 API 2026-02-14 22:15:06 -08:00
Chris Lu e29a7f1741 cleanup(iam): remove temporary debug logging from STS and S3 API (redo) 2026-02-14 22:14:33 -08:00
Chris Lu f1bf60d288 faster 2026-02-13 14:09:30 -08:00
Chris Lu ac242d04ee one time manual run 2026-02-11 13:20:52 -08:00
Chris Lu 21543134c8 fix manual build process 2026-02-11 13:20:41 -08:00
Chris Lu 7151181d54 fix flaky tests 2026-02-11 12:23:35 -08:00
Chris Lu db76eb26e7 compile 2026-02-09 21:06:07 -08:00
Chris Lu 4ccc7668ce admin: resolve merge conflicts 2026-02-09 20:56:20 -08:00
Chris Lu be26ce74ce s3tables: support multi-level namespace normalization 2026-02-09 19:42:31 -08:00
Chris Lu 0b80f055c2 Merge branch 'fix/8270-leader-not-elected' 2026-02-09 18:15:59 -08:00
Chris Lu af8273386d 4.12 2026-02-09 18:15:19 -08:00
Chris Lu 15d0a46679 Normalize and deduplicate master peer addresses 2026-02-09 18:10:54 -08:00
Chris Lu ae27e17e6f Canonicalize self peer entry to avoid raft self-alias panic 2026-02-09 18:07:01 -08:00
Chris Lu 02dac23119 Fix master leader detection when grpc ports change 2026-02-09 18:00:09 -08:00
Chris Lu 2ed5a8f65c add tests 2026-02-09 01:37:56 -08:00
Chris Lu 5a279c4d2f fmt 2026-02-08 21:19:00 -08:00
Chris Lu 0c89185291 4.10 2026-02-08 21:16:58 -08:00
Chris Lu bbcb8b7590 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2026-02-08 20:05:24 -08:00
Chris Lu 3bb9493a5b Enhance Iceberg catalog browsing UI 2026-02-08 00:00:02 -08:00
Chris Lu d9e3fb2b8e Add Iceberg table details view 2026-02-07 23:59:49 -08:00
Chris Lu 217d977579 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2026-02-06 13:14:30 -08:00
Chris Lu 833bcde9f3 test: add Trino Iceberg catalog integration test
- Create test/s3/catalog_trino/trino_catalog_test.go with TestTrinoIcebergCatalog
- Tests integration between Trino SQL engine and SeaweedFS Iceberg REST catalog
- Starts weed mini with all services and Trino in Docker container
- Validates Iceberg catalog schema creation and listing operations
- Uses native S3 filesystem support in Trino with path-style access
- Add workflow job to s3-tables-tests.yml for CI execution
2026-02-05 16:10:31 -08:00
Chris Lu e39a4c2041 fix flaky test 2026-02-04 23:16:31 -08:00
Chris Lu 27d5f44415 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2026-02-04 12:44:54 -08:00
Chris Lu 000e2bd4a9 logging and debugging 2026-02-04 12:44:52 -08:00
Chris Lu 2ff1cd9fc9 format 2026-02-03 18:39:01 -08:00
Chris Lu 5a5cc38692 4.09 2026-02-03 17:56:25 -08:00
Chris Lu 746df25164 fix formatting 2026-02-03 00:27:20 -08:00
Chris Lu 7169fe585d Delete verify_gc_empty_test.go 2026-02-03 00:20:38 -08:00
Chris Lu 1c1f80358f templ 2026-02-03 00:10:43 -08:00
Chris Lu 330bd92ddc 4.08 2026-02-02 20:44:13 -08:00
Chris Lu ba8816e2e1 4.08 2026-02-02 20:36:03 -08:00
Chris Lu 3b9e367c1a templ 2026-02-02 20:34:37 -08:00
Chris Lu c7c2d8d606 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2026-01-29 20:03:26 -08:00
Chris Lu 88c27615c4 /table-buckets 2026-01-29 20:03:17 -08:00
Chris Lu 94e0b902f9 shell: update fs.verify and volume.fsck for new BFS signature
Updated dependent commands to match the refactored
doTraverseBfsAndSaving signature and use context for channel sends.
2026-01-29 14:42:10 -08:00
Chris Lu 3b05efbdbc shell: fix potential deadlock in fs.meta.save BFS traversal
Refactored doTraverseBfsAndSaving to use context cancellation.
If the saving process fails, the traversal is stopped immediately
to prevent workers from blocking on the output channel.
2026-01-29 14:42:10 -08:00
Chris Lu b91427c30f filer: preserve existing TTL during CreateEntry/UpdateEntry gRPC calls
This ensures that metadata load correctly restores entries with their
original TTL instead of overwriting with the default from filer.conf.
Fixes #8159.
2026-01-29 14:42:09 -08:00
Chris Lu 9e575822a3 Merge master into s3tables-by-claude (resolve conflicts, keep s3tables changes) 2026-01-28 19:47:42 -08:00
Chris Lu d399113e0c test: fix duplicate subtest names in permissions_test.go
Rename duplicate 'combined * and ?' test cases to include singular/plural suffix
for clarity and to support targeted test runs.
2026-01-28 19:42:19 -08:00
Chris Lu c106532b79 fix: prevent MiniClusterCtx race conditions in command shutdown
Capture global MiniClusterCtx into local variables before goroutine/select
evaluation to prevent nil dereference/data race when context is reset to nil
after nil check. Applied to filer, master, volume, and s3 commands.
2026-01-28 19:42:16 -08:00
Chris Lu a4217dff5f s3tables: enhance DeleteTable authorization with policy checking
Fetch and evaluate table policies in DeleteTable handler to support policy-based
delegation. Aligns authorization behavior with GetTable and ListTables handlers
instead of only checking ownership.
2026-01-28 19:42:12 -08:00
Chris Lu 745a7e40a6 s3tables: improve bucket policy error handling in DeleteTableBucket
Explicitly handle ErrAttributeNotFound vs other errors when fetching bucket policy.
Return errors for non-expected failures to prevent masking filer issues and
ensure correct authorization decisions.
2026-01-28 19:42:08 -08:00
Chris Lu d5ce6a4cda s3tables: refactor bucket name validation into single function
Combine length, character, and reserved pattern validation into validateBucketName()
which returns descriptive error messages. Keep isValidBucketName() for backward
compatibility. This simplifies handler validation and provides better error reporting.
2026-01-28 19:42:01 -08:00
Chris Lu fe66d00ab0 Merge branch 'master' into s3tables-by-claude 2026-01-28 19:40:46 -08:00
Chris Lu 549b65785d refactor 2026-01-28 18:55:16 -08:00
Chris Lu 590e7efbef s3tables: Separate table name pattern constant for clarity
Define a separate tableNamePatternStr constant for the table name component in
the ARN regex, even though it currently has the same value as
tableNamespacePatternStr. This improves code clarity and maintainability, making
it easier to modify if the naming rules for tables and namespaces diverge in the
future.
2026-01-28 18:40:02 -08:00
Chris Lu 78c00e313a go fmt 2026-01-28 18:34:32 -08:00
Chris Lu f5d26b803b s3tables: Fix ListTables authorization and policy parsing
Make ListTables authorization consistent with GetTable/CreateTable:

1. ListTables authorization now evaluates policies instead of owner-only checks:
   - For namespace listing: checks namespace policy AND bucket policy
   - For bucket-wide listing: checks bucket policy
   - Uses CanListTables permission framework

2. Remove owner-only filter in listTablesWithClient that prevented policy-based
   sharing of tables. Authorization is now enforced at the handler level, so all
   tables in the namespace/bucket are returned to authorized callers (who have
   access either via ownership or policy).

3. Add flexible PolicyDocument.UnmarshalJSON to support both single-object and
   array forms of Statement field:
   - Handles: {"Statement": {...}}
   - Handles: {"Statement": [{...}, {...}]}
   - Improves AWS IAM compatibility

This ensures cross-account table listing works when delegated via bucket/namespace
policies, consistent with the authorization model for other operations.
2026-01-28 18:27:37 -08:00
Chris Lu 25b0f86bda s3tables: Fix ownership consistency across handlers
Address three related ownership consistency issues:

1. CreateNamespace now sets OwnerAccountID to bucketMetadata.OwnerAccountID
   instead of request principal. This prevents namespaces created by
   delegated callers (via bucket policy) from becoming unmanageable, since
   ListNamespaces filters by bucket owner.

2. CreateTable now:
   - Fetches bucket metadata to use correct owner for bucket policy evaluation
   - Uses namespaceMetadata.OwnerAccountID for namespace policy checks
   - Uses bucketMetadata.OwnerAccountID for bucket policy checks
   - Sets table OwnerAccountID to namespaceMetadata.OwnerAccountID (inherited)

3. GetTable now:
   - Fetches bucket metadata to use correct owner for bucket policy evaluation
   - Uses metadata.OwnerAccountID for table policy checks
   - Uses bucketMetadata.OwnerAccountID for bucket policy checks

This ensures:
- Bucket owner retains implicit "owner always allowed" behavior even when
  evaluating bucket policies
- Ownership hierarchy is consistent (namespace owned by bucket, table owned by namespace)
- Cross-principal delegation via policies doesn't break ownership chains
2026-01-28 18:03:47 -08:00
Chris Lu b049e883e1 go fmt 2026-01-28 17:51:02 -08:00
Chris Lu c99e8d4152 s3tables: Remove duplicate bucket extraction logic in helper
Move bucket name extraction outside the if/else block in
extractResourceOwnerAndBucket since the logic is identical for both
ResourceTypeTable and ResourceTypeBucket cases. This reduces code
duplication and improves maintainability.

The extraction pattern (parts[1] from /tables/{bucket}/...) works for
both resource types, so it's now performed once before the type-specific
metadata unmarshaling.
2026-01-28 17:47:14 -08:00
Chris Lu 3dcaee56aa Revert "ci: Pin GitHub Actions to commit SHAs for s3-tables-tests"
This reverts commit 01da26fbcb.
2026-01-28 17:43:11 -08:00
Chris Lu 21584e4ac8 s3tables: Add resource ARN validation to policy evaluation
Implement resource-specific policy validation to prevent over-broad
permission grants. Add matchesResource and matchesResourcePattern functions
to validate statement Resource fields against specific resource ARNs.

Add new CheckPermissionWithResource function that includes resource ARN
validation, while keeping CheckPermission unchanged for backward compatibility.

This enables policies to grant access to specific resources only:
- statements with Resource: "arn:aws:s3tables:...:bucket/specific-bucket/*"
  will only match when accessing that specific bucket
- statements without Resource field match all resources (implicit *)
- resource patterns support wildcards (* for any sequence, ? for single char)

For future use: Handlers can call CheckPermissionWithResource with the
target resource ARN to enforce resource-level access control.
2026-01-28 17:41:22 -08:00
Chris Lu 01da26fbcb ci: Pin GitHub Actions to commit SHAs for s3-tables-tests
Update all action refs to use pinned commit SHAs instead of floating tags:
- actions/checkout: @v6 → @8e8c483 (v4)
- actions/setup-go: @v6 → @0c52d54 (v5)
- actions/upload-artifact: @v6 → @65d8626 (v4)

Pinned SHAs improve reproducibility and reduce supply chain risk by
preventing accidental or malicious changes in action releases. Aligns
with repository conventions used in other workflows (e.g., go.yml).
2026-01-28 17:40:40 -08:00
Chris Lu 2c45b69775 s3tables: Fix remaining policy error handling in namespace and bucket handlers
Replace silent error swallowing (err == nil) with proper error distinction
for bucket policy reads. Now properly checks ErrAttributeNotFound and
propagates other errors as internal server errors.

Fixed 5 locations:
- handleCreateNamespace (policy fetch)
- handleDeleteNamespace (policy fetch)
- handleListNamespaces (policy fetch)
- handleGetNamespace (policy fetch)
- handleGetTableBucket (policy fetch)

This prevents masking of filer issues when policies cannot be read due
to I/O errors or other transient failures.
2026-01-28 17:39:44 -08:00
Chris Lu b7bba7e7dc s3tables: Generate ARNs using resource owner account ID
Change ARN generation to use resource OwnerAccountID instead of caller
identity (h.getAccountID(r)). This ensures ARNs are stable and consistent
regardless of which principal accesses the resource.

Updated generateTableBucketARN and generateTableARN function signatures
to accept ownerAccountID parameter. All call sites updated to pass the
resource owner's account ID from metadata.

This prevents ARN inconsistency issues when multiple principals have
access to the same resource via policies.
2026-01-28 17:38:22 -08:00
Chris Lu e7b2869aa9 s3tables: Use policy framework for GetTable authorization
Replace strict ownership check with policy-based authorization in GetTable.
Now checks both table and bucket policies for GetTable permission, allowing
authorized non-owners to read table metadata.

Authorization logic:
- Table policy grants GetTable → allowed
- Bucket policy grants GetTable → allowed
- Otherwise → 404 NotFound (no access disclosed)

Maintains security through policy evaluation while enabling read delegation.
2026-01-28 17:37:12 -08:00
Chris Lu bea0f8eda0 s3tables: Use policy framework for table creation authorization
Replace strict ownership check in CreateTable with policy-based authorization.
Now checks both namespace and bucket policies for CreateTable permission,
allowing delegation via resource policies while still respecting owner bypass.

Authorization logic:
- Namespace policy grants CreateTable → allowed
- Bucket policy grants CreateTable → allowed
- Otherwise → denied (even if same owner)

This enables cross-principal table creation via policies while maintaining
security through explicit allow/deny semantics.
2026-01-28 17:36:53 -08:00
Chris Lu cf5043a9f9 s3tables: Normalize action names to include service prefix
Add automatic normalization of operations to full IAM-style action names
(e.g., 's3tables:CreateTableBucket') in CheckPermission(). This ensures
policy statements using prefixed actions (s3tables:*) correctly match
operations evaluated by permission helpers.

Also fixes incorrect r.Context() passed to GetIdentityNameFromContext
which expects *http.Request. Now passes r directly.
2026-01-28 17:36:16 -08:00
Chris Lu 08bd1e2563 s3tables: Pre-validate namespace to return 400 instead of 500
Move validateNamespace call outside of filerClient.WithFilerClient closure
so that validation errors return HTTP 400 (InvalidRequest) instead of 500
(InternalError).

Before: Validation error inside closure → treated as internal error → 500
After: Validation error before closure → handled as bad request → 400

This provides correct error semantics: namespace validation is an input
validation issue, not a server error.
2026-01-28 17:03:04 -08:00
Chris Lu 8eee6b2a0e s3tables: Fix bucket policy error handling in permission checks
Replace error-swallowing pattern where all errors from getExtendedAttribute
were ignored for bucket policy reads. Now properly distinguish between:

- ErrAttributeNotFound: Policy not found is expected; continue with empty policy
- Other errors: Return internal server error and stop processing

Applied fix to all bucket policy reads in:
- handleDeleteTableBucketPolicy (line 220)
- handleTagResource (line 313)
- handleUntagResource (line 405)
- handleListTagsForResource (line 488)
- And additional occurrences in closures

This prevents silent failures and ensures policy-related errors are surfaced
to callers rather than being silently ignored.
2026-01-28 17:02:59 -08:00
Chris Lu fe856928c4 s3tables: Add t field to TestCluster for logging
Add *testing.T field to TestCluster struct and initialize it in
startMiniCluster. This allows Stop() to properly log warnings when
cluster shutdown times out. Includes the t field in the test cluster
initialization and restores the logging statement in Stop().
2026-01-28 17:02:53 -08:00
Chris Lu 6658a655f6 clean up 2026-01-28 17:00:42 -08:00
Chris Lu c5eadadf5a s3tables: Fix vet error - remove undefined c.t reference in Stop()
The TestCluster.Stop() method doesn't have access to testing.T object.
Remove the log statement and keep the timeout handling comment for clarity.
The original intent (warning about shutdown timeout) is still captured in
the code comment explaining potential issues.
2026-01-28 16:44:16 -08:00
Chris Lu 1e18c01a78 go fmt 2026-01-28 16:42:46 -08:00
Chris Lu 3e8d2a0a71 s3tables: Use policy_engine wildcard matcher for complete IAM compatibility
Replace the custom suffix-only wildcard implementation in matchesActionPattern
and matchesPrincipal with the policy_engine.MatchesWildcard function from
PR #8052. This enables full wildcard support including:

- Middle wildcards: s3tables:Get*Table matches GetTable
- Question mark wildcards: Get? matches any single character
- Combined patterns: s3tables:*Table* matches any action containing 'Table'

Benefits:
- Code reuse: eliminates duplicate wildcard logic
- Complete IAM compatibility: supports all AWS wildcard patterns
- Performance: uses efficient O(n) backtracking algorithm
- Consistency: same wildcard behavior across S3 API and S3 Tables

Add comprehensive unit tests covering exact matches, suffix wildcards,
middle wildcards, question marks, and combined patterns for both action
and principal matching.
2026-01-28 16:37:31 -08:00
Chris Lu dbf6465b0e s3tables: Add log message when cluster shutdown times out
The timeout path (2 second wait for graceful shutdown) was silent. Add a
warning log message when it occurs to help diagnose flaky test issues and
indicate when the mini cluster didn't shut down cleanly.
2026-01-28 16:24:15 -08:00