mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-27 07:20:44 +01:00
fix(s3): stop S3 Tables routes from swallowing buckets named "buckets" or "get-table" (#9566)
* fix(s3): stop S3 Tables routes from swallowing buckets named "buckets" or "get-table"
The S3 Tables REST endpoints share top-level paths with the regular S3
API (/buckets for ListTableBuckets/CreateTableBucket, /get-table for
GetTable). They are registered first on the same router as the bucket
subrouter, so a path-style request such as GET /buckets?list-type=2 on
a bucket actually named "buckets" matched ListTableBuckets and returned
JSON. AWS SDK V2 (and Hadoop s3a / Spark) then failed XML parsing with
"Unexpected character '{' (code 123) in prolog".
Disambiguate by requiring the AWS V4 credential scope to name the
s3tables service on the colliding routes. Regular S3 SDKs sign with
service=s3, S3 Tables SDKs sign with service=s3tables, and the scope is
present in both the Authorization header and the X-Amz-Credential query
parameter for presigned URLs, so the matcher works for both flavors.
ARN-bearing S3 Tables routes (/buckets/<arn>, /namespaces/<arn>, etc.)
already cannot collide because colons are not valid in bucket names, so
they are left untouched.
* fix(s3): accept AWS JSON RPC content type as S3 Tables intent signal
The Iceberg catalog integration tests send unsigned PUT /buckets with
Content-Type: application/x-amz-json-1.1 to create table buckets. With
only the credential-scope check, those requests fell through to the
regular S3 CreateBucket handler and the suite went red on this branch.
Extend the matcher so a request is recognized as S3 Tables when either:
- its AWS V4 credential scope names SERVICE=s3tables; or
- it carries the canonical AWS JSON RPC 1.1 content type and is
unsigned (a request explicitly signed for SERVICE=s3 still wins).
The regular S3 SDKs do not send application/x-amz-json-1.1, so the
signal is safe for the colliding paths (/buckets, /get-table).
Also add an AWS SDK V2 for Go integration test under
test/s3/sdk_v2_routing/ that drives the SDK's own XML deserializer
against a bucket literally named "buckets" and "get-table" — the SDK
errors before the test asserts if the server returns the wrong body
shape. Wired up via .github/workflows/s3-sdk-v2-routing-tests.yml,
mirroring the etag/acl workflow.
* s3api: extend service matcher to all S3 Tables routes; simplify scope check
- Apply serviceMatcher to every S3 Tables route, not just the bare-path
ones. ARN-bearing paths could otherwise be hit by an S3 object key
that starts with arn:aws:s3tables:..., inside a bucket named
"buckets", "namespaces", "tables", or "tag". One matcher everywhere
closes both collision classes.
- Replace strings.Split + index lookup with strings.Contains for the
credential-scope check. The scope shape is fixed at
AK/DATE/REGION/SERVICE/aws4_request, slashes only delimit components,
and access keys are alphanumeric — so /s3tables/ matches iff SERVICE
is exactly s3tables. Existing unit cases (including the
access-key-substring case) still pass.
- Read the GetObject body in the SDK v2 routing test with io.ReadAll;
the single Read could return short and make the equality check flaky.
* s3api: drop content-type fallback; sign s3 tables harness traffic instead
The content-type fallback in isS3TablesSignedRequest let an anonymous
regular-S3 request whose body type is application/x-amz-json-1.1 hit
an S3 Tables route when the path-style object key happened to be
shaped like an S3 Tables ARN (e.g. PutObject on bucket "buckets"
with key arn:aws:s3tables:.../bucket/foo/policy). Narrow the matcher
back to the AWS V4 credential scope so only requests signed for
SERVICE=s3tables match the S3 Tables routes.
Update the Iceberg catalog test harness — the only caller still
sending unsigned PUT /buckets — to sign with SERVICE=s3tables. The
mini instance runs in default-allow mode, so the signature itself is
not verified; only the credential scope matters for the route match.
Drop the stale unit cases for the JSON-RPC content-type signal and
the routing test that exercised unsigned harness traffic.
This commit is contained in:
1 parent
cfc08fbf6c
commit
f72983c1fd
6 files changed
+616
-24
No files matched your search
@@ -17,6 +17,9 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws/credentials"
|
||||
v4 "github.com/aws/aws-sdk-go/aws/signer/v4"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/test/testutil"
|
||||
)
|
||||
|
||||
@@ -451,19 +454,22 @@ func icebergPath(prefix, path string) string {
|
||||
return withPrefix
|
||||
}
|
||||
|
||||
// createTableBucket creates a table bucket via the S3Tables REST API
|
||||
// createTableBucket creates a table bucket via the S3Tables REST API.
|
||||
// The request is AWS V4 signed for SERVICE=s3tables so the S3 Tables
|
||||
// route matcher accepts it; signing with regular SERVICE=s3 would let
|
||||
// the request fall through to the S3 CreateBucket handler.
|
||||
func createTableBucket(t *testing.T, env *TestEnvironment, bucketName string) {
|
||||
t.Helper()
|
||||
|
||||
// Use S3Tables REST API to create the bucket
|
||||
endpoint := fmt.Sprintf("http://localhost:%d/buckets", env.s3Port)
|
||||
|
||||
reqBody := fmt.Sprintf(`{"name":"%s"}`, bucketName)
|
||||
|
||||
req, err := http.NewRequest(http.MethodPut, endpoint, strings.NewReader(reqBody))
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create request: %v", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/x-amz-json-1.1")
|
||||
signS3TablesRequest(t, req, reqBody)
|
||||
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
@@ -480,6 +486,20 @@ func createTableBucket(t *testing.T, env *TestEnvironment, bucketName string) {
|
||||
t.Logf("Created table bucket %s", bucketName)
|
||||
}
|
||||
|
||||
// signS3TablesRequest signs req with AWS V4 for SERVICE=s3tables. The
|
||||
// underlying weed mini instance runs in default-allow mode so the
|
||||
// signature itself is not verified; only the credential scope matters,
|
||||
// because the S3 Tables route matcher requires SERVICE=s3tables to
|
||||
// distinguish S3 Tables traffic from regular S3 calls on the same paths.
|
||||
func signS3TablesRequest(t *testing.T, req *http.Request, body string) {
|
||||
t.Helper()
|
||||
creds := credentials.NewStaticCredentials("test-ak", "test-sk", "")
|
||||
signer := v4.NewSigner(creds)
|
||||
if _, err := signer.Sign(req, strings.NewReader(body), "s3tables", "us-east-1", time.Now()); err != nil {
|
||||
t.Fatalf("Failed to sign S3 Tables request: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// randomSuffix returns a short random hex suffix for unique resource naming.
|
||||
func randomSuffix() string {
|
||||
return fmt.Sprintf("%x", time.Now().UnixNano()&0xffffffff)
|
||||
|
||||
Reference in new issue
Block a user