mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-22 22:11:39 +02:00
1e3c5e6fe1e1f62d3dec5cf2b3bf6210d75f9579
548
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Lu
|
1e3c5e6fe1 |
fuse integration: fail posix test targets when go test fails
Every suite pipes go test through tee under /bin/sh, so the recipe status was tee's and test failures stayed green. Run recipes under bash -o pipefail and emit the color escapes with printf since bash echo does not expand them. |
||
|
Chris Lu
|
1cf097de9a |
fuse integration: reference posix_Makefile explicitly in recursive make calls
GNU make does not propagate -f through MAKEFLAGS, so bare $(MAKE) re-read the default Makefile in this directory. |
||
|
Chris Lu
|
e284bdc6e1 | fuse integration: size readv buffer to the written segment | ||
|
Chris Lu
|
c607adbe6a | fuse integration: skip non-executable weed path candidates | ||
|
Devin
|
5fa30f6d4e |
fuse integration: add POSIX compliance test suite
Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Javier de la TorreandJavier de la Torre
|
de2fc03c7d |
s3tables: accept the external-volume verification object at a table bucket's root (#11900)
* s3tables: accept the external-volume verification object at a table bucket's root Snowflake verifies an S3-compatible external volume before it creates or writes Iceberg tables in it (SYSTEM$VERIFY_EXTERNAL_VOLUME, also run implicitly): it writes, reads, lists and deletes one small object, <base>/verify_<epoch milliseconds>_<number>, a 21-byte body. A table bucket refuses every key outside the Iceberg layout, so the check failed and Snowflake could not use a table bucket as an external volume. PUT, GET, HEAD, DELETE and DeleteObjects now accept exactly that key in a table bucket, narrowly enough that it cannot be used to store data: - at the bucket root only, so nothing is encoded in a prefix, written into a hidden tree or placed on a namespace or table name; - one shape: verify_<13 digits of epoch ms within one hour of the server's clock>_<1-10 digits>; - a PUT must declare a Content-Length of at most 1 KiB (no chunked or undeclared body) and carry no user metadata, tagging, Content-Disposition, Cache-Control, Expires or website redirect. Multipart, copy, rename, ACL, tagging and POST policy writes keep the plain layout rule. Buckets that are not table buckets are unaffected. Tests: table_bucket_verify_object_test.go, the key accepted (with the Iceberg layout as the positive control and the plain rule still refusing it), 15 look-alikes refused (prefixed, hidden tree, wrong shape, outside the window, ordinary keys), and seven PUT refusals through PutObjectHandler. * s3tables: keep the verification key off namespaces and narrow its PUT A namespace may be named like a verification key (verify_<ms>_<n>), and the key then names the namespace's directory. A PUT replaced that directory with a prefix object, dropping the namespace's catalog metadata; a DELETE removed an empty namespace. The key now gets the plain layout rule when it names a directory or its entry cannot be read, and a PUT of it never falls back to a prefix object, so a namespace created after the check is not replaced either. The time window applies to PUT only: GET, HEAD and DELETE take a key of any date, so an object a check left behind for more than an hour can still be read and removed. A verification PUT also refuses an extraction request and the stored Content-Language, Response-Content-Disposition and Content-Encoding headers (aws-chunked framing alone is still accepted). The integration test runs the check's PUT, HEAD, GET, list and DELETE against a table bucket, and the PUT and DELETE over a namespace of the key's shape. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
dependabot[bot]
|
efa9d304d2 |
build(deps): bump github.com/klauspost/compress from 1.18.0 to 1.18.7 in /test/kafka/kafka-client-loadtest (#11879)
build(deps): bump github.com/klauspost/compress Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.18.0 to 1.18.7. - [Release notes](https://github.com/klauspost/compress/releases) - [Commits](https://github.com/klauspost/compress/compare/v1.18.0...v1.18.7) --- updated-dependencies: - dependency-name: github.com/klauspost/compress dependency-version: 1.18.7 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
1253b97303 |
build(deps): bump github.com/klauspost/compress from 1.15.9 to 1.18.7 in /test/kafka/kafka-client-loadtest/tools (#11866)
build(deps): bump github.com/klauspost/compress Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.15.9 to 1.18.7. - [Release notes](https://github.com/klauspost/compress/releases) - [Commits](https://github.com/klauspost/compress/compare/v1.15.9...v1.18.7) --- updated-dependencies: - dependency-name: github.com/klauspost/compress dependency-version: 1.18.7 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
a1d8739464 |
build(deps): bump github.com/klauspost/compress from 1.17.0 to 1.18.7 in /test/kafka/simple-consumer (#11869)
build(deps): bump github.com/klauspost/compress Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.17.0 to 1.18.7. - [Release notes](https://github.com/klauspost/compress/releases) - [Commits](https://github.com/klauspost/compress/compare/v1.17.0...v1.18.7) --- updated-dependencies: - dependency-name: github.com/klauspost/compress dependency-version: 1.18.7 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
GAURIandGAURI
|
da0d423bd9 |
Add OLake to README and Admin UI; fix catalog_olake path-style docs (#11858)
* README: mention OLake in the Iceberg lakehouse section * catalog_olake: fix path-style prose; admin UI: list OLake as a REST catalog client * admin UI: restore generated file import block, keep only the string change --------- Co-authored-by: GAURI <174253719+gauriiiiiiiiiiii@users.noreply.github.com> |
||
|
Chris LuandDevin
|
79daffb2d2 |
test: retry retryable 503s in cold-start assign test (#11852)
The master intentionally sheds an assign that triggered volume growth once its wait window elapses (503 + Retry-After); on a slow CI box cold-start growth can outlast it, so the single-shot assertion flaked. Honor the documented retry contract like a real client, within a 60s deadline. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
10ce19cb17 |
test: poll for TopologyId convergence in multi-master failover tests
A restarted goraft master wipes its raft state and only learns the TopologyId after the leader re-admits it and replays the entry. That trails HTTP readiness by several seconds, so the one-shot check in assertTopologyIdConsistent raced the propagation (CI run 39199033687: TopologyId applied on the follower ~2s after the test gave up). |
||
|
Chris Lu
|
8ac5133abe |
admin: default to port 12535, keep 23646 as a best-effort compat listener (#11816)
* admin: default to port 12535, keep 23646 as a best-effort compat listener The default admin HTTP port 23646 derives a worker gRPC port of 33646 (port + GrpcPortOffset), inside the Linux default ephemeral range (32768+). One of our own startup dials can claim 33646 before the admin binds it, killing startup. This is the same failure #10209 fixed for the test port allocator, still reachable via the mini default (#10838) and via standalone `weed admin` defaults. Move the default admin port to 12535 (gRPC 22535) everywhere, keeping the http+10000 convention so worker -admin derivation, the Rust worker and host:http.grpc addressing keep working: - weed admin -port, weed mini -admin.port - weed worker -admin, the plugin worker fallback, Rust worker and weed-lance-worker defaults - help text, README, helm values, docker compose, tests So old workers, bookmarks and scripts keep working, admin also serves the same handler and the worker gRPC server on the previous default (-port.legacy / -admin.port.legacy, default 23646, 0 disables). The legacy gRPC port binds best-effort: it lives in the ephemeral range, so a failed bind only warns instead of reintroducing the startup failure. weed mini reserves it early like the primary port and keeps it out of its own port relocation. grpc.Server tracks every listener handed to Serve, so the compat listener stops with the primary. Also warn when an explicitly configured admin port puts the derived gRPC port at or above the ephemeral floor, the other fix suggested in the issue. Supersedes the pinned-grpc approach in #11814: pinning -admin.port.grpc off the +10000 convention breaks external workers that derive the admin gRPC port from the HTTP address. ClickHouse/ClickHouse#115512 pins -admin.port on their side; this makes that pin unnecessary. Fixes #10838 * admin: let services win the legacy ports, close compat listener on failure Review findings on the compat listeners: - Do not put the legacy ports in mini's reservedPorts: an explicit -s3.port=23646 or -master.port.grpc=33646 must be honored, not rejected or stolen by the early reservation. Mini now checks the finalized port set and skips only the conflicting compat listener, so HTTP and gRPC compat degrade independently and the service keeps its port. - Also skip the legacy HTTP listener when a mini service owns the port: a wildcard-bound service can still leave loopback bindable, which would let the compat listener shadow it for loopback clients. - Close legacyServer's raw listener in the deferred cleanup: Serve may not have registered it yet when startAdminServer returns early. - Helm: add admin.legacyPort (default 23646) and expose the legacy http/grpc ports through the admin Service, container ports and NetworkPolicy so older workers resolved via the Service keep working. * helm: only pass -port.legacy when it differs from the binary default ct install deploys the released image with the new chart, and 4.48 does not know -port.legacy, so the admin pod crash-loops. Omitting the flag at the default value is behavior-identical on new binaries (their default is also 23646) and keeps older releases installable; legacyPort: 0 or a custom port still renders the flag. |
||
|
Chris Lu
|
887b1aac76 |
ec: carry the source volume's TTL expiry into the worker-written .vif (#11777)
* ec: carry the source volume's TTL expiry into the worker-written .vif ReadVolumeFileStatus now reports the volume's ExpireAtSec (last write + TTL), and the admin worker stamps it into the .vif it distributes with the shards. Previously every worker-encoded .vif had expireAtSec 0, so EC volumes of TTL collections never expired. * ec: report expiry from the Rust volume server and cover the .vif carry The seaweed-volume drop-in server omitted expire_at_sec from its ReadVolumeFileStatus response, so worker-encoded volumes sourced from it lost the TTL. Fall back to the nested VolumeInfo expiry for servers that predate the top-level field, and pin the nonzero/zero carry with a test. |
||
|
Chris Lu
|
921535411b |
s3: default allowDeleteBucketNotEmpty to false (#11769)
* s3: default allowDeleteBucketNotEmpty to false DeleteBucket on a non-empty bucket silently destroyed every object, while AWS returns 409 BucketNotEmpty. Clients like 'aws s3 rb' rely on that refusal as the 'delete only if empty' safety net. Deployments that need the recursive delete can opt back in with -allowDeleteBucketNotEmpty=true. * test: pin allowDeleteBucketNotEmpty=true on versioning config1 The config1 run relied on the flag's old true default to contrast with config2's false. Pin it so the two modes stay distinct. * test: drop redundant allowDeleteBucketNotEmpty=false in s3 integration test False is the default now, so TestS3DeleteBucketNotEmpty exercises the out-of-box behavior. |
||
|
|
6840371774 |
sftp: keep a file's existing bytes when it is opened for write without truncate (#11723)
* sftp: keep a file's existing bytes when it is opened for write without truncate * sftp: preserve sparse gaps and refuse unreadable entries when copying existing content The copy used a single chunk-stream read, which ends at the first sparse gap: the tail past a hole was dropped, and a remote-only entry with no local chunks copied zero bytes, so closing the handle uploaded an empty file. Read through PrepareStreamContent, which fills unstored regions with zeros, and refuse entries that carry no local bytes at all. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: report a failed destination write from a completed chunk fetch retriedStreamFetchChunkData broke out of the retry loop on a clean HTTP end before checking writeErr, so a destination write that failed mid- stream returned a short write and nil error. Check writeErr first. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * sftp: cover remote and all-holes sizes when copying existing content Take the declared size from RemoteEntry when it exceeds FileSize, pad inline content to the declared size, and treat a chunkless local file as all holes rather than unreadable. A remote-only entry still refuses the copy, now checked with a zero FileSize as well. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * sftp: size the existing-content copy the way filer.FileSize does A locally shortened file still names its older, larger remote object; taking RemoteSize unconditionally padded the copy back out to it. filer.FileSize only honors the remote size when the remote is newer, and also covers the chunk-extent size. A remote-only entry still refuses the copy whenever it points at remote bytes. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Martijn HoekstraandClaude Opus 5.5
|
1076ebdf4a |
test: add build constraints to Unix-only files (#11712)
test/fuse_dlm, test/fuse_p2p/peer_chunk_sharing_test.go and unmaintained/check_disk_size use syscall.Flock, syscall.Stat_t and syscall.Statfs, which do not exist on Windows. Without constraints, gopls and go vet on a Windows host report them as undefined. fuse_p2p/framework_test.go already carries `linux || darwin`; the FUSE test files now match it. check_disk_size uses the same constraint as weed/stats/disk_supported.go. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Chris Lu
|
c023493cf0 |
s3api: tolerate session tokens on statically configured credentials (#11694)
* s3api: consolidate session token extraction into extractSessionToken Three call sites duplicated the same header/header/query lookup; share one helper named after the existing s3tables equivalent. * s3api: tolerate session tokens on statically configured credentials Credential vendors like Unity Catalog emit a session token with every vended credential, including static ones (UC's StaticAwsCredentialGenerator only engages when s3.sessionToken is set). Requests signed by a configured access key were routed to STS validation and rejected, so static credential vending never worked against SeaweedFS. Resolve the access key first: when it maps to a configured credential the signature alone authenticates the request, and the attached token is marked ignored so authorization does not route it into the STS session-policy path. STS-issued access keys are never in the static map, so temporary credentials still validate their token exactly as before. * s3api: cover static credentials carrying a session token * test: exercise UC static credential vending against SeaweedFS s3.sessionToken.0 selects UC's StaticAwsCredentialGenerator, which vends the configured keys verbatim. The vended session token is foreign to SeaweedFS and previously failed SigV4; now it round-trips through temporary-table-credentials into real S3 I/O. * s3api: reject temporary credentials in GetFederationToken after auth Token presence alone cannot distinguish a temporary credential from a statically configured one carrying a vended token. Move the check behind verifyV4Signature and key it on the operative session token so tolerated tokens keep the caller eligible. * s3api: test GetFederationToken with authenticated temporary credentials Sign the rejection cases with real session credentials so they reach the post-auth check, and cover a vended static credential being accepted. * test: check vended-credential delete error in UC integration test |
||
|
Chris LuandDevin
|
440249a270 |
test: stop the compat server even when startup fails (#11659)
* test: stop the compat server even when startup fails test-with-server depended on start-server as a prerequisite, so a failed startup skipped the recipe and the spawned weed mini kept the port. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: verify the recorded PID is weed before stopping it A prerequisite failure leaves a stale weed-server.pid in place; the PID may have been reused by an unrelated process. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
30cf53265b |
deps: pin seaweedfs/goexif at v1.0.3 and stop dependabot re-bumping it (#11648)
* deps: pin github.com/seaweedfs/goexif back to v1.0.3 The fork's newest tagged release is v1.0.3; the v2.0.0+incompatible requirement resolved to older, untagged code and breaks isolated builds that fetch it from the proxy. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: stop dependabot bumping seaweedfs/goexif past its latest tag Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test/kafka: settle goexif at v1.0.3 too The kafka test module recorded v2.0.0+incompatible in its own requires, so MVS kept selecting it over the pinned v1.0.3 in the root module. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
56fbc3deb5 |
s3api: align S3/IAM error responses with AWS (#11632)
* s3err: add InvalidArgument and AuthorizationHeaderMalformed codes Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: answer unrecognized bucket PUT sub-resources with 501 A PUT on a bucket carrying an unrecognized query (logging, metrics, intelligent-tiering, ...) fell through to the bare CreateBucket route and returned BucketAlreadyOwnedByYou or re-created the bucket. AWS answers these with NotImplemented. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: reject malformed copy-source and multipart PUT parameters A malformed X-Amz-Copy-Source or a non-numeric partNumber fell through to the plain PutObject route and stored the body as a regular object. Answer them with InvalidArgument-class errors instead of writing data. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: verify x-amz-content-sha256 against the streamed body A PUT carrying a hex or base64 payload hash now streams through a verifier that reports a mismatch once the stream is exhausted, instead of storing an object that does not match its declared hash. The error is deferred so intermediate reads that drop (n>0, err) results cannot silently swallow it. Sentinel values (unsigned/streaming payloads) remain exempt and malformed values fail fast. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: map truncated SigV4 headers to the error for the missing field AWS answers an Authorization header missing Credential= with InvalidArgument and one missing or malformed Signature= with AuthorizationHeaderMalformed, instead of a generic MissingFields. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: answer IAM/STS failures in the query-protocol envelope Embedded IAM and STS routes now report authentication, form-parse and authorization failures with the IAM ErrorResponse body instead of the S3 Error envelope, so IAM SDK clients can parse them. Requests signed for s3 keep the S3 envelope, keyed off the credential scope. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: return 403 AccessDenied when the request has no Date Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: answer throttling rejections as SlowDown ErrTooManyRequest and ErrRequestBytesExceed reported made-up codes; AWS serves these throttling rejections as SlowDown. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: reject versionId requests on buckets that never had versioning GET, HEAD and DELETE carrying a non-empty versionId on an unversioned bucket now fail with InvalidArgument instead of being answered as a plain object request. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: answer DeleteObjects over 1000 keys with MalformedXML Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: InvalidArgument for non-numeric or out-of-range part numbers partNumber=abc, 0 and >10000 all resolve to InvalidArgument, matching AWS, instead of InvalidPart. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iam: correct LimitExceeded, InvalidAction and ServiceFailure mappings LimitExceeded is a conflict (409), an unknown Action is InvalidAction (404) rather than NotImplemented, and internal failures report the IAM receiver fault type. Applies to both the embedded IAM endpoint and the standalone iamapi server. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iam: refuse DeleteUser while access keys remain Deleting a user with live credentials orphaned its access keys; AWS answers DeleteConflict until they are removed first. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test/s3: add S3/IAM error-response compatibility harness * s3: return InvalidArgument for malformed x-amz-content-sha256 A header value that decodes to neither 32-byte hex nor base64 is a malformed argument, not a hash mismatch. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: stop spawned mini when readiness times out A slow-starting server otherwise survives the failure path and keeps the S3 port occupied for the next run. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: register atexit cleanup before setup A failed setup previously skipped cleanup, leaking the bucket and IAM user on persistent servers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: reject unrouted subresources on the DELETE bucket catch-all PutBucketHandler gained the same guard when the route-level check moved into the handlers; DeleteBucketHandler was missed, so an authorized DELETE /bucket?logging could delete the bucket instead of answering NotImplemented. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: tolerate unset fixture variables in cleanup and cover DELETE ?logging Cleanup now runs its IAM/multipart steps only when setup reached them, so an early setup failure still removes the bucket. Added a DELETE bucket-subresource case asserting NotImplemented and bucket survival. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: fail a case when its side-effect check reports a regression A non-empty check note now fails the case, so a deleted bucket or an object created by a malformed request cannot slip through behind a passing status check. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
3a65365f6e |
build(deps): bump org.apache.spark:spark-core_2.12 from 3.5.7 to 3.5.8 in /test/java/spark (#11616)
build(deps): bump org.apache.spark:spark-core_2.12 in /test/java/spark Bumps org.apache.spark:spark-core_2.12 from 3.5.7 to 3.5.8. --- updated-dependencies: - dependency-name: org.apache.spark:spark-core_2.12 dependency-version: 3.5.8 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
2b057eccbf |
build(deps): bump com.fasterxml.jackson.core:jackson-databind from 2.22.2 to 2.22.3 in /test/java/spark (#11548)
build(deps): bump com.fasterxml.jackson.core:jackson-databind Bumps [com.fasterxml.jackson.core:jackson-databind](https://github.com/FasterXML/jackson-databind) from 2.22.2 to 2.22.3. - [Commits](https://github.com/FasterXML/jackson-databind/compare/jackson-databind-2.22.2...jackson-databind-2.22.3) --- updated-dependencies: - dependency-name: com.fasterxml.jackson.core:jackson-databind dependency-version: 2.22.3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Khris RichardsonandClaude Opus 5.5
|
895d49b55b |
s3/iam: manage roles through the IAM API, with an opt-in persistent role store (#11522)
* s3/iam: manage roles through the IAM API, with an opt-in persistent role store
Roles could only come from the IAM config file: the S3 server pinned the
role store to memory and the embedded IAM API had no role actions, so a
role could not be created, retrusted or revoked without editing the file
and restarting every gateway.
Role store
- Read the `roleStore` key (the IAMConfig field already existed). With an
IAM config file the default stays memory; with none it is the filer, as
for OIDC providers, so zero-config clusters keep runtime-created roles.
- Roles from the IAM config file never go into a persistent role store,
which outlives the file and may be shared by S3 servers with different
files. They are served from memory beneath the store, as OIDC providers
are: a stored role of the same name takes precedence, and deleting it
restores the file's. A config-file role cannot be changed or deleted
through the API (UnmodifiableEntity), and removing one from the file
removes it at the next start. An in-memory store holds them as records,
as before. They have no creation time, so CreateDate is omitted rather
than reporting when this server started. SetRoleStore installs a store
the same way, so a store set after startup keeps the config-file roles,
as SetOIDCProviderStore does for providers.
- Watch /etc/iam/roles and drop the cached role definitions on change. The
cached filer store otherwise serves a peer's stale role for up to its 5m
TTL, which keeps a revoked trust policy in force on the other gateways.
- Role stores wrap ErrRoleNotFound for a missing role; the filer store
used to report any failed lookup as "role not found". CreateRole proceeds
only on a confirmed absence, so an unreadable store cannot let it write
over an existing role.
IAM actions
- CreateRole, GetRole, ListRoles, DeleteRole, UpdateAssumeRolePolicy,
AttachRolePolicy, DetachRolePolicy, ListAttachedRolePolicies. The reads
are allowed in read-only mode.
- A role defined in the config file is reloaded from it at every start, so
changing or deleting it through the API is refused (UnmodifiableEntity)
rather than silently reverted.
- DeleteRole with policies attached is refused (DeleteConflict), as on AWS.
- Role names follow AWS's rules ([\w+=,.@-]{1,64}); a role is stored as
<name>.json in the filer, so this also keeps a name from leaving the role
store's directory. At most 10 managed policies per role (AWS's default
quota; MaxManagedPoliciesPerUser is 10 too), LimitExceeded beyond.
- DeletePolicy is refused (DeleteConflict) while a role attaches the
policy, as it already is for users and groups: roles attach policies by
name, so a policy created later under the deleted one's name would
otherwise take effect on the role.
- Role paths other than "/" and role tags are not stored, so they are
refused rather than dropped.
Role IDs and sessions
- Roles get a unique RoleId when first stored (random, AWS AROA form),
kept across updates; a config-file role gets a stable ID derived from its
name, since it is created again at every start.
- Sessions issued through AssumeRoleWithWebIdentity, AssumeRoleWithCredentials
and AssumeRole carry the role's ID (claim "rid"), and a request under a role
whose current ID differs is denied. Resolving a session's policies by role
name let a session outlive its role: once a role was deleted, a role later
created under the same name — with a different trust policy and different
policies — revived every unexpired session of the old one with the new
role's permissions. Sessions issued before this change carry no ID and are
unaffected until they expire.
Integration test (test/s3/iam, run with `make start-services`):
TestWebIdentityWithProviderAndRoleManagedThroughIAMAPI configures an OIDC
provider, a managed policy and a role entirely through the IAM API against a
JWKS served by the test, then checks the trusted subject gets credentials
scoped to the attached policy; another subject, a token signed by another
key, an unsigned token and a token for another audience are refused; and UpdateAssumeRolePolicy moves the
trust at once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: bind every role session to its role, and change roles atomically
Review follow-ups.
Session binding
- The role-ID check ran only when a session carried no policy names, and
AssumeRole embeds the role's attached policies, so those sessions kept
their permissions after the role was deleted or recreated. The check
now runs for every session carrying a role ID, before policy selection.
- A named role that cannot be resolved at issuance gets no session,
instead of one with no role ID (which nothing binds).
- A config-file role's ID is derived from its name and trust policy, not
the name alone: a different role put in the file under the same name
gets a new ID, while an unchanged role keeps its sessions across restarts.
Role writes
- RoleStore gains UpdateRole, a read-modify-write that lands only if the
role is unchanged since the read, and otherwise re-reads and retries. The
filer store uses the filer's write conditions (IF_NOT_EXISTS for a new
role, IF_ENTRY_EQUAL otherwise). CreateRole, UpdateAssumeRolePolicy and
Attach/DetachRolePolicy all go through it, so two gateways no longer
overwrite each other's changes, a change racing a delete no longer
writes the role back, and of two concurrent creates one gets
EntityAlreadyExists.
- The filer store's ListRoles pages past 1,000 entries and fails on a
broken stream instead of returning what arrived, so DeletePolicy's
attachment check sees every role. ListRoles skips a role deleted between
listing and reading it.
- CreateRole validates first; a failed write is ServiceFailure, not
InvalidInput. Any Tags.* parameter is refused, not only the first key.
- ExecuteAction's skipPersist covers the S3ApiConfiguration only; the
comment now says so. Role and OIDC provider actions write their own stores.
Each fix has a test that fails without it. Against a real filer with two
gateways, concurrent AttachRolePolicy calls lost 1-4 of 8 attachments per
run before this change and none after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: one role snapshot per decision; DeleteRole is atomic; watch a custom role store path
Review follow-ups.
- Authorization evaluates the policies of the role definition the session's
binding was checked against, instead of reading the role again: a role
replaced in between cannot lend a session its policies.
- AssumeRole and AssumeRoleWithLDAPIdentity issue the session from the
definition whose trust admits the caller (IAMManager.ResolveRoleForPrincipal),
and take its ID, duration cap and embedded policies from that same
definition. A role replaced after the caller's trust check by one that does
not trust the caller now yields AccessDenied, not a session bound to the
replacement.
- A RoleUpdate that returns nil deletes the role, on the same condition as a
write: the filer store deletes with ObjectTransaction on IF_ENTRY_EQUAL,
routed and locked like the conditional CreateEntry. DeleteRole decides
against the role it deletes, so a policy attached meanwhile on another
server is a DeleteConflict, and a delete never removes a role written
after its check.
- S3 servers watch the role store's configured basePath, not only
/etc/iam/roles, so a custom path also drops peers' cached roles on change.
Each has a test that fails without it. Live against a real filer: DeleteRole
refuses while a policy is attached and removes the entry once detached; all
test/s3/iam CI stages pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: state which roles DeletePolicy's attachment check can see
RolesAttachingPolicy sees the stored roles and this server's config-file
roles. A role defined only in another server's IAM config file is invisible
to it, so a config-file role that attaches a managed policy is protected
only on the servers whose file defines it. The doc comment now says so and
how to avoid it: keep such roles in every server's file, or attach only
config-file policies to config-file roles.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* iam: note that a role store set after startup is not watched for peer changes
S3 servers build their metadata watch list once, at startup, from the role
store installed then. SetRoleStore's doc now says that a filer-backed store
installed later with a different basePath is not watched, so peers' changes
to it reach this server's cached roles only when the cache expires.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
Chris LuandDevin
|
757917f564 |
filer: evict remote-cached objects under storage pressure (#11515)
* filer: identify remote-mounted entries safe to drop under disk pressure ListEvictableRemoteEntries walks every mounted directory directly on the filer store (no lazy remote listing) and returns entries that hold local chunks fully synchronized with remote, ordered oldest-cached first. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: evict remote-cached chunks oldest-first and vacuum the garbage uncacheRemoteEntry applies the same transition remote.uncache does - cleared chunks plus a reset LastLocalSyncTsNs under the entry path lock - and evictRemoteCachedEntries serializes passes over all mounts until a byte target is met. Aged victims are preferred; a second pass accepts any synchronized cached entry when aged ones cannot cover the request, since a failed read is worse than a dropped hot object. Cleared chunks only become disk space after compaction, so reclaimRemoteCacheSpace pairs each pass with a rate-limited VacuumVolume call that also picks up orphaned partial fills. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: trigger remote cache eviction under storage pressure A periodic check (30s) reads disk usage from master topology and evicts remote-mounted cached chunks once any disk crosses -filer.remoteCacheEvictThreshold (default 0.9; 0 disables), with a vacuum pass to reclaim the tombstoned needles. The cold-read cache path also kicks the same reclaim when a fill fails on exhausted volumes - the request still falls back to streaming from the remote, but the cache stops being permanently wedged full. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: flush deletion queue before remote cache vacuum Vacuum ran immediately after eviction while evicted file IDs still sat in the asynchronous deletion queue, so compaction saw no garbage and the cache stayed wedged. Flush the queue synchronously first and shorten the vacuum cooldown so sustained pressure does not wait five minutes between reclaim passes. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover remote cache eviction under capacity pressure Unit tests pin the eligibility filter and oldest-first ordering; the integration test runs a constrained two-node setup that saturates the cache, verifies the oldest synced entry is evicted and vacuumed, and that a later read re-caches it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: coalesce remote cache reclaim passes A failed cache fill used to queue behind any in-flight eviction, stacking full mount traversals during a write-failure storm. Skip the pass when one is already running; the caller falls back to streaming from remote regardless. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: stop the remote cache janitor on shutdown The eviction ticker kept running after Shutdown closed the metadata store and could traverse a closed store. Give the janitor a context cancelled from Shutdown and propagate it into its master RPCs and traversals. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: vacuum only tombstoned volumes and retry deferred passes VacuumVolume with no volume id swept every collection, compacting volumes unrelated to the cache fill that failed. Now the reclaim path collects the vids of file ids actually flushed from the deletion queue and compacts only those. Vids that land inside the vacuum cooldown stay in a pending set the janitor retries on each tick, so chunks evicted just after a sweep are not stranded until the next pressure event. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: count only pressured disks when evicting remote cache The janitor measured the largest excess on one disk but let bytes on healthy disks satisfy the reclaim target. Split the topology disk view per physical disk and count only chunk bytes whose volumes sit on an over-threshold disk; entries contributing nothing there are skipped. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: compare remote cache sync time at nanosecond precision Second-precision mtime comparisons let a local write in the same second as the last sync still qualify as evictable, discarding unsynced changes. Compare LastLocalSyncTsNs against full-precision mtime (mtime_ns round-trips through the entry codec), and apply the same fix to remote.uncache's inline check. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: invalidate remote sync stamp on local content change A local overwrite that keeps the remote entry's LastLocalSyncTsNs looks evictable even though the remote copy no longer matches, and some write paths stamp mtime at second precision so a timestamp comparison cannot catch it. UpdateEntry now clears the stamp when chunks change without a fresh stamp, leaving replicated updates authoritative. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: bound remote cache master rpcs and vacuum all evicted garbage VolumeList and VacuumVolume now run under a 30s context so a stalled master cannot wedge the eviction janitor. The targeted vacuum drops the garbage threshold so volumes with under 10% deleted bytes still compact. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: tolerate straggler fills in remote cache eviction test Detached fills from the concurrent wave keep racing the final checks: live chunks legitimately fill both volumes, and a re-cached object can be evicted again before its commit is observed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: start remote cache eviction loop after filer init The janitor's first tick dereferences fs.filer; starting the goroutine before NewFiler assigns it could panic when startup exceeds an interval. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: keep remote cache vacuum intent across retries Evicted entries now record their chunk volumes for vacuum directly, so the intent survives whoever consumes the shared deletion queue first. A pending volume keeps several vacuum attempts so tombstones that land late are still compacted, and the janitor retries pending volumes under the reclaim mutex instead of flushing unrelated deletes every tick. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: bound each remote cache vacuum request independently A shared 30s deadline across pending volumes let one slow compaction cancel the rest. Each VacuumVolume now gets its own context, and pending volumes keep more attempts since the master reports request acceptance rather than compaction. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: tighten remote cache reclamation bound Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: treat chunk timestamp changes as content changes chunksEqual now also compares ModifiedTsNs so an update that rewrites a chunk record still invalidates the remote sync stamp. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: run remote cache queue flush under the reclaim context BatchDelete for flushed file ids now uses the caller's context instead of context.Background(), so a reclaim pass bounded by shutdown or timeout stops its deletes too. Other callers keep their existing behavior. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: scope remote cache vacuum to evicted volumes The flush no longer feeds the shared deletion queue's ids into the pending set — only evicted chunks' volumes are tracked, so ordinary deletions no longer pick up repeated vacuum attempts. The flush also runs under a shutdown-immune bounded context and is skipped when no volume is pending. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: retry remote cache vacuums even after unmount Pending volumes were only retried while a remote mount existed; removing the last mount skipped every later pass and left evicted bytes allocated. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: reclaim partial cache fills that run out of capacity A fill that fails midway queues its written chunks for deletion, but when no entries remain evictable the reclaim pass found no pending volumes and skipped the flush and vacuum entirely, leaving the partial garbage to the slow periodic vacuum while the disk stayed full. Mark the failed fill's chunk volumes pending so the pass tombstones and compacts them even when nothing was evicted. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
67b19b2db5 |
build(deps): bump com.fasterxml.jackson.core:jackson-databind from 2.22.1 to 2.22.2 in /test/java/spark (#11511)
build(deps): bump com.fasterxml.jackson.core:jackson-databind Bumps [com.fasterxml.jackson.core:jackson-databind](https://github.com/FasterXML/jackson-databind) from 2.22.1 to 2.22.2. - [Commits](https://github.com/FasterXML/jackson-databind/compare/jackson-databind-2.22.1...jackson-databind-2.22.2) --- updated-dependencies: - dependency-name: com.fasterxml.jackson.core:jackson-databind dependency-version: 2.22.2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Ping QiuandClaude Opus 5
|
635f69a821 |
test(s3tables): add an OLake Iceberg catalog integration test (#11441)
OLake (github.com/datazip-inc/olake) is a CDC and ingestion engine that writes Apache Iceberg. It covers two paths none of the existing catalog suites reach. It is a strict Java Iceberg client. OLake does not write Iceberg from Go — its Go process spawns a Java sidecar over gRPC and writes through the official Apache Iceberg library, because the Go library has no equality deletes and CDC needs them. That makes it the client class weed/s3api/iceberg's metadata compliance backfill exists to serve: the one that fails with "Cannot parse missing long current-snapshot-id" when spec-required keys are omitted. And it produces equality deletes. Its upsert path commits operation=overwrite with an equality-delete file and a delete manifest. ClickHouse, Doris, Trino, Spark and DuckDB all only append, so nothing else in this directory exercises a delete manifest at all. Six subtests: the destination check reaches SUCCEEDED and actually loads the REST catalog; discover enumerates the source; a full sync commits a snapshot; PyIceberg reads back what the Java writer committed; an update plus a re-sync records an overwrite carrying equality deletes with a delete manifest in the current snapshot; and the catalog does not rewrite manifests the official Java writer produced, which gives the manifest-repair path a negative test to go with the ClickHouse positive one. What it deliberately does not assert is a delete-applied read. PyIceberg refuses to scan a table carrying equality deletes (apache/iceberg#6568) while reading its metadata fine, and an engine that can apply them costs a multi-gigabyte image. Recording the commit correctly is the catalog's contract; applying deletes on read is the engine's. The README says so, says the read half was verified by hand once with StarRocks, and warns against later "upgrading" this to a PyIceberg rows read — which would either fail or, if PyIceberg ever starts skipping deletes instead of raising, pass by not looking. The workflow job asserts the suite actually ran rather than trusting a green exit, for the same reason: at least one top-level PASS and zero SKIP. No product change — the destination config is the generic catalog_type=rest with the standard OAuth2 client-credentials flow, and s3_path_style is not even set, since OLake turns it on itself whenever s3_endpoint is non-empty. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Chris Lu
|
b77c42ff32 | test: use BaseEndpoint for the checksum test client | ||
|
Chris Lu
|
17ad5a1419 |
s3: accept FULL_OBJECT checksums without per-part checksums at complete
COMPOSITE uploads must still carry every part checksum in the complete request, but FULL_OBJECT uploads may instead supply the whole-object checksum in an x-amz-checksum-* request header. Compare that header against the computed object checksum and return BadDigest on mismatch, matching AWS. |
||
|
Chris Lu
|
b03419ee92 |
s3: reject UploadPart checksum algorithms conflicting with the upload
An UploadPart that explicitly selects a different checksum algorithm than the one declared at CreateMultipartUpload would store a checksum CompleteMultipartUpload could never accept. Reject the conflict up front with InvalidRequest, matching AWS. |
||
|
Chris Lu
|
f849b7c823 |
s3: validate per-part checksums in CompleteMultipartUpload
Parse the Checksum* elements of each completed part and enforce what AWS does for uploads created with x-amz-checksum-algorithm: every part must carry a checksum in the complete request (InvalidRequest when missing, BadDigest when it differs from the stored part checksum), and an x-amz-checksum-type header must match the upload resolved checksum type (BadDigest). Add the issue-11401 reproduction as a regression test. |
||
|
Chris Lu
|
f15b980976 |
s3: UploadPart inherits the checksum algorithm of its multipart upload
AWS computes a checksum for every part of an upload created with x-amz-checksum-algorithm, even when the part request carries no checksum headers. Mirror that: when the part request specifies no algorithm, apply the one stored on the upload entry so the part entry keeps a checksum CompleteMultipartUpload can fold into the object checksum. |
||
|
Chris Lu
|
01bb3b3053 |
s3api: add Snowflake s3compat API integration tests (#11394)
* s3api: add Snowflake s3compat API integration tests Run the upstream snowflakedb/snowflake-s3compat-api-test-suite against a local SeaweedFS server in CI. test/s3/snowflake/run.sh starts weed server with S3 (-s3.autoCreateBucket=false so missing-bucket PUTs return NoSuchBucket), prepares the fixtures the suite needs (versioned bucket, deny-all-policy bucket, >1000-object prefix), clones the suite, patches it to path-style addressing, and runs mvn -Dtest=S3CompatApiTest. The suite also exposed that GetBucketLocation returned 404 NoSuchBucket for a malformed bucket name; validate the name first and return 400 InvalidBucketName like AWS. * test: harden snowflake s3compat runner per review - Pin the upstream suite to a tested commit (SUITE_REV) instead of the moving default branch - Bind the test server to loopback only - Require the AccessDenied error code when verifying the denied bucket - Fix README so go install runs in a subshell - checkout with persist-credentials: false - Make the concurrency group unique per PR, and widen path filters to the storage/operation/wdclient/cluster/pb packages the S3 stack uses * test: advertise loopback ip for snowflake test server -ip.bind 127.0.0.1 alone left the volume server advertising the host's primary address, so chunk uploads were refused. Also set -ip 127.0.0.1 and disable the Iceberg/Lance listeners so the harness is loopback-only and does not collide with other local services. |
||
|
Chris Lu
|
37bf1cd91d |
volume: validate copy/tail source addresses before dialing (#11390)
* pb: stop exiting the process on malformed server addresses ServerToGrpcAddress and GrpcAddressToServerAddress called glog.Fatalf when hostAndPort could not parse the port, which os.Exit(255)ed the whole process. A caller-supplied copy or tail source address reached this path synchronously in the serving goroutine, so one anonymous VolumeCopy with a non-numeric port terminated the volume server. Log the parse error and return the input unchanged instead: the dial or request that consumes the address then fails as an ordinary error. * volume: validate copy and tail source addresses before dialing VolumeCopy, VolumeEcShardsCopy and VolumeTailReceiver dial a caller-supplied source address (SourceDataNode / SourceVolumeServer) with no endpoint validation, so an anonymous caller could aim the volume server at loopback, link-local (cloud metadata) or other unintended destinations and read dial behavior back as a connectivity oracle. Apply the same peer-target deny list FetchAndWriteNeedle uses for replica targets: the source must be a bare host:port whose host is not loopback, link-local or unspecified; cluster peers stay reachable on private networks, and -volume.allowUntrustedRemoteEndpoints opts out. The loopback-using copy tests set the flag to keep exercising the copy path in process. * rust volume: validate copy and tail source addresses before dialing Mirror the Go guard on the Rust volume server: volume_copy, volume_ec_shards_copy and volume_tail_receiver dial a caller-supplied source address, so run it through validate_replica_target first (bare host:port; no loopback, link-local or unspecified hosts; private peers stay allowed). --volume.allowUntrustedRemoteEndpoints opts out; the test fixture and the Rust test-cluster launcher set it so loopback sources in tests keep working. * volume: pin validated copy/tail source addresses at dial time validateReplicaTarget resolves the source hostname once, but the gRPC client resolved it again at connect, leaving a DNS-rebinding window for hostname sources. The copy and tail source dials now run through the same guardedDialerPolicy the remote-storage path uses, so every resolved address is re-checked against the replica deny list (private peers allowed) immediately before the TCP connect. guardedDialerPolicy also moves to util.OutboundDialContext so the guarded path keeps the -ip.bind source binding the default gRPC dialer had. The Rust volume server mirrors this with connect_guarded, a tonic connector that resolves, re-checks each address, and connects to the first passing IP; handlers use it whenever the untrusted-endpoint opt-out is off. A handler-level test now exercises the enabled validation branches for all three source-taking RPCs. * pb: return empty server address for malformed grpc addresses GrpcAddressToServerAddress used to return the unparseable input on a hostAndPort failure, so a malformed raft address (e.g. "host:abc") flowed into admin dashboard master maps unchanged. Return an empty string instead, skip empty conversions at the two raft-cluster merge sites, and drop the now-stale comment about the fatal exit the earlier commit removed. * test: opt erasure-coding loopback clusters out of the remote endpoint guard The erasure-coding suites drive VolumeEcShardsCopy / VolumeCopy between volume servers bound to 127.0.0.1, which the copy/tail source guard now rejects by default. Pass -volume.allowUntrustedRemoteEndpoints to the test volume launches, matching what the volume_server framework harnesses already do. * admin: only claim fallback master leadership on an empty raft response A nonempty RaftListClusterServers response whose entries were all rejected left masterMap empty, so the fallback marked the reachable current master as leader the same way a genuinely empty (non-raft) response does. Track whether the successful response returned zero servers and only promote the fallback master then. |
||
|
Chris Lu
|
0ca1c19821 |
s3api: unify auth error handling across s3tables, iceberg and lance (#11381)
* s3api: fail closed when S3 Tables signature verification fails * s3api: avoid nil Account dereference in S3 Tables auth log * iceberg: return auth error instead of falling back to DefaultAllow * lance: return auth error instead of falling back to DefaultAllow * s3api: stop trusting client-supplied s3-account-id The header is set by the server after successful authentication; scrub inbound values alongside the other internal headers, and apply the same admin guard to the header fallback branch of getAccountID that the identity branch already has. * test: cover table-catalog auth wrappers and principal resolution * test: configure anonymous identity where catalog clients do not sign * s3api: scrub s3-account-id after signature verification |
||
|
Chris Lu
|
f40687b34e |
s3: tighten STS session token handling (#11383)
* s3api: test that a session token must not reveal its credential * sts: derive secret access key with HMAC keyed on the signing key * s3api: stop accepting STS session tokens as bearer credentials * security: reject STS session tokens on filer and admin gRPC auth * test: sign s3/iam framework requests with the session credential * s3api: exercise the real auth pipeline in the end-to-end harness |
||
|
Chris Lu
|
4fd67001d9 |
security: require go 1.26.6 and bump vulnerable deps (#11374)
* security: require go 1.26.6 and bump vulnerable deps A dependency scan of the 4.47 release flagged the bundled toolchain and modules: - github.com/golang/go < 1.26.6 (CVE-2026-39821, CVE-2026-56853, CVE-2026-56859, CVE-2026-56862, CVE-2026-56864, CVE-2026-56865, CVE-2026-33818, CVE-2026-46600): raise the go directive to 1.26.6 so every built artifact requires the fixed toolchain. - google.golang.org/grpc (CVE-2026-84445, CVE-2026-84304): move to the fixed dev pseudo-version; released tags through v1.85.0-dev remain in the affected range. - github.com/pelletier/go-toml/v2 <= v2.4.2 (unbounded parser recursion): v2.4.3. - alpine libcrypto3/libssl3 < 3.5.8-r0 (CVE-2026-75803, CVE-2026-63073, CVE-2026-63075, CVE-2026-63076, CVE-2026-63072, CVE-2026-54874, CVE-2026-18798, CVE-2026-14456, CVE-2026-14457): the release images already apk-upgrade the final stage; extend the same to the telemetry and admin-integration images. Same bumps applied to the test/kafka, test/sftp, kafka-client-loadtest, and telemetry/server modules. * telemetry: send integration test report above the 10 GiB floor The collect endpoint keeps reports only when TotalDiskBytes >= proto.MinDiskBytes, but the integration test still sent 1 GiB, so the server counted the report and skipped storing it. No cluster_id series was ever created and /metrics lacked seaweedfs_telemetry_volume_servers. Send just above the floor (via proto.MinDiskBytes so it cannot silently drift again) so the expected per-cluster metrics are exported. |
||
|
David ChristopherandChris Lu
|
e4ca0d09e7 |
s3: preserve versions for POST policy uploads (#11316)
* s3: preserve versions for POST policy uploads Route POST policy uploads through the existing version-aware write helpers and validate promoted Object Lock headers before writing. Return the generated version ID when versioning is enabled, return x-amz-version-id: null when versioning is suspended, and omit the header when versioning has never been enabled. * s3: reuse versioning helpers in POST policy handler Route the POST policy handler through the existing getVersioningState and isObjectLockEnabled helpers instead of open-coding the object-lock forces-versioning-enabled rule, matching the PUT path. Drop the x-amz-version-id: null response header for suspended versioning; the PUT handler omits it and the S3 PutObject sample response for suspended buckets does not include it. Trim the moved fileSize comment. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
0e82b4e351 |
s3: populate Initiated timestamp in ListMultipartUploads (#11313)
* s3: populate Initiated timestamp in ListMultipartUploads ListMultipartUploads returned each upload with only Key and UploadId, omitting the Initiated timestamp. Clients such as GeeseFS rely on this field to expire stale uploads and crash on its absence. Set Initiated from the upload directory entry creation time so repeated listings preserve the original initiation time. * test/s3: verify Initiated timestamp in ListMultipartUploads Add an integration test that initiates a multipart upload, lists it, and asserts the Initiated field is populated and preserved across repeated listings rather than reflecting the listing time. |
||
|
Chris Lu
|
f4bad510c9 |
test/ec: pin rack in seedAndSpread volume.grow to stop silent no-ops (#11299)
seedAndSpread() calls `volume.grow -dataNode X` without pinning the rack.
The master's grow picks the rack by weighted-random when -rack is unset,
and only one of the three racks holds the requested data node, so an
unpinned grow lands on the wrong rack two times out of three. The
VolumeGrow RPC swallows the "No matching data node" failure for
non-cache collections, so those grows count as success without creating
a volume. The per-server cap (maxGrowsPerServer=4) is then exhausted by
silent no-ops before the volumes ever spread, and seedAndSpread times
out with "volumes never spread across >=2 disks on all 3 nodes".
Pin -dataCenter dc1 and -rack rack{i} alongside -dataNode so every grow
reaches the target node. This removes the timing-sensitive assumption
that made TestECVacuumDuplicateShardClaimAcrossDisks flaky.
|
||
|
Chris Lu
|
5d8a463b3e |
test/ec: fix EC interruption matrix slot exhaustion (#11295)
* test/ec: fix EC interruption matrix slot exhaustion The EC integration test cluster (test/erasure_coding/chaos_lifecycle_test.go) configured each disk with -max 4 and the seedAndSpread spread loop fired volume.grow -count 4 every 2 s with no per-server cap. Because the master topology lags the volume.grow writes, the loop re-fired before the prior grow was visible, over-filling disks to capacity. A full disk leaves zero free EC shard slots (failing the cluster-wide capacity check with "no free ec shard slots") and drops the source disk below the encode's FreeVolumeCount >= 2 health check (failing with "no healthy replicas"), which aborted ec.encode before any phase marker printed and made every encode scenario in TestECInterruptionMatrix fail. Three changes to the test cluster: 1. Raise -max from 4 to 8 per disk so the source disk always retains FreeVolumeCount >= 2 for ec.encode's 14-shard generation (2 volume-slot equivalents) even after the spread loop and multiple encodes. 2. Switch the spread loop from -count 4 to -count 1 so each grow lands exactly one volume on the volume server's least-loaded disk, giving deterministic cross-disk spreading instead of relying on a single multi-volume grow to fan out. 3. Cap grows per server at 4 so heartbeat lag cannot run away and over-fill disks before the master registers the prior grow. 4. Pass -minFreeSpace 0 so the test is not falsely gated by the physical disk's free-space percentage on the host running CI (the EC shard slot calculation separately enforces a 90 % disk-usage cap via balancer.DiskTooFullAfter, which already guards against an over-set maxVolumeCount on a physically full disk). Verified locally by running TestECInterruptionMatrix twice (all encode, decode, and balance scenarios pass, including the previously failing encode@Deletingoriginalvolumes). * test/ec: only count successful grows toward the spread cap A failed volume.grow (e.g. a transient collectTopologyInfo or VolumeGrow RPC error) would otherwise consume one of the four permitted attempts without creating any volume, exhausting the retry budget and leaving the loop to only poll until the Eventually timeout. Increment the per-server counter only when commandGrow.Do returns nil. |
||
|
Chris Lu
|
3b4a681e53 |
test(fuse_failover): dump chunk list and hex on append corruption (#11285)
* test(fuse_failover): dump chunk list and hex on append corruption
The failover append test (TestAppendWhileVolumeServerRestarts) failed
in CI with an 8-byte NUL region at offset 632 that appeared in both
the writer mount and the filer own view, but the failure message
only showed a quoted-string window around the divergence. That is
not enough to tell which chunk covered the zeroed bytes or which
volume server held it, so the next recurrence would be just as
unattributable.
Add a FileChunkList helper that reads the filer resolved chunk
list, and on failure dump:
- every chunk fid, offset, size, volume id, and current master
holders, flagging the chunk that covers the first divergence;
- a hex+ASCII dump of the writer mount around the divergence so
the exact zero-filled region is visible byte-for-byte.
No production code is touched; this only makes the test fail louder.
* test(fuse_failover): preserve diagnostic collection errors
Address review feedback from CodeRabbit and Greptile on PR #11285:
- FileChunkList now returns the wrapped ParseUint error when
fid.volume_id is zero and the file_id prefix is invalid, matching
FileVolumeIds instead of silently keeping vid=0 (which would
query /dir/lookup?volumeId=0 and report the wrong holders).
- dumpChunkList captures the VolumeHolders error and renders it as
'lookup failed: ...' so a failed master request is distinguishable
from a successful lookup with no holders (both previously showed
'holders=[unknown]').
- runChaosAppend captures the writer-mount read error and includes
it in the failure message so an unavailable writer view is not
mistaken for corrupted content.
|
||
|
Nguyễn Đăng Minh Lực
|
c968084b34 |
iceberg: fix OAuth token expiry handling (401 + token-exchange + configurable TTL) (#11242)
* iceberg: return 401 for invalid or expired Bearer tokens BUG-0001: when the OAuth JWT expired, Server.Auth fell through to the S3 SigV4 authenticator, which rejects the "Authorization: Bearer" scheme with NotImplemented — a 501. Iceberg clients (Java OAuth2Manager, pyiceberg) only refresh tokens on 401, so they retried the dead token forever: RisingWave sinks stalled and Doris catalog queries failed every token TTL (1h) until the client process was restarted. A request carrying a Bearer header is an Iceberg REST client: answer 401 (+ WWW-Authenticate: Bearer, RFC 6750) when the token fails, and only fall through to the S3 authenticator when no Bearer header is present. * iceberg: make OAuth token TTL configurable via ICEBERG_OAUTH_TOKEN_EXPIRY BUG-0001 follow-up: production evidence shows Iceberg Java 1.10.x clients (RisingWave connector node, Doris FE) never re-fetch tokens on 401 — the sink stalled again on token expiry even with the 501→401 fix, and no POST /v1/oauth/tokens appeared in server logs across dozens of retries. 401 is necessary but not sufficient for these clients. The TTL was hardcoded to 3600 with no knob. Read the expiry (seconds) from ICEBERG_OAUTH_TOKEN_EXPIRY, defaulting to 3600, so deployments can issue longer-lived tokens (e.g. 86400) to survive client restart cycles. * iceberg: support OAuth token exchange (RFC 8693) for client refresh Decompiling the Iceberg Java 1.10.1 client bundled with Doris FE showed the missing half of BUG-0001: OAuth2Manager refreshes via token-exchange (AuthConfig.exchangeEnabled defaults to true — the client_credentials re-fetch branch only runs with exchange disabled), so a server that only accepts client_credentials leaves Iceberg clients unable to ever refresh their token, regardless of 401 correctness. Accept grant_type=urn:ietf:params:oauth:grant-type:token-exchange on POST /v1/oauth/tokens: verify the subject_token signature against the issuing credential, allow exchange within a recovery grace window (max(2*TTL, 1h), capped 24h) so clients holding tokens that expired while the grant was unsupported recover without a restart, and mint a fresh access token with the configured TTL. * iceberg: harden OAuth token exchange and Bearer matching per review - match the Bearer scheme case-insensitively (RFC 7235), like authenticateBearer already does - accept optional client authentication on the token-exchange grant (Basic or form credentials, bound to the subject token's client); expired subject tokens now require it. Iceberg Java's proactive refresh sends Bearer-only headers, so the grant cannot require it - reject subject tokens without an exp claim, and re-check the issuer on the verified claims - unauthenticated exchange cannot extend the lifetime past the subject token's own expiry (no chain-refresh from a leaked token) - return 400 invalid_grant per RFC 6749 §5.2 (was 401) - include issued_token_type on exchange responses (RFC 8693) - clamp ICEBERG_OAUTH_TOKEN_EXPIRY to 365d so Duration math cannot overflow into already-expired tokens * iceberg: give authenticated token exchanges a fresh full TTL The remaining-lifetime cap only guards unauthenticated (Bearer-only) exchanges; an authenticated client renewing a live token must get the full configured TTL, matching client_credentials. * iceberg: reject token exchange when no lifetime remains A Bearer-only exchange with under a second of subject lifetime would mint a token with expires_in: 0. Reject with invalid_grant instead. * iceberg: pin near-expiry test token to the next second boundary jwt/v5 serializes exp at one-second precision, so a 300 ms offset can round into the current second and route the test through the expired branch instead of the ttlSeconds<=0 guard. Mint the subject with the next whole-second expiry: live at exchange time, deterministically under a second of remaining lifetime. * iceberg: drop internal ticket reference from comments * iceberg: clamp oversized OAuth TTLs on 32-bit platforms strconv.Atoi on an int-sized value fails with ErrRange on 386, so an oversized ICEBERG_OAUTH_TOKEN_EXPIRY silently fell back to the default instead of clamping. Parse in 64-bit space and clamp, then narrow. * iceberg: make OAuth TTL narrowing explicit * iceberg: disable legacy OAuth in PyIceberg integration tests |
||
|
|
4b41329e12 |
rust volume: stop a VolumeCopy whose caller has gone (#11188)
* rust volume: stop a VolumeCopy whose caller has gone VolumeCopy runs its copy in a detached tokio::spawn and reports progress with the send error discarded, so nothing observes the client leaving. When the caller cancels the RPC -- which weed-admin's batch balance does routinely, starting far more copies than it finishes -- the server streamed the whole volume from the source, wrote it to disk, and mounted it. The destination is then left holding a volume nobody took delivery of: its index cache is never reclaimed, and under replication=000 one volume id ends up on two servers, both writable, which concurrent writes can diverge. Three checks now reach the task: - Every chunk in copy_file_from_source, via the sender's is_closed(). This is the one that matters in practice. The first progress report is 128MB in, so for a smaller volume -- the ordinary balance move -- no send ever happens and its result says nothing; only the closed channel does. The sender is passed for the .idx and .vif copies too, with reporting gated separately, so those phases notice as well. - The throttle sleep, which for a throttled copy runs for seconds at a time, now races the sender's closed() instead of being slept through. - Immediately before mount_volume, and once at the top of the task. Cancellation surfaces as an ordinary Err(Status::cancelled), so it lands in the existing error branch that already removes the partial .dat/.idx/ .vif and the .note. That branch also logs now: the error otherwise went to a channel nobody was reading, leaving the operator with the balancer's "delete that copy, then re-run the move" and no cause. This also clears the stranded read-only sources reported on the issue. They are downstream of the orphan mount, not a separate defect: LiveMoveVolume's cleanup probes the target before undoing the freeze (volume_move.go:95, "the server can finish the copy and mount the target even when the client loses the stream"), and when it finds a mounted copy it cannot attribute, or cannot delete, it deliberately keeps the source readonly rather than risk two writable replicas -- the messages at volume_move.go:110 and :123. With nothing mounted on the target the probe reports clean and the freeze is undone. On Go parity: the progress send result is honoured here too, matching `return false` in volume_grpc_copy.go. But that report is Go's only abort signal, and measured against a 120MiB volume -- above the throttler's activation threshold, below the 128MiB report interval -- a Go destination mounts an abandoned copy as well. The issue's premise that Go aborts holds only above the report interval. The Rust side now stops in both cases; the Go behaviour is worth its own issue. Tests: the integration test runs against both implementations and is green on Go, red on Rust before this change. Its 192MiB fixture is sized for two separate constraints, documented at the fixture: IoBytePerSecond is a no-op below ~100ms of wall clock (64MiB copies in ~110ms on a tmpfs loopback cluster), and the payload must exceed the 128MiB report interval for the Go leg to pass at all. The two Rust unit tests cover what the integration test cannot reach: cancellation detected with no progress report at all, and the cleanup of the partial files plus the .note. Fixes #11186 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122W3eqt6gmLUMxmRoZdPAb * rust volume: surface VolumeCopy cancellation as Status::cancelled copy_file_from_source returned Result<_, String>, so the per-chunk cancellation path -- the one that matters in practice for volumes below the 128MB report interval -- was wrapped to Status::internal at the call sites. The spawn logging branch then classified it as a generic failure instead of the intended "abandoned by caller", defeating the logging change in the same PR for the case that occurs most often. Return Result<_, Status> from copy_file_from_source: Status::cancelled for caller-gone, Status::internal for the existing errors. Drop the .map_err(|e| Status::internal(e)) at the three call sites. The unit test now asserts the code is Cancelled, not just the message text, so the classification is locked in. * rust volume: close cancellation gaps in VolumeCopy Address two review findings on the same PR: 1. Roll back a mount that races a departing caller. The pre-mount is_closed() check cannot close the window between the check and mount_volume: if the receiver drops in that gap, the volume mounts and the final tx.send(Ok(...)) fails, but its error was discarded (let _ =), so the task returned Ok(()) and the error branch never ran. The destination then held an orphaned mounted replica — the exact defect this PR prevents. Fix: track a mounted flag. The final send now checks its result; on failure it returns Status::cancelled, and the error branch calls store.delete_volume (which unmounts AND removes the files) when mounted is true, instead of only unlinking. 2. Observe cancellation while awaiting the source stream. The per-chunk is_closed() check only runs after stream.message().await returns. A stalled source (slow disk, partition, GC pause) never delivers a chunk, so a caller that has already left cannot preempt the read: the task, the source connection, and the partial files (including the .note) all outlive the caller indefinitely. Fix: race stream.message() against progress_tx.closed() in a tokio::select!, so a departing caller preempts a stalled source. Adds test_volume_copy_after_mount_cancellation_rolls_back_mount to cover the after-mount rollback path. cargo test --release green (497 + 5 + 1 + 28). * rust volume: keep remote data on after-mount rollback, race RPC startup Two review findings on the after-mount rollback added in |
||
|
Chris Luanddevin-ai-integration[bot]
|
811b8b5734 |
make the remote-mount cache wait configurable per mount (#11168)
* add a per-mount cache_wait_ms to the remote storage mount mapping A read of an uncached remote-only object waits on a hardcoded size tier before it can fall back to the origin, so every ranged read of a large remote-only object pays that wait. Carry the wait in the mount mapping so it can be tuned, or set to zero, per mount. * resolve the cache wait of an uncached remote-only read from its mount The wait came only from the object size, so an operator could not trade cache hits for time to first byte. Both read paths now resolve the mount covering the object and let its cache_wait_ms replace the size tiers. * read straight from the remote when a mount waits zero for its cache A mount used as a streaming source pays the cache wait on every ranged read of an object too large to finish caching, and the caching itself is wasted work. A zero wait now skips the cache call, so both read paths go to the origin immediately. * let remote.mount set the cache wait of a mount remote.mount -cacheWait=0 turns a mount into a streaming source, and any other duration trades cache hits against time to first byte. * keep the size based wait for a version-specific read A read pinned to a version cannot fall back to the origin, since the mounted remote only holds the current key, so a mount that opts out of caching would leave it on the 503 retry loop forever. * let the operator allow a remote-only read to dial an internal endpoint The remote-mount read paths in the filer and the S3 gateway always refused an endpoint resolving to a loopback or private host, so a mount backed by an internal S3 could never be read from its origin, only through the local cache. Both now take the allowance the volume server already has, still off by default. * skip the background cache of a mount that waits zero for its cache GetObjectHandler kicks off caching for every remote-only read, so a mount serving as a streaming source kept downloading whole objects even though no read ever waited for them. * cover a zero cache wait end to end The read has to reach a real origin, so the harness also opts the filer and the S3 gateway into dialing the loopback remote it already allows for the volume server. * resolve the S3 cache wait once so the background cache follows it too The background cache that GetObjectHandler starts read the mount on its own, so it skipped a version-specific read that the foreground path still waits for. Both now ask the same resolver. * answer 404 when the origin of a zero-wait read is gone Metadata can outlive the object it points at, and with no cache to fill the read would sit on the 503 retry path forever. The remote backends already report a missing object as ErrRemoteObjectNotFound. * open the origin at write time for a multipart range Every part of a multipart Range is prepared before any is written, so opening eagerly would hold one origin connection per part and leak the ones already opened when a later part fails to open. * reject a cache wait shorter than a millisecond The mapping stores milliseconds, so -cacheWait=500us truncated to zero and silently turned caching off instead of waiting. * restore the doc comment of cacheRemoteObjectForStreamingWithShortTimeout Extracting the wait resolver left its comment on the new function. * stat the origin before committing a multipart range Opening at write time keeps no connection through the preparation, but it also moved a failure past the point where the multipart body picks the response status, so a gone origin truncated a 206 instead of answering 404. One stat up front puts the status back. * stat the origin once per request Every part of a multipart Range is prepared on its own, so the preflight ran once per range instead of once per read. * map Azure and GCS stream not-found to ErrRemoteObjectNotFound ReadFileAsStream on Azure and GCS returned provider-specific not-found errors instead of ErrRemoteObjectNotFound, so a zero-wait read of a deleted object was misclassified as a transient cache failure and retried indefinitely. Map BlobNotFound and ErrObjectNotExist the same way StatFile already does. * Update weed/remote_storage/gcs/gcs_storage_client.go Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
24b8646ec3 |
volume: let evacuation proceed on a server in maintenance mode (#11145)
Maintenance mode exists to fence a volume server so it can be evacuated without taking new writes (#7977), but the gate added in #8115 also rejected the RPCs evacuation issues against the source: VolumeMarkReadonly (the first step of every move, and the failure reported in #11066), VolumeDelete (the last step), and VolumeEcShardsDelete (the last step for EC shards). volumeServer.evacuate, volume.move and ec.balance therefore all failed on exactly the server they were meant to drain. Those three RPCs only remove data or restrict the server further, the same class as DeleteCollection and the unmount RPCs that were never gated, so they are exempted from the maintenance check in both the Go and Rust volume servers. Everything that adds data or reopens the server for writes (AllocateVolume, WriteNeedleBlob, BatchDelete, VolumeCopy, ReceiveFile, EC generate/copy/rebuild, vacuum, tiering, VolumeMarkWritable) stays blocked. A side effect is that scrub can now fence broken volumes readonly on a server already in maintenance. Fixes #11066 Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
52db7a6aee |
test: write more files than the master grows volumes at a time (#11131)
The cached-location test needs two files on one volume, but it wrote six files into the six volumes a 001 layout starts with, and every so often each file landed on its own volume and the test had nothing to probe. Seven files leave no way to spread them out. Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs |
||
|
Chris Lu
|
eed5e8cdf6 |
s3: return the multipart object checksum in the CompleteMultipartUpload response (#11101)
* s3: return the multipart object checksum in the CompleteMultipartUpload body S3 carries the flexible-checksum members of CompleteMultipartUploadResult in the XML body, not in response headers, so every SDK read back an empty checksum from an upload that asked for one. Claude-Session: https://claude.ai/code/session_01Huux1uh7JxAbf8yypMYrMk * s3: echo the checksum algorithm and type from CreateMultipartUpload The upload directory already records both, but the response dropped them, so a client could not confirm which checksum its parts had to carry. Claude-Session: https://claude.ai/code/session_01Huux1uh7JxAbf8yypMYrMk * test: multipart upload reports the object checksum it was asked for Covers every algorithm end to end: the create response echoes the algorithm and type, the complete response carries the checksum, and it matches what a later HEAD reports. Claude-Session: https://claude.ai/code/session_01Huux1uh7JxAbf8yypMYrMk |
||
|
dependabot[bot]
|
1c57734e58 |
build(deps): bump google.golang.org/grpc from 1.81.1 to 1.83.1 in /test/kafka (#11078)
build(deps): bump google.golang.org/grpc in /test/kafka Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.81.1 to 1.83.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](https://github.com/grpc/grpc-go/compare/v1.81.1...v1.83.1) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.83.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
40b3d32fe5 |
test: metadata operations on unlinked open files and removed open directories (#11075)
The POSIX suites skirt this: pjdfstest's unlink/14.t covers only fstat and pread on an unlinked descriptor — its driver has no fchmod at all and never opens a directory. Pin the full rule in the FUSE integration suite: ftruncate, fchmod, futimes, fstat, and the f*xattr calls keep working between the removal of the last name and the final close, for a file after unlink and a directory after rmdir, with nlink 0 and the changes visible to a following fstat. Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK |