test-with-server depended on start-server as a prerequisite, so a failed
startup skipped the recipe and the spawned weed mini kept the port.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A non-empty check note now fails the case, so a deleted bucket or an
object created by a malformed request cannot slip through behind a
passing status check.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Cleanup now runs its IAM/multipart steps only when setup reached them, so
an early setup failure still removes the bucket. Added a DELETE
bucket-subresource case asserting NotImplemented and bucket survival.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
PutBucketHandler gained the same guard when the route-level check moved
into the handlers; DeleteBucketHandler was missed, so an authorized
DELETE /bucket?logging could delete the bucket instead of answering
NotImplemented.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A failed setup previously skipped cleanup, leaking the bucket and IAM
user on persistent servers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A slow-starting server otherwise survives the failure path and keeps
the S3 port occupied for the next run.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A header value that decodes to neither 32-byte hex nor base64 is a
malformed argument, not a hash mismatch.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Deleting a user with live credentials orphaned its access keys; AWS
answers DeleteConflict until they are removed first.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
LimitExceeded is a conflict (409), an unknown Action is InvalidAction
(404) rather than NotImplemented, and internal failures report the IAM
receiver fault type. Applies to both the embedded IAM endpoint and the
standalone iamapi server.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
partNumber=abc, 0 and >10000 all resolve to InvalidArgument, matching
AWS, instead of InvalidPart.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
GET, HEAD and DELETE carrying a non-empty versionId on an unversioned
bucket now fail with InvalidArgument instead of being answered as a
plain object request.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
ErrTooManyRequest and ErrRequestBytesExceed reported made-up codes;
AWS serves these throttling rejections as SlowDown.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Embedded IAM and STS routes now report authentication, form-parse and
authorization failures with the IAM ErrorResponse body instead of the
S3 Error envelope, so IAM SDK clients can parse them. Requests signed
for s3 keep the S3 envelope, keyed off the credential scope.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
AWS answers an Authorization header missing Credential= with
InvalidArgument and one missing or malformed Signature= with
AuthorizationHeaderMalformed, instead of a generic MissingFields.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A PUT carrying a hex or base64 payload hash now streams through a
verifier that reports a mismatch once the stream is exhausted, instead
of storing an object that does not match its declared hash. The error
is deferred so intermediate reads that drop (n>0, err) results cannot
silently swallow it. Sentinel values (unsigned/streaming payloads)
remain exempt and malformed values fail fast.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A malformed X-Amz-Copy-Source or a non-numeric partNumber fell through
to the plain PutObject route and stored the body as a regular object.
Answer them with InvalidArgument-class errors instead of writing data.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A PUT on a bucket carrying an unrecognized query (logging, metrics,
intelligent-tiering, ...) fell through to the bare CreateBucket route
and returned BucketAlreadyOwnedByYou or re-created the bucket. AWS
answers these with NotImplemented.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cosi): add get/list/watch on bucketclasses to enable static provisioning
* fix(cosi): bind provisioner deployment to the chart-created service account
The deployment referenced a release-prefixed service account name while the
chart creates and binds "seaweedfs-objectstorage-provisioner", so the RBAC
grants never reached the provisioner pod.
---------
Co-authored-by: Jonas Onshuus <jons@dips.no>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3api: extract quoteETag from setEtag
Consolidate ETag quoting so the XML response builders can share it.
* s3api: quote ETag in CopyObjectResult XML
AWS returns the ETag quoted in the copy result body, matching the ETag header. Fixesseaweedfs/seaweedfs#11622
* s3api: quote ETag in CopyPartResult XML
UploadPartCopy returned the raw ETag in the XML body while the response header and other APIs return it quoted. Fixesseaweedfs/seaweedfs#11622
* s3api: test ETag quoting in copy responses
* s3api: assert quoted ETag on the wire in copy response tests
* s3api: use strconv.Quote in quoteETag
Addresses CodeQL 'potentially unsafe quoting' on string concatenation.
* volume: return error instead of panicking when .dat open fails
When backend.OpenVolumeFile returns an error (e.g. disk below
-minFreeSpace), dataFile is nil. Calling backend.NewDiskFile(nil)
immediately after caused a nil-pointer panic in f.Stat()/f.Name().
Move the existing error check to run right after OpenVolumeFile, before
NewDiskFile is called, so the error is returned cleanly.
Fixesseaweedfs/seaweedfs#11615
* volume: share .dat load error handling
Extract datFileLoadError helper so open and create paths share one check.
* volume: trim dat-open-fail test comments
* volume(rust): cover unopenable .dat load path
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* build(deps): bump github.com/apache/iceberg-go
Bumps [github.com/apache/iceberg-go](https://github.com/apache/iceberg-go) from 0.6.1-0.20260817192109-c2105090c9e2 to 0.7.0.
- [Release notes](https://github.com/apache/iceberg-go/releases)
- [Commits](https://github.com/apache/iceberg-go/commits/v0.7.0)
---
updated-dependencies:
- dependency-name: github.com/apache/iceberg-go
dependency-version: 0.7.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* iceberg: write delete manifests with NewManifestWriter
iceberg-go 0.7.0 validates that delete entries only land in
delete-content manifests, so the WriteManifest + byte-level content
patch workaround no longer works: WriteManifest always creates a
data-content writer and rejects delete entries outright.
Use NewManifestWriter with WithManifestWriterContent instead and
dispatch entries through Add/Existing/Delete by status. The returned
ManifestFile now carries writer-computed counts, partitions and
min-sequence-number, so the manual ManifestFile rebuild is dropped.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* helm: drop fromToml dependency in security-configmap.yaml (fixes#11611)
fromToml was added to Helm in v3.17.0 (helm/helm#12026, merged
2024-09-12, one day after v3.16.0 was cut). This chart declares no
minimum Helm version (no Chart.yaml kubeVersion, nothing in the
README), and the call in security-configmap.yaml:21 is an
unconditional *parse*-time failure on Helm < v3.17.0 - Go's
text/template parses a file's entire body before evaluating any
{{if}}, so this breaks the chart (any topology, any values) even when
securityConfigEnabled is false and the ConfigMap would render nothing.
Replaces the fromToml-based dig lookup with a small regex-based helper
(seaweedfs.existingTomlKey) that reads the same four "key = ..." JWT
signing-key values out of a previously-rendered security.toml,
preserving the existing fallback-to-random behavior exactly.
Verified:
- helm lint (v3.16.3 and v4.3.0): clean
- helm template with chart defaults: byte-identical output to the
unpatched chart rendered via Helm v4 (which has fromToml) - the
disabled/default path is untouched
- helm template with security enabled, no prior ConfigMap: identical
structure to the unpatched chart (helm v4), modulo the expected
random key
- Real helm install + helm upgrade round trip (live lookup, since
"helm template" never evaluates lookup, even under the original
fromToml code): the JWT signing key is identical across both
releases - confirms key persistence across upgrades is preserved,
not just "renders without erroring"
- helm template with chart defaults, Helm v3.16.3: previously failed
with a parse error naming fromToml as undefined; now renders
successfully
Fixes#11611.
* helm: harden existingTomlKey against commented key lines and CRLF
Addresses two review findings from greptile-apps on PR #11614:
- The key-line regex matched the first "key = ..." anywhere in the
section block, including a commented-out "# key = ..." line, which
would shadow a real active key on a hand-edited or otherwise
non-chart-generated ConfigMap. Anchored to line start with the
Go regexp multiline flag ((?m)^key...), which a line starting with
"#" cannot match.
- The section-header match required an exact "]\n", so a ConfigMap
with CRLF line endings would fail to match the block at all and
regenerate the key instead of reusing it. Changed to "]\r?\n".
Also adds a CI test ("Verify JWT signing key persistence across
upgrades") exercising all of this end to end with a real
helm install -> edit the live ConfigMap -> helm upgrade cycle,
matching the existing "Verify SFTP host key secret lifecycle" test's
shape: both edge cases are reproduced against a real ConfigMap and
asserted on the post-upgrade rendered security.toml.
Verified locally (same commands as the new CI step) against a real
cluster before pushing.
* helm: preserve JWT keys across supported TOML layouts
* ci: use setup-python interpreter for JWT upgrade checks
* helm: preserve keys under quoted TOML section headers
* helm: ignore unrelated quoted TOML section headers
The standalone s3 Deployment hardcoded terminationGracePeriodSeconds and
offered no way to set a container lifecycle or the Deployment strategy, so
operators could not add a preStop delay to drain endpoints before SIGTERM
or hold maxUnavailable at 0 during rollouts.
Add s3.updateStrategy, s3.lifecycle and s3.terminationGracePeriodSeconds.
Defaults render the same manifest as before.
Standalone weed s3 only shut its servers down when shutdownCtx was set,
which only weed mini does. On SIGTERM the interrupt hooks ran and the
process exited with requests still in flight, so clients saw connection
resets during rolling restarts.
Register an interrupt hook that drains every S3 HTTP(S) listener
(including the local and unix socket ones, and the Iceberg and Lance
servers) for up to 15s alongside a bounded gRPC GracefulStop, then closes
the S3 API server, reusing the filer's shutdown helper. Serve exits now
join that shutdown so the process does not exit mid-drain, and the
secondary listeners tolerate http.ErrServerClosed instead of exiting
fatally.
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: read GET/HEAD needles off the store lock, and only once
The GET/HEAD handler read the needle synchronously on the tokio worker
while holding store.read(): first a stream-info read that loaded the
whole record just to parse its meta, then, for every needle that was not
streamed (small, compressed, chunk manifest, image ops), a second full
read. For a tiered volume each read is an S3 GET under the store lock,
and a writer queued behind it parks every other store reader.
The regular-volume read now runs in spawn_blocking. Under the store guard
it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat
handle or the remote backend, offset, size); the guard is dropped before
any needle data I/O. No data-file lease is held across the read either,
since a writer waits for one while holding the store write lock. The
index size decides the read, as in Go's readNeedle: a HEAD, a ranged read
or a needle above the stream threshold reads only its header and meta
tail (ReadNeedleMeta) and hands off to StreamingBody or the range path;
everything else is read in full once, with its checksum verified. A
compressed or manifest needle found by the meta read is then read in
full once. The range-from-source read also moves to spawn_blocking.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stream needle chunks without the store lock
StreamingBody::poll_frame took store.read() and find_volume for every
chunk to compare the volume's compaction revision, dup'd the source
handle, and allocated a fresh chunk buffer. With -hasSlowRead=false the
stream also holds a data-file read lease for its whole life, while a
writer waits for that lease under store.write(): the next chunk's
store.read() then waits for the writer and the writer for the stream.
The per-chunk re-lookup was also wrong. The stream reads a handle opened
at plan time, which pins the .dat inode the offset was resolved against;
a vacuum commit renames a new file over .dat and leaves that inode
untouched. The re-looked-up offset belongs to the new file but was read
from the old inode, so a stream whose needle a vacuum moved ended in a
checksum error. The pinned offset stays valid, so the check, and with
it every store access, is dropped, along with the now unused
re_lookup_needle_data_offset and the revision fields of the read plan.
The source is shared as an Arc instead of dup'd per chunk, and the chunk
buffer is a BytesMut that the blocking read hands back with its result,
so its allocation is reclaimed once the previous frame has been written.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: split get_or_head_handler_inner into phases
get_or_head_handler_inner was a ~650-line function. Its middle resolved
the needle and set five mutable flags (stream_info, can_stream,
can_handle_head_from_meta, can_handle_range_from_source, bypass_cm) that
three if-let reply paths then re-tested, each re-checking stream_info.
It is now a 126-line orchestrator over named phases: reject_read_jwt,
proxy_missing_volume, wait_for_download_slot, parse_read_request,
read_ec_needle / read_volume_needle, etag_and_last_modified,
not_modified_response, read_response_headers, and the reply phases
stream_response, head_from_meta_response, range_from_source_response,
buffered_payload and buffered_response. The read phases return a
ReadPlan whose ReadStrategy enum (Stream, HeadFromMeta, RangeFromSource,
Buffered) carries the NeedleStreamInfo only on the variants that use it,
so the reply is one match instead of three flag checks.
Pure refactor: every status code, header and header order, error text,
metric increment, lock and data-file lease scope, spawn_blocking
boundary and side-effect order is unchanged. Phases that can end the
request return ControlFlow<Response, T>. A Range header that is not
visible ASCII still falls through to the buffered path, as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stop a needle stream once its volume becomes unavailable
Taking the store lock out of StreamingBody also dropped its per-chunk
unavailable_error() check. With -hasSlowRead a writer can take the
data-file lease between chunks, fail its fsync and its truncate, and mark
the volume unavailable; the stream then kept serving the rest of the
needle from its pinned handle.
The volume's io_unavailable reason is now an Arc-shared leaf mutex that
the read plan hands to the stream. Each chunk checks it under its
data-file lease, where the writer marks it, and fails with the same
"volume is unavailable: <reason>" error the old check returned.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: read a non-ASCII or empty Range header as Go does
A Range value with a byte >= 0x80 (obs-text, which hyper accepts) failed
HeaderValue::to_str at both range gates. For a needle served as stored the
handler had already chosen a meta-only read, so it fell through to the
buffered path with no payload and answered 200 with an empty body; a
compressed or EC needle answered 200 with the full body. Go's parseRange
fails on a byte it can neither trim nor parse and answers 416 "invalid
range", and trims Unicode whitespace such as NBSP into a normal 206. An
empty Range value was also a 200 with an empty body, where Go sends the
whole payload.
Read Range once with from_utf8_lossy, dropping an empty value, and hand
that one value to the read plan and to both range gates. A replaced byte
never parses, so it is a 416; str::trim trims the same Unicode whitespace
as strings.TrimSpace. A range read from the data file now always answers
itself instead of falling through with an empty needle. The buffered path
answers HEAD before it looks at Range, as Go's writeResponseContent does,
so an EC HEAD with a Range is a 200 with the full length.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: apply Range to chunk manifests and forward raw headers when proxying
A GET of a chunk manifest assembled the object and always answered 200 with
the whole body, ignoring Range. Go serves the expanded manifest through
writeResponseContent, which answers HEAD first and then hands Range to
ProcessRangeRequest: 206 for one range, multipart/byteranges for several,
416 for an unsatisfiable or unparsable one. try_expand_chunk_manifest now
returns the assembled body and headers, and the caller answers through
buffered_response, the same helper the buffered needle path uses.
A proxied read forwarded a request header only if HeaderValue::to_str
succeeded, so a Range with an obs-text byte was dropped and the target
answered 200 with the full body. Go copies every header value as is.
Forward the raw HeaderValue for every header.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: reset Content-Type on range errors as net/http does
Go's http.Error sets text/plain and nosniff unconditionally; keeping the
needle's MIME type on a 416 mislabels the error body. Match it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: count manifest downloads toward the limit
Expanded manifests hard-coded track_download off, so chunked-file reads
bypassed in-flight byte accounting while every other path honored the
download limiter.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A draining filer stayed in the master's lock ring until its process
exited, while gRPC GracefulStop was already refusing new connections. S3
gateways and peer filers kept routing object-write locks and owner-routed
writes to it for the whole graceful-stop window.
On shutdown the filer now sends leave_lock_ring on its open KeepConnected
stream. The master removes it from the lock ring only; it stays a cluster
member so peers keep following its metadata log through the drain. Once
the ring update without it arrives, the filer has already transferred its
locks to the new owners, and it keeps serving through the prior-owner
window (plus a second for peers that apply the update later) before gRPC
and HTTP begin draining. The whole leave is bounded at 10s so slow lock
transfers or a stuck stream cannot hold up the drain. A lone filer, or a
master that ignores the message, falls back to the previous behavior.
* fix(s3): honor object ACLs for anonymous reads
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
* s3: drop unsigned session tokens on deferred anonymous reads
An unsigned GET/HEAD carrying X-Amz-Security-Token is anonymous, not a
session request; strip the token before authentication and authorization
so it cannot influence identity or policy evaluation.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: bound metadata log flush retries during shutdown
On SIGTERM the filer could hang forever in Shutdown: the final
LocalMetaLogBuffer flush retries appendToFile indefinitely, and with
the master already down each AssignVolume attempt just kept failing.
WaitForShutdown never returned, the interrupt hook never reached
os.Exit, and the half-dead filer kept its ports bound.
Thread a context through appendToFile/assignAndUpload and switch to
a 15s-bounded context once the filer is stopping: the flush abandons
with a log line instead of retrying forever. Normal operation keeps
the unbounded retry so no metadata is dropped while running.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: share one shutdown deadline across all pending meta log flushes
Review feedback on the per-flush timeout: a deadline armed at flush start
could already be expired when shutdown arrived, the first append of a flush
still ran unbounded, each queued window got a fresh budget (16 windows *
15s), and an abandoned window still advanced the flushed watermark as if it
had landed.
Rework to a single shared flush context on the Filer, cancelled once by
Shutdown via AfterFunc. Every append - the in-flight one and every queued
window - observes the same deadline, so the whole drain is bounded at 15s.
A flush that gives up reports its dropped bytes through the new
LogBuffer.NoteFlushDropped, and loopFlush then skips the offset/timestamp
advance and subscriber notifications for that window.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: bound append attempts and commit uploaded pieces detached
Store operations check then drop request cancellation, so a stalled
backend could still hold flushFn past the shutdown deadline; run each
append attempt on its own goroutine and give up on it at the deadline.
Once a piece is uploaded, commit its entry on a detached context so the
expired deadline cannot strand the chunk.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: keep shutdown flushes synchronous
Detaching the append attempt let flushFn return while the goroutine
still held the pooled flush buffer and could commit after the metadata
store closed; abandonment is only safe for the cancelable assign/upload
phase, which the shared flush context already bounds.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: read GET/HEAD needles off the store lock, and only once
The GET/HEAD handler read the needle synchronously on the tokio worker
while holding store.read(): first a stream-info read that loaded the
whole record just to parse its meta, then, for every needle that was not
streamed (small, compressed, chunk manifest, image ops), a second full
read. For a tiered volume each read is an S3 GET under the store lock,
and a writer queued behind it parks every other store reader.
The regular-volume read now runs in spawn_blocking. Under the store guard
it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat
handle or the remote backend, offset, size); the guard is dropped before
any needle data I/O. No data-file lease is held across the read either,
since a writer waits for one while holding the store write lock. The
index size decides the read, as in Go's readNeedle: a HEAD, a ranged read
or a needle above the stream threshold reads only its header and meta
tail (ReadNeedleMeta) and hands off to StreamingBody or the range path;
everything else is read in full once, with its checksum verified. A
compressed or manifest needle found by the meta read is then read in
full once. The range-from-source read also moves to spawn_blocking.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stream needle chunks without the store lock
StreamingBody::poll_frame took store.read() and find_volume for every
chunk to compare the volume's compaction revision, dup'd the source
handle, and allocated a fresh chunk buffer. With -hasSlowRead=false the
stream also holds a data-file read lease for its whole life, while a
writer waits for that lease under store.write(): the next chunk's
store.read() then waits for the writer and the writer for the stream.
The per-chunk re-lookup was also wrong. The stream reads a handle opened
at plan time, which pins the .dat inode the offset was resolved against;
a vacuum commit renames a new file over .dat and leaves that inode
untouched. The re-looked-up offset belongs to the new file but was read
from the old inode, so a stream whose needle a vacuum moved ended in a
checksum error. The pinned offset stays valid, so the check, and with
it every store access, is dropped, along with the now unused
re_lookup_needle_data_offset and the revision fields of the read plan.
The source is shared as an Arc instead of dup'd per chunk, and the chunk
buffer is a BytesMut that the blocking read hands back with its result,
so its allocation is reclaimed once the previous frame has been written.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: split get_or_head_handler_inner into phases
get_or_head_handler_inner was a ~650-line function. Its middle resolved
the needle and set five mutable flags (stream_info, can_stream,
can_handle_head_from_meta, can_handle_range_from_source, bypass_cm) that
three if-let reply paths then re-tested, each re-checking stream_info.
It is now a 126-line orchestrator over named phases: reject_read_jwt,
proxy_missing_volume, wait_for_download_slot, parse_read_request,
read_ec_needle / read_volume_needle, etag_and_last_modified,
not_modified_response, read_response_headers, and the reply phases
stream_response, head_from_meta_response, range_from_source_response,
buffered_payload and buffered_response. The read phases return a
ReadPlan whose ReadStrategy enum (Stream, HeadFromMeta, RangeFromSource,
Buffered) carries the NeedleStreamInfo only on the variants that use it,
so the reply is one match instead of three flag checks.
Pure refactor: every status code, header and header order, error text,
metric increment, lock and data-file lease scope, spawn_blocking
boundary and side-effect order is unchanged. Phases that can end the
request return ControlFlow<Response, T>. A Range header that is not
visible ASCII still falls through to the buffered path, as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stop a needle stream once its volume becomes unavailable
Taking the store lock out of StreamingBody also dropped its per-chunk
unavailable_error() check. With -hasSlowRead a writer can take the
data-file lease between chunks, fail its fsync and its truncate, and mark
the volume unavailable; the stream then kept serving the rest of the
needle from its pinned handle.
The volume's io_unavailable reason is now an Arc-shared leaf mutex that
the read plan hands to the stream. Each chunk checks it under its
data-file lease, where the writer marks it, and fails with the same
"volume is unavailable: <reason>" error the old check returned.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: read a non-ASCII or empty Range header as Go does
A Range value with a byte >= 0x80 (obs-text, which hyper accepts) failed
HeaderValue::to_str at both range gates. For a needle served as stored the
handler had already chosen a meta-only read, so it fell through to the
buffered path with no payload and answered 200 with an empty body; a
compressed or EC needle answered 200 with the full body. Go's parseRange
fails on a byte it can neither trim nor parse and answers 416 "invalid
range", and trims Unicode whitespace such as NBSP into a normal 206. An
empty Range value was also a 200 with an empty body, where Go sends the
whole payload.
Read Range once with from_utf8_lossy, dropping an empty value, and hand
that one value to the read plan and to both range gates. A replaced byte
never parses, so it is a 416; str::trim trims the same Unicode whitespace
as strings.TrimSpace. A range read from the data file now always answers
itself instead of falling through with an empty needle. The buffered path
answers HEAD before it looks at Range, as Go's writeResponseContent does,
so an EC HEAD with a Range is a 200 with the full length.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: apply Range to chunk manifests and forward raw headers when proxying
A GET of a chunk manifest assembled the object and always answered 200 with
the whole body, ignoring Range. Go serves the expanded manifest through
writeResponseContent, which answers HEAD first and then hands Range to
ProcessRangeRequest: 206 for one range, multipart/byteranges for several,
416 for an unsatisfiable or unparsable one. try_expand_chunk_manifest now
returns the assembled body and headers, and the caller answers through
buffered_response, the same helper the buffered needle path uses.
A proxied read forwarded a request header only if HeaderValue::to_str
succeeded, so a Range with an obs-text byte was dropped and the target
answered 200 with the full body. Go copies every header value as is.
Forward the raw HeaderValue for every header.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: fetch only the chunks a manifest Range needs, like Go
A ranged GET of a chunk manifest fetched every chunk, assembled the whole
object and then sliced it, so reading a few bytes of a large object cost a
read of all of it, and a 416 still fetched everything. Go serves a manifest
through ChunkedFileReader, which seeks to each range and reads only the
chunks under it.
For a GET with a Range, try_expand_chunk_manifest now parses the ranges
against the manifest size, fetches only the chunks whose declared window
overlaps one of them (none when the reply carries no body), and answers
through handle_range_request_with, the reader-based core that
handle_range_request now wraps, so 206/416/multipart stay one code path.
The reader replays assembly: chunks clamped as before, later chunks over
earlier ones, zeros in gaps. HEAD, no-Range GETs and GETs that crop or
resize an image still assemble the whole object.
A missing chunk outside the requested ranges no longer turns a ranged GET
into a 500, as in Go; a missing chunk inside them still does, before any
headers are sent.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: reword a comment codespell flags
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: keep only range-covered bytes of fetched manifest chunks
A ranged GET retained every overlapping chunk's full contents; 1,000
overlapping 8 MiB chunks could pin ~8 GiB for a one-byte response. Clip
each fetched chunk to the bytes the requested ranges can actually read,
preserving the later-chunks-overwrite and zero-fill-gap semantics.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: bucket ranged manifest parts by range
Serving a multipart range scanned every retained part. Bucket the kept
intersections by their range so one range only reads its own parts.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* kafka: honor notification.kafka.event_types
Kafka published every filer event and ignored the filter the webhook notifier already uses.
Co-authored-by: Cursor <cursoragent@cursor.com>
* notification: share event-type classification between queues
Kafka duplicated the webhook's event classification verbatim; move it to
the notification package so the two queues cannot drift. Webhook keeps
its typed eventType wrappers over the shared helpers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* image: add an optional public image processing gateway
* image: fix representation metadata and processing bounds
* image: restrict passthrough to non-executable media types
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* image: tighten source media-type validation
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* image: write passthrough body on the inner response writer
CodeQL still flagged the passthrough write: the content type was set on
the wrapper while the body reached w.ResponseWriter, so the validated
header could not be associated with the write. Set headers and copy the
body on the same inner writer.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* image: serve processed output on the inner response writer
* image: reject XML source types and unsafe conditional metadata
---------
Co-authored-by: zhaoyuchen <yc.zhao@yinzon.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: read GET/HEAD needles off the store lock, and only once
The GET/HEAD handler read the needle synchronously on the tokio worker
while holding store.read(): first a stream-info read that loaded the
whole record just to parse its meta, then, for every needle that was not
streamed (small, compressed, chunk manifest, image ops), a second full
read. For a tiered volume each read is an S3 GET under the store lock,
and a writer queued behind it parks every other store reader.
The regular-volume read now runs in spawn_blocking. Under the store guard
it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat
handle or the remote backend, offset, size); the guard is dropped before
any needle data I/O. No data-file lease is held across the read either,
since a writer waits for one while holding the store write lock. The
index size decides the read, as in Go's readNeedle: a HEAD, a ranged read
or a needle above the stream threshold reads only its header and meta
tail (ReadNeedleMeta) and hands off to StreamingBody or the range path;
everything else is read in full once, with its checksum verified. A
compressed or manifest needle found by the meta read is then read in
full once. The range-from-source read also moves to spawn_blocking.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stream needle chunks without the store lock
StreamingBody::poll_frame took store.read() and find_volume for every
chunk to compare the volume's compaction revision, dup'd the source
handle, and allocated a fresh chunk buffer. With -hasSlowRead=false the
stream also holds a data-file read lease for its whole life, while a
writer waits for that lease under store.write(): the next chunk's
store.read() then waits for the writer and the writer for the stream.
The per-chunk re-lookup was also wrong. The stream reads a handle opened
at plan time, which pins the .dat inode the offset was resolved against;
a vacuum commit renames a new file over .dat and leaves that inode
untouched. The re-looked-up offset belongs to the new file but was read
from the old inode, so a stream whose needle a vacuum moved ended in a
checksum error. The pinned offset stays valid, so the check, and with
it every store access, is dropped, along with the now unused
re_lookup_needle_data_offset and the revision fields of the read plan.
The source is shared as an Arc instead of dup'd per chunk, and the chunk
buffer is a BytesMut that the blocking read hands back with its result,
so its allocation is reclaimed once the previous frame has been written.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: split get_or_head_handler_inner into phases
get_or_head_handler_inner was a ~650-line function. Its middle resolved
the needle and set five mutable flags (stream_info, can_stream,
can_handle_head_from_meta, can_handle_range_from_source, bypass_cm) that
three if-let reply paths then re-tested, each re-checking stream_info.
It is now a 126-line orchestrator over named phases: reject_read_jwt,
proxy_missing_volume, wait_for_download_slot, parse_read_request,
read_ec_needle / read_volume_needle, etag_and_last_modified,
not_modified_response, read_response_headers, and the reply phases
stream_response, head_from_meta_response, range_from_source_response,
buffered_payload and buffered_response. The read phases return a
ReadPlan whose ReadStrategy enum (Stream, HeadFromMeta, RangeFromSource,
Buffered) carries the NeedleStreamInfo only on the variants that use it,
so the reply is one match instead of three flag checks.
Pure refactor: every status code, header and header order, error text,
metric increment, lock and data-file lease scope, spawn_blocking
boundary and side-effect order is unchanged. Phases that can end the
request return ControlFlow<Response, T>. A Range header that is not
visible ASCII still falls through to the buffered path, as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: stop a needle stream once its volume becomes unavailable
Taking the store lock out of StreamingBody also dropped its per-chunk
unavailable_error() check. With -hasSlowRead a writer can take the
data-file lease between chunks, fail its fsync and its truncate, and mark
the volume unavailable; the stream then kept serving the rest of the
needle from its pinned handle.
The volume's io_unavailable reason is now an Arc-shared leaf mutex that
the read plan hands to the stream. Each chunk checks it under its
data-file lease, where the writer marks it, and fails with the same
"volume is unavailable: <reason>" error the old check returned.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* volume server: read a non-ASCII or empty Range header as Go does
A Range value with a byte >= 0x80 (obs-text, which hyper accepts) failed
HeaderValue::to_str at both range gates. For a needle served as stored the
handler had already chosen a meta-only read, so it fell through to the
buffered path with no payload and answered 200 with an empty body; a
compressed or EC needle answered 200 with the full body. Go's parseRange
fails on a byte it can neither trim nor parse and answers 416 "invalid
range", and trims Unicode whitespace such as NBSP into a normal 206. An
empty Range value was also a 200 with an empty body, where Go sends the
whole payload.
Read Range once with from_utf8_lossy, dropping an empty value, and hand
that one value to the read plan and to both range gates. A replaced byte
never parses, so it is a 416; str::trim trims the same Unicode whitespace
as strings.TrimSpace. A range read from the data file now always answers
itself instead of falling through with an empty needle. The buffered path
answers HEAD before it looks at Range, as Go's writeResponseContent does,
so an EC HEAD with a Range is a 200 with the full length.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* fix(s3): initialize destination ACLs for CopyObject
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
* fix(s3): re-check routed self-copy eligibility on the locked read
routeInPlace was decided on the pre-lock entry, but the PATCH body
re-reads the entry. A concurrent write changing file mode or MIME in
between left the routed PATCH installing new ACL keys while Attributes
kept the stale mode. Evaluate eligibility against the re-read entry and
retry the self-copy under the distributed lock when it no longer
qualifies.
* fix(s3): guard metadata self-copies against concurrent writes
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
* ci: raise s3api unit-test timeout to 9m
The suite crossed the 5m binary timeout on the hosted runner (local run
is ~4.3m and still growing). The job-level limit is already 10m.
* ci: allow setup time before the S3 API test suite
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
---------
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* admin: delete persisted state when scan cancels pending tasks
Each detection cycle cancels every pending task of a type before
re-detecting it, and the cancel path saved the cancelled task back to
disk. Nothing ever removed those files, so -dataDir/tasks gained one
orphaned .pb per candidate volume per scan cycle.
Cancelled is terminal, so drop the file the same way CompleteTask does
for completed/failed tasks. The cancelled entry stays in memory for the
UI until the next purge.
Refs #11595
* admin: delete persisted state when CancelTask cancels a pending task
The manual cancel path only updated memory, leaving the pending .pb on
disk where a restart would resurrect the cancelled task as pending and
the file would linger until then. Delete it like the scan-cycle cancel
path now does.
* admin: count cancelled tasks toward task retention cleanup
CleanupOldTasks and ConfigPersistence.CleanupCompletedTasks only
filtered completed/failed tasks, so cancelled entries were exempt from
retention in both memory and on disk. Treat all terminal states alike;
nil CompletedAt entries also count and sort last, so they are pruned
first.
* admin: run task file retention in the periodic cleanup loop
cleanupCompletedTasks had no callers, so the on-disk retention bound
never ran during uptime. Invoke it from performCleanup alongside the
in-memory CleanupOldTasks sweep.
* admin: guard task state writes against stale saves and failed deletes
saveTaskState runs after mq.mutex is released, so the task may have gone
terminal in between; a delayed pending save could then recreate the file
a cancel just deleted and resurrect the task on restart. Skip saving
non-terminal snapshots once the live task is terminal or gone.
If a cancel file removal fails, fall back to writing the cancelled
snapshot so the file is terminal rather than pending. deleteTaskState now
returns its error, and CancelTask captures task.Status while still
holding the queue lock.
* admin: serialize task file check+write against cancel deletes
The saveTaskState guard still had a check-then-write window: a pending
snapshot could pass the terminal check before a cancel deleted the file,
then write it back after. A persistMu on the queue now covers the
check+save and the cancel paths' delete (with its terminal-state
fallback), so the two cannot interleave for the same task.
* s3api: thread filer client through the versioned-listing collector
findVersionsRecursively now binds one SeaweedFilerClient for the whole
recursive walk instead of re-resolving a filer on every list/lookup call,
and the collector's list/getEntry/scanLatestVersionEntry/getObjectVersionList
helpers go through it. No behavior change; this also lets tests drive
collectVersions with a stubbed client.
* s3api: keep collecting versions while pending names can sort into the page
ListObjectVersions walked the filer in directory-entry name order and
stopped as soon as maxKeys+1 items were collected, sorting only that
partial set. Filer names do not match key order: "a.copy.versions" sorts
before "a.versions" while key "a.copy" sorts after "a", so a page
boundary inside the earlier-walked sibling's versions permanently skipped
the later key.
Track the largest key collected (maxKey) and, once the collector is full,
keep walking until entry names pass the ceiling of names that can still
resolve to keys at or below it; the ceiling reaches through the prefix
versions of maxKey. Entries whose subtree can only hold keys above maxKey
are skipped. Versions of an in-bound object are collected in full so its
position in the sorted page is exact.
Fixes seaweedfs#11594
* s3api: resume versioned listings at the earliest covering name prefix
computeStartFrom mapped the key marker straight to an entry name (or cut
it at the first '/'), which skips sibling directories that are a prefix
of the marker below '0' - for marker "d.x" the listing resumed at name
"d.x", skipping directory "d" whose keys "d/*" all sort after it.
Resume at the earliest remainder prefix ending at a byte below '0' ('/',
'.', '-' and friends), so every directory whose subtree can still hold
keys past the marker is revisited; already-returned keys inside are
filtered by the existing marker checks as before.
* s3api: regression test for versioned-listing pagination order
Drive collectVersions against a stubbed filer holding the issue-11594
layout - "a.copy.versions" listing before "a.versions", plus a "d/"
subtree next to "d.x" - and assert that every page size from 1 up
reproduces the unpaginated ordering with no lost or duplicated entries.
Also updates TestComputeStartFrom for the new earliest-prefix resume and
gives testFilerClient a LookupDirectoryEntry stub.
* s3api: inject list/getEntry functions into the version collector
Pinning one SeaweedFilerClient for the whole walk dropped per-call
failover: previously each s3a.list resolved a filer through
WithFilerClient, so a mid-walk filer failure could fall back to a
healthy peer. Inject s3a.list/s3a.getEntry as function fields instead -
production keeps the failover behavior, tests can still stub.
* s3api: keep scanning marker for a later covering prefix
A leading byte below '0' (marker .hidden/file) has no non-empty prefix
at index 0, but a deeper separator still does - resuming at .hidden/file
skipped the .hidden directory and its keys after file. Continue the scan
instead of bailing on the first byte.
* s3api: persist ACLs on PutObject uploads
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
* s3api: fix PutObject ACL edge cases found in review
- Only enforce BucketOwnerEnforced when explicitly configured; buckets
without a stored ownership control keep accepting upload ACLs
- Ignore ACL query parameters on SigV2 requests, which do not sign them
- Mirror signed-query ACL values into headers after authentication so
grant parsing and resolveFileMode agree on presigned uploads
- Validate only caller-supplied grantees against the account registry;
default grants now work for accounts outside the local registry
- Reject unknown grantee keys and accept comma-separated grantee lists
without spaces in ParseCustomAclHeader
- Guard against identities without an account
* s3api: harden upload ACL parsing and authorization
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
* s3api: evaluate upload ACL grantees individually in policies
A comma-joined grant header or a signed query parameter reached policy
conditions as one value, so a deny on a later grantee did not fire. Split
grant headers into per-grantee values for policy evaluation and share the
grantee pair parser with ParseCustomAclHeader.
* s3api: keep raw grant header values visible to policy conditions
Exact-match conditions written against the signed header value stopped
matching once grantees were split for evaluation. Preserve the original
wire values alongside the per-grantee values so deny policies fire on
either granularity.
* s3api: evaluate upload ACL grants as one canonical list in policies
Conditions on s3:x-amz-grant-* now see a single comma-separated canonical
grant list identical for a single line, repeated header lines, or a signed
query parameter. This keeps StringEquals allows and exact-list or
allowlist (StringNotEquals) denies accurate regardless of wire encoding.
* s3api: preserve upload ACL denies and align policy checks
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
* s3api: retain upload owner grants and literal policy values
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
---------
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix 11586
* Update filer_server_handlers_write_autochunk.go
* filer: fix inline append races, empty files, and stale ETags
Serialize the append read-modify-write on the entry lock so concurrent
appends merge instead of losing content, keep small appends to empty
files inline, tolerate legacy entries whose metadata size differs from
their content, and set the entry digest so appended inline files keep a
real ETag.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
