A rename copied right away left its pending source to remote.sync,
which released it only while the destination was still remote-only.
Cached before the sync got there, the destination took the other path
and the source stayed on the remote for good. Release it there too,
with the same checks.
The filer deleted a pending entry's source after copying it on a later
rename, but that rename locks only its own paths: another rename can
copy onto the source key before its entry is visible, and the cleanup
then deleted that copy. And deleting a pending entry never released its
source at all.
The filer now leaves marked sources alone. remote.sync releases one
when the pending entry is deleted, or when a later rename was copied
right away, and only while the live object is still the one the
entry's stamp describes and no remote-only entry holds the path: a
copy that landed there since has a different ETag and is kept.
When a remote-only entry is renamed over another remote-only file and
the deferred copy fails, it reads the source key through its marker.
Renamed again the same way, the marker keeps naming that source, but:
- remote.sync completed each rename from the event's old key. For the
second one that is the first target's old object, which it copied
over the destination before clearing the marker. It now copies from
the source the entry's marker names.
- For the first rename it found no entry at the new path and deleted
the old key, which the later entry still reads. When the event's new
entry carried a marker and the path is gone, it now leaves the key to
the entry that moved on.
- When the filer copies a pending entry immediately on a later rename,
it now also releases the source the copy read, after the commit and
with the same checks as the old key.
Follow-ups to reading a pending remote rename copy from its source key:
- Only the filer sets the marker. The HTTP upload and tagging handlers
store Seaweed-* request headers as extended attributes, so they skip
this one, and a marker that leads out of the entry's own mount is
ignored, so it cannot point reads at another remote.
- A pending entry renamed again copies from the marker's source, not
from its own key, which still holds the old target's bytes. A further
deferred rename keeps naming the original source; an immediate copy
clears the marker.
- When remote.sync finds the destination already holding the content,
it stamps the entry (clearing the marker) before deleting the source,
and deletes the source only once the entry no longer reads it.
- A read that resolved the source just before the copy finished and
the source went away retries with the entry as it is now.
An object at a deleted key is not always the one the delete meant: a
rename copy can land before its filer entry is visible, and a rewrite
can upload over it. When the event recorded no remote stamp, or the
live stat cannot confirm the recorded etag, the delete now keeps the
object instead of treating it as a late upload.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A remote-only rename over a remote-only target queued the new entry's
event with the source's stamp before the deferred copy ran. Peers
replaying it saved a stamp the copy had already superseded, and the
sync serving such a peer rejected the destination as a match and tried
to copy the deleted source. The event now carries the copy's result
and drops the pending-source marker once it lands.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A remote-only rename over a remote-only target commits before its copy
lands, so reads on the new path served the target's old object until the
copy stamped the entry — and indefinitely when the copy failed without
remote.sync running. The committed entry now records the renamed-from
path, and remote-only reads resolve that object's key while the stamp
is pending; the copy's stamp and the sync's completion stamp both clear
the marker.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Deleting a changed object when no entry holds the path also deletes a
copy a directory rename has written but not yet made visible, while a
delete of that path waits in the sync; the rename then removes the old
key and the content is gone. Keep any object whose ETag changed since
the delete, as before. An upload that finishes after the delete leaks
its object instead, which is the cheaper failure.
- The deferred copy stamps the entry with a plain UpdateEntry, which
publishes no event, so the queued rename event still carried the old
key's stamp to peer filers and subscribers. Give it the copy's
RemoteEntry before it is sent. The two notify loops share the code now.
- remote.sync kept any object whose ETag changed since the delete. With
no entry at the path that is an upload which finished after the
delete, and keeping it leaves it on the remote for good. Look up the
path first and keep a changed object only when an entry holds it.
The SQL stores keep the transaction in the context and refuse every
call through it once it has committed. The work that runs after
CommitTransaction used that context, so on postgres2 the deferred
old-key delete could not look up the old path and kept the object, and
finishRemoteCopies could not read or stamp the entry. Both failed with
"sql: transaction has already been committed or rolled back" on every
copied rename; remote.sync cleaned up after them.
Keep the context from before the transaction for the work after the
commit. The rename test store gets a strict mode that fails calls
through a committed transaction the way database/sql does.
A rename can copy another entry's content onto a key between the filer's
check and the remote delete, and the delete then removes the copy. The
delete event carries the object's etag; delete only when the live object
still matches it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Copying the source object onto the target's key before the transaction
committed destroyed the target's only content, which a rolled-back
commit cannot restore. When the target is remote-only the copy now runs
after the commit: the new entry keeps the old key's stamp until the copy
lands and re-stamps it, the target's object is kept meanwhile, and a
failed copy keeps the old object so remote.sync can finish it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The deferred delete must only run once the lookup shows no remote-only
entry at the old path. A failed lookup shows nothing, and an extra
object loses nothing.
A directory rename does not lock its children's paths, so between the
commit and the deferred delete of a copied child's old key another
rename can put a remote-only entry there, whose content is then only in
that object. Check the store before deleting.
Removing the old directory on the remote deletes everything under its
prefix. Run after the commit, that could also take an object another
rename copied in under the same path meanwhile. When a child was copied,
leave the prefix alone and delete only the copied children's old keys.
Review follow-ups:
- The old key of a copied remote-only entry, and the old directory of a
rename that copied one, are now deleted once the rename has committed.
A rename that fails after the copy leaves the source entry and its
object in place, plus an extra object at the new key.
- Every part of a multipart copy is conditioned on the ETag the copy
started from, so an object overwritten mid-copy fails the copy
instead of mixing versions.
- Both copy paths keep the source's server-side encryption settings.
- filer.remote.sync no longer deletes a key that a remote-only entry
holds by the time the delete event is processed: a file deleted and
then replaced by a remote-only rename has its content only there.
Renaming a file under a remote.mount whose content exists only on the
remote (never cached, or remote.uncache'd) lost it: the new entry kept
the old RemoteEntry, deleting the old entry deleted the old object, and
with no chunks nothing ever wrote the new key. filer.remote.sync could
not finish the rename either, since the old object was already gone.
moveSelfEntry now copies such an entry's object to the new key before
it touches anything, and the new entry gets the RemoteEntry of the
copy. The copy is a new optional RemoteStorageObjectCopier interface,
implemented for S3 with CopyObject up to 5 GB and UploadPartCopy above
that; the multipart copy carries the source's metadata and, where the
remote supports tagging, its tags. When an existing target is
overwritten, its delete keeps the remote object (the copy replaced it)
and its event is marked metadata-only so the sync leaves it alone too.
A remote-only rename that leaves its mount, or whose remote cannot
copy, is refused instead of deleting the only copy. Empty entries and
entries with local data rename as before.
UpdateFileMetadata builds its copy source with the same helper, which
escapes each key segment and keeps the slashes.
* s3api: preserve remote object ETags across cache fills
* s3api: align remote copy validators and reject stale ETags
* s3api: keep remote ETag through metadata touches, drop it after resizes
The remote-ETag check required Mtime to match RemoteMtime, so a utimens
on a cold remote object returned an empty ETag until a later cache fill
restored the remote value. It also trusted LastLocalSyncTsNs alone, so a
setattr resize that leaves chunks and the stamp untouched kept serving
the old remote ETag for different local bytes.
Require FileSize to match RemoteSize for both the synced and remote-only
paths and stop comparing modification times: content changes already
surface through cleared sync stamps, chunks, or inline content, while a
metadata-only touch leaves the remote bytes authoritative. Regression
tests cover touched cold and cached entries and resized cached entries.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* fix(s3api): accept max-keys up to 2147483647 and cap list pages at 1000
* add tests for ListObjects max-keys range and 1000 cap
* refactor(s3api): return uint16 from parseMaxKeys to avoid narrowing casts
* s3api: register the azurekms KMS provider under its build tag
weed/kms/azure registers itself with weed/kms from an init(), but the
blank import that triggers it was commented out in auth_credentials.go
(TODO: Fix Azure SDK compatibility issues). Nothing else imports the
package, so building with -tags azurekms still ends with
"KMS provider 'azure' not registered" at startup for any config using
"type": "azure".
Move the import into weed/s3api/kms_azure.go behind //go:build azurekms so
default builds stay free of the Azure SDK, and drop the dead comment.
Closes#11686
* ci: compile and test azurekms-gated code
weed/kms/azure is excluded from every default build, so nothing in CI
compiled it; that is how the provider shipped unregistered. Build the
tree and run the kms tests with -tags azurekms on every Go change.
---------
Co-authored-by: Yi-111-a <>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* worker: reject balance moves onto servers without the volume's disk type
Proposals are bucketed by disk type at detection time, but queued or
out-of-band jobs can still target a server that lacks the volume's disk.
The copy then reads the source disk only to be rejected by the target's
VolumeCopy, and the failure is retried on every scheduling cycle.
Fetch the master's topology once per job and check each move's target
against the volume's disk type before executing, so an incompatible move
fails before any data is copied.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* worker: cover the balance disk-type guard
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* worker: match old-topology nodes in the balance disk-type guard
Topology entries from older masters omit Address and carry a plain
host:port Id, so checkMoveDiskType never matched grpc-suffixed move
nodes and silently skipped the disk-type check. Also compare each node
via pb.NewServerAddressFromDataNode, which reconstructs the same
host:port.grpcPort form from Id and GrpcPort.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: apply bucket default encryption when volume data encryption is enabled
When -s3.encryptVolumeData (s3a.cipher) is enabled, putToFiler skipped
checking and applying bucket default encryption due to a '!s3a.cipher'
guard. Volume-level data encryption and object-level Server-Side
Encryption (SSE-S3 / SSE-KMS) operate at different layers, and explicit
SSE headers already work alongside volume encryption.
Remove the '!s3a.cipher' guard so PutObject without explicit SSE headers
inherits bucket default encryption regardless of volume data encryption.
Add regression test TestPutObjectAppliesBucketDefaultEncryptionWithVolumeCipher.
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
* s3api: decrypt the volume cipher on direct SSE chunk reads
fetchFullChunk, fetchChunkViewData, and createEncryptedChunkReader fetched
raw bytes over HTTP, so volume-encrypted chunks reached SSE-S3/KMS/C
decryptors still ciphered. Route them through fetchChunkData: encrypted or
compressed chunks go through RetriedFetchChunkData (cipher-aware, slices
plaintext space for views); plain chunks keep the streaming range read.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: cover cipher-aware chunk reads with a fake volume server
The fake volume now serves stored GETs with Range support, and
TestFetchChunkDataDecryptsVolumeCipher verifies full-chunk and view reads
return plaintext for ciphered chunks while plain chunks still slice via HTTP
ranges.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: guard fake volume server stored map with its mutex
The HTTP handler goroutine read v.stored while test goroutines wrote
it, a data race go test -race can flag. Lock v.mu around the map read
and the test writes.
---------
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* kms/azure: fix encrypt/decrypt round trip for Azure Key Vault
The provider stored the wrapped data key as string(encryptResult.Result)
in the JSON envelope, sent the encryption context as AAD to an RSA-OAEP
key, and passed a full key URL to the azkeys client in the name
position. Each of those breaks a round trip on its own.
- split the key URL into the (name, version) pair azkeys expects before
Encrypt, Decrypt and GetKey; a plain name still resolves to the latest
version, and decrypt keeps the stored version so objects stay readable
after a key rotation
- store the wrapped key base64-encoded and decode it strictly, so the JSON
envelope no longer replaces the raw bytes with U+FFFD
- stop sending AAD: Key Vault rejects it on RSA-OAEP with BadParameter,
so the encryption context is logged and dropped instead
* kms/azure: reject a key URL that names another vault
splitKeyID dropped the host of a full Key Vault URL, so a key ID from a
different vault silently resolved to this vault's same-named key. Return
an error instead of encrypting under a key the caller did not ask for.
* kms/azure: bind encryption context via an envelope digest
RSA-OAEP rejects AAD, so removing it left the encryption context
unauthenticated: a wrapped key could be decrypted under a different
object's context. Record a sha256 digest of the marshaled context in
the envelope's provider_specific field on encrypt and verify it before
calling Decrypt, restoring the binding without AAD.
* kms/azure: treat an explicit :443 port as the same vault
* kms/azure: accept an absent context digest only for empty contexts
* kms/azure: normalize both hosts when comparing key URLs to the vault
splitKeyID stripped :443 and a trailing dot from the configured vault but
only :443 from the key URL, so a key URL naming the same vault with a
trailing DNS dot was rejected before Azure was ever called. Compare both
sides through vaultHost so they are normalized identically.
* kms/azure: reject non-key vault URLs in splitKeyID, document digest limits
A Key Vault URL that does not name a key under /keys/, has an empty key
name, or carries extra path segments now fails fast instead of being
passed to the client as a key name, where it would surface as a
confusing vault-side error.
Also note that the envelope context digest is a client-side mismatch
check, not vault-authenticated AAD, and only allocate providerSpecific
when a context is present.
* ci: compile and test azurekms-gated code
weed/kms/azure is excluded from every default build, so nothing in CI
compiled it; that is how the provider shipped unregistered. Build the
tree and run the kms tests with -tags azurekms on every Go change.
---------
Co-authored-by: Yi-111-a <>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: fail the request, not the process, when chunks cannot be resolved
CompactFileChunks and ViewFromChunks discarded the error from
NonOverlappingVisibleIntervals and iterated the nil interval list it
returns on failure. A CreateEntry or UpdateEntry whose context was
cancelled therefore panicked in SeparateGarbageChunks and took the whole
filer down, losing the metadata log it had not yet flushed.
Both functions now return the error. cleanupChunks fails the request
with it, the mount flush paths return it, the replication sinks and the
S3 SSE range reader return it, and a ChunkStreamReader that cannot build
its view fails every read with it and reports it as its SourceError
instead of reading as an empty file.
Fixes#11682
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* filer: propagate chunk resolution failures in scan paths
NonOverlappingVisibleIntervals can fail without a cancelled context; the
query engine and log-store callers discarded the error and dereferenced
nil intervals.
* query: propagate chunk resolution errors instead of skipping files
When a manifest cannot be resolved, ReadParquetStatistics,
countLiveLogRowsExcludingParquetSources, and computeLiveLogMinMax logged a
warning and skipped the failed file, so queries returned understated counts
and wrong min/max values. Surface the errors instead: the fast-path
aggregation collector now returns a DataSourceError so the engine falls
back to a full scan, and the row-count helpers return the error rather
than a partial total.
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(s3api): accept max-keys up to 2147483647 and cap list pages at 1000
* add tests for ListObjects max-keys range and 1000 cap
* refactor(s3api): return uint16 from parseMaxKeys to avoid narrowing casts
* fix(s3api): validate ListObjectVersions max-keys and honor max-keys=0
* add test for ListObjectVersions max-keys=0
* fix(s3api): widen uint16 max-keys for ListObjectVersions
* filer: fail writes when the existing-entry lookup fails
CreateEntry only returned a FindEntry error for exclusive creates; a
regular write treated a transient store error as "entry not found" and,
on upsert stores, replaced the entry without loading its old chunks —
leaving them out of the cleanup process. Return any lookup error except
ErrNotFound for both paths.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: test that a lookup failure fails a regular write
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: count index-load metrics only for live-key removals
The in-memory and leveldb offset loaders counted a deletion for every
tombstone row replayed, including re-deletes of already-deleted keys
(runtime logDelete only counts a live removal), and added the returned
old size to the byte counters even when it was a tombstone sentinel —
uint64(-1) wraps DeletionByteCounter. The leveldb offset loader also
counted every index row as a file and its bytes unconditionally. Gate
the counters the same way the runtime paths do so a reload reports the
same numbers the runtime counters hold.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: derive index metrics from latest-row state in the metric loader
The leveldb metric loader walked the index in reverse and incremented
FileCounter for every row (inflating FileCount with tombstones) and
DeletionCounter for every superseded row plus every deleted-key row
(double counting both). Per key, the runtime deleted count is exactly
(valid rows) - (1 when the latest row is live), so count live-latest
rows via the same bloom filter and derive both deletion counters; this
matches the runtime logPut/logDelete counters exactly instead of
drifting with tombstone history.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* server: warn instead of failing volume copies on counter drift
checkCopyCounts compared the source's in-memory counters against the
target's fresh-load counters, but counters computed by different
loaders drifted apart for identical .idx data, so a good copy was
rejected (issue #11678). The file-size checks already gate integrity;
log the mismatch instead of failing the copy.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: assert MaxFileKey parity in the metric loader test
MaxFileKey derives from row order alone, so unlike the bloom-filtered
deletion counters it must always match the runtime counters exactly.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A remote-only entry whose object was deleted from the remote storage
outside the filer answered GET with 500 and stayed in the filer. The
remote's not-found was lost on the way: the backends' ReadFile returned
it as an untyped error, so FetchAndWriteNeedle failed with codes.Unknown
and nothing downstream could tell it from any other failure.
- remote_storage: GCS, S3 (NoSuchKey) and Azure (BlobNotFound) reads
return ErrRemoteObjectNotFound. GCS reports a missing bucket the same
way as a missing object, so it confirms the bucket with a listing.
- volume server and filer: the not-found crosses gRPC as codes.NotFound
carrying the sentinel's text, and the filer's cache RPC returns
codes.NotFound, which the S3 gateway already maps to NoSuchKey.
- s3api: the origin fallback answers NoSuchKey on a confirmed not-found.
- filer: a confirmed not-found removes the stale entry, so the lazy
remote-metadata cache converges on the remote. Only remote-only files
outside .versions and without an active object lock are removed, only
if unchanged since the fetch (checked on the object's write owner,
under the lock S3 object writes take), and with a metadata-only
delete: the filer skips its inline remote delete, and the delete
events' entries carry a marker that makes filer.remote.sync and
filer.remote.gateway skip their remote delete, while filer.sync still
replicates it. The replicated DeleteEntryRequest carries
keep_remote_object, so the destination's delete events are marked too.
The store drops the marker from every write, so clients cannot plant
it.
* s3api: resolve volume data encryption in CopyObject SSE flows (#11646)
* s3api: honor bucket-default KMS key on copy and fix transformed-upload metadata
- Synthesize the destination bucket's default encryption as request
headers before any SSE evaluation, so the configured KMS key ID and
bucket-key setting reach the copy paths instead of only a boolean.
- uploadTransformedChunkData returns the upload result so callers record
the uploader's cipher key AND compression decision; a wrongly cleared
IsCompressed made transformed copies unreadable.
- decompressChunkVolumeCipher fails loudly when a compressed chunk does
not decompress, instead of uploading still-compressed bytes marked
uncompressed.
- copyMultipartSSECChunk now strips the volume cipher and re-encrypts on
upload like the other transform paths.
- The ciphered inner upload now uses the caller's private BytesBuffer.
* s3api: extract copy bucket-default header synthesis for coverage
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: test bucket-default encryption header synthesis on copy
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: validate copy encryption headers before bucket defaults and resolve empty KMS key
* s3api: name the AWS-managed SSE-KMS default key once
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(filer): end metadata subscriptions before gRPC GracefulStop on shutdown
GracefulStop waits for every open stream. The filer's own MetaAggregator
subscription and, under -s3, the S3 gateway's IAM subscription live in
the same process and only end when the filer shuts down, which happens
after GracefulStop - so every shutdown sat out the full 15s timeout.
StopSubscriptions ends them (and any started later) right after leaving
the lock ring; the handlers derive their context from the stream and
this signal. Measured on weed server -s3: stop 25.6s -> 10.7s, and
0.6s with -volume.preStopSeconds=0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FTq6bDpgdQfqQagQsUUvdw
* fix(filer): end stopped subscriptions with Unavailable, not cleanly
A subscription that StopSubscriptions cut short returned nil. The client
reads that as io.EOF, "caught up, done", and util.RetryUntil - which the
S3 gateway and mount follow with - stops on nil. A separate S3 gateway
therefore never resubscribed after its filer restarted (measured: 0
subscriptions in 25s after the restart; upstream master and this fix: it
is back within a second).
endOfSubscription now turns that clean end into codes.Unavailable, which
is what a dropped connection looks like, so followers reconnect. Errors
pass through, and so does the end of a stream the client closed.
A subscriber that has stopped reading is still left to GracefulStop's
timeout: its handler sits in stream.Send, which only the end of the
stream releases, and grpc-go does not allow Send after the handler
returns. Documented at StopSubscriptions. Shutdown under weed server -s3
is unchanged at 10.7s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FTq6bDpgdQfqQagQsUUvdw
* fix(filer): interrupt in-flight subscription reads and end them Unavailable
StopSubscriptions only reached the wait points: a handler replaying the
ring or the persisted log kept streaming until the pass ended, and a pass
cut short at a ctx.Err() boundary (the replay semaphore, chunk ref
collection, ref sends) returned a wrapped context.Canceled, which reads as
codes.Canceled on the wire - a code transient-error classifiers treat as
non-retryable.
eachLogEntryFn and sendRefsBatched now check the subscription context per
entry/batch, and endOfSubscription reports a context.Canceled cut short by
the stop signal as codes.Unavailable, same as a clean end it interrupted.
* filer: honor the subscription context in the pipelined sender
A subscriber that stops reading wedges sendLoop in stream.Send; Send and
Close then blocked past StopSubscriptions, so GracefulStop still waited out
its timeout. Both now return when the subscription context ends, letting the
handler return and gRPC tear the stream down.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: stop sendLoop from starting sends after the subscription ends
With Send and Close released by the subscription context, sendLoop could
still issue a stream.Send after the handler returned. Gate every send on
the context so at most the in-flight Send overlaps teardown - and the
stream's own teardown is what unblocks it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: assert sendLoop exits in the cancel test
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: consolidate session token extraction into extractSessionToken
Three call sites duplicated the same header/header/query lookup; share
one helper named after the existing s3tables equivalent.
* s3api: tolerate session tokens on statically configured credentials
Credential vendors like Unity Catalog emit a session token with every vended credential, including static ones (UC's StaticAwsCredentialGenerator only engages when s3.sessionToken is set). Requests signed by a configured access key were routed to STS validation and rejected, so static credential vending never worked against SeaweedFS.
Resolve the access key first: when it maps to a configured credential the signature alone authenticates the request, and the attached token is marked ignored so authorization does not route it into the STS session-policy path. STS-issued access keys are never in the static map, so temporary credentials still validate their token exactly as before.
* s3api: cover static credentials carrying a session token
* test: exercise UC static credential vending against SeaweedFS
s3.sessionToken.0 selects UC's StaticAwsCredentialGenerator, which vends the configured keys verbatim. The vended session token is foreign to SeaweedFS and previously failed SigV4; now it round-trips through temporary-table-credentials into real S3 I/O.
* s3api: reject temporary credentials in GetFederationToken after auth
Token presence alone cannot distinguish a temporary credential from a
statically configured one carrying a vended token. Move the check behind
verifyV4Signature and key it on the operative session token so tolerated
tokens keep the caller eligible.
* s3api: test GetFederationToken with authenticated temporary credentials
Sign the rejection cases with real session credentials so they reach the
post-auth check, and cover a vended static credential being accepted.
* test: check vended-credential delete error in UC integration test
* volume: answer chunk-manifest HEAD from the chunk-size sum
* volume: size chunk-manifest reads by the sum of chunk sizes
* volume: saturate the sum of chunk-manifest Range lengths
* volume: fetch remote chunk-manifest ranges with a bounded Range
* volume: fmt chunk-manifest handlers
* volume: cap whole-window chunk reads at the visible window
* volume: reject an over-cap chunk span and an illegal manifest MIME
* volume: take the manifest transform extension from the URL first
Go's tryHandleChunkedFile keeps the extension parsed from the URL —
including a fid suffix like /vid,fid.png — and only falls back to the
resolved filename's extension when the URL has none. The manifest path
now does the same, so a transform requested on a fid-suffixed URL runs
as it does in Go.
Also: compute the chunk-size sum once, drop an unused remote_read_for
parameter, share the crop/resize predicate with parse_read_request, and
settle a couple of clippy lints.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3api: revoke the identities a config reload no longer declares
A static config file reload merges with isFullState=false, so an identity
deleted from -s3.config kept authenticating with all of its access keys until
the process restarted. An operator revoking a leaked key got no error and no
revocation.
The file is the source of truth for the identities it declares, so a reload now
drops the ones that have left it, together with their access keys and those of
their service accounts. staticIdentityNames was only ever added to, which kept a
removed name protected as well; the names the file itself declares are tracked
separately from the AWS environment credentials, so a reload never revokes what
the file never declared.
* s3api: keep a file reload from replacing the dynamic store
Removing the last identity from a config file emptied staticIdentityNames, so
useStaticConfig turned false and the next reload of that file took the replace
path: every filer-managed identity and its access keys disappeared until a
dynamic reload brought them back.
A static config file is authoritative for the identities it declares and never
for the dynamic store, so a file load now always merges. The first file load at
startup takes the merge path as well, from an empty state.
Reported by the Devin and Greptile reviews of this PR.
TestReloadStaticConfigWithoutIdentitiesKeepsDynamic reloads an emptied file twice
and asserts that a filer-managed identity and its access key survive both; the
existing test now also checks that the service account key was loaded before the
reload and that the environment identity's key still works after it.
* s3api: clear the environment credentials in the emptied-file test
The AWS environment identity is static and is re-added after every merge, so on
a runner that has AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set it kept
hasStaticConfig true once the file was emptied, and the next reload still took
the merge path: the regression this test guards passed unnoticed.
Clear both variables whatever the runner has set. With the pre-fix condition
(hasStaticConfig alone) and the variables present in the environment, the test
now fails with "reload 2: a filer-managed identity must survive a reload of an
emptied file".
Reported by the Greptile review of this PR.
* s3api: verify under the write lock before re-committing an ambiguous routed PUT
A routed PUT whose response was lost after the owner committed, or whose
transaction returned a store error, fell straight into the lock path and
re-sent the same entry. A concurrent PUT that had superseded the commit in
the meantime had already deleted this entry's chunks as old, so the
re-commit restored metadata pointing at dead needles and the object read
back 404 permanently.
The lock path now resolves the routed attempt's outcome inside the write
lock first: a stored entry with the uploaded chunks means the route
landed; a different stored entry or an unresolved lookup refuses the
re-commit with ServiceUnavailable so the client retries with a fresh
upload; only a proven-absent entry falls through to the normal create.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: tighten ambiguous routed PUT recovery
- a match found only after an unanswered filer is not authoritative;
the unreachable filer may hold a newer entry, so refuse instead of
finalizing on it (both recovery paths)
- a failed post-recovery finalization now rolls the recovered entry
back, matching the create path's undo
- a proven-absent entry is only re-committed after probing that the
uploaded chunks' needles are still alive; a committed-and-deleted
PUT would otherwise write back an entry pointing at reclaimed
needles
- the .versions latest pointer no longer flips back to an older
version when a newer one committed while the write was uncertain
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: stamp late versions noncurrent when a newer latest pointer wins
The keep-newer early return in updateLatestVersionInDirectory left the
version just stored without ExtNoncurrentSinceNsKey, so the lifecycle
engine could never age it out.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: only a failure past mutation 0 makes a routed PUT ambiguous
A deterministic refusal at the PUT mutation (e.g. "existing entry is a
directory") applied nothing, but marking it ambiguous sent the lock-path
fallback through conservative recovery, which found the directory entry
and refused with 503 instead of the correct 409.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: keep all transaction errors ambiguous; route directory conflicts through the lock path
* s3: verify uploaded chunks before re-committing over a directory
A stored directory does not prove the routed PUT never committed: the
route can land, a delete can remove the entry and its chunks, and a
nested write can recreate the directory before recovery takes the lock.
Re-committing then stores an entry pointing at dead needles. Probe the
uploaded chunks first and refuse with ServiceUnavailable when they can
no longer be verified.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: skip cache lock when a cacher cannot be removed yet
SingleChunkCacher.readChunkAt defers removeConsumed on every read, and
every unpin retries removal too. Each call acquired the ReaderCache
mutex even though the removal conditions are plain atomics, so a busy
cache paid a lock acquisition per read just to discover readers>0.
Check the atomics before locking: when a cacher is not yet consumable
the removal is impossible and the lock round trip is pure contention.
Removals still run under the lock via removeConsumedLocked, so the
attach-vs-remove race keeps its existing serialization.
Ref #11676
* filer: guard stream position with a per-stream mutex, not the cache lock
chunkStream.cacher is only ever shared by concurrent ReadAt calls on one
ChunkReadAt, yet pin, unpin, releaseIfFinished, and releaseStream all
mutated it under the ReaderCache mutex that serializes every reader in
the process. Each read that switched chunks took that global lock to pin
the new chunk, released it, then reacquired it in unpin() just to drop
the old chunk's pin and retry removal; releaseIfFinished and
releaseStream did the same lock-detach-unlock-relock dance. At high
small-GET concurrency that turned per-request stream bookkeeping into a
global convoy (#11676).
Give chunkStream its own mutex and drop the cache lock from the pin
lifecycle entirely: pin/detach serialize on stream.mu, the pin counter
and consumable checks are already atomics, and removeConsumed only
takes the cache lock when a cacher is actually removable. The map lock
now guards only map membership and read registration.
* filer: look up cached chunks under the read lock
readChunkAt serialized every small read on the write lock even though
the common paths are read-only: an existing downloader just needs its
read registered, and a chunkCache hit needs no map access at all. Each
GET also paid the lock a second time to reach the chunkCache check.
Use the read lock for the downloader lookup and registration; the
registered read keeps the buffer alive against a concurrent destroy,
and only error eviction, insertion, and removal need the write lock.
The chunkCache probe runs lock-free, and the insert path re-checks the
map under the write lock to cover a downloader registered in between.
* filer: start the chunk download outside the map lock
The insert path held the ReaderCache write lock across goroutine spawn
and the cacheStartedCh handshake, so every downloader miss serialized
against the startup of a fetch goroutine. Register the cacher in the
map under the lock, then start the download after releasing it; a fetch
that fails early still lands in the map and is evicted by the next
reader's completed-error check.
* filer: test that stream pin lifecycle stays off the cache lock
Regression coverage for the contention fix: pin and releaseStream on a
non-removable cacher must complete while the ReaderCache lock is held
by another goroutine.
* filer: pin the stream under the map lock
Between read registration and stream.pin the cacher showed zero pins,
so a budget eviction in that gap could pick a chunk the stream was just
attaching to and the stream's next slice refetched it. The pin counter
is an atomic and stream.mu is never held while acquiring the cache
lock, so pinning inside the map hold is deadlock-free and closes the
window.
* s3api: reject SSE headers PutObject cannot honor, as CopyObject does
PutObject and CreateMultipartUpload did not check the server-side
encryption headers they were given. An x-amz-server-side-encryption
value that names no method, such as "aes:kms", matched none of the SSE
paths, so the object was stored unencrypted and the request answered
200. SSE-C together with x-amz-server-side-encryption was also accepted,
and one of the two silently won.
CopyObject already rejects both through validateEncryptionCompatibility.
Run the same check, with the same error codes, before PutObject and
CreateMultipartUpload store anything.
* s3api: answer rejected SSE headers with InvalidArgument, as S3 does
S3 rejects an unknown x-amz-server-side-encryption value and SSE-C
combined with another method with InvalidArgument. PutObject and
CreateMultipartUpload now return that code, with S3's messages, through
two new error codes; CopyObject keeps its own.
Also run ceph/s3-tests' test_put_obj_enc_conflict_c_s3, _c_kms and
_bad_enc_kms in CI, which check both handlers' responses end to end.
* s3api: close the remaining ways a PUT could skip requested encryption
- A repeated x-amz-server-side-encryption header is rejected: the
encryption paths apply only the first value, so extra values could
hide the method the client asked for.
- KMS options (key id, encryption context, bucket key) are rejected
unless the method is aws:kms, matching the S3 InvalidArgument error.
- CreateMultipartUpload validates before auto-create, so a refused
upload cannot leave a bucket behind.
- Directory markers encrypt their inline content through the shared
SSE path and record the same entry metadata as regular objects,
instead of storing requested-encrypted bytes in plaintext.
* s3api: read back what the SSE marker write stores
- Repeated SSE-C and KMS option headers are rejected alongside a
repeated x-amz-server-side-encryption, closing the same first-value
bypass for customer-key and KMS fields.
- Algorithm validity is checked before KMS options so an unsupported
value keeps the "not supported" error.
- serveDirectoryContent decrypts marker content with the stored SSE
metadata and returns the SSE headers, so an encrypted marker reads
back what was written; its Content-Length now reflects the bytes
actually served.
* s3api: HEAD of a marker skips decryption and keeps the stored size
HEAD returns no body, so it now runs only the SSE-C key check instead of
decrypting — matching HeadObjectHandler and avoiding a KMS round-trip —
and chunk-backed directory entries report Attributes.FileSize again
rather than the length of their (empty) inline content.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: keep GET Content-Length to the bytes serveDirectoryContent writes
The FileSize override described chunk-backed markers on HEAD, but GET
sends only the inline content, so it promised bytes it never wrote.
* s3api: stream a chunk-backed directory on GET like any object
A directory promoted over an uploaded object keeps the object chunks
with empty inline content, so serving only entry.Content made GET
deliver nothing while HEAD reported FileSize. GET now routes those
entries through the regular volume-server stream, keeping the two
methods consistent.
* s3api: answer a busy volume read with RequestBytesExceed
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* helm: add Gateway API HTTPRoute for master, volume, filer, s3 and admin
Each component that offers an Ingress can now be exposed through a
Gateway API HTTPRoute instead, via <component>.httpRoute. It is disabled
by default, so existing renders are unchanged.
parentRefs, hostnames, labels and annotations pass through as given.
Each rule may set matches, filters, timeouts and backendRefs, and a rule
without backendRefs routes to the component's own service and port,
following all-in-one mode the same way the Ingress templates do. An
empty rules list yields a single rule sending all traffic there.
Signed-off-by: younsl <cysl@kakao.com>
* helm: fix HTTPRoute backend selection for S3 on filer and all-in-one
- s3: with S3 on the filer the route targets filer.s3.port, the port the
s3 Service exposes. The all-in-one Service is chosen only when
allInOne.s3 is enabled, so a standalone S3 next to all-in-one routes
to the s3 Service.
- master: render in all-in-one mode and route to the all-in-one Service,
like the volume and filer routes.
- Name routes with seaweedfs.componentName, keeping them within 63
characters like the Services they point to.
- CI: cover the filer S3 port and the all-in-one master route.
Signed-off-by: younsl <cysl@kakao.com>
* helm: document the Gateway API HTTPRoute values in the chart README
Signed-off-by: younsl <cysl@kakao.com>
* helm: link the Gateway API docs from the chart README
Signed-off-by: younsl <cysl@kakao.com>
* helm: scope the filer S3 port note in the chart README
Signed-off-by: younsl <cysl@kakao.com>
---------
Signed-off-by: younsl <cysl@kakao.com>
The handler sends the header non-blocking and the test stops on fetch
errors instead of waiting on a channel that may never be fed.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: stop the compat server even when startup fails
test-with-server depended on start-server as a prerequisite, so a failed
startup skipped the recipe and the spawned weed mini kept the port.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: verify the recorded PID is weed before stopping it
A prerequisite failure leaves a stale weed-server.pid in place; the PID
may have been reused by an unrelated process.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
calculateSeedSignature called verifyV4Signature with
shouldCheckPermissions=true, which runs VerifyActionPermission — a
check that does not consult bucket policies. Every caller of
newChunkedReader is already behind the Auth middleware, which does
evaluate them, so a principal allowed only by the bucket policy was
authorized upstream and then denied when the handler built the body
reader: aws-chunked PutObject/UploadPart (botocore's default shape
over TLS, PyArrow's over HTTP as well) failed with AccessDenied.
The reader now verifies only the signature; a wrong secret is still
SignatureDoesNotMatch. The non-streaming paths already work this way.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
rewrite_position_delete_files emitted a schema-less parquet file whose
file_path column was DELTA_LENGTH_BYTE_ARRAY. PyIceberg reads delete
files with file_path as a dictionary column, which PyArrow cannot
decode from that encoding, so every table the worker rewrote failed
scans outright; readers that resolve columns by field id found none.
The row struct now declares the spec's reserved field ids
(2147483546 file_path, 2147483545 pos) and dictionary-encodes
file_path, which is also the compact choice for a column repeating one
data file's path.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Orphan cleanup built its referenced set from snapshot manifests plus the
current and previous metadata files, then walked all of metadata/ and
data/. Table statistics (Puffin) and partition statistics files are
listed in the table metadata's statistics and partition-statistics
lists, not in any snapshot, so once they passed the safety window
remove_orphans deleted them while the metadata still pointed at them.
Mark the files in both lists referenced as well.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: answer a denied write with 403 when the caller can read the entry
Update/Delete/Rename answered every authorization refusal as not-found so
the denial leaked no existence signal. For a caller allowed to GetTable
(or GetView) the same entry, the veil hides nothing it could not load —
yet a refused write was still answered 404, so clients saw a table they
just loaded reported as missing.
When the write check fails, re-check read permission on the same entry:
readable entries get 403 AccessDenied; invisible ones keep the
not-found answer.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg: map UpdateTable errors through writeManagerError on commit paths
CommitTable and CommitTransaction answered any non-conflict UpdateTable
failure, including AccessDenied and NoSuchTable, with a bare 500. A 500
reads as outcome-unknown to clients (PyIceberg raises
CommitStateUnknownException) where a refused commit is a plain
ForbiddenException, matching what the drop and rename handlers already
emit through the same mapper.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: keep the not-found veil over views and match the real read check on renames
UpdateTable and DeleteTable read shared metadata without checking the
entry kind, so a denied write on a view reported 403 to a GetTable-only
caller where a missing name reports 404. The rename visibility check
also fed resource tags into GetView evaluation that the real GetView
path never supplies.
* s3tables: gofmt handler_table.go
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Alex Russell