mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-13 01:28:37 +02:00
acb44a68a97a26bc5d0816c285c9bfb0f093f6a3
15474
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
acb44a68a9 |
fix(volume): skip unnecessary needle data reads on delete (#11711)
* fix(volume): do not read a needle's data to delete it A delete read the needle's whole payload from disk only to discard it. The delete handler and BatchDelete need the needle's cookie and its chunk-manifest flag, which live in the header and trailer, but fetched them with a full needle read, the same call a GET uses. readNeedle now reads the header and trailer only when MustMetaOnly is set, whatever the needle's size, and the delete paths ask for that. The Rust volume server honours must_meta_only the same way for its delete paths. A compressed needle or a chunk manifest still gets the full read, as before, because their callers need the bytes. HEAD is unchanged. It used to set MustMetaOnly, which nothing read; it now asks for AttemptMetaOnly alone, so it keeps reading a needle up to PagedReadLimit whole. On a spinning disk that one read is cheaper than a header read followed by a trailer read. Version 1 needles are always read in full. They have no data size field and no metadata after the data, which ReadNeedleMeta relies on, so a metadata-only read of one failed with "invalid needle meta size". That already happened for version 1 needles above PagedReadLimit read with AttemptMetaOnly. A delete therefore no longer verifies the data CRC of the needle it removes. The cookie check is unchanged. * fix(volume): read transformed images fully on HEAD too A HEAD carrying resize or crop params reports the transformed length, which a metadata read cannot produce. Go and Rust both answered such HEADs from the source header, returning the stored size instead of the cropped or resized one. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
6927de6003 |
seaweed-worker: mint the admin bearer token like the Go worker
The admin control stream now requires mTLS or a token signed by jwt.filer_signing.key; the Rust worker gets --filer-signing-key and --filer-signing-key-expires-after-seconds, mirroring the Go worker's config names, and attaches the token to the stream metadata. |
||
|
Chris Lu
|
613a286cde |
worker: present the admin bearer token on control streams
Workers mint a SeaweedFilerAdminClaims token when jwt.filer_signing.key is in their configuration, matching the authentication the admin server now requires. |
||
|
Chris Lu
|
55ac46f7a9 |
admin: authenticate worker and plugin gRPC streams
Worker registration previously ran unauthenticated on any interface the listener bound. The stream now requires a verified mTLS client certificate or a Bearer token signed by jwt.filer_signing.key, and falls back to local clients only when no key is configured. |
||
|
Chris Lu
|
3f8035deb7 |
security: share the admin gRPC peer authentication check
The local-peer fallback and bearer-token validation used by the filer IAM gRPC path move into weed/security so every admin-plane gRPC service can enforce the same contract. A client certificate verified by the server's TLS config now also counts as authenticated. |
||
|
Chris Lu
|
a4c8d31cb8 |
s3api: ignore response header overrides on anonymous requests
AWS requires response-* query parameters to be signed; an anonymous caller must not choose how a stored object is served. |
||
|
Chris LuandDevin
|
6bac506842 |
sftp: terminate the missing-ancestor permission walk at the root
A path whose ancestors are all absent recursed on the root forever,
since Dir("/") is "/".
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Chris LuandDevin
|
2c031d9d63 |
sftp: check write permission on the target file, not only its parent
newFileWriter authorized uploads against the directory, so a file with a more specific read-only ACL or mode could still be overwritten. The same parent-directory fallback used for nonexistent paths now applies through CheckFilePermission on the target itself. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
d69f45ee83 |
s3: validate multipart initiate ACLs like PUT
CreateMultipartUpload accepts canned and custom ACLs on initiation; run them through the same authorization path as PutObject so an explicit ACL requires s3:PutObjectAcl, and persist the validated grants on the upload entry for completion to apply. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
3aaf80b33b |
s3: stop trusting internal ACL headers in ParseS3Metadata
Multipart initiation stored client-supplied Seaweed-X-Amz-Acl and Seaweed-X-Amz-Owner request headers onto the upload entry, and CompleteMultipartUpload copied them to the finished object, letting a caller forge ownership or a public ACL without s3:PutObjectAcl. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
85f04b54fd |
s3tables: check data-path ancestors for catalog markers on DeleteTable
The decoupled-delete guard only inspects the resolved data path itself, but the catalog marker lives on a table's name path, so a location that resolves inside another table's directory (its marker-less data subdirectory) slips through and the recursive purge destroys the other table's data. Walk the data path's ancestors within the bucket and refuse when any carries a table or view catalog entry. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
6f93f87446 |
s3tables: confine caller-supplied metadataLocation to the table's namespace
CreateTable/RegisterTable/UpdateTable (and the view equivalents) validate the location only for bucket and traversal, so a tenant can register a table whose location resolves into another namespace's directory and DeleteTable will purge it. Require the location to live under the caller's namespace subtree, the boundary per-namespace delegation uses. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
scooby87
|
73949daa98 |
filer: keep sequential prefetch within prefetchCount chunks (#11674)
ChunkReadAt.ReadAt asks the ReaderCache to prefetch after every read, and MaybeCache counted only the downloads it newly started toward the window, so each call reached prefetchCount chunks further than the last. A reader consuming a chunk in several reads, as the S3 GET path does with a 256 KiB copy buffer over 4 MiB chunks, ran the read-ahead up to the ReaderCache downloader limit and kept most of a large object in memory, enough to get an S3 gateway OOM-killed mid-response. Count every chunk in the window, downloading or not, so the read-ahead stays at prefetchCount chunks past the reader. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Chris LuandDevin
|
603a1221a7 |
filer: converge peer metadata replication on restart (#11698)
* filer: skip replayed peer events older than the local entry Replay deleted and re-inserted every received event unconditionally, so a restarted filer that rewound its subscription offset re-applied stale peer writes over newer local state (values reverted, deleted entries came back and pointed at already-vacuumed chunks). Compare the event's TsNs against the local entry's Mtime and skip per-path, so a move event can still remove its source while keeping a newer destination. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: add replay last-writer-wins regression tests Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: stop peer subscriptions and flush offsets on shutdown Filer.Shutdown closed the store while subscription goroutines were still mid-stream; their deferred offset save was never guaranteed to run, so a restart rewound each peer's stream back to the last periodic flush and replayed stale events. MetaAggregator.Shutdown now closes every peer's stop channel and waits for the loops so the saved offset covers what was replayed, and the reconnect sleep is interruptible by the stop channel. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: test that shutdown stops and waits for peer loops Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: order replay by persisted per-path commit timestamps Review feedback found the mtime comparison still lets stale replay win: a local delete removes the entry it would have been compared against, an entry mtime is client-controllable, and the check ran unlocked so a local write could land between it and the store mutation. - Record the event timestamp per mutated path in the store KV on every emitted mutation, plus a delete record for sources of deletes and renames. The record survives the entry, so a replayed create older than a local delete no longer resurrects it; ancestor delete records cover children swept without per-child events. - Replay compares the event timestamp against that record under the same per-path entry locks local RPC mutations hold, falling back to entry mtime only for entries written before records existed. Lookup errors now abort the event instead of being treated as an absent entry. - MetaAggregator.Shutdown marks the aggregator stopped under peerChansLock so a master update cannot start a subscription that outlives the wait, and waits for the loops without the five-second cap now that every blocking stage — signature RPC, bootstrap traversal, retry sleeps, and per-event processing — cancels on the stop channel. * filer: pin peer offset at failed replay, fix commit-record context and key size A replayed event that exhausted its retries (or was interrupted by shutdown) was still followed by advancing the peer watermark, so the persisted offset skipped the unapplied event forever. replicateMetadataChange now reports the outcome: errPeerSubscriptionStopped aborts the subscription so the deferred save keeps the last completed event, and an exhausted retry records a barrier the persisted offset and reconnect resume point cannot pass while the stream keeps flowing. Commit records also failed on two fronts: NotifyUpdateEvent runs after CommitTransaction with the same context, so SQL stores wrote MpTs/MpDl through an already-committed *sql.Tx; and metaPathKey embedded the whole path, overflowing the 766-char name column SQL KV stores use, which the KvPut fallback update then silently dropped. The writes now use a fresh context and a sha256-hashed fixed-size key. * filer: lock HTTP delete handler and replay against ancestor paths The HTTP DeleteHandler called DeleteEntryMetaAndData without the shared entry lock, so a peer replay could still interleave with a local delete and its commit record. It now holds the same lock the gRPC DeleteEntry handler does. Replay also locked only the exact event paths, but its ordering check consults ancestor delete records: a local recursive delete holds only the ancestor's lock, so a stale child create could check before the tombstone landed and then reinsert. lockReplayPaths now locks every ancestor of both event paths in one lexicographic order — parents sort before children, as the rename handler's two-path order already does — while never locking the root, which cannot be deleted and would serialize all replays. * filer: record replay commit before the replacement insert If the same-path update DeleteEntry succeeded but InsertEntry failed, the path sat deleted with no commit record until the retry arrived, and a stale peer create could resurrect it in the gap. The commit record now lands immediately after the delete — without a delete record, since an update is not a delete — and again after the successful insert. * filer: lock replay ancestors shared, not exclusive Exclusive locks on every ancestor serialized all replays under a shared parent — unrelated objects in the same bucket could no longer apply in parallel. Ancestors only need to exclude an exclusive holder (a local delete or rename of the ancestor), so they now take shared locks; a path the event changes stays exclusive even when it is also the other path's ancestor. * filer: order peer bootstrap on the sender's commit timestamp traversePeerMetadata compared localCommitTsNs — a commit-clock value — against the entry's mtime, which a client can set freely: a peer change committed recently with an old mtime was skipped, and a stale entry mtime-stamped in the future overwrote a newer local commit. TraverseBfsMetadataResponse now carries the sender's per-path commit timestamp (recorded MpTs record, entry mtime as the unrecorded fallback). The bootstrap applies an entry only when its commit is newer than the local one and records the applied timestamp, so replayed events order against it — closing the gap between bootstrap content and the event stream. A peer without the field sends 0 and keeps the mtime fallback, preserving rolling-upgrade behavior. * filer: keep failed commit records pending until they land notePathChange ran after the mutation committed and only logged a KvPut failure: a delete whose MpTs/MpDl record never landed left no tombstone, so a stale replayed create could resurrect the path while the store was unreachable. Failed records now stay in metaTsPending and a repair pass retries them until they land. localCommitTsNs, deletedAfterTsNs and CommitTsNs consult the pending record, so ordering holds in the window between the committed mutation and the durable record. A per-path flag bounds the repair to one goroutine, and respawns cover merges landing between its last pass and the flag release. * filer: keep only unwritten work in the pending commit record The pending record merged every obligation into one timestamp and flag, and a later successful write left it untouched: a repair pass could then write an older timestamp over a newer one, and a merged delete inherited the newer change's time, which the ancestor check reads as a newer delete. The record now holds a change and a delete timestamp of its own, both discharged under the record's lock alongside the store writes that cover them, so a repair writes only what is still owed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: fail traversal on a commit-record read error CommitTsNs answered a store read failure with the entry mtime, a value clients can set arbitrarily far ahead; a peer bootstrapping on it would record the minted time and reject real updates below it. The lookup now returns the error so the traversal restarts instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
f11cecc697 |
helm: keep the S3 config checksum stable across the first upgrade (fixes #11719) (#11720)
* helm: keep the S3 config checksum stable across the first upgrade (fixes #11719) checksum/s3config hashed the rendered s3-secret.yaml. That template keeps its generated S3 keys with lookup, so rendering it again for the checksum drew new keys whenever lookup found nothing: on install the filer, S3 and all-in-one pods each carried a checksum of a different set of keys than the Secret got, and the first upgrade with unchanged values corrected it and restarted them. Hash the values s3-secret.yaml is built from instead: the enable and auth switches, existingConfigSecret, credentials, the legacy-secret settings, the audit log configs and the Secret name. Only those, so scaling the S3 gateway or changing its resources does not restart the filer. A change to any of them still restarts the pods. Keys that the chart generated stay out of the hash; they only change if the Secret is deleted by hand. The new render test fails on the old template (two checksums for one Secret, different on every render) and passes now; it also checks that the checksum follows the credentials and filer.s3.auditLogConfig and ignores s3.replicas and s3.resources. On kind, an upgrade with unchanged values no longer replaces the filer pod. * helm: check the S3 checksum per workload, all-in-one included Each workload's checksum/s3config is now read on its own (--show-only), so a workload that loses the annotation fails the test instead of being hidden by sort -u, and the all-in-one deployment gets the same checks. Verified against two broken copies: one without the filer annotation, one with the old all-in-one checksum; the test fails on both. The helper comment now says what happens when the Secret is deleted to rotate generated keys. * ci(helm): fail the S3 checksum test on a missing or unrendered checksum A failed helm template or a render without the annotation produced an empty checksum, and the inequality assertions then passed against it. s3_checksum now returns non-zero unless it found a checksum, and the changed-value renders are assigned before comparing so failures cannot hide inside test. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Ege Zambelli <egezamb@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Martijn HoekstraandClaude Opus 5.5
|
1076ebdf4a |
test: add build constraints to Unix-only files (#11712)
test/fuse_dlm, test/fuse_p2p/peer_chunk_sharing_test.go and unmaintained/check_disk_size use syscall.Flock, syscall.Stat_t and syscall.Statfs, which do not exist on Windows. Without constraints, gopls and go vet on a Windows host report them as undefined. fuse_p2p/framework_test.go already carries `linux || darwin`; the FUSE test files now match it. check_disk_size uses the same constraint as weed/stats/disk_supported.go. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Chris LuandDevin
|
bd0bc06df7 |
mount: unwind metadata lookups on FUSE interrupt (#11717)
* mount: propagate request context through metadata lookups The FUSE interrupt channel never reached the filer RPC: lookupEntry ran its LookupEntry under context.Background(), so a stalled filer parked ops like ListXAttr forever even after the kernel gave up on them. Thread a context from each op's cancel channel through maybeReadEntry, maybeLoadEntry, lookupEntry, ensureDirectoryVisited and waitForPendingAsyncFlush; the existing withFilerClient attempt bound turns it into both an interrupt and a deadline. A caller cancellation now surfaces as EINTR instead of waiting out the RPC. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: reuse ctxFromCancel for the read path's interrupt bridge Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: regression tests for canceled and stalled filer lookups Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: thread ctx through AcquireHandle Open, Create and Mknod all carry the request's cancel channel; let it bound the pending-flush wait and metadata load inside AcquireHandle too. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: keep flush waits and write sequences uninterruptible The cancel-derived context must only cover metadata reads: an interrupted Unlink or Rename that skips the pending-flush wait can race the async write back into existence, and Link/Rename mutations that run under a canceled context abort mid-sequence, strand the rollback, and leave wrong hard-link counts. Restore the blocking flush wait and run each op's write phase on context.Background(). Rmdir and Unlink also propagate EINTR from the parent lookup instead of proceeding past a permission check the interrupt prevented. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: interrupt shared directory builds and directory reads Waiters behind another request's singleflight directory build blocked until it finished even when interrupted; switch the wait to DoChan so the request context can abandon it. ReadDir paths now carry the request context through EnsureListingFresh, cache listings and direct reads, and map its cancellation to EINTR rather than EIO. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: assert exact statuses in filer-stall regression tests Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: let read-side flush waits release on interrupt waitForPendingAsyncFlush reports whether the flush actually finished: lookupEntry and AcquireHandle return EINTR when the request is abandoned mid-wait, while Unlink and Rename keep waiting on context.Background() so the delete/rename still sees the flushed state. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: propagate interrupt through remaining lookup checks Rename's source and destination permission lookups now return EINTR on interrupt instead of skipping checkStickyBit or masking an existing target, and the readdir preload visit maps cancellation to EINTR like the other branches. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: fail closed on errored rename prerequisite lookups * mount: stop rmdir on interrupted target lookup, fail closed on dir lookup errors --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
50a9312e1e |
iceberg maintenance: v3 row lineage so v3 tables compact and rewrite manifests (#11653)
* iceberg maintenance: commit v3 row lineage on the three commit paths
iceberg-go 0.7.0 refuses every snapshot on a format-version 3 table
that lacks first-row-id and added-rows ("invalid row lineage"), so no
v3 table was ever compacted or had its manifests rewritten; worse,
passing 0 to the manifest list would have handed new files row IDs
existing rows already hold.
Each commit now plans first-row-id from the table's next-row-id, lets
the v3 manifest-list writer assign manifest first_row_id from there,
and reports the assigned range as the snapshot's added-rows so
next-row-id advances by exactly that. The commit guard also treats a
moved next-row-id or a changed format version as a stale plan, like a
moved head: a commit to another branch advances next-row-id too, and
manifests written for the old version must not be committed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: keep row identity through v3 compaction
A rewrite that moves rows into a new data file must not renumber them.
On v3 tables the merge now appends _row_id and
_last_updated_sequence_number (spec field ids 2147483540/2147483539)
to the output and fills each row with the value its input
materializes, else the inherited one: the input file's first_row_id
plus the row's position before any delete, and the manifest entry's
data sequence number. Inputs are matched with their lineage columns
set aside, so a file an earlier compaction wrote merges with one an
engine wrote; a duplicate lineage column or a reserved name under
another field id fails the bin instead of writing a doubled column.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: skip rewrite_position_delete_files on v3 tables
The spec forbids adding position-delete files to a format-version 3
table (v3 writes deletion vectors), and this rewrite adds new ones.
Detection no longer proposes the operation and the operation returns a
skip result; compaction still applies and removes existing position
deletes.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: keep Puffin deletion vectors out of position-delete reads
A Puffin file in a delete manifest is a deletion vector, not a
position-delete file: reading it as Parquet fails, and merging the data
file it covers without applying the vector resurrects deleted rows.
Skip Puffin entries in the position-delete collection and the delete
rewrite group scan, and exclude the files a vector covers from
compaction candidates - in compactDataFiles and in eligibility
detection alike. A vector whose referenced_data_file is missing or
unresolvable fails the run rather than silently compacting whatever it
points at.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: name a merged file's own first row ID
Rewritten rows keep their IDs, so a merged data file's first_row_id is
the _row_id of its first written row - not the next free range the
manifest list would otherwise assign to the entry at commit. Read the
first materialized value back off the merged output and stamp it on the
manifest entry.
Add an end-to-end test that populates a v3 table whose entries carry no
first_row_id, so the compact path exercises manifest first_row_id
inheritance through the stored manifest list, then commits and reads
the result back through the real manifest readers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: locate the _row_id column by index, not row position
Repeated columns shift parquet.Row positions, so leaf.ColumnIndex is not
the value's slot — a first row containing a list or map saved an unrelated
value as the merged file's first_row_id.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: read the returned row before treating EOF as empty
parquet-go can return a row alongside io.EOF; a one-row merged file then
lost its first_row_id and the manifest list would allocate a fresh range.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: avoid nil dereference in first-row-id failure message
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
github-actions[bot]
|
1bc430a4ec | docs: regenerate star history chart | ||
|
Chris Lu
|
63d2024b82 |
filer.remote.sync: upload a rewrite that lands in the remote mtime's second (#11715)
* filer.remote.sync: resolve same-second updates with the sync stamp shouldSendToRemote compared remote_mtime to the entry mtime in whole seconds, so a rewrite landing in the same second as the previous upload was dropped and never retried. When the two seconds are equal, fall back to the full-precision LastLocalSyncTsNs stamp, the same check remote.uncache uses to refuse uncaching an unsynced entry. An entry without the stamp is treated as unsynced and sent. Fixes #11713 * filer.remote.sync: restamp the entry after a metadata-only remote update A same-second metadata edit now reaches UpdateFileMetadata, but without advancing LastLocalSyncTsNs the entry stays dirty and every later event re-sends the metadata update. Stamping it through updateLocalEntry keeps the same IF_ENTRY_EQUAL guard as an upload stamp. * filer.remote.sync: stamp a cloned RemoteEntry in updateLocalEntry Callers can pass the event's own RemoteEntry; mutating its stamp before the expected entry is cloned put the new stamp into IF_ENTRY_EQUAL while the filer still holds the old one, so every restamp was rejected as stale. Stamping a clone keeps the precondition on the stored stamp and leaves the caller's RemoteEntry untouched. |
||
|
|
72da592b27 |
fix(filer): prevent nil pointer dereference in CompactFileChunks when context is cancelled (#11690)
* filer: prevent nil pointer dereference in CompactFileChunks when context is cancelled Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com> * filer: propagate chunk resolution errors through ViewFromChunks A failed manifest resolution produced a nil interval list; with the nil guards this silently became an empty view, so replication sinks would write empty replicas and report success. Return the error so callers fail instead, and make ChunkStreamReader report it on first read. --------- Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
dependabot[bot]
|
cfbfb60817 |
build(deps): bump github.com/ydb-platform/ydb-go-sdk-auth-environ from 0.5.2 to 0.5.3 (#11666)
build(deps): bump github.com/ydb-platform/ydb-go-sdk-auth-environ Bumps [github.com/ydb-platform/ydb-go-sdk-auth-environ](https://github.com/ydb-platform/ydb-go-sdk-auth-environ) from 0.5.2 to 0.5.3. - [Changelog](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/blob/master/CHANGELOG.md) - [Commits](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/compare/v0.5.2...v0.5.3) --- updated-dependencies: - dependency-name: github.com/ydb-platform/ydb-go-sdk-auth-environ dependency-version: 0.5.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
9a65616cd0 |
filer: announce to master only once gRPC serves, and leave before it stops (#11700)
* wdclient: gate master registration until the server is serving The first KeepConnected send registers the client into the master's cluster list, which is what routes S3 and peer traffic to it. A filer connects to the master long before its gRPC port serves (peer discovery, store bootstrap), so it announced itself while still refusing connections. announceCh defers only the registration send, letting the stream machinery connect early without joining the list; ListMasters exposes the configured masters for callers that cannot wait on the stream. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: announce to the master only once gRPC serves NewFilerServer previously registered on the master's filer list during startup, before the gRPC listener was serving — and on a fresh store before peer bootstrap finished — so a restarting filer could be routed to while refusing connections. The announce channel is closed right after grpcS.Serve starts. ListExistingPeerUpdates now iterates the configured masters directly since GetMaster would block on the gated stream. On shutdown, cancelling the master context ends the KeepConnected stream during the leave phase, so the master drops the filer node before gRPC stops accepting; fs.Shutdown cancels it as a fallback. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * wdclient: test that the announce gate delays registration Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * wdclient: unblock queries behind the announce gate The gate holds the registration send, but setting currentMaster only after the first response left GetMaster blocked for the whole gated window. Any startup path that queries the master before serving (e.g. loading a chunked filer.conf) deadlocked the constructor, so the gate never opened. Mark the master queryable once the stream is established; registration and leader redirect still wait on the gate and first response as before. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: keep the fullest peer list from any master ListExistingPeerUpdates returned the first non-nil answer, so a follower master could seed the peer list with a stale partial view before the leader replied. Keep the longest response instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: keep master queries usable after the shutdown disconnect Leaving the filer list early ends the KeepConnected stream, which cleared currentMaster and left the final metadata-log flush unable to reach the master during drain. Keep the last serving address queryable after a canceled disconnect, and move the Shutdown fallback cancel behind the filer's own shutdown so callers that skip LeaveLockRing flush against a live stream. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * cluster: propagate a bounded context through ListExistingPeerUpdates The helper dialed and queried masters on context.Background, so one unresponsive master could stall the whole peer scan and ignore caller cancellation. Pass ctx through, bound each master query in the filer's scan, and stop when the caller cancels. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: wait for cluster removal before draining gRPC Cancelling the KeepConnected stream only signals removal; the master drops the filer when its stream handler unwinds. Until then new requests can still route to this filer while its listener is closing. Poll the configured masters, bounded, until none still lists this filer. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * wdclient: serialize master discovery access ListMasters made peer scans a second concurrent accessor of mc.masters alongside tryAllMasters, while ServerDiscovery refreshes and reads its address list unsynchronized. Guard the refresh and every instance copy behind one lock. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * cluster, filer: prefer the leader's peer list and report lookup errors Keeping the longest answer let a stale ex-leader's list win: after a leader change the old master retains registrations until its streams end, so a fresh filer could bootstrap from an unreachable peer. Ask the masters for the raft leader and take its list; only when the leader is unreachable fall back to the fullest answer. ListExistingPeerUpdates also returned an empty slice on RPC failure, which a fresh filer read as "no peers" and skipped bootstrap, serving without the existing files. Propagate the error and retry the scan during startup, before the announce gate opens. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: bound the removal wait and retry unconfirmed masters Each master query got its own two-second timeout while the five-second deadline was only checked between scans, so three stalled queries could stretch the wait past its budget. A failed query also counted as confirmed removal, letting gRPC drain while a master still listed the filer. Derive every query context from one overall deadline, track which masters confirmed absence, and retry the rest until they answer or the budget expires. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * wdclient: run the SRV lookup outside the masters lock ListMasters and tryAllMasters held mastersLock across the DNS lookup, so a slow resolver stalled GetMasters and ListMasterSet readers past their request deadlines. Split the lookup from the list swap: resolve the SRV record unlocked, then lock only to save it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * wdclient: bound WaitUntilConnected in the announce-gate test The wait ran on the caller context with no deadline, so a regression that left currentMaster unset hung the whole test run instead of failing the check. Give it the same three-second budget as the assertions that follow. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * wdclient: order SRV refreshes and reuse the cached list on shutdown * filer: confirm an empty leader peer list before accepting it * filer: announce only after the startup peers' replay catches up --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
985d0473bf |
build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.109.1 to 1.114.1 (#11669)
build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.109.1 to 1.114.1. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.109.1...service/s3/v1.114.1) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-version: 1.114.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
6d692ab013 |
mount: fail writes over to the next filer instead of hanging on a dead one (#11696)
* mount: bound each filer call so a wedged filer cannot hold the failover walk WithFilerClient callers run their RPCs on context.Background(), so a filer whose transport stalls without refusing (stopped peer, black-holed connection) blocks the call forever and the walk never reaches the remaining filer addresses. Run each attempt under a timeout; on expiry drop the cached channel so the abandoned call is unblocked and the next address is tried. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: time out wedged mutation streams and rotate to the next filer doUnary and Rename wait on stream Send/Recv under context.Background(); a dead transport never delivers an error, so writes hang without ever reaching the unary fallback. Bound the waits and tear the stream down on timeout; teardown also advances the sticky filer index so the next stream opens against a different filer instead of redialing the dead one. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: cover filer failover past a wedged filer Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: cancel filer attempts by context and unwind them before retry Review follow-ups on the filer failover: - callFiler derives a per-attempt context instead of only abandoning the wait, so a timed-out attempt's RPCs die on their own without tearing down the shared cached channel. The walk now also waits for the attempt's callback to unwind before trying the next filer, so two runs of the same closure cannot race on the caller's captured results. Callbacks that ignore the attempt context are still forced out by dropping the channel, as a last resort after a grace period. - teardownStream captures the dying stream's filer index and rotates it while holding m.mu, so it can never rotate a successor generation's filer by mistake. - The unary rename fallback bounds silence on its dedicated stream the way the mux does, instead of trusting Recv on a wedged filer. - downloadRemoteEntry opts out of the metadata-call bound: a healthy whole-file download can legitimately outlast it. * mount: count only wire silence against the rename timeout Applying a rename event can outlast the silence bound on a healthy filer — a directory rename migrates an open child's DLM lock under another writer. Stop the mux's response timer for the local work and re-arm it before the next wait; the fallback stream's watchdog defers while the handler runs instead of cancelling a live stream. * mount: restart the rename fallback's silence bound after each event The bound covers the wait for the next response, not the rename's whole lifetime — a healthy rename that keeps sending events can outlast it. Re-arm after each handled event; the defer-while-handling stays for local work that outlasts the interval on its own. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
aa4252d491 | docs: regenerate star history chart | ||
|
dependabot[bot]
|
02f2042360 |
build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.33.1 to 1.33.6 (#11665)
build(deps): bump github.com/aws/aws-sdk-go-v2/config Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.33.1 to 1.33.6. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.33.1...config/v1.33.6) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-version: 1.33.6 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
dd812413fe |
build(deps): bump github.com/jhump/protoreflect from 1.18.0 to 1.18.1 (#11667)
Bumps [github.com/jhump/protoreflect](https://github.com/jhump/protoreflect) from 1.18.0 to 1.18.1. - [Release notes](https://github.com/jhump/protoreflect/releases) - [Commits](https://github.com/jhump/protoreflect/compare/v1.18.0...v1.18.1) --- updated-dependencies: - dependency-name: github.com/jhump/protoreflect dependency-version: 1.18.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Zhe ChenandChris Lu
|
547befc070 |
s3api: preserve remote object ETags across cache fills (#11704)
* s3api: preserve remote object ETags across cache fills * s3api: align remote copy validators and reject stale ETags * s3api: keep remote ETag through metadata touches, drop it after resizes The remote-ETag check required Mtime to match RemoteMtime, so a utimens on a cold remote object returned an empty ETag until a later cache fill restored the remote value. It also trusted LastLocalSyncTsNs alone, so a setattr resize that leaves chunks and the stamp untouched kept serving the old remote ETag for different local bytes. Require FileSize to match RemoteSize for both the synced and remote-only paths and stop comparing modification times: content changes already surface through cleared sync stamps, chunks, or inline content, while a metadata-only touch leaves the remote bytes authoritative. Regression tests cover touched cold and cached entries and resized cached entries. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
dependabot[bot]
|
17ba8b3c7c |
build(deps): bump github.com/klauspost/compress from 1.19.2 to 1.20.1 (#11668)
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.19.2 to 1.20.1. - [Release notes](https://github.com/klauspost/compress/releases) - [Commits](https://github.com/klauspost/compress/compare/v1.19.2...v1.20.1) --- updated-dependencies: - dependency-name: github.com/klauspost/compress dependency-version: 1.20.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
homeofe
|
59db4ddc6c | build(deps): bump golang.org/x/net from 0.58.0 to 0.61.0 (#11705) | ||
|
Bharat Kaurav
|
e7ae0d6049 |
s3api: accept max-keys up to 2147483647 and cap list pages at 1000 (#11702)
* fix(s3api): accept max-keys up to 2147483647 and cap list pages at 1000 * add tests for ListObjects max-keys range and 1000 cap * refactor(s3api): return uint16 from parseMaxKeys to avoid narrowing casts |
||
|
|
fac021544e |
s3api: register the azurekms KMS provider under its build tag (#11691)
* s3api: register the azurekms KMS provider under its build tag weed/kms/azure registers itself with weed/kms from an init(), but the blank import that triggers it was commented out in auth_credentials.go (TODO: Fix Azure SDK compatibility issues). Nothing else imports the package, so building with -tags azurekms still ends with "KMS provider 'azure' not registered" at startup for any config using "type": "azure". Move the import into weed/s3api/kms_azure.go behind //go:build azurekms so default builds stay free of the Azure SDK, and drop the dead comment. Closes #11686 * ci: compile and test azurekms-gated code weed/kms/azure is excluded from every default build, so nothing in CI compiled it; that is how the provider shipped unregistered. Build the tree and run the kms tests with -tags azurekms on every Go change. --------- Co-authored-by: Yi-111-a <> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
448de42b05 |
worker: reject balance moves onto servers without the volume's disk type (#11695)
* worker: reject balance moves onto servers without the volume's disk type Proposals are bucketed by disk type at detection time, but queued or out-of-band jobs can still target a server that lacks the volume's disk. The copy then reads the source disk only to be rejected by the target's VolumeCopy, and the failure is retried on every scheduling cycle. Fetch the master's topology once per job and check each move's target against the volume's disk type before executing, so an incompatible move fails before any data is copied. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * worker: cover the balance disk-type guard Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * worker: match old-topology nodes in the balance disk-type guard Topology entries from older masters omit Address and carry a plain host:port Id, so checkMoveDiskType never matched grpc-suffixed move nodes and silently skipped the disk-type check. Also compare each node via pb.NewServerAddressFromDataNode, which reconstructs the same host:port.grpcPort form from Id and GrpcPort. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
d050464316 |
s3api: apply bucket default encryption when volume data encryption is enabled (#11681)
* s3api: apply bucket default encryption when volume data encryption is enabled When -s3.encryptVolumeData (s3a.cipher) is enabled, putToFiler skipped checking and applying bucket default encryption due to a '!s3a.cipher' guard. Volume-level data encryption and object-level Server-Side Encryption (SSE-S3 / SSE-KMS) operate at different layers, and explicit SSE headers already work alongside volume encryption. Remove the '!s3a.cipher' guard so PutObject without explicit SSE headers inherits bucket default encryption regardless of volume data encryption. Add regression test TestPutObjectAppliesBucketDefaultEncryptionWithVolumeCipher. Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com> * s3api: decrypt the volume cipher on direct SSE chunk reads fetchFullChunk, fetchChunkViewData, and createEncryptedChunkReader fetched raw bytes over HTTP, so volume-encrypted chunks reached SSE-S3/KMS/C decryptors still ciphered. Route them through fetchChunkData: encrypted or compressed chunks go through RetriedFetchChunkData (cipher-aware, slices plaintext space for views); plain chunks keep the streaming range read. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: cover cipher-aware chunk reads with a fake volume server The fake volume now serves stored GETs with Range support, and TestFetchChunkDataDecryptsVolumeCipher verifies full-chunk and view reads return plaintext for ciphered chunks while plain chunks still slice via HTTP ranges. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: guard fake volume server stored map with its mutex The HTTP handler goroutine read v.stored while test goroutines wrote it, a data race go test -race can flag. Lock v.mu around the map read and the test writes. --------- Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
c96eb7b8ee |
kms/azure: fix encrypt/decrypt round trip for Azure Key Vault (#11692)
* kms/azure: fix encrypt/decrypt round trip for Azure Key Vault The provider stored the wrapped data key as string(encryptResult.Result) in the JSON envelope, sent the encryption context as AAD to an RSA-OAEP key, and passed a full key URL to the azkeys client in the name position. Each of those breaks a round trip on its own. - split the key URL into the (name, version) pair azkeys expects before Encrypt, Decrypt and GetKey; a plain name still resolves to the latest version, and decrypt keeps the stored version so objects stay readable after a key rotation - store the wrapped key base64-encoded and decode it strictly, so the JSON envelope no longer replaces the raw bytes with U+FFFD - stop sending AAD: Key Vault rejects it on RSA-OAEP with BadParameter, so the encryption context is logged and dropped instead * kms/azure: reject a key URL that names another vault splitKeyID dropped the host of a full Key Vault URL, so a key ID from a different vault silently resolved to this vault's same-named key. Return an error instead of encrypting under a key the caller did not ask for. * kms/azure: bind encryption context via an envelope digest RSA-OAEP rejects AAD, so removing it left the encryption context unauthenticated: a wrapped key could be decrypted under a different object's context. Record a sha256 digest of the marshaled context in the envelope's provider_specific field on encrypt and verify it before calling Decrypt, restoring the binding without AAD. * kms/azure: treat an explicit :443 port as the same vault * kms/azure: accept an absent context digest only for empty contexts * kms/azure: normalize both hosts when comparing key URLs to the vault splitKeyID stripped :443 and a trailing dot from the configured vault but only :443 from the key URL, so a key URL naming the same vault with a trailing DNS dot was rejected before Azure was ever called. Compare both sides through vaultHost so they are normalized identically. * kms/azure: reject non-key vault URLs in splitKeyID, document digest limits A Key Vault URL that does not name a key under /keys/, has an empty key name, or carries extra path segments now fails fast instead of being passed to the client as a key name, where it would surface as a confusing vault-side error. Also note that the envelope context digest is a client-side mismatch check, not vault-authenticated AAD, and only allocate providerSpecific when a context is present. * ci: compile and test azurekms-gated code weed/kms/azure is excluded from every default build, so nothing in CI compiled it; that is how the provider shipped unregistered. Build the tree and run the kms tests with -tags azurekms on every Go change. --------- Co-authored-by: Yi-111-a <> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
b2eefb8a73 |
filer: fail the request, not the process, when chunks cannot be resolved (#11685)
* filer: fail the request, not the process, when chunks cannot be resolved CompactFileChunks and ViewFromChunks discarded the error from NonOverlappingVisibleIntervals and iterated the nil interval list it returns on failure. A CreateEntry or UpdateEntry whose context was cancelled therefore panicked in SeparateGarbageChunks and took the whole filer down, losing the metadata log it had not yet flushed. Both functions now return the error. cleanupChunks fails the request with it, the mount flush paths return it, the replication sinks and the S3 SSE range reader return it, and a ChunkStreamReader that cannot build its view fails every read with it and reports it as its SourceError instead of reading as an empty file. Fixes #11682 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * filer: propagate chunk resolution failures in scan paths NonOverlappingVisibleIntervals can fail without a cancelled context; the query engine and log-store callers discarded the error and dereferenced nil intervals. * query: propagate chunk resolution errors instead of skipping files When a manifest cannot be resolved, ReadParquetStatistics, countLiveLogRowsExcludingParquetSources, and computeLiveLogMinMax logged a warning and skipped the failed file, so queries returned understated counts and wrong min/max values. Surface the errors instead: the fast-path aggregation collector now returns a DataSourceError so the engine falls back to a full scan, and the row-count helpers return the error rather than a partial total. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Bharat Kaurav
|
774c9927a7 |
s3api: validate ListObjectVersions max-keys and honor max-keys=0 (#11703)
* fix(s3api): accept max-keys up to 2147483647 and cap list pages at 1000 * add tests for ListObjects max-keys range and 1000 cap * refactor(s3api): return uint16 from parseMaxKeys to avoid narrowing casts * fix(s3api): validate ListObjectVersions max-keys and honor max-keys=0 * add test for ListObjectVersions max-keys=0 * fix(s3api): widen uint16 max-keys for ListObjectVersions |
||
|
Chris LuandDevin
|
c3d13c60e9 |
filer: fail writes when the existing-entry lookup fails (#11699)
* filer: fail writes when the existing-entry lookup fails CreateEntry only returned a FindEntry error for exclusive creates; a regular write treated a transient store error as "entry not found" and, on upsert stores, replaced the entry without loading its old chunks — leaving them out of the cleanup process. Return any lookup error except ErrNotFound for both paths. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: test that a lookup failure fails a regular write Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
4bf5e92b64 |
storage: unify needle-map counters across index loaders and warn on copy-count drift (#11701)
* storage: count index-load metrics only for live-key removals The in-memory and leveldb offset loaders counted a deletion for every tombstone row replayed, including re-deletes of already-deleted keys (runtime logDelete only counts a live removal), and added the returned old size to the byte counters even when it was a tombstone sentinel — uint64(-1) wraps DeletionByteCounter. The leveldb offset loader also counted every index row as a file and its bytes unconditionally. Gate the counters the same way the runtime paths do so a reload reports the same numbers the runtime counters hold. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * storage: derive index metrics from latest-row state in the metric loader The leveldb metric loader walked the index in reverse and incremented FileCounter for every row (inflating FileCount with tombstones) and DeletionCounter for every superseded row plus every deleted-key row (double counting both). Per key, the runtime deleted count is exactly (valid rows) - (1 when the latest row is live), so count live-latest rows via the same bloom filter and derive both deletion counters; this matches the runtime logPut/logDelete counters exactly instead of drifting with tombstone history. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * server: warn instead of failing volume copies on counter drift checkCopyCounts compared the source's in-memory counters against the target's fresh-load counters, but counters computed by different loaders drifted apart for identical .idx data, so a good copy was rejected (issue #11678). The file-size checks already gate integrity; log the mismatch instead of failing the copy. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * storage: assert MaxFileKey parity in the metric loader test MaxFileKey derives from row order alone, so unlike the bloom-filtered deletion counters it must always match the runtime counters exactly. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Peter Dodd
|
5f1a742726 |
fix: return NoSuchKey and drop stale entries when a remote-mounted object is gone from the remote (#11680)
A remote-only entry whose object was deleted from the remote storage outside the filer answered GET with 500 and stayed in the filer. The remote's not-found was lost on the way: the backends' ReadFile returned it as an untyped error, so FetchAndWriteNeedle failed with codes.Unknown and nothing downstream could tell it from any other failure. - remote_storage: GCS, S3 (NoSuchKey) and Azure (BlobNotFound) reads return ErrRemoteObjectNotFound. GCS reports a missing bucket the same way as a missing object, so it confirms the bucket with a listing. - volume server and filer: the not-found crosses gRPC as codes.NotFound carrying the sentinel's text, and the filer's cache RPC returns codes.NotFound, which the S3 gateway already maps to NoSuchKey. - s3api: the origin fallback answers NoSuchKey on a confirmed not-found. - filer: a confirmed not-found removes the stale entry, so the lazy remote-metadata cache converges on the remote. Only remote-only files outside .versions and without an active object lock are removed, only if unchanged since the fetch (checked on the object's write owner, under the lock S3 object writes take), and with a metadata-only delete: the filer skips its inline remote delete, and the delete events' entries carry a marker that makes filer.remote.sync and filer.remote.gateway skip their remote delete, while filer.sync still replicates it. The replicated DeleteEntryRequest carries keep_remote_object, so the destination's delete events are marked too. The store drops the marker from every write, so clients cannot plant it. |
||
|
|
d3dc03c85a |
s3api: resolve volume data encryption in CopyObject SSE flows (#11646) (#11683)
* s3api: resolve volume data encryption in CopyObject SSE flows (#11646) * s3api: honor bucket-default KMS key on copy and fix transformed-upload metadata - Synthesize the destination bucket's default encryption as request headers before any SSE evaluation, so the configured KMS key ID and bucket-key setting reach the copy paths instead of only a boolean. - uploadTransformedChunkData returns the upload result so callers record the uploader's cipher key AND compression decision; a wrongly cleared IsCompressed made transformed copies unreadable. - decompressChunkVolumeCipher fails loudly when a compressed chunk does not decompress, instead of uploading still-compressed bytes marked uncompressed. - copyMultipartSSECChunk now strips the volume cipher and re-encrypts on upload like the other transform paths. - The ciphered inner upload now uses the caller's private BytesBuffer. * s3api: extract copy bucket-default header synthesis for coverage Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: test bucket-default encryption header synthesis on copy Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: validate copy encryption headers before bucket defaults and resolve empty KMS key * s3api: name the AWS-managed SSE-KMS default key once --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
ea65746947 |
fix(filer): end metadata subscriptions before gRPC GracefulStop on shutdown (#11663)
* fix(filer): end metadata subscriptions before gRPC GracefulStop on shutdown GracefulStop waits for every open stream. The filer's own MetaAggregator subscription and, under -s3, the S3 gateway's IAM subscription live in the same process and only end when the filer shuts down, which happens after GracefulStop - so every shutdown sat out the full 15s timeout. StopSubscriptions ends them (and any started later) right after leaving the lock ring; the handlers derive their context from the stream and this signal. Measured on weed server -s3: stop 25.6s -> 10.7s, and 0.6s with -volume.preStopSeconds=0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FTq6bDpgdQfqQagQsUUvdw * fix(filer): end stopped subscriptions with Unavailable, not cleanly A subscription that StopSubscriptions cut short returned nil. The client reads that as io.EOF, "caught up, done", and util.RetryUntil - which the S3 gateway and mount follow with - stops on nil. A separate S3 gateway therefore never resubscribed after its filer restarted (measured: 0 subscriptions in 25s after the restart; upstream master and this fix: it is back within a second). endOfSubscription now turns that clean end into codes.Unavailable, which is what a dropped connection looks like, so followers reconnect. Errors pass through, and so does the end of a stream the client closed. A subscriber that has stopped reading is still left to GracefulStop's timeout: its handler sits in stream.Send, which only the end of the stream releases, and grpc-go does not allow Send after the handler returns. Documented at StopSubscriptions. Shutdown under weed server -s3 is unchanged at 10.7s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FTq6bDpgdQfqQagQsUUvdw * fix(filer): interrupt in-flight subscription reads and end them Unavailable StopSubscriptions only reached the wait points: a handler replaying the ring or the persisted log kept streaming until the pass ended, and a pass cut short at a ctx.Err() boundary (the replay semaphore, chunk ref collection, ref sends) returned a wrapped context.Canceled, which reads as codes.Canceled on the wire - a code transient-error classifiers treat as non-retryable. eachLogEntryFn and sendRefsBatched now check the subscription context per entry/batch, and endOfSubscription reports a context.Canceled cut short by the stop signal as codes.Unavailable, same as a clean end it interrupted. * filer: honor the subscription context in the pipelined sender A subscriber that stops reading wedges sendLoop in stream.Send; Send and Close then blocked past StopSubscriptions, so GracefulStop still waited out its timeout. Both now return when the subscription context ends, letting the handler return and gRPC tear the stream down. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: stop sendLoop from starting sends after the subscription ends With Send and Close released by the subscription context, sendLoop could still issue a stream.Send after the handler returned. Gate every send on the context so at most the in-flight Send overlaps teardown - and the stream's own teardown is what unblocks it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: assert sendLoop exits in the cancel test Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
c023493cf0 |
s3api: tolerate session tokens on statically configured credentials (#11694)
* s3api: consolidate session token extraction into extractSessionToken Three call sites duplicated the same header/header/query lookup; share one helper named after the existing s3tables equivalent. * s3api: tolerate session tokens on statically configured credentials Credential vendors like Unity Catalog emit a session token with every vended credential, including static ones (UC's StaticAwsCredentialGenerator only engages when s3.sessionToken is set). Requests signed by a configured access key were routed to STS validation and rejected, so static credential vending never worked against SeaweedFS. Resolve the access key first: when it maps to a configured credential the signature alone authenticates the request, and the attached token is marked ignored so authorization does not route it into the STS session-policy path. STS-issued access keys are never in the static map, so temporary credentials still validate their token exactly as before. * s3api: cover static credentials carrying a session token * test: exercise UC static credential vending against SeaweedFS s3.sessionToken.0 selects UC's StaticAwsCredentialGenerator, which vends the configured keys verbatim. The vended session token is foreign to SeaweedFS and previously failed SigV4; now it round-trips through temporary-table-credentials into real S3 I/O. * s3api: reject temporary credentials in GetFederationToken after auth Token presence alone cannot distinguish a temporary credential from a statically configured one carrying a vended token. Move the check behind verifyV4Signature and key it on the operative session token so tolerated tokens keep the caller eligible. * s3api: test GetFederationToken with authenticated temporary credentials Sign the rejection cases with real session credentials so they reach the post-auth check, and cover a vended static credential being accepted. * test: check vended-credential delete error in UC integration test |
||
|
Eliah RusinandChris Lu
|
924a683397 |
volume server: answer chunk-manifest reads from the sum of chunk sizes (#11628)
* volume: answer chunk-manifest HEAD from the chunk-size sum * volume: size chunk-manifest reads by the sum of chunk sizes * volume: saturate the sum of chunk-manifest Range lengths * volume: fetch remote chunk-manifest ranges with a bounded Range * volume: fmt chunk-manifest handlers * volume: cap whole-window chunk reads at the visible window * volume: reject an over-cap chunk span and an illegal manifest MIME * volume: take the manifest transform extension from the URL first Go's tryHandleChunkedFile keeps the extension parsed from the URL — including a fid suffix like /vid,fid.png — and only falls back to the resolved filename's extension when the URL has none. The manifest path now does the same, so a transform requested on a fid-suffixed URL runs as it does in Go. Also: compute the chunk-size sum once, drop an unused remote_read_for parameter, share the crop/resize predicate with parse_read_request, and settle a couple of clippy lints. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Paolo Valletta
|
4d306e1279 |
s3api: revoke the identities a config reload no longer declares (#11679)
* s3api: revoke the identities a config reload no longer declares A static config file reload merges with isFullState=false, so an identity deleted from -s3.config kept authenticating with all of its access keys until the process restarted. An operator revoking a leaked key got no error and no revocation. The file is the source of truth for the identities it declares, so a reload now drops the ones that have left it, together with their access keys and those of their service accounts. staticIdentityNames was only ever added to, which kept a removed name protected as well; the names the file itself declares are tracked separately from the AWS environment credentials, so a reload never revokes what the file never declared. * s3api: keep a file reload from replacing the dynamic store Removing the last identity from a config file emptied staticIdentityNames, so useStaticConfig turned false and the next reload of that file took the replace path: every filer-managed identity and its access keys disappeared until a dynamic reload brought them back. A static config file is authoritative for the identities it declares and never for the dynamic store, so a file load now always merges. The first file load at startup takes the merge path as well, from an empty state. Reported by the Devin and Greptile reviews of this PR. TestReloadStaticConfigWithoutIdentitiesKeepsDynamic reloads an emptied file twice and asserts that a filer-managed identity and its access key survive both; the existing test now also checks that the service account key was loaded before the reload and that the environment identity's key still works after it. * s3api: clear the environment credentials in the emptied-file test The AWS environment identity is static and is re-added after every merge, so on a runner that has AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set it kept hasStaticConfig true once the file was emptied, and the next reload still took the merge path: the regression this test guards passed unnoticed. Clear both variables whatever the runner has set. With the pre-fix condition (hasStaticConfig alone) and the variables present in the environment, the test now fails with "reload 2: a filer-managed identity must survive a reload of an emptied file". Reported by the Greptile review of this PR. |
||
|
github-actions[bot]
|
c4f2f514fa | docs: regenerate star history chart | ||
|
Chris LuandDevin
|
5d80ac39b8 |
s3api: verify under the write lock before re-committing an ambiguous routed PUT (#11649)
* s3api: verify under the write lock before re-committing an ambiguous routed PUT A routed PUT whose response was lost after the owner committed, or whose transaction returned a store error, fell straight into the lock path and re-sent the same entry. A concurrent PUT that had superseded the commit in the meantime had already deleted this entry's chunks as old, so the re-commit restored metadata pointing at dead needles and the object read back 404 permanently. The lock path now resolves the routed attempt's outcome inside the write lock first: a stored entry with the uploaded chunks means the route landed; a different stored entry or an unresolved lookup refuses the re-commit with ServiceUnavailable so the client retries with a fresh upload; only a proven-absent entry falls through to the normal create. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: tighten ambiguous routed PUT recovery - a match found only after an unanswered filer is not authoritative; the unreachable filer may hold a newer entry, so refuse instead of finalizing on it (both recovery paths) - a failed post-recovery finalization now rolls the recovered entry back, matching the create path's undo - a proven-absent entry is only re-committed after probing that the uploaded chunks' needles are still alive; a committed-and-deleted PUT would otherwise write back an entry pointing at reclaimed needles - the .versions latest pointer no longer flips back to an older version when a newer one committed while the write was uncertain Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: stamp late versions noncurrent when a newer latest pointer wins The keep-newer early return in updateLatestVersionInDirectory left the version just stored without ExtNoncurrentSinceNsKey, so the lifecycle engine could never age it out. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: only a failure past mutation 0 makes a routed PUT ambiguous A deterministic refusal at the PUT mutation (e.g. "existing entry is a directory") applied nothing, but marking it ambiguous sent the lock-path fallback through conservative recovery, which found the directory entry and refused with 503 instead of the correct 409. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: keep all transaction errors ambiguous; route directory conflicts through the lock path * s3: verify uploaded chunks before re-committing over a directory A stored directory does not prove the routed PUT never committed: the route can land, a delete can remove the entry and its chunks, and a nested write can recreate the directory before recovery takes the lock. Re-committing then stores an entry pointing at dead needles. Probe the uploaded chunks first and refuse with ServiceUnavailable when they can no longer be verified. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
e337443176 |
filer: cut ReaderCache mutex contention on small reads (#11677)
* filer: skip cache lock when a cacher cannot be removed yet SingleChunkCacher.readChunkAt defers removeConsumed on every read, and every unpin retries removal too. Each call acquired the ReaderCache mutex even though the removal conditions are plain atomics, so a busy cache paid a lock acquisition per read just to discover readers>0. Check the atomics before locking: when a cacher is not yet consumable the removal is impossible and the lock round trip is pure contention. Removals still run under the lock via removeConsumedLocked, so the attach-vs-remove race keeps its existing serialization. Ref #11676 * filer: guard stream position with a per-stream mutex, not the cache lock chunkStream.cacher is only ever shared by concurrent ReadAt calls on one ChunkReadAt, yet pin, unpin, releaseIfFinished, and releaseStream all mutated it under the ReaderCache mutex that serializes every reader in the process. Each read that switched chunks took that global lock to pin the new chunk, released it, then reacquired it in unpin() just to drop the old chunk's pin and retry removal; releaseIfFinished and releaseStream did the same lock-detach-unlock-relock dance. At high small-GET concurrency that turned per-request stream bookkeeping into a global convoy (#11676). Give chunkStream its own mutex and drop the cache lock from the pin lifecycle entirely: pin/detach serialize on stream.mu, the pin counter and consumable checks are already atomics, and removeConsumed only takes the cache lock when a cacher is actually removable. The map lock now guards only map membership and read registration. * filer: look up cached chunks under the read lock readChunkAt serialized every small read on the write lock even though the common paths are read-only: an existing downloader just needs its read registered, and a chunkCache hit needs no map access at all. Each GET also paid the lock a second time to reach the chunkCache check. Use the read lock for the downloader lookup and registration; the registered read keeps the buffer alive against a concurrent destroy, and only error eviction, insertion, and removal need the write lock. The chunkCache probe runs lock-free, and the insert path re-checks the map under the write lock to cover a downloader registered in between. * filer: start the chunk download outside the map lock The insert path held the ReaderCache write lock across goroutine spawn and the cacheStartedCh handshake, so every downloader miss serialized against the startup of a fetch goroutine. Register the cacher in the map under the lock, then start the download after releasing it; a fetch that fails early still lands in the map and is evicted by the next reader's completed-error check. * filer: test that stream pin lifecycle stays off the cache lock Regression coverage for the contention fix: pin and releaseStream on a non-removable cacher must complete while the ReaderCache lock is held by another goroutine. * filer: pin the stream under the map lock Between read registration and stream.pin the cacher showed zero pins, so a budget eviction in that gap could pick a chunk the stream was just attaching to and the stream's next slice refetched it. The pin counter is an atomic and stream.mu is never held while acquiring the cache lock, so pinning inside the map hold is deadlock-free and closes the window. |
||
|
|
838c554e33 |
s3api: reject SSE headers PutObject cannot honor, as CopyObject does (#11642)
* s3api: reject SSE headers PutObject cannot honor, as CopyObject does PutObject and CreateMultipartUpload did not check the server-side encryption headers they were given. An x-amz-server-side-encryption value that names no method, such as "aes:kms", matched none of the SSE paths, so the object was stored unencrypted and the request answered 200. SSE-C together with x-amz-server-side-encryption was also accepted, and one of the two silently won. CopyObject already rejects both through validateEncryptionCompatibility. Run the same check, with the same error codes, before PutObject and CreateMultipartUpload store anything. * s3api: answer rejected SSE headers with InvalidArgument, as S3 does S3 rejects an unknown x-amz-server-side-encryption value and SSE-C combined with another method with InvalidArgument. PutObject and CreateMultipartUpload now return that code, with S3's messages, through two new error codes; CopyObject keeps its own. Also run ceph/s3-tests' test_put_obj_enc_conflict_c_s3, _c_kms and _bad_enc_kms in CI, which check both handlers' responses end to end. * s3api: close the remaining ways a PUT could skip requested encryption - A repeated x-amz-server-side-encryption header is rejected: the encryption paths apply only the first value, so extra values could hide the method the client asked for. - KMS options (key id, encryption context, bucket key) are rejected unless the method is aws:kms, matching the S3 InvalidArgument error. - CreateMultipartUpload validates before auto-create, so a refused upload cannot leave a bucket behind. - Directory markers encrypt their inline content through the shared SSE path and record the same entry metadata as regular objects, instead of storing requested-encrypted bytes in plaintext. * s3api: read back what the SSE marker write stores - Repeated SSE-C and KMS option headers are rejected alongside a repeated x-amz-server-side-encryption, closing the same first-value bypass for customer-key and KMS fields. - Algorithm validity is checked before KMS options so an unsupported value keeps the "not supported" error. - serveDirectoryContent decrypts marker content with the stored SSE metadata and returns the SSE headers, so an encrypted marker reads back what was written; its Content-Length now reflects the bytes actually served. * s3api: HEAD of a marker skips decryption and keeps the stored size HEAD returns no body, so it now runs only the SSE-C key check instead of decrypting — matching HeadObjectHandler and avoiding a KMS round-trip — and chunk-backed directory entries report Attributes.FileSize again rather than the length of their (empty) inline content. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: keep GET Content-Length to the bytes serveDirectoryContent writes The FileSize override described chunk-backed markers on HEAD, but GET sends only the inline content, so it promised bytes it never wrote. * s3api: stream a chunk-backed directory on GET like any object A directory promoted over an uploaded object keeps the object chunks with empty inline content, so serving only entry.Content made GET deliver nothing while HEAD reported FileSize. GET now routes those entries through the regular volume-server stream, keeping the two methods consistent. * s3api: answer a busy volume read with RequestBytesExceed --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |