mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-18 20:18:27 +02:00
446e6f3444e8cf9099f69763aa59c5b262fb2e17
15510
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandDevin
|
446e6f3444 |
s3tables: scope OIDC-derived account IDs to the token issuer (#11774)
* s3tables: scope OIDC-derived account IDs to the token issuer Table ownership keyed off the bare sub claim, so two identity providers issuing the same sub collided on one account - a caller could act on tables owned by a different provider's user. OIDC bearer tokens now carry the same issuer-namespaced (sub, iss) hash STS sessions already expose as aws:userid, and getAccountID prefers it over the raw claims. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: keep legacy owners working and scrub client-subject claims - OIDC bearer tokens without a subject are rejected, and a token-supplied aws:userid claim is always overwritten with the verified (sub, iss) hash, so a forged claim can no longer stand in for the principal. - Owners written before issuer scoping hold a bare subject; permission checks now re-anchor that subject under the caller's issuer, so the same (sub, iss) pair keeps its ownership while scoped owners remain issuer-isolated. The token issuer is asserted into claims alongside sub and carried through PolicyContext. * s3tables: anchor legacy subjects to the configured issuer set * s3: enumerate trusted issuers from registered providers too * s3: fail closed when the OIDC provider store cannot be listed trustedOIDCIssuers kept the registered issuers when ListOIDCProviders failed, presenting a partial set as complete. A deployment trusting one registered issuer A plus a stored issuer B would, on a listing error, report only A — the single-issuer shape under which a legacy bare subject re-anchors — letting a caller from A claim a stored subject that belongs to B and gain owner access. With no provider store the registered providers are provably complete, so keep returning them; with a store, a failed listing means the real set is unknown, so return none. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
348116b2c4 |
filer: sweep the local subtree on a replayed directory delete (#11785)
* filer: sweep the local subtree on a replayed directory delete A recursive delete emits only the directory's own key, so peers applied it with a single DeleteEntry. Children the peer held that the origin had not listed stayed behind unlisted but still readable, and their needles were never deleted. Sweep the local children on a replayed directory delete and drop their chunks when the origin deleted chunks. * filer: walk the subtree on a replayed directory delete DeleteFolderChildren removes only direct children in stores keyed by immediate parent, so the sweep now lists and deletes each descendant entry itself. Entries the local filer committed after the event survive — with the directories that hold them — and hard-linked files keep their chunks for the origin's own events to settle. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: key replayed subtree chunk deletes to each entry's path The sweep queued every descendant's chunks under the deleted directory's storage rule and skipped directory entries entirely. Chunks now collect keyed by their owning path, so per-path rules apply to each file — and a directory carrying its own chunks (a file promoted by an S3 child) frees them too. * filer: fence kept replayed dirs and flush collected chunks on error A directory kept because a newer descendant survived still needs its delete timestamp recorded, or a stale replayed create can resurrect another child under it. And chunks collected for deleted children must be flushed even when the sweep fails partway — the entries are gone and a retry cannot list them again. Collection now also runs only after the entry delete succeeds. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: flush swept chunk references in batches --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
5b4a707c43 |
shell: keep volume.tier.compact replicas on the same remote object (#11783)
* shell: keep volume.tier.compact replicas on the same remote object The compact cycle ran on a single replica: it deleted the shared remote object up front and never touched the siblings, leaving their .vif/.idx pointing at a removed object. It also remounted the working replica into a writable local volume mid-operation. Freeze all replicas first, keep the shared object while siblings still reference it, replicate the new metadata to every replica, and let the last tiered replica retire the old object. * storage: surface remote object delete failures from volume destroy Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: keep tier.compact replicas frozen until every copy lands A delete between the source unfreeze and a sibling's index copy could land past the copied snapshot, leaving the replica serving deleted data. Unfreeze all replicas only after every resync completes; a remote volume's readonly mark is inert, so the freeze adds no extra cost. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * storage: abort volume destroy when remote delete fails destroyLocked removed local files even when the backend delete errored, leaving a registered replica with no .dat/.idx and a leaked remote object. Run the remote delete before local cleanup and bail out on failure so the volume stays fully usable and the delete can be retried. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: mirror destroy ordering for remote delete failures Propagate a failed remote-object delete instead of warning past it, and reinsert the volume into the disk location registry so a still-intact replica is not unregistered. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: recover partial tier.compact freezes and report held replicas A failed freeze left already-marked replicas readonly with nothing changed, and later stage failures returned without saying the replicas stay frozen or how to resume. Roll back the partial mark when freezing fails, note in every mid-flight error that replicas remain readonly and rerun is the recovery path, and report how many replicas a partial unfreeze released. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: retire abandoned remote objects during tier.compact resync A resync overwrote each replica's .vif, orphaning its previous remote object; an interrupted run could leave replicas on different keys, and only the last tiered replica's object was deleted. Read every replica's remote file list up front, count holders per object, and delete-before- copy whichever replica is an object's last holder, so shared and stranded objects alike are retired instead of leaked. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: keep collection members registered when destroy fails --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
d3a97f76bd |
s3: fix PutObject If-Match under suspended versioning (#11780)
* s3: keep suspended-versioning conditional writes off the routed condition A suspended PUT writes the null-version slot at the regular path while the entry the conditional headers evaluate lives under .versions/. The routed WriteCondition checked the request's etag against that empty slot, so a correct If-Match failed with 412. Passing uniqueWritePath sends conditional requests through the lock path, whose precondition resolves the same entry the read path serves; unconditional writes still route. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer_pb: add condition_alt_key to ObjectTransactionRequest * filer: evaluate a transaction condition against a fallback entry condition_alt_key names the entry consulted when the primary is absent, so a suspended-versioning write can check 'the null slot if present, else the latest version' atomically under the lock. * s3: route suspended-versioning conditional writes with a fallback key The gateway names the latest version file as condition_alt_key, so the transaction checks the entry the read path serves — null slot first, latest version when it is absent — under the same lock that applies the write. Conditional suspended PUTs route again instead of splitting check and write across a gateway lock that routed writes skip. * filer: resolve the condition fallback under the transaction lock The gateway named the pointed-at version file as condition_alt_key, resolved before the transaction ran: a concurrent version delete could move the latest pointer between the gateway read and the locked check, and a delete-marker latest read as an existing object. The alt key is now the .versions directory and the filer resolves the latest-version pointer under the same lock that serializes the write — a delete marker resolves to absent, and unresolvable pointer state fails closed. * filer: fail closed on alt-key lookups and skip directory primaries A transient FindEntry error returning a nil entry read as "versions dir absent" and let If-None-Match writes pass. Check the error before the entry. And a bare directory at the object path satisfied the primary lookup, so the .versions fallback never ran — directories are not the object, treat them as an absent slot. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: scope directory-as-absent conditions to alt-key callers --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
cdc6976e3a | docs: regenerate star history chart | ||
|
Chris Lu
|
921535411b |
s3: default allowDeleteBucketNotEmpty to false (#11769)
* s3: default allowDeleteBucketNotEmpty to false DeleteBucket on a non-empty bucket silently destroyed every object, while AWS returns 409 BucketNotEmpty. Clients like 'aws s3 rb' rely on that refusal as the 'delete only if empty' safety net. Deployments that need the recursive delete can opt back in with -allowDeleteBucketNotEmpty=true. * test: pin allowDeleteBucketNotEmpty=true on versioning config1 The config1 run relied on the flag's old true default to contrast with config2's false. Pin it so the two modes stay distinct. * test: drop redundant allowDeleteBucketNotEmpty=false in s3 integration test False is the default now, so TestS3DeleteBucketNotEmpty exercises the out-of-box behavior. |
||
|
Chris LuandDevin
|
651546ae74 |
mount: release flock locks before the close-time flush (#11726)
* mount: release flock locks before the close-time flush Release ran the dirty-handle flush before dropping the file description's flock locks. A flush parked on a dead filer kept a dead holder's lock in the table, so a restarted process taking flock on the same file saw "file already locked" until the mount restarted. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: finish the blocked-flush release test deterministically Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: let the release test's parked flush talk to a live filer Marking the handle deleted raced with the flush's unlocked early read of isDeleted. Pointing the fixture at a live fake filer lets the flush complete on its own once the handle lock frees, no flag write needed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
dependabot[bot]andChris Lu
|
82f0a37efe |
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.8.0 to 1.8.2 (#11740)
* build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.8.0 to 1.8.2. - [Release notes](https://github.com/Azure/azure-sdk-for-go/releases) - [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.8.0...sdk/azidentity/v1.8.2) --- updated-dependencies: - dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob dependency-version: 1.8.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * test: restore loadBucketMetadataFromFiler stub after TestGetBucketMetadata TestGetBucketMetadata replaced the package-level loadBucketMetadataFromFiler with a stub that sleeps one second per load and never restored it. Every test running after it in the same binary paid that sleep on each bucket metadata cache miss: TestPutObjectUploadACL (~245s) and TestCopyObjectDestinationACL (~46s) each created a fresh BucketRegistry per subtest, so the stub fired hundreds of times. In CI this stretched the Test job past 11 minutes and left it exposed to runner cancellation. Restore the real loader via t.Cleanup so the slow stub is scoped to the test that needs it. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
dependabot[bot]
|
9aae0bb702 |
build(deps): bump helm/kind-action from 1.15.0 to 1.15.1 (#11747)
Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.15.0 to 1.15.1. - [Release notes](https://github.com/helm/kind-action/releases) - [Commits](https://github.com/helm/kind-action/compare/v1.15.0...v1.15.1) --- updated-dependencies: - dependency-name: helm/kind-action dependency-version: 1.15.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
80060e1bd6 |
build(deps): bump github.com/a-h/templ from 0.3.1020 to 0.3.1070 (#11742)
Bumps [github.com/a-h/templ](https://github.com/a-h/templ) from 0.3.1020 to 0.3.1070. - [Release notes](https://github.com/a-h/templ/releases) - [Commits](https://github.com/a-h/templ/compare/v0.3.1020...v0.3.1070) --- updated-dependencies: - dependency-name: github.com/a-h/templ dependency-version: 0.3.1070 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
72aa17ff02 |
build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.47.1 to 1.47.3 (#11741)
Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.47.1 to 1.47.3. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.47.1...v1.47.3) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2 dependency-version: 1.47.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
e6cf190596 |
ec: re-plan balance moves when a target reports no space left (#11759)
* ecbalancer: add Node.SetNoReceive to exclude a node as a move target A node that has refused a shard copy at runtime (e.g. 'no space left' on the actual disk, issue #11753) can no longer be trusted as a destination for the remainder of a balance run, but it may still hold shards that must move away. SetNoReceive excludes the node from target selection and from receiving-side capacity accounting, while leaving it eligible as a move source. * ec: re-plan balance moves when a target reports no space left Moves are planned from a heartbeat snapshot, but concurrent copies let the volume server's real disk state diverge from it, so a planned target can refuse the copy with 'no space left' mid-run and abort the whole balance (issue #11753). Track the refusing targets during execution, mark them no-receive, re-collect the topology, and plan the remaining moves again. The loop terminates because a marked node can never be picked as a target again. * volume server: return ResourceExhausted when no disk can take an EC shard A failed EcShardsCopy carried code=Unknown even though the cause is exhausted local capacity; ResourceExhausted describes the refusal and lets clients distinguish it from internal errors. The message text is kept so message-based matching still works (the Rust volume server reports the same text under Internal). * ec: replay applied moves onto the re-collected snapshot and surface failures Addresses review on the re-plan loop: a fresh topology read can still describe an applied move while heartbeats lag, so journal each applied move and replay it onto the next snapshot; retry only when every failure was a 'no space left' refusal (an unrelated failure aborts instead of being hidden); and exclude refused nodes' slots from the free-space precheck so a balance with no remaining receiver fails instead of reporting success for an empty plan. * ec: carry per-disk state when replaying applied moves A move that landed on an explicit TargetDisk loses that attribution when replayed onto a stale snapshot, leaving the disk's free slots and shard count unchanged; the next plan could pick the same disk, hit another refusal, and over-exclude the node. Replay the shard onto the target disk's entry and move the disk-level counters, and restore the source disk's counters on removal. * ecbalancer: add Node.SetDiskNoReceive to exclude a single disk as a target A copy with an explicit disk_id can be refused while the node's other disks still have room; a disk_id 0 refusal already means the server searched every disk, so the scopes differ. disk.noReceive removes that disk from receiving capacity and disk selection while leaving the node and its sibling disks eligible. * ec: record explicit-disk refusals as disk-scoped no-receive recordMoveResult now distinguishes the two refusal scopes: TargetDisk>0 marks only (node, disk), while TargetDisk==0 marks the whole node since the server's auto-pick found no usable disk. The free-slot precheck subtracts refused disk slots, and the replan log names refused destinations as node:diskN. * ecbalancer: skip no-receive disks in pickBestDiskOnNode The balance phases pick the destination disk through pickBestDiskOnNode, which ranked disks by score only; a refused disk with a low shard count would be re-picked, fail again, and abort the run since no new destination was rejected. * volume_move: export IsNoSpaceLeftError for move callers Both the ec.balance replanner and the worker ec_balance task need to recognize the volume server's no-space refusal; keep one matcher next to the move RPCs instead of a private copy in weed/ec. * worker: retry pinned-disk EC moves with a server-picked disk on no space ec_balance detection always pins TargetDisk, so a full disk fails the job even when a sibling disk has room. On a no-space refusal to the pinned disk, retry the move once with TargetDisk 0 so the volume server picks a disk on the same node; the planner's node choice stands. |
||
|
Youngju KIMandChris Lu
|
ee2d98f664 |
mini: note mini.options and Docker filer.toml pitfalls in help (#11755)
* mini: note mini.options and Docker filer.toml pitfalls in help Two behaviors of "weed mini" surprise users and are not mentioned in its help text. Flags are saved to <dir>/mini.options and applied again on the next start to every flag that still has its default value. Removing a flag, or passing it with its default value, therefore keeps the saved value. The official Docker image ships /etc/seaweedfs/filer.toml, which sets the leveldb2 store to /data/filerldb2. When filer.toml is found, the filer ignores the default store dir derived from -dir, so moving -dir away from /data leaves the metadata in /data. If /data is not persisted, buckets look empty after the container is recreated. Only the help text changes. Signed-off-by: YoungjuKim <fjvbn2003@gmail.com> * mini: update help notes for computed-only mini.options --------- Signed-off-by: YoungjuKim <fjvbn2003@gmail.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
18b235da0d |
mini: surface saved options and warn when filer store escapes -dir (#11758)
mini: persist only computed options, and warn when filer store escapes -dir - saveMiniConfiguration no longer writes command-line flags to mini.options; the file now only pins values mini computed itself (currently master.volumeSizeLimitMB), so a dropped flag can no longer resurrect on the next start. Existing files still apply once and are then rewritten without the saved flags. - applyConfigFileOptions skips any flag explicitly provided on the command line or via WEED_* env vars, so passing a flag always wins over a saved value, including at its default value. - mini prints the saved options it applies at startup (secret values redacted), since quietMiniLogs hides the previous V(1)/V(2) messages. - warn once at startup when a filer.toml points a local filer store (leveldb/leveldb2/leveldb3, including path-specific <store>.<id> sections) outside -dir and -master.dir: the filer skips the -dir-derived store dir whenever a filer.toml exists, and the metadata is lost on container recreate if that path is not persisted. |
||
|
github-actions[bot]
|
4c81015bd3 | docs: regenerate star history chart | ||
|
Chris Lu
|
a5ccefb661 |
filer: escape LIKE wildcards in SQL list prefixes (#11757)
* filer: escape LIKE wildcards in SQL list prefixes ListDirectoryPrefixedEntries bound prefix+"%" straight into a LIKE clause, so '_' and '%' in object names (e.g. Langfuse keys like call_03_...) acted as wildcards: wrong results and a ~40x slower range scan because (dirhash, name) can no longer be narrowed. Escape '!'/'%'/'_' in the prefix and declare ESCAPE '!' in the generated list SQL so every abstract_sql backend (mysql, mysql2, postgres, postgres2, sqlite) matches literally with identical semantics. '!' is used instead of '\' because SQLite shares SqlGenMysql but would reject the MySQL-style '\\' literal, and '\' would still need escaping itself. * filer: escape LIKE wildcards in YDB list prefixes Same wildcard-in-prefix bug as the abstract_sql backends: YQL LIKE treated '_' and '%' in the prefix as wildcards. Apply the shared LikePattern helper and declare ESCAPE '!' (YQL allows any escape char except '%', '_' and '\'). * test: cover literal '_'/'%'/'!' prefixes in sqlite filer listing End-to-end regression test through SqliteStore against a real database; runs in CI under the sqlite build tag. |
||
|
|
103328e70f |
filer: create parent directories for lazily fetched remote entries (#11727)
* filer: create parent directories for lazily fetched remote entries A remote-mounted object first read through the filer was persisted without its parent directories, so remote.uncache and the remote cache evictor, which walk down from the mount directory, never reached it: its cached chunks could not be evicted. Persist the missing parents with the entry. With those directories in place, the empty-folder cleaner leaves folders under a remote mount alone: deleting one removes its whole prefix on the remote, which can still hold objects that were never read locally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: keep remote objects at parent keys and remote prefixes of empty folders Look up a lazily fetched entry's missing parents through lazy fetch before creating them, so a remote object stored at a parent's key is materialized and promoted to a prefix object rather than hidden behind a new directory. Deleting a folder that is empty in the filer is metadata-only under a remote mount, both for a non-recursive delete and for the expiry cleanup of empty parents: the mount is populated lazily, so the folder can still hold remote objects never read, and removing its remote directory deletes every object under that prefix. This replaces the empty-folder cleaner's own guard, which covered only one of those paths. A metadata-only delete records no remote-deletion tombstone, locally or from its metadata event: nothing is owed to the remote, so lazy reads stay open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: stop the lazy parent lookup at a store error Only a parent the store reports as not found is missing; a parent it cannot read ends the walk, so the lookup does not stat ancestors above it on the remote. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: stat each missing remote parent once A lazy fetch that finds k missing parents asked FindEntry for each, and every nested fetch re-walked its own still-missing ancestors, re-stating remote keys already proven absent: 1+k(k+1)/2 remote stats for one first read. The flagged parent lookups now skip their own walk; their entries are created by the same CreateEntry that already persists the chain. * filer: materialize parents for lazily listed remote entries maybeLazyListFromRemote persisted children and the listing directory with skipCreateParentDir, so a first access that goes through the list path still left entries unreachable to a walk from the mount directory. Materialize the missing ancestors the same way the lazy fetch does, and let the synced_at directory create its parent chain. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Jay HemnaniandChris Lu
|
1562621c5e |
filer: record the size of a multipart upload saved inside the entry (#11734)
* filer: record the size of a multipart upload saved inside the entry * filer: guard save-inside multipart append and check Content-MD5 Addressing review feedback: - Reject a save-inside multipart append when the target is chunk backed. The buffered content cannot be attached to existing chunks, so previously the request grew FileSize without storing the appended bytes, and reads returned zeros for the added range. - Compare the request Content-MD5 against the buffered content before saving, matching the regular multipart path, so a bad digest no longer commits the file. - Extend the size test to check the saved entry MD5, and cover the rejected append against chunk-backed and inline targets plus the bad-digest rejection. * filer: return 400 for unsupported append modes The save-inside append rejection and the existing inline append and offset write rejections are deterministic client errors, but the autoChunk default case mapped them to 500. Give them shared constants and return 400 instead, and assert the status in the tests. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Jay Hemnani
|
3659c369e0 | webdav: read small files stored inline in the entry (#11733) | ||
|
Robin Tweedie
|
e12ec94747 | fix(shell): compare fractional evacuation destination ratios (#11752) | ||
|
dependabot[bot]
|
9e56651808 |
build(deps): bump github.com/go-redsync/redsync/v4 from 4.17.0 to 4.18.0 (#11743)
Bumps [github.com/go-redsync/redsync/v4](https://github.com/go-redsync/redsync) from 4.17.0 to 4.18.0. - [Commits](https://github.com/go-redsync/redsync/compare/v4.17.0...v4.18.0) --- updated-dependencies: - dependency-name: github.com/go-redsync/redsync/v4 dependency-version: 4.18.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
d167521270 |
build(deps): bump helm/chart-testing-action from 2.8.0 to 2.9.0 (#11745)
Bumps [helm/chart-testing-action](https://github.com/helm/chart-testing-action) from 2.8.0 to 2.9.0. - [Release notes](https://github.com/helm/chart-testing-action/releases) - [Commits](https://github.com/helm/chart-testing-action/compare/v2.8.0...v2.9.0) --- updated-dependencies: - dependency-name: helm/chart-testing-action dependency-version: 2.9.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
8608a4e75d |
build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.151.1 to 3.155.1 (#11739)
build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 Bumps [github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) from 3.151.1 to 3.155.1. - [Release notes](https://github.com/ydb-platform/ydb-go-sdk/releases) - [Changelog](https://github.com/ydb-platform/ydb-go-sdk/blob/master/CHANGELOG.md) - [Commits](https://github.com/ydb-platform/ydb-go-sdk/compare/v3.151.1...v3.155.1) --- updated-dependencies: - dependency-name: github.com/ydb-platform/ydb-go-sdk/v3 dependency-version: 3.155.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
896e4dd192 |
build(deps): bump github/codeql-action from 4.38.2 to 4.38.3 (#11746)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.38.2 to 4.38.3. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v4.38.2...v4.38.3) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.38.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
84512927ab | docs: regenerate star history chart | ||
|
github-actions[bot]
|
590a75b9d6 | docs: regenerate star history chart | ||
|
github-actions[bot]
|
8093ecf4c8 | docs: regenerate star history chart | ||
|
|
6840371774 |
sftp: keep a file's existing bytes when it is opened for write without truncate (#11723)
* sftp: keep a file's existing bytes when it is opened for write without truncate * sftp: preserve sparse gaps and refuse unreadable entries when copying existing content The copy used a single chunk-stream read, which ends at the first sparse gap: the tail past a hole was dropped, and a remote-only entry with no local chunks copied zero bytes, so closing the handle uploaded an empty file. Read through PrepareStreamContent, which fills unstored regions with zeros, and refuse entries that carry no local bytes at all. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: report a failed destination write from a completed chunk fetch retriedStreamFetchChunkData broke out of the retry loop on a clean HTTP end before checking writeErr, so a destination write that failed mid- stream returned a short write and nil error. Check writeErr first. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * sftp: cover remote and all-holes sizes when copying existing content Take the declared size from RemoteEntry when it exceeds FileSize, pad inline content to the declared size, and treat a chunkless local file as all holes rather than unreadable. A remote-only entry still refuses the copy, now checked with a zero FileSize as well. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * sftp: size the existing-content copy the way filer.FileSize does A locally shortened file still names its older, larger remote object; taking RemoteSize unconditionally padded the copy back out to it. filer.FileSize only honors the remote size when the remote is newer, and also covers the chunk-extent size. A remote-only entry still refuses the copy whenever it points at remote bytes. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
d67fc47a61 |
build(deps): bump rustls from 0.23.37 to 0.23.45 in /seaweed-volume (#11707)
Bumps [rustls](https://github.com/rustls/rustls) from 0.23.37 to 0.23.45. - [Release notes](https://github.com/rustls/rustls/releases) - [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md) - [Commits](https://github.com/rustls/rustls/compare/v/0.23.37...v/0.23.45) --- updated-dependencies: - dependency-name: rustls dependency-version: 0.23.45 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
mortezaandMorteza
|
d0f3fad133 |
ec: avoid redundant needle.GetActualSize call in LocateEcShardNeedle (#11629)
LocateEcShardNeedle passed needle.GetActualSize(size, version) to LocateEcShardNeedleInterval, which calls needle.GetActualSize internally. This caused header and padding to be added twice to needle size. Fixes #3864 Co-authored-by: Morteza <m.mohammadi@rahkar.team> |
||
|
Чепурнов СемёнandChepurnov Simon
|
624a13ecc6 |
EC worker: pin jobs to replica-host workers, late-bind local source (#11718)
* feat: prefer same-host worker and local EC replica Admin executor pick ranked idle workers only, so a job whose volume lived on full-03 could still run on full-01 and pull the whole .dat across the LAN. Rank workers whose Hello host matches any replica first (fallback to others if the local worker is full). Detection named Sources[0] / the lowest server id; execute then SyncAndSelectBestReplica and could switch the copy source to another rack. After sync, if a replica on this machine has a complete file count, copy from that URL so affinity and pull stay on one host. * feat: pin scheduler jobs to replica-host workers until load K Reserve prefers the least-loaded replica-host worker and waits while that load is below 5 instead of spilling to a free remote worker. K=0 restores immediate spill. Waterfill uses reservation load so four idle preferred workers each get a job. * feat: park scheduled jobs on per-worker admin queues Assign a detection batch to the least-loaded replica-host worker in one pass, then let ExecutionConcurrency goroutines pop a home-runnable job or steal when that home queue reaches K. Dispatch no longer holds a job in reserve-backoff, so a free worker is not blocked by another home's pin. Tracked jobs stay pending/queued and record the home worker for the admin UI. * fix: rehome queued jobs when the home worker dies A live home still pins below K. A home missing from live executors is stolen immediately onto a free worker so a short orphaned queue does not sit until the dispatch window ends. * feat: wait before stealing a dead home worker queue Orphaned queues are held for 30s so a worker reconnect blink does not dump jobs onto other machines. After the grace, steal without K as before. Zero disables the wait. A home that comes back during grace keeps its queue. * fix: late-bind EC from to worker hostname After replicas are consistent (equal FileCount), rebind encode source to a volume server on the same hostname as worker -address. Any port on that host is local. Diverged FileCount still uses the max replica. * feat: assign homeless jobs to global FIFO Jobs whose replica hostname has no registered worker skip pin and orphan grace. Free workers take own home, then steal at K/orphan, then leftover FIFO. * fix: take homeless FIFO before steal for no-home workers Workers without a replica-host match take leftover first; steal (K/orphan) only when FIFO is empty. Home workers keep own-queue, steal, then leftover. --------- Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com> |
||
|
|
492e700d1f |
Volume grow avg fill threshold (#11725)
* master: add conditional grow by average fill ratio to VolumeGrow gRPC Callers that pre-grow volumes on fill level currently have to list all volumes of a collection and sum sizes themselves, which is expensive on clusters with millions of volumes. The master already tracks each writable volume's effective size in memory, so it can answer the question directly. Add grow_avg_fill_threshold to VolumeGrowRequest: when set (e.g. 0.6), growth only happens once the layout's writable volumes have reached that average fill ratio. A layout with no writable volumes skips the check, since that is the case growth exists to fix. VolumeGrowResponse now reports the observed avg_fill and writable_count whether or not growth was triggered. * master: unit tests for WritableAvgFill and FindVolumeLayout Covers the average-fill computation (mixed fills, no writables, a writable volume without a size-tracking entry yet) and that FindVolumeLayout resolves existing layouts without creating missing collections. * address review: report avg_fill and writable_count for every VolumeGrow The stats were only read when grow_avg_fill_threshold was set, so a request without a threshold got AvgFill=0 and WritableCount=0 even when writable volumes existed. Read the layout stats unconditionally and use the threshold only to decide whether growth happens. * master: scope the grow fill check to the requested target A VolumeGrow aimed at a data center, rack or node must not skip growth because fill elsewhere keeps the layout average below the threshold. Measure only writables serving the target; a target with none still grows. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
b5ee5b871e |
sftp: fail a directory listing on a stream error, and page past hidden entries (#11724)
* sftp: fail a directory listing on a stream error, and page past hidden entries * sftp: test a fully hidden listing page Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
6fb21c9ec0 |
filer: copy a remote-only entry on the remote when renaming it (#11709)
* filer: copy a remote-only entry on the remote when renaming it
Renaming a file under a remote.mount whose content exists only on the
remote (never cached, or remote.uncache'd) lost it: the new entry kept
the old RemoteEntry, deleting the old entry deleted the old object, and
with no chunks nothing ever wrote the new key. filer.remote.sync could
not finish the rename either, since the old object was already gone.
moveSelfEntry now copies such an entry's object to the new key before
it touches anything, and the new entry gets the RemoteEntry of the
copy. The copy is a new optional RemoteStorageObjectCopier interface,
implemented for S3 with CopyObject up to 5 GB and UploadPartCopy above
that; the multipart copy carries the source's metadata and, where the
remote supports tagging, its tags. When an existing target is
overwritten, its delete keeps the remote object (the copy replaced it)
and its event is marked metadata-only so the sync leaves it alone too.
A remote-only rename that leaves its mount, or whose remote cannot
copy, is refused instead of deleting the only copy. Empty entries and
entries with local data rename as before.
UpdateFileMetadata builds its copy source with the same helper, which
escapes each key segment and keeps the slashes.
* filer: delete a copied rename's old object only after the commit
Review follow-ups:
- The old key of a copied remote-only entry, and the old directory of a
rename that copied one, are now deleted once the rename has committed.
A rename that fails after the copy leaves the source entry and its
object in place, plus an extra object at the new key.
- Every part of a multipart copy is conditioned on the ETag the copy
started from, so an object overwritten mid-copy fails the copy
instead of mixing versions.
- Both copy paths keep the source's server-side encryption settings.
- filer.remote.sync no longer deletes a key that a remote-only entry
holds by the time the delete event is processed: a file deleted and
then replaced by a remote-only rename has its content only there.
* filer: do not sweep the old prefix of a directory rename that copied
Removing the old directory on the remote deletes everything under its
prefix. Run after the commit, that could also take an object another
rename copied in under the same path meanwhile. When a child was copied,
leave the prefix alone and delete only the copied children's old keys.
* filer: keep an old key a remote-only entry has retaken by cleanup time
A directory rename does not lock its children's paths, so between the
commit and the deferred delete of a copied child's old key another
rename can put a remote-only entry there, whose content is then only in
that object. Check the store before deleting.
* filer: keep a renamed entry's old object when the path lookup fails
The deferred delete must only run once the lookup shows no remote-only
entry at the old path. A failed lookup shows nothing, and an extra
object loses nothing.
* filer: copy over a remote-only rename target only after the commit
Copying the source object onto the target's key before the transaction
committed destroyed the target's only content, which a rolled-back
commit cannot restore. When the target is remote-only the copy now runs
after the commit: the new entry keeps the old key's stamp until the copy
lands and re-stamps it, the target's object is kept meanwhile, and a
failed copy keeps the old object so remote.sync can finish it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote.sync: match the delete event's etag before deleting the object
A rename can copy another entry's content onto a key between the filer's
check and the remote delete, and the delete then removes the copy. The
delete event carries the object's etag; delete only when the live object
still matches it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: finish a rename's remote work outside the committed transaction
The SQL stores keep the transaction in the context and refuse every
call through it once it has committed. The work that runs after
CommitTransaction used that context, so on postgres2 the deferred
old-key delete could not look up the old path and kept the object, and
finishRemoteCopies could not read or stamp the entry. Both failed with
"sql: transaction has already been committed or rolled back" on every
copied rename; remote.sync cleaned up after them.
Keep the context from before the transaction for the work after the
commit. The rename test store gets a strict mode that fails calls
through a committed transaction the way database/sql does.
* filer: carry a deferred copy's stamp in the rename event; delete orphaned uploads
- The deferred copy stamps the entry with a plain UpdateEntry, which
publishes no event, so the queued rename event still carried the old
key's stamp to peer filers and subscribers. Give it the copy's
RemoteEntry before it is sent. The two notify loops share the code now.
- remote.sync kept any object whose ETag changed since the delete. With
no entry at the path that is an upload which finished after the
delete, and keeping it leaves it on the remote for good. Look up the
path first and keep a changed object only when an entry holds it.
* remote.sync: keep a changed object even with no entry at its path
Deleting a changed object when no entry holds the path also deletes a
copy a directory rename has written but not yet made visible, while a
delete of that path waits in the sync; the rename then removes the old
key and the content is gone. Keep any object whose ETag changed since
the delete, as before. An upload that finishes after the delete leaks
its object instead, which is the cheaper failure.
* filer: read a pending remote rename copy from the source key
A remote-only rename over a remote-only target commits before its copy
lands, so reads on the new path served the target's old object until the
copy stamped the entry — and indefinitely when the copy failed without
remote.sync running. The committed entry now records the renamed-from
path, and remote-only reads resolve that object's key while the stamp
is pending; the copy's stamp and the sync's completion stamp both clear
the marker.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: stamp the rename event with the deferred copy's result
A remote-only rename over a remote-only target queued the new entry's
event with the source's stamp before the deferred copy ran. Peers
replaying it saved a stamp the copy had already superseded, and the
sync serving such a peer rejected the destination as a match and tried
to copy the deleted source. The event now carries the copy's result
and drops the pending-source marker once it lands.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote.sync: delete only the object the event's stamp describes
An object at a deleted key is not always the one the delete meant: a
rename copy can land before its filer entry is visible, and a rewrite
can upload over it. When the event recorded no remote stamp, or the
live stat cannot confirm the recorded etag, the delete now keeps the
object instead of treating it as a late upload.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: tighten the pending rename source marker
Follow-ups to reading a pending remote rename copy from its source key:
- Only the filer sets the marker. The HTTP upload and tagging handlers
store Seaweed-* request headers as extended attributes, so they skip
this one, and a marker that leads out of the entry's own mount is
ignored, so it cannot point reads at another remote.
- A pending entry renamed again copies from the marker's source, not
from its own key, which still holds the old target's bytes. A further
deferred rename keeps naming the original source; an immediate copy
clears the marker.
- When remote.sync finds the destination already holding the content,
it stamps the entry (clearing the marker) before deleting the source,
and deletes the source only once the entry no longer reads it.
- A read that resolved the source just before the copy finished and
the source went away retries with the entry as it is now.
* filer, remote.sync: follow a pending rename's source across renames
When a remote-only entry is renamed over another remote-only file and
the deferred copy fails, it reads the source key through its marker.
Renamed again the same way, the marker keeps naming that source, but:
- remote.sync completed each rename from the event's old key. For the
second one that is the first target's old object, which it copied
over the destination before clearing the marker. It now copies from
the source the entry's marker names.
- For the first rename it found no entry at the new path and deleted
the old key, which the later entry still reads. When the event's new
entry carried a marker and the path is gone, it now leaves the key to
the entry that moved on.
- When the filer copies a pending entry immediately on a later rename,
it now also releases the source the copy read, after the commit and
with the same checks as the old key.
* remote.sync: release a pending rename's source, the filer leaves it
The filer deleted a pending entry's source after copying it on a later
rename, but that rename locks only its own paths: another rename can
copy onto the source key before its entry is visible, and the cleanup
then deleted that copy. And deleting a pending entry never released its
source at all.
The filer now leaves marked sources alone. remote.sync releases one
when the pending entry is deleted, or when a later rename was copied
right away, and only while the live object is still the one the
entry's stamp describes and no remote-only entry holds the path: a
copy that landed there since has a different ETag and is kept.
* remote.sync: release a pending source when the renamed entry is cached
A rename copied right away left its pending source to remote.sync,
which released it only while the destination was still remote-only.
Cached before the sync got there, the destination took the other path
and the source stayed on the remote for good. Release it there too,
with the same checks.
* remote.sync: release a pending source only after the destination is safe
|
||
|
|
41260a55f6 |
master: record writableVolumeCount on HTTP /dir/assign like gRPC Assign (#11716)
* master: extract the periodic per-layout grow decision Move the must-grow and crowded-step decision of the periodic growth scan in ProcessGrowRequest into periodicGrowCount, a pure function of the layout's lastGrowCount, writable and crowded counts, so it can be tested without running the scan loop or growing real volumes. No behavior change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: record writableVolumeCount on HTTP /dir/assign The periodic growth scan (#5781) keeps a layout supplied ahead of demand only when it knows how many writable volumes clients want: lastGrowCount, which gRPC Assign records with SetLastGrowCount. HTTP /dir/assign parsed writableVolumeCount but only used it to size the emergency grow when no writable volume was left, so the scan always saw lastGrowCount 0 for HTTP-only writers. Since #5821 removed the per-assign threshold grow, such a layout only grew by the rack-aware step of 2, and only once every writable volume was crowded. When a batch of volumes filled in lockstep, those 2 took every write and could fill before the next scan, leaving no writable volumes until an assign-triggered emergency grow landed, with 503s for every other assign meanwhile. Record writableVolumeCount on HTTP assigns as gRPC Assign does. Clients that do not send it are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: reject unknown rack/dataNode before recording grow targets An assign naming a rack or data node outside the topology can never be served, but lastGrowCount was recorded first. The periodic growth scan drops the requested placement, so it kept topping the layout up with volumes the requester could not use. Both assign paths now validate the placement like they already validate the data center. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: scope placement existence checks to the requested path A rack in another data center or a data node under another rack passed validation yet could never serve the request, and the grow count was still recorded. The rack and data-node lookups now take the requested parents, matching the filters the placement engine applies. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
3031ab5d97 |
EC placer: walk extra shards on freeSlots, keep 2+2 (#11714)
* fix: rotate EC extra shards on pickTarget count ties
pickTarget used to keep the first sorted key when shard counts tied,
so 10+4 onto 4 racks always put the two fours on full-01/full-02.
Tie-break on remaining freeSlots (same idea as pickNodeInRackEligible)
so the leftover shards walk across racks over a series of volumes.
* fix: pay within-rack EC shard debt on Place
Score nodes by live EcShards plus this Place's assignments, not
this-volume counts or EC-volume presence. Allow SameRackCount+1
only after a sibling already holds a shard, so 4-on-2 becomes 3+1
while 3-on-2 stays 2+1. Rack pickTarget tie-break is unchanged.
* fix: even-spread EC disks, pay debt with remainders only
Do not pack 3+1: three shards on one disk is one disk-loss from
unrecoverable. Place this volume evenly first; leftover shards
(2+1) go to the thinner live count. 4-on-2 stays 2+2.
* fix: score Place extras by freeSlots then liveShards
Even-per-volume and SameRackCount stay; remainder prefers capacity,
then shard debt. No 3+1.
* fix: keep shard-debt fixture capacity-consistent
freeSlots = cap-shards so extras keep paying the thinner disk
across volumes instead of oscillating after the first pair.
* refactor: document dropped mFree; switch EcBalance node pick
Keep host even (mCount). Remainder is disk freeSlots then liveShards so
thin disks fill first. SameRackCount / 2+2 unchanged. pickTarget unchanged.
* test: cover Place 10+4 extras on thin disks/racks
Seed liveShards/freeSlots imbalance on 1-1-{4,6,7,8,15}, 4-2-3, 5-2-2, 6-1-2, 7-1-2. Remainder tails must miss fat first ids; remainder 0 freezes the live gap.
* fix(ecbalancer): count only matching-type slots when picking a move target
Aggregate freeSlots let a rack, machine, or node whose free capacity sat
on another disk type win placement; pickBestDiskOnNode then found no
eligible disk and the move went out with TargetDisk 0, which the copy
endpoint reads as automatic selection onto any disk.
Score candidates on freeSlotsForType — per-disk metadata when present,
the node aggregate when absent — so a wrong-type destination is never
chosen, and cover pickBestNodeForVolume's tie-breaks and the wrong-type
plan case.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec_balance: spread free slots per disk type, keeping the remainder
Dividing the node's free slots evenly over all its non-full disks floored
a leftover slot to zero on every disk — one free slot across two disks
made the node look full. It also gave disks of other types a share of
slots counted only for the requested type. Each type's slots now spread
over that type's disks with ceil division, so the total is preserved and
every usable slot stays visible to the planner.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
|
acb44a68a9 |
fix(volume): skip unnecessary needle data reads on delete (#11711)
* fix(volume): do not read a needle's data to delete it A delete read the needle's whole payload from disk only to discard it. The delete handler and BatchDelete need the needle's cookie and its chunk-manifest flag, which live in the header and trailer, but fetched them with a full needle read, the same call a GET uses. readNeedle now reads the header and trailer only when MustMetaOnly is set, whatever the needle's size, and the delete paths ask for that. The Rust volume server honours must_meta_only the same way for its delete paths. A compressed needle or a chunk manifest still gets the full read, as before, because their callers need the bytes. HEAD is unchanged. It used to set MustMetaOnly, which nothing read; it now asks for AttemptMetaOnly alone, so it keeps reading a needle up to PagedReadLimit whole. On a spinning disk that one read is cheaper than a header read followed by a trailer read. Version 1 needles are always read in full. They have no data size field and no metadata after the data, which ReadNeedleMeta relies on, so a metadata-only read of one failed with "invalid needle meta size". That already happened for version 1 needles above PagedReadLimit read with AttemptMetaOnly. A delete therefore no longer verifies the data CRC of the needle it removes. The cookie check is unchanged. * fix(volume): read transformed images fully on HEAD too A HEAD carrying resize or crop params reports the transformed length, which a metadata read cannot produce. Go and Rust both answered such HEADs from the source header, returning the stored size instead of the cropped or resized one. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
6927de6003 |
seaweed-worker: mint the admin bearer token like the Go worker
The admin control stream now requires mTLS or a token signed by jwt.filer_signing.key; the Rust worker gets --filer-signing-key and --filer-signing-key-expires-after-seconds, mirroring the Go worker's config names, and attaches the token to the stream metadata. |
||
|
Chris Lu
|
613a286cde |
worker: present the admin bearer token on control streams
Workers mint a SeaweedFilerAdminClaims token when jwt.filer_signing.key is in their configuration, matching the authentication the admin server now requires. |
||
|
Chris Lu
|
55ac46f7a9 |
admin: authenticate worker and plugin gRPC streams
Worker registration previously ran unauthenticated on any interface the listener bound. The stream now requires a verified mTLS client certificate or a Bearer token signed by jwt.filer_signing.key, and falls back to local clients only when no key is configured. |
||
|
Chris Lu
|
3f8035deb7 |
security: share the admin gRPC peer authentication check
The local-peer fallback and bearer-token validation used by the filer IAM gRPC path move into weed/security so every admin-plane gRPC service can enforce the same contract. A client certificate verified by the server's TLS config now also counts as authenticated. |
||
|
Chris Lu
|
a4c8d31cb8 |
s3api: ignore response header overrides on anonymous requests
AWS requires response-* query parameters to be signed; an anonymous caller must not choose how a stored object is served. |
||
|
Chris LuandDevin
|
6bac506842 |
sftp: terminate the missing-ancestor permission walk at the root
A path whose ancestors are all absent recursed on the root forever,
since Dir("/") is "/".
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Chris LuandDevin
|
2c031d9d63 |
sftp: check write permission on the target file, not only its parent
newFileWriter authorized uploads against the directory, so a file with a more specific read-only ACL or mode could still be overwritten. The same parent-directory fallback used for nonexistent paths now applies through CheckFilePermission on the target itself. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
d69f45ee83 |
s3: validate multipart initiate ACLs like PUT
CreateMultipartUpload accepts canned and custom ACLs on initiation; run them through the same authorization path as PutObject so an explicit ACL requires s3:PutObjectAcl, and persist the validated grants on the upload entry for completion to apply. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
3aaf80b33b |
s3: stop trusting internal ACL headers in ParseS3Metadata
Multipart initiation stored client-supplied Seaweed-X-Amz-Acl and Seaweed-X-Amz-Owner request headers onto the upload entry, and CompleteMultipartUpload copied them to the finished object, letting a caller forge ownership or a public ACL without s3:PutObjectAcl. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
85f04b54fd |
s3tables: check data-path ancestors for catalog markers on DeleteTable
The decoupled-delete guard only inspects the resolved data path itself, but the catalog marker lives on a table's name path, so a location that resolves inside another table's directory (its marker-less data subdirectory) slips through and the recursive purge destroys the other table's data. Walk the data path's ancestors within the bucket and refuse when any carries a table or view catalog entry. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
6f93f87446 |
s3tables: confine caller-supplied metadataLocation to the table's namespace
CreateTable/RegisterTable/UpdateTable (and the view equivalents) validate the location only for bucket and traversal, so a tenant can register a table whose location resolves into another namespace's directory and DeleteTable will purge it. Require the location to live under the caller's namespace subtree, the boundary per-namespace delegation uses. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
scooby87
|
73949daa98 |
filer: keep sequential prefetch within prefetchCount chunks (#11674)
ChunkReadAt.ReadAt asks the ReaderCache to prefetch after every read, and MaybeCache counted only the downloads it newly started toward the window, so each call reached prefetchCount chunks further than the last. A reader consuming a chunk in several reads, as the S3 GET path does with a 256 KiB copy buffer over 4 MiB chunks, ran the read-ahead up to the ReaderCache downloader limit and kept most of a large object in memory, enough to get an S3 gateway OOM-killed mid-response. Count every chunk in the window, downloading or not, so the read-ahead stays at prefetchCount chunks past the reader. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Chris LuandDevin
|
603a1221a7 |
filer: converge peer metadata replication on restart (#11698)
* filer: skip replayed peer events older than the local entry Replay deleted and re-inserted every received event unconditionally, so a restarted filer that rewound its subscription offset re-applied stale peer writes over newer local state (values reverted, deleted entries came back and pointed at already-vacuumed chunks). Compare the event's TsNs against the local entry's Mtime and skip per-path, so a move event can still remove its source while keeping a newer destination. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: add replay last-writer-wins regression tests Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: stop peer subscriptions and flush offsets on shutdown Filer.Shutdown closed the store while subscription goroutines were still mid-stream; their deferred offset save was never guaranteed to run, so a restart rewound each peer's stream back to the last periodic flush and replayed stale events. MetaAggregator.Shutdown now closes every peer's stop channel and waits for the loops so the saved offset covers what was replayed, and the reconnect sleep is interruptible by the stop channel. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: test that shutdown stops and waits for peer loops Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: order replay by persisted per-path commit timestamps Review feedback found the mtime comparison still lets stale replay win: a local delete removes the entry it would have been compared against, an entry mtime is client-controllable, and the check ran unlocked so a local write could land between it and the store mutation. - Record the event timestamp per mutated path in the store KV on every emitted mutation, plus a delete record for sources of deletes and renames. The record survives the entry, so a replayed create older than a local delete no longer resurrects it; ancestor delete records cover children swept without per-child events. - Replay compares the event timestamp against that record under the same per-path entry locks local RPC mutations hold, falling back to entry mtime only for entries written before records existed. Lookup errors now abort the event instead of being treated as an absent entry. - MetaAggregator.Shutdown marks the aggregator stopped under peerChansLock so a master update cannot start a subscription that outlives the wait, and waits for the loops without the five-second cap now that every blocking stage — signature RPC, bootstrap traversal, retry sleeps, and per-event processing — cancels on the stop channel. * filer: pin peer offset at failed replay, fix commit-record context and key size A replayed event that exhausted its retries (or was interrupted by shutdown) was still followed by advancing the peer watermark, so the persisted offset skipped the unapplied event forever. replicateMetadataChange now reports the outcome: errPeerSubscriptionStopped aborts the subscription so the deferred save keeps the last completed event, and an exhausted retry records a barrier the persisted offset and reconnect resume point cannot pass while the stream keeps flowing. Commit records also failed on two fronts: NotifyUpdateEvent runs after CommitTransaction with the same context, so SQL stores wrote MpTs/MpDl through an already-committed *sql.Tx; and metaPathKey embedded the whole path, overflowing the 766-char name column SQL KV stores use, which the KvPut fallback update then silently dropped. The writes now use a fresh context and a sha256-hashed fixed-size key. * filer: lock HTTP delete handler and replay against ancestor paths The HTTP DeleteHandler called DeleteEntryMetaAndData without the shared entry lock, so a peer replay could still interleave with a local delete and its commit record. It now holds the same lock the gRPC DeleteEntry handler does. Replay also locked only the exact event paths, but its ordering check consults ancestor delete records: a local recursive delete holds only the ancestor's lock, so a stale child create could check before the tombstone landed and then reinsert. lockReplayPaths now locks every ancestor of both event paths in one lexicographic order — parents sort before children, as the rename handler's two-path order already does — while never locking the root, which cannot be deleted and would serialize all replays. * filer: record replay commit before the replacement insert If the same-path update DeleteEntry succeeded but InsertEntry failed, the path sat deleted with no commit record until the retry arrived, and a stale peer create could resurrect it in the gap. The commit record now lands immediately after the delete — without a delete record, since an update is not a delete — and again after the successful insert. * filer: lock replay ancestors shared, not exclusive Exclusive locks on every ancestor serialized all replays under a shared parent — unrelated objects in the same bucket could no longer apply in parallel. Ancestors only need to exclude an exclusive holder (a local delete or rename of the ancestor), so they now take shared locks; a path the event changes stays exclusive even when it is also the other path's ancestor. * filer: order peer bootstrap on the sender's commit timestamp traversePeerMetadata compared localCommitTsNs — a commit-clock value — against the entry's mtime, which a client can set freely: a peer change committed recently with an old mtime was skipped, and a stale entry mtime-stamped in the future overwrote a newer local commit. TraverseBfsMetadataResponse now carries the sender's per-path commit timestamp (recorded MpTs record, entry mtime as the unrecorded fallback). The bootstrap applies an entry only when its commit is newer than the local one and records the applied timestamp, so replayed events order against it — closing the gap between bootstrap content and the event stream. A peer without the field sends 0 and keeps the mtime fallback, preserving rolling-upgrade behavior. * filer: keep failed commit records pending until they land notePathChange ran after the mutation committed and only logged a KvPut failure: a delete whose MpTs/MpDl record never landed left no tombstone, so a stale replayed create could resurrect the path while the store was unreachable. Failed records now stay in metaTsPending and a repair pass retries them until they land. localCommitTsNs, deletedAfterTsNs and CommitTsNs consult the pending record, so ordering holds in the window between the committed mutation and the durable record. A per-path flag bounds the repair to one goroutine, and respawns cover merges landing between its last pass and the flag release. * filer: keep only unwritten work in the pending commit record The pending record merged every obligation into one timestamp and flag, and a later successful write left it untouched: a repair pass could then write an older timestamp over a newer one, and a merged delete inherited the newer change's time, which the ancestor check reads as a newer delete. The record now holds a change and a delete timestamp of its own, both discharged under the record's lock alongside the store writes that cover them, so a repair writes only what is still owed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: fail traversal on a commit-record read error CommitTsNs answered a store read failure with the entry mtime, a value clients can set arbitrarily far ahead; a peer bootstrapping on it would record the minted time and reject real updates below it. The lookup now returns the error so the traversal restarts instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |