mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 15:41:15 +02:00
9d5525e7470351c77007f0671f37a41367d0162f
14973
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Lu
|
9d5525e747 |
master: keep periodic volume growth to the data centers a layout lives in (#11060)
* master: keep the periodic growth scan to data centers hosting the layout The rack-aware scan planned growth for every data center in the topology, so a collection pinned to one DC (fs.configure -dataCenter) sprouted volumes in all the others within one scan cycle. Plan only for data centers already hosting the layout's volumes; an empty DC gets its volumes from the DC-constrained assign that first asks for them. The lastGrowCount divisor likewise counts only the racks the scan can plan for. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 * master: pin periodic must-grow growth to a single-DC layout's data center The must-grow and crowded paths of the periodic loop grow with no DataCenter, so even with the scan fixed a pinned collection's volumes could still land in any DC once lastGrowCount demands more writables. Stamp the grow request with the layout's data center when its volumes all live in one; layouts spanning DCs keep unconstrained growth. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 * master: never pin growth of a cross-DC-replicated layout A layout whose replication spans data centers cannot legitimately live in one DC; observing a single hosting DC there means the other DCs are down. Do not encode that outage as a placement constraint. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 * master: bound the hosting-DC walk by the answer it needs listVolumeDataCenters walked every location of the layout under accessLock — ~190ms for a million volumes, twice per layout per cycle, stalling assigns behind the read lock. Stop once enough distinct DCs answer the caller's question: two for the single-DC check, the topology's DC count for the scan. A spanning million-volume layout now finishes in microseconds; only a layout truly confined to fewer DCs still pays a full walk, the same cost class as the under-replication count this loop already takes each cycle. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 |
||
|
Chris Lu
|
8873f9775c |
shell: reset noLock in the admin script dispatcher too (#11059)
Three dispatchers reuse one CommandEnv: the interactive shell, the master's maintenance script runner, and the plugin worker's admin script handler. The first two were fixed; this is the third. It changes nothing today -- ForceNoLock already exempts this path -- so it is here to keep the rule the same everywhere rather than resting on that exemption staying in place. |
||
|
Chris Lu
|
1996c6aec6 |
volume: open volume files with O_NOATIME (#11055)
* volume server: open volume files with O_NOATIME Nothing reads the atime of .dat, .idx, .sdx, or EC files, but every needle read still dirtied the inode: even relatime writes atime on the first read after each write, so an actively written volume paid a metadata write per read/write cycle, and strictatime mounts paid one per read. Open the serving handles with O_NOATIME, falling back to a plain open when the file belongs to another owner (EPERM). Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD * seaweed-volume: mirror the O_NOATIME volume file opens Same change as the Go volume server: serving handles for .dat, .idx, .sdx, .ecx, .ecj, and shard files open with O_NOATIME on Linux, with a plain-open fallback on EPERM. Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD * route the tier-down and recreate .dat opens through the no-atime helper Review caught the Rust tier-down swap opening the local .dat directly. The Go swapToLocalDatBackend and the zero-length read-only .dat recreate in maybeWriteSuperBlock had the same gap: all three install long-lived serving handles. Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD |
||
|
Chris Lu
|
0c59c0fb05 |
master: scope the startup capacity shed to a truly empty topology (#11058)
* master: scope the startup capacity shed to a truly empty topology The retryable "no volume server capacity registered yet" shed checked capacity for the requested disk type, so a cluster serving only other media -- where that capacity will never register -- shed every assign until the client's deadline instead of failing fast. An unsteered write to such a cluster hung for its full HTTP deadline and surfaced "context deadline exceeded" in place of "No writable volumes". Shed only while no disk type has any registered capacity, and name the unserved medium in the fast failure. Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R * master: name the unserved medium for every fail-fast caller The diagnostic sat in the growth-initiator block, so a follower joining an in-flight growth and a growth-disabled master failed the same way with only the generic pick error. Wrap at the fail-fast break instead, which every caller reaches, and cover all three paths in the test. Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R |
||
|
Chris Lu
|
721499a05a |
release: judge downstream releases by their run, excluding runner-queue time (#11057)
Claude-Session: https://claude.ai/code/session_01Y228KU8MLsGmfpcbGxgjwh |
||
|
github-actions[bot]
|
79b8720213 | 4.45 4.45 | ||
|
Chris Lu
|
0f4a7d0803 |
shell: keep noLock to the command that set it (#11052)
noLock says "this invocation changes nothing" -- volume.balance, volume.move,
volume.copy, volume.merge and volume.fix.replication all set it for a dry run,
and none clears it. The CommandEnv is created once and reused by both
dispatchers, the interactive shell and the master's maintenance script runner,
so a simulation left every later command unlocked:
volume.balance -noLock # changes nothing
volume.move ... # mutates, and skips its lock
Reset before dispatch in both, where the invocation begins. forceNoLock is
untouched: that is set once, deliberately, for a trusted path.
|
||
|
Chris Lu
|
87474c2f21 |
s3: let attached policies authorize CreateBucket (#11049)
* s3: resolve admin bucket subresources to their specific S3 actions Encryption, requestPayment, publicAccessBlock and ownershipControls requests reached the policy engines as s3:*, so only a policy granting all of s3 could authorize them. Map each subresource to its AWS action, with DELETE sharing the PUT permission as AWS does. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ * s3: authorize CreateBucket as s3:CreateBucket in the policy engine A plain bucket-level PUT is registered with ACTION_ADMIN, which resolved to s3:*, so no attached policy short of s3:* could match it. Federated sessions whose policy explicitly allowed s3:CreateBucket were always denied while the same policy worked for object operations. Resolve it to s3:CreateBucket, like DeleteBucket already resolves. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ |
||
|
Chris Lu
|
b8049bc633 |
shell: keep fs.mergeVolumes from spinning past the finished moves (#11050)
* filer_pb: walk a re-delivered directory only once in TraverseBfs A directory handed back twice by a listing (a page-boundary race with concurrent renames, or a store whose ordering misbehaves) was enqueued twice; the second walk re-lists the same subtree and can keep the traversal from ever terminating. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ * filer_pb: fail a directory listing whose pagination stops advancing A full page ending on the very name the cursor started from re-fetches the same page forever; a store whose listing order does not advance past the cursor turns any full-directory read into a silent infinite loop. Return an error naming the stuck cursor instead. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ * shell: skip foreign-collection manifests in fs.mergeVolumes Every manifest chunk in the namespace was resolved, downloading its manifest needle, even when the merge plan only touches one collection. Sub-chunks live in the manifest's own collection, so a manifest on a volume outside the plan's collections cannot reference a source volume; skip it and spare a cluster-wide download pass that looks like a hang after the real moves finish. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ |
||
|
Chris Lu
|
23d424248d |
sts: session duration no longer clamped to the web identity token exp (#11048)
* sts: session duration no longer clamped to the web identity token exp The assumed-role session lifetime is governed by DurationSeconds and the configured tokenDuration/maxSessionLength, matching AWS. Clamping to the already-verified token's exp made short-lived id_tokens (GitLab issues ~2-minute ones) yield unusable sessions regardless of configuration. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ * sts: cover session duration against short-lived web identity tokens The mock OIDC provider now carries the token exp through to the identity like the real provider, so the integration test would catch the clamp. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ |
||
|
dependabot[bot]andChris Lu
|
49ee13635b |
build(deps): bump google.golang.org/grpc from 1.84.0-dev.0.20260723093437-b6eac429d7b6 to 1.85.0-dev (#11043)
build(deps): bump google.golang.org/grpc Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.84.0-dev.0.20260723093437-b6eac429d7b6 to 1.85.0-dev. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](https://github.com/grpc/grpc-go/commits/v1.85.0-dev) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.85.0-dev dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
dependabot[bot]
|
8bd5ec37d1 |
build(deps): bump google.golang.org/api from 0.293.0 to 0.294.0 (#11044)
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.293.0 to 0.294.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.293.0...v0.294.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-version: 0.294.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
1971fb8cb0 |
build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.43.5 to 1.45.1 (#11046)
Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.43.5 to 1.45.1. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.43.5...v1.45.1) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2 dependency-version: 1.45.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
8c3695fd4a |
build(deps): bump actions/setup-java from 5 to 6 (#11047)
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 5 to 6. - [Release notes](https://github.com/actions/setup-java/releases) - [Commits](https://github.com/actions/setup-java/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/setup-java dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
3a3d513cb8 |
build(deps): bump github/codeql-action from 4.37.8 to 4.37.9 (#11045)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.8 to 4.37.9. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v4.37.8...v4.37.9) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.9 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
86ee10a080 |
build(deps): bump modernc.org/sqlite from 1.56.0 to 1.57.0 (#11042)
Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.56.0 to 1.57.0. - [Changelog](https://gitlab.com/cznic/sqlite/blob/master/CHANGELOG.md) - [Commits](https://gitlab.com/cznic/sqlite/compare/v1.56.0...v1.57.0) --- updated-dependencies: - dependency-name: modernc.org/sqlite dependency-version: 1.57.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
909f5cabc7 |
build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.147.1 to 3.151.1 (#11041)
build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 Bumps [github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) from 3.147.1 to 3.151.1. - [Release notes](https://github.com/ydb-platform/ydb-go-sdk/releases) - [Changelog](https://github.com/ydb-platform/ydb-go-sdk/blob/master/CHANGELOG.md) - [Commits](https://github.com/ydb-platform/ydb-go-sdk/compare/v3.147.1...v3.151.1) --- updated-dependencies: - dependency-name: github.com/ydb-platform/ydb-go-sdk/v3 dependency-version: 3.151.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Guang Jiong LouandChris Lu
|
f740210235 |
get volume topology info without volume details (#11036)
* get volume topology info without volume details Signed-off-by: lou <alex1988@outlook.com> * master: rename VolumeListRequest.without_volumes to topology_only The field shapes the reply rather than selecting volumes, and it leaves out the ec shards too, which the old name denied. Match the message's *_only style and say what a master that predates the field does with it. Claude-Session: https://claude.ai/code/session_01QHnaNRgxnjzZsiz7WTFML5 * master: refuse topology_only combined with a volume selector A topology_only request that also names a collection or volume ids contradicts itself, and answering either half in silence surprises the caller. Answer InvalidArgument from both VolumeList and its stream, before the stream sends its header. Claude-Session: https://claude.ai/code/session_01QHnaNRgxnjzZsiz7WTFML5 --------- Signed-off-by: lou <alex1988@outlook.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
32df246a81 |
mq: fix idle-cleanup shard deadlock that permanently wedges the broker's topic map (#11051)
mq: remove emptied topics after the cleanup iteration, not inside it cleanupIdlePartitions called manager.topics.Remove from inside manager.topics.IterCb. IterCb holds the shard's read lock while running the callback, and Remove takes the same shard's write lock, so removing an emptied topic self-deadlocked the cleanup goroutine. The pending writer then blocked every later reader of that shard, permanently hanging ListTopicsInMemory and, for shard-mates, TopicExistsInMemory. On the Kafka gateway this surfaced as flaky e2e consumer-group tests: one minute after any earlier topic went idle, the broker's first 'Removing empty topic' wedged the map, every gateway ListTopics/TopicExists RPC burned its full 5s timeout, Metadata could no longer finish inside kafka-go's 5s coordinator deadline, and consumer groups looped in PreparingRebalance until the test timed out. Collect the emptied topic keys during the iteration and remove them afterwards via RemoveCb, re-checking emptiness under the shard lock so a topic that just gained a partition is kept. Claude-Session: https://claude.ai/code/session_014yA6c8JQcY6MqPXCT13yYA |
||
|
Chris Lu
|
d3b8030a69 |
master: shed assigns retryably until volume servers register capacity (#11032)
An assign arriving before any volume server has heartbeated saw zero available space and failed outright with a plain error no client retries, so the first write to a fresh bucket answered 500 while the cluster was still starting. Distinguish a topology with no registered capacity from a genuinely full one: fail fast only when registered capacity is exhausted, and shed ResourceExhausted otherwise so the client's retry budget rides out the startup window. Claude-Session: https://claude.ai/code/session_018G9kWFgy8BaBAEkYV3YL9n |
||
|
Chris Lu
|
9bafeb6139 |
ec: refuse to mount a 0-byte shard file when the index has entries (#11030)
* ec: refuse to mount a 0-byte shard file when the index has entries The startup scan already skips (and eventually deletes) zero-sized shard files as residue of a failed copy, but the mount RPC path opens the file directly with no size check, so an explicit VolumeEcShardsMount over a truncated file registers a size-0 claim. A registered empty shard serves nothing while advertising ownership: with placement pinned to the owning disk, it would keep attracting re-copies to a file that was never valid. The one legitimate 0-byte shard is the empty volume's: encoding a volume with no live needles produces a 0-byte .ecx and 0-byte shards, and that mount must keep working (TestMountEcShards_EmptyEcxMountsSuccessfully). So the gate compares against the index: AddEcVolumeShard (Go) and EcVolume::add_shard (Rust) refuse a 0-byte shard file only when the volume's .ecx has entries. Go's AddEcVolumeShard grows an error return for this; the loader cleans up the refused shard and, when it just created the EcVolume, unregisters that too. The mount loop already collects non-ENOENT failures per disk and keeps scanning, so a sibling disk holding a real copy still wins. Regression tests in both trees: an empty shard beside an index with entries is refused and leaves nothing registered; an empty shard of an empty volume still mounts. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 * ec: release the duplicate shard when a mount retry re-loads it Review follow-up: AddEcVolumeShard keeps the existing shard and reports added=false for a shard this disk already registered, but the loader discarded that result, so every retried LoadEcShard leaked the duplicate it had just opened — an fd and a mount-gauge increment per retry. Release both and return the existing volume. Regression test pins the gauge. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 * ec: close the test DiskLocation instead of only its EC volumes Review follow-up: DiskLocation.Close() also stops the background goroutine NewDiskLocation starts; closeEcVolumes left it running for the rest of the test process. Both uses are this PR's own tests. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 * rust: unregister the just-created EcVolume when its first mount is refused Review follow-up: when the first mount of a volume rejects its shard (e.g. the new 0-byte-beside-nonempty-index refusal), the Rust mount path had already inserted the EcVolume and propagated the error without removing it — a zero-shard registration advertising a mount that serves no data while pinning the .ecx/.ecj descriptors (and, since placement's mounted tier keys off it, steering shard placement at this disk). Remove it on the way out, exactly as the Go loader already does; a volume that already holds shards keeps them (the RPC's first-error-aborts contract). Regression test covers both. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 * rust: skip already mounted shards on a mount retry Review follow-up: EcVolume::add_shard replaces self.shards[id] for a shard the volume already holds, and the mount loop then bumps the ec_shards gauge although the mounted count did not grow — gauge drift on every mount retry, and a serving fd swapped for no reason. Skip shard ids the volume already reports, mirroring Go's AddEcVolumeShard added=false handling. Regression test pins the gauge across a duplicate mount (unique collection label: the gauge is process-global and tests run in parallel). Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 |
||
|
Chris Lu
|
74b520113e |
ec: pin auto-selected shard placement to the disk that already owns the shard (#11029)
* ec: pin auto-selected shard placement to the disk that already owns the shard A multi-disk server legitimately mounts one EC volume on several disks, so FindEcShardTargetLocation's per-volume tiers tie at "mounted" and the free-shard-count tie-break decides — pointing at whichever disk is emptier, not at the disk that already holds the shard being placed. A re-copy of a shard the server already has (a retried ec.balance / ec.rebuild move) then lands on a sibling disk, and both disks register the same (volume, shard id): the shard is reported to the master from two disk ids, and which claimant serves reads or survives a later unmount/delete becomes an accident of Locations order. Add a tier above "mounted": a disk that already claims one of the shard ids being placed wins, ahead of the space filters too — re-copying in place needs no new shard slot, and a genuinely full disk should fail the write rather than silently split the claim. Applied to the Go selector and the VolumeEcShardsCopy auto-select (ReceiveFile refuses mounted EC volumes, so no claim can exist there) and mirrored in the Rust volume server. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 * ec: refuse a copy batch whose shards are already owned by different disks Review follow-up: ownership-aware selection ranks a mixed-owner batch (shard 0 on disk A, shard 2 on disk B — the legitimate multi-disk spread) into one destination, so the copy would still duplicate the losing disk's claim. No production caller sends such a batch (balance moves one shard, rebuild and encode copy shards the target lacks), so fail closed: report every owning disk via Store.EcShardOwnerDisks and refuse the copy with an error naming them, telling the caller to split per shard or pass disk_id. Go and Rust, with unit tests for the owner-reporting contract. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 |
||
|
Lorentz Kinde
|
83a929720a |
fix: Move cluster.LiveLock.generation to first field fix 32bit alignment (#11028)
On 32-bit systems the generation fields becomes unaligned. Not an issue on 64bit. Discovered on 32bit raspbian, validated by manual patch (in addition to test). |
||
|
Chris Lu
|
88c873ecd4 |
ec: uniform shard block layout (#10932)
* ec: uniform shard block layout An EC volume is striped as 1GiB blocks until less than one row remains, then 1MiB blocks, and consecutive blocks land on different shards. With ec.encode's -fullPercent 95 against the 30GiB default limit, ~30% of every volume sits in that 1MiB tail, so a 4MB filer chunk there is five stripes on five servers. New encodes now use one block per shard, sized ceil(datSize/dataShards) rounded up to 1MiB and recorded in the .vif (EcShardConfig.block_size, also carried by the .ecsum manifest). A needle now maps to one shard unless it is larger than the block or straddles a boundary. The chosen size equals the legacy layout's padded shard length for every input, so shard sizes, capacity math, and the shard-size credibility checks are unchanged; only the byte placement moved. Reads, decode, and scrub resolve the block sizes from the volume's .vif; absence keeps the legacy interpretation, so existing EC volumes read exactly as before. Rebuild is layout-agnostic. weed fix -ecx recovers the layout from the .vif, else the .ecsum sidecar, and with neither de-stripes under both candidate layouts and keeps the one that indexes more valid needles. Same change in the Rust volume server, which now also streams the encode in 256KB sub-batches like Go instead of allocating whole blocks, and computes the large-row count as shardSize/largeBlock to match Go on exact multiples. On a 26MB fixture both encoders produce byte-identical shards, and a Go-written .vif parses in Rust with the block size intact. * ec: resolve the rust ecx rebuild through the recorded layout The Rust rebuild path regenerated a lost .ecx by scanning the logical .dat through a hand-rolled pure-1MiB striping, which was already wrong for legacy volumes with large-block rows and is wrong for any uniform volume with a block past 1MiB. Route the scan through locate_data with the .vif-recorded block size, the same mapping the read path uses. Also seed the new tests' random data instead of the deprecated global math/rand.Read. * ec: fail the Rust ecx rebuild on any shard read error A read error mid-scan published the entries collected so far as a successful .ecx, and read_at's byte count was ignored so a legal short read passed as complete — a truncated or failing shard could produce a silently incomplete recovery index. Exact-read semantics in read_from_data_shards, error propagation in the needle walk, and a truncated-shard regression test. * ec: fail the mount on an unreadable or malformed vif Both servers silently fell back to the legacy layout when an existing .vif could not be read or parsed. Every new encode records a positive uniform block size there, so the fallback mounted the same shards with legacy offset math and could return wrong data. Absent stays legal (legacy volumes predate the sidecar), and a zero-byte stub still reads as absent (Go's MaybeLoadVolumeInfo convention, now mirrored in Rust); a present-but-unreadable or malformed .vif fails the mount instead. * ec: bound the reconstruct fan-out of one needle's intervals A degraded interval fans out a read to every reachable shard location, each with a buffer the size of the interval. Reading a needle's intervals in parallel multiplied that by the interval concurrency: a needle spanning 8 blocks could hold 8 x MaxShardCount remote reads and buffers at once, where the sequential version peaked at MaxShardCount. Give each needle a single reconstruct budget its intervals share, held for the buffer's lifetime, so separate reads stay independent but one read cannot multiply its own fan-out. * ec: drop the duplicated shard-size formula calculateExpectedShardSize reimplemented the padding rule that UniformBlockSize already owns — TestUniformBlockSizeMatchesLegacyShardSize asserts the two agree for every input — so a change to the rule would have had to be made in both. Defer to the helper, keeping the historic answer for an empty .dat. * ec: resolve the shard block layout from whatever records it Four places still answered the layout question by inference when a record of it was available, or accepted an answer that was not one: - A mount with no .vif defaulted to the legacy layout; the bitrot sidecar records the same config at encode time, so take it when present, as weed fix -ecx already does. The vif itself is now parsed once per mount rather than twice. - The Rust ecx rebuild derived its row count from the padded shard extent, which under the legacy layout reads a shard that is an exact large-block multiple as one row too many. Pass the encode-time .dat size from the .vif and keep the extent as the fallback. - weed fix -ecx read the block size outside the EC-config guard (collapsing the unknown sentinel into a definitive legacy), only wrote the recovered layout back when the .vif was absent rather than unusable, and broke a scan tie by candidate order instead of the documented reach. - The uniform layout tripped writeDatFile's large-block ambiguity guard, which cannot apply when the large and small blocks are the same size. * ec: give the index-recovery tests a parseable vif The fixtures wrote the literal bytes "volinfo" as the source .vif and the recovery copies it verbatim, so the receiving server then mounted the volume from a .vif it could not parse. That used to pass by silently defaulting to the legacy layout; a mount now refuses a vif it cannot read, which is what the tests were exercising all along without meaning to. * ec: validate the layout a vif records, not just its syntax Review follow-ups on the mount-strictness change: - A .vif can parse and still record a block size no encoder could have produced (negative, or not a whole number of small blocks). Both servers took it and mapped every read through it. ValidateBlockSize / the Rust mirror now refuse the mount, the same way an unparseable vif does; 0 stays valid as the legacy two-tier layout. - The bitrot-sidecar fallback accepted parity_shards == 0 and summed the counts in their own width, so values near the ceiling wrapped past the MaxShardCount bound. Require both counts and sum in a wider type. - weed fix -ecx treated a config with only DataShards > 0 as usable, so a half-written .vif suppressed the recovery paths AND survived the rewrite. Require a complete, in-range config before trusting it. - Returning the vif-load error left the .ecx and .ecj descriptors open; repeated mount attempts on malformed metadata could exhaust them. * ec: refuse to act on a layout the metadata does not establish - The worker encode only logged a failed .vif write and skipped it in the distribution set, and treated the .ecsum write as best-effort. A worker whose disk filled after the much larger shards landed could still distribute, mount, verify shard inventory, and delete the source replicas — leaving holders with shards whose geometry nothing records. Both writes and both inclusions are encode success conditions now. - A generation-matching .ecsum that disagreed with the .vif geometry only disabled checksums in Go, and in Rust was not compared at all, so protection stayed On while reads used the other layout. Both files record the layout their generation was encoded with, so a disagreement now fails the mount. * ec: reject an invalid recorded block size in weed fix -ecx A .vif with valid shard counts but a negative or unaligned block size was marked usable: a positive invalid value pinned the scan to a geometry that de-stripes to garbage, and a negative one ran the dual scan but left the invalid .vif in place afterwards. Validate it with the same rule the mount applies, and when it fails leave the layout unknown so the scan recovers it and the file is rewritten. * ec: validate the sidecar layout weed fix -ecx recovers from The .ecsum fallback was taken on DataShards > 0 alone, so a CRC-valid sidecar carrying the wrong generation, an incomplete ratio, or an unaligned block size would pin the reconstruction to one incorrect uniform-layout candidate instead of letting the dual scan decide. Require generation 0, a complete in-range ratio, and a valid block size; anything less leaves the layout unknown, which is the answer that still recovers by scanning. * ec: let only a genuinely absent sidecar choose the legacy layout With no .vif the bitrot sidecar is the only record of a volume's layout, and the mount fallback read a failed load, an unusable config, or a sidecar stamped for another generation as "assume legacy". A uniform generation-0 volume could therefore mount with legacy or another generation's geometry and answer reads with the wrong bytes. Present-but-unusable now fails the mount; only actual absence keeps the legacy defaults. Shared as EcShardConfigFromSidecar so every caller reads the sidecar the same way. * ec: treat a recorded-but-impossible layout as corruption, not as legacy - A .vif whose ecShardConfig is PRESENT but records an impossible ratio was answered with the default 10+4 and the legacy block layout, in both languages. That reads a uniform volume's shards at the wrong offsets and returns the wrong bytes. Only an entirely absent config still means "this predates the record"; a present one that cannot be true fails the mount. - The shard-count bound summed two uint32 counts as int, which wraps on a 32-bit build: 0x7fffffff + 0x7fffffff lands at -2 and slips under MaxShardCount. ValidEcShardCounts sums in uint64, and every EC call site that checked a recorded ratio now goes through it. * ec: rebuild on the geometry the sidecar records, and flag it when it disagrees The rebuild RPC passes BackgroundECContext, so RebuildEcFiles resolves the layout itself — and it resolved a missing or invalid .vif to the default 10+4 with the legacy block size. Two consequences: a 12+4 volume was reconstructed through a 10+4 matrix, which produces wrong bytes and never regenerates shards 14-15; and the chosen geometry then contradicted a valid uniform sidecar, which loadRebuildSidecar reported as BitrotOff — silently skipping the input and regenerated-shard checksum checks precisely when the volume had already lost its metadata. The layout now resolves from the bitrot sidecar (found across the server's disks, not just beside the base name) before falling back to the defaults, and a present-but-impossible ratio fails instead of being replaced. A sidecar that contradicts the chosen geometry is BitrotInvalid, which the existing unsafeIgnoreSidecar override still lets an operator push past. * ec: let the Rust rebuild read metadata off a sibling disk read_ec_shard_config searches only the location the rebuild writes into, so a volume whose .vif or generation-0 .ecsum sits on another of the server's disks resolved to the default 10+4 with the legacy block layout — the Rust half of the geometry-guessing the Go rebuild just stopped doing. It then reconstructs a custom-ratio or uniform volume through the wrong Reed-Solomon matrix and de-striping geometry. The rebuild now looks for the .vif in its own location and then each sibling, falls back to the generation-0 sidecar wherever that lives, and only defaults when neither exists anywhere. The encode-time .dat size the ecx rebuild needs is resolved the same way. * ec: resolve a rebuild's vif from every directory that may hold it RebuildEcFiles probed only <data-base>.vif. The caller knows the selected location's index directory and the sibling locations, but passed neither for metadata: additionalDirs carried shard directories only, and were searched for shards and the checksum sidecar. A split -dir/-dir.idx layout, or a disk holding only shards, therefore resolved a pre-sidecar custom-ratio volume to 10+4 and reconstructed through the wrong matrix — never regenerating shards 14-15. The caller now hands over the index and sibling directories, and the resolver probes the vif across all of them, matching what the Rust resolver already does for both the vif and the sidecar. * ec: make every rebuild consumer agree on the layout it resolved - The post-rebuild bitrot backfill re-derived the geometry from this directory's .vif alone and dropped the block size entirely, so a rebuild that resolved its layout from a sibling, the sidecar, or a uniform vif wrote a manifest describing a DIFFERENT layout — one later mounts reject, or that covers only the default shard count. The layout is resolved once now, through an exported ResolveRebuildECContext, and the rebuild and the backfill share that answer. - The Rust rebuild collected only each location's data directory, so a sibling's INDEX directory — where a split -dir/-dir.idx layout keeps .ecx/.ecj/.vif — was never probed, and a custom-ratio volume still resolved to 10+4 with the legacy layout. Both directories of every location are carried now, deduped against the rebuild's own. - A shard delivery can bring the checksum manifest with it, but the receive path only writes the file: a server that already had the volume mounted kept its resolved protection state (off) until a remount. The mount RPC re-resolves it once the shards it describes have been added. * ec: cover the rebuild's directory search with tests Reviewers flagged the sibling index directory twice, and the fix that closed it had no test of its own: the assembly sat inline in the rebuild handler, reachable only through a gRPC call against a populated store. Lifting it into rebuildSearchDirs / select_rebuild_location makes the rule assertable — a sibling contributes BOTH its data and its index directory, a shared index directory is listed once, and the rebuild's own data directory never repeats. Writing the Rust cases surfaced that the two implementations do not agree on where the rebuild's own index directory belongs, and both are right: Go's resolver takes a single directory list, so that directory has to be inside it, while Rust's takes the rebuild's data and index directories as their own arguments and would search them twice. The tests now state which contract each side is holding to, so neither drifts into the other's shape. Pure refactor otherwise; no behaviour change. * ec: search the index directory for the layout sidecar The Rust resolver looked for the generation-0 .ecsum in the rebuild's data directory and the sibling list, but not in the rebuild's own index directory — while the .vif lookup directly above it did, and Go's findBitrotSidecar has always checked both bases. On a split -dir/-dir.idx location that directory is where the metadata lives, and callers leave it out of the sibling list precisely because it is passed here separately, so nothing searched it. With no .vif anywhere the sidecar is the only surviving record of the layout. Missing it resolved a 12+4 uniform volume to 10+4 with the legacy striping — the test added here fails with (10, 4, 0) against the old code — and the rebuild then reconstructs through the wrong matrix and writes .ecx offsets that no reader can follow. * ec: let the rebuild see its own index directory The Rust rebuild takes a single flat directory list — the shape Go's RebuildEcFiles uses — so it cannot be handed the rebuild location's index directory separately the way the layout resolvers are, and the handler was passing the sibling list, which deliberately omits exactly that directory. On a split -dir/-dir.idx location that is where .ecx and .vif live, so the shard and index lookups could not see them. Go has always carried that directory in additionalDirs; this lines the two call sites up. * ec: let a config-free vif fall through to the layout sidecar A .vif that carries no ecShardConfig answers nothing about the layout, so it is no more informative than an absent one — but both trees treated its mere existence as the end of the search. Go went straight to the 10+4 legacy defaults without consulting the sidecar at all; Rust returned whatever ec_shard_config_from could make of a single directory. A 12+4 uniform volume with a legacy config-free vif therefore resolved as 10+4 legacy, and every read landed at the wrong shard offset. The sidecar lookup was also single-directory on both sides, while a split -dir/-dir.idx layout keeps .vif and .ecsum with the INDEX. Go's findBitrotSidecar has always taken both bases; the callers here passed only the data base, and the Rust bitrot resolver derived its path from the data base alone. Rust's layout resolver now takes a candidate directory list — data, index, then any siblings — and searches all of it, which also removes the early return that made the vif's presence decisive. load_vif_info_across_dirs reported `dir` even when load_vif_info had found the vif in `dir_idx`. Nothing reads that field today, so this changes no behaviour; it stops the next caller that resolves the rest of the volume's metadata against the answer from being sent to a disk holding none of it. Absence stays legal throughout: a volume with neither record is genuinely legacy. Present-but-unusable still fails the mount, now in the config-free-vif branch too. * ec: activate a delivered sidecar on every per-disk runtime A vid mounts as one EcVolume per disk, each with its own resolved protection state, but the post-delivery reload used the first-match lookup and so touched exactly one of them. The siblings kept reporting no protection until a remount — and since shard distribution deduplicates the metadata files onto the first target disk for a node, the runtime that got the .ecsum is not necessarily the one the lookup returns. Iterate every runtime instead, via a new FindAllEcVolumes and its Rust mut equivalent. Combined with each runtime now resolving its sidecar against its index directory as well as its data directory, a server sharing one -dir.idx across its disks activates all of them from the single delivered copy. The Rust volume server had no post-mount reload at all; it gets one here, matching Go. * ec: resolve the delivered sidecar across every EC metadata directory Reloading every per-disk runtime, added last round, did not by itself make the delivered manifest reachable. Startup mirroring copies .ecx/.ecj/.vif to every shard-bearing disk so each mounts self-contained, but deliberately not .ecsum, and a repair delivers exactly one copy. Each runtime was resolving against its own two directories, so every sibling of the disk that received the file kept reporting no protection however often it reloaded. Resolve one authoritative copy across every EC metadata directory instead of duplicating the file. Mirroring .ecsum would have to keep pace with a file that is rewritten as shards are repaired, and would not help the reported case at all: the delivery happens at runtime, and mirroring only runs at startup. The regression test pins both halves — a reload restricted to the volume's own directories still finds nothing, and the same reload given the server's metadata directories turns protection on. * ec: ask every directory before writing a TOFU baseline After a rebuild the opportunistic backfill asks whether this volume already has a checksum manifest, and answered from the data base alone. A split -dir/-dir.idx layout keeps the sidecar with the index, and a multi-disk server may keep it on a sibling, so an existing manifest read as absent. The consequence is worse than a missed read. On a false "no" the backfill writes a fresh sidecar at the data base from whatever the shards say right now — and the data base is the first candidate every resolver checks, so that TOFU baseline shadows the real manifest rather than sitting beside it. A shard that was silently corrupt gets blessed, and the record that would have caught it stops being consulted. FindBitrotSidecar exports the search the package already used internally, so the question is asked of the data base, the index base and the sibling disks — the same candidates the rebuild resolves its layout from. * ec: refuse a shard block size no encoder could have produced weed fix -ecx derived one from the raw shard extent, so a truncated or partially copied shard wrote a .vif that NewEcVolume then permanently refuses — the volume the tool was run to rescue could never mount again. An extent that is not a whole number of small blocks cannot have come from a uniform encode, so it is no longer offered as a candidate, and nothing unvalidated reaches the .vif. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: derive the .vif's dat size and block size from one measurement VolumeEcShardsGenerate stat'ed the .dat before the encode while WriteEcFiles stat'ed it again to size the blocks. A write landing between the two produced a .vif whose own two fields describe different files. WriteEcFiles now leaves both on the context, and fills a placeholder context in place so the caller can read them back. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: keep the source volume until every holder serves its shard layout The uniform layout rides in a .vif field older volume servers never knew: they discard it, mount the shards as legacy and return wrong bytes with nothing erroring, and the shard files are the same length either way so no other check notices. The upgrade order lived only in the release note. VolumeEcShardsInfo now reports the block size the holder actually serves, in both the Go and Rust servers, and the pre-delete verification refuses to drop the source unless every reachable holder echoes the one the shards were encoded with — while a rollback still exists. A server that predates the field answers 0, which is the negative answer. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: drop the rebuild's dead block-size parameters generateMissingEcFiles never reads largeBlockSize/smallBlockSize — Reed-Solomon reconstruction is layout-agnostic — so passing the legacy constants only advertised a layout the rebuild does not use. Also move UniformBlockSize's doc off ValidateBlockSize. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: warn about EC defaults only when the mount used them The "vif file not found, using defaults" warning fired even after the bitrot sidecar supplied a non-default layout, sending anyone triaging wrong bytes after the legacy layout the volume never mounted on. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: stat the distributed bitrot sidecar once The strict check re-stat'ed the file immediately before the stat that already gates inclusion, and a failed sidecar write now fails the encode outright, so the first could only fire on a deletion between the two lines. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: say what the reconstruct budget actually bounds A shard's buffer stays in bufs until its interval reconstructs, which is after the read that filled it released its permit, so the semaphore bounds round trips in flight and not retained bytes. Peak memory is the intervals reconstructing at once times the shards each reaches times the interval size. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * test: let the fake volume server report its delivered EC layout The pre-delete verification now asks each holder which shard block layout it serves, and a fake that always answered "unset" looked exactly like a volume server too old to know the field. Distribution ships the .vif to every holder alongside its shards, so read the layout back out of it as a real holder does. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 |
||
|
Guang Jiong LouandChris Lu
|
93666c90e9 |
filter by volume ids (#10983)
* filter by volume ids * master: carry the volume ids VolumeList asks about in one repeated field One id and a list of them ask the same question, so field 2 holds the list rather than standing beside a second field that supersedes it. Claude-Session: https://claude.ai/code/session_011qAmAdhrYvnzGkw7A9N4mP --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
3967ca23be | rust: cover the READS scrub reconstruction path (#11027) | ||
|
Chris LuandLisandro Pin
|
fcc2ea61d3 |
ec: scrub a volume through its parity data (#11006)
* Introduce a new `READS` scrub mode. `READS` performs a full volume scrub but, unlike `FULL`, it will attempt to reconstruct data for missing/damaged shard intervals from other shards in the cluster when necessary. The goal of this check is to ensure that EC volume contents _are readable by Seaweed_ even on a degraded storage state, by exercising parity data which is not read in `FULL` mode. This is useful not only to validate data is user-readable, but also to detect potential parity shard issues which may be difficult to pinpoint otherwise - particularly for older volumes lacking sidecar data, and hence unaffected by `CHECKSUM` scrubs. For regular volumes, this operation is equivalent to `FULL`. Example: ``` > ec.shard.unmount --volumeId=1 --shardId=0,3,11 --delete --apply Live shard topology for volume ID 1 (14 shards): 0@10.200.18.89:9001 1@10.200.18.89:9002 2@10.200.18.89:9003 3@10.200.18.89:9004 4@10.200.18.89:9005 5@10.200.18.89:9006 6@10.200.18.89:9007 7@10.200.18.89:9008 8@10.200.18.89:9009 9@10.200.18.89:9013 10@10.200.18.89:9010 11@10.200.18.89:9011 12@10.200.18.89:9012 13@10.200.18.89:9020 Will unmount + delete 3 shard(s): 0@10.200.18.89:9001 3@10.200.18.89:9004 11@10.200.18.89:9011 Unmounting shard 0@10.200.18.89:9001 for volume ID 1... Deleting shard 0@10.200.18.89:9001 for volume ID 1... Unmounting shard 3@10.200.18.89:9004 for volume ID 1... Deleting shard 3@10.200.18.89:9004 for volume ID 1... Unmounting shard 11@10.200.18.89:9011 for volume ID 1... Deleting shard 11@10.200.18.89:9011 for volume ID 1... All done! > ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=full using FULL mode Scrubbing 10.200.18.89:9002 (1/1)... Scrubbed 6 EC files and 1 volumes on 1 nodes Got scrub failures on 1 EC volumes and 1 EC shards :( Affected volumes: 10.200.18.89:9002:1 Affected shards: 10.200.18.89:9002:1:0 > ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=reads using READS mode Scrubbing 10.200.18.89:9002 (1/1)... Scrubbed 6 EC files and 1 volumes on 1 nodes ``` * ec: report the shards a READS scrub had to rebuild A READS scrub that recovers an interval was recording nothing, so a volume missing three shards came back clean and nobody repaired it. The unreadable shard is now recorded before the rebuild is attempted: READS reports the same broken shards as FULL and differs only in whether the needles themselves failed, which is the signal worth having - shards are gone, data is still there. forceDeletedNeedlesCheck now applies to READS as well, in the shell and in the RPC guard: it runs the same needle walk as FULL. Regenerated the proto instead of hand-editing it, so the pancis typo (which protoc-gen-go-grpc emits into eight other files here) and the header whitespace stay as generated. Mirrors into the Rust volume server, which also now honors force_deleted_needles_check rather than hardcoding it off. Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq * ec: answer a deleted needle from a READS rebuild as deleted #11020 gave the Rust recovery a deleted flag alongside its bytes, and it answers a deleted needle with no bytes at all. The READS scrub appended that empty answer, which does not compile against the new signature and, once it did, would leave the needle short and report the size mismatch as damage. Zero-fill the interval instead, the way the direct read beside it already does: the assembled needle then reaches read_bytes as the delete-state mismatch the walk already tolerates. Go takes the same branch off the flag its recovery returns, rather than discarding it. Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq --------- Co-authored-by: Lisandro Pin <lisandro.pin@proton.ch> |
||
|
Chris Lu
|
ba5b14b457 |
master, filer, s3api: bound the collection deletes that strand a caller (#11026)
* master: bound each volume server DeleteCollection, and finish the fan-out A collection delete fanned out to every volume server holding it with context.Background(), so a server that accepted the connection and then went quiet held the whole delete open with nothing to end it. Each RPC is bounded now, on the same budget allocateVolumeTimeout gives the other master-to-volume-server admin RPC. The volume server runs the delete to completion regardless of the request context, so giving up costs the confirmation and not the deletion. The walk itself is the caller's, not a per-server one: - It outlives the caller. A cancelled request must not abandon a destructive fan-out part-done, with volumes left behind and no request still running to come back for them. - It no longer stops at the first server that refuses, which left the collection on every server after it in the list. The first failure is still what is reported, and the collection stays in the topology so a later delete comes back for the rest. - It sends one RPC per server rather than one per replica. ListVolumeServers reports a node once for every replica it holds, while DeleteCollection removes the whole collection from the server it reaches, so a collection with thousands of volumes repeated the same whole-collection delete thousands of times over. Both passes run too. Returning after a failed normal pass left the collection's EC shards in place with nothing left to retry them. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * master: delete the EC shards behind /col/delete too The HTTP handler carried its own copy of the volume-server walk and only ever ran the normal pass, so a collection deleted through it kept its EC shards. It shares the gRPC path now, which also gets it the bounded RPCs and the one-per-server fan-out. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * filer: bound the collection delete a bucket delete leaves behind Deleting a bucket entry deletes its collection afterwards, deliberately detached from the request so a client that hangs up cannot strand the bucket's volumes. Detached meant unbounded, though: with the master down or mid-election the wait for a leader has nothing to end it, so the handler parks, and the client retrying behind it parks another. It keeps outliving the request and now carries a deadline of its own. The budget bounds the wait, not the work: the master keeps deleting on its own fan-out once asked, so giving up costs the confirmation. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * s3api: bound the collection RPCs a bucket creation and deletion issue Neither carried a deadline, so a transient failure anywhere down the chain held the S3 request open until the client gave up on it. Both budgets are taken outside the filer failover walk, so one budget covers the whole walk rather than granting each filer a fresh one. The walk itself stops when that budget is spent, and stops without blaming anyone: the caller's own expiry is not evidence against the filer that was answering, and the next filer has no time left to answer in either. Recorded as a filer failure, a slow master upstream would flag every filer in the walk, and the three failures that open the circuit take unrelated object reads down with them. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * s3api: a failed collection listing no longer fails a bucket creation PutBucket lists collections to notice a leftover one it is about to reuse. The result feeds a warning and nothing else -- s3a.exists is what decides whether the bucket already exists -- yet a transient failure of that listing returned 500 and refused the creation. It is advisory now, so a failure is logged and the creation continues, exactly as it does when the listing returns false. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP |
||
|
Chris Lu
|
7dc3835b02 |
s3: an abort answered mid-part no longer leaves the upload completable (#11025)
* s3: reject a part whose upload was aborted while its body was in flight The upload-exists check runs before the part body is read. An abort answered during the read deletes the upload directory, and the part write that follows re-creates it, so the aborted upload is listed nowhere yet completes. Re-check after the write: only createMultipartUpload stamps the destination key on .uploads/<id>, so a directory without it is one the part write resurrected. Drop it along with the part and answer NoSuchUpload. Claude-Session: https://claude.ai/code/session_01ByZ49KQdUtHhmjG6SpNczT * s3: reject a copied part whose upload was aborted mid-copy UploadPartCopy has the same window as UploadPart: the upload-exists check runs before the bytes are copied, and the part write that follows re-creates the directory an abort removed. Both the re-encryption and the raw-copy path re-check before answering. Claude-Session: https://claude.ai/code/session_01ByZ49KQdUtHhmjG6SpNczT * s3: do not complete an upload whose directory holds no upload record A .uploads/<id> directory that a part write created rather than createMultipartUpload carries no destination key, no owner and no encryption settings. Completing one turned stray parts into an object; answer NoSuchUpload instead. Claude-Session: https://claude.ai/code/session_01ByZ49KQdUtHhmjG6SpNczT * s3: log the part left behind when the resurrected directory survives abortMultipartUpload can fail to remove what the part write re-created. The client still hears NoSuchUpload, since the upload is gone either way and a retry would only write another part, but the leftover is worth a line. Claude-Session: https://claude.ai/code/session_01ByZ49KQdUtHhmjG6SpNczT |
||
|
Chris Lu
|
c858e01a09 |
ec: split the shard-interval recovery into a gather and a rebuild (#11005)
* ec: split the shard-interval recovery into a gather and a rebuild Recovering an interval is now one function doing the local seeding, the waved peer fetch, the shard accounting and the Reed-Solomon rebuild, under a memory budget. Splitting the gather from the rebuild makes the rebuild a plain function over a set of intervals, which is testable on its own and reusable by the parity checks a full scrub wants. The rebuild refuses a parity target, and the caller checks that before the gather so a doomed target costs no fan-out. ReconstructData rebuilds data shards only, so asking it for a parity shard returned no error and left the slot nil, and the caller copied that out as a successful read of zeroes. Only data shard ids reach here today, so this is a guard, not a live fix. Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq * ec: rebuild only the EC shard the read asked for ReconstructData rebuilds every missing data shard. The gather stops as soon as DataShards intervals are in hand, so on a distributed volume it routinely finishes holding parity where data is missing -- and each of those data shards is then rebuilt into an interval-sized buffer, decoded, and never read. Ask for the one shard the read needs. The budget covers it now too: DataShards gathered plus the one the rebuild allocates. It never covered the rebuild's output, and with ReconstructData that output was up to ParityShards buffers. The required mask is Total() long rather than DataShards. reedsolomon documents both lengths, but its presence scan walks every shard and indexes the short mask past its end, so the documented short form panics whenever a parity shard is absent - which here it usually is. Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq |
||
|
Chris Lu
|
cd5013f116 |
Re-check an EC shard map a failed read has disproved (#11023)
* Re-check an EC shard map a failed read has disproved A read that fails against a cached location drops that shard from the map, which leaves it one short of complete -- and a map one short is trusted for seven more minutes. So a moment's trouble between volume servers cost minutes in which every read of that shard skipped the direct fetch and paid for a Reed-Solomon recovery instead, at DataShards times the memory and the peer load. Mark the map when a read disproves it, and re-check a marked map on the same eleven-second footing as one that never had enough shards to begin with. The mark clears on refresh, so it buys one prompt re-check rather than a master lookup per read. The tiers move into a helper; they were three overlapping conditions in one expression, and the reading of them was not obvious. Rust keeps the entry rather than dropping it -- a dead peer fails fast on the next attempt, and it was the freshness window, not the entry, hiding a shard that had moved. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Invalidate the location of an EC shard whose own read failed Recovery fans out to the other shards, so the one whose direct read just failed is the only location nothing ever invalidates: a shard that moved to another server was reconstructed on every read until the map's own window expired, up to thirty-seven minutes for a map still complete. Mark the map there too. The entry stays -- a moved shard's old holder fails fast, and the next refresh is seconds away. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Consume the stale mark before the lookup, not after A read that fails while the master is answering has disproved the very map that answer is about to install, and clearing the mark on the refresh's return swallowed it. Clear it where it is acted on instead. A lookup that then fails loses the mark, which costs nothing: the refresh time is only advanced on success, so the next read looks up regardless. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Judge the shard map and consume its mark in one critical section Reading the mark and clearing it were two separate acquisitions, so a mark raised between them was cleared by a refresh that had not seen it. In Go that gap was a few instructions; in Rust the mark was read when the read first snapshotted the volume and cleared at the decision point, with the local interval reads in between. Take both under one hold. Rust needs a mutex rather than an atomic to do it, and no longer carries the mark through the snapshot. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Put the stale mark back when the lookup does not answer for it Consuming the mark up front assumed the lookup would supersede it. A lookup that fails, or comes back with fewer than DataShards holders, supersedes nothing: the map is unchanged, its refresh time unadvanced, and with the mark gone the map a read had disproved is trusted for its full window again on the strength of a lookup that never landed. Put the mark back on both branches. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN |
||
|
Chris Lu
|
624deaf3a4 |
mount: implement fallocate (#11021)
* mount: implement fallocate instead of reporting it unsupported Fallocate answered ENOSYS, so the kernel marked the mount as having no fallocate and returned EOPNOTSUPP. glibc then fell back to its emulation, which preads a byte from every block already inside the file to see if it is allocated; on a write-only descriptor that pread is EBADF, and posix_fallocate returned it. Volume space is assigned when a write is flushed, so nothing can be reserved up front: a range inside the file is answered OK untouched, and one past the end grows the file the way a truncate would. A mode we cannot honor is refused with ENOTSUP, not ENOSYS, so the kernel keeps sending the ones we do. Claude-Session: https://claude.ai/code/session_01XG6sAAdkqTwWffqK5h4rH9 * mount: let a fallocate that allocates nothing past the quota and worm guards A range already inside the file, and any FALLOC_FL_KEEP_SIZE request, reserve no space and rewrite no entry, but the preflight refused them with ENOSPC on a full mount and EPERM on a worm-enforced file. Decide the no-op first and guard only the growth. Claude-Session: https://claude.ai/code/session_01XG6sAAdkqTwWffqK5h4rH9 * mount: charge a fallocate growth to the uncommitted byte counter Write charges the counter by how much the file grew, so the writes that fill a range fallocate already extended charge nothing and the real-time quota check never sees that data — only the periodic filer refresh does. Count the growth where it happens. Claude-Session: https://claude.ai/code/session_01XG6sAAdkqTwWffqK5h4rH9 * mount: charge a truncate-up growth to the uncommitted byte counter Same gap Fallocate had: Write charges the counter by how much the file grew, so the writes that fill a range ftruncate already extended charge nothing and the real-time quota check never sees that data. Count the growth where it happens; a shrink still leaves the counter alone, since it is only ever raised and then reset by the periodic filer refresh. Claude-Session: https://claude.ai/code/session_01XG6sAAdkqTwWffqK5h4rH9 |
||
|
Chris Lu
|
7bb0a1c127 |
s3: replay a delete whose reply the transport dropped (#11022)
* s3: stop retrying a delete the filer refused for a non-empty folder The filer looked and the children are there, so the answer will not change. retryFilerOp spent six attempts and up to 3.1s of backoff on it before the caller could act on the condition it was already holding. Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD * s3: thread the request context through the unversioned delete path doDeleteEntry issued every DeleteEntry on context.Background(), so an S3 client that hung up left the gateway working on its behalf, out of reach of both cancellation and the per-request retry allowance that DeleteMultipleObjectsHandler installs. Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD * s3: treat a cancelled filer RPC as terminal, not transient isRetryableFilerErr matched context.Canceled and DeadlineExceeded by sentinel, which only holds while the error is still local. Once it has crossed gRPC it is a status, so an abandoned request was retried six times on behalf of a caller that had already gone. Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD * s3: replay a delete whose reply the transport dropped A delete is idempotent at the filer, which answers an entry that is already gone with an empty resp.Error, so a reply lost in transit can be reissued rather than surfaced. Surfaced, it becomes a 500 on the bucket delete, which boto3 resends and is then answered NoSuchBucket, or a per-key InternalError inside the 200 of a multi-object delete, which no SDK retries at all. The replay runs through retryFilerOp, so it draws on the allowance the request already installs rather than paying a backoff per key, and stops for a caller that has gone. rm and rmObject re-enter WithFilerClient per attempt, so each one walks the failover list again on a connection the failed attempt had invalidated; the multi-object loop holds one client for the batch, so there the replay reuses it. Classification stays structural. The filer reports its own refusals in resp.Error, which carries no status and has the deleted path - and, for a recursive delete, the children it stopped on - formatted into it, so no key name can steer the decision either way. rm and rmObject now take the caller's context. Cleanup and rollback paths pass context.Background() deliberately: they have to run whether or not the caller is still waiting. Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD * s3: share one retry allowance across multipart completion cleanup The unused-entry loop deletes once per entry, and each delete now retries, so a filer that stays unavailable held the response for 3.1s per entry after the object was already committed. Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD |
||
|
Chris Lu
|
af6f69740c |
Read metadata log chunks the way the mount reads every other chunk (#11018)
* Replay metadata log chunks the way the mount reads every other chunk The subscription's log-chunk replay built its own lookup, which always resolves volume server addresses. A mount started with -volumeServerAccess=filerProxy cannot reach those, so every fresh subscription failed on the previous minute's persisted segment and resubscribed a second later, forever. Take the lookup from the caller instead; the mount hands over the one it uses for file reads, which also keeps publicUrl and the bounded location cache in play. Claude-Session: https://claude.ai/code/session_01NGqrxYj7cHUpSrL249n3Z6 * Keep a log chunk read failure off the filer connection A metadata subscriber reads persisted log chunks over HTTP from volume servers and hands whatever went wrong back as the subscription's error. "connection refused" from a volume server then matched the transport patterns that decide a gRPC channel is dead, so every failed replay closed the shared filer ClientConn and cancelled the assign and upload RPCs riding on it with "the client connection is closing". Mark those read failures so they are judged for what they are. Claude-Session: https://claude.ai/code/session_01NGqrxYj7cHUpSrL249n3Z6 |
||
|
Chris Lu
|
95248f7492 |
Bound the memory an EC shard recovery holds (#11020)
* Reconstruct an EC shard from the shards already on this server recoverOneRemoteEcShardInterval only ever fanned out to the cached shard locations, so a server holding shards of the volume still fetched them over gRPC from itself -- and when the peers were unreachable it could not reconstruct at all, even holding the whole volume on local disk. Seed the Reed-Solomon buffers from the locally mounted shards first; each one is a peer round trip, and an interval-sized buffer, the fan-out no longer needs. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Fetch only the EC shards reconstruction still needs The recovery fan-out read every surviving shard, so a 10+4 volume pulled 13 interval-sized buffers to feed Reed-Solomon 10 -- a third more memory held, and a third more load asked of peers that were, by definition, already having trouble. Fetch what is missing, and widen only when some of those reads fail. A shard reporting the needle deleted ends the walk: the rest would only answer the same. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Bound the bytes EC recovery holds in flight Recovery is the one read path that multiplies the served bytes: it holds an interval-sized buffer per shard until Reed-Solomon runs, and a peer that is slow to fail keeps them all alive for the whole gRPC timeout. Nothing bounded how many of those fan-outs ran at once, so a transient problem between volume servers turned every read into a DataShards-fold allocation and the server died of it -- 64 concurrent 4MB intervals pin 3.6GB, and that is a small burst. Charge each recovery against a process-wide budget, so a burst queues on the semaphore instead of on the heap. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Answer a deleted EC needle as deleted, not as a failed recovery A holder reporting the needle deleted is authoritative: deletes are never invented and never undone. Recovery already collected that flag, then dropped it on the branch where too few shards came back -- so a read of a deleted needle that had to recover surfaced as "cannot recover shard", and the volume server answered 500 where it owed a 404. Carry the flag out of the shortfall, and let it decide ahead of the error it came with. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Check the encode run of a locally seeded EC shard in Rust The Rust recovery seeded Reed-Solomon straight from the mounted shards, without the encode-run check the remote reads and Go's readLocalEcShardInterval both apply. A volume remounted from a newer encode between the read's snapshot and its recovery would have fed mixed-generation bytes into the reconstruction. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Say what the recovery budget actually guarantees Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Seed Rust EC recovery from shards on every local disk find_ec_volume returns the first disk's EcVolume, so a reconciled volume whose shards are split across data dirs had the siblings ignored and could report "cannot recover" while holding enough shards locally. Resolve each shard together with the disk that owns it, the way Go's recovery already does, and check that owner's encode run. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN |
||
|
Chris Lu
|
23241cf0f1 |
Let filer.sync move past a chunk the source cluster no longer has (#11019)
* Name the failure when the source cluster cannot locate a chunk's volume LookupFileId formatted a nil err into the message it returned, so the only thing a caller could do with "no locations for this volume" was match on the text. Return a typed error instead. Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK * Fail a source chunk read on a failure status instead of copying the error page ReadPart never looked at the response status, so a volume server answering 404 for a needle vacuum had removed came back as a successful read whose body was the error page. The caller counted those bytes as file content and reported a size mismatch — a corruption claim about data the source had simply lost — and a 404 from one replica ended the search instead of trying the next. Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK * Stop retrying a chunk the source cluster can no longer produce A chunk whose volume vacuum has removed fails the same way on every attempt, but the retry loop had no way to say so and kept going forever. The sync job holding it never finished, so it pinned the offset watermark at the event ahead of it and filer.sync never checkpointed again — alive, quiet, and permanently behind. Wait the source out for a grace period long enough to cover a volume server restart or a master failover, then give up and mark the failure permanent. Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK * Let replication continue past an entry whose source data is gone An entry the source can no longer read holds the sync offset forever: the event fails on every replay, so the checkpoint never moves past it and every later event stays uncheckpointed, however long the sync keeps running. Nothing brings those bytes back, so skip the entry with an error naming it and carry on. Skip only while the source is demonstrably still serving other chunks. A volume with no locations reads the same whether it was vacuumed away or every replica is down, and during a cluster-wide outage that answer comes back for every chunk — skipping then would drop live files wholesale. Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK * Propagate a missing source chunk instead of waiting when supersession is unverifiable An incremental sink's dated target keys cannot be mapped back to a source path, so nothing here can tell a chunk the source lost from one a later version already replaced. Waiting out the grace period would stall every vacuumed needle for half an hour; hand the failure to the caller, which has the event's real source key. Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK * Wait out a gone volume once, not once per file it held A volume vacuum removed took every file it held with it, and each chunk was timing its own grace period. With a bounded chunk executor those waits serialize, so one gone volume holding many files stalls the sync for far longer than the grace period — the wedge again, only slower. Track the wait per source volume on the sink instead: the first chunk to find it unlocatable starts the clock, every later chunk inherits it and gives up as soon as it has run out, and a chunk the source does serve clears it. Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK * Probe the source with a read, not a lookup, before writing an entry off A lookup only proves the source master still has the topology. If every volume server is unreachable while the master still lists them, the probe passed and the sink wrote off an entry whose data was merely out of reach. Read the probe chunk instead, and say in the log that the entry stays unreplicated. Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK |
||
|
Chris Lu
|
60893c5ef3 |
Classify a filer error before a user-controlled path is wrapped into it (#11004)
* util, pb: classify a filer error by the status the server sent DoSeaweedListWithSnapshot wrapped a failed ListEntries with %v, dropping the gRPC status, so IsTransientError fell back to matching substrings against a message that now held the caller's path. Keep the status with %w and let it decide, reading the server's own text rather than the wrapper's. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * s3: keep the bucket and prefix out of the list retry decision A bucket named transport, or a prefix under logs/unavailable/, made a PermissionDenied listing look transient and got it retried; a key holding the not-found sentence suppressed a retry that should have run. Both checks now read the filer's status, and only fall back to the text when there is none. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * filer, s3: classify a delete failure before the path is wrapped into it The filer put the non-empty-folder marker behind its own "delete directory %s" wrapper and the gateway matched it as a substring, so a key named after the marker turned a real delete failure into the demote-the-marker no-op and the request answered 204. Keep the marker leading the message that crosses the wire, turn it back into a sentinel where the response is read, and match that. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU |
||
|
Chris Lu
|
9e06e1d0f9 |
Report a delete the filer rejected instead of answering success (#11003)
* s3tables: report a delete the filer rejected deleteDirectory discarded DeleteEntryResponse and checked only the transport error, so DeleteTable, DeleteNamespace, DeleteView and DeleteTableBucket answered 200 for a delete the filer refused. Call filer_pb.DoRemove, which reads resp.Error and still treats a missing entry as success. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * admin: report a delete the filer rejected The bucket delete, the file browser handlers and the topic retention purger all discarded DeleteEntryResponse, so a delete the filer refused came back as success. Call filer_pb.DoRemove, which reads resp.Error. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * credential: report a delete the filer rejected DeleteUser, DeletePolicy and the full-sync cleanup loops discarded DeleteEntryResponse, so a rejected delete answered success and left the credential file in place. The service account path in the same store already read resp.Error; the rest now do too, via filer_pb.DoRemove where not-found is already tolerated. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * shell: report a delete the filer rejected remote.configure -delete, remote.cache and the remote metadata sync discarded DeleteEntryResponse, so a rejected delete printed as removed. Call filer_pb.DoRemove, which reads resp.Error. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * mq: report a delete the filer rejected The consumer offset group purge and the coordinator assignment delete discarded DeleteEntryResponse. Call filer_pb.DoRemove, which reads resp.Error. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * iam: count only the revocation entries the filer actually deleted The expiry sweep discarded DeleteEntryResponse, so a rejected delete was counted as purged and the entry stayed. Call filer_pb.DoRemove, which reads resp.Error, matching the role and provider stores beside it. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * mount: fail rmdir when the unary fallback delete was rejected The streaming branch turns DeleteEntryResponse.Error into an error, the unary fallback dropped it, so rmdir of a non-empty directory answered OK off the stream and ENOTEMPTY on it. Surface it in both. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * s3tables: fail DeleteTableBucket when the directory delete is refused The handler only failed when both the leaf entry and the directory delete failed, so a refused bucket directory delete still answered 200 with the bucket in place. The directory is the bucket, so it decides; the leaf entry stays best-effort. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU |
||
|
Chris Lu
|
742b2f5896 |
s3: share one retry allowance across a batch delete (#11001)
Every key in a multi-object delete drives its own retryFilerOp, so a filer that is briefly unhealthy multiplied one op's ~3.1s of backoff by a key count the client picks. The batch now carries a single allowance in its context, sized to one op's worst case; once it is spent the remaining keys fail fast with a per-key error instead of holding the request goroutine. A single-object delete carries no allowance and keeps its full retries. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU |
||
|
Chris Lu
|
902a12fd6f |
wdclient: bound the wait for a master leader by the caller's context (#11002)
* wdclient: bound the wait for a master leader by the caller's context WithClient waited on GetMaster with context.Background(), so a caller that arrived while no master leader was known parked in a 200ms poll loop until one appeared, whatever deadline it had already set on the RPC. Each retry above it then left another goroutine in the same wait. Take the context in WithClient and WithClientCustomGetMaster and hand it to GetMaster, and stop the retry loop once it is done. The dial keeps context.Background(): fn brings its own RPC context, so a cancellation seen here cannot be attributed to the shared connection. Call sites pass whatever they hold: the request context in the filer's CollectionList, DeleteCollection and Statistics handlers and in the credential store's propagation, the operation context in the shell's s3.bucket.delete and the kafka gateway's broker and filer discovery, and context.Background() where there is none - the shell commands, the admin dashboard wrapper, and the exclusive locker's initial lease. The locker's release keeps its own uncancelled context so a slow unlock cannot turn into a ghost lock. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * wdclient: test that WithClient gives up with the caller's context Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * wdclient: cut the master retry backoff short when the caller gives up util.Retry sleeps unconditionally between attempts, so a transient error arriving just before the caller's deadline still cost it a full backoff step. Use the context-aware util.RetryWithBackoff, the same helper the volume lookup in this file already uses. Two call sites went with it: the shell's lock-holder lookup builds its three second bound before WithClient so it also covers finding the leader, as its comment already promised, and the filer's post-delete collection cleanup goes back to an uncancelled context - the entry is already gone, so a caller that hung up must not leave the collection behind. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * wdclient: test that a cancel during backoff ends the retry Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU |
||
|
Chris Lu
|
d850f36513 |
s3: distinguish a failed bucket lookup from a missing bucket on HEAD (#11000)
HeadBucket treated any lookup error as ErrNoSuchBucket, so a transient filer failure answered 404 instead of 500 and clients stopped retrying. Split the two cases the way the bucket policy handlers already do. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU |
||
|
Chris Lu
|
eed3c27d15 |
volume: cut the memory a server holding millions of volumes still uses (#10999)
* volume: stop the .vif guard depending on which entry the scan handed over A volume has both an .idx and a .vif, and loadExistingVolume skipped a .vif next to an .ecx as EC shard metadata. That was only ever correct because os.ReadDir sorted .idx ahead of .vif: an interrupted encode, where the .idx is still there, has to reach validateEcVolume to be reclaimed. Ask for the .idx instead of trusting the order. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * volume: walk volume directories in batches instead of listing them whole os.ReadDir builds, and sorts, a slice of every entry before the caller sees the first one. A disk holding millions of volumes has a .dat, .idx and .vif per volume, so each startup scan costs hundreds of MB of peak heap that the runtime is slow to hand back -- and there are several of them before the first volume loads. Walk in batches instead, and keep only the entries each scan acts on: loadAllEcShards now sorts and stats the shard and index files alone rather than every file on the disk. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * volume: skip the sibling-.dat scan when no EC volume is loaded pruneIncompleteEcWithSiblingDat only ever prunes EC volumes that are loaded, but it first walks every disk and keys a map by every .dat on the server. On a store with no EC volumes at all that is millions of map entries built to answer no question. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * volume: stop keeping a departure message for every volume The report state held a VolumeShortInformationMessage per volume copy so a departure could be named, but almost no volume ever departs. Hold a handle to the identity instead -- volumes share very few distinct ones -- and build the message on the way out. Measured over a populated report state: 195 -> 83 bytes per volume. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * rust volume: stop keeping a whole volume message per volume held The send loop kept a VolumeInformationMessage for every volume just to notice mounts and unmounts, and rebuilt the map from scratch on every beat. Keep the identity a delta names, which is what the Go report state keeps for the same reason. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * rust volume: keep only the EC files the shard scan acts on load_all_ec_shards named every file on the disk twice -- once in the dedup set and once in the sorted vector -- before deciding it only wanted .ec?? and .ecx. Filter while reading instead. Mirrors the same change in loadAllEcShards. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * volume: share the strings every .vif repeats A tiered volume's .vif names its replication and its backend, and every decode allocates a fresh copy, so a server holding millions of them holds millions of copies of the same handful of names. Route them through the interning table the volume info decode already uses. The remote key names one volume and is left alone. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy |
||
|
Chris Lu
|
fdd8bd9478 |
s3: reject a request that names two operations (#10987)
The router matches bucket subresource routes in registration order while the IAM action resolver matches its own list in a different order, so a request carrying two operation subresources is authorized as one operation and served as another. `PUT /bucket?policy&tagging` resolves to s3:PutBucketTagging and runs PutBucketPolicy, letting an identity delegated bucket tagging install an arbitrary bucket policy. The same mismatch reaches PutBucketCors, PutBucketLifecycle, PutBucketVersioning, PutObjectLockConfiguration, PutBucketRequestPayment and the policy and cors deletes. Reject the ambiguity where the other pre-routing checks live, so neither list has to stay in step with the other. Keys that modify an operation rather than select one -- versionId, partNumber, prefix -- still combine freely. |
||
|
Chris Lu
|
99cf7a66df |
shell: remove the directories emptied by volume.fsck's filer entry purge (#10992)
* shell: remove the directories emptied by volume.fsck's filer entry purge volume.fsck -findMissingChunksInFiler -reallyDeleteFilerEntries deleted the orphan entries but left their parent directories behind, so a namespace accumulated empty directories that had to be cleaned up by hand. Remember the parent of every purged entry and, once the purge is done, walk up from each one deleting the directories that are now empty. The delete is non-recursive, so the filer itself rejects a directory that still has children; a bucket and a directory that is an S3 object of its own are left alone. Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc * shell: keep a directory volume.fsck saw change under it The empty-directory sweep read the entry to spot an S3 directory key object and then deleted unconditionally, so a directory promoted to an object in between was removed anyway. Delete with the mtime the lookup returned, leaving the filer to skip a directory that has changed since. Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc * shell: leave a directory volume.fsck just saw written for the next run The mtime the delete is conditioned on has second resolution, so a write landing in the same second as the one already on the directory is indistinguishable from it and the directory would still be deleted. Skip a directory modified within the last few seconds. A write after the lookup then always carries a later second than the one the delete carries, and the sweep picks the directory up on the next run. Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc * shell: skip a directory volume.fsck cannot condition a delete on A zero mtime disables the delete's condition at the filer, so a directory whose entry carries none was removed unconditionally and a concurrent promotion to an S3 object went with it. Leave such a directory alone. Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc * shell: hold volume.fsck's quiet period to the cutoff second itself Mtime keeps whole seconds, so a directory whose mtime lands on the cutoff second was written up to a second after it. Skip that directory too, so the quiet period fails closed. Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc |
||
|
Chris Lu
|
2a97e08caa |
s3: cover the directory marker key with object lock (#10988)
* s3: enforce object lock when deleting a directory marker The key "dir/" is deleted the unversioned way, ahead of the branches that enforce Object Lock, so a principal with plain delete permission could remove a key the gateway was reporting as COMPLIANCE-retained -- retention set through PutObjectRetention is stored on the directory entry and served back by GetObjectRetention, only the delete ignored it. The same path also takes any key ending in "/" regardless of size, while a PUT only makes a marker of one up to 1KiB. A larger one is a genuine versioned object, and deleting it here dropped its whole history after the versioned delete of the same key had been refused. Enforce in the marker delete itself, so the single, versioned and multi-object delete paths are all covered. * s3: apply object lock headers on a directory marker PUT The trailing-slash branch runs before the versioning and Object Lock handling, so it accepted x-amz-object-lock-* headers and stored none of them: a bucket owner could believe a key was retained while nothing recorded it, and an invalid mode or a past retention date that a regular key rejects came back 200 here. Validate the headers the way the regular path does, store what they ask for beside the owner the same callback already sets, and refuse to replace a key that is already retained. * s3: check every version a marker delete would remove The marker delete clears any history under the key in one recursive removal, while the lock check ahead of it resolves the latest version only. A version retained under an unretained one was taken with the rest, so enforce against each version the removal covers. * test: pin the marker lock refusals to AccessDenied A bare require.Error passes on any failure, including one that has nothing to do with the lock. Assert the code, the key the batch delete reports, and that the marker survives each refusal. * s3: check the history entries a version list leaves out The version list skips an entry without a version id, while the removal takes it with the rest, so an entry an older build left unnamed escaped the check. Walk the history directly instead, and refuse when an unnamed entry is still under a retention or a legal hold of its own. * s3: let a governance bypass reach an unnamed history entry The unnamed branch refused every active retention, so a caller allowed to bypass governance could not clear one, which the named path lets through. Refuse a legal hold and compliance mode as before, and take the bypass into account for governance. * s3: keep the object lock decision in one place The unnamed history entry had to repeat the retention and legal hold rules inline because the enforcement helper only takes a key to look up. Split the part that judges an entry out of it and call that from both. * s3: guard a marker PUT on the entry it replaces The overwrite check resolved the key's latest version, but mkdir builds a fresh entry for the marker itself, dropping the lock metadata the old one carried. Once the key had a history, an unlocked version answered for a retained marker and a plain PUT replaced it. Judge the entry the write is about to replace instead; a versioned write of the same key still adds a version, which is its own to allow. * s3: guard a marker delete on the entry it removes The check ran against the key rather than the entry, so once the key had a history it answered with a version and the retention recorded on the marker itself went unseen. Judge the entry that is about to be removed, the same way the PUT side now does; the versions under it are still covered by the walk that follows. * s3: take the object write lock for a marker PUT The overwrite check read the entry that the mkdir after it replaces, so two marker PUTs could both pass while one was still unlocked. The marker delete already runs under this lock; hold it across the check and the mkdir so the entry cannot change in between, and so the two paths are serialized against each other. |
||
|
Chris Lu
|
ab8b34720a |
s3tables: delete only the location the dropped table owns (#10986)
DeleteTable authorizes the named table, then recursively purges the data path derived from its stored MetadataLocation. That location is supplied by the caller at create/register time and never bound to the table, so a tenant allowed to drop one table could point it at a table in a sibling namespace and have the delete destroy that table's catalog entry and data files. A legitimately decoupled location -- a rename source, or a leftover the name was reused over -- has had its catalog attributes stripped, so a surviving metadata marker identifies a path that belongs to another entry. Refuse those, alongside the existing ancestor refusal. |
||
|
Chris Lu
|
0b5fff2ccd |
filer, s3: reuse the volume server's guarded remote-storage client builder (#10990)
* volume: build the guarded remote storage client through a shared helper Fold the endpoint validation, credential check and rebinding-safe dialer that FetchAndWriteNeedle applies before dialing a caller-supplied remote storage endpoint into a single BuildGuardedRemoteStorageClient helper, so other callers that dial the same endpoints can reuse it. No behavior change on this path. Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN * filer: build the remote-mount stream client through the guarded helper streamFromRemote serves a cold remote-only entry straight from its mounted origin. Build its client through BuildGuardedRemoteStorageClient so the same endpoint checks the volume server applies cover this read path too. Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN * s3: build the remote-mount stream client through the guarded helper openRemoteStream serves a remote-mounted object straight from its origin when the local read cannot. Build its client through the same guarded helper so the endpoint checks apply here as well. Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN |
||
|
Chris Lu
|
28862c866e |
Authorize an Iceberg table create before it writes (#10991)
* s3tables: share one CreateTable authorization gate CreateTable and RegisterTable each carried their own copy of the name validation, policy load and permission check. Fold them into authorizeCreateTable, and expose it on the Manager for callers that write into a table bucket before the table itself is registered. Claude-Session: https://claude.ai/code/session_01QiJkka1T2NAWDWq4JQ8Vuy * iceberg: authorize a table create before it writes Stage-create returns before the S3Tables registration that authorizes a create, and the plain create writes its metadata file before reaching it, so a caller who may not create the table could still leave a staged template, a marker and a v1.metadata.json in the target bucket - and get vended credentials for a location of their choosing. Run the CreateTable gate as soon as the table is known to be absent. Claude-Session: https://claude.ai/code/session_01QiJkka1T2NAWDWq4JQ8Vuy * iceberg: authorize a create-on-commit the same way A commit against a table that does not exist creates it, writing the metadata file first and only then reaching the registration that checks the caller may create it. Denied callers saw a 500 for what is a 403. Claude-Session: https://claude.ai/code/session_01QiJkka1T2NAWDWq4JQ8Vuy * iceberg: pin that identity actions reach the create gate The manager request is built from the caller's own context, so an identity whose actions carry the permission still passes. Worth a test: a fresh context here would silently deny every such caller. Claude-Session: https://claude.ai/code/session_01QiJkka1T2NAWDWq4JQ8Vuy |
||
|
Chris Lu
|
bc06505b40 |
mount: keep metadata operations working on an unlinked open file (#10989)
* mount: serve metadata ops from the open handle of an unlinked file ftruncate on a descriptor whose file was unlinked failed with ENOENT: maybeReadEntry resolved the inode to a path first, and unlink had already dropped it. GetAttr worked around that with its own handle fallback; SetAttr and the xattr handlers had none. Look the handle up first and let it answer whether or not a name still points at the inode. GetAttr keeps reporting nlink 0 there, now off the empty path. Claude-Session: https://claude.ai/code/session_01U1R8BM4bVT46KwPDEj2Ega * mount: read an open handle's attributes under the handle lock too GetAttr held only the LockedEntry lock, which covers the async uploader's chunk appends but not Write or the metadata flush: those rewrite size, times and the whole chunk slice under the handle lock, so FileSize could walk a slice mid-reassignment. The branch this replaced took both locks; take both here, outer handle lock first, as Read and Lseek do. Claude-Session: https://claude.ai/code/session_01U1R8BM4bVT46KwPDEj2Ega * mount: report nlink 0 from SetAttr for an unlinked open file The kernel caches the attributes a SETATTR reply carries, so an ftruncate on an unlinked file left fstat reporting nlink 1 until the cache expired, even though GetAttr had it right. Both replies go through the same rule. Claude-Session: https://claude.ai/code/session_01U1R8BM4bVT46KwPDEj2Ega |
||
|
Chris Lu
|
e9a464840c |
webdav: describe a listed entry the way clients expect (#10993)
* webdav: name the entry, not its path, in a listing DAV:displayname carried the full path of every entry. A client that takes displayname for the child's name - Windows Explorer does - then looks for /dir/name under /dir and finds nothing, so a folder shows up empty while the root, where the two spellings differ only by a leading slash, still lists. Readdir now builds its entries with toFileInfo like stat does, so a listing and a lookup describe a child the same way, and the wrapper that was trimming the sub-folder back off a name goes away with it. Claude-Session: https://claude.ai/code/session_01XCeuCWpF9xo9CfyHvCQE9c * webdav: derive an ETag when nothing hashed the entry Uploads through this gateway carry no content MD5, so filer.ETag comes back empty and every file in a PROPFIND answered with an empty DAV:getetag, which is not a valid entity-tag. Report it as unimplemented instead, the way the sub-folder wrapper already did, and webdav falls back to modification time and size. The wrapper's copy went with it - it swallowed the stat error a caller was meant to see. Claude-Session: https://claude.ai/code/session_01XCeuCWpF9xo9CfyHvCQE9c |