mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-11 17:10:40 +02:00
1971fb8cb0b361cd27df07186ea014c92bd44e49
2044
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Guang Jiong LouandChris Lu
|
f740210235 |
get volume topology info without volume details (#11036)
* get volume topology info without volume details Signed-off-by: lou <alex1988@outlook.com> * master: rename VolumeListRequest.without_volumes to topology_only The field shapes the reply rather than selecting volumes, and it leaves out the ec shards too, which the old name denied. Match the message's *_only style and say what a master that predates the field does with it. Claude-Session: https://claude.ai/code/session_01QHnaNRgxnjzZsiz7WTFML5 * master: refuse topology_only combined with a volume selector A topology_only request that also names a collection or volume ids contradicts itself, and answering either half in silence surprises the caller. Answer InvalidArgument from both VolumeList and its stream, before the stream sends its header. Claude-Session: https://claude.ai/code/session_01QHnaNRgxnjzZsiz7WTFML5 --------- Signed-off-by: lou <alex1988@outlook.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
d3b8030a69 |
master: shed assigns retryably until volume servers register capacity (#11032)
An assign arriving before any volume server has heartbeated saw zero available space and failed outright with a plain error no client retries, so the first write to a fresh bucket answered 500 while the cluster was still starting. Distinguish a topology with no registered capacity from a genuinely full one: fail fast only when registered capacity is exhausted, and shed ResourceExhausted otherwise so the client's retry budget rides out the startup window. Claude-Session: https://claude.ai/code/session_018G9kWFgy8BaBAEkYV3YL9n |
||
|
Chris Lu
|
74b520113e |
ec: pin auto-selected shard placement to the disk that already owns the shard (#11029)
* ec: pin auto-selected shard placement to the disk that already owns the shard A multi-disk server legitimately mounts one EC volume on several disks, so FindEcShardTargetLocation's per-volume tiers tie at "mounted" and the free-shard-count tie-break decides — pointing at whichever disk is emptier, not at the disk that already holds the shard being placed. A re-copy of a shard the server already has (a retried ec.balance / ec.rebuild move) then lands on a sibling disk, and both disks register the same (volume, shard id): the shard is reported to the master from two disk ids, and which claimant serves reads or survives a later unmount/delete becomes an accident of Locations order. Add a tier above "mounted": a disk that already claims one of the shard ids being placed wins, ahead of the space filters too — re-copying in place needs no new shard slot, and a genuinely full disk should fail the write rather than silently split the claim. Applied to the Go selector and the VolumeEcShardsCopy auto-select (ReceiveFile refuses mounted EC volumes, so no claim can exist there) and mirrored in the Rust volume server. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 * ec: refuse a copy batch whose shards are already owned by different disks Review follow-up: ownership-aware selection ranks a mixed-owner batch (shard 0 on disk A, shard 2 on disk B — the legitimate multi-disk spread) into one destination, so the copy would still duplicate the losing disk's claim. No production caller sends such a batch (balance moves one shard, rebuild and encode copy shards the target lacks), so fail closed: report every owning disk via Store.EcShardOwnerDisks and refuse the copy with an error naming them, telling the caller to split per shard or pass disk_id. Go and Rust, with unit tests for the owner-reporting contract. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 |
||
|
Chris Lu
|
88c873ecd4 |
ec: uniform shard block layout (#10932)
* ec: uniform shard block layout An EC volume is striped as 1GiB blocks until less than one row remains, then 1MiB blocks, and consecutive blocks land on different shards. With ec.encode's -fullPercent 95 against the 30GiB default limit, ~30% of every volume sits in that 1MiB tail, so a 4MB filer chunk there is five stripes on five servers. New encodes now use one block per shard, sized ceil(datSize/dataShards) rounded up to 1MiB and recorded in the .vif (EcShardConfig.block_size, also carried by the .ecsum manifest). A needle now maps to one shard unless it is larger than the block or straddles a boundary. The chosen size equals the legacy layout's padded shard length for every input, so shard sizes, capacity math, and the shard-size credibility checks are unchanged; only the byte placement moved. Reads, decode, and scrub resolve the block sizes from the volume's .vif; absence keeps the legacy interpretation, so existing EC volumes read exactly as before. Rebuild is layout-agnostic. weed fix -ecx recovers the layout from the .vif, else the .ecsum sidecar, and with neither de-stripes under both candidate layouts and keeps the one that indexes more valid needles. Same change in the Rust volume server, which now also streams the encode in 256KB sub-batches like Go instead of allocating whole blocks, and computes the large-row count as shardSize/largeBlock to match Go on exact multiples. On a 26MB fixture both encoders produce byte-identical shards, and a Go-written .vif parses in Rust with the block size intact. * ec: resolve the rust ecx rebuild through the recorded layout The Rust rebuild path regenerated a lost .ecx by scanning the logical .dat through a hand-rolled pure-1MiB striping, which was already wrong for legacy volumes with large-block rows and is wrong for any uniform volume with a block past 1MiB. Route the scan through locate_data with the .vif-recorded block size, the same mapping the read path uses. Also seed the new tests' random data instead of the deprecated global math/rand.Read. * ec: fail the Rust ecx rebuild on any shard read error A read error mid-scan published the entries collected so far as a successful .ecx, and read_at's byte count was ignored so a legal short read passed as complete — a truncated or failing shard could produce a silently incomplete recovery index. Exact-read semantics in read_from_data_shards, error propagation in the needle walk, and a truncated-shard regression test. * ec: fail the mount on an unreadable or malformed vif Both servers silently fell back to the legacy layout when an existing .vif could not be read or parsed. Every new encode records a positive uniform block size there, so the fallback mounted the same shards with legacy offset math and could return wrong data. Absent stays legal (legacy volumes predate the sidecar), and a zero-byte stub still reads as absent (Go's MaybeLoadVolumeInfo convention, now mirrored in Rust); a present-but-unreadable or malformed .vif fails the mount instead. * ec: bound the reconstruct fan-out of one needle's intervals A degraded interval fans out a read to every reachable shard location, each with a buffer the size of the interval. Reading a needle's intervals in parallel multiplied that by the interval concurrency: a needle spanning 8 blocks could hold 8 x MaxShardCount remote reads and buffers at once, where the sequential version peaked at MaxShardCount. Give each needle a single reconstruct budget its intervals share, held for the buffer's lifetime, so separate reads stay independent but one read cannot multiply its own fan-out. * ec: drop the duplicated shard-size formula calculateExpectedShardSize reimplemented the padding rule that UniformBlockSize already owns — TestUniformBlockSizeMatchesLegacyShardSize asserts the two agree for every input — so a change to the rule would have had to be made in both. Defer to the helper, keeping the historic answer for an empty .dat. * ec: resolve the shard block layout from whatever records it Four places still answered the layout question by inference when a record of it was available, or accepted an answer that was not one: - A mount with no .vif defaulted to the legacy layout; the bitrot sidecar records the same config at encode time, so take it when present, as weed fix -ecx already does. The vif itself is now parsed once per mount rather than twice. - The Rust ecx rebuild derived its row count from the padded shard extent, which under the legacy layout reads a shard that is an exact large-block multiple as one row too many. Pass the encode-time .dat size from the .vif and keep the extent as the fallback. - weed fix -ecx read the block size outside the EC-config guard (collapsing the unknown sentinel into a definitive legacy), only wrote the recovered layout back when the .vif was absent rather than unusable, and broke a scan tie by candidate order instead of the documented reach. - The uniform layout tripped writeDatFile's large-block ambiguity guard, which cannot apply when the large and small blocks are the same size. * ec: give the index-recovery tests a parseable vif The fixtures wrote the literal bytes "volinfo" as the source .vif and the recovery copies it verbatim, so the receiving server then mounted the volume from a .vif it could not parse. That used to pass by silently defaulting to the legacy layout; a mount now refuses a vif it cannot read, which is what the tests were exercising all along without meaning to. * ec: validate the layout a vif records, not just its syntax Review follow-ups on the mount-strictness change: - A .vif can parse and still record a block size no encoder could have produced (negative, or not a whole number of small blocks). Both servers took it and mapped every read through it. ValidateBlockSize / the Rust mirror now refuse the mount, the same way an unparseable vif does; 0 stays valid as the legacy two-tier layout. - The bitrot-sidecar fallback accepted parity_shards == 0 and summed the counts in their own width, so values near the ceiling wrapped past the MaxShardCount bound. Require both counts and sum in a wider type. - weed fix -ecx treated a config with only DataShards > 0 as usable, so a half-written .vif suppressed the recovery paths AND survived the rewrite. Require a complete, in-range config before trusting it. - Returning the vif-load error left the .ecx and .ecj descriptors open; repeated mount attempts on malformed metadata could exhaust them. * ec: refuse to act on a layout the metadata does not establish - The worker encode only logged a failed .vif write and skipped it in the distribution set, and treated the .ecsum write as best-effort. A worker whose disk filled after the much larger shards landed could still distribute, mount, verify shard inventory, and delete the source replicas — leaving holders with shards whose geometry nothing records. Both writes and both inclusions are encode success conditions now. - A generation-matching .ecsum that disagreed with the .vif geometry only disabled checksums in Go, and in Rust was not compared at all, so protection stayed On while reads used the other layout. Both files record the layout their generation was encoded with, so a disagreement now fails the mount. * ec: reject an invalid recorded block size in weed fix -ecx A .vif with valid shard counts but a negative or unaligned block size was marked usable: a positive invalid value pinned the scan to a geometry that de-stripes to garbage, and a negative one ran the dual scan but left the invalid .vif in place afterwards. Validate it with the same rule the mount applies, and when it fails leave the layout unknown so the scan recovers it and the file is rewritten. * ec: validate the sidecar layout weed fix -ecx recovers from The .ecsum fallback was taken on DataShards > 0 alone, so a CRC-valid sidecar carrying the wrong generation, an incomplete ratio, or an unaligned block size would pin the reconstruction to one incorrect uniform-layout candidate instead of letting the dual scan decide. Require generation 0, a complete in-range ratio, and a valid block size; anything less leaves the layout unknown, which is the answer that still recovers by scanning. * ec: let only a genuinely absent sidecar choose the legacy layout With no .vif the bitrot sidecar is the only record of a volume's layout, and the mount fallback read a failed load, an unusable config, or a sidecar stamped for another generation as "assume legacy". A uniform generation-0 volume could therefore mount with legacy or another generation's geometry and answer reads with the wrong bytes. Present-but-unusable now fails the mount; only actual absence keeps the legacy defaults. Shared as EcShardConfigFromSidecar so every caller reads the sidecar the same way. * ec: treat a recorded-but-impossible layout as corruption, not as legacy - A .vif whose ecShardConfig is PRESENT but records an impossible ratio was answered with the default 10+4 and the legacy block layout, in both languages. That reads a uniform volume's shards at the wrong offsets and returns the wrong bytes. Only an entirely absent config still means "this predates the record"; a present one that cannot be true fails the mount. - The shard-count bound summed two uint32 counts as int, which wraps on a 32-bit build: 0x7fffffff + 0x7fffffff lands at -2 and slips under MaxShardCount. ValidEcShardCounts sums in uint64, and every EC call site that checked a recorded ratio now goes through it. * ec: rebuild on the geometry the sidecar records, and flag it when it disagrees The rebuild RPC passes BackgroundECContext, so RebuildEcFiles resolves the layout itself — and it resolved a missing or invalid .vif to the default 10+4 with the legacy block size. Two consequences: a 12+4 volume was reconstructed through a 10+4 matrix, which produces wrong bytes and never regenerates shards 14-15; and the chosen geometry then contradicted a valid uniform sidecar, which loadRebuildSidecar reported as BitrotOff — silently skipping the input and regenerated-shard checksum checks precisely when the volume had already lost its metadata. The layout now resolves from the bitrot sidecar (found across the server's disks, not just beside the base name) before falling back to the defaults, and a present-but-impossible ratio fails instead of being replaced. A sidecar that contradicts the chosen geometry is BitrotInvalid, which the existing unsafeIgnoreSidecar override still lets an operator push past. * ec: let the Rust rebuild read metadata off a sibling disk read_ec_shard_config searches only the location the rebuild writes into, so a volume whose .vif or generation-0 .ecsum sits on another of the server's disks resolved to the default 10+4 with the legacy block layout — the Rust half of the geometry-guessing the Go rebuild just stopped doing. It then reconstructs a custom-ratio or uniform volume through the wrong Reed-Solomon matrix and de-striping geometry. The rebuild now looks for the .vif in its own location and then each sibling, falls back to the generation-0 sidecar wherever that lives, and only defaults when neither exists anywhere. The encode-time .dat size the ecx rebuild needs is resolved the same way. * ec: resolve a rebuild's vif from every directory that may hold it RebuildEcFiles probed only <data-base>.vif. The caller knows the selected location's index directory and the sibling locations, but passed neither for metadata: additionalDirs carried shard directories only, and were searched for shards and the checksum sidecar. A split -dir/-dir.idx layout, or a disk holding only shards, therefore resolved a pre-sidecar custom-ratio volume to 10+4 and reconstructed through the wrong matrix — never regenerating shards 14-15. The caller now hands over the index and sibling directories, and the resolver probes the vif across all of them, matching what the Rust resolver already does for both the vif and the sidecar. * ec: make every rebuild consumer agree on the layout it resolved - The post-rebuild bitrot backfill re-derived the geometry from this directory's .vif alone and dropped the block size entirely, so a rebuild that resolved its layout from a sibling, the sidecar, or a uniform vif wrote a manifest describing a DIFFERENT layout — one later mounts reject, or that covers only the default shard count. The layout is resolved once now, through an exported ResolveRebuildECContext, and the rebuild and the backfill share that answer. - The Rust rebuild collected only each location's data directory, so a sibling's INDEX directory — where a split -dir/-dir.idx layout keeps .ecx/.ecj/.vif — was never probed, and a custom-ratio volume still resolved to 10+4 with the legacy layout. Both directories of every location are carried now, deduped against the rebuild's own. - A shard delivery can bring the checksum manifest with it, but the receive path only writes the file: a server that already had the volume mounted kept its resolved protection state (off) until a remount. The mount RPC re-resolves it once the shards it describes have been added. * ec: cover the rebuild's directory search with tests Reviewers flagged the sibling index directory twice, and the fix that closed it had no test of its own: the assembly sat inline in the rebuild handler, reachable only through a gRPC call against a populated store. Lifting it into rebuildSearchDirs / select_rebuild_location makes the rule assertable — a sibling contributes BOTH its data and its index directory, a shared index directory is listed once, and the rebuild's own data directory never repeats. Writing the Rust cases surfaced that the two implementations do not agree on where the rebuild's own index directory belongs, and both are right: Go's resolver takes a single directory list, so that directory has to be inside it, while Rust's takes the rebuild's data and index directories as their own arguments and would search them twice. The tests now state which contract each side is holding to, so neither drifts into the other's shape. Pure refactor otherwise; no behaviour change. * ec: search the index directory for the layout sidecar The Rust resolver looked for the generation-0 .ecsum in the rebuild's data directory and the sibling list, but not in the rebuild's own index directory — while the .vif lookup directly above it did, and Go's findBitrotSidecar has always checked both bases. On a split -dir/-dir.idx location that directory is where the metadata lives, and callers leave it out of the sibling list precisely because it is passed here separately, so nothing searched it. With no .vif anywhere the sidecar is the only surviving record of the layout. Missing it resolved a 12+4 uniform volume to 10+4 with the legacy striping — the test added here fails with (10, 4, 0) against the old code — and the rebuild then reconstructs through the wrong matrix and writes .ecx offsets that no reader can follow. * ec: let the rebuild see its own index directory The Rust rebuild takes a single flat directory list — the shape Go's RebuildEcFiles uses — so it cannot be handed the rebuild location's index directory separately the way the layout resolvers are, and the handler was passing the sibling list, which deliberately omits exactly that directory. On a split -dir/-dir.idx location that is where .ecx and .vif live, so the shard and index lookups could not see them. Go has always carried that directory in additionalDirs; this lines the two call sites up. * ec: let a config-free vif fall through to the layout sidecar A .vif that carries no ecShardConfig answers nothing about the layout, so it is no more informative than an absent one — but both trees treated its mere existence as the end of the search. Go went straight to the 10+4 legacy defaults without consulting the sidecar at all; Rust returned whatever ec_shard_config_from could make of a single directory. A 12+4 uniform volume with a legacy config-free vif therefore resolved as 10+4 legacy, and every read landed at the wrong shard offset. The sidecar lookup was also single-directory on both sides, while a split -dir/-dir.idx layout keeps .vif and .ecsum with the INDEX. Go's findBitrotSidecar has always taken both bases; the callers here passed only the data base, and the Rust bitrot resolver derived its path from the data base alone. Rust's layout resolver now takes a candidate directory list — data, index, then any siblings — and searches all of it, which also removes the early return that made the vif's presence decisive. load_vif_info_across_dirs reported `dir` even when load_vif_info had found the vif in `dir_idx`. Nothing reads that field today, so this changes no behaviour; it stops the next caller that resolves the rest of the volume's metadata against the answer from being sent to a disk holding none of it. Absence stays legal throughout: a volume with neither record is genuinely legacy. Present-but-unusable still fails the mount, now in the config-free-vif branch too. * ec: activate a delivered sidecar on every per-disk runtime A vid mounts as one EcVolume per disk, each with its own resolved protection state, but the post-delivery reload used the first-match lookup and so touched exactly one of them. The siblings kept reporting no protection until a remount — and since shard distribution deduplicates the metadata files onto the first target disk for a node, the runtime that got the .ecsum is not necessarily the one the lookup returns. Iterate every runtime instead, via a new FindAllEcVolumes and its Rust mut equivalent. Combined with each runtime now resolving its sidecar against its index directory as well as its data directory, a server sharing one -dir.idx across its disks activates all of them from the single delivered copy. The Rust volume server had no post-mount reload at all; it gets one here, matching Go. * ec: resolve the delivered sidecar across every EC metadata directory Reloading every per-disk runtime, added last round, did not by itself make the delivered manifest reachable. Startup mirroring copies .ecx/.ecj/.vif to every shard-bearing disk so each mounts self-contained, but deliberately not .ecsum, and a repair delivers exactly one copy. Each runtime was resolving against its own two directories, so every sibling of the disk that received the file kept reporting no protection however often it reloaded. Resolve one authoritative copy across every EC metadata directory instead of duplicating the file. Mirroring .ecsum would have to keep pace with a file that is rewritten as shards are repaired, and would not help the reported case at all: the delivery happens at runtime, and mirroring only runs at startup. The regression test pins both halves — a reload restricted to the volume's own directories still finds nothing, and the same reload given the server's metadata directories turns protection on. * ec: ask every directory before writing a TOFU baseline After a rebuild the opportunistic backfill asks whether this volume already has a checksum manifest, and answered from the data base alone. A split -dir/-dir.idx layout keeps the sidecar with the index, and a multi-disk server may keep it on a sibling, so an existing manifest read as absent. The consequence is worse than a missed read. On a false "no" the backfill writes a fresh sidecar at the data base from whatever the shards say right now — and the data base is the first candidate every resolver checks, so that TOFU baseline shadows the real manifest rather than sitting beside it. A shard that was silently corrupt gets blessed, and the record that would have caught it stops being consulted. FindBitrotSidecar exports the search the package already used internally, so the question is asked of the data base, the index base and the sibling disks — the same candidates the rebuild resolves its layout from. * ec: refuse a shard block size no encoder could have produced weed fix -ecx derived one from the raw shard extent, so a truncated or partially copied shard wrote a .vif that NewEcVolume then permanently refuses — the volume the tool was run to rescue could never mount again. An extent that is not a whole number of small blocks cannot have come from a uniform encode, so it is no longer offered as a candidate, and nothing unvalidated reaches the .vif. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: derive the .vif's dat size and block size from one measurement VolumeEcShardsGenerate stat'ed the .dat before the encode while WriteEcFiles stat'ed it again to size the blocks. A write landing between the two produced a .vif whose own two fields describe different files. WriteEcFiles now leaves both on the context, and fills a placeholder context in place so the caller can read them back. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: keep the source volume until every holder serves its shard layout The uniform layout rides in a .vif field older volume servers never knew: they discard it, mount the shards as legacy and return wrong bytes with nothing erroring, and the shard files are the same length either way so no other check notices. The upgrade order lived only in the release note. VolumeEcShardsInfo now reports the block size the holder actually serves, in both the Go and Rust servers, and the pre-delete verification refuses to drop the source unless every reachable holder echoes the one the shards were encoded with — while a rollback still exists. A server that predates the field answers 0, which is the negative answer. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: drop the rebuild's dead block-size parameters generateMissingEcFiles never reads largeBlockSize/smallBlockSize — Reed-Solomon reconstruction is layout-agnostic — so passing the legacy constants only advertised a layout the rebuild does not use. Also move UniformBlockSize's doc off ValidateBlockSize. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: warn about EC defaults only when the mount used them The "vif file not found, using defaults" warning fired even after the bitrot sidecar supplied a non-default layout, sending anyone triaging wrong bytes after the legacy layout the volume never mounted on. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: stat the distributed bitrot sidecar once The strict check re-stat'ed the file immediately before the stat that already gates inclusion, and a failed sidecar write now fails the encode outright, so the first could only fire on a deletion between the two lines. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: say what the reconstruct budget actually bounds A shard's buffer stays in bufs until its interval reconstructs, which is after the read that filled it released its permit, so the semaphore bounds round trips in flight and not retained bytes. Peak memory is the intervals reconstructing at once times the shards each reaches times the interval size. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * test: let the fake volume server report its delivered EC layout The pre-delete verification now asks each holder which shard block layout it serves, and a fake that always answered "unset" looked exactly like a volume server too old to know the field. Distribution ships the .vif to every holder alongside its shards, so read the layout back out of it as a real holder does. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 |
||
|
Chris LuandLisandro Pin
|
fcc2ea61d3 |
ec: scrub a volume through its parity data (#11006)
* Introduce a new `READS` scrub mode. `READS` performs a full volume scrub but, unlike `FULL`, it will attempt to reconstruct data for missing/damaged shard intervals from other shards in the cluster when necessary. The goal of this check is to ensure that EC volume contents _are readable by Seaweed_ even on a degraded storage state, by exercising parity data which is not read in `FULL` mode. This is useful not only to validate data is user-readable, but also to detect potential parity shard issues which may be difficult to pinpoint otherwise - particularly for older volumes lacking sidecar data, and hence unaffected by `CHECKSUM` scrubs. For regular volumes, this operation is equivalent to `FULL`. Example: ``` > ec.shard.unmount --volumeId=1 --shardId=0,3,11 --delete --apply Live shard topology for volume ID 1 (14 shards): 0@10.200.18.89:9001 1@10.200.18.89:9002 2@10.200.18.89:9003 3@10.200.18.89:9004 4@10.200.18.89:9005 5@10.200.18.89:9006 6@10.200.18.89:9007 7@10.200.18.89:9008 8@10.200.18.89:9009 9@10.200.18.89:9013 10@10.200.18.89:9010 11@10.200.18.89:9011 12@10.200.18.89:9012 13@10.200.18.89:9020 Will unmount + delete 3 shard(s): 0@10.200.18.89:9001 3@10.200.18.89:9004 11@10.200.18.89:9011 Unmounting shard 0@10.200.18.89:9001 for volume ID 1... Deleting shard 0@10.200.18.89:9001 for volume ID 1... Unmounting shard 3@10.200.18.89:9004 for volume ID 1... Deleting shard 3@10.200.18.89:9004 for volume ID 1... Unmounting shard 11@10.200.18.89:9011 for volume ID 1... Deleting shard 11@10.200.18.89:9011 for volume ID 1... All done! > ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=full using FULL mode Scrubbing 10.200.18.89:9002 (1/1)... Scrubbed 6 EC files and 1 volumes on 1 nodes Got scrub failures on 1 EC volumes and 1 EC shards :( Affected volumes: 10.200.18.89:9002:1 Affected shards: 10.200.18.89:9002:1:0 > ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=reads using READS mode Scrubbing 10.200.18.89:9002 (1/1)... Scrubbed 6 EC files and 1 volumes on 1 nodes ``` * ec: report the shards a READS scrub had to rebuild A READS scrub that recovers an interval was recording nothing, so a volume missing three shards came back clean and nobody repaired it. The unreadable shard is now recorded before the rebuild is attempted: READS reports the same broken shards as FULL and differs only in whether the needles themselves failed, which is the signal worth having - shards are gone, data is still there. forceDeletedNeedlesCheck now applies to READS as well, in the shell and in the RPC guard: it runs the same needle walk as FULL. Regenerated the proto instead of hand-editing it, so the pancis typo (which protoc-gen-go-grpc emits into eight other files here) and the header whitespace stay as generated. Mirrors into the Rust volume server, which also now honors force_deleted_needles_check rather than hardcoding it off. Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq * ec: answer a deleted needle from a READS rebuild as deleted #11020 gave the Rust recovery a deleted flag alongside its bytes, and it answers a deleted needle with no bytes at all. The READS scrub appended that empty answer, which does not compile against the new signature and, once it did, would leave the needle short and report the size mismatch as damage. Zero-fill the interval instead, the way the direct read beside it already does: the assembled needle then reaches read_bytes as the delete-state mismatch the walk already tolerates. Go takes the same branch off the flag its recovery returns, rather than discarding it. Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq --------- Co-authored-by: Lisandro Pin <lisandro.pin@proton.ch> |
||
|
Chris Lu
|
ba5b14b457 |
master, filer, s3api: bound the collection deletes that strand a caller (#11026)
* master: bound each volume server DeleteCollection, and finish the fan-out A collection delete fanned out to every volume server holding it with context.Background(), so a server that accepted the connection and then went quiet held the whole delete open with nothing to end it. Each RPC is bounded now, on the same budget allocateVolumeTimeout gives the other master-to-volume-server admin RPC. The volume server runs the delete to completion regardless of the request context, so giving up costs the confirmation and not the deletion. The walk itself is the caller's, not a per-server one: - It outlives the caller. A cancelled request must not abandon a destructive fan-out part-done, with volumes left behind and no request still running to come back for them. - It no longer stops at the first server that refuses, which left the collection on every server after it in the list. The first failure is still what is reported, and the collection stays in the topology so a later delete comes back for the rest. - It sends one RPC per server rather than one per replica. ListVolumeServers reports a node once for every replica it holds, while DeleteCollection removes the whole collection from the server it reaches, so a collection with thousands of volumes repeated the same whole-collection delete thousands of times over. Both passes run too. Returning after a failed normal pass left the collection's EC shards in place with nothing left to retry them. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * master: delete the EC shards behind /col/delete too The HTTP handler carried its own copy of the volume-server walk and only ever ran the normal pass, so a collection deleted through it kept its EC shards. It shares the gRPC path now, which also gets it the bounded RPCs and the one-per-server fan-out. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * filer: bound the collection delete a bucket delete leaves behind Deleting a bucket entry deletes its collection afterwards, deliberately detached from the request so a client that hangs up cannot strand the bucket's volumes. Detached meant unbounded, though: with the master down or mid-election the wait for a leader has nothing to end it, so the handler parks, and the client retrying behind it parks another. It keeps outliving the request and now carries a deadline of its own. The budget bounds the wait, not the work: the master keeps deleting on its own fan-out once asked, so giving up costs the confirmation. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * s3api: bound the collection RPCs a bucket creation and deletion issue Neither carried a deadline, so a transient failure anywhere down the chain held the S3 request open until the client gave up on it. Both budgets are taken outside the filer failover walk, so one budget covers the whole walk rather than granting each filer a fresh one. The walk itself stops when that budget is spent, and stops without blaming anyone: the caller's own expiry is not evidence against the filer that was answering, and the next filer has no time left to answer in either. Recorded as a filer failure, a slow master upstream would flag every filer in the walk, and the three failures that open the circuit take unrelated object reads down with them. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * s3api: a failed collection listing no longer fails a bucket creation PutBucket lists collections to notice a leftover one it is about to reuse. The result feeds a warning and nothing else -- s3a.exists is what decides whether the bucket already exists -- yet a transient failure of that listing returned 500 and refused the creation. It is advisory now, so a failure is logged and the creation continues, exactly as it does when the listing returns false. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP |
||
|
Chris Lu
|
60893c5ef3 |
Classify a filer error before a user-controlled path is wrapped into it (#11004)
* util, pb: classify a filer error by the status the server sent DoSeaweedListWithSnapshot wrapped a failed ListEntries with %v, dropping the gRPC status, so IsTransientError fell back to matching substrings against a message that now held the caller's path. Keep the status with %w and let it decide, reading the server's own text rather than the wrapper's. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * s3: keep the bucket and prefix out of the list retry decision A bucket named transport, or a prefix under logs/unavailable/, made a PermissionDenied listing look transient and got it retried; a key holding the not-found sentence suppressed a retry that should have run. Both checks now read the filer's status, and only fall back to the text when there is none. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * filer, s3: classify a delete failure before the path is wrapped into it The filer put the non-empty-folder marker behind its own "delete directory %s" wrapper and the gateway matched it as a substring, so a key named after the marker turned a real delete failure into the demote-the-marker no-op and the request answered 204. Keep the marker leading the message that crosses the wire, turn it back into a sentinel where the response is read, and match that. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU |
||
|
Chris Lu
|
902a12fd6f |
wdclient: bound the wait for a master leader by the caller's context (#11002)
* wdclient: bound the wait for a master leader by the caller's context WithClient waited on GetMaster with context.Background(), so a caller that arrived while no master leader was known parked in a 200ms poll loop until one appeared, whatever deadline it had already set on the RPC. Each retry above it then left another goroutine in the same wait. Take the context in WithClient and WithClientCustomGetMaster and hand it to GetMaster, and stop the retry loop once it is done. The dial keeps context.Background(): fn brings its own RPC context, so a cancellation seen here cannot be attributed to the shared connection. Call sites pass whatever they hold: the request context in the filer's CollectionList, DeleteCollection and Statistics handlers and in the credential store's propagation, the operation context in the shell's s3.bucket.delete and the kafka gateway's broker and filer discovery, and context.Background() where there is none - the shell commands, the admin dashboard wrapper, and the exclusive locker's initial lease. The locker's release keeps its own uncancelled context so a slow unlock cannot turn into a ghost lock. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * wdclient: test that WithClient gives up with the caller's context Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * wdclient: cut the master retry backoff short when the caller gives up util.Retry sleeps unconditionally between attempts, so a transient error arriving just before the caller's deadline still cost it a full backoff step. Use the context-aware util.RetryWithBackoff, the same helper the volume lookup in this file already uses. Two call sites went with it: the shell's lock-holder lookup builds its three second bound before WithClient so it also covers finding the leader, as its comment already promised, and the filer's post-delete collection cleanup goes back to an uncancelled context - the entry is already gone, so a caller that hung up must not leave the collection behind. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * wdclient: test that a cancel during backoff ends the retry Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU |
||
|
Chris Lu
|
0b5fff2ccd |
filer, s3: reuse the volume server's guarded remote-storage client builder (#10990)
* volume: build the guarded remote storage client through a shared helper Fold the endpoint validation, credential check and rebinding-safe dialer that FetchAndWriteNeedle applies before dialing a caller-supplied remote storage endpoint into a single BuildGuardedRemoteStorageClient helper, so other callers that dial the same endpoints can reuse it. No behavior change on this path. Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN * filer: build the remote-mount stream client through the guarded helper streamFromRemote serves a cold remote-only entry straight from its mounted origin. Build its client through BuildGuardedRemoteStorageClient so the same endpoint checks the volume server applies cover this read path too. Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN * s3: build the remote-mount stream client through the guarded helper openRemoteStream serves a remote-mounted object straight from its origin when the local read cannot. Build its client through the same guarded helper so the endpoint checks apply here as well. Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN |
||
|
Chris Lu
|
e9a464840c |
webdav: describe a listed entry the way clients expect (#10993)
* webdav: name the entry, not its path, in a listing DAV:displayname carried the full path of every entry. A client that takes displayname for the child's name - Windows Explorer does - then looks for /dir/name under /dir and finds nothing, so a folder shows up empty while the root, where the two spellings differ only by a leading slash, still lists. Readdir now builds its entries with toFileInfo like stat does, so a listing and a lookup describe a child the same way, and the wrapper that was trimming the sub-folder back off a name goes away with it. Claude-Session: https://claude.ai/code/session_01XCeuCWpF9xo9CfyHvCQE9c * webdav: derive an ETag when nothing hashed the entry Uploads through this gateway carry no content MD5, so filer.ETag comes back empty and every file in a PROPFIND answered with an empty DAV:getetag, which is not a valid entity-tag. Report it as unimplemented instead, the way the sub-folder wrapper already did, and webdav falls back to modification time and size. The wrapper's copy went with it - it swallowed the stat error a caller was meant to see. Claude-Session: https://claude.ai/code/session_01XCeuCWpF9xo9CfyHvCQE9c |
||
|
Chris Lu
|
eb3bbfeb1f |
filer: apply the path's storage rule TTL on every write path (#10963)
* filer: cover the storage rule TTL on the object transaction write path An object written through ObjectTransaction used to land with ttlSec 0 even under an fs.configure TTL rule, while the same object written through CreateEntry got the rule's TTL. Guard the shared stamping so the two paths cannot drift apart again. * filer: apply the path's storage rule to an appended entry AppendToEntry resolved the storage option from the path - so its chunks land on a TTL volume under an fs.configure TTL rule - but never stamped the rule's TTL on the entry it creates, leaving an entry that outlives its data. Route it through applyStorageDefaultsToEntry, which now feeds the entry's own TTL into the option so the placement an existing entry's appended chunks get is unchanged. * filer: apply the path's storage rule to a completed TUS upload The PATCH path resolves the storage option from the target, so a TUS upload into an fs.configure TTL prefix writes its chunks to a TTL volume, but completion built the final entry with ttlSec 0 - the entry outlived the data it pointed at. Stamp it through applyStorageDefaultsToEntry, which also subsumes the hand-rolled read-only check and supplies the rule's name-length limit. * filer: apply the destination's storage option TTL to a copied entry The copy handler re-uploads the source's chunks under the destination's storage option, so a copy into an fs.configure TTL prefix already lands its data on a TTL volume. The entry, though, carried the source's ttlSec - 0 for a source outside the prefix, or the source's own TTL where the two rules differ - so it never expired with the data it pointed at. Take the TTL from the same option the chunks were placed with, after the data-only copy has restored the destination's metadata. |
||
|
Chris Lu
|
a02c0024e5 |
master: cap the reported capacity at what the disks hold (#10960)
* master: cap the reported capacity at what the disks hold Statistics reported max volume count times the volume size limit, which is how many volumes the cluster is allowed to place, not how much space it has. A cluster given far more slots than its disks can fill reported a capacity it could never reach -- 65536 slots at 30GB read as 1.9PB on a 460GB disk -- and the number never moved, since writing data changes neither the slot count nor the size limit. The volume servers already report each filesystem's total and free bytes in their heartbeats, so bound the answer by what they say is left. * mount: keep the last known sizes when filer statistics fails A failed Statistics call returned before df's answer was filled in, so a mount whose filer or master was briefly unreachable reported an empty filesystem rather than the sizes it already had. * master: drop the disk ceiling when a volume server does not report A cluster part way through an upgrade has volume servers that predate the disk bytes in the heartbeat. Summing only the ones that answered left the quiet server's free space out of the total, and the server holding the room is exactly the one that could make the cluster read as full. Answer with the disks only when every one of them reported. |
||
|
Chris Lu
|
44115c1051 |
filer: stop TUS uploads from turning into garbage (#10945)
* filer: store TUS sub-chunks through the regular chunk writer A TUS sub-chunk was written with one assigned file id, retried up to three times against that same id, and abandoned on failure: an attempt that had landed on some replicas left a needle no session record and no entry ever references, unreclaimable by vacuum. dataToChunkWithSSE, which the regular write path uses per chunk, assigns a fresh file id per attempt and hands back the file ids of failed attempts, which are now freed the way the regular write path frees them. * filer: retry a chunk write on a fresh volume when the server 5xxs The filer's chunk writer assigns a fresh file id per attempt but only retried transient network errors, so a volume filling up and turning read-only mid-write failed the whole request even though the very next assignment would have landed elsewhere. Every other write client already routes this through ShouldReassignUpload; the filer's own write path now does the same, for regular uploads and TUS sub-chunks alike. * filer: export the chunk deletion queue The filer test harness in weed/server builds filer.Filer as a struct literal, so any code path reaching DeleteChunks dereferenced a nil queue. Exported like the neighboring DeletionRetryQueue so the harness can arm it. * filer: complete a TUS upload whose chunk records overlap A PATCH retried while its predecessor was still storing a sub-chunk - a proxy timeout with an immediate retry is enough - records the same range twice. HEAD computes Upload-Offset as the covered watermark and reported the upload fully received, but completion demanded exactly adjacent records and failed every attempt: the client concluded success from offset == length, no entry was created, and the session eventually expired, turning the entire upload into deleted needles for the vacuum to chew through. Completion now validates gapless coverage with the same watermark HEAD uses. A record extending coverage joins the entry - the read path resolves partial overlaps by ModifiedTsNs, and the raced copies carry identical bytes - while a fully covered duplicate is freed once the entry lands. * filer: allow one mutating TUS request per session at a time Nothing stopped two PATCHes from writing the same range concurrently: both loaded the same offset, both passed the conflict check, and both recorded their sub-chunks. A client whose request timed out in a proxy retries immediately while the server side is still storing the buffered sub-chunk, which is exactly that race. A session now accepts one PATCH or DELETE at a time, the way tusd locks uploads; a concurrent one is refused with 423 Locked, which TUS clients retry, and HEAD keeps answering so progress polling is unaffected. The chunk state is loaded under the claim, so a retried PATCH sees every record its predecessor left and conflicts cleanly instead of duplicating data. * test: cover a TUS PATCH raced by its own retry Stalls a PATCH mid-body over a raw connection, retries the same range while it is in flight, and expects the retry refused with 423 Locked; the upload then resumes from the reported offset and the final content must be intact. * filer: never free a TUS duplicate the entry still references Coverage is computed from ranges, so a record fully covered by another is treated as a duplicate no matter which needle it names. A malformed record naming a file id the entry keeps would have had that needle freed right after the entry landed - the corruption this change set exists to stop. The duplicates are now freed in one batch, skipping any file id the entry references; their records go with the session directory. * test: bound the raw TUS connection reads http.ReadResponse on the stalled PATCH's connection blocked until the whole go test timeout if the filer never answered. * filer: free the needles of chunk write attempts a retry replaced A volume server stores the needle locally and only then fans out to the replicas, so a replication failure 5xxs with the data already written. Each attempt assigns its own file id, so once a later attempt lands elsewhere nothing references the earlier ones: the caller only sees the chunk that succeeded, and the failed ids were dropped. They are now freed the way the caller frees them when the whole write fails. Retrying on a 5xx makes this reachable on every read-only or full volume, which is exactly the condition that filled the reporter's volumes. |
||
|
孙超
|
c80664ec21 |
s3: propagate storage rule fsync to volume server uploads (#10906)
The storage rule's fsync decision was computed by the filer (detectStorageOption -> rule.Fsync) and applied on the filer's own HTTP write path, but was never carried onto the chunk uploads S3 issues: the AssignVolumeResponse had no fsync field, so the s3api client could not learn the decision, and the chunked upload URL was hardcoded without it. Every S3 write to a path with fsync configured went to the volume server as a non-fsync write. Carry the decision through the assign response: - filer.proto: AssignVolumeResponse gains bool fsync, filled from the storage option the assign resolved. - operation.AssignResult gains Fsync, so uploadChunk can append ?fsync=true to the volume server upload URL (single and replica fan-out paths). - The S3 PUT/UploadPart assignFunc, the S3 copy path, the admin file browser upload, and the Iceberg worker assign functions all forward the response field. Adds TestUploadReaderInChunksAppendsFsyncWhenAssigned. |
||
|
Chris Lu
|
74038e1b14 |
master: don't let a dead KeepConnected handler close its successor's channel (#10900)
A client that reconnects before the old handler exits re-registers the same client name, and addClient overwrites the map entry. The old handler's deferred deleteClient then closed whatever channel the map held under that name: the new, live stream's. Receiving from a closed channel returns nil immediately and forever, so the new handler's send loop degenerated into sending empty responses at wire speed, pinning a core on each side until the client killed the connection. deleteClient now closes the channel its own handler registered and leaves the map entry alone unless it still points to that channel. This also closes the previously orphaned old channel, whose drain goroutine used to leak. The send loop treats a closed channel as an exit instead of a message stream. |
||
|
Chris Lu
|
0f85d005ad |
server: 416 only when no requested range overlaps, with Content-Range, and the Rust mirror (#10889)
* filer, volume server: return 416 when no requested range overlaps the content * seaweed-volume: return 416 when no requested range overlaps the content * server: check the range test error, use the request context, fix the no-overlap comment boundary |
||
|
Chris Lu
|
173adbc291 |
master: never re-seed a raft cluster over committed state under -raftBootstrap (#10883)
* master: never re-seed a raft cluster over committed state -raftBootstrap deleted logs.dat, stable.dat and snapshots on every start and then bootstrapped a fresh cluster. Since hashicorp raft only snapshots after 8192 log entries, the TopologyId lives in the log, not in a snapshot, so the pre-wipe snapshot recovery found nothing and each restart minted a new cluster identity. A master that came up while it could not reach its peers seeded a rival cluster; when the two logs met, SetTopologyId's split-brain guard fatally stopped every master holding the other id, and the master layer crash-looped with no quorum. Bootstrapping is genesis. Drop the wipe and the inline bootstrap. The first master in -peers already mints a cluster once it has confirmed no peer has a leader, so the flag has nothing left to do and is now ignored; keeping that one master the sole bootstrap authority is what stops a partition from minting two clusters, so the flag must not widen it either. A master with state rejoins its peers, and one whose data dir was reset is admitted by the sitting leader instead of forking again. * test: cover -raftBootstrap restarts in the multi-master suite Three masters start with -raftBootstrap, the way the helm chart renders it on every master on every roll, and the cluster has to hold one TopologyId after they all restart. /dir/status is proxied to the leader, so each master's own view of the identity is read out of its log, which is where a fork shows up. Before the fix the hashicorp case minted a new id on each restart. |
||
|
Junker der Provinz
|
5ebc9c9f4b | server: reject a Range start offset equal to the file size (#10898) | ||
|
Chris Lu
|
c1a993bc3b |
filer: keep the TUS sub-chunks that already landed when a write fails (#10876)
* filer: keep the TUS sub-chunks that already landed when a write fails A PATCH is split into 4MB sub-chunks, and each one is recorded in the session as soon as it is stored. The session listing is what HEAD reports as Upload-Offset and what the final entry is assembled from, so a record is a promise that the data behind it exists. When a later sub-chunk failed - a read-only volume, or a client that hung up mid-body - the error path deleted the needles of every sub-chunk the same PATCH had written but left their records in place. The resuming client was then told to continue past bytes the filer had just queued for deletion, and the upload completed into a gapless manifest pointing at needles that were gone: HEAD returned the right size, GET died mid-body once a vacuum reclaimed them. Recorded sub-chunks now stay, which is what resumption expects: the client picks up at the offset the session reports, and an upload that is abandoned frees its chunks with the session. * filer: drop a TUS chunk's record before freeing its data filer.CreateEntry can return an error with the entry already inserted - the parent-directory pass runs after the insert and keeps the entry when it fails. A failed saveTusChunk therefore does not mean the record is absent, and deleting the needle outright left the same corruption the resume path used to cause: a session record pointing at data that is gone. Remove the record first and only free the needle once it is gone. A record lost with its data still stored merely leaks, which the vacuum and fsck paths already account for. * test: cover a TUS PATCH that is cut off mid-body Resets the connection after one 4MB sub-chunk has landed, resumes from the offset the session reports, and vacuums before reading the file back, so anything the filer deleted behind a kept record shows up as a short read. |
||
|
Chris Lu
|
35d53a20f6 |
master: let the leader admit a master that starts with no raft state (#10865)
* master: answer with the leader raft already knows Topo.Leader() backs off for up to 20 seconds waiting for an election. Callers that a health probe or a client is blocked on cannot afford that: /cluster/status, /cluster/healthz and /readyz all sit past the probe timeout of both the helm chart and the operator, so a master that is still joining looks dead rather than joining, and the kubelet restarts it. informNewLeader and SendHeartbeat hold the client on a master that cannot serve it, exactly when it should move on to find the one that can. Answer these from MaybeLeader instead, which reports what raft knows right now. MaybeLeader takes over the "am I the leader myself" fallback that Leader() used to apply on top of it, so one non-blocking call is still correct; Leader() keeps the backoff for callers that must wait. * master: let the leader admit a master that starts with no raft state Neither raft implementation lets a server outside the configuration campaign: goraft's promotable() requires a non-empty log, and hashicorp rejects vote requests from a candidate that is not in its configuration. A master that comes up with fresh state therefore cannot elect itself in — the leader has to pull it in. Nothing did. The peer list is static, rendered from the replica count, so scaling it up leaves the sitting leader running the old list with no idea the new masters exist. Under goraft they wait forever. Under hashicorp they are worse off: each bootstraps a cluster of its own from the new list, and two of them form a quorum next to the live leader, with their own TopologyId. That is the split brain SetTopologyId kills a master over. Admit the peer where it registers instead. Only the leader gets past the IsLeader check in KeepConnected, and a joining master's client lands there, so that is the moment it joins. The broadcast OnPeerUpdate rides on is not enough on its own: it only reaches masters already connected, which is why a leader that came up first missed both newcomers. RaftAddServer grew a goraft branch on the way, so cluster.raft.add stops silently doing nothing on the default raft, and RaftRemoveServer with it. Bootstrapping is now one call for both implementations, made only after the peers confirm nobody has a leader, and retried until this master is in rather than checked once and dropped. * master: do not evict a peer that is still in -peers The hashicorp leader drops a master from the raft configuration as soon as it stops answering pings. A master that is merely restarting answers nothing, so an ordinary bounce shrinks the quorum behind the operator's back — and then races its own return: the master comes back, registers, gets re-admitted, and the eviction lands after it. A randomized start/stop walk lands on it. Two of three masters running, the leader evicts the one that just went down, the restart re-adds it, the removal commits late and takes the leader's own leadership with it. What is left is a two-server configuration whose other half is down, and a running master that nobody will ask for a vote — no quorum, no way back until the third master returns. -peers is what declares membership. updatePeers already reconciles the configuration against it on every leadership change, and an operator who really means to drop a master can say so with cluster.raft.remove, so keep the eviction for masters that are no longer listed at all. * test: bounce masters at random and hold the election to it Twelve rounds of stopping or starting a random master, on both raft implementations, checking the two things an election must never get wrong: two masters claiming leadership at once, and a quorum that comes back without agreeing on one. The cluster's identity has to survive the whole walk, since a master that re-mints a TopologyId is the split brain SetTopologyId kills its peers over. The seed is random and logged, so a failure names the walk that reproduces it. Below a quorum the walk moves straight on. A master that has lost its quorum cannot commit anything, and goraft only checks whether it still has one on an election-timeout ticker, after its peers have been quiet for a full timeout — measured taking over 30 seconds to step down. That direction belongs to TestTwoMastersDownAndRestart, which was giving it ten seconds and would have started failing on a slower machine; it now waits on that behaviour explicitly rather than sleeping twice and hoping. WaitForTopologyId returns the id it waited for. Reading it separately raced the leader applying the raft entry that carries it, which shows up as an empty id right after an election rather than as a wrong one. |
||
|
Chris Lu
|
0c95137528 |
filer: stop aggregated metadata subscribers from spinning on a peer watermark hold (#10863)
* fix(filer): stop logging a held aggregated read as an error An aggregated subscriber may not read past the peers' low-watermark, and it stops at the first entry beyond it by returning a sentinel from the read callback. LoopProcessLogData logs every callback error, so on a cluster that keeps writing - where there is almost always an entry newer than the watermark - every read wrote an ERROR line naming the entry it stopped at, thousands per minute per filer. Mark the stop as control flow: an error wrapping StopReadingError is handed back to the caller unlogged, and the held-read sentinel wraps it. * fix(filer): release an aggregated watermark hold on peer progress A held read waited on the aggregated buffer's data channel, which the next write signalled - but a write cannot release a hold, only a peer reporting further progress can. On a cluster that keeps writing the loop therefore re-ran a whole pass per arriving event, log file listing and all, and held again on the same entry every time. Signal held readers from the meta aggregator instead, whenever a low-watermark rises: a peer reporting, or one dropped past its removal grace. The retry interval stays as the backstop for what no watermark covers. Count the holds so a parked subscriber stays visible. * fix(filer): floor how often an aggregated watermark hold releases Peers advance their delivery watermark on every event they stream, so releasing a hold on every advance is the same pass-per-event storm as releasing on every write, just without the log lines - and each pass lists a day of log files. Floor the release at 20ms. Advances inside the floor collapse into one release, which then delivers everything they covered. * fix(filer): pace a peer's delivery claim by what its subscribers hold at A filer's local metadata stream carries an idle heartbeat to its peer aggregators, and each peer turns it into that filer's delivery low-watermark. Aggregated subscribers hold at the minimum across peers, so a filer quiet enough to fall back on the heartbeat parked every subscriber in the cluster up to a keepalive interval - 5 seconds - behind live writes. With nine filers, most of them quiet at any moment, the minimum sat there permanently. Pace that heartbeat at 200ms once the filer has peers. It stays a keepalive, at the keepalive interval, for a filer with none. * fix(filer): wake each aggregated hold on its own watermark A persisted-log read is held by what the peers have flushed, an in-memory read by what they have delivered, but both parked on one channel closed whenever either minimum rose. Peers advance their delivery watermark on every event they stream, so a flush-held reader woke at the coalescing floor to re-list a day of log files and park again on the same entry - the storm this set out to fix, in the one place asymmetric peer progress still reached. Signal the two separately and park each read on the one that bounds it. |
||
|
|
5d5fcdf07b |
fix(filer): bound aggregated metadata reads by peer watermarks (#10803)
* fix(filer): watermark-bound aggregated metadata subscription against multi-source merge races The aggregated metadata subscription (SubscribeMetadata) merges per-filer sources that become readable at independent paces, but tracks its progress with a single scalar cursor. Once the cursor passes a timestamp T, anything a source materializes below T afterwards is silently skipped: a peer recovering from a stall re-inserts its backlog late (late ring merge), and a source's flush can land a log file, or a later chunk of the same file, after a subscriber's disk pass listed the files (late persisted-log landing). This is the residual documented in #10501. Bound the subscriber's two read paths by what every source has provably made visible, each with its own watermark: - Delivery low-watermark -> in-memory reads. The meta aggregator tracks, per subscribed peer (self included), the newest timestamp received on that peer's stream - real events, or idle heartbeats (peer streams now opt into ClientSupportsIdleHeartbeat). The aggregated ring is complete up to the minimum across peers; in-memory reads hold at it. - Flush low-watermark -> persisted-log reads. Each filer reports its local log-buffer flush watermark on its stream: a new flushed_ts_ns response field, carried on idle heartbeats and on periodic flush reports (gated on ClientSupportsIdleHeartbeat). Disk passes freeze the minimum across peers before listing the log files and hold at it; the day-boundary cursor jump and the metadata-chunks ref listing are bounded the same way, the latter at minute-file granularity. - Held reads keep the cursor at the last entry actually delivered and retry; the retry re-lists the log files, which is what picks up a late-landing file. Both watermarks are relaxed by the settled horizon (2 x LogFlushInterval) as a liveness escape, so a peer stalled beyond it delays subscribers by at most the horizon instead of forever - any loss that escape allows was unconditional before. With reads held at the flush watermark, a disk advance below it is proven complete on every peer's disk, so the unproven-crossing counter now only counts crossings the horizon escape allowed past a stalled peer. Live delivery on the aggregated stream may lag by up to the idle-heartbeat interval when some peers are quiet; SubscribeLocalMetadata consumers are unaffected. * fix(filer): resume evicted aggregated readers from an original-space disk anchor The aggregated ring rewrites out-of-order peer arrivals to its head, so a subscriber tailing it advances its cursor in bumped (arrival) timestamps, while persisted logs keep original timestamps. When a slow reader's unread window is evicted (e.g. a peer backlog flooding in after a stall) and the reader falls back to disk, resuming from the bumped cursor skips every original-space entry below it that memory never delivered - reproduced as a ~66% silent loss on a 3-filer cluster with one peer's stream frozen for ~70s while the subscriber lagged. Track a disk anchor: the newest original-space position the stream is proven complete through. Disk passes advance it directly; contiguous memory reads advance it to the peers' delivery low-watermark observed before the read (per-peer streams are ordered, so everything with an original timestamp at or below that watermark had already arrived and was delivered). A reader kicked off the ring resumes the disk pass from the anchor instead of the bumped cursor - redelivering what memory already sent is within the subscription's at-least-once contract, skipping what it never sent is not. * fix(filer): close review findings on the peer-watermark subscription bounds Four correctness holes found in review, one generated-file cleanup: - The flush-through claim could assert durability for events still on their way into the buffer: an event is timestamped before notification work that can block, and only then appended. Track stamped-but-unappended events on the Filer (the stamp shares a lock with the reader, and appends are bumped monotonically past the buffer head), and cap the reported flush watermark just below the oldest in-flight stamp. - Removing a peer deleted its watermark entries while its stream kept running: its next signal recreated the deleted entry, which then pinned the low-watermark forever once the stream died. Watermarks now advance only for tracked peers, and peer removal cancels the subscription context so the stream stops feeding the aggregated buffer promptly. - The pipelined sender folded flush reports (TsNs 0 reads as far behind) into batch Events tails, where the aggregator's nil-notification guard dropped them - a busy backlog replay could starve the flush watermark until the settled-horizon escape opened a loss window. Control messages are now unbatchable on the sender, and the receiver also reads watermark state off nested batch entries as belt and braces. - A give-up skip's cursor was not anchored, so the next eviction rewind undid the counted decision and re-entered the same park forever when the evicted window carried bumped timestamps. The anchor now follows give-up skips; an anchored cursor makes the rewind a no-op and keeps the gap machinery's re-arm onto the retained window reachable. - Regenerated-file churn from a different protoc-gen-go-vtproto version is dropped: the vtproto file is upstream's, plus only the flushed_ts_ns marshal/size/unmarshal cases in the same generator style. New tests pin the in-flight floor, the no-resurrection rule for removed peers, and that control messages are never nested in batches. * fix(filer): keep a removed peer's watermarks through a grace period Deleting a peer's watermark entries the moment the master removes it reopened the loss the watermarks exist to prevent: a filer frozen or partitioned long enough to miss master heartbeats is removed from the cluster, its unflushed events still exist, and with its entries gone the low-watermarks snap forward to the healthy peers - subscribers advance past the absent peer's window and its late-landing log files are silently skipped. Reproduced on a 3-filer cluster: freezing two filers for ~70s got them removed ~28s in, and a catching-up subscriber lost their entire overlapping window. Removal now only marks the peer; its watermarks keep participating in the low-watermarks for a grace period (2 x LogFlushInterval, matching the subscribe loops' settled horizon, which already bounds a stale watermark's influence meanwhile). A re-added peer clears the mark and continues its values monotonically - the flap case costs nothing. A peer that stays gone is dropped when the grace expires, so a decommission cannot pin the low-watermarks, and a dropped peer's straggling signals cannot resurrect its entry. * fix(filer): cap delivery heartbeats by the in-flight floor; harden stamps Second review pass on the watermark bounds: - Idle heartbeats on the local stream claimed delivery-completeness through "now" while an event could still sit stamped-but-unappended behind blocking notification work. A peer aggregator turns that claim into its delivery low-watermark, so it could advance (and anchor credits with it) past an event that had not been streamed yet. The heartbeat timestamp is now capped just below the oldest in-flight stamp, like the flush claim already was. - In-flight stamps are forced monotonic against the registry's own history, so a wall-clock step backwards cannot slip a new stamp under an already-sampled floor. The cross-goroutine ordering still shares the meta log's global forward-clock assumption; the comments now say so instead of overclaiming. - Duplicate removal notifications no longer refresh a removed peer's grace deadline: the first removal time wins, so a decommissioned peer cannot sit in the watermark sets forever on repeated updates. - A failed buffer append clears the event's in-flight stamp on purpose: the event is dropped from the change stream entirely (a pre-existing defect of the append path, loudly logged), and a watermark waiting for it would pin this filer's claims forever. The comments now state the decision instead of implying the failure cannot happen. * docs(filer): tighten the watermark comments Comment-only: compress the narrative comments added on this branch down to their load-bearing invariants, and fix one stale sentence (peer removal no longer deletes the watermark entries immediately). No code changes. * fix(filer): subscribe to the local filer before remote peers Self's events reach the aggregated buffer only through the aggregator's own subscription to it, but bootstrap only seeded the peers the master already listed - and self's master registration races that listing, so the watermark set could hold remote peers without self. Once the remotes signalled, the low-watermarks would claim completeness for a stream that was still missing a merge source, letting aggregated subscribers advance past the local filer's events before its subscription started. Seed self first, unconditionally: before that the watermark set is empty (a documented safe state - reads hold at the settled horizon), and after it the set can never be remotes-only. The later master update for self, or a duplicate in the listed peers, is a no-op via the already-followed check in OnPeerUpdate. * fix(filer): fence watermark claims against wall-clock regression Record issued heartbeat/flush claims in the in-flight registry and stamp later events above them, so a backward clock step cannot land an event under a watermark a peer has already advanced to. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(filer): re-check the buffer head after fencing heartbeat claims An event appended between the caught-up check and the delivery claim was covered by the claim but not yet sent on the stream. The claims fence later stamps, so re-checking the head after them proves every covered event was already sent before the heartbeat. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(filer): cross the aggregated ring's pre-subscription range only on proof The eviction gate and the gap proofs read "nothing evicted yet" as "memory holds everything after the cursor". That is false for the merge-fed aggregated ring, which is born empty while every peer's history sits on disk: before the ring's first real eviction, a subscriber whose cursor was still below the bounded chunk pass's listing stop was served the ring's earliest entry inclusively, silently skipping the withheld pre-restart files - and the idle-wait callback credited the delivery low-watermark to the disk anchor in the same disconnected state. Mark everything at or below the subscriptions' start as evicted when the aggregator is built, credit the anchor only once the run is connected to the ring, and give the aggregated gap pass a real proof to cross the marked boundary with: each disk pass's proven coverage (the peer flush low-watermark capped by the pass's listing bound). An empty pass whose proof reaches the eviction watermark crosses to it silently - no park, no loss counter - so the mark costs a bounded catch-up delay instead of the 15-minute give-up. * fix(filer): keep shipped chunk tails at or below the hold point A log file spans past its named minute (window start plus up to a flush interval), and chunk-mode clients apply a shipped file whole - so a file tail past the hold point can become a persisted client checkpoint beyond what every peer has proven, and a crash inside that window resumes past another peer's late-but-in-contract flush. Stop the ref listing a minute plus a flush interval below the hold; the withheld band is served by the memory pass (ring retention far exceeds it) or by later passes as the hold advances, so freshness is unchanged. A frozen peer flushing one window that spans its whole freeze can still overshoot; that residual is bounded by the freeze and needs a crash inside it. * docs(filer): trim the review-fix comments --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
3cf7d306a5 |
Give the WebDav chunk reader a bounded, invalidatable location cache (#10801)
* mount: re-resolve volume locations after a failed chunk read NewChunkGroup passed nil as the ReaderCache's CacheInvalidator, so retryFetchAfterCacheInvalidation was dead code on the FUSE read path. A mount that cached a volume's locations while one server was down kept retrying that server after it died, then returned EIO, even though the master and filer both resolved the live replica. The S3 gateway already passes its filerClient; do the same for the mount. * test: FUSE integration tests for volume server failover One mount appends while a second tails, and a volume server is killed, started or restarted mid-stream against a 001-replicated cluster of three volume servers. Automates the scenario matrix reported for Docker Swarm mounts, including the large-file variant and a no-chaos control. * test: report the filer's own view when append content mismatches A mismatch between what the writer wrote and what the reader sees can come from either side's cache. Read the file back through the filer's HTTP handler as well, and let the mount verbosity be raised from the environment, so a failing run says which layer lost the data. * test: wait for the reader mount to converge before comparing A mount caches metadata for about a second, so reading the file the instant the writer's last close returned can legitimately come back short. Poll the reader until it matches or the timeout expires; content that is wrong rather than merely late never converges and still fails, now with the writer's mount and the filer's own view alongside it. * test: detect a failover cluster child that exited at startup Signal(0) succeeds for a zombie and nothing reaped these children until shutdown, so a process that died on startup looked alive until the readiness timeout expired. Reap each child as it is started and consult the result. * test: read a file the killed volume server actually holds Placement decides which two of three servers back each volume, so killing volume N and reading readfile-N could pass without the victim ever holding a replica of it. Resolve each file's volumes through the filer and the master, and pick one the victim backs, preferring a file the reader has not cached. * ci: stop persisting checkout credentials in the failover workflow The job does not use the token after cloning. Also tag the README's command block as bash and match the timeout the workflow actually uses. * test: discard the ignored errors errcheck flags in the failover harness * test: resolve manifests when mapping a file to its volumes A manifest chunk's own fid names the volume holding the manifest, not the volumes holding the data, so a large enough file would point the failover victim at the wrong server. * test: pin the stale-location recovery path with a primed reader Reading a file for the first time after a server dies proves nothing: the lookup is fresh and returns the survivor. Kill one holder and wait for the master to drop it, read a file on that volume so the reader caches the lone survivor, restart the first server, then kill the survivor. The reader's only cached location is now dead while the data is live elsewhere, which is the case the invalidator exists for: EIO without it, recovery with it. * filer: re-look-up a chunk's locations as soon as they all fail A read that fails against every location it was given is far more likely to be holding a stale list than to be hitting a cluster that is briefly slow, but the retry loops spent the whole backoff ladder, about 13 s, before the caller got a chance to invalidate and look the chunk up again. Give the loops a refresh hook and let the reader cache invalidate on the first fully failed pass, so recovery starts in milliseconds. Clients without an invalidator keep the old behavior. The filer's streaming read path has its own fetch loop and is not covered. * webdav: give the chunk reader a bounded, invalidatable location cache WebDav resolved chunk locations through filer.LookupFn, whose own doc asks long-running processes to prefer wdclient.FilerClient: its cache is unbounded, and it has no way to invalidate an entry, so the reader cache was constructed with a nil invalidator and a WebDav server that had cached a location kept reading from it after the volume moved or died. Use FilerClient, as the mount and the S3 gateway already do. * filer: refresh locations on the random-read path too readChunkSliceAt bypasses the chunk cacher in random-access mode and fetches the range directly, which left it without the invalidation the cacher does: a random reader parked on a stale location had no way back at all. Hoist the refresh hook onto the reader cache so both paths share it. * filer: compare chunk locations as a set, not in order Lookups shuffle the locations they return, so comparing positionally reads a reshuffle of the very same replicas as a fresh set and spends an immediate retry on locations that just failed. weed/filer already had an order-independent comparison for this; move it next to the retry loops so both callers share one helper. |
||
|
Chris Lu
|
6fda8c67f3 |
Guard the gcs credential path in FetchAndWriteNeedle like the other backends (#10796)
* volume: accept only static-key gcs credentials on the fetch request An inline credentials document of a federated type points the SDK at a url, file or executable of the caller's choosing for the token exchange, so the request-supplied value is no longer just a key. * volume: guard the gcs token endpoint like the other remote endpoints Inline credentials pick where the token request goes, so route the gcs client through the same deny-list and rebinding-safe dialer used for S3 and azure. * rust volume: pin that gcs has no credential-driven dial path * volume: only check gcs credentials on a gcs remote conf Only the gcs backend reads that field, so another backend carrying a stale value should not fail the request. * gcs: load credentials with the type the caller expects The untyped loader is deprecated because it reads whatever the document claims to be; callers handling credentials they do not control now name the types they accept. |
||
|
Chris Lu
|
fbd85d31b0 |
ec.decode: check the rebuilt .dat is complete before the shards can be deleted (#10768)
A decode ends by deleting the shards it read, and the only thing standing between that and a bad reconstruction is verifyDecodedVolumeBeforeDelete, which asks whether .dat and .idx are non-empty. A .dat truncated to a single byte passes, and the shards -- the only other copy of everything past the cut -- are deleted on the strength of it. The server already knows the answer it never checks: FindDatFileSize returns the extent the EC index references, and WriteDatFile rebuilds to it. Compare the two once the file is written and fail the decode instead of reporting a short volume as a good one. Longer than the extent still verifies -- padding is not missing data -- so only a genuinely short rebuild is rejected. Needle counts cannot answer this: .idx is written from .ecx, so the count matches by construction and a truncated .dat still reports every needle. |
||
|
Chris Lu
|
602746f51d |
test: EC lifecycle chaos harness, with four fixes it found (#10763)
* ec: let the encode's balance see a migrating volume's shards across disk-type buckets Shard generation writes beside the source .dat, so a cross-tier encode (source on hdd, -diskType=ssd) leaves the fresh shards in the source disk-type bucket. The encode's internal balance ingested only the target bucket, saw no shards, and planned no moves; the spread guard then correctly aborted the encode (and before that guard existed, the shards silently stayed clumped on the generation host in the wrong tier). EcBalance now takes the encode batch as migratingVolumeIds and ingests those volumes' shards from every bucket, while everything else keeps the bucket filter so a plain ec.balance never drags deliberately tiered shards onto another disk type. The in-memory model delete also becomes bucket-agnostic: a node holds a given shard in exactly one bucket, and a bucket-scoped delete missed cross-bucket moves in the dry-run model. * volume: decode reads shard 0 from its resolved path, not the EC volume's base dir On a multi-disk server a volume's shards can sit on several disks; the store registers each shard with its own path and CollectEcShards resolves them, but FindDatFileSize derived the .ec00 path from the EcVolume's base directory. When shard 0 lived on a sibling disk, VolumeEcShardsToVolume failed with 'open ...ec00: no such file or directory' and ec.decode aborted. * ec: decode re-copies shards the topology claims but the target does not hold An interrupted earlier decode or balance can leave the master believing the decode target holds a shard whose file never landed: the mount registered but the partial copy was cleaned, or the file was swept. The collect step took the topology's word for it, excluded the shard from the copy set, and the decode failed with 'missing shard'. Probe the target's live inventory (VolumeEcShardsInfo) and treat anything it cannot serve as still-to-copy. * ec: decode discovers shards across disk-type buckets Shards sit wherever encode generation and balance left them: a cross-tier encode leaves them in the source disk-type bucket, a partial migration straddles buckets. ec.decode scoped its shard discovery to the -diskType bucket and reported a decodable volume as having no shards at all. Union across buckets, the way the encode's shard verification already does. * test: EC chaos lifecycle harness Randomized, seeded sequences of the EC lifecycle against a live cluster in the production-shaped layout: multiple data disks per server, a separate -dir.idx directory so .ecx/.ecj sidecars are shared across disks, and a tagged ssd tier. Operations cover encode (hdd and ssd targets), balance, shard damage plus rebuild, decode, re-encode, deletes, scrub, tier moves, crash-restarts, sidecar fault injections (a data-dir .vif pushed into the shared idx dir; a stale-generation shard planted beside a newer encode), and interruptions: a real weed shell subprocess killed mid-encode, mid-decode, and mid-balance, with the recovery re-run required to converge. One invariant holds after every step: every stored byte reads back identical and every deleted needle stays deleted. EC_CHAOS_SEED and EC_CHAOS_STEPS make runs reproducible and scalable. A known gap is tolerated and logged rather than fixed here: a shard mounted on two disks of one node (orphan adoption after an interrupted copy) is invisible to ec.balance's dedup and unaddressable by ec.shard.unmount's shard@address form, so no cleanup path exists yet. * test: fail payload-corruption checks on the test goroutine t.Fatalf inside require.Eventually's condition runs on the poller's goroutine, where Goexit kills only that goroutine and the corruption message can be lost behind a generic timeout. Record the mismatch, end the polling, and fail on the test goroutine. Also assert the full shard count in the cross-bucket decode-discovery test. |
||
|
Chris Lu
|
d713ab49f9 |
volume: validate replica targets and restrict gcs credentials in FetchAndWriteNeedle (#10755)
* volume: validate replica upload targets in FetchAndWriteNeedle The replica leg forwarded the fetched needle to a caller-supplied address without checking it, so a malformed target could redirect the upload to an unintended host or path. Require each replica target to be a bare host:port whose host is not loopback / link-local / unspecified, reusing the address deny-list; cluster peers legitimately sit on private networks, so RFC 1918 / CGNAT stay allowed and -volume.allowUntrustedRemoteEndpoints still opts out. Validate every target up front so a bad one fails the request before the local write, and upload through a client that re-checks the resolved address at connect time so a replica hostname cannot rebind to a blocked address after validation. Mirrored in Rust (validation moved ahead of the local write; the Rust S3 path's connect-time re-check is still a follow-up there). * volume: only accept inline gcs credentials in FetchAndWriteNeedle The gcs credentials value on this request could name a local filesystem path, which the SDK reads from disk. Accept only inline JSON here; the server-side GOOGLE_APPLICATION_CREDENTIALS env var still supplies a path. The Rust volume server has no gcs backend, so there is nothing to mirror. |
||
|
Chris Lu
|
9125b9c835 |
volume: extend the remote-endpoint guard to the azure backend (#10754)
* remote_storage/azure: allow a per-request HTTP client Thread an optional *http.Client through NewAzBlobClient and add azure.MakeWithHTTPClient, mirroring the S3 backend. When set, the client overrides the azblob transport so a caller can pin the dial path. The existing makers pass nil, so behavior is unchanged. * volume: extend the remote-endpoint guard to the azure backend The endpoint validation and rebinding-safe dialer in FetchAndWriteNeedle covered the S3-SDK backends. The azure backend also dials a caller-supplied AzureEndpoint, so route both families through a single guardedRemoteClient helper that returns the endpoint each backend dials and a constructor bound to the guarded HTTP client. azure is guarded only when AzureEndpoint is set; an empty endpoint derives the public host from the account. -volume.allowUntrustedRemoteEndpoints still opts out. * rust volume: assert the azure endpoint has no remote-client path The Rust volume server has no azure backend, so make_remote_storage_client rejects the type before any client is built. Add a regression test pinning that invariant. |
||
|
Chris Lu
|
7f27c572c4 |
log_buffer: end bounded reads that find the buffer empty (#10750)
A bounded LoopProcessLogData (stopTsNs set) on a buffer that never took a write since process start fell into the ResumeFromDiskError branch, which never checks stopTsNs when ReadFromDiskFn is nil and HasData() is false. The read parked on the notification loop forever while the subscription's idle heartbeats kept the stream looking alive, so a bounded SubscribeMetadata pass on a freshly restarted idle filer never completed. Terminate like the caught-up path does, returning a nil error: leaking the pending ResumeFromDiskError would latch the filer's outer loop into its gap machinery, which parks the bounded subscriber all over again. |
||
|
Chris Lu
|
4f50c5b0d4 |
feat: throughput limits for replicate, EC shard, and worker-driven moves (#10749)
* feat: throughput limits for replicate, EC shard, and worker-driven moves VolumeCopy was the only rate-limitable transfer; EC shard copies, replica creation, and worker-driven moves all ran at whatever the receiving server's maintenance rate allowed, with no per-operation control. - proto: VolumeEcShardsCopyRequest and the balance / ec_balance task params and configs gain io_byte_per_second; 0 keeps today's behavior (the volume server's own maintenance rate governs). - volume server: VolumeEcShardsCopy throttles with one WriteThrottler per request, shared across the shard, .ecx, .ecj, .vif, and .ecsum copies so the limit caps the transfer as a whole - the same shape as VolumeCopy. - volume_move: ReplicateVolume accepts the limit; EcMoveOptions carries it through MoveEcShards/CopyAndMountEcShards into the copy request, with fake-client tests asserting propagation. - shell: ec.balance gains -ioBytePerSecond; volume.tier.move's replication top-up honors the command's existing -ioBytePerSecond instead of running unthrottled. - worker: balance and ec_balance configs gain io_byte_per_second (surfaced in the admin config schema), carried through detection and plugin job parameters into task params and handed to the shared mover; batch balance jobs inherit the limit from their detection results. The limit is per copy stream, so maxParallelization multiplies the aggregate ceiling. * worker plugins: expose io_byte_per_second in the plugin config and derive it The plugin-driven detection path derives its task Config from the plugin configuration values, and both balance and ec_balance left IoBytePerSecond at zero there - a configured limit silently reverted to the server maintenance rate. Both derive functions now read the field (clamped at zero), and the plugin descriptors expose it with defaults so the configuration form carries it. |
||
|
Chris Lu
|
db5a086d04 |
read cold remote objects straight from the origin while caching (#10731)
* refactor: extract remote mount resolution into shared helpers * refactor: share the adaptive remote cache wait policy * filer: stream cold remote reads from the origin while caching * s3: stream cold remote reads from the origin instead of 503 retries * test: cover the S3 origin stream-through path * remote mounts: match on path components and prefer the longest mount * fail short origin streams instead of silently truncating * s3: try the origin before failing a cold read on a local cache error * s3: gate origin streaming on the entry's resolved version * return the cache RPC's NotFound as a canonical status and classify it everywhere * filer: keep multipart-range cold reads on the retry path |
||
|
Chris Lu
|
5b519489c1 |
remote_storage: build all S3-compatible clients through one constructor (#10720)
* remote_storage: build S3-compatible clients through one constructor The eight non-s3 S3-SDK providers each duplicated the AWS session setup and only the s3 maker could take a custom *http.Client. Route every S3-compatible type (s3, wasabi, b2, aliyun, tencent, baidu, filebase, storj, contabo) through MakeWithHTTPClient with a single options table, and add S3CompatibleEndpoint so callers can resolve the endpoint a given type dials. No behavior change. * volume: apply the remote-endpoint check to all S3-compatible providers FetchAndWriteNeedle validated the endpoint and used the pinned dialer only for type "s3". Every S3-SDK backend (wasabi, b2, aliyun, tencent, baidu, filebase, storj, contabo) dials a caller-supplied endpoint through the same client, so gate on S3CompatibleEndpoint to apply the same check uniformly. -volume.allowUntrustedRemoteEndpoints still opts out. * volume: don't route the guarded remote-endpoint client through a proxy The guarded client exists to dial the validated endpoint directly and re-check the resolved IP at connect time. With http.ProxyFromEnvironment set, the dialer only validates the proxy's address while the proxy re-resolves the endpoint host, which reopens the rebinding window. Drop the proxy on this path; operators that need one can opt out with -volume.allowUntrustedRemoteEndpoints. |
||
|
Chris Lu
|
c6e1387f59 |
shell: multi-target fs.mergeVolumes and volume.mark -readonlyCanDelete (#10706)
* shell: fs.mergeVolumes distributes one volume across multiple -toVolumeId targets * volume: volume.mark -readonlyCanDelete rejects writes but keeps accepting deletes * seaweed-volume: mirror readonlyCanDelete volume state |
||
|
Chris Lu
|
365d3e9e87 |
filer: TUS concatenation extension (#10702)
* filer: TUS creation accepts Upload-Concat partial uploads * filer: TUS final uploads concatenate completed partials * filer: TUS concatenation tests * filer: consumed marker pins TUS chunk ownership on completion * filer: TUS session delete decides chunk ownership after removing the session info * filer: TUS completion persists the consumed marker before creating the entry * filer: TUS completion re-verifies the session after persisting the consumed marker * filer: serialize TUS session ownership transitions per filer * filer: surface failed TUS consumed-marker rollbacks |
||
|
Chris Lu
|
753cb8cda8 |
master: stop copying the cluster to name it (#10700)
* topology: name a node's volumes without copying them ToVolumeLocations reads a volume id off every volume in the cluster, and got there through GetVolumes, which copies a whole storage.VolumeInfo per volume to be read for four bytes of it. Every client that connects asks for this. At 800k volumes the walk goes from 94.6MB to 16.0MB, which is the ids themselves. * master: log why a client send failed, not what was sent The message names every volume on a newly connected node, so a client going away had the master format a protobuf that size into text -- through the one log level that is always on. The error is the part worth having. |
||
|
Chris Lu
|
46ce8cbe84 |
master: stream volume listings (#10676)
* master: stream volume listings A listing of 800k volumes is 36MB on the wire but 305MB as messages, and the master built all of it, then held it while grpc encoded it. Two of those at once is most of a small master's heap, and the maintenance scanner asks every 30 minutes. The topology goes out first, listing nothing, then its volumes in batches, so the master holds a batch rather than a cluster: 341MB of live heap for one listing becomes 4.4MB. It allocates much the same either way -- what changes is how much of it has to be live at once, which is what sets the heap ceiling. Batches are built under their disk's lock and sent outside it, so a slow reader stalls the stream rather than the topology. They therefore do not share one instant, which a single listing did not either: it takes each disk's lock in turn, so a volume moving during either can be seen twice or not at all. The client helper hides which kind of master answered: one too old for the stream is asked the old way and its reply cut into the same batches. Either way the topology handed over lists no volumes, so a caller cannot come to depend on finding them there. * admin: stream the listing the maintenance scan reads It asks for every volume in the cluster every 30 minutes. Reassembling it client-side keeps the scan identical -- ActiveTopology splits disks by the disk ids on the volumes, so it needs them in the topology -- while the master no longer builds the whole reply to send it. |
||
|
Chris Lu
|
00c5572e8c |
volume: decode IPv6 transition addresses in the remote-endpoint guard (#10683)
* volume: decode IPv6 transition addresses in the remote-endpoint guard checkBlockedIP normalized only ::ffff: mapped IPv4, so NAT64 (64:ff9b::/96), 6to4 (2002::/16), Teredo (2001:0000::/32), and IPv4-compatible (::/96) addresses that embed an internal IPv4 (loopback, 169.254.169.254, RFC 1918) passed the endpoint guard even though the plain IPv4 forms are refused. Extract the embedded IPv4 from those forms and re-check it against the deny list, which covers both the up-front validation and the dial-time guard. Mirrored in the Rust volume server. * volume: require the full NAT64 well-known prefix before decoding Only 64:ff9b::/96 carries the embedded IPv4 in the low 32 bits, so also require bytes 4-11 to be zero before treating an address as NAT64; other 64:ff9b: prefixes place the IPv4 elsewhere and are left untouched. Add public-target coverage for 6to4, Teredo, and IPv4-compatible so every decoder is exercised on both a blocked and an allowed destination. Mirrored in the Rust volume server. |
||
|
Chris Lu
|
3911e4c548 | master: keep a racing registration out of a dying collection (#10677) | ||
|
Chris Lu
|
a2ff9cca27 |
master: let VolumeList ask for the volumes it wants (#10674)
* master: let VolumeList ask for the volumes it wants The request carried nothing, so every caller was answered with the whole cluster. A dashboard opening one volume's page, or a capacity probe adding up one bucket, was served all 800k of them and threw away the rest -- and the master built every one of those messages first. The topology, its disks and their counters are still reported in full: a caller reading free space or replica placement needs the cluster whichever volumes it asked about. Only what is listed under a disk is selected, ec shards included. An empty collection and a zero volume id take everything, the way volume.list already reads its own -collectionPattern and -volumeId, so a caller that forgets to narrow is answered too much rather than answered wrongly. That leaves the default collection unnameable, since it is the one the empty string names, so it gets a field of its own. An older client sends none of it and is answered exactly as before. * admin: ask the master for the volume the page is showing A volume's detail page was pulling every volume in the cluster to find one and its replicas, and discarding the rest. * admin: ask the master for the ec volume the page is showing Same as the volume detail page: one volume's shards were found by pulling every ec shard in the cluster. * s3: ask the master for the bucket's own collection The SOSAPI capacity probe summed one collection's volumes out of a listing of every volume in the cluster. Cluster capacity still comes out the same: it is read from the disk counters, which a filtered listing reports in full. * topology: read the disk usage counters atomically They are written with atomic.AddInt64 from heartbeats but were read plainly by the two listings and by FreeSpace, and the map they sit in was iterated without the lock its neighbour takes. Under -race a listing concurrent with a heartbeat trips on both. |
||
|
Chris Lu
|
f09e8345c6 |
storage: stop keeping the remote storage key on the master (#10672)
A master decides nothing from it. Every caller that read it was asking whether a volume is remote, which the backend name answers, and the value itself is reported on demand by the server holding the volume, through the volume info in ReadVolumeFileStatus. It is also the one string here that cannot be shared: unique per volume, so unlike the collection and backend names it carries its own characters for every volume a master tracks. VolumeInfo goes from 136 bytes to 120. 800k volumes registered from a heartbeat that has been over the wire go from 214 to 163 B/volume when tiered. The volume server's own status page keeps showing the key, now read from the volume it holds rather than relayed through a master, which is also where the other volume server implementation reads it. The heartbeat digest drops it on the same grounds: a change to something the master does not hold cannot make its copy stale. Both implementations and their shared vectors move together, and the field-coverage test now names what is deliberately not retained rather than being loosened. |
||
|
Chris Lu
|
567052bfb6 |
s3: take bucket sizes from the master's summary (#10664)
* pb: ask the master what each collection holds Callers tracking usage were sent every volume in the cluster to add up themselves, which is the master's largest single allocation. * topology: summarise what each collection holds One pass over the topology, allocating per collection rather than per volume. Regular volumes count once each for logical totals and once per replica for physical, taken from the lookup index, which is already keyed by volume and so needs no set of seen ids. Ec shards are node-local so their sizes sum, while the file and delete counts describe the volume and resolve once every holder has been seen. Replicas of one volume disagree while a write is landing or a heartbeat is late. Walking a full listing took whichever replica the map iteration reached first, so the answer moved between runs; this takes the largest, which is stable and never reports usage below what some replica already holds. * s3: take bucket sizes from the master's summary The bucket size metrics pulled the whole volume list once a minute and added it up, which cost the master 184.6MB of allocation and 17.8MB on the wire for six numbers per collection. VolumeList over 550k volumes 184.6 MB allocated, 17.8 MB on the wire CollectionStatistics 176 bytes allocated, 47 bytes on the wire The aggregation moves to the master with it, so the cases the removed tests covered are now asserted against it directly. * topology: count the replica holding the most live data Quotas are enforced on size less deletions, and the replica with the biggest raw size can be the one that has deleted the most. Counting it reported a bucket smaller than it is and would leave one writable over its quota, which is the opposite of what picking the largest was meant to guarantee. * topology: cap a volume's deletions at what it holds Live usage is read as a collection's size less its deletions, so a volume reporting more deleted bytes than it has cancels live bytes belonging to other volumes in the same bucket and reports it smaller than it is. Replica selection already floored that volume's own live size at zero; the totals have to agree with it. |
||
|
Chris Lu
|
a2ffc7aadf |
heartbeat: keep the master current through collection churn (#10657)
* heartbeat: name departed volumes in delta heartbeats * master: release the lookup index with a deleted collection * master: keep a fresh grow safe from the report that raced it * volume: name the volumes a deleted collection took with it Deleting a collection left the master to work out what went by omission from the next full volume list, which it no longer gets: heartbeats carry the whole list only when the master asks for it. The volumes a bucket's churn creates and destroys between two of those requests are never named in either direction, so the master keeps counting their slots as occupied and a cluster that creates and drops collections quickly runs its free-slot accounting dry -- assigns fail with no free volumes left while the disk holds a handful of volumes. The destroy path already knows exactly which volumes it removed, so send them down the same channel every other deletion uses. * rust: name the volumes a deleted collection took with it Mirrors the Go volume server. The notify path derives its deltas by diffing snapshots, so a collection delete that does not wake it is invisible until the master next asks for the whole list. |
||
|
Chris Lu
|
3a61debaa5 |
filer: rebuild peer metadata subscriptions after a master reconnect (#10648)
* filer: keep the existing peer subscription on a repeated add A cluster node add for a peer that is already followed restarted the subscription, dropping the metadata events between the two runs. * master: tell a connecting client the current cluster membership Cluster node updates are only broadcast to the clients connected at that moment. A filer that lost its master stream while a peer came back never learned about the peer, and stopped replicating its metadata for good. * test: a filer joining the master learns about the filers already there * test: a filer resubscribes to a peer that registered while it was disconnected Runs the reported sequence against real processes: filer2 leaves, filer1 is paused and its master stream is broken, filer2 registers again, and filer1 has to replicate from it after reconnecting. |
||
|
Chris Lu
|
37f3dff677 |
volume: validate the file extension in CopyFile and ReceiveFile (#10644)
* volume: validate the file extension in CopyFile and ReceiveFile
CopyFile and ReceiveFile build an on-disk path from the client-supplied
Ext. Both are intentionally ungated for cluster-internal peers, so a
value like "/../../x" is joined onto the volume directory and, once
path-cleaned, resolves outside it -- an EC-shard receive can then write,
and CopyFile read, anywhere the process can reach.
Constrain Ext to a real suffix (a leading dot followed by alphanumerics)
before it is used to build any path, so it can no longer carry a
separator or a parent reference.
* test: use an alphanumeric missing-file extension in the copy variants
The not-found and stop-offset-zero cases used ".definitely-missing" as a
deliberately absent source. The extension is now validated, and the hyphen
makes it invalid, so switch to ".missing" -- still a nonexistent file, but a
real extension shape.
* volume: validate the collection in CopyFile and ReceiveFile
The client-supplied Collection is folded into the on-disk path as
"<collection>_<vid>" by VolumeFileName and EcShardBaseFileName, both joined
with path.Join / util.Join. A Collection carrying a separator, e.g.
"../../x", therefore path-cleans to a target outside the volume directory,
the same escape the extension check just closed. Reject a collection that is
a bare parent reference or holds a separator; ordinary names ('.', '-' and
all) still pass.
|
||
|
Chris Lu
|
e9cde3e4b1 |
master: gate raft membership RPCs behind the admin whitelist (#10649)
* master: evict a dead peer via the local raft handle OnPeerUpdate only runs on the leader, and the AddVoter branch right above mutates the local raft directly. The remove branch instead dialed our own RaftRemoveServer back over gRPC. Drop the self-dial and remove the peer through the local handle, matching the add path. This also leaves operator tooling as the only caller of the RaftRemoveServer RPC. * master: require whitelist auth for raft membership RPCs RaftAddServer, RaftRemoveServer and RaftLeadershipTransfer rewrite raft quorum but had no caller check beyond "am I the leader". Any client that could reach the master gRPC port could add an unreachable phantom voter and stall the write path. Gate the three on the admin whitelist, mirroring the volume server's checkGrpcAdminAuth. With no whitelist configured the guard allows every caller, so default and single-master deployments are unaffected; operators who set -whiteList get these RPCs locked down to it. The leader's own dead-peer eviction no longer dials these RPCs, so the only remaining callers are operator tooling. |
||
|
Chris Lu
|
ce7d388639 |
heartbeat: send only the volumes that changed (#10640)
* pb: let a heartbeat carry only the volumes that changed A partial list cannot travel in volumes: a master that did not understand it would read the absences as deletions. So changes get their own field, used only once the master has said it compares digests and can tell when it has fallen behind. * master: apply the volumes a heartbeat reports as changed Only the named volumes are touched. A full report says the server holds exactly these; a changed report says nothing about the ones it leaves out, so absence must not read as removal. Also advertises that the master compares digests, which is what lets a server stop sending its whole list. Advertising it once per connection means a server reconnecting to a master that does not is back to full lists straight away. * volume: send only the volumes that changed once the master accepts them The whole list goes on every heartbeat until the master says it compares digests, and again whenever it asks, so a master that cannot tell when it has fallen behind never has to. has_no_volumes stays derived from a full list alone. Deriving it from what a heartbeat happens to carry would make a quiet one read as a server that had lost every volume, and the master would drop them all. The digest still covers every volume held rather than the ones sent, which is what lets the master confirm that applying the changes left it current. Reporting state is per-connection: a server that reconnects, or reaches a different master, starts again from the full list. * volume: let the zero reporting state stand for having told no master anything A Store built as a literal, which tests do, left the reporting state nil and panicked on the first heartbeat. As a value its zero form already means nothing has been reported to anyone, which is exactly the state that sends the whole list. * rust: send only the volumes that changed once the master accepts them Mirrors the Go volume server, with one hazard the Go side does not have: mount and unmount deltas here are derived by diffing successive heartbeats, so a heartbeat that carries a partial list would report every volume it left out as unmounted. Collecting now returns the full set alongside the message, and every site that diffs uses that rather than what went on the wire. * volume: do not let a full-list request be lost to the heartbeat it raced The request arrived while a heartbeat was already being built as a delta, and committing that heartbeat cleared it, so the master waited for another digest mismatch before asking again. Count the requests and clear only the one the heartbeat answered. * rust: stop marking volumes reported by a heartbeat that is thrown away The state-notify path collected a heartbeat only to diff its volume list, then sent a delta message of its own and dropped the one it had collected. Once collecting recorded what the master had been told, every mount or unmount silently marked the changed volumes as sent, and the master learned of them only after a digest mismatch. Snapshotting no longer records anything, and no longer expires ec volumes whose deletion that path was already discarding. * master: announce only the volumes a change actually brought Every changed volume was broadcast as a new location. Volumes grow constantly and growth moves no location, so on a busy cluster that told every connected client about volumes it could already reach, filling bounded broadcast queues and pushing out the topology updates that matter. * master: ask for the full list when only one can repair the master Delta heartbeats stop the full report, and with it the only thing that re-registers a volume the lookup index lost. The volume server cannot see that divergence and its digest cannot show it, so the master now checks its own two indexes agree and asks for the list when they do not. A node reporting one volume id twice is kept on full lists for the same reason rather than merely skipped: its digest can never be verified, so nothing else would tell the master what it had stopped holding. * master: keep the volume options on every heartbeat response A volume server takes them from whatever response arrives, and preallocate is a bare bool with no way to tell off from unmentioned. A response sent to ask for the volume list therefore turned preallocation off until the server reconnected. Responses sent mid-stream now start from the configured options rather than being built field by field. * master: announce a volume the lookup index had lost Repairing the index makes the volume servable again, but clients were told it went when the node dropped out and nothing told them otherwise: the disk map still held it, so it did not count as an arrival. Reaching the lookup index is what makes a volume servable, so recovering an entry there is an arrival as far as clients are concerned, on both the full report and the changed-volume path. |
||
|
Chris Lu
|
cab666fca1 |
filer: configurable TUS max upload size and session expiry (#10638)
* make TUS max upload size and session expiry configurable * default TUS session expiry to 24h |
||
|
Chris Lu
|
5ec813b4f1 |
topology: follow a volume that moved between a server's disks (#10628)
* topology: follow a volume that moved between a server's disks The heartbeat diff asked only whether a volume id was reported anywhere on the node, so a volume that moved to a disk of another type stayed on the disk it left as well. The master then held two copies of it forever: the volume count was overstated, and GetVolumesById returned whichever disk the map iterated first, so lookups could hand back the disk the volume had already left. Track which disk types the heartbeat named each volume on, and treat a volume named on another disk as absent from this one. Disk types are interned to an index because a server reports a handful of them across hundreds of thousands of volumes. A volume named on two disks at once is a stale twin rather than a move, and is still kept on both -- dropping one would tell the master a replica vanished. Only a volume named twice on one disk type is unrepresentable, so that is now what marks the node, rather than any repeat of an id. * master: do not tell clients a moved volume left the node A volume moved between a node's disks is removed from one and added to the other, so it lands in both lists of the same heartbeat. Clients apply additions before deletions, so the removal wins and they end up with no location for a volume that never went anywhere. Skip removals for volumes the node still holds, as the ec shard paths already do, and update the topology before judging the delta removals so an unmount that really did happen is still reported. * trim the comments on this change to the parts that are not evident * master: judge a volume removal on normal replicas alone HasVolumesById answers for ec shards as well, so a replica encoded into ec shards looked like it was still on the node and clients were never told the normal location had gone. They hold normal and ec locations separately and prefer the normal one from the same generation, so that location would have gone on shadowing the shards. |
||
|
Chris Lu
|
6d08b08f37 |
heartbeat: carry a volume digest and verify it (#10627)
* pb: carry a volume digest on the heartbeat The full volume list is the only way a master notices a volume that vanished without a delta, so it cannot simply be dropped. A digest gives the same guarantee without the list, and a way back to the list when they disagree. The digest has explicit presence: a server holding no volumes reports 0, which has to stay distinguishable from a server that does not compute one at all. * volume: report a digest of the volumes each heartbeat carries Digests exactly what goes on the wire: volumes skipped as quarantined, phantom or expired are absent from both the list and the digest, so the master compares against the same set the server meant to report. Runs the master's own hash over the master's own conversion of the message, so the two ends cannot drift into disagreeing about a field. * master: check the reported volume digest and ask for the list on a mismatch Compared after everything the heartbeat carried has been applied, so agreement means the master is current rather than that nothing changed. Servers reporting no digest are untouched, and a mismatch on a heartbeat that already carried the full list is reported rather than answered: there is nothing further to ask for, so asking again would loop. Nodes reporting one volume id twice are skipped for the same reason. * rust: report the heartbeat volume digest Mirrors the Go volume server. The master compares this against a digest it computes itself, so the hash has to agree byte for byte across the two implementations, not merely be a hash of the same fields: report_hash_vectors pins it against values generated by the Go side, and the ttl and replica placement narrowing the master applies when it decodes a message is applied here too rather than assumed away. A drift there would not corrupt anything, but every volume server on this implementation would report a digest the master can never match and fall back to sending its whole volume list forever, which is the cost the digest exists to avoid. * master: pin what the digest check does to each kind of report The upgrade story rests on these: a server that reports no digest is never asked for anything, so the two sides can be upgraded in either order, and a disagreement that resending cannot fix is reported rather than re-asked, so it cannot loop. * topology: enumerate the digest coverage test from the message The list of fields was written out by hand, so a field added to VolumeInformationMessage later would fall outside the digest while the test went on passing, and a change to it would never reach the master. Walk the message descriptor instead. Some fields are narrowed or normalised on the way into VolumeInfo, so the smallest change to the wire value can land back on the stored one; the test offers several values per field and asks only that some change is visible. |
||
|
Chris Lu
|
b46946ece5 |
filer: list directories without decoding chunk lists (#10616)
* filer: decode a listed entry without building its chunk list
A readdir reads attributes and never looks at chunks, but decoding an
entry builds the whole chunk list first: four allocations per chunk, all
of it thrown away. On a directory of ordinary 4MB-chunked files that is
most of what listing costs.
DecodeAttributesOnly walks the wire format and hands everything except
the chunks to the generated unmarshaller, so new fields in filer.proto
need no attention here. The chunks are still measured, because the S3
copy and multipart paths deliberately store a zero FileSize and let the
chunk extents define the size, but nothing is allocated to do it.
The blob is only re-encoded once a chunk is actually seen, so an entry
without any -- every directory, for one -- is unmarshalled where it lies
and pays nothing for the walk.
Listings opt in through the context, the way the lazy remote paths
already do; a store that ignores it stays correct.
chunks full attrs-only allocs
0 312.8n 310.1n ~ 1 -> 1
1 686.1n 411.1n -40.07% 7 -> 1
4 1.742u 667.4n -61.69% 24 -> 1
16 5.770u 1.544u -73.25% 86 -> 1
64 25.23u 6.004u -76.20% 328 -> 1
* mount: list directories with chunk lists omitted
The two meta cache listings behind a readdir are the only callers, and
neither reads a chunk. On 200k single-chunk files one enumeration goes
from 364ms to 277ms and drops a million allocations.
The read-through listing still fetches whole entries from the filer,
which would need the request to say it wants attributes only.
* mount: give the readdir benchmark's entries a chunk
Chunkless entries made the decode look far cheaper than it is, which is
the part of a listing worth measuring.
* filer: let a listing ask for entries without their chunk lists
The read-through readdir fetches whole entries over gRPC, and for a wide
directory the chunk lists are most of what crosses the wire and most of
what the client then unmarshals. A 4MB-chunked file is 113 bytes of
entry against 46 without its chunk.
ListEntriesRequest gains omit_chunks. The size a client needs is already
in the attributes, where the store decode folded the chunk extents in,
so dropping the list costs the client nothing.
The filer still reads the entries whole. A listing is where a TTL-expired
entry gets collected and deleted, and deleting one needs its chunks to
find the data, so omitting them there would leak. Only the response is
trimmed.
The hint moves to filer_pb so one context flag serves both transports:
the gRPC request sets omit_chunks, and a listing served from the local
store skips building the chunks. Cache population is unaffected either
way, since EnsureVisited starts from its own context.
* filer: reject a chunk the full decoder would reject
The walk skipped a chunk's bytes without looking inside them, so a
FileChunk carrying a corrupt nested fid, or a string that is not valid
UTF-8, sailed past the listing decoder while every other read of the same
entry still failed. The file listed with a plausible size and then gave
EIO on open, and corruption that used to fail the listing loudly was
hidden instead.
The chunk bytes are the one part of the blob the generated unmarshaller
never sees, so the two checks it would have made are made here: a
submessage has to parse, and a proto3 string has to be valid UTF-8.
FileChunk's only submessages are FileIds of scalars, so walking them is a
complete check. A descriptor-driven test fails if FileChunk ever gains a
field of either kind that the walk does not know to check, which is the
part that keeps this honest as filer.proto grows.
Taking the scratch buffer lazily, only once a chunk is actually dropped,
also takes the pool out of the path for entries that have none. Those
were measurably slower than the full decoder before; they are now level
with it. Each chunk's length prefix is parsed once rather than twice.
chunks full attrs-only vs base
0 171.4n 176.9n ~ (p=0.670)
1 366.6n 259.4n -29.24%
4 1.034u 500.2n -51.60%
16 3.905u 1.464u -62.52%
64 13.48u 5.195u -61.46%
* filer: carry the size before dropping chunks over the wire
Dropping the chunk list assumed every store folds the chunk extents into
FileSize when it decodes. A store that keeps entries as JSON rather than
as an encoded Entry never re-derives it, so an object written with a zero
FileSize kept its real size only in the chunks, and stripping them left
the client reading the file as empty. Stamp the size into the attributes
first, which costs nothing and does not depend on how the store loaded
the entry.
* mount: test that the readdir context reaches the store decode
Everything else exercises the decoder directly, so a refactor that
stopped threading the context would have reverted the whole thing with
every test still passing.
The benchmark's chunks also carried a constant legacy FileId, which
BeforeEntrySerialization reparses over Fid on the way in, so all 200k
entries stored one byte-identical chunk rather than the varying fixture
it looked like.
|