mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-15 19:10:48 +02:00
c194924d13de12e2eb21e94f7987bb07fecc98b5
89
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Yaroslav HalchenkoandClaude Code 2.1.217 / Claude Opus 4.7
|
490379bff3 |
Add codespell support with configuration and typo fixes (#10393)
* Add GitHub Actions workflow for codespell on master * Add rudimentary codespell config * Tune codespell config: skip generated code, ignore camelCase, whitelist domain terms Add camelCase/PascalCase regex to ignore common Go/Rust/JS identifiers like allLocations, publishErr, ReadInside, FlushInterval. Also skip templ-generated *_templ.go files, and whitelist a handful of short/domain-specific words (visibles, fo, te, ser, bject, unparseable, keep-alives, tread, anc, ue) that show up as false positives across the tree. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix ambiguous typos and protect false positives Fixes typos that codespell reports with multiple candidate suggestions (so `codespell -w` cannot auto-apply them), plus one inline pragma and one config entry to protect legitimate identifiers. Manual fixes (single correct answer chosen from context): - pattens -> patterns (5x) in filer/upload/shell flag help strings - finded -> found (2x) in tarantool storage.lua comment - spacify -> specify (2x) in helm chart values.yaml comment - wether -> whether in skiplist.go docstring - simpe -> simple in mq schema test case name False-positive protection: - Add `//codespell:ignore` next to `source GET's` (possessive of HTTP verb) in s3api_object_handlers_copy_stream.go - Whitelist `auther` in .codespellrc — it's a local variable meaning "authenticator" in weed/security/tls.go, not a typo of "author". Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Extend codespell ignore list: .git-meta path and thirdparty groupId Also skip `.git-meta` (scratch dir for commit messages that may contain typo words verbatim) and whitelist `thirdparty` — it appears as the literal Maven groupId `org.apache.hadoop.thirdparty` in hdfs3 poms and cannot be renamed. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [DATALAD RUNCMD] Fix non-ambiguous typos with codespell -w Auto-applied fixes to the 44 remaining single-suggestion typos across docs, comments, log messages, tests, config, and one Java pom. === Do not change lines below === { "chain": [], "cmd": "uvx codespell -w", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [], "pwd": "." } ^^^ Do not change lines above ^^^ * Revert breaking codespell fixes; whitelist unknwon and atleast Two of the auto-applied `codespell -w` fixes were false positives that would break the build/tests: - go.mod: `github.com/unknwon/goconfig` is a real Go module path — the upstream author's GitHub handle is literally `unknwon`. Renaming to `unknown` would fail dependency resolution. - test/benchmark/fuse_db/bin/{sqlite_verify.py,run_mysql.sh,run_sqlite.sh}: `atleast` is a literal CLI mode value (a string constant compared and passed as a positional argument). Rewriting to `at least` splits it into two arguments and breaks the mode check. Reverted those files and whitelisted both words in .codespellrc so future runs won't re-suggest the same broken fixes. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris Lu
|
5a54beac80 |
EC decode: read shards with the encode-time block layout (#10385)
* erasure_coding: WriteDatFile takes the encode-time dat size for the shard block layout * volume server: derive EC decode layout from the encode-time dat size, not the live extent * erasure_coding: test decode after tail deletions shrink the live extent below a large-block row * seaweed-volume: write_dat_file_from_shards takes the encode-time dat size for the shard block layout * seaweed-volume: derive EC decode layout from the encode-time dat size, not the live extent * seaweed-volume: test decode after tail deletions shrink the live extent below a large-block row * erasure_coding: reject decoding with no data shards * worker: record the encode-time dat size in the .vif * erasure_coding: fall back to the shard-derived layout only when the encode-time dat size is missing * erasure_coding: reject an ambiguous shard-derived block layout * seaweed-volume: fall back to the shard-derived layout only when the encode-time dat size is missing * seaweed-volume: reject an ambiguous shard-derived block layout |
||
|
Chris Lu
|
267f595660 |
batch delete: align the shard test and Rust server with continue-past-mismatch (#10349)
Commit
|
||
|
|
8bff3b3213 |
fix(volume): reject overflowing needle ID deltas (#10342)
* fix: reject overflowing needle ID deltas Problem: Parsing a file ID with a delta can wrap a valid maximum needle ID back to zero without returning an error. Root cause: Needle.ParsePath added the parsed uint64 delta without checking whether the sum exceeded the needle ID range. Fix: Compare the delta with the remaining uint64 capacity before addition and return a contextual overflow error when it does not fit. Validation: go test ./weed/storage/needle -run ^TestNeedleParsePathRejectsDeltaOverflow$ -count=1; go test ./weed/storage/needle -count=1; git diff --check 10cdaf381875492a2c752d1038797e96ff18208f..HEAD Co-authored-by: Codex <noreply@openai.com> * fix: propagate needle ID delta parse errors Co-authored-by: Codex <noreply@openai.com> * print the needle id in hex in the delta overflow error * batch delete: keep processing after a cookie mismatch * rust volume: reject overflowing needle id deltas --------- Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
bd9b5c25ff |
rust volume: verify the .dat ends at the last indexed needle (#10320)
* rust volume: verify the .dat ends at the last indexed needle The Go loader quarantines a volume whose .dat extends past the last indexed needle - the leftover of a torn shutdown - but the Rust check only verified header fields of the trailing index entries and never compared file sizes, so a torn tail loaded clean and writable. Appends land at the raw file end, and past a misaligned tail the next needle sits at an offset the 8-byte-unit .idx encoding rounds down, pointing the index a few bytes before the needle. Replace the last-10-entries walk with the current Go shape: find the entry physically last in the .dat (append-ordered fast path, max-offset scan for key-sorted rebuilds), verify that needle - tombstones with their on-disk Size=0 - and require the file to end exactly at it, marking the volume read-only otherwise. Claude-Session: https://claude.ai/code/session_01XgGXMLknzaNgQzHMyo2Vhb * rust volume: buffer the .idx max-offset scan The slow path read one 16-byte entry per syscall; a BufReader batches the sequential scan like Go's WalkIndexFile does. Claude-Session: https://claude.ai/code/session_01XgGXMLknzaNgQzHMyo2Vhb |
||
|
Chris Lu
|
c1a1e3c1e3 |
shell: volume.tier.upload keeps volume replicas (#10314)
* volume: copying a remote-backed volume only needs space for the index VolumeCopy sized its target-location check by the source .dat even when that .dat lives in a cloud tier and only .idx/.vif land locally, so re-replicating a tiered volume demanded the full remote size in free disk. Require the index size instead. * shell: volume.tier.upload keeps volume replicas Tiering a replicated volume deleted every replica but the upload source, leaving one server holding the only .idx and the only .vif that knows the remote object key — losing that server orphaned the volume even though its data sat intact in the cloud. Replicate the uploaded .idx/.vif onto the other replica servers instead (VolumeCopy skips the .dat for remote-backed volumes), so all replicas serve reads from the same remote object and the volume keeps its replica count. An already-tiered replica is preferred as the upload source, so a rerun after a partial failure reuses the existing remote object instead of uploading a second copy under a new key. * shell: group tier upload locations instead of re-prepending * rust volume: copying a remote-backed volume only needs space for the index Mirror the Go VolumeCopy change: size the free-location check by the source .idx when the .dat lives in a cloud tier, since only .idx/.vif land locally. |
||
|
Eliah RusinandChris Lu
|
c006dc563e |
ec: remove .ecsum sidecars on destroy / shard delete; align Go and Rust cleanup (#10307)
* fix(rust-volume): remove .ecsum sidecars on EC destroy / shard delete Rust EcVolume::destroy removed shards and .ecx/.ecj/.vif but left bitrot checksum sidecars (.ecsum / .ecsum.v*). On clusters that run weed-volume (not Go weed volume), collection.delete therefore orphans every sidecar while correctly wiping shards — observed live on 4.39 (14/14 .ecsum survived after collection.delete on a freshly encoded EC volume). Go Destroy already calls RemoveBitrotSidecars; this brings Rust to parity: - hoist remove_bitrot_sidecars into ec_bitrot (shared helper) - call it from EcVolume::destroy for dir / dir_idx / ecx_actual_dir - call it from Store::delete_ec_shards when a disk has no remaining shards - unit test: test_destroy_removes_bitrot_sidecar * rust volume: gate the shard-delete sidecar sweep on a local shard removal Only sweep a disk's .ecsum when this delete actually removed a shard file there, matching Go's found gate: a delete that never touched a disk must not strip a sidecar it does not own — a shared -dir.idx sibling with surviving shards, or an ec.rebuild index-prep copy that lands .ecx/.ecsum before any shard. The shard-presence probe now treats unexpected stat errors as "exists" so a transient failure cannot orphan-classify live shards, and check_all_ec_shards_deleted reuses it. * rust volume: destroy() sidecar sweep needs only the data and idx bases ecx_actual_dir is always one of the two, so the third branch could never run; this is now exactly Go Destroy()'s two-base sweep. * rust volume: call the shared sidecar removal helper directly * rust volume: unit-test remove_bitrot_sidecars Mirrors Go's TestRemoveBitrotSidecars: legacy and versioned sidecars are removed, a shard file and a longer-vid sidecar survive, absent is success. * rust volume: keep the shared idx-base sidecar while a sibling disk has shards One -dir.idx serves every location, so emptying one disk must not sweep <idx>/<vol>.ecsum out from under a sibling that still holds shards. Nothing reads the idx-base sidecar today, but .ecx shows index-dir files are real; this keeps the defensive sweep safe if a writer ever lands one there. * ec shard delete: keep the shared idx-base sidecar while a sibling disk has shards One -dir.idx serves every disk, so emptying one disk must not sweep <idx>/<vol>.ecsum out from under a sibling that still holds shards of the volume — the same gate the Rust volume server applies. A status error counts as in-use so a transient failure never strips it early. * rust volume: drop a shard-only disk's stale .vif with the node's last shard Go's removeEcSharedIndexFiles also clears the data-base .vif in the all-shards-gone pass, gated on .idx absence so a disk still hosting the source volume keeps its live .vif; the Rust delete path left it behind. Unexpected stat errors count as .idx-present so a transient failure never strips a live volume's .vif. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
6f816c955d |
volume.fsck: fix orphan purge against the rust volume server (#10289)
* rust volume: accept odd-length needle id hex in file ids Go formats the needle id with strconv.FormatUint and parses it back with strconv.ParseUint, neither of which pads to an even number of hex digits. hex::decode rejected such file ids with "Odd number of digits", so volume.fsck could not purge orphans from a rust volume server. Parse the needle id and cookie with from_str_radix, matching Go's ParseNeedleId and ParseCookie. * storage: emit even-length needle id hex in NeedleId.FileId volume.fsck and volume.check_disk build purge file ids here with unpadded FormatUint hex, while every other fid formatter strips whole leading zero bytes. Pad to even length so the output matches the canonical fid format and strict hex parsers accept it. |
||
|
Chris Lu
|
e98cbfc8f1 |
seaweed-volume: async, buffered writes in VolumeEcShardsCopy (#10237)
* seaweed-volume: async, buffered writes in VolumeEcShardsCopy The EC-shards-copy RPC handler wrote each streamed chunk to disk with a synchronous std::fs::File::write_all inside the async handler, blocking a Tokio worker thread for the duration of every write — noticeable for a large .ecx on a slow or busy disk. Factor the five near-identical receive-and-write loops (.ec shards, .ecx, .ecj, .vif, .ecsum) into drain_copy_stream_to_file, which uses tokio::fs + BufWriter for async, buffered I/O. Behavior is otherwise unchanged: the .ecj append mode, the .ecsum byte count and 0-byte-file cleanup, and all error messages are preserved. Claude-Session: https://claude.ai/code/session_01Ny5Rt1ph9VWeKmfY936GtF * seaweed-volume: remove partial copy target on error in EC-shards-copy Follow-up: drain_copy_stream_to_file now deletes the destination file on any recv/write/flush error, so a failed VolumeEcShardsCopy no longer leaves a truncated .ecNN/.ecx/.ecj/.vif/.ecsum on disk for a later reader to trip on. Matches receive_file / the Go volume server. Best-effort cleanup; the original stream error is still returned. Claude-Session: https://claude.ai/code/session_01Ny5Rt1ph9VWeKmfY936GtF |
||
|
Chris Lu
|
c332323b01 |
rust volume: pin rustls to aws-lc-rs so TLS gRPC startup doesn't panic (#10233)
aws-lc-rs and ring both get linked transitively, so rustls can't auto-select a crypto provider and tonic's client TLS panics the moment the volume server dials a master over TLS. Install aws-lc-rs as the process default in main(), matching the provider the server config already uses. |
||
|
|
cc4043c9d2 |
fix(volume [rust]): compare live compaction_revision instead of stale last_compact_revision (#10189)
* fix(volume [rust]): compare live compaction_revision instead of stale last_compact_revision * fix(volume [rust]): compare live compaction_revision instead of stale last_compact_revision - unit tests * s3: invalidate stale reader cache locations on chunk read failure (#10156) * s3: invalidate stale reader cache locations on chunk read failure * filer: share the chunk-read self-heal across reader cache and streaming paths The reader cache retry added a third copy of the invalidate-relookup-compare-retry dance already inlined in PrepareStreamContentWithThrottler and duplicated in retryWithCacheInvalidation. Extract retryFetchWithFreshLocations and route all three through it, parameterized by the refetch primitive. * filer: drop redundant completedTimeNew store in reader cache success path startCaching already stamps completedTimeNew unconditionally before the fetchErr branch; the second store inside the success branch is dead. * filer: make NewReaderCache cache invalidator an explicit parameter The variadic ...CacheInvalidator only ever read the first element, so a caller could pass two and silently get one. Take a single explicit argument and have the non-S3 callers pass nil. * filer: inject reader cache chunk fetch as a struct field Replace the process-global readerCacheFetchChunkData test seam with a per-instance fetchChunkDataFn field defaulted in NewReaderCache, matching how lookupFileIdFn is already wired. Tests set the field on the cache instead of swapping a shared global. * filer: log the location count, not full URLs, on self-heal retry --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> * fix(shell): honor explicit fs.mergeVolumes from/to direction (#10159) * fix(shell): honor explicit fs.mergeVolumes from/to direction mergeVolumes only ever merged a smaller volume into a larger one. When the user named both -fromVolumeId and -toVolumeId with the source larger than the target, the planner produced an empty plan and the command printed just "max volume size: N MB" and moved nothing. Build the requested pair directly when both ids are given, instead of routing through the size-descending heuristic. Read-only, empty, and wrong-collection endpoints are rejected with a clear error rather than a silent no-op. * fix(shell): allow fs.mergeVolumes into an empty target volume Merging chunks into an empty volume is valid, e.g. consolidating data into a freshly created or recently vacuumed volume. Only reject an empty source, which has nothing to move. * fix(shell): reject self-map in directed mergeVolumes planner createMergePlan with from == to returned a {vid: vid} self-merge when called directly. Guard it in the planner so it is correct independent of the Do entrypoint. * fix(volume [rust]): compare compaction_revision in u32, not truncated u16 `req.compaction_revision as u16` truncates any request value above 65535, so a stale revision of 65537 aliases to a live revision of 1 and the "is compacted" guard wrongly passes. Widen the volume's revision to u32 and compare there, matching Go's uint32(v.CompactionRevision) != req.CompactionRevision. --------- Co-authored-by: adri <adri@digitalunited.net> Co-authored-by: Aleksey <48918167+MilanFun@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
cf64cafc3b |
volume: drop stale volume-location cache on under-replication (#10185)
* volume: drop stale volume-location cache on under-replication A replicated write looks up the volume's locations and caches them for 10 minutes. When the master briefly reports fewer replicas than the copy count (e.g. a stale heartbeat drops a just-added volume), that under-replicated result got cached, so every write failed with "replicating operations is less than replication copy count" until the entry expired -- long after the master re-registered the replica. Invalidate the cached entry when the location count is below the copy count, so the next write re-queries the master and recovers as soon as it heals. * volume: mirror the replication copy-count guard in seaweed-volume do_replicated_request accepted a write even when the master reported fewer locations than the volume's copy count, silently under-replicating. Reject it, matching Go's GetWritableRemoteReplications. lookup_volume is uncached, so the next write recovers as soon as the missing replica re-registers. |
||
|
Chris Lu
|
f6032cf23d |
fix(ec): read chunk-manifest chunks stored on EC volumes (rust volume server) (#10187)
* fix(ec): read chunk-manifest chunks stored on EC volumes Chunk-manifest expansion read every chunk through store.read_volume_needle, which only resolves a local regular volume. Once a chunk's volume is EC-encoded, that lookup returns NotFound and the GET fails 500 with "read chunk ...: not found", so a chunked object over an EC tier is unreadable even though its parity is intact and reconstructable. Resolve each chunk to wherever it lives — a local regular volume, a local EC volume (reconstruct-on-read from the surviving shards), or a peer via master lookup — matching Go's ChunkedFileReader, which never assumes chunks are local regular needles. * fix(ec): validate the chunk cookie on local manifest chunk reads A chunk fetched from a peer is cookie-checked by that peer's GET handler, but the local regular and EC reads returned data without comparing the needle's cookie to the one in the chunk fid. Check it, matching the main GET paths, so a stale or guessed id can't serve another needle's bytes. * fix(ec): clamp manifest chunk copy to its declared size Expansion writes each chunk into result[offset..] by offset, so a chunk whose bytes exceed its declared size could overwrite the next chunk's window. Clamp the copy to chunk.size (and reject a negative size) so an over-long or malformed chunk stays within its own range. |
||
|
Chris Lu
|
ce3ba31bcd |
fix(ec): reject oversized bitrot payload before narrowing to u32 (#10172)
save_bitrot_sidecar writes payload.len() into the header as a u32; guard against a payload > 1 GiB (which would silently truncate the length field), mirroring Go's SaveBitrotSidecar maxBitrotPayloadSize check. The check uses encoded_len() before serializing, so an oversized manifest never allocates a large buffer. Never triggers for a real sidecar (a few KB). Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo |
||
|
Chris Lu
|
b4a99b996d |
feat(ec): EC bitrot CHECKSUM scrub on the Rust volume server (#10154)
* proto: add EC bitrot checksum messages + CHECKSUM scrub mode Mirror weed/pb/volume_server.proto byte-for-byte (field numbers + types) so the .ecsum sidecar payload is wire-identical across the Go and Rust binaries: EcBitrotProtection / EcShardChecksums / ChecksumAlgorithm, VolumeScrubMode.CHECKSUM=4, and VolumeEcShardsCopyRequest.copy_ecsum_file. No code uses them yet — the .ecsum format, producer, mount-load, copy, and scrub land in following commits. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): port the .ecsum bitrot checksum module ec_bitrot.rs mirrors weed/storage/erasure_coding/ec_bitrot.go: the .ecsum sidecar format (14-byte big-endian ECSU header + CRC32C over a prost-serialized EcBitrotProtection payload), the per-shard per-block CRC32C producer (ShardChecksumBuilder), save/load with payload self-integrity, manifest validation, status resolution, and verify_shard_file_blocks for the CHECKSUM scrub. A byte-exact test pins the serialized bytes against the Go reference's identical constant so a format drift in either binary fails loudly. Producer wiring (encode/vacuum), mount-load, copy, and the mode-4 dispatch land in following commits. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * test(ec): pin .ecsum sidecar bytes for cross-binary interop Deterministic EcBitrotProtection -> exact on-disk bytes, asserted against a canonical constant on BOTH sides (this test and ec_bitrot.rs), so a format drift in either binary fails its own suite rather than silently desyncing a Go-written .ecsum from a Rust-written one. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): write the .ecsum bitrot sidecar during EC encode write_ec_files now feeds each shard's bytes through a per-shard ShardChecksumBuilder as it writes them, then persists the generation-0 sidecar (<base>.ecsum) alongside the shards — mirroring weed's WriteEcFiles + SaveBitrotSidecar. Best-effort: a failed sidecar write leaves the generation unprotected rather than failing the encode. A test confirms the produced sidecar validates and its per-block CRCs match every on-disk shard. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): load the .ecsum at mount + EcVolume::checksum_scrub EcVolume now loads and validates its generation-0 .ecsum sidecar at mount, caching the parsed protection + BitrotStatus (Off/On/Invalid), and exposes bitrot_protection() mirroring Go's EcVolume.BitrotProtection(). checksum_scrub() verifies every locally-held shard's raw bytes against the sidecar block CRCs — the only path that exercises cold parity shards — reporting mismatched shards without mutating anything; a wholesale mismatch beyond parity is flagged as a suspect sidecar rather than mass shard corruption. Mirrors Go's ChecksumScrub. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(scrub): dispatch EC CHECKSUM (mode 4) to checksum_scrub Accept VolumeScrubMode.CHECKSUM=4 and route it to EcVolume::checksum_scrub, accumulating blocks scanned + mismatched shards into the scrub response, plus the CHECKSUM scrub-mode metric label. Read-only bitrot verification over local shards. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): copy the .ecsum sidecar during VolumeEcShardsCopy Honor copy_ecsum_file: when set, copy the generation-0 .ecsum alongside the shards so protection travels with them, mirroring Go's non-2PC copy path. Tolerant of a missing source (empty stream) — the 0-byte file is dropped so mount sees no sidecar (protection off) rather than a truncated/invalid one. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): remove the .ecsum sidecars when destroying an EC volume remove_ec_volume_files now clears <base>.ecsum (and any versioned .ecsum.v<N>) from the data and idx dirs, so a vid reuse can't load a stale sidecar. Mirrors Go's removeBitrotSidecars. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * style(ec): align bitrot comments and test setup for merge-cleanliness Match the shared bitrot code (write_ec_files, encode_one_batch, checksum_scrub, the encode sidecar test) to the canonical wording/layout so the volume-server Rust port stays line-aligned across trees, keeping periodic merges conflict-free. No behavior change. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo |
||
|
Chris Lu
|
9550b830d0 |
worker: project the moved volume when gating on disk fullness (#10171)
The disk-fullness gate only rejected destinations already at/above the mark, so a server just under it could take a large volume and overshoot. Project the selected volume's bytes onto the candidate: if the move would cross the mark, drop that destination for the rest of the cycle and re-pick instead of overshooting. Also note the per-location capacity-summing assumption on the Rust heartbeat side, to match the Go store.go comment. |
||
|
Chris Lu
|
77bf2a3ab0 |
volume.balance: gate on real physical disk usage (fixes #10160) (#10162)
* shell: add volume.balance -byDiskUsage to balance by actual data The default balancer ranks servers by slot density, dividing used volumes by MaxVolumeCount. When MaxVolumeCount is configured higher than the disk can hold, a physically near-full server looks nearly empty and gets picked as the move target, so balancing drains less-full servers onto an already-full one. -byDiskUsage ranks servers by the actual data they hold (sum of volume sizes) instead, so the fullest-by-data server is treated as full and balancing drains it. It assumes comparable disk sizes per disk type and still respects each server's free volume slots. Default behavior is unchanged. * plumb physical disk usage into topology, gate volume.balance on it Volume servers now report each disk's filesystem total/free bytes in the heartbeat, and the master stores them in DiskInfo. volume.balance uses them to skip any move target whose disk is already near full (-maxDiskUsagePercent, default 90), so an over-configured maxVolumeCount can no longer make a physically full server look empty and get drained onto. The gate judges each server against its own disk, so heterogeneous disk sizes are fine; servers that do not report bytes fall back to slot-only behavior. Rust seaweed-volume mirrors the heartbeat reporting. * admin: report real physical disk capacity when volume servers provide it The dashboard estimated server capacity as maxVolumeCount * volumeSizeLimit, which overstates it when maxVolumeCount is set higher than the disk holds. Prefer the filesystem capacity now reported per disk, falling back to the estimate for servers that do not report it. * worker: gate automatic balance on physical disk fullness too The maintenance balance worker selects the least slot-utilized server as the move destination, so an over-configured maxVolumeCount makes a physically full server look empty and get drained onto — the same defect as the shell command. Now that DiskInfo carries real disk bytes, skip any destination whose disk is at/above 90% used (per server, against its own disk); a full server can still be a source. When every candidate destination is full, create no tasks. Servers that do not report disk bytes are not gated. * balance: share the physical-disk-fullness gate between shell and worker The shell volume.balance command and the maintenance balance worker each grew their own copy of the disk-fullness gate (targetDiskTooFull / destinationDiskTooFull) and a maxDiskUsagePercent=90 constant. Pull both into weed/topology/balancer (DiskTooFullAfter + DefaultMaxDiskUsagePercent) so the policy has one home and the two balancers can't drift. * balance: harden the physical-disk gate Guard against a nil DiskInfo in the byte/slot lookups. Let a zero disk-capacity report clear previously stored bytes (0 means "not reported" for bytes, unlike maxVolumeCount), so a server that stops reporting falls back to slot-only instead of trusting stale capacity. In the worker, charge each planned move's bytes to its destination within a detection cycle so the gate sees a target fill up rather than only its heartbeat-time free space. Note the per-location capacity summing assumes one location per filesystem (the used ratio the gate relies on stays correct regardless; absolute capacity can over-report). |
||
|
Chris Lu
|
41d6c821ba |
feat(topology): report empty disks (per-disk type + capacity in heartbeat) (#10166)
* fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk
DiskId 0 doubles as the first physical disk (Locations[0]) and the
protobuf "unset" default. SplitByPhysicalDisk folded every DiskId-0
record onto the aggregate DiskId whenever that was non-zero, so on a
multi-disk node the first disk's volumes merged into whichever disk
held volumes[0]: the node reported one fewer disk, the sibling showed
~2x volumes, and per-disk max was smeared across the survivors. This
surfaced as cluster.status and volume.list undercounting disks.
Only treat 0 as unset when no record carries a non-zero DiskId; with a
mix, 0 is a real disk and keeps its own entry.
* fix(admin): resolve physical disk 0 in active-topology indexes
rebuildIndexes re-derived each volume/EC record's physical disk id with
the same "DiskId 0 means unset" heuristic SplitByPhysicalDisk used, so
the two agreed only by sharing the bug. Now that SplitByPhysicalDisk
keeps disk 0 distinct, the duplicated heuristic would fold disk-0 records
onto a sibling while at.disks kept them on disk 0; GetVolumeLocations and
GetECShardLocations then matched no record and silently dropped every
volume and EC shard on the first disk, starving balance and EC tasks.
Build the indexes from the same SplitByPhysicalDisk reconstruction that
builds at.disks, so the keys always resolve. One source of truth instead
of a parallel normalize.
* fix(ec): allow physical disk 0 as preferred EC shard target
pickBestDiskOnNode gated its result on bestDiskId != 0, but 0 is both a
valid physical disk and the uint32 zero value, so a best-scoring disk 0
was discarded and the non-matching fallback returned instead. Gate on
bestScore.
* test(admin): cover EC-shard index resolution for physical disk 0
rebuildIndexes builds ecShardIndex the same way as volumeIndex; pin the EC
path too so a shard on disk 0 keeps resolving via GetECShardLocations.
* proto: per-disk type/capacity in DiskTag, DiskInfo.physical_disks
DiskTag gains type + max_volume_count so the heartbeat can describe every
physical disk, including ones holding no volumes or EC shards. DiskInfo
gains physical_disks so the master can hand the full per-type disk set to
per-physical-disk consumers.
* feat(volume): report each physical disk's type and capacity
CollectHeartbeat fills DiskTag.type and the per-disk effective max for
every location, so the master can account for disks that hold no volumes
or EC shards yet. Rust heartbeat mirrors it.
* feat(master): surface empty disks in the per-physical-disk view
The master records each disk's type and max from DiskTags and lists them
on DiskInfo.physical_disks per type, including disks with no volumes or
EC shards. SplitByPhysicalDisk enumerates that full set and gives each
disk its exact max, so cluster.status, volume.list and the admin
topology count and can target empty disks. Without physical_disks the
even-split fallback is unchanged.
* fix(master): clamp per-disk free at zero for over-allocated disks
In the exact-max path FreeVolumeCount could go negative when a disk holds
more volumes than its max; a negative would reduce the node's summed free
and block placement on healthy disks. Clamp at 0.
* fix(master): rebuild disk tags fresh each heartbeat
DiskTags is the full authoritative per-disk list every heartbeat, so
rebuild dn.diskTags from scratch like dn.diskBackends; merging left stale
entries for removed disks.
* fix(master): keep zero-capacity disks in physical_disks
A disk reporting max 0 (an unavailable disk) is a valid physical disk,
not a signal to drop it. List every disk of the type, but only emit
physical_disks when the node reports real per-disk capacity, so an older
server sending all zeros still falls back to the aggregate split.
* test(volume): cover disk-space-low per-disk max in heartbeat
Assert DiskTag.max_volume_count follows the used-slots override when a
location is low on space, matching the per-type max_volume_counts.
* chore: trim comments on the empty-disk change
Drop narration; keep only the non-obvious why (disk-0 sentinel, exact-max
free clamp, EC slots not subtracted, all-zeros fallback).
* refactor(master): merge per-disk tags and capacity into one map
diskTags and diskBackends were parallel maps keyed by the same DiskId and
filled together from DiskTags. Fold them into one diskMetas map of
{tags, type, max}.
* refactor(proto): per-disk max as a map keyed by disk id
physical_disks was a repeated {disk_id, max_volume_count} whose fields
duplicated DiskInfo's own disk_id/max_volume_count. A map<uint32,int64>
keyed by disk id expresses "max per disk" directly, drops the extra
PhysicalDiskInfo message, and the consumer reads it as the disk set.
* docs(proto): note DiskInfo.disk_id's two meanings
Identity on a per-physical-disk DiskInfo (from SplitByPhysicalDisk),
representative fallback on the type-keyed aggregate.
|
||
|
Chris Lu
|
8f2a2abae4 |
fix(ec): correct EC FULL scrub for deleted needles, shard-location cache, and parity coverage (#10152)
* fix(ec): correct EC FULL scrub for deleted needles + shard-location cache
Addresses review findings on the EC FULL distributed scrub:
- Remote EC reads now thread Go's (bytes, is_deleted) contract. A runtime EC
delete keeps the .ecx size positive (the delete lives in .ecj/memory), so the
raw-index walk verifies the needle, and its header interval is usually remote;
the peer answers is_deleted with no payload. The scrub zero-fills that interval
(so the needle reaches read_bytes -> SizeMismatch{0} -> the delete-state
suppression), the serving direct read short-circuits to not-found, and
reconstruction EXCLUDES the shard instead of feeding zeros into Reed-Solomon.
- The walk skips size.is_deleted() (not just is_tombstone), so a -originalSize
.ecx entry (pre-encode delete) can't yield empty intervals or panic parse_header.
- Restore Go's < data_shards completeness guard (per-volume, custom-ratio aware)
and per-shard merge in the location cache instead of clobber-with-partial.
- Abort the scrub with an error on mid-scan unmount instead of a false-CLEAN.
- Hoist the refreshed location map once instead of cloning it per needle.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(scrub): keep RS parity check in EC FULL until CHECKSUM lands
The per-needle FULL walk only reads live data-shard intervals, so it can't catch
bitrot in a parity shard or an unwalked cold region. Run verify_ec_shards
alongside the walk, gated on all-shards-local (single-node EC), via spawn_blocking.
A deliberate temporary divergence from Go FULL; moves to mode 4 (CHECKSUM) once
the .ecsum subsystem lands.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
|
||
|
Chris Lu
|
d18b85ef61 |
feat(scrub): EC FULL scrub — distributed local+remote needle walk (#10149)
* feat(ec): add scrub_ec_volume_distributed (FULL EC scrub, local+remote) Ports Go's Store.ScrubEcVolume: walk the raw .ecx, verify every needle across local AND remote shards without decoding (report faults, don't heal), with the #10130 deleted-needle size-mismatch suppression gated on a force flag. Reuses the read path's lock-drop + no-reconstruct read_remote_ec_shard_interval so no !Send store guard is held across an .await. Walks the unmasked index (scrub_snapshot_under_lock locates from the raw (offset, size), not locate_needle) so logically-deleted-but-present needles are still byte-verified, matching Go. Refreshes shard locations once up front and hard-fails on a master-lookup error rather than retrying per needle. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(scrub): dispatch EC FULL (mode 2) to the distributed needle walk FULL ran a local-only Reed-Solomon parity check; route it to the per-needle local+remote walk instead, mirroring Go. The handler collects vids under a brief lock then releases it: FULL self-locks per needle (it awaits remote reads), INDEX/LOCAL re-acquire a brief lock. verify_ec_shards is retained but no longer wired to a mode. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo |
||
|
Chris Lu
|
acbb6f7550 |
fix(scrub): don't flag offset-0 logical tombstones in volume scrub (#10148)
* fix(scrub): don't flag offset-0 logical tombstones in volume scrub A remote-tier delete records a tombstone at .idx offset 0 with no physical .dat bytes. Full scrub double-flagged a healthy remote-tiered volume with deletes: scrubVolumeData counted the tombstone's GetActualSize(-1)=32 toward totalRead (want > physical .dat), and CheckIndexFile treated it as occupying [0,31] and flagged the first live needle as overlapping. Skip offset-0 logical tombstones from both the size reconcile and the overlap check; they are still counted for the index-size check. Local deletes (offset != 0) are unaffected. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * fix(scrub): mirror offset-0 logical tombstone handling into Rust Same fix as the Go volume_checking.go + idx/check.go change: Volume::scrub skips offset-0 logical tombstones from total_read, and check_index_file excludes them from the overlap check (still counted for the index-size check). Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo |
||
|
Chris Lu
|
c9f2ef9ef7 |
fix(ec): suppress deleted-needle size mismatch in EC LOCAL scrub (#10147)
* fix(ec): suppress deleted-needle size mismatch in EC LOCAL scrub EcVolume.ScrubLocal reassembles each fully-local needle and ReadBytes-checks it, but appended every error unconditionally. A needle the .ecx still reports live while its reassembled on-disk header carries size 0 (delete state disagrees between index and header) is not corruption — the LOCAL twin of the #10130 fix for the FULL path. Suppress the ErrorSizeMismatch in that case; genuine (non-zero) size mismatches and CRC/tail errors are still reported. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * fix(ec): mirror EC LOCAL scrub deleted-needle suppression into Rust Same suppression as the Go EcVolume.ScrubLocal change: a NeedleError::SizeMismatch whose on-disk header size is 0 against a live index entry is a delete-state disagreement, not corruption. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo |
||
|
Chris Lu
|
473f7b2367 |
feat(scrub): EC LOCAL needle walk (split from FULL) (#10144)
* feat(ec): extract locate_ec_shard_needle_interval Mirrors Go's EcVolume.LocateEcShardNeedleInterval; reused by locate_needle and the upcoming local scrub walk. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): add EcVolumeShard::to_ec_shard_info Mirrors Go's ToEcShardInfo. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): add EcVolume::scrub_local Walk the .ecx and verify each needle against the locally-held shards, reading interval-by-interval (reusing one chunk buffer); CRC-check only fully-local needles, report short/unreadable local shards, and abort the scan on a structural size mismatch. Mirrors Go's EcVolume.ScrubLocal. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(scrub): dispatch EC LOCAL (mode 3) to scrub_local Splits the mode 2|3 arm: FULL (2) keeps the Reed-Solomon parity check; LOCAL (3) now runs the per-needle local-shard walk. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo |
||
|
Chris Lu
|
dccc015a1f |
fix(scrub): walk the on-disk .idx in Volume::scrub (source-of-truth parity) (#10143)
* refactor(scrub): extract open_index_for_scrub shared by scrub_index Mirrors Go's openIndex, shared by ScrubIndex and the upcoming Scrub rewrite. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * fix(scrub): walk the on-disk .idx in Volume::scrub scrub walked the deduped in-memory map, so total_read undercounted the physical .dat on any volume with overwrites or deletes and the size reconcile falsely flagged healthy volumes broken. Walk every .idx row instead (matching Go's scrubVolumeData): count all rows, CRC-verify live needles, skip deleted, and reconcile against the .dat. Holds one data-file read lock and reads via the unlocked path, like Go's Scrub. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo |
||
|
adriandadri
|
1df7a0e653 |
fix(volume [rust] + ec): search sibling disk locations when rebuilding missing EC shards + .ecx files (#10145)
* fix(volume [rust] + ec): search sibling disk locations when rebuilding missing EC shards * fix(volume [rust] + ec): apply sibling-disk shard lookup to .ecx rebuild as well * fix(volume [rust] + ec): include rebuild_dir in .ecx rebuild's shard search dirs --------- Co-authored-by: adri <adri@digitalunited.net> |
||
|
Chris Lu
|
72009c607b |
fix(scrub): align Rust INDEX scrub to Go's idx.CheckIndexFile (#10142)
* feat(idx): add check_index_file mirroring Go idx.CheckIndexFile Index-only structural check: walk the on-disk index, sort by (offset, size), flag overlapping needles, and verify the file is a whole number of entries. No data-file reads. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * refactor(ec): use idx::check_index_file in EcVolume::scrub_index Drops the inline walk/sort/overlap copy. Walks a private fd so the structural scan never moves the shared ecx_file cursor (read positionally elsewhere). Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * fix(scrub): make Volume::scrub_index an index-only check on the on-disk .idx INDEX mode walked the deduped in-memory map and read .dat headers — more than the cheap-INDEX contract allows, yet missing Go's overlap and size-multiple structural checks. Route it through idx::check_index_file so it matches Go's Volume.ScrubIndex and the INDEX<LOCAL<FULL cost tiering holds. Ports openIndex's zero-size-index guard (a populated .dat with an empty .idx is corruption) and takes the data-file read lock for a consistent snapshot. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo |
||
|
Chris Lu
|
0e293c9b0a |
docs(scrub): correct backwards FULL/LOCAL mode comments in Rust (#10141)
docs(scrub): correct backwards FULL(2)/LOCAL(3) mode comments The proto enum is FULL=2, LOCAL=3; two comments had them swapped. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo |
||
|
Chris Lu
|
270ac332ff |
fix(ec): honor wide EC ratios in Rust read_ec_shard_config (#10140)
* fix(ec): cap EcShardConfig at MAX_SHARD_COUNT, not TOTAL_SHARDS_COUNT read_ec_shard_config rejected any .vif ratio summing past 14 shards and silently fell back to 10/4, so wider EC volumes ran against the wrong shard set. Match Go's MaxShardCount(32) bound. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * docs(ec): correct stale 0..14 shard-count comments Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo |
||
|
jk2lx
|
96f93d8e3b |
fix(rust-volume): parse master lookup when publicUrl is omitted (#10128)
Master /dir/lookup JSON omits publicUrl when empty (Go json omitempty). The Rust volume server required the field, so serde failed with "lookup parse failed: error decoding response body" and cross-DC replicated writes failed. Default publicUrl to empty, fall back to url for peer filtering, and normalize addresses with to_http_address before excluding the local peer (so host:port.grpcPort forms do not match self incorrectly). |
||
|
Chris Lu
|
c2668fbc64 |
fix(volume): make tier-down crash-safe and serve from local (Rust) (#10113)
* fix(volume): fsync .vif and downloaded tier .dat (Rust) save_volume_info wrote the .vif with a plain write and no fsync, and the tier download never synced the .dat it wrote. Either could be lost on a crash before the tier-down path acts on them. fsync both, matching the Go volume server's util.WriteFile and DownloadFile. * fix(volume): swap to local before deleting remote on tier-down (Rust) The tier-down path deleted the shared remote object before trimming the .vif, so a crash in between left the volume's .vif pointing at a deleted object. It also dropped the remote backend only on the delete path and never opened the downloaded local .dat, so reads broke until reload and a keep-remote download kept serving from the slow remote object. Trim the .vif and swap to the local .dat on both paths, bracketed by directory fsyncs, before removing the remote object; gate only the object removal on keep_remote_dat_file. Matches the Go volume server's crash-safe ordering. |
||
|
Chris Lu
|
66620a1ab8 |
fix(volume): serve reads from remote after tier upload (Rust) (#10112)
After VolumeTierMoveDatToRemote uploaded the .dat, the volume closed its local backend but never opened the remote one, leaving both dat_file and remote_dat_file empty. The needle read path has no lazy reopen, so reads returned "dat file not open" until the volume reloaded. Switch to the remote backend right after saving the .vif, the same as the Go volume server's LoadRemoteFile, so the volume keeps serving from remote storage immediately after tiering. |
||
|
Chris Lu
|
2efc0e1656 |
ec: recover EC shards whose .ecx index lives only on a peer server (#10108)
* ec: recover EC shards whose .ecx index lives only on a peer server A volume server that boots with EC shard files on disk but no .ecx index on any local disk cannot mount the shards, so the master never learns about them. ec.rebuild works off master-registered shards, so it sees the volume as short and gives up even though the shard data is intact. Add an operator-triggered recovery: VolumeEcShardsMount gains a recover_missing_index flag that makes the volume server fetch the missing .ecx (plus .ecj/.vif) from a peer holding it and mount the on-disk shards. ec.rebuild runs this across the cluster before planning, so orphaned shards register and the rebuild sees the true shard set. .ecx is an immutable encode-time index, identical on every holder. .ecj is a per-holder deletion journal that differs across holders, so the recovered node adopts the source peer's deletion view, like a balanced or rebuilt shard does. * ec: mirror missing-index recovery into the Rust volume server Port the #10104 recovery to seaweed-volume so the Rust volume server self-heals the same layout: EC shards on disk with the .ecx index only on a peer. Adds collect_ec_volumes_missing_index / mount_recovered_ec_shards to the store, recover_missing_ec_indexes (master LookupEcVolume + peer CopyFile fetch + mount) to the server, and the recover_missing_index flag on VolumeEcShardsMount. .ecx is the immutable encode-time index, identical on every holder. .ecj is a per-holder deletion journal, so the recovered node adopts the source peer's deletion view, matching the Go path. |
||
|
|
130a5dffc3 |
fix (Volume [Rust]): stream copy_file and volume_incremental_copy instead of buffering the whole file in memory (#10110)
* fix(volume): stream copy_file from disk instead of buffering whole file copy_file pushed every 2MB chunk into a Vec and only then returned tokio_stream::iter(results), so serving a near-limit volume as a copy source (e.g. during volume.fix.replication) held the entire .dat resident and could OOM the process. Stream chunks through a bounded mpsc channel from a spawn_blocking reader instead; caps memory at ~16MB per transfer with backpressure. * fix(volume): stream volume_incremental_copy from disk instead of buffering Same buffering pattern as copy_file: every 2MB chunk was pushed into a Vec and only then returned via tokio_stream::iter, holding the entire delta resident. Stream the byte range from an owned file handle through a bounded mpsc channel, mirroring the copy_file fix. * test(volume): cover streaming copy_file and volume_incremental_copy Adds a multi-chunk .dat fixture and tests asserting both handlers stream in 2MB chunks (multiple messages), reassemble byte-for-byte, carry modified_ts_ns only on the first copy_file message, and honor stop_offset. * address review: use u64 byte counters; stream local incremental copy without holding the store lock - copy_file/volume_incremental_copy: track remaining bytes and offsets as u64 instead of casting uint64 stop_offset/dat_size through i64 (CodeRabbit). - volume_incremental_copy: for local volumes open the .dat and stream directly with no lock held; only remote/tiered volumes take the per-chunk read_dat_slice path, so a remote S3 read is never performed while holding the store read lock (Gemini). * volume (Rust): stream tiered incremental copy off the store lock, open .dat under it Capture the reader for volume_incremental_copy while the volume lookup is still under the store read lock: an open File for local volumes, a cloned remote backend handle for tiered ones. Then drop the lock and stream with none held. Opening under the lock pins the reader to the volume that exists now, so a concurrent delete/recreate can't stream from the wrong file, and a slow S3 fetch for a tiered .dat no longer blocks store writers (the remote path previously re-took the store lock per chunk). Use a non-uniform copy-test payload so chunk reassembly catches duplicated or reordered chunks a repeated byte would hide. * volume (Rust): return empty when incremental-copy start offset is past the .dat A corrupt needle index could locate an offset beyond the captured .dat size, underflowing the dat_size - start_offset subtraction (panic in debug, wrap in release). Guard it up front like the other empty-delta early returns. --------- Co-authored-by: adri <adri@digitalunited.net> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
bc257fe72e |
volume: detect phantom volumes held open as deleted FDs (#10011)
* volume: detect phantom volumes held open as deleted FDs
Add disk-file validation in heartbeat collection to prevent reporting
phantom volumes that exist in memory but are deleted from disk. This
unblocks re-replication when files are unlinked while the volume server
holds them open via file descriptors.
Cache disk checks per-volume with 30-second TTL to avoid syscall overhead.
Implement in both Go and Rust volume servers.
* volume: make last_disk_check_ns field public for heartbeat access
* volume: only check for phantom volumes when size > 0
Skip phantom volume detection for zero-size volumes (e.g., test volumes).
Phantom volumes only occur when disk files are deleted while the process
holds them open via FDs - which requires the volume to have had actual data.
Test volumes with zero size should not trigger disk file existence checks.
* volume: only check for phantom volumes when size > 0
Skip phantom volume detection for zero-size volumes (e.g., test volumes).
Phantom volumes only occur when disk files are deleted while the process
holds them open via FDs - which requires the volume to have had actual data.
Test volumes with zero size should not trigger disk file existence checks.
* volume: only check for phantom volumes if file_count > 0
Use file_count as the indicator for whether a volume held actual data,
rather than volume size. Phantom volumes only occur when a volume that
had files is deleted while the process holds open file descriptors.
Test volumes with no file count won't trigger the phantom detection check.
* volume: stat the .dat with its extension when detecting phantom volumes
DataFileName()/IndexFileName() return the extensionless base path, so os.Stat
saw every volume's files as missing and dropped it from the heartbeat, leaving
the master with no locations and breaking deletes/lookups. Stat FileName(".dat")
instead, skip remote-tiered volumes whose .dat lives in cloud storage, and
re-check a missing file every heartbeat rather than caching the negative.
|
||
|
Chris Lu
|
b5ecfcd28c |
volume: validate remote S3 endpoints in FetchAndWriteNeedle (Rust) (#10001)
* volume: validate remote S3 endpoints in FetchAndWriteNeedle (Rust) Port the Go volume server's SSRF guard to the Rust volume server. The gRPC FetchAndWriteNeedle reads from a caller-supplied S3 endpoint, so an unguarded server can be pointed at loopback / link-local / RFC1918 / CGNAT / cloud-metadata hosts to read internal services. Resolve and reject those endpoints unless -volume.allowUntrustedRemoteEndpoints is set (default off), mirroring weed/server/volume_grpc_remote.go. Connect-time re-validation against DNS rebinding (Go's guardedDialer) is not yet ported: the aws-sdk-s3 client builds its own connector, so the up-front resolve-and-check leaves a narrow TOCTOU window. Left as a follow-up. * volume: harden remote endpoint guard (IPv4-mapped IPv6, all S3 types) Address SSRF review feedback: - Normalize IPv4-mapped IPv6 (::ffff:a.b.c.d) to IPv4 before the deny checks, so ::ffff:127.0.0.1 / ::ffff:169.254.169.254 no longer slip past the IPv4 rules. - Validate the endpoint for every S3-compatible backend, not just type "s3"; wasabi/backblaze/aliyun/... all dial a caller-supplied endpoint through the same client. Skip validation when the endpoint is empty (the provider default, e.g. real AWS S3, which cannot reach internal hosts). * volume: set allow_untrusted_remote_endpoints in integration-test state The tests/http_integration.rs VolumeServerState literal was missed, which broke cargo test compilation (it builds the integration tests, unlike the cargo test --lib used locally). |
||
|
Chris Lu
|
284796c7b6 |
fix(ec): fence stale-worker EC shard cleanup by encode generation (#9953)
* feat(ec): add encode_ts_ns to the EC task params, shard-unmount, and shard-delete RPCs The generation fence for stale EC-worker cleanup needs the encode generation on three messages: ErasureCodingTaskParams (admin issues it), VolumeEcShardsUnmountRequest, and VolumeEcShardsDeleteRequest (the worker carries it to the volume server). Additive fields only; 0 preserves the existing unfenced behavior. Mirror the two volume-server fields in the Rust volume server's proto copy. * feat(ec): issue the EC encode generation from the admin and carry it on the worker Stamp each EC proposal's encode_ts_ns from the admin's per-cycle DetectionSequence (a single-clock value) so generations are globally ordered even though detection runs on a rotating worker. The worker writes that generation into the distributed .vif and passes it on its shard unmount/delete RPCs; it falls back to a local timestamp for the .vif only on the unfenced legacy/shell path (keeping the read guard on). * fix(ec): fence the stale-worker EC shard unmount and teardown by generation A reaped-but-still-running EC worker's cleanupStaleEcShards issued a generation-blind unmount + full teardown that could unmount and then overwrite a newer run's live shards on a shared node. Both RPCs now carry the encode generation: the volume server unmounts/deletes a disk only when its .vif generation is strictly older than the request, and preserves a same-or-newer generation, a generation-0 (recovered or pre-upgrade) volume, and an unreadable .vif. Unload is per-disk, never node-wide. Request generation 0 keeps the blanket teardown for the shell pre-encode cleanup and pre-upgrade callers. Mirrored in the Rust volume server. * test(ec): cover the generation-fenced teardown and unmount End-to-end volume-server tests: a fenced FullTeardown wipes a strictly- older generation, preserves a newer one, preserves a generation-0 volume, and blanket-wipes on request generation 0; the gen-aware unmount preserves a same-or-newer mounted generation; and the .vif generation reader handles present/absent/no-config cases. * test(ec): pin the fenced .vif==teardown generation and the unreadable-.vif preserve A fenced run must stamp the admin generation verbatim into the .vif so it matches the generation sent on the teardown RPCs; add a regression test that sets the task generation and asserts the .vif carries it exactly. Also cover the present-but-unparseable .vif case (reads as generation 0, preserved) and correct the readEcGenerationTsNs docstring accordingly. * fix(ec): surface EC full-teardown filesystem errors in the Rust volume server remove_ec_volume_files(_full_teardown) discarded every fs::remove_file error, so a teardown that failed on permissions or a full disk still returned full_teardown_done=true and left stale artifacts to collide with the next encode. Return io::Result, ignore NotFound, propagate the first real error, and have the teardown RPC surface it -- matching the Go contract. The best-effort reconcile/load-cleanup callers keep ignoring it. * refactor(ec): reuse the EC volume lookup on unmount and short-circuit the gen read Address review: the Rust unmount fence reuses the ec_vol it already fetched instead of a second find_ec_volume; the Go .vif generation reader breaks out of the data/idx loop early when the two dirs are the same. |
||
|
Chris Lu
|
da243b9423 |
fix(ec): group orphan-source completeness by encode generation (topology encode_ts_ns) (#9952)
* feat(ec): carry the encode generation through the topology heartbeat Add encode_ts_ns (field 14) to VolumeEcShardInformationMessage and populate it from each EC volume's .vif identity. The volume server emits it on the full and incremental heartbeats; the master stores it on EcVolumeInfo and re-emits it via GetTopologyInfo, so the admin/worker layer can see which encode run produced each shard set. Field 14 avoids the enterprise fork's reserved 10-13. Mirror the proto field and both heartbeat emit sites in the Rust volume server. * fix(ec): group orphan-source shard completeness by encode generation countExistingEcShardsForVolume ORed EcIndexBits across every disk, so two interrupted encode runs whose shard sets overlap unioned into a false-complete set -- triggering the orphaned-source delete while no single generation was actually complete. Group shards by encode_ts_ns and return the largest single generation's count, so the trigger fires only when one run holds the full set. Shards from pre-upgrade servers (encode_ts_ns==0) form their own bucket. The heartbeat carries one encode_ts_ns per (volume, disk), so this separates generations on different disks; same-disk mixing is prevented upstream by the pre-encode artifact wipe and the cross-run read guard. * fix(ec): guard against a nil Ec shard info entry in the generation count Defensive: a manually-constructed or corrupted topology could carry a nil entry in EcShardInfos. Skip it rather than dereference. * fix(ec): carry the encode generation on the EC shard unmount delta The mount delta sets EncodeTsNs; the unmount deletion delta left it 0. Populate it from the Ec volume before unloading so both incremental deltas are consistent (the Rust volume server already does this via its snapshot diff). |
||
|
Chris Lu
|
1e858d8af0 |
fix(ec): make ec.decode write-path crash-safe and atomic (#9949)
* fix(ec): check decode .idx writes and fsync decoded .dat/.idx WriteIdxFileFromEcIndex silently dropped io.Copy and Write errors, so a short or failed write of the reconstructed .idx went unnoticed and the caller proceeded to delete the source EC shards. Propagate those errors. Also fsync the decoded .dat and .idx before returning, so the bytes are durable before the shards that produced them are removed cluster-wide. Mirror the .idx fsync into the Rust volume server (its .dat already syncs and its writes already propagate errors). * fix(ec): publish decoded .dat/.idx atomically via temp file and rename WriteDatFile and WriteIdxFileFromEcIndex wrote in place at the final name with O_TRUNC. A crash mid-write left a truncated .dat/.idx at the final name beside the still-present EC shards; on restart that partial file could be mounted as the live volume even though the shards held the real data. Write to a .tmp file, fsync it, then rename into place and fsync the directory, so the final name is only ever absent or complete. A failed decode removes its own temp file rather than leaking it. Add util.FsyncDir as the shared directory-fsync primitive and reuse the Rust volume server's fsync_dir for the mirrored change. * fix(ec): propagate .ecj read errors in the Rust decoder Path::exists returned false for any error (permission denied, transient IO), silently skipping the deletion journal and resurrecting deleted needles as live. Read the journal directly and treat only NotFound as absent, propagating other errors. The Go decoder already behaves this way (FileExists returns false only for IsNotExist, then the open surfaces other errors). * fix(ec): remove rename destination on Windows in the Rust decoder publish std::fs::rename does not replace an existing file on every Windows version. Remove the destination first under a Windows guard before the atomic publish rename, matching the compaction commit path. |
||
|
Chris Lu
|
339a597e7e |
fix(vacuum): crash-safe compaction commit with a durable .cpc marker, fsync-before-rename, and a reload fence (#9944)
* storage: make vacuum/compaction commit crash-safe with a durable .cpc marker A crash mid-compaction-commit could lose or corrupt volume data. The two-rename commit (.cpd->.dat, .cpx->.idx) was not atomic, fsync results were discarded before renaming over a healthy .dat, a stale .ldb could poison the needle map, and a duplicate/late commit could delete the live .dat/.idx outright. Introduce a durable .cpc commit marker so the swap is atomic across a crash: - CommitCompact writes and fsyncs the .cpc marker after makeupDiff fsyncs the .cpd/.cpx, then runs applyCompactSwap: an existence-guarded rename of .cpd->.dat and .cpx->.idx, a directory fsync, removal of the stale .ldb/.rdb, and finally removal of the marker. - reconcileCompactState recovers an interrupted commit on load: roll forward (finish the renames) when the marker is present, roll back (delete the orphan .cpd/.cpx) when it is absent. It runs from a directory pre-pass keyed on .cpd/.cpc existence, since the per-volume loader is keyed on .idx/.vif and misses the marker-only and already-renamed-.idx states. - applyCompactSwap verifies BOTH .cpd and .cpx exist before touching the live files, so a stale-state commit (including the Windows RemoveAll-then-rename path) errors without deleting anything. - Error-check the fsyncs that gate the swap: the .cpd close-fsync and .cpx fsync in copyDataBasedOnIndexFile, the makeupDiff .idx fsync, and MemDb.SaveToIdx. - generateLevelDbFile rebuilds from offset 0 when the stored watermark sits past the end of the .idx, instead of replaying zero entries and poisoning the needle map. - removeVolumeFiles and cleanupCompact sweep the .cpc marker; cleanup refuses to unlink the temp files while a marker is present. Mirror the commit-marker, fsync-before-rename, guard, and load/reconcile logic in the Rust volume server. * storage: don't reconcile an already-loaded volume's compaction state on reload reconcileCompactStates runs in loadExistingVolumes, which is re-invoked at runtime on SIGHUP (Store.LoadNewVolumes). For a volume that is already loaded and mid-vacuum, its .cpd/.cpx are live temp files, not crash leftovers -- rolling them back would clobber the in-flight compaction (and remove a live .ldb out from under an open handle). Skip any vid already present in the volume map; genuine startup recovery runs before any volume is loaded, so the map is empty then. Mirrored in the Rust volume server. Also drop the .note keepVif change that crept into this branch; it belongs to the replica-copy/verify workstream and is restored to master's behavior here so the two changes don't collide. * storage: roll a compaction commit forward per-file, not all-or-nothing A crash after the .cpd->.dat rename but before .cpx->.idx leaves .cpd gone, .cpx and .cpc present, and a stale .idx. The roll-forward required BOTH temp files, so it skipped the swap and cleared the marker, pairing the fresh .dat with the stale .idx (index corruption). Finish whichever temp file remains: extract finishCompactSwap to rename .cpd->.dat and/or .cpx->.idx independently; applyCompactSwap keeps the both-present guard for the normal commit. Existence in the Rust mirror is checked robustly so a transient error never skips the swap. * seaweed-volume: propagate directory fsync failures on the compaction commit path fsync_dir dropped every sync_all error, so the commit could proceed with an undurable marker or rename and a later restart could recover the wrong generation. Return the error and check it at the commit call sites (marker write and the swap), matching the Go fsyncDir which already propagates. Directory fsync stays a no-op on Windows, where it is unsupported. * storage: overflow-safe stale-watermark check when rebuilding the leveldb index watermark*NeedleMapEntrySize can overflow uint64 for a corrupted watermark and wrap below the file size, defeating the stale-.ldb guard. Compare in entries (watermark > size/NeedleMapEntrySize) instead, which is equivalent and cannot overflow. LevelDb-backed needle map is Go-only; no Rust mirror. * storage: propagate idxFile.Close error when writing the compacted index SaveToIdx writes the .cpx that is renamed to .idx at commit; a discarded Close error (buffered data not flushed) could leave a partially-written index after a crash. Surface it in the same durability gate as the fsync. |
||
|
Chris Lu
|
c2591b4395 |
fix(replication): verify-before-destroy in VolumeCopy, check.disk, and over-replication trim (#9943)
* volume: verify before destroy in VolumeCopy and replication repair Four data-safety fixes around copy/repair paths that could destroy or resurrect data before verifying the source or survivors. (a) VolumeCopy no longer deletes a pre-existing local replica up front. The delete is deferred until ReadVolumeFileStatus on the source succeeds, so a transient source outage (or a retry after one) can no longer wipe a healthy destination replica. Gated on source readability only; size/count comparisons are intentionally not used because they invert legitimately after divergent vacuum/compaction. Mirrored in the Rust volume server. (b) volume.check.disk no longer resurrects vacuumed-deleted needles. A key present-and-live on the source but entirely absent on the target is ambiguous: it may be a genuine missing write, or a needle deleted on the target and then vacuumed (its index entry and any tombstone are gone). An individual needle AppendAtNs has no monotonic relation to a vacuum watermark, so the old cutoff heuristic could not tell them apart. Without positive proof the absence is a missing write, the safe default is to NOT push it back. Tradeoff: a real missing write may go unrepaired until a tombstone-aware path exists, but we never raise back deleted data. (c) Over-replication trim no longer resurrects needles or removes the wrong replica. The pre-delete sync now runs read-only (divergence check only) instead of writing the doomed replica's needles into the survivor. pickOneReplicaToDelete only ever removes the smallest of multiple healthy writable replicas; it refuses the trim when doing so would leave only read-only/integrity-flagged survivors, since file_count>0 alone cannot prove the survivor's .dat is readable. (d) Incomplete-volume (.note) cleanup keeps the shared .vif when an .ecx for the same vid coexists on the disk, so removing an interrupted regular copy cannot strip a coexisting EC volume's info file. VolumeCopy now surfaces .note write/remove errors instead of ignoring them. In the Rust volume server (where a persisting note is actually reachable) the .note check moves below the empty-stub sweep and EC validation, keeps the .vif on EC coexistence, and the mount path fails when a .note still persists. * shell: scope the over-replication writable-survivor guard to the trim path only The writable-survivor guard (never trim down to a read-only survivor) lived inside the shared pickOneReplicaToDelete, so it also gated the misplaced-volume relocation via pickOneMisplacedVolume -- a misplaced read-only volume (e.g. a full one) would silently stop being rebalanced. Extract pickSmallestReplica for the relocation path (which deletes-and-recreates and must act on read-only replicas), and keep the writable-survivor guard only in pickOneReplicaToDelete used by the over-replication trim. * seaweed-volume: recompute keep_vif after invalid-EC cleanup in the .note path keep_vif used the pre-validation ecx_exists snapshot, so when the EC-validation step above removed the invalid .ecx/shards, the .note cleanup still preserved a now-orphaned .vif. Re-check .ecx existence at cleanup time, matching the Go hasEcxFile re-check. * shell: keep placement when picking an over-replication victim to delete The trim picked the smallest writable replica without regard to placement, so it could delete the only replica in a required failure domain (e.g. with "100" and replicas dc1 + two in dc2, deleting dc1 leaves both survivors in dc2). Prefer a writable replica whose removal still satisfies placement, falling back to the smallest writable only when none does. |
||
|
Chris Lu
|
aabd44fbb5 |
[volume] preserve volume data mtime across tier moves (#9947)
* fix(tier): preserve volume data modification time * fix(tier): best-effort restore of data mtime on download A failed Chtimes should not abort an otherwise complete tier-down; warn and continue, matching the EC copy path. * fix(tier): preserve volume data mtime in rust volume server Mirror the Go fix: store the source .dat mtime on upload instead of the upload time, and restore it on the downloaded .dat. Without this a tiered-then-restored volume loads last_modified_ts_seconds from the upload/download time, extending its TTL across a restart or remount. * fix(tier): read source mtime via DiskFile.GetStat() GetStat() is nil-safe when the backend is closed concurrently and skips a redundant stat syscall; its cached modTime is the on-disk mtime a reload reads, since every .dat write or Chtimes is followed by a DiskFile (re)open. * fix(tier): surface mtime-restore failures on rust tier-down set_file_mtime now returns io::Result; the tier-down path warns on a failed restore instead of dropping it silently, so a wrong local .dat mtime (and the TTL drift it causes) is observable. Matches the Go download. The EC copy path keeps its best-effort silence. |
||
|
Chris Lu
|
f724828bcb |
fix(ec): never delete recoverable EC shards on startup/reconcile (the non-empty-.dat sibling of the stub bug) (#9941)
* fix(ec): never delete recoverable shards on startup/reconcile (size-direction + byte-exact .dat)
EC startup validation and the cross-disk reconcile could delete the only
copy of distributed-EC shards whenever a non-empty .dat sat beside them.
This is the same data-loss class as the empty-.dat-stub fix, now for a
real (non-empty) stale or partial .dat.
validateEcVolume: the discriminating signal is the shard size relative to
the .dat's full encode, not the shard count.
- shards smaller than expected: an interrupted local encode left partial
shards and the .dat is the complete source -> reclaim the .dat.
- shards equal to expected: a valid (or still-distributing) EC volume ->
keep; the shards may be the only copy.
- shards larger than expected: the .dat is the stale/partial side (e.g. an
interrupted decode left a half-written .dat next to the real shards) ->
keep.
Previously any size mismatch, a low shard count beside a .dat, or a
transient stat error returned "delete", wiping sole-copy shards. Now every
ambiguity (size mismatch in either direction, inconsistent shard sizes,
transient I/O error, partial shard set) keeps the data; only a credible
full source .dat with no partial set to lose is reclaimed.
handleFoundEcxFile: a shard load failure (corrupt/locked .ecx, EMFILE
during a mass restart, transient I/O) no longer deletes the EC files when a
.dat exists -- it only unloads and keeps the files for retry. All deletion
authority now flows through validateEcVolume.
pruneIncompleteEcWithSiblingDat: count shards NODE-WIDE (a set split across
sibling disks summing to >= dataShards is independently recoverable and is
left alone), and require the sibling .dat to byte-exactly match the size
.vif recorded at encode time before deleting -- the prior "at least this
big, or bigger than a superblock" gate could trust a stale .dat and wipe
sole-copy shards. EC encode records the source size in .vif, so this gate
works for real volumes; older volumes without it fail safe (kept).
Rust volume server mirrors all of the above: size-direction + keep-on-
ambiguity in validate_ec_volume, keep-on-load-failure in
handle_found_ecx_file, and the node-wide + byte-exact gate in the prune.
The Rust validate/prune paths now resolve the data-shard count from the
volume's own .vif instead of hardcoding 10+4, so custom-ratio volumes are
not mis-sized and wrongly deleted on reboot.
Existing tests that encoded the old (unsafe) "delete on low count / size
mismatch" behavior are updated to the safe expectation, and new regression
tests cover the partial-decode-.dat-keeps-shards and transient-error-keeps
cases (Go and Rust); they fail on the pre-fix code.
* fix(ec): record DatFileSize in planted EC .vif for the prune test; trim comments
The multi-disk lifecycle e2e test planted a partial EC leftover with an
empty .vif, so the byte-exact prune gate (which a real encoded volume
satisfies via its recorded source size) kept it instead of cleaning up.
Record DatFileSize + the EC ratio in the planted .vif, matching production.
Also condense the verbose comments added in this change to the repo's
concise style.
|
||
|
Chris Lu
|
18cdb3819b |
fix(ec): crash-safe ecx-journal fold and shard rebuild (fsync before publish, no short-read-as-success) (#9938)
* fix(ec): make ecx-journal fold and shard rebuild crash-safe Two EC rebuild paths could silently lose or corrupt data: RebuildEcxFile folded the .ecj deletion journal into .ecx (in-place WriteAt tombstones) and then unlinked the journal without flushing the .ecx writes first. A crash could persist the unlink ahead of the tombstones, resurrecting deleted needles on the next load. It also read journal records with a bare n!=size break, so a torn tail silently dropped the remaining tombstones before the unlink. Now: read records with io.ReadFull (io.EOF ends cleanly, a torn tail aborts and leaves .ecj in place for retry), fsync .ecx before removing the journal. rebuildEcFiles treated a zero/short ReadAt as a clean end-of-input and discarded the read error, so a truncated or unreadable input shard produced truncated regenerated shards that were then published as restored redundancy; the regenerated shards were also never fsynced on the no-sidecar path. Now: derive the expected shard size from the present inputs up front (rejecting a divergent/zero-size input), drive the loop by that size, fail on any short read or short write, and fsync every regenerated shard before it is mounted/renamed. Rust volume server mirrors the rebuild fix: rebuild_ec_files now checks the read_at byte count (it previously discarded it, the same truncation bug). The Rust ecx fold already synced .ecx before removing the journal. Custom EC ratios are unaffected: the shard size derives from the input shards and the loop uses the .vif-resolved data/parity counts, never a hardcoded 10+4. * storage: close ecx journal files via defer in RebuildEcxFile Per review: a single deferred Close per file replaces the per-error-path manual closes, so new early returns cannot leak descriptors. The journal is still closed explicitly before its unlink since Windows cannot delete an open file; the deferred second Close is a harmless no-op. |
||
|
Chris Lu
|
34f9b91d69 |
fix(storage): never let an empty .dat delete healthy distributed EC shards (#9930)
* fix(storage): never let an empty .dat delete healthy distributed EC shards A leftover empty .dat stub (a phantom from the pre-fix loader; zero needles) next to a distributed EC volume's local shards made startup classify the volume as an interrupted local encode: validateEcVolume requires >= dataShards local shards when a .dat is present, fails with the 1-2 shards a distributed volume keeps per disk, and the cleanup deletes those shards -- the only copies of that part of the volume. Repeated across restart waves this destroys enough shards cluster-wide to make the volume unrecoverable. Go: - loadExistingVolume: hoist the empty-stub sweep above the EC presence checks. Previously the .vif-next-to-.ecx guard returned before the sweep ever ran, so exactly the dangerous layout (stub + .ecx + local shards) kept its stub and then lost its shards in loadAllEcShards. - validateEcVolume / checkDatFileExists: treat a .dat <= a superblock (zero needles) as absent. An empty .dat cannot be the encode source, so it must never gate shard deletion; this also covers stubs without a .vif, which the sweep cannot prove are EC leftovers. Rust mirror (seaweed-volume): the same gate in validate_ec_volume and check_dat_file_exists (the Rust sweep already ran before validation); the volume-load skip keeps a plain existence check so fresh, needle-less volumes still load. Regression tests in Go and Rust reproduce the production layout (a zero-byte .dat beside .ecx/.ecj and two shards of a 10+4 volume, with and without a .vif) and fail without the fix with the shards deleted. * fix(ec): gate source volume deletion on a recoverable shard set After EC encode, the shell command and the (plugin) worker task refused to delete the source volume unless every shard was present, and aborted otherwise -- leaving the source .dat next to live shards, exactly the mixed state the startup cleanup mishandles. Replace the full-set requirement with a recoverability gate shared by both callers (RequireRecoverableShardSet): deleting a non-empty source .dat requires at least dataShards distinct shards cluster-wide. Below that the source is kept and the encode fails as before. A degraded but recoverable set (>= dataShards, < total) now proceeds with a warning instead of aborting: the missing shards can be rebuilt from the survivors, while keeping the source would preserve the dangerous mixed state. Empty stub replicas are still swept unguarded (OnlyEmpty) -- an empty .dat has nothing to lose. dataShards/totalShards stay parameters so enterprise custom EC ratios share the helper verbatim. * test(ec): use recoverable shard verification gate |
||
|
Chris Lu
|
4f8af455bf |
feat(storage): sweep leftover empty EC .dat stubs on volume server startup (#9927)
* feat(storage): sweep leftover empty EC .dat stubs on volume server startup An EC volume keeps no local .dat. The pre-fix loader left empty 8-byte superblock .dat stubs next to EC metadata (one per lone .vif). Left in place each loads as a phantom empty volume, and the same vid's stub on two disks of one server blocks Rust startup via the duplicate-vid check in Store::add_location -- the prior fix stops creating new stubs but does not clean up existing ones. On startup, when a .dat is empty (<= a superblock, i.e. zero needles) and its .vif marks the volume erasure-coded, remove the stub (+ empty .idx) instead of loading it. The real data is in the EC shards, so the empty stub holds nothing to lose. Non-EC empty .dat files (e.g. freshly allocated volumes) are left alone. Done in both Rust (load_existing_volumes) and Go (loadExistingVolume), with regression tests that fail without the sweep. * refactor(storage): extract empty EC .dat stub sweep into its own function Move the startup stub-sweep into remove_empty_ec_dat_stub (Rust) and removeEmptyEcDatStub + vifIsEcVolume (Go) for clearer logic, and look up the .vif in both the data and idx directories (each read at most once) so a stub is still found when -dir.idx is configured. Adds direct tests for the idx-directory lookup on both engines. |
||
|
Chris Lu
|
79ac279fe1 |
fix(ec): don't mix EC shards from different encode runs (#9880)
* feat(ec): add encode_ts_ns to EC shard metadata and the shard read RPC EcShardConfig and VolumeEcShardReadRequest gain an int64 encode_ts_ns (encode time in unix nanos). It rides in .vif and the read request so a read can be scoped to the encode run that produced the index. * fix(ec): stamp each encode and reject cross-run shard reads Generate stamps EncodeTsNs into the volume's .vif. Reads carry it to the shard's owning volume (resolved together via FindEcVolumeWithShard, so a multi-disk server validates the disk that actually serves the bytes) and reject a shard from a different encode run, recovering from parity. A zero on either side (pre-upgrade volume) skips the guard. * fix(ec): stamp the encode identity on the worker-generated .vif The worker-local encode path now writes EncodeTsNs (and the resolved EC ratio) into the .vif, so the read guard is not silently off for volumes encoded by the maintenance worker. * fix(ec): wipe stale EC artifacts before re-encoding VolumeEcShardsGenerate evicts any in-memory EcVolume for the volume and removes its on-disk shard/index/sidecar files before writing fresh ones, so a retried encode never builds on a partial prior run and the unlink frees the inodes instead of leaving open fds serving old bytes. * fix(ec): unmount EC shards across all disks UnmountEcShards walked only the first disk holding the shard, leaving a duplicate copy mounted on a sibling disk (split-disk reconciled volumes) still serving and heartbeating. Traverse every disk and emit one deletion delta per disk. * fix(ec): delete orphan shards without a local .ecx deleteEcShardIdsForEachLocation gated shard-file removal on a local .ecx, so it could not clean an orphan .ecNN left by a failed copy on a disk with no index. Delete the requested shard files unconditionally; the index-file (.ecx/.ecj/.vif) routing stays gated as before. * fix(ec): clear stale EC shards cluster-wide before re-encoding ec.encode unmounts and deletes EC shards for the target volumes on every node before regenerating: fatal for the shards the topology reports (mounted leftovers), best-effort for the rest (a sweep that catches unmounted failed-copy orphans). A down node is a no-op. * fix(ec): don't nil EC fds on close so reads can't race eviction A reader resolves an EcVolume/shard under the lock then reads after it is released, so an eviction that nils ecxFile/ecdFile would race that read and panic. Close the fds without nilling the fields: the field is now write-once (no data race) and a concurrent read hits a closed fd, getting a clean error that the caller recovers from parity. * fix(ec): wipe stale EC artifacts on every disk and surface failures The pre-encode wipe only deleted beside the source volume, so a stale shard on a sibling disk survived and could be mounted against the new index at reconcile. Sweep every disk. Removal also ignored os.Remove errors, reporting a failed cleanup as success and letting a stale shard join the next generation; surface the first real failure (treating already-gone as success) from removeStaleEcArtifacts and the shard delete. * fix(ec): log when a local shard is skipped for a different encode run The cross-run guard returned errShardNotLocal, indistinguishable in logs from a genuinely-absent shard. Add a V(1) line naming both EncodeTsNs so operators can tell "wrong encode generation" from "shard not here". * fix(ec): surface metadata removal failures in the shard delete path deleteEcShardIdsForEachLocation still dropped os.Remove errors on the .ecx/.ecj/.vif/sidecar cleanup. A surviving stale .ecx is the orphan-index condition this path prevents, so route those through removeFileIfExists and return the first real failure instead of reporting cleanup as success. * fix(ec): fail orphan cleanup when a reachable node's delete fails The pre-encode orphan sweep swallowed every error for unreported (node, volume) pairs. That is only safe for an unreachable node, which cannot receive this encode's new generation. A reachable node whose delete genuinely failed (permission/IO) keeps an orphan shard that a later copy re-stamps with the new run's volume-level .vif identity, so the read guard would accept stale data. Surface those; stay best-effort only for unreachable nodes (gRPC Unavailable / no status). * fix(ec): guard ecjFile under its lock in the EC delete path EcVolume.Close nils ecjFile under ecjFileAccessLock; a delete that resolved its .ecx lookup before a concurrent eviction (the generate-time UnloadEcVolume) could then reach the journal append with a nil fd. Bail with a clear "volume closed" error under the lock instead. * fix(ec): reject an unstamped shard when the caller has an encode identity The read guard required both identities nonzero, so a current (stamped) caller accepted a holder with identity 0 and could be served a stale pre-upgrade shard. Reject when the caller is stamped and the holder differs (including unstamped); stay lenient only when the caller itself has no identity (pre-upgrade reader). A skipped shard recovers from parity. * fix(ec): full-teardown delete so cluster cleanup wipes a whole generation The pre-encode cluster sweep deleted only the listed canonical shards on remote nodes, leaving index/sidecar (and, on builds with versioned generations, those too) behind. Add a full_teardown flag to VolumeEcShardsDelete that evicts the volume and wipes every EC artifact for it on every disk via removeStaleEcArtifacts; the shell and worker pre-encode cleanup paths set it. Other delete callers (balance/decode/repair) are unchanged. * fix(ec): take ecjFileAccessLock before the nil-check in Sync and Close Sync and Close read ev.ecjFile before acquiring ecjFileAccessLock while Close nils it under the lock, a data race on the field. Take the lock first, then nil-check inside, in both. * fix(ec): acknowledge full_teardown so a pre-upgrade server can't fake success An old volume server silently ignores full_teardown and returns success for an ordinary delete, so the caller wrongly believes the generation was wiped and copies a fresh gen-0 onto an unwiped node. Echo full_teardown_done in the response; the worker destination cleanup fails when it is absent, and the shell cluster sweep fails for a reported (mounted) leftover while staying best-effort for an unreported node. encode_ts_ns stays an accepted transient (an old server just skips the new read guard, no regression). * fix(ec): fail the pre-encode sweep for any reachable node that can't ack teardown A reachable pre-upgrade server ignores full_teardown and returns success without wiping an orphan, which a later copy then folds into the new generation. Treat a missing full_teardown_done ack as fatal for every reachable node (best-effort only for a gRPC-unreachable one), not just for topology-reported pairs. * fix(ec): return the served shard identity and validate it client-side The encode identity was only enforced server-side, so a pre-upgrade server ignored the request field and served bytes unchecked. Echo the served shard's EncodeTsNs on every read response chunk and have the client reject a mismatch (including 0 from an old server), so the guard holds regardless of server version; a rejected read recovers from parity. * fix(ec): reject a short/empty remote shard read instead of serving zeros doReadRemoteEcShardInterval accepted an immediate EOF or a short stream and returned success with a partly zero-filled, unvalidated buffer (the server stamps the identity only on chunks that carry bytes). A non-deleted interval must arrive whole: require n == len(buf), exempting the is_deleted short-circuit (n=0), matching readLocalEcShardInterval's local check. A short read now fails so the caller recovers from parity. * test(ec): fake volume server echoes the full_teardown acknowledgement The worker now fails a teardown delete that isn't acknowledged (so a pre-upgrade server can't silently skip the wipe). The fake server's no-op VolumeEcShardsDelete returned an empty response, which the worker read as a skipped teardown and aborted the encode. Echo full_teardown_done. * feat(ec): mirror the encode-run identity guard + full_teardown into the Rust volume server The Go volume server stamps an encode-run identity (encode_ts_ns) into the .vif and rejects a read served from a shard of a different run; full_teardown wipes a whole generation and acknowledges it. The Rust volume server had none of it. Mirror the shared logic: load encode_ts_ns from the .vif onto the EcVolume, stamp it on every read response, and reject a request/response mismatch on both the server and the distributed-read client (recovering from parity); handle full_teardown by evicting the volume and wiping every EC artifact on each disk, echoing full_teardown_done so the caller can detect a server that ignored it. * fix(ec): remove a stale .vif on full teardown of a shard-only node A shard copy installs shards + .ecx before .vif, so an interrupted copy after a teardown could mount the new files under the previous run's identity / version / shard ratio / dat_file_size carried by the surviving .vif. Remove .vif during full teardown, gated on .idx absence so a source-volume holder keeps its live .vif. In Rust this lives in a teardown-only helper so the reconcile / load- fallback paths (which share the base removal) still preserve .vif. * fix(ec): treat a missing teardown ack as fatal, not as an unreachable node isNodeUnreachable returned true for any non-gRPC-status error, so a reachable pre-upgrade server's missing full_teardown_done ack (a plain error) was classified unreachable and the unreported pair was silently skipped. Classify only a real codes.Unavailable as unreachable, and wrap the missing ack in a sentinel the sweep treats as fatal regardless. A genuinely down node still surfaces as Unavailable from the RPC and stays best-effort. * fix(ec): reject a short shard read in the local EC needle reader read_ec_shard_needle ignored the byte count from shard.read_at and appended the whole pre-sized buffer, so a truncated shard's zero-filled tail passed the later length check and parsed as garbage. Require n == buf.len() per interval, erroring on a short read like the local interval reader already does. * fix(ec): probe reachability before skipping a node that returns Unavailable The pre-encode sweep skipped any node whose teardown delete returned codes.Unavailable, but a reachable volume server in maintenance mode also returns that code for the maintenance-gated delete, so its stale EC files were left behind on a node that can still receive the new generation. Confirm with a non-maintenance-gated empty-target Ping: skip only when the node fails the probe too (genuinely unreachable). * fix(ec): use try_exists for the teardown .vif .idx guard The teardown-only .vif removal gated on Path::exists(), which returns false on a permission/IO stat error, so a stat failure on a present .idx would read as a shard-only node and delete the live source volume's .vif. Gate on try_exists() == Ok(false) instead, preserving the sidecar on any stat error. * fix(ec): only skip a sweep node when a Ping confirms it is transport-down The pre-encode sweep skipped a node whenever its teardown delete and a liveness Ping both failed, but it treated ANY Ping error as down — an application-level Internal/ResourceExhausted, or Unimplemented from a pre-Ping server, left a reachable node's stale generation in place. Classify the Ping tri-state and skip only when it transport-fails with codes.Unavailable; a reachable or inconclusive node stays fatal. * fix(ec): exclude sweep-skipped nodes from the encode's rebalance The pre-encode sweep skips a genuinely-down node best-effort, but the rebalance then recollected the current topology — a node that recovered between the two could become a copy target and receive the new generation while still holding its stale, never-cleared shards. Have the sweep return the skipped set and exclude those nodes from the rebalance for this encode, so a node we could not clean cannot receive the new generation. Standalone ec.balance is unaffected. * fix(ec): re-sweep recovered nodes before generation so they aren't stranded A node skipped as down by the pre-encode sweep is excluded from the rebalance, but it can recover and become the generation host — mounting all shards locally, then being excluded from distribution. Union-only verification accepts all shards on one node and deletes the originals: a single point of failure. Re-sweep the skipped nodes just before generation; one whose teardown now succeeds leaves the skipped set and rebalances normally, while a node still down stays skipped. * fix(ec): abort the encode if a selected source is still skipped after re-sweep The re-sweep un-skips a recovered node, but the source was selected before it and a node can stay down through the re-sweep then recover just in time to be the generation host — mounting all shards locally while still excluded from the rebalance, which union-only verification accepts before deleting the originals. Abort the encode when a selected source remains skipped after the re-sweep. * fix(ec): batch delete returns retriable 503 when a volume became EC mid-batch If a volume is not EC at the batch-delete classification but is encoded to EC and its .dat deleted before the regular-volume mutation, the mutation returns an exact "not found" that the filer chunk-GC treats as completed, dropping the delete. Recheck EC presence under the mutation lock and return a retriable 503 with the "try again" token so the filer requeues it onto the EC path. * fix(ec): recheck EC state before the regular batch-delete mutation ec.encode mounts EC shards (copied from the .dat) before deleting the originals, so a volume can be EC while its .dat still exists. The batch delete only rechecked EC after a NotFound, so a successful regular-volume delete in that window wrote a tombstone to the soon-removed .dat — the delete was lost and the needle resurrected from the pre-tombstone shards. Recheck has_ec_volume under the write lock before delete_volume_needle and return a retriable 503 so the filer requeues onto the EC path. * fix(volume): make the metrics push test independent of test order test_push_metrics_once asserted the pushed body contains the request-counter family without ever touching the counter — a CounterVec with no children emits nothing, so the assertion only held when another test had already created a labelset in the shared registry. Create one in the test itself. |
||
|
Chris Lu
|
1c9039d3ac |
fix(seaweed-volume): stop EC shard deletion from phantom .dat on restart (#9874)
* fix(seaweed-volume): stop EC shard deletion from phantom .dat on restart On startup load_existing_volumes() scans .vif/.idx entries (not just .dat). For distributed EC, a volume's .vif can be mirrored onto a disk whose .ecx lives on a sibling disk, so the per-disk ecx check is false and the loader falls through to Volume::new, which always creates the .dat if missing -> a phantom 8-byte superblock stub. The store-level prune_incomplete_ec_with_sibling_dat then treats that stub as the authoritative source and deletes the real EC shards on sibling disks. Go guards the same case (disk_location.go: 'Without this guard NewVolume below would create a phantom empty .dat') but only same-disk. Fix A (root cause): in load_existing_volumes, don't create a .dat during load. Skip the entry when there is no local .dat AND the .vif does not reference remote files -- remote-tiered volumes have no local .dat but must still load via the remote path. Uses the robust check_dat_file_exists helper so a transient stat error doesn't skip a real volume. New volumes go through create_volume(). Covers the cross-disk .vif/.ecx split Go's same-disk hasEcxFile() misses. Fix B (defense in depth, Go + Rust): when the EC .vif records no source size (dat_file_size==0), require the sibling .dat to be strictly larger than a bare superblock, so an empty 8-byte stub can't pass the credibility gate. Previously it fell back to SUPER_BLOCK_SIZE, which an 8-byte stub exactly meets. Adds regression tests reproducing the cross-disk lone-.vif phantom and the 8-byte stub gate; updates an existing prune test to use a real collection so its .ecx lookup matches the loaders. * fix(storage): don't create phantom .dat from lone .vif on Go volume load Mirror Fix A on the Go side. loadExistingVolume scans .vif/.idx entries, and for distributed EC a .vif can be mirrored onto a disk whose .ecx is on a sibling disk. The same-disk hasEcxFile() guard does not fire there, so the loader falls through to NewVolume(createDatIfMissing=true) and writes an 8-byte phantom .dat, which the sibling-.dat prune then uses to delete the real EC shards on sibling disks. Skip the entry when there is no local .dat AND the .vif has no remote file (via MaybeLoadVolumeInfo); remote-tiered volumes have no local .dat but must still load. Adds TestLoneVifDoesNotCreatePhantomDat (fails without the guard) and TestRemoteTier_DiskScanLoadsRemoteOnlyVolume (fails if the guard skips a remote-only volume). |
||
|
Chris Lu
|
ab7be7867d |
security: hot-reload JWT signing keys on SIGHUP (#9826)
* security: reload JWT signing keys on SIGHUP Signing keys were read once in the server constructors and never refreshed. After a key rotation (Secret update, divergent reads) the in-memory key stayed stale and every request kept failing "wrong jwt" until the affected process was restarted. Add Guard.UpdateSigningKeys and call it from the master, volume and filer reload paths and the s3 reload hook, next to the existing whitelist refresh. Make the global chunk-read JWT cache reloadable via an atomic swap, and register the master's Reload with grace.OnReload -- it was never wired, so the master ignored SIGHUP entirely. Mirror the same refresh in the Rust volume server's SIGHUP handler. * security: swap signing keys behind an atomic pointer Addresses review feedback on the in-place key swap: SigningKey is a []byte, so reassigning the Guard fields while a request handler reads them is a data race that can tear the multi-word slice header and read out of bounds. Hold the four signing-key fields in an immutable signingConfig snapshot behind atomic.Pointer; UpdateSigningKeys swaps the whole pointer, so a reader sees either the old keys or the new ones. Reads go through new SigningKey/ExpiresAfterSec/ReadSigningKey/ReadExpiresAfterSec accessors. The Rust guard is already safe: every read and the SIGHUP write go through the shared RwLock<Guard>. * security: fold whitelist + auth state into the atomic snapshot Review follow-up. UpdateSigningKeys still wrote isWriteActive while the request path read it (and the whitelist maps) unsynchronized, so a SIGHUP under load could expose an inconsistent mix of activation bits and whitelist contents. Move all hot-reloadable Guard state -- keys, expirations, whitelist, and the activation flags -- into a single immutable guardState swapped behind one atomic.Pointer. The Update* methods take a small mutex to serialize the read-modify-write; readers stay lock-free. The concurrency test now also rotates the whitelist and probes IsWhiteListed under -race. Also read each signing key once per branch in the volume/filer JWT auth checks, so a reload landing mid-check can't take the allow-fast-path after auth was enabled or verify against a different key than the branch saw. |
||
|
Chris Lu
|
e264e9883e |
fix(seaweed-volume): bound request body and stored-content expansion to prevent OOM under load (#9780)
* fix(seaweed-volume): bound request body and stored-content expansion to prevent OOM The Rust volume server buffered the entire upload body with to_bytes(usize::MAX) and only checked the file-size limit afterward, so a single large upload — or many concurrent uploads, since the in-flight byte throttle defaults to 0 (unlimited) — could exhaust memory and get the process OOM-killed under load. The read path had two more single-request OOM vectors: `vec![0u8; manifest.size]` allocated from an attacker-controlled chunk-manifest size, and gzip decompression was unbounded (gzip bomb). - Bound the upload body read by file_size_limit_bytes (plus a margin for multipart framing), mirroring Go's io.LimitReader(sizeLimit+1), and reject oversize before the whole body is buffered. - Validate manifest.size (reject negative / oversized) before allocating. - Cap gzip output in maybe_decompress_gzip and route the inline GzDecoder sites through it. * fix(seaweed-volume): address review - chunk offset, 32-bit cast, decompress errors - Validate chunk.offset before indexing in chunk-manifest expansion: a negative offset wrapped to a huge usize and underflowed `end - offset` (panic from a crafted manifest). Reject negative, skip out-of-range, use saturating math. - Use usize::try_from for the upload body limit instead of `as usize`, so a >usize::MAX file_size_limit on 32-bit caps at usize::MAX rather than silently truncating to a tiny value. - maybe_decompress_gzip now returns Result<_, GunzipError> distinguishing a decode failure (callers fall back to raw bytes, as before) from hitting the size cap (TooLarge), which now returns 413 instead of silently serving the still-compressed bytes. * fix(seaweed-volume): inflate manifest chunks into the result window to cap peak memory The chunk-manifest expansion still doubled memory: `result` was already allocated at manifest.size (<=2 GiB) and each compressed chunk was inflated into a separate Vec (also up to 2 GiB), so a single request could peak near 4 GiB. Decompress compressed chunks directly into their result[offset..] window (bounded by the remaining space) so a chunk never allocates a second large buffer; peak stays at ~manifest.size. Bytes past the window are dropped (matching the prior truncation), and a fully-undecodable chunk still falls back to its raw bytes. * fix(seaweed-volume): fall back to raw chunk bytes on any decode failure Per review: the gzip fallback must run on any decode error, not only when no bytes were decoded. Clear the partially-written output and copy the chunk's raw bytes (truncated to the window), restoring the prior decode-failure behavior. |
||
|
Chris Lu
|
dfa86b4313 |
volume: keep volume writable after a deletion-tail compaction (#9776)
makeupDiff replays post-snapshot changes onto the compacted volume. For a replayed deletion it appended a tombstone to the new .dat but recorded the .idx entry with offset 0. When that deletion is the last replayed change the tombstone lands at the .dat tail, and the post-commit integrity check skips offset-0 entries, so it sees 32 trailing bytes it can't account for and flips the volume read-only, reloading it as a SortedFileNeedleMap instead of the writable map. Record the tombstone's real .dat offset, matching the normal delete path; the needle map still treats it as deleted off the negative size, so lookups are unchanged. Mirror the same fix into the Rust volume server. |