* volume: rebuild a missing .idx from the .dat
Pointing -dir.idx at a directory that holds no index aborted the whole
volume server: checkIdxFile found no .idx and load() called glog.Fatalf.
Every row of the index is derivable from the .dat, so walk it in append
order and write the index back, which reproduces byte for byte what the
server's own writes had left in the old directory.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: keep the index co-located with the data in the Rust server
Go's load() drops back to the data directory when an .idx already sits
beside the .dat, so naming a --dir.idx does not strand a pre-existing
index. Rust had no such adjustment: it opened the new directory with
create, and the volume came up on an empty index with every needle
invisible.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: rebuild a missing .idx from the .dat in the Rust server
Mirrors the Go side. Rust did not abort on a missing index the way
checkIdxFile did; it opened the new directory with create and mounted the
volume on an empty index, so every needle read as missing while the .dat
still held the data. Walk the .dat in append order and write the index
back, byte for byte what the server's own writes had left behind.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: stop the idx rebuild at a zero-padded .dat tail
An all-zero needle header is unwritten space, not a record. Go's .dat walk
keeps reading past it and would index a truncated data file's tail as
millions of needle 0 rows; the Rust walk already stops there. Stop the Go
rebuild at the same place.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: create the -dir.idx directory when it does not exist
Rust's DiskLocation creates the index directory as it takes it; Go only
resolved the path, so naming a directory that does not exist yet left every
volume unable to open or rebuild its index and took the server down.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: stop the idx rebuild at a torn .dat record
A crash between writing a needle's header and its body leaves a record
whose declared size runs past the end of .dat. Indexing it puts a row in
the .idx that points at bytes that do not exist, which fails every read of
that needle and trips the past-EOF check on the next load. Stop at the
first record that does not fit, in both servers.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: stop the idx rebuild at a negative-size header
A corrupt header whose size field is negative makes the .dat walk advance
backwards: NeedleBodyLength adds the negative size, so the next offset is
lower than the current one. The Go walk then reads at a negative offset and
the rebuild fails, which puts the volume server right back to exiting at
startup; the Rust walk seeks past EOF and truncates the index instead.
A negative size is never a record, so stop there.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: skip a volume whose index cannot be rebuilt, do not exit
glog.Fatalf calls os.Exit(255), so a rebuild that could not write -- a full
or read-only index directory -- put the server right back to dying at
startup for one bad volume. Return the error instead: loadExistingVolume
logs it and skips that volume, which is what the remote-volume branch just
above already does and what the Rust loader has always done.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: create the index directory from the rebuild too
The rebuild is the first thing to write into a fresh -dir.idx, and it runs
before the loaders that create the directory on their way to opening .idx.
Create it in both rebuilds so the ordering does not matter.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* ci: let codespell past the sme variable in the mount tests
weedfs_stream_mutate_error_test.go names its *streamMutateError local
sme, which codespell reads as a misspelling of same/some. It is an
identifier, so exempt it beside the other variable-name entries.
Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
* volume: resolve the disk IO slow-latency threshold per disk
volume.toml keys [volume.disk.io.slow.latency] by disk type, but the
threshold was chosen once per server by switching on the raw -disk flag.
-disk is comma-separated, one entry per -dir, so a multi-disk server
matched no case and silently took the hdd threshold.
Carry the table on DiskIOProbeConfig and resolve it in CheckDiskSpace
from the location's own DiskType. A type with no entry keeps falling
back to the hdd threshold.
* volume: run the disk IO probe on multi-directory volume servers
The probe was disabled whenever more than one -dir was configured,
because a single server-wide slow-latency threshold could not describe
disks of different types. The threshold is per disk now, and the rest of
the probe already is: diskRegistry is keyed by directory, each
DiskLocation runs its own CheckDiskSpace, and Store consults
isDiskUnavailable per location.
* volume: reject duplicate -dir entries
Nothing deduplicated -dir, so the same directory listed twice produced two
DiskLocations that each loaded every volume in it, appending to the same .dat
under two independent locks. Compare directory identity with os.SameFile
rather than the path, so a symlink or bind mount aliasing an earlier entry is
rejected as well.
* volume: cover the per-disk slow-latency handoff
SlowLatencyFor has a test, but nothing asserted that CheckDiskSpace feeds it
the location's own disk type. Probe through a seam so the resolved threshold
is observable, and check hdd, ssd, nvme, the empty type, and an unlisted tag.
* Implement RPC skeleton for regular/EC volumes scrubbing.
See https://github.com/seaweedfs/seaweedfs/issues/8018 for details.
* Minor proto improvements for `ScrubVolume()`, `ScrubEcVolume()`:
- Add fields for scrubbing details in `ScrubVolumeResponse` and `ScrubEcVolumeResponse`,
instead of reporting these through RPC errors.
- Return a list of broken shards when scrubbing EC volumes, via `EcShardInfo'.
Chris Lu