mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-04 02:57:44 +01:00
Chris Lu
49e83a26cb
* feat(seaweed-volume): auto-load EC shards on startup
The Rust volume server's load_existing_volumes only scanned .dat
files; EC shards on disk stayed invisible until something explicitly
issued VolumeEcShardsMount. Strict superset of the issue
seaweedfs/seaweedfs#9212 reports for Go: after a fresh restart, every
local EC shard was missing from the master's view.
Port loadAllEcShards from weed/storage/disk_location_ec.go:
- DiskLocation::load_all_ec_shards walks Directory (and IdxDirectory
if separate) sorted, groups .ec?? shard files by (collection, vid),
validates and mounts each group when its matching .ecx is found.
- handle_found_ecx_file: validate_ec_volume + mount_ec_shards path,
with cleanup when .dat exists and validation fails (incomplete
encoding) or load fails.
- check_orphaned_shards: cleans up shard remnants whose .ecx never
arrived AND whose stale .dat is still present (interrupted
encoding); leaves them on disk otherwise so cross-disk
reconciliation / operator recovery can find them.
- check_dat_file_exists / parse_collection_volume_id /
parse_ec_shard_extension: small helpers mirroring Go's checkDatFileExists,
parseCollectionVolumeId, and the `\.ec\d{2,3}` regex.
- Wire through load_existing_volumes after the .dat scan; failures
log but don't fail the disk's startup.
Tests:
- test_parse_ec_shard_extension covers .ec00–.ec255 and the rejection
of .ec0, .ec999, .ecx, .ecj, .dat, and missing leading dot.
- test_load_all_ec_shards_mounts_pairs_with_ecx: shards + .ecx + .vif
on disk get mounted into ec_volumes after load_existing_volumes.
- test_load_all_ec_shards_keeps_orphan_shards_when_no_dat: orphan
shards (no .ecx, no .dat) stay on disk untouched
(distributed-EC scenario).
- test_load_all_ec_shards_cleans_orphan_shards_when_dat_exists:
orphan shards alongside a stale .dat get cleaned up
(interrupted-encoding scenario).
Prerequisite for porting the cross-disk orphan-shard reconciliation
in seaweedfs/seaweedfs#9244 to Rust.
* fix(seaweed-volume): dedupe filenames when scanning data + idx dirs
load_all_ec_shards scans both `directory` and `idx_directory` (when
they differ) so the loop can pair `.ec??` shards with their `.ecx`
regardless of which dir owns the index. If the same filename is
present in both — possible in idempotent legacy layouts that
pre-date `-dir.idx` — the previous implementation processed it
twice. mount_ec_shards increments the per-shard `ec_shards` metric
inside the loop, so a duplicated `.ec??` entry would double-count
the gauge.
Use a HashSet<String> while accumulating entries so each filename
is processed exactly once.
Reported in PR #9251 review by @gemini-code-assist.
* fix(seaweed-volume): drive partial-mount cleanup through unmount_ec_shards
handle_found_ecx_file calls mount_ec_shards which adds shards one at
a time. mount_ec_shards increments the `ec_shards` gauge per shard
that successfully attaches. If mount fails halfway, plain
ec_volumes.remove(vid) drops the EcVolume but leaves the gauge
incremented for whatever did mount.
Drive the cleanup branches through unmount_ec_shards instead — it
mirror-decrements the gauge per shard and only then drops the
EcVolume. Same shape applied to both .dat-exists and distributed-EC
fallbacks.
Reported in PR #9251 review by @gemini-code-assist.
* docs(seaweed-volume): clarify parse_ec_shard_extension shard-id range
Doc previously said `.ec00`–`.ec999` but the implementation rejects
any shard id > 255 (matches the `EcVolumeShard` u8 typed shard id
and Go's `strconv.ParseInt(... 10, 64)` + `> 255` guard). Fix the
doc to say `.ec00`–`.ec255` and explain why the 3-digit form is
still recognised.
Reported in PR #9251 review by @coderabbitai.