Chris Lu
3b4a681e53
* test(fuse_failover): dump chunk list and hex on append corruption
The failover append test (TestAppendWhileVolumeServerRestarts) failed
in CI with an 8-byte NUL region at offset 632 that appeared in both
the writer mount and the filer own view, but the failure message
only showed a quoted-string window around the divergence. That is
not enough to tell which chunk covered the zeroed bytes or which
volume server held it, so the next recurrence would be just as
unattributable.
Add a FileChunkList helper that reads the filer resolved chunk
list, and on failure dump:
- every chunk fid, offset, size, volume id, and current master
holders, flagging the chunk that covers the first divergence;
- a hex+ASCII dump of the writer mount around the divergence so
the exact zero-filled region is visible byte-for-byte.
No production code is touched; this only makes the test fail louder.
* test(fuse_failover): preserve diagnostic collection errors
Address review feedback from CodeRabbit and Greptile on PR #11285:
- FileChunkList now returns the wrapped ParseUint error when
fid.volume_id is zero and the file_id prefix is invalid, matching
FileVolumeIds instead of silently keeping vid=0 (which would
query /dir/lookup?volumeId=0 and report the wrong holders).
- dumpChunkList captures the VolumeHolders error and renders it as
'lookup failed: ...' so a failed master request is distinguishable
from a successful lookup with no holders (both previously showed
'holders=[unknown]').
- runChaosAppend captures the writer-mount read error and includes
it in the failure message so an unavailable writer view is not
mistaken for corrupted content.
FUSE volume server failover tests
Integration tests for what happens to FUSE mounts when a volume server goes away, comes back, or restarts underneath in-flight IO. They automate the manual matrix reported in discussion #10206: one mount appends to a file while a second mount tails it, and a volume server is stopped, started or restarted mid-stream.
The cluster is 1 master (-defaultReplication=001), 3 volume servers, 1 filer
and 2 mounts, all as local processes. With 001 every chunk has a copy on two of
the three servers, so losing any single server must be invisible to both mounts.
| Reported scenario | Test |
|---|---|
| control: append + tail with nothing failing | TestAppendWithoutChaos |
| read a file while one volume server is down | TestReadWithVolumeServerDown |
| "STOP volumes": append + tail, kill a server mid-stream | TestAppendWhileVolumeServerStops |
| "Start volumes": append + tail with a server down, start it mid-stream | TestAppendWhileVolumeServerStarts |
| "Re-start volumes": append + tail, restart a server mid-stream | TestAppendWhileVolumeServerRestarts |
| part 2: large file copy instead of small appends | TestLargeWriteWhileVolumeServerStops |
Volume servers are dropped with SIGKILL, the closest local equivalent of a Swarm task disappearing from the overlay network: no deregistration, and the address stops answering.
Running
go build -o weed/weed ./weed
WEED_BINARY=$PWD/weed/weed go test -v -count=1 -timeout=30m ./test/fuse_failover/...
Needs FUSE and, on Linux, a working /dev/fuse. Logs from a failed run are
copied to /tmp/seaweedfs-fuse-failover-logs/.