Chris Lu
52db7a6aee
The cached-location test needs two files on one volume, but it wrote six files into the six volumes a 001 layout starts with, and every so often each file landed on its own volume and the test had nothing to probe. Seven files leave no way to spread them out. Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
FUSE volume server failover tests
Integration tests for what happens to FUSE mounts when a volume server goes away, comes back, or restarts underneath in-flight IO. They automate the manual matrix reported in discussion #10206: one mount appends to a file while a second mount tails it, and a volume server is stopped, started or restarted mid-stream.
The cluster is 1 master (-defaultReplication=001), 3 volume servers, 1 filer
and 2 mounts, all as local processes. With 001 every chunk has a copy on two of
the three servers, so losing any single server must be invisible to both mounts.
| Reported scenario | Test |
|---|---|
| control: append + tail with nothing failing | TestAppendWithoutChaos |
| read a file while one volume server is down | TestReadWithVolumeServerDown |
| "STOP volumes": append + tail, kill a server mid-stream | TestAppendWhileVolumeServerStops |
| "Start volumes": append + tail with a server down, start it mid-stream | TestAppendWhileVolumeServerStarts |
| "Re-start volumes": append + tail, restart a server mid-stream | TestAppendWhileVolumeServerRestarts |
| part 2: large file copy instead of small appends | TestLargeWriteWhileVolumeServerStops |
Volume servers are dropped with SIGKILL, the closest local equivalent of a Swarm task disappearing from the overlay network: no deregistration, and the address stops answering.
Running
go build -o weed/weed ./weed
WEED_BINARY=$PWD/weed/weed go test -v -count=1 -timeout=30m ./test/fuse_failover/...
Needs FUSE and, on Linux, a working /dev/fuse. Logs from a failed run are
copied to /tmp/seaweedfs-fuse-failover-logs/.