mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 15:41:15 +02:00
4b41329e12
* rust volume: stop a VolumeCopy whose caller has gone VolumeCopy runs its copy in a detached tokio::spawn and reports progress with the send error discarded, so nothing observes the client leaving. When the caller cancels the RPC -- which weed-admin's batch balance does routinely, starting far more copies than it finishes -- the server streamed the whole volume from the source, wrote it to disk, and mounted it. The destination is then left holding a volume nobody took delivery of: its index cache is never reclaimed, and under replication=000 one volume id ends up on two servers, both writable, which concurrent writes can diverge. Three checks now reach the task: - Every chunk in copy_file_from_source, via the sender's is_closed(). This is the one that matters in practice. The first progress report is 128MB in, so for a smaller volume -- the ordinary balance move -- no send ever happens and its result says nothing; only the closed channel does. The sender is passed for the .idx and .vif copies too, with reporting gated separately, so those phases notice as well. - The throttle sleep, which for a throttled copy runs for seconds at a time, now races the sender's closed() instead of being slept through. - Immediately before mount_volume, and once at the top of the task. Cancellation surfaces as an ordinary Err(Status::cancelled), so it lands in the existing error branch that already removes the partial .dat/.idx/ .vif and the .note. That branch also logs now: the error otherwise went to a channel nobody was reading, leaving the operator with the balancer's "delete that copy, then re-run the move" and no cause. This also clears the stranded read-only sources reported on the issue. They are downstream of the orphan mount, not a separate defect: LiveMoveVolume's cleanup probes the target before undoing the freeze (volume_move.go:95, "the server can finish the copy and mount the target even when the client loses the stream"), and when it finds a mounted copy it cannot attribute, or cannot delete, it deliberately keeps the source readonly rather than risk two writable replicas -- the messages at volume_move.go:110 and :123. With nothing mounted on the target the probe reports clean and the freeze is undone. On Go parity: the progress send result is honoured here too, matching `return false` in volume_grpc_copy.go. But that report is Go's only abort signal, and measured against a 120MiB volume -- above the throttler's activation threshold, below the 128MiB report interval -- a Go destination mounts an abandoned copy as well. The issue's premise that Go aborts holds only above the report interval. The Rust side now stops in both cases; the Go behaviour is worth its own issue. Tests: the integration test runs against both implementations and is green on Go, red on Rust before this change. Its 192MiB fixture is sized for two separate constraints, documented at the fixture: IoBytePerSecond is a no-op below ~100ms of wall clock (64MiB copies in ~110ms on a tmpfs loopback cluster), and the payload must exceed the 128MiB report interval for the Go leg to pass at all. The two Rust unit tests cover what the integration test cannot reach: cancellation detected with no progress report at all, and the cleanup of the partial files plus the .note. Fixes #11186 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122W3eqt6gmLUMxmRoZdPAb * rust volume: surface VolumeCopy cancellation as Status::cancelled copy_file_from_source returned Result<_, String>, so the per-chunk cancellation path -- the one that matters in practice for volumes below the 128MB report interval -- was wrapped to Status::internal at the call sites. The spawn logging branch then classified it as a generic failure instead of the intended "abandoned by caller", defeating the logging change in the same PR for the case that occurs most often. Return Result<_, Status> from copy_file_from_source: Status::cancelled for caller-gone, Status::internal for the existing errors. Drop the .map_err(|e| Status::internal(e)) at the three call sites. The unit test now asserts the code is Cancelled, not just the message text, so the classification is locked in. * rust volume: close cancellation gaps in VolumeCopy Address two review findings on the same PR: 1. Roll back a mount that races a departing caller. The pre-mount is_closed() check cannot close the window between the check and mount_volume: if the receiver drops in that gap, the volume mounts and the final tx.send(Ok(...)) fails, but its error was discarded (let _ =), so the task returned Ok(()) and the error branch never ran. The destination then held an orphaned mounted replica — the exact defect this PR prevents. Fix: track a mounted flag. The final send now checks its result; on failure it returns Status::cancelled, and the error branch calls store.delete_volume (which unmounts AND removes the files) when mounted is true, instead of only unlinking. 2. Observe cancellation while awaiting the source stream. The per-chunk is_closed() check only runs after stream.message().await returns. A stalled source (slow disk, partition, GC pause) never delivers a chunk, so a caller that has already left cannot preempt the read: the task, the source connection, and the partial files (including the .note) all outlive the caller indefinitely. Fix: race stream.message() against progress_tx.closed() in a tokio::select!, so a departing caller preempts a stalled source. Adds test_volume_copy_after_mount_cancellation_rolls_back_mount to cover the after-mount rollback path. cargo test --release green (497 + 5 + 1 + 28). * rust volume: keep remote data on after-mount rollback, race RPC startup Two review findings on the after-mount rollback added inff51f6a: 1. The rollback called delete_volume(vid, false, false), i.e. keep_remote_data=false. A remote-tier copy .vif points at the same cloud object the source replica references, so destroying the abandoned destination with keep_remote_data=false deletes the source remote data via Volume::destroy backend.delete_file_blocking. Use keep_remote_data=true, matching the pre-spawn delete_volume at the top of volume_copy. 2. client.copy_file(copy_req).await (the initial RPC establishment) was not raced against progress_tx.closed(). If the source stalls before sending response headers, the per-message select! added inff51f6ais never reached, and the task, source connection, and preallocated files outlive a departed caller. Race the RPC establishment against progress_tx.closed() the same way. cargo test --release green (497 + 5 + 1 + 28). * rust volume: race master-configuration wait against caller cancellation try_get_master_configuration().await was the last un-raced await in the VolumeCopy task before copy_file_from_source. A stalled master (or slow leader election) would hold the task and its .note past a departing caller, since the per-chunk cancellation checks are never reached. Race it against tx.closed() the same way the source stream reads already are. cargo test --release green (497 + 5 + 1 + 28). * rust volume: fix after-mount rollback test to actually reach that path The previous version of test_volume_copy_after_mount_cancellation_rolls_back_mount dropped the response immediately after volume_copy returned, so tx.is_closed() was already true at the spawn first check and the task returned before mount_volume. The test was green for the wrong reason: the mounted flag, the rollback, and the keep_remote_data=true line were all uncovered. Use the store write lock as a seam: take it immediately after volume_copy returns so the task runs the copy to completion with the caller still attached, passes the pre-mount is_closed() check, then parks entering the mount block. Drop the response (caller gone) and release the guard: the task mounts, fails the final tx.send, and must roll back via delete_volume. Verified by setting the rollback guard to if false: the test fails with "destination still holds a mounted volume". With the rollback enabled, probe eprintlns confirmed the full path: about to mount -> mounted = true -> final send failed -> rolling back mount -> rollback done. cargo test --release green (497 + 5 + 1 + 28). --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>
Volume Server Integration Tests
This package contains integration tests for SeaweedFS volume server HTTP and gRPC APIs.
Run Tests
Run tests from repo root:
go test ./test/volume_server/... -v
If a weed binary is not found, the harness will build one automatically.
Optional environment variables
WEED_BINARY: explicit path to theweedexecutable (disables auto-build).VOLUME_SERVER_IT_KEEP_LOGS=1: keep temporary test directories and process logs.
Current scope (Phase 0)
- Shared cluster/framework utilities
- Matrix profile definitions
- Initial HTTP admin endpoint checks
- Initial gRPC state/status checks
More API coverage is tracked in /Users/chris/dev/seaweedfs2/test/volume_server/DEV_PLAN.md.