mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-23 06:20:48 +02:00
refactor: share volume and EC shard move logic between shell and workers (#10727)
* operation: add shared volume_move package for volume and EC shard moves The shell commands (volume.move, volume.balance, ec.balance, tier moves) and the maintenance workers (balance, ec_balance) each carried their own copy of the move RPC sequences, and the copies had drifted: the worker verified the target before deleting the source but dropped the disk type and IO throttle; the shell passed those but deleted the source unverified. volume_move.Mover carries the merged sequences, keeping the stricter behavior from each side: - LiveMoveVolume: check-then-hard-freeze the source (VolumeStatus's IsReadOnly also covers low-disk and readonly-but-can-delete states, which still accept needle deletes), copy with disk type and IO throttle, tail, verify the target is not behind the source before the destructive source delete (a target that is ahead holds writes it accepted during the tail and the move commits to keep them), and restore the source's writability when a failure precedes the delete and this move did the freezing. Aborts clean up the incomplete target copy; a failed cleanup or an ambiguous source delete keeps the source readonly (ErrSourceKeptReadonly) so callers do not thaw a source next to a possibly-authoritative copy. With a readonly source, an existing or unknown-state target refuses the move outright: no client-side observation can prove such a copy is a stale remnant rather than the authoritative copy of an unfinished move. - MoveEcShards: copy with the .ecx/.ecj/.vif/.ecsum sidecars, mount, verify the target registered every shard before unmount+delete on the source, and reject same-server moves (the EC delete is server-wide). Server identity is the grpc endpoint (SameServer), so node:8080 and node:8080.18080 compare equal while test servers sharing a degenerate HTTP address stay distinct; addresses are validated non-fatally before dialing and before being embedded in copy/tail requests, since both the client dialer and the receiving server normalize them through a parser that aborts the process on a malformed port. The Rust volume server's codes.NotFound counts as a definitively absent probe answer alongside the Go server's plain-error code Unknown. All RPCs go through an injectable ClientFunc, so the sequences are unit tested against a fake volume server client: RPC order, request fields, and that verification failures keep the source intact. * shell, worker: delegate volume and EC shard moves to operation/volume_move LiveMoveVolume and the copy/tail/delete/mark-writable helpers become thin wrappers over the shared mover, keeping their signatures; the EC helpers keep their per-step output and delegate the RPCs. BalanceTask and ECBalanceTask keep their parameter validation, progress reporting, and guards (same-node cross-disk rejection, dedup keep-node verification, shard ids range-checked before the uint8 narrowing) and hand the RPC sequences to the mover. volume.tier.move skips its thaw-on-failure when the mover deliberately kept the source readonly, since reopening the replicas beside a possibly-authoritative target copy would fork the volume. The tail-failure tolerance moves inside the mover: a failed tail is tolerated only when the volume was already readonly before the move began, backstopped by a stability re-read across the idle window, so volume.balance's -skipTailError-by-readonly heuristic and tier-move's unconditional skip both become the same authoritative rule. * volume_move: keep the source readonly when a failed copy leaves a target of unknown origin A failed copy can leave a complete, mounted copy on the target (the server finishes after the client loses the stream). The abort probed the target only when its pre-copy state was known-absent; an unknown prior state skipped both the probe and the cleanup and then reopened the source - two writable replicas of one volume, diverging from the next write on. The abort now probes the target on every failed copy and restores the source only when the target provably holds nothing. A copy whose provenance cannot be proven (unknown prior state, a pre-existing replica, or an unreachable target) is never deleted, and the source stays readonly with ErrSourceKeptReadonly naming the recovery. * test: teach the plugin worker harness the shared move sequence The fake volume server lacked VolumeStatus, which the shared mover now issues before freezing the source, and the batch execution test's status-read accounting predates the pre-copy target probe and the verification reads. Mirrors the harness the enterprise tree already carries.
This commit is contained in:
Chris Lu authored and GitHub committed
1 parent
c55ef964d9
commit
0799084e98
17 files changed
+1978
-618
No files matched your search
@@ -2,20 +2,18 @@ package shell
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/wdclient"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/operation"
|
||||
"github.com/seaweedfs/seaweedfs/weed/operation/volume_move"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/util"
|
||||
|
||||
"google.golang.org/grpc"
|
||||
)
|
||||
@@ -42,7 +40,7 @@ func (c *commandVolumeMove) Help() string {
|
||||
1. This command marks the source volume as read-only, copies it to the target volume server, and records the last entry timestamp.
|
||||
2. This command asks the target volume server to mount the new volume.
|
||||
3. This command asks the target volume server to tail the source volume for updates after the timestamp, for 1 minutes to drain any in-flight requests.
|
||||
4. This command asks the source volume server to delete the source volume.
|
||||
4. This command verifies the target volume matches the source, then asks the source volume server to delete the source volume.
|
||||
|
||||
The option "-disk [hdd|ssd|<tag>]" can be used to change the volume disk type.
|
||||
The option "-timeout" fails the whole move if it does not finish in time.
|
||||
@@ -81,7 +79,7 @@ func (c *commandVolumeMove) Do(args []string, commandEnv *CommandEnv, writer io.
|
||||
|
||||
volumeId := needle.VolumeId(*volumeIdInt)
|
||||
|
||||
if sourceVolumeServer == targetVolumeServer {
|
||||
if volume_move.SameServer(sourceVolumeServer, targetVolumeServer) {
|
||||
return fmt.Errorf("source and target volume servers are the same!")
|
||||
}
|
||||
|
||||
@@ -92,170 +90,32 @@ func (c *commandVolumeMove) Do(args []string, commandEnv *CommandEnv, writer io.
|
||||
defer cancel()
|
||||
}
|
||||
|
||||
return LiveMoveVolume(ctx, commandEnv.option.GrpcDialOption, writer, volumeId, sourceVolumeServer, targetVolumeServer, 5*time.Second, *diskTypeStr, *ioBytePerSecond, false)
|
||||
return LiveMoveVolume(ctx, commandEnv.option.GrpcDialOption, writer, volumeId, sourceVolumeServer, targetVolumeServer, 5*time.Second, *diskTypeStr, *ioBytePerSecond)
|
||||
}
|
||||
|
||||
// LiveMoveVolume moves one volume from one source volume server to one target volume server, with idleTimeout to drain the incoming requests.
|
||||
func LiveMoveVolume(ctx context.Context, grpcDialOption grpc.DialOption, writer io.Writer, volumeId needle.VolumeId, sourceVolumeServer, targetVolumeServer pb.ServerAddress, idleTimeout time.Duration, diskType string, ioBytePerSecond int64, skipTailError bool) (err error) {
|
||||
|
||||
log.Printf("copying volume %d from %s to %s", volumeId, sourceVolumeServer, targetVolumeServer)
|
||||
lastAppendAtNs, leftReadonly, err := copyVolume(ctx, grpcDialOption, writer, volumeId, sourceVolumeServer, targetVolumeServer, diskType, ioBytePerSecond, false)
|
||||
if err != nil {
|
||||
return fmt.Errorf("copy volume %d from %s to %s: %v", volumeId, sourceVolumeServer, targetVolumeServer, err)
|
||||
}
|
||||
|
||||
// A move aborted after the copy must restore the source writability, or
|
||||
// the source volume is left permanently readonly.
|
||||
var sourceDeleteStarted bool
|
||||
defer func() {
|
||||
if err == nil || !leftReadonly {
|
||||
return
|
||||
}
|
||||
if !sourceDeleteStarted {
|
||||
// The target copy may be missing tailed entries; remove it so the
|
||||
// restored source stays the only replica.
|
||||
deleteCtx, deleteCancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
|
||||
defer deleteCancel()
|
||||
if dErr := deleteVolume(deleteCtx, grpcDialOption, volumeId, targetVolumeServer, false, true); dErr != nil {
|
||||
// Restoring the source while the stale target stays mounted
|
||||
// risks divergent replicas; re-running the move recovers both.
|
||||
log.Printf("volume %d is left readonly on %s: failed to delete the incomplete copy on %s: %v; re-run volume.move to recover", volumeId, sourceVolumeServer, targetVolumeServer, dErr)
|
||||
return
|
||||
}
|
||||
}
|
||||
restoreCtx, restoreCancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
|
||||
defer restoreCancel()
|
||||
if wErr := markVolumeWritable(restoreCtx, grpcDialOption, volumeId, sourceVolumeServer, true, false); wErr != nil {
|
||||
log.Printf("failed to restore volume %d writable on %s: %v", volumeId, sourceVolumeServer, wErr)
|
||||
}
|
||||
}()
|
||||
|
||||
log.Printf("tailing volume %d from %s to %s", volumeId, sourceVolumeServer, targetVolumeServer)
|
||||
if err = tailVolume(ctx, grpcDialOption, volumeId, sourceVolumeServer, targetVolumeServer, lastAppendAtNs, idleTimeout); err != nil {
|
||||
if skipTailError {
|
||||
fmt.Fprintf(writer, "tail volume %d from %s to %s: %v\n", volumeId, sourceVolumeServer, targetVolumeServer, err)
|
||||
} else {
|
||||
return fmt.Errorf("tail volume %d from %s to %s: %v", volumeId, sourceVolumeServer, targetVolumeServer, err)
|
||||
}
|
||||
}
|
||||
|
||||
log.Printf("deleting volume %d from %s", volumeId, sourceVolumeServer)
|
||||
sourceDeleteStarted = true
|
||||
if err = deleteVolume(ctx, grpcDialOption, volumeId, sourceVolumeServer, false, true); err != nil {
|
||||
return fmt.Errorf("delete volume %d from %s: %v", volumeId, sourceVolumeServer, err)
|
||||
}
|
||||
|
||||
log.Printf("moved volume %d from %s to %s", volumeId, sourceVolumeServer, targetVolumeServer)
|
||||
return nil
|
||||
func LiveMoveVolume(ctx context.Context, grpcDialOption grpc.DialOption, writer io.Writer, volumeId needle.VolumeId, sourceVolumeServer, targetVolumeServer pb.ServerAddress, idleTimeout time.Duration, diskType string, ioBytePerSecond int64) (err error) {
|
||||
return volume_move.NewMover(grpcDialOption).LiveMoveVolume(ctx, volumeId, sourceVolumeServer, targetVolumeServer, volume_move.VolumeMoveOptions{
|
||||
DiskType: diskType,
|
||||
IoBytePerSecond: ioBytePerSecond,
|
||||
IdleTimeout: idleTimeout,
|
||||
Writer: writer,
|
||||
})
|
||||
}
|
||||
|
||||
func copyVolume(ctx context.Context, grpcDialOption grpc.DialOption, writer io.Writer, volumeId needle.VolumeId, sourceVolumeServer, targetVolumeServer pb.ServerAddress, diskType string, ioBytePerSecond int64, restoreWritable bool) (lastAppendAtNs uint64, leftReadonly bool, err error) {
|
||||
|
||||
// check to see if the volume is already read-only and if its not then we need
|
||||
// to mark it as read-only and then before we return we need to undo what we
|
||||
// did
|
||||
var shouldMarkWritable bool
|
||||
defer func() {
|
||||
if !shouldMarkWritable {
|
||||
return
|
||||
}
|
||||
if !restoreWritable && err == nil {
|
||||
leftReadonly = true
|
||||
return
|
||||
}
|
||||
|
||||
// Restoring writability must outlive a cancelled copy, or the source
|
||||
// is left permanently readonly on abort, as balance_task.go already
|
||||
// guards against.
|
||||
restoreCtx, restoreCancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
|
||||
defer restoreCancel()
|
||||
clientErr := operation.WithVolumeServerClient(false, sourceVolumeServer, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
|
||||
_, writableErr := volumeServerClient.VolumeMarkWritable(restoreCtx, &volume_server_pb.VolumeMarkWritableRequest{
|
||||
VolumeId: uint32(volumeId),
|
||||
})
|
||||
return writableErr
|
||||
})
|
||||
if clientErr != nil {
|
||||
log.Printf("failed to mark volume %d as writable after copy from %s: %v", volumeId, sourceVolumeServer, clientErr)
|
||||
}
|
||||
}()
|
||||
|
||||
err = operation.WithVolumeServerClient(false, sourceVolumeServer, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
|
||||
resp, statusErr := volumeServerClient.VolumeStatus(ctx, &volume_server_pb.VolumeStatusRequest{
|
||||
VolumeId: uint32(volumeId),
|
||||
})
|
||||
if statusErr == nil && !resp.IsReadOnly {
|
||||
shouldMarkWritable = true
|
||||
_, readonlyErr := volumeServerClient.VolumeMarkReadonly(ctx, &volume_server_pb.VolumeMarkReadonlyRequest{
|
||||
VolumeId: uint32(volumeId),
|
||||
Persist: false,
|
||||
})
|
||||
return readonlyErr
|
||||
}
|
||||
return statusErr
|
||||
})
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
err = operation.WithVolumeServerClient(true, targetVolumeServer, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
|
||||
stream, replicateErr := volumeServerClient.VolumeCopy(ctx, &volume_server_pb.VolumeCopyRequest{
|
||||
VolumeId: uint32(volumeId),
|
||||
SourceDataNode: string(sourceVolumeServer),
|
||||
DiskType: diskType,
|
||||
IoBytePerSecond: ioBytePerSecond,
|
||||
})
|
||||
if replicateErr != nil {
|
||||
return replicateErr
|
||||
}
|
||||
for {
|
||||
resp, recvErr := stream.Recv()
|
||||
if recvErr != nil {
|
||||
if recvErr == io.EOF {
|
||||
break
|
||||
} else {
|
||||
return recvErr
|
||||
}
|
||||
}
|
||||
if resp.LastAppendAtNs != 0 {
|
||||
lastAppendAtNs = resp.LastAppendAtNs
|
||||
} else {
|
||||
fmt.Fprintf(writer, "%s => %s volume %d processed %s\n", sourceVolumeServer, targetVolumeServer, volumeId, util.BytesToHumanReadable(uint64(resp.ProcessedBytes)))
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
})
|
||||
|
||||
return
|
||||
func copyVolume(ctx context.Context, grpcDialOption grpc.DialOption, writer io.Writer, volumeId needle.VolumeId, sourceVolumeServer, targetVolumeServer pb.ServerAddress, diskType string, ioBytePerSecond int64, restoreWritable bool) (lastAppendAtNs uint64, err error) {
|
||||
return volume_move.NewMover(grpcDialOption).CopyVolume(ctx, volumeId, sourceVolumeServer, targetVolumeServer, diskType, ioBytePerSecond, restoreWritable, writer)
|
||||
}
|
||||
|
||||
func tailVolume(ctx context.Context, grpcDialOption grpc.DialOption, volumeId needle.VolumeId, sourceVolumeServer, targetVolumeServer pb.ServerAddress, lastAppendAtNs uint64, idleTimeout time.Duration) (err error) {
|
||||
|
||||
return operation.WithVolumeServerClient(true, targetVolumeServer, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
|
||||
_, replicateErr := volumeServerClient.VolumeTailReceiver(ctx, &volume_server_pb.VolumeTailReceiverRequest{
|
||||
VolumeId: uint32(volumeId),
|
||||
SinceNs: lastAppendAtNs,
|
||||
IdleTimeoutSeconds: uint32(idleTimeout.Seconds()),
|
||||
SourceVolumeServer: string(sourceVolumeServer),
|
||||
})
|
||||
return replicateErr
|
||||
})
|
||||
|
||||
return volume_move.NewMover(grpcDialOption).TailVolume(ctx, volumeId, sourceVolumeServer, targetVolumeServer, lastAppendAtNs, idleTimeout)
|
||||
}
|
||||
|
||||
// deleteVolume removes the volume from sourceVolumeServer. When keepRemoteData
|
||||
// is true, the cloud-tier object backing the volume is left intact — used on
|
||||
// the source side of a move where another server is taking over the same .vif.
|
||||
func deleteVolume(ctx context.Context, grpcDialOption grpc.DialOption, volumeId needle.VolumeId, sourceVolumeServer pb.ServerAddress, onlyEmpty bool, keepRemoteData bool) (err error) {
|
||||
return operation.WithVolumeServerClient(false, sourceVolumeServer, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
|
||||
_, deleteErr := volumeServerClient.VolumeDelete(ctx, &volume_server_pb.VolumeDeleteRequest{
|
||||
VolumeId: uint32(volumeId),
|
||||
OnlyEmpty: onlyEmpty,
|
||||
KeepRemoteData: keepRemoteData,
|
||||
})
|
||||
return deleteErr
|
||||
})
|
||||
return volume_move.NewMover(grpcDialOption).DeleteVolume(ctx, volumeId, sourceVolumeServer, onlyEmpty, keepRemoteData)
|
||||
}
|
||||
|
||||
func markVolumeWritable(ctx context.Context, grpcDialOption grpc.DialOption, volumeId needle.VolumeId, sourceVolumeServer pb.ServerAddress, writable, persist bool) (err error) {
|
||||
@@ -300,44 +160,10 @@ func markVolumeReplicasWritable(ctx context.Context, grpcDialOption grpc.DialOpt
|
||||
|
||||
// replicateVolumeToServer copies a volume from sourceAddress to targetAddress via the VolumeCopy gRPC stream.
|
||||
func replicateVolumeToServer(ctx context.Context, grpcDialOption grpc.DialOption, writer io.Writer, volumeId needle.VolumeId, sourceAddress, targetAddress pb.ServerAddress, diskType string) error {
|
||||
return operation.WithVolumeServerClient(false, targetAddress, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
|
||||
stream, replicateErr := volumeServerClient.VolumeCopy(ctx, &volume_server_pb.VolumeCopyRequest{
|
||||
VolumeId: uint32(volumeId),
|
||||
SourceDataNode: string(sourceAddress),
|
||||
DiskType: diskType,
|
||||
})
|
||||
if replicateErr != nil {
|
||||
return replicateErr
|
||||
}
|
||||
for {
|
||||
resp, recvErr := stream.Recv()
|
||||
if recvErr != nil {
|
||||
if recvErr == io.EOF {
|
||||
break
|
||||
}
|
||||
return recvErr
|
||||
}
|
||||
if resp.ProcessedBytes > 0 {
|
||||
fmt.Fprintf(writer, "volume %d processed %s bytes\n", volumeId, util.BytesToHumanReadable(uint64(resp.ProcessedBytes)))
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return volume_move.NewMover(grpcDialOption).ReplicateVolume(ctx, volumeId, sourceAddress, targetAddress, diskType, writer)
|
||||
}
|
||||
|
||||
// configureVolumeReplication sets the replication setting on a volume at the given server.
|
||||
func configureVolumeReplication(ctx context.Context, grpcDialOption grpc.DialOption, volumeId needle.VolumeId, targetAddress pb.ServerAddress, replicationString string) error {
|
||||
return operation.WithVolumeServerClient(false, targetAddress, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
|
||||
resp, configureErr := volumeServerClient.VolumeConfigure(ctx, &volume_server_pb.VolumeConfigureRequest{
|
||||
VolumeId: uint32(volumeId),
|
||||
Replication: replicationString,
|
||||
})
|
||||
if configureErr != nil {
|
||||
return configureErr
|
||||
}
|
||||
if resp.Error != "" {
|
||||
return errors.New(resp.Error)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return volume_move.NewMover(grpcDialOption).ConfigureVolumeReplication(ctx, volumeId, targetAddress, replicationString)
|
||||
}
|
||||
Reference in new issue
Block a user