shell: keep volume.tier.compact replicas on the same remote object (#11783)

* shell: keep volume.tier.compact replicas on the same remote object

The compact cycle ran on a single replica: it deleted the shared remote
object up front and never touched the siblings, leaving their .vif/.idx
pointing at a removed object. It also remounted the working replica into
a writable local volume mid-operation.

Freeze all replicas first, keep the shared object while siblings still
reference it, replicate the new metadata to every replica, and let the
last tiered replica retire the old object.

* storage: surface remote object delete failures from volume destroy

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: keep tier.compact replicas frozen until every copy lands

A delete between the source unfreeze and a sibling's index copy could
land past the copied snapshot, leaving the replica serving deleted
data. Unfreeze all replicas only after every resync completes; a remote
volume's readonly mark is inert, so the freeze adds no extra cost.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* storage: abort volume destroy when remote delete fails

destroyLocked removed local files even when the backend delete errored,
leaving a registered replica with no .dat/.idx and a leaked remote
object. Run the remote delete before local cleanup and bail out on
failure so the volume stays fully usable and the delete can be retried.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: mirror destroy ordering for remote delete failures

Propagate a failed remote-object delete instead of warning past it, and
reinsert the volume into the disk location registry so a still-intact
replica is not unregistered.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: recover partial tier.compact freezes and report held replicas

A failed freeze left already-marked replicas readonly with nothing
changed, and later stage failures returned without saying the replicas
stay frozen or how to resume. Roll back the partial mark when freezing
fails, note in every mid-flight error that replicas remain readonly and
rerun is the recovery path, and report how many replicas a partial
unfreeze released.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: retire abandoned remote objects during tier.compact resync

A resync overwrote each replica's .vif, orphaning its previous remote
object; an interrupted run could leave replicas on different keys, and
only the last tiered replica's object was deleted. Read every replica's
remote file list up front, count holders per object, and delete-before-
copy whichever replica is an object's last holder, so shared and
stranded objects alike are retired instead of leaked.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: keep collection members registered when destroy fails

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

weed/shell/command_volume_tier_compact_test.go

@@ -5,6 +5,7 @@ import (
 
 	"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
 	"github.com/seaweedfs/seaweedfs/weed/storage/needle"
+	"github.com/seaweedfs/seaweedfs/weed/wdclient"
 	"github.com/stretchr/testify/assert"
 )
 
@@ -253,3 +254,89 @@ func TestCollectRemoteVolumesWithInfoInvalidPattern(t *testing.T) {
 	_, err := collectRemoteVolumesWithInfo(topo, "[invalid")
 	assert.Error(t, err)
 }
+
+func buildTestTopologyWithReplicas() *master_pb.TopologyInfo {
+	mkDn := func(id string, volumes ...*master_pb.VolumeInformationMessage) *master_pb.DataNodeInfo {
+		return &master_pb.DataNodeInfo{
+			Id:       id,
+			GrpcPort: 18080,
+			DiskInfos: map[string]*master_pb.DiskInfo{
+				"": {VolumeInfos: volumes},
+			},
+		}
+	}
+	remote := func(vid uint32) *master_pb.VolumeInformationMessage {
+		return &master_pb.VolumeInformationMessage{
+			Id:                vid,
+			Collection:        "col1",
+			RemoteStorageName: "s3.default",
+			RemoteStorageKey:  "col1/vid.dat",
+		}
+	}
+	local := func(vid uint32) *master_pb.VolumeInformationMessage {
+		return &master_pb.VolumeInformationMessage{Id: vid, Collection: "col1"}
+	}
+	return &master_pb.TopologyInfo{
+		DataCenterInfos: []*master_pb.DataCenterInfo{
+			{
+				RackInfos: []*master_pb.RackInfo{
+					{
+						DataNodeInfos: []*master_pb.DataNodeInfo{
+							mkDn("n1:8080", remote(1)),
+							mkDn("n2:8080", remote(1), local(2)),
+							mkDn("n3:8080", remote(1)),
+						},
+					},
+				},
+			},
+		},
+	}
+}
+
+func TestCollectVolumeTierCompactLocations(t *testing.T) {
+	topo := buildTestTopologyWithReplicas()
+
+	tiered, local := collectVolumeTierCompactLocations(topo, needle.VolumeId(1), "col1")
+	assert.Equal(t, 3, len(tiered))
+	assert.Equal(t, 0, len(local))
+
+	tiered, local = collectVolumeTierCompactLocations(topo, needle.VolumeId(2), "col1")
+	assert.Equal(t, 0, len(tiered))
+	assert.Equal(t, 1, len(local))
+
+	// a different collection or volume id matches nothing
+	tiered, local = collectVolumeTierCompactLocations(topo, needle.VolumeId(1), "other")
+	assert.Equal(t, 0, len(tiered))
+	assert.Equal(t, 0, len(local))
+}
+
+func TestPlanTierCompactResync(t *testing.T) {
+	loc := func(url string) wdclient.Location {
+		return wdclient.Location{Url: url, GrpcPort: 1}
+	}
+	urls := func(locs []wdclient.Location) (out []string) {
+		for _, l := range locs {
+			out = append(out, l.Url)
+		}
+		return
+	}
+
+	// single replica: nothing to resync, nothing to delete (its download
+	// already removed the object)
+	order, deleter := planTierCompactResync([]wdclient.Location{loc("a")}, nil)
+	assert.Empty(t, order)
+	assert.Empty(t, deleter)
+
+	// two tiered replicas: the sibling is the last holder and goes last
+	order, deleter = planTierCompactResync([]wdclient.Location{loc("a"), loc("b")}, nil)
+	assert.Empty(t, order)
+	assert.Equal(t, []string{"b"}, urls(deleter))
+
+	// three tiered replicas plus one local copy: middle replicas resync in
+	// order, the last tiered replica deletes and is resynced last
+	order, deleter = planTierCompactResync(
+		[]wdclient.Location{loc("a"), loc("b"), loc("c")},
+		[]wdclient.Location{loc("d")})
+	assert.Equal(t, []string{"b", "d"}, urls(order))
+	assert.Equal(t, []string{"c"}, urls(deleter))
+}