fix(ec): preserve source disk type across EC encoding (#9423) (#9449)

* fix(ec): carry source disk type on VolumeEcShardsMount (#9423)

When EC shards land on a target whose disk type differs from the
source volume's, master heartbeats wrongly reported under the target
disk's type. Add source_disk_type to VolumeEcShardsMountRequest; the
target server applies it to the in-memory EcVolume via SetDiskType so
the mount notification and steady-state heartbeat both carry the
source's disk type. Empty value falls back to the location's disk
type (used by disk-scan reload paths).

The override is not persisted with the volume — disk type stays an
environmental property and .vif remains portable.

* fix(ec): plumb source disk type through plugin worker (#9423)

Add source_disk_type to ErasureCodingTaskParams (field 8; 7 reserved),
populate it from the metric the detector already collects, thread it
through ec_task into the MountEcShards helper, and forward it on the
VolumeEcShardsMount RPC.

* fix(ec): mirror source disk type plumbing in rust volume server (#9423)

The volume_ec_shards_mount handler now forwards source_disk_type into
mount_ec_shard → DiskLocation::mount_ec_shards. When non-empty it
overrides ec_vol.disk_type (and each mounted shard's disk_type) via
the new set_disk_type method; empty value keeps the location's disk
type, so disk-scan reload and reconcile paths are unchanged.

Also picks up two pre-existing proto drifts that 'make gen' synced
from weed/pb (LockRingUpdate in master.proto, listing_cache_ttl_seconds
in remote.proto).

* feat(ec): bias placement toward preferred disk type (#9423)

Add DiskCandidate.DiskType and PlacementRequest.PreferredDiskType.
When PreferredDiskType is non-empty, SelectDestinations partitions
suitable disks into matching/fallback tiers and runs the rack/server/
disk-diversity passes on the matching tier first; the fallback tier
is only consulted if the matching pool can't satisfy ShardsNeeded.
PlacementResult.SpilledToOtherDiskType lets callers warn on spillover.

Empty PreferredDiskType keeps the existing single-pool behavior.

* fix(ec): plumb source disk type into placement planner (#9423)

diskInfosToCandidates now copies DiskInfo.DiskType into the placement
candidate, and ecPlacementPlanner.selectDestinations forwards
metric.DiskType as PreferredDiskType so EC shards land on disks
matching the source volume's disk type when possible. A glog warning
fires when placement had to spill to other disk types.

* test(ec): integration coverage for source-disk-type plumbing (#9423)

store_ec_disk_type_test exercises Store.MountEcShards end-to-end: a
shard physically lives on an HDD location, MountEcShards is called
with sourceDiskType="ssd", and the test asserts that the in-memory
EcVolume, the mounted shard, the NewEcShardsChan notification, and
the steady-state heartbeat all report under the source's disk type.
A companion test pins the empty-source path so disk-scan reload
keeps the location's disk type.

detection_disk_type_test exercises the worker plumbing: with a
cluster of nodes carrying both HDD and SSD disks, planECDestinations
must place every shard on SSD when metric.DiskType="ssd"; with only
one SSD node and 13 HDD nodes it must still satisfy a 10+4 layout
via spillover (and log a warning).

* revert(ec): drop unrelated proto drift in seaweed-volume/proto (#9423)

make gen pulled two pre-existing OSS changes into the rust proto
tree (LockRingUpdate / by_plugin in master.proto,
listing_cache_ttl_seconds in remote.proto). Reviewers flagged it as
scope creep — none of the rust EC fix references those fields.
Restore both files to origin/master so this branch only touches
EC-related symbols.

* fix(ec placement): treat empty disk type as hdd and skip used racks on spill (#9423)

partitionByDiskType used raw string comparison, so a PreferredDiskType
of "hdd" never matched candidates whose DiskType is "" (the
HardDriveType sentinel that weed/storage/types uses). EC encoding of
an HDD source would spill onto any HDD reporting "" even when the
cluster has plenty of matching capacity. Normalize both sides
through normalizeDiskType, which lowercases and folds "" → "hdd",
mirroring types.ToDiskType without taking a dependency on it.

selectFromTier's rack-diversity pass also kept revisiting racks the
preferred tier had already used when running on the fallback tier,
which negated PreferDifferentRacks on spillover. Skip racks already
in usedRacks so fallback placements still spread onto new racks.

* fix(ec): empty-source remount must not clobber existing disk type (#9423)

mount_ec_shards_with_idx_dir runs more than once per vid (RPC mount,
disk-scan reload, orphan-shard reconcile). After an RPC sets the
source-derived disk type, any later call passing source_disk_type=""
was resetting ec_vol.disk_type back to the location's value, which
reintroduces the heartbeat drift this PR is meant to fix. Only
default to the location's disk type when the EC volume is fresh
(no shards mounted yet); otherwise leave the recorded type alone so
empty-source reloads preserve whatever the original mount RPC set.

weed/storage/erasure_coding/ec_volume.go

@@ -246,6 +246,24 @@ func (ev *EcVolume) Destroy() {
 	os.Remove(ev.FileName(".vif"))
 }
 
+// DiskType returns the disk type the EC volume currently reports under.
+// Defaults to the physical location's disk type; orchestrators can override
+// it via SetDiskType so the volume keeps reporting under the source
+// volume's disk type after encoding (#9423).
+func (ev *EcVolume) DiskType() types.DiskType {
+	return ev.diskType
+}
+
+// SetDiskType overrides the EC volume's reported disk type and propagates
+// to its mounted shards. Intended for the orchestrator-driven mount path
+// (VolumeEcShardsMount); not persisted across restarts.
+func (ev *EcVolume) SetDiskType(d types.DiskType) {
+	ev.diskType = d
+	for _, s := range ev.Shards {
+		s.DiskType = d
+	}
+}
+
 func (ev *EcVolume) FileName(ext string) string {
 	switch ext {
 	case ".ecx", ".ecj":

weed/storage/erasure_coding/placement/placement.go

@@ -10,6 +10,7 @@ package placement
 import (
 	"fmt"
 	"sort"
+	"strings"
 )
 
 // DiskCandidate represents a disk that can receive EC shards
@@ -18,6 +19,7 @@ type DiskCandidate struct {
 	DiskID     uint32
 	DataCenter string
 	Rack       string
+	DiskType   string // disk type (hdd/ssd/...) — empty means HardDrive
 
 	// Capacity information
 	VolumeCount    int64
@@ -62,6 +64,13 @@ type PlacementRequest struct {
 	// PreferDifferentRacks when true, spreads shards across different racks
 	// before using multiple servers in the same rack
 	PreferDifferentRacks bool
+
+	// PreferredDiskType, when non-empty, biases placement toward disks of
+	// this type. Disks of the preferred type are exhausted (subject to the
+	// other diversity preferences) before disks of any other type are
+	// considered. Empty means no disk-type bias — all suitable disks form a
+	// single pool, matching pre-#9423 behavior.
+	PreferredDiskType string
 }
 
 // PlacementResult contains the selected destinations for EC shards
@@ -77,12 +86,25 @@ type PlacementResult struct {
 	ShardsPerServer map[string]int
 	ShardsPerRack   map[string]int
 	ShardsPerDC     map[string]int
+
+	// SpilledToOtherDiskType is set when PlacementRequest.PreferredDiskType
+	// was non-empty but the preferred-type pool could not satisfy
+	// ShardsNeeded, so placement had to spill onto disks of other types.
+	// Callers can log a warning when this is true.
+	SpilledToOtherDiskType bool
 }
 
 // SelectDestinations selects the best disks for EC shard placement.
 // This is the main entry point for EC placement logic.
 //
-// The algorithm works in multiple passes:
+// Disk-type preference (#9423): when config.PreferredDiskType is non-empty,
+// suitable disks are partitioned into a matching-type tier and a
+// fallback tier. Each tier is run through the diversity passes below;
+// the fallback tier is only consulted if the matching tier runs out of
+// candidates before ShardsNeeded is satisfied. Empty PreferredDiskType
+// processes all suitable disks as one tier, preserving prior behavior.
+//
+// Within each tier, the algorithm works in multiple passes:
 // 1. First pass: Select one disk from each rack (maximize rack diversity)
 // 2. Second pass: Select one disk from each unused server in used racks (maximize server diversity)
 // 3. Third pass: Select additional disks from servers already used (maximize disk diversity)
@@ -100,9 +122,6 @@ func SelectDestinations(disks []*DiskCandidate, config PlacementRequest) (*Place
 		return nil, fmt.Errorf("no suitable disks found after filtering")
 	}
 
-	// Build indexes for efficient lookup
-	rackToDisks := groupDisksByRack(suitable)
-
 	result := &PlacementResult{
 		SelectedDisks:   make([]*DiskCandidate, 0, config.ShardsNeeded),
 		ShardsPerServer: make(map[string]int),
@@ -114,7 +133,83 @@ func SelectDestinations(disks []*DiskCandidate, config PlacementRequest) (*Place
 	usedServers := make(map[string]bool) // nodeID -> bool
 	usedRacks := make(map[string]bool)   // "dc:rack" -> bool
 
-	// Pass 1: Select one disk from each rack (maximize rack diversity)
+	// Partition suitable into preferred-disk-type / fallback tiers.
+	// Process the preferred tier first; only spill to fallback when the
+	// preferred pool can't satisfy ShardsNeeded.
+	preferredTier, fallbackTier := partitionByDiskType(suitable, config.PreferredDiskType)
+	selectFromTier(preferredTier, result, usedDisks, usedServers, usedRacks, config)
+	if config.PreferredDiskType != "" && len(result.SelectedDisks) < config.ShardsNeeded && len(fallbackTier) > 0 {
+		before := len(result.SelectedDisks)
+		selectFromTier(fallbackTier, result, usedDisks, usedServers, usedRacks, config)
+		if len(result.SelectedDisks) > before {
+			result.SpilledToOtherDiskType = true
+		}
+	}
+
+	// Calculate final statistics
+	result.ServersUsed = len(usedServers)
+	result.RacksUsed = len(usedRacks)
+	dcSet := make(map[string]bool)
+	for _, disk := range result.SelectedDisks {
+		dcSet[disk.DataCenter] = true
+	}
+	result.DCsUsed = len(dcSet)
+
+	return result, nil
+}
+
+// partitionByDiskType splits disks into (matching, fallback) based on the
+// preferred disk type. If preferred is empty, everything goes into the
+// matching tier and fallback is empty — i.e. existing single-pool behavior.
+//
+// Empty DiskCandidate.DiskType is treated as HardDriveType ("hdd") to
+// mirror weed/storage/types.ToDiskType's normalization, so a
+// PreferredDiskType of "hdd" matches disks reporting "" — otherwise EC
+// shards from an HDD source would always spill onto disks that happen to
+// report their type as "" (HardDriveType).
+func partitionByDiskType(disks []*DiskCandidate, preferred string) (matching, fallback []*DiskCandidate) {
+	if preferred == "" {
+		return disks, nil
+	}
+	pref := normalizeDiskType(preferred)
+	for _, d := range disks {
+		if normalizeDiskType(d.DiskType) == pref {
+			matching = append(matching, d)
+		} else {
+			fallback = append(fallback, d)
+		}
+	}
+	return matching, fallback
+}
+
+// normalizeDiskType lower-cases the input and folds "" to "hdd" so the
+// HardDriveType sentinel ("") and explicit "hdd"/"HDD" all compare equal.
+func normalizeDiskType(t string) string {
+	t = strings.ToLower(t)
+	if t == "" {
+		return "hdd"
+	}
+	return t
+}
+
+// selectFromTier runs the three diversity passes against `tier`, mutating
+// `result` and the used* maps in place. Passes stop as soon as ShardsNeeded
+// is reached. The function is a no-op when the tier is empty or the result
+// already has enough shards, so it is safe to call once per tier.
+func selectFromTier(tier []*DiskCandidate, result *PlacementResult,
+	usedDisks, usedServers, usedRacks map[string]bool,
+	config PlacementRequest) {
+
+	if len(tier) == 0 || len(result.SelectedDisks) >= config.ShardsNeeded {
+		return
+	}
+
+	rackToDisks := groupDisksByRack(tier)
+
+	// Pass 1: Select one disk from each rack (maximize rack diversity).
+	// When this is the fallback tier (preferred tier already populated
+	// usedRacks), skip those racks so the spillover still spreads onto
+	// new racks instead of doubling up on ones already picked.
 	if config.PreferDifferentRacks {
 		// Sort racks by number of available servers (descending) to prioritize racks with more options
 		sortedRacks := sortRacksByServerCount(rackToDisks)
@@ -122,6 +217,9 @@ func SelectDestinations(disks []*DiskCandidate, config PlacementRequest) (*Place
 			if len(result.SelectedDisks) >= config.ShardsNeeded {
 				break
 			}
+			if usedRacks[rackKey] {
+				continue
+			}
 			rackDisks := rackToDisks[rackKey]
 			// Select best disk from this rack, preferring a new server
 			disk := selectBestDiskFromRack(rackDisks, usedServers, usedDisks, config)
@@ -166,9 +264,9 @@ func SelectDestinations(disks []*DiskCandidate, config PlacementRequest) (*Place
 	// Pass 3: Fill remaining slots from already-used servers (different disks)
 	// Use round-robin across servers to balance shards evenly
 	if len(result.SelectedDisks) < config.ShardsNeeded {
-		// Group remaining disks by server
+		// Group remaining disks by server (within this tier)
 		serverToRemainingDisks := make(map[string][]*DiskCandidate)
-		for _, disk := range suitable {
+		for _, disk := range tier {
 			if !usedDisks[getDiskKey(disk)] {
 				serverToRemainingDisks[disk.NodeID] = append(serverToRemainingDisks[disk.NodeID], disk)
 			}
@@ -220,17 +318,6 @@ func SelectDestinations(disks []*DiskCandidate, config PlacementRequest) (*Place
 			addDiskToResult(result, disk, usedDisks, usedServers, usedRacks)
 		}
 	}
-
-	// Calculate final statistics
-	result.ServersUsed = len(usedServers)
-	result.RacksUsed = len(usedRacks)
-	dcSet := make(map[string]bool)
-	for _, disk := range result.SelectedDisks {
-		dcSet[disk.DataCenter] = true
-	}
-	result.DCsUsed = len(dcSet)
-
-	return result, nil
 }
 
 // filterSuitableDisks filters disks that are suitable for EC placement

weed/storage/erasure_coding/placement/placement_test.go

@@ -0,0 +1,143 @@
+package placement
+
+import (
+	"strconv"
+	"testing"
+)
+
+// makeDisk builds a DiskCandidate with sensible defaults; tests override
+// only the fields they care about.
+func makeDisk(node, rack, diskType string, diskID uint32) *DiskCandidate {
+	return &DiskCandidate{
+		NodeID:         node,
+		DiskID:         diskID,
+		DataCenter:     "dc1",
+		Rack:           rack,
+		DiskType:       diskType,
+		VolumeCount:    0,
+		MaxVolumeCount: 100,
+		FreeSlots:      100,
+	}
+}
+
+func disksByType(disks []*DiskCandidate) map[string]int {
+	out := map[string]int{}
+	for _, d := range disks {
+		out[d.DiskType]++
+	}
+	return out
+}
+
+func newRequest(shards int, preferred string) PlacementRequest {
+	return PlacementRequest{
+		ShardsNeeded:           shards,
+		PreferDifferentServers: true,
+		PreferDifferentRacks:   true,
+		PreferredDiskType:      preferred,
+	}
+}
+
+// Plenty of SSD disks available: placement should fill entirely from SSD
+// when PreferredDiskType="ssd", leaving HDDs untouched and not flagging
+// spillover.
+func TestSelectDestinations_PrefersMatchingDiskType(t *testing.T) {
+	var disks []*DiskCandidate
+	for i := 0; i < 6; i++ {
+		disks = append(disks, makeDisk("ssd-"+strconv.Itoa(i), "r"+strconv.Itoa(i%3), "ssd", uint32(i)))
+	}
+	for i := 0; i < 6; i++ {
+		disks = append(disks, makeDisk("hdd-"+strconv.Itoa(i), "r"+strconv.Itoa(i%3), "", uint32(i)))
+	}
+
+	result, err := SelectDestinations(disks, newRequest(4, "ssd"))
+	if err != nil {
+		t.Fatalf("SelectDestinations: %v", err)
+	}
+	if got := len(result.SelectedDisks); got != 4 {
+		t.Fatalf("selected %d disks, want 4", got)
+	}
+	if counts := disksByType(result.SelectedDisks); counts["ssd"] != 4 {
+		t.Fatalf("disk-type counts = %v, want ssd=4", counts)
+	}
+	if result.SpilledToOtherDiskType {
+		t.Fatalf("SpilledToOtherDiskType should be false when preferred pool was sufficient")
+	}
+}
+
+// Only one SSD disk available but 4 shards needed: placement must consume
+// the SSD first, then spill to HDD for the remainder, and report spillover.
+func TestSelectDestinations_SpillsWhenPreferredScarce(t *testing.T) {
+	disks := []*DiskCandidate{
+		makeDisk("ssd-0", "r0", "ssd", 0),
+		makeDisk("hdd-0", "r1", "", 0),
+		makeDisk("hdd-1", "r2", "", 0),
+		makeDisk("hdd-2", "r3", "", 0),
+	}
+
+	result, err := SelectDestinations(disks, newRequest(4, "ssd"))
+	if err != nil {
+		t.Fatalf("SelectDestinations: %v", err)
+	}
+	if got := len(result.SelectedDisks); got != 4 {
+		t.Fatalf("selected %d disks, want 4", got)
+	}
+	counts := disksByType(result.SelectedDisks)
+	if counts["ssd"] != 1 || counts[""] != 3 {
+		t.Fatalf("disk-type counts = %v, want ssd=1 hdd=3", counts)
+	}
+	if !result.SpilledToOtherDiskType {
+		t.Fatalf("SpilledToOtherDiskType should be true after falling back to HDD")
+	}
+}
+
+// PreferredDiskType="hdd" must match disks whose DiskType is "" (the
+// HardDriveType sentinel) — otherwise EC encoding of an HDD source would
+// always spill onto HDDs that happen to report disk_type="" even though
+// the cluster has plenty of matching capacity.
+func TestSelectDestinations_PreferredHddMatchesEmptyDiskType(t *testing.T) {
+	disks := []*DiskCandidate{
+		makeDisk("hdd-0", "r0", "", 0),  // HardDriveType sentinel
+		makeDisk("hdd-1", "r1", "", 0),  // HardDriveType sentinel
+		makeDisk("ssd-0", "r2", "ssd", 0),
+	}
+
+	result, err := SelectDestinations(disks, newRequest(2, "hdd"))
+	if err != nil {
+		t.Fatalf("SelectDestinations: %v", err)
+	}
+	if got := len(result.SelectedDisks); got != 2 {
+		t.Fatalf("selected %d disks, want 2", got)
+	}
+	// Both selected disks must be HDD-reporting (i.e. DiskType == ""),
+	// and no spillover should have been required.
+	for _, d := range result.SelectedDisks {
+		if d.DiskType != "" {
+			t.Errorf("selected disk %s has DiskType=%q, want \"\" (HardDriveType)", d.NodeID, d.DiskType)
+		}
+	}
+	if result.SpilledToOtherDiskType {
+		t.Fatalf("SpilledToOtherDiskType should be false when HDD pool matches preferred=hdd")
+	}
+}
+
+// Empty PreferredDiskType: pre-#9423 behavior, single pool, no spillover
+// flag regardless of disk-type mix.
+func TestSelectDestinations_EmptyPreferredDiskTypeKeepsPriorBehavior(t *testing.T) {
+	disks := []*DiskCandidate{
+		makeDisk("ssd-0", "r0", "ssd", 0),
+		makeDisk("hdd-0", "r1", "", 0),
+		makeDisk("hdd-1", "r2", "", 0),
+		makeDisk("ssd-1", "r3", "ssd", 0),
+	}
+
+	result, err := SelectDestinations(disks, newRequest(3, ""))
+	if err != nil {
+		t.Fatalf("SelectDestinations: %v", err)
+	}
+	if got := len(result.SelectedDisks); got != 3 {
+		t.Fatalf("selected %d disks, want 3", got)
+	}
+	if result.SpilledToOtherDiskType {
+		t.Fatalf("SpilledToOtherDiskType should never be set when PreferredDiskType is empty")
+	}
+}

weed/storage/erasure_coding/shard_distribution.go

@@ -222,7 +222,9 @@ func DistributeEcShards(volumeID uint32, collection string, targets []*worker_pb
 }
 
 // MountEcShards mounts EC shards on destination servers using an assignment map.
-func MountEcShards(volumeID uint32, collection string, shardAssignment map[string][]string, dialOption grpc.DialOption, logger logger) error {
+// sourceDiskType is forwarded to VolumeEcShardsMount so the resulting EC volume
+// reports under the source's disk type rather than the destination's (#9423).
+func MountEcShards(volumeID uint32, collection string, shardAssignment map[string][]string, sourceDiskType string, dialOption grpc.DialOption, logger logger) error {
 	if shardAssignment == nil {
 		return fmt.Errorf("shard assignment not available for mounting")
 	}
@@ -263,9 +265,10 @@ func MountEcShards(volumeID uint32, collection string, shardAssignment map[strin
 		err := operation.WithVolumeServerClient(false, pb.ServerAddress(destNode), dialOption,
 			func(client volume_server_pb.VolumeServerClient) error {
 				_, mountErr := client.VolumeEcShardsMount(context.Background(), &volume_server_pb.VolumeEcShardsMountRequest{
-					VolumeId:   volumeID,
-					Collection: collection,
-					ShardIds:   shardIds,
+					VolumeId:       volumeID,
+					Collection:     collection,
+					ShardIds:       shardIds,
+					SourceDiskType: sourceDiskType,
 				})
 				return mountErr
 			})