volume: drop stale volume-location cache on under-replication (#10185)

* volume: drop stale volume-location cache on under-replication

A replicated write looks up the volume's locations and caches them for 10
minutes. When the master briefly reports fewer replicas than the copy count
(e.g. a stale heartbeat drops a just-added volume), that under-replicated
result got cached, so every write failed with "replicating operations is less
than replication copy count" until the entry expired -- long after the master
re-registered the replica.

Invalidate the cached entry when the location count is below the copy count, so
the next write re-queries the master and recovers as soon as it heals.

* volume: mirror the replication copy-count guard in seaweed-volume

do_replicated_request accepted a write even when the master reported fewer
locations than the volume's copy count, silently under-replicating. Reject it,
matching Go's GetWritableRemoteReplications. lookup_volume is uncached, so the
next write recovers as soon as the missing replica re-registers.

seaweed-volume/src/server/handlers.rs

@@ -546,6 +546,23 @@ async fn do_replicated_request(
     .await
     .map_err(|e| format!("lookup volume failed: {}", e))?;
 
+    // Mirror Go's GetWritableRemoteReplications: reject when the master reports fewer replicas than
+    // the copy count. lookup_volume is uncached, so recovery is immediate once the replica re-registers.
+    let copy_count = {
+        let store = state.store.read().unwrap();
+        store.find_volume(VolumeId(vid)).map_or(1, |(_, v)| {
+            v.super_block.replica_placement.get_copy_count()
+        })
+    };
+    if locations.len() < copy_count as usize {
+        return Err(format!(
+            "replicating operations [{}] is less than volume {} replication copy count [{}]",
+            locations.len(),
+            vid,
+            copy_count
+        ));
+    }
+
     let self_http = to_http_address(&state.self_url);
     let remote_locations: Vec<_> = locations
         .into_iter()