Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-11-09 05:18:23 +01:00
Code Issues Packages Projects Releases Wiki Activity

ec.rebuild: remount shards the rebuild replaced, keep regenerated copies (#12193)

Browse Source 
* volume: remount ec shards the rebuild replaced under a mounted handle

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec.rebuild: keep copied shards the rebuild regenerated in place

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: reopen regenerated ec shards in place instead of remounting

UnmountEcShards and MountEcShards send deltas through separate buffered
channels, so a remount can reach the master as add-then-delete and drop
the repaired shard from lookups until the next full EC heartbeat.
ReopenEcShard swaps the mounted shard's descriptor under the volume lock
instead: registration, disk type, and heartbeat state never flip, and
readers holding the old shard degrade to a clean error the parity path
already recovers from.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: reopen rebuilt ec shards in place, mirroring Go

volume_ec_shards_rebuild recreates a shard file under a mounted handle
whose descriptor still serves the detached inode (a mounted shard whose
file was deleted under it). DiskLocation::reopen_ec_shard swaps the
mounted descriptor for a fresh open, matching Go's ReopenEcShard — the
registration and heartbeat view never flip.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: fail ec rebuild when a regenerated shard cannot be reopened

* rust volume: fail ec shards rebuild when a rebuilt shard cannot be reopened

* volume: repoint mounted ec shards after rebuild across disks

A mounted shard whose file was deleted under the mount is "missing" to
the rebuild: the regenerated file lands on the rebuild location while
the handle keeps serving the detached inode on another disk. Reopening
the mounted disk then failed the whole RPC, so the caller never saw the
rebuilt ids and its deferred cleanup deleted the regenerated copies.

Repoint every mounted handle of the rebuilt volume instead of only this
run's outputs: reopen in place when the rebuilt file is on the mounted
disk, otherwise drop the stale handle and remount the shard from
whichever local disk holds the file via a delta-free load split out of
MountEcShards — a same-server move, so the master's registration never
flips. A shard that can be neither reopened nor remounted is left
unmounted rather than serving replaced bytes, and the repoint step no
longer fails the RPC, so RebuiltShardIds always reaches the caller.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: repoint mounted ec shards after rebuild across disks

Mirror the Go repoint loop: iterate every mounted shard of the rebuilt
volume, reopen in place when the rebuilt file is on the mounted disk,
and otherwise drop the stale handle and remount the shard from
whichever local disk holds it. The repoint no longer fails the RPC, so
rebuilt_shard_ids still reaches the caller and its copy cleanup keeps
the regenerated shards.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: still rebuild ecx and repoint mounts when no shard is missing

* master: keep a node in an ec shard's lookup while a disk still holds it

A rebuild that relocates a mounted shard between a node's disks reports
the move as a delete on the source disk plus an add on the destination.
Registering first is a no-op because shard lookups key on the node, then
the unregister dropped the node outright — and steady-state heartbeats
never restore it, so readers and later rebuilds lose the repaired copy.
Skip the delete while another disk on the node still holds the shard.

UnregisterDataNode now clears the node's own ec records before dropping
its registrations, so the retained-shard check cannot skip the teardown,
and each unregister gets a ShardsInfo copy since DeleteEcShard drains
the shared one.

* rust volume: fold the journal on a mounted index instead of rewriting it

The rebuild handler regenerated .ecx through File::create even while the
volume was mounted, truncating the index under readers that search the
open descriptor with its old size — a live lookup could hit EOF mid-scan.
Go folds the deletion journal in place on a mounted volume; do the same
through rebuild_ecx_from_journal and reserve the full regen for the
unmounted path, where nothing holds the file open to interrupt.

* volume, rust volume: report rebuilt shards on late failures, fix rust corrupt discovery

Two review findings the repoint rework left open:

- A RebuildEcxFile failure still returns a plain error after shards were
  regenerated, so the caller discards the response holding the rebuilt
  ids and the deferred cleanup deletes copies the rebuild regenerated
  in place. The Go handler now packs the ids into the gRPC status
  details and the shell pulls them back out on error; the Rust handler
  carries them in grpc-status-details-bin the same way.
- The Rust rebuild still counted a sidecar-corrupt shard as present, so
  it fed bad bytes into reconstruction and never regenerated the shard.
  Discovery now loads the generation-0 .ecsum the way Go does, verifies
  every present shard, and reclassifies mismatches (and zero-size
  residue) as rebuild targets written to a .rebuilding temp file,
  verified, and atomically renamed over the original only after every
  output passes the sidecar check. Honors unsafe_ignore_sidecar, added
  to VolumeEcShardsRebuildRequest to match the Go proto.

* volume: repoint sibling journals sharing a folded .ecj after rebuild

RebuildEcxFile unlinks the deletion journal and the mounted path only
reopens its own handle — a sibling mount journaling into the same file
keeps appending to the detached inode, and those deletes never reach a
path-based reader again. Reopen every mounted volume sharing the index
base before leaving the rebuild.

* rust volume: repoint sibling journals sharing a folded .ecj after rebuild

rebuild_ecx_from_journal replaces the journal and reopens only its own
handle; mounts on sibling disks sharing that .ecj keep the detached
handle, so their next appends are lost at restart. Adopt the folded
journal on every mounted runtime sharing the path before releasing the
store lock, clearing their in-memory set — every journaled id is now an
.ecx tombstone.

* ec: swap the deletion journal handle only after the new file opens

A failed reopen left ecjFile nil, so every later delete returned
"ec volume closed". Open first, then close and swap — mirroring the
Rust adopt_folded_journal ordering.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec.rebuild: fail the rebuild when a sibling journal cannot be repointed

A warning left the sibling mount journaling to a detached inode while
the RPC reported success. Propagate the error like the Rust handler's
adopt_folded_journal map_err.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec: keep deletes loud when the journal reopen cannot open the replacement

Reverts the swap-on-success ordering: a detached handle kept on failure
lets deletes append and sync to the removed inode, then vanish at
restart. Closing first also frees the descriptor before the open.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: drop the folded journal handle before reopening it

adopt_folded_journal and the post-fold recreate now clear ecj_file
before opening the replacement, so a failed reopen leaves deletes
erroring instead of appending to the detached inode. Mirrors Go's
ReopenDeletionJournal.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec: drop the cached journal size when its reopen fails

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
Chris LuandDevin authored and GitHub committed 2026-10-29 00:58:30 -07:00
1 parent 5c92a54a12
commit 2c510cc30e
17 files changed
+1202 -110
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

seaweed-volume/src/server/grpc_server.rs
+337 -55
View File
@@ -16,6 +16,7 @@ use crate::pb::filer_pb;
use crate::pb::master_pb;
use crate::pb::volume_server_pb;
use crate::pb::volume_server_pb::volume_server_server::VolumeServer;
use crate::storage::erasure_coding::ec_bitrot;
use crate::storage::erasure_coding::ec_shard::{ShardId, shard_id_try_from};
use crate::storage::erasure_coding::ecj_merge::EcjIdDecoder;
use crate::storage::needle::needle::{self, Needle};
@@ -61,6 +62,42 @@ fn is_not_leader_status(e: &Status) -> bool {
e.message().contains("Not current leader")
}
/// A gRPC error that still carries the ids of shards the rebuild wrote before
/// failing. The Go shell pulls VolumeEcShardsRebuildResponse out of the status
/// details so its deferred cleanup keeps the copies the rebuild regenerated
/// in place instead of deleting them as orphans — a plain error drops the
/// response holding them. `details` is the serialized google.rpc.Status that
/// the grpc-status-details-bin trailer carries.
fn status_internal_with_rebuild_ids(message: String, rebuilt_shard_ids: &[u32]) -> Status {
#[derive(Clone, PartialEq, prost::Message)]
struct RpcStatus {
#[prost(int32, tag = "1")]
code: i32,
#[prost(string, tag = "2")]
message: String,
#[prost(message, repeated, tag = "3")]
details: Vec<prost_types::Any>,
}
use prost::Message;
let resp = volume_server_pb::VolumeEcShardsRebuildResponse {
rebuilt_shard_ids: rebuilt_shard_ids.to_vec(),
};
let status = RpcStatus {
code: tonic::Code::Internal as i32,
message: message.clone(),
details: vec![prost_types::Any {
type_url: "type.googleapis.com/volume_server_pb.VolumeEcShardsRebuildResponse"
.to_string(),
value: resp.encode_to_vec(),
}],
};
Status::with_details(
tonic::Code::Internal,
message,
status.encode_to_vec().into(),
)
}
/// Deadline from the caller's `grpc-timeout` header (TimeoutValue +
/// H|M|S|m|u|n unit, per the gRPC-over-HTTP2 spec). Tonic's GrpcTimeout layer
/// drops an over-deadline handler future anyway; parsing it here lets a queued
@@ -3197,63 +3234,215 @@ impl VolumeServer for VolumeGrpcService {
})?;
let total_shards = data_shards + parity_shards;
// Check which shards are missing (check rebuild dir and all other dirs)
let mut missing: Vec<u32> = Vec::new();
for shard_id in 0..total_shards as u8 {
let shard = crate::storage::erasure_coding::ec_shard::EcVolumeShard::new(
&rebuild_dir,
collection,
vid,
shard_id,
);
let mut found = std::path::Path::new(&shard.file_name()).exists();
if !found {
for other_dir in &other_dirs {
let other_shard = crate::storage::erasure_coding::ec_shard::EcVolumeShard::new(
other_dir, collection, vid, shard_id,
);
if std::path::Path::new(&other_shard.file_name()).exists() {
found = true;
break;
// Discover shard files like Go's findShardFile: the rebuild's own data
// and index directories first, then every sibling directory. Track
// each present shard's path — a shard the sidecar reclassifies as
// corrupt is replaced at ITS discovered path, not at the rebuild dir.
let shard_file_path = |dir: &str, shard_id: ShardId| -> String {
crate::storage::erasure_coding::ec_shard::EcVolumeShard::new(
dir, collection, vid, shard_id,
)
.file_name()
};
let mut shard_dirs: Vec<String> = vec![rebuild_dir.clone()];
if !rebuild_idx_dir.is_empty() && rebuild_idx_dir != rebuild_dir {
shard_dirs.push(rebuild_idx_dir.clone());
}
shard_dirs.extend(other_dirs.iter().cloned());
let mut found: Vec<Option<String>> = vec![None; total_shards as usize];
for shard_id in 0..total_shards as ShardId {
found[shard_id as usize] = shard_dirs
.iter()
.map(|d| shard_file_path(d, shard_id))
.find(|p| std::path::Path::new(p).exists());
}
// Load the generation-0 checksum sidecar the way Go's
// loadRebuildSidecar does: absent, wrong-generation, or recording no
// geometry => Off; a self-integrity, geometry, or manifest failure =>
// Invalid; otherwise On.
let data_base = crate::storage::volume::volume_file_name(&rebuild_dir, collection, vid);
let idx_base = crate::storage::volume::volume_file_name(&rebuild_idx_dir, collection, vid);
let sidecar_dir_refs: Vec<&str> = other_dirs.iter().map(|s| s.as_str()).collect();
let sidecar_path =
ec_bitrot::find_bitrot_sidecar(0, &data_base, &idx_base, &sidecar_dir_refs);
let (prot, bitrot_status) = if sidecar_path.is_empty() {
(None, ec_bitrot::BitrotStatus::Off)
} else {
match ec_bitrot::load_bitrot_sidecar(&sidecar_path) {
Err(e) => {
tracing::warn!(path = %sidecar_path, error = %e, "bitrot: sidecar self-integrity failed");
(None, ec_bitrot::BitrotStatus::Invalid)
}
Ok(prot) => {
if prot.generation != 0 || prot.ec_shard_config.is_none() {
(None, ec_bitrot::BitrotStatus::Off)
} else if !ec_bitrot::geometry_matches(
&prot,
data_shards as usize,
parity_shards as usize,
block_size,
) {
tracing::warn!(path = %sidecar_path, "bitrot: sidecar records a different layout than the rebuild uses");
(None, ec_bitrot::BitrotStatus::Invalid)
} else if let Err(e) = ec_bitrot::validate_manifest(
&prot,
data_shards as usize,
parity_shards as usize,
) {
tracing::warn!(path = %sidecar_path, error = %e, "bitrot: sidecar manifest invalid");
(None, ec_bitrot::BitrotStatus::Invalid)
} else {
(Some(prot), ec_bitrot::BitrotStatus::On)
}
}
}
if !found {
missing.push(shard_id as u32);
};
let unsafe_ignore_sidecar = req.unsafe_ignore_sidecar;
// Present/missing/corrupt classification — mirrors pass 1 of Go's
// generateMissingEcFiles plus its bitrot verify-and-exclude.
let mut missing: Vec<u32> = Vec::new();
let mut corrupt_owned: Vec<(u32, String)> = Vec::new();
let mut present = vec![false; total_shards as usize];
let mut present_count = 0usize;
for shard_id in 0..total_shards as usize {
match &found[shard_id] {
None => missing.push(shard_id as u32),
Some(p) => {
let size = std::fs::metadata(p).map(|m| m.len()).unwrap_or(0);
if size == 0 {
// Zero-sized residue is a failed operation's leftover,
// not a shard: exclude it from inputs and regenerate
// over it in place, like a reclassified-corrupt shard.
tracing::warn!(shard = shard_id, path = %p, "zero-sized shard; excluding from rebuild inputs and regenerating");
corrupt_owned.push((shard_id as u32, p.clone()));
} else {
present[shard_id] = true;
present_count += 1;
}
}
}
}
if missing.is_empty() {
return Ok(Response::new(
volume_server_pb::VolumeEcShardsRebuildResponse {
rebuilt_shard_ids: vec![],
},
));
match bitrot_status {
ec_bitrot::BitrotStatus::Invalid => {
if !unsafe_ignore_sidecar {
return Err(Status::internal(format!(
"bitrot sidecar for {} is malformed/unverifiable; refusing to rebuild (pass unsafe_ignore_sidecar to override)",
base_name
)));
}
tracing::warn!(base = %base_name, "bitrot sidecar is malformed/unverifiable; proceeding because unsafe_ignore_sidecar is set");
}
ec_bitrot::BitrotStatus::On if !unsafe_ignore_sidecar => {
let prot = prot.as_ref().unwrap();
let mut corrupt: Vec<u32> = Vec::new();
for shard_id in 0..total_shards as usize {
if !present[shard_id] {
continue;
}
let Some(entry) = ec_bitrot::shard_checksums(prot, shard_id as u32) else {
continue;
};
let p = found[shard_id].as_ref().unwrap();
match ec_bitrot::verify_shard_file_blocks(p, entry, prot.block_size as i64) {
Err(e) => {
// A read error means we cannot trust this shard as
// a Reed-Solomon input: exclude it rather than
// silently feeding possibly-corrupt bytes into
// reconstruction.
tracing::warn!(shard = shard_id, error = %e, "bitrot: failed to verify present shard; excluding it");
corrupt.push(shard_id as u32);
}
Ok(mismatched) if !mismatched.is_empty() => {
corrupt.push(shard_id as u32);
}
_ => {}
}
}
if !corrupt.is_empty() {
// Wholesale-mismatch guard (RS-arbiter conservative form):
// localized bitrot touches a few shards; a stale/wrong
// sidecar mismatches more than parity_shards.
if corrupt.len() > parity_shards as usize {
return Err(Status::internal(format!(
"bitrot sidecar suspect for {}: {}/{} present shards mismatch (> parity {}); refusing to rebuild (pass unsafe_ignore_sidecar to override)",
base_name,
corrupt.len(),
present_count,
parity_shards
)));
}
if present_count - corrupt.len() < data_shards as usize {
return Err(Status::internal(format!(
"bitrot: only {} verified-good shards for {}, need {} data shards; sidecar may be stale (pass unsafe_ignore_sidecar to override)",
present_count - corrupt.len(),
base_name,
data_shards
)));
}
for shard_id in corrupt {
tracing::warn!(
shard = shard_id,
"bitrot: present shard fails checksum; excluding from rebuild inputs and regenerating"
);
let p = found[shard_id as usize].take().unwrap();
present[shard_id as usize] = false;
present_count -= 1;
corrupt_owned.push((shard_id, p));
}
}
}
_ => {}
}
// Rebuild missing shards, searching all locations for input shards.
// Pass other_dirs so shards on sibling disks are found even when the
// primary rebuild dir doesn't hold them. This one takes a single flat
// list — the shape Go's RebuildEcFiles uses — so unlike the resolvers
// above it cannot be handed the rebuild's own index directory
// separately, and a split -dir/-dir.idx location keeps its .ecx and
// .vif there. Go's additionalDirs carries that directory for the same
// reason.
// Everything the rebuild (re)writes: genuinely missing plus
// reclassified-corrupt. The response — and any failure after this
// point — must report the whole set, since the caller keeps copies
// the rebuild regenerated in place.
let mut rebuilt: Vec<u32> = missing.clone();
rebuilt.extend(corrupt_owned.iter().map(|(id, _)| *id));
rebuilt.sort_unstable();
if !rebuilt.is_empty() && present_count < data_shards as usize {
return Err(Status::internal(format!(
"not enough shards to rebuild {}: found {} shards, need at least {} (data shards), missing shards: {:?}",
base_name, present_count, data_shards, rebuilt
)));
}
// Rebuild missing and corrupt shards, searching all locations for
// input shards. This takes a single flat list — the shape Go's
// RebuildEcFiles uses — so unlike the resolvers above it cannot be
// handed the rebuild's own index directory separately, and a split
// -dir/-dir.idx location keeps its .ecx and .vif there. Go's
// additionalDirs carries that directory for the same reason. Nothing
// missing skips only the regeneration: a retry after a failed earlier
// attempt still owes the .ecx rebuild and the mounted-handle repoint
// below.
let mut rebuild_search_dirs: Vec<String> = other_dirs.clone();
if !rebuild_idx_dir.is_empty() && rebuild_idx_dir != rebuild_dir {
rebuild_search_dirs.push(rebuild_idx_dir.clone());
}
let other_dir_refs: Vec<&str> = rebuild_search_dirs.iter().map(|s| s.as_str()).collect();
crate::storage::erasure_coding::ec_encoder::rebuild_ec_files(
&rebuild_dir,
collection,
vid,
&missing,
data_shards as usize,
parity_shards as usize,
&other_dir_refs,
)
.map_err(|e| Status::internal(format!("RebuildEcFiles: {}", e)))?;
if !rebuilt.is_empty() {
crate::storage::erasure_coding::ec_encoder::rebuild_ec_files(
&rebuild_dir,
collection,
vid,
&missing,
&corrupt_owned,
data_shards as usize,
parity_shards as usize,
&other_dir_refs,
prot.as_ref().filter(|_| {
bitrot_status == ec_bitrot::BitrotStatus::On && !unsafe_ignore_sidecar
}),
)
.map_err(|e| Status::internal(format!("RebuildEcFiles: {}", e)))?;
}
// Rebuild .ecx; use idx_directory with fallback to data directory
let ecx_base = format!("{}/{}", rebuild_idx_dir, base_name);
@@ -3276,20 +3465,113 @@ impl VolumeServer for VolumeGrpcService {
}
ecx_dir_refs.extend(other_dir_refs.iter().copied());
crate::storage::erasure_coding::ec_encoder::rebuild_ecx_file(
&ecx_rebuild_dir,
collection,
vid,
data_shards as usize,
block_size,
dat_file_size,
&ecx_dir_refs,
)
.map_err(|e| Status::internal(format!("RebuildEcxFile: {}", e)))?;
// A mounted volume folds its .ecj into the live .ecx in place —
// the same path Go takes under LockDeletionJournal — so readers
// searching the open index never watch it get truncated. Only an
// unmounted index takes the full regen; nothing holds it open to
// interrupt. Either failure still owes the caller the regenerated
// ids — the shards are already written and its deferred cleanup
// would delete the copies as strays without them.
let mut folded = false;
{
let mut store = self.state.store.write().unwrap_or_else(|e| e.into_inner());
let loc_idx = store
.locations
.iter()
.position(|loc| loc.directory == rebuild_dir);
if let Some(loc_idx) = loc_idx {
if let Some(ev) = store.locations[loc_idx].find_ec_volume_mut(vid) {
let journal_path = ev.ecj_file_name();
ev.rebuild_ecx_from_journal().map_err(|e| {
status_internal_with_rebuild_ids(
format!("RebuildEcxFile: {}", e),
&rebuilt,
)
})?;
folded = true;
// The fold replaced a journal other disks may share:
// repoint their handles before dropping the store lock,
// or their next delete lands on the detached inode.
for (i, loc) in store.locations.iter_mut().enumerate() {
if i == loc_idx {
continue;
}
if let Some(other) = loc.find_ec_volume_mut(vid) {
if std::path::Path::new(&other.ecj_file_name())
== std::path::Path::new(&journal_path)
{
other.adopt_folded_journal().map_err(|e| {
status_internal_with_rebuild_ids(
format!("RebuildEcxFile: {}", e),
&rebuilt,
)
})?;
}
}
}
}
}
}
if !folded {
crate::storage::erasure_coding::ec_encoder::rebuild_ecx_file(
&ecx_rebuild_dir,
collection,
vid,
data_shards as usize,
block_size,
dat_file_size,
&ecx_dir_refs,
)
.map_err(|e| {
status_internal_with_rebuild_ids(format!("RebuildEcxFile: {}", e), &rebuilt)
})?;
}
// Repoint every mounted handle of this volume at the file now on
// disk, not just this run's outputs — mirrors Go's repoint loop in
// VolumeEcShardsRebuild: a shard an earlier rebuild wrote on another
// disk is "present" this run and never re-enters `missing`, yet its
// mounted handle still serves the detached inode.
{
let mut store = self.state.store.write().unwrap_or_else(|e| e.into_inner());
for loc_idx in 0..store.locations.len() {
let mounted: Vec<ShardId> = match store.locations[loc_idx].find_ec_volume(vid) {
Some(ev) => ev
.shards
.iter()
.enumerate()
.filter_map(|(i, s)| s.as_ref().map(|_| i as ShardId))
.collect(),
None => continue,
};
for shard_id in mounted {
if store.locations[loc_idx]
.reopen_ec_shard(vid, shard_id)
.is_ok()
{
continue;
}
// The mounted disk lost the file and the rebuild wrote it
// elsewhere — or it is unreadable in place. Drop the stale
// handle and remount from whichever local disk holds the
// rebuilt file; a shard that cannot be remounted stays
// unmounted rather than serving replaced bytes.
store.locations[loc_idx].unmount_ec_shards(vid, &[shard_id]);
if let Err(e) = store.mount_ec_shard(vid, collection, shard_id, "") {
tracing::warn!(
"rebuilt ec shard {}.{} left unmounted: {}",
vid.0,
shard_id,
e
);
}
}
}
}
Ok(Response::new(
volume_server_pb::VolumeEcShardsRebuildResponse {
rebuilt_shard_ids: missing,
rebuilt_shard_ids: rebuilt,
},
))
}