mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-19 10:54:08 +01:00
volume: reclaim staged EC shard generations left by the 2PC switch (#11501)
* volume: remove staged EC generation files on teardown and shard delete The 2PC generation switch stages each run as <base>.ecNN.v<N> plus versioned .ecx/.ecj/.vif files. Nothing on the volume server removes them: isEcDataShardFile only recognises the exact .ecNN name, so the staged files are invisible to every bookkeeping pass, and even full_teardown's wipe-all path left them behind. Each re-encode therefore leaks a full shard set per shard-holding disk. RemoveEcGenerationFiles sweeps <base>.ec*.v<N> and <base>.vif.v<N>, optionally keeping generations at or above a threshold; teardown and the reconcile wipe remove every generation, and a per-shard delete removes that shard's staged generations too. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: delete staged EC generations older than N via VolumeEcShardsDelete After a 2PC generation switch commits, the superseded generation's <base>.*.v<N> files sit on disk with no cleanup path: teardown removes everything, and a per-shard delete only touches the named shards, so the executor had no RPC that reclaims just the staged leftovers. delete_generations_older_than removes staged generation files strictly below the threshold on every disk. Versioned files are never mounted, so nothing is unloaded first; the committed generation and the canonical files are preserved. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: mirror staged EC generation cleanup Parity with the Go volume server: remove_ec_generation_files sweeps <base>.ec*.v<N> and <base>.vif.v<N> staged by the 2PC switch, called by remove_ec_volume_files (which covers both teardown paths) and the new delete_generations_older_than request field; delete_ec_shards removes a shard's staged generations along with the canonical file. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: match staged generation filenames literally filepath.Glob interprets metacharacters in the collection part of the base name, so a collection like a[bc] could match another volume's staged files (or miss its own). Scan the directory and compare names literally instead, mirroring the Rust read_dir implementation. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: report generation-sweep errors and drop the store lock first - snapshot the location base names under the read lock and run the filesystem sweep after dropping it, so a slow disk cannot stall the store; - record per-entry read_dir errors in remove_ec_generation_files and propagate them from remove_ec_shard_generations instead of flatten() skipping them; - warn when a staged-shard generation fails to delete rather than reporting success with files left behind. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: fail shard delete when the staged-generation listing fails A transient ReadDir failure fell back to removing canonical shard names only: staged .v<N> files survived while the RPC still reported success, leaving the leak invisible to retrying callers. ENOENT still means the disk simply has no such directory; other listing errors now propagate. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: propagate staged-generation removal failures delete_ec_shards logged remove_ec_shard_generations errors and the RPC returned success while staged .v<N> files remained, diverging from the Go handler which surfaces the failure. The sweep keeps processing the remaining shards, retains the first error, and volume_ec_shards_delete maps it to Status::internal so callers can retry. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: notify state change even when the shard sweep errors delete_ec_shards already deletes and unmounts the shards before returning a staged-generation failure, so returning early skipped volume_state_notify and the master kept routing to them until the next heartbeat. Notify before propagating the error. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
150a69fe11
commit
43fd5b8d82
13 files changed
+582
-29
No files matched your search
@@ -195,6 +195,104 @@ impl ShardBits {
|
||||
}
|
||||
}
|
||||
|
||||
/// Parses the generation of a 2PC-staged `<base>.v<N>` file: `None` means the
|
||||
/// name is not a generation file of `base`.
|
||||
pub fn ec_file_generation(name: &str, base: &str) -> Option<u32> {
|
||||
let suffix = name.strip_prefix(&format!("{}.v", base))?;
|
||||
match suffix.parse::<u32>() {
|
||||
Ok(g) if g > 0 => Some(g),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Removes 2PC generation files staged under `base`:
|
||||
/// `<base>.ecNN.v<N>`, `<base>.ecx.v<N>`, `<base>.ecj.v<N>`, `<base>.ecsum.v<N>`
|
||||
/// and `<base>.vif.v<N>`. `generations_older_than == 0` removes every
|
||||
/// generation; otherwise only generations strictly below it. Returns the
|
||||
/// first real removal failure. Mirrors Go's `RemoveEcGenerationFiles`.
|
||||
pub fn remove_ec_generation_files(base: &str, generations_older_than: u32) -> io::Result<()> {
|
||||
let path = std::path::Path::new(base);
|
||||
let (Some(parent), Some(fname)) = (path.parent(), path.file_name()) else {
|
||||
return Ok(());
|
||||
};
|
||||
let ec_prefix = format!("{}.ec", fname.to_string_lossy());
|
||||
let vif_name = format!("{}.vif", fname.to_string_lossy());
|
||||
let mut first_err: Option<io::Error> = None;
|
||||
let mut record = |res: io::Result<()>| {
|
||||
if let Err(e) = res
|
||||
&& first_err.is_none()
|
||||
{
|
||||
first_err = Some(e);
|
||||
}
|
||||
};
|
||||
match fs::read_dir(parent) {
|
||||
Ok(entries) => {
|
||||
for entry in entries {
|
||||
let entry = match entry {
|
||||
Ok(entry) => entry,
|
||||
Err(e) => {
|
||||
// A skipped entry means an incomplete sweep; report it
|
||||
// instead of pretending the cleanup finished.
|
||||
record(Err(e));
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let name = entry.file_name().to_string_lossy().into_owned();
|
||||
let Some((artifact, _)) = name.rsplit_once(".v") else {
|
||||
continue;
|
||||
};
|
||||
if artifact != vif_name && !artifact.starts_with(&ec_prefix) {
|
||||
continue;
|
||||
}
|
||||
let Some(generation) = ec_file_generation(&name, artifact) else {
|
||||
continue;
|
||||
};
|
||||
if generations_older_than > 0 && generation >= generations_older_than {
|
||||
continue;
|
||||
}
|
||||
record(match fs::remove_file(entry.path()) {
|
||||
Err(e) if e.kind() != io::ErrorKind::NotFound => Err(e),
|
||||
_ => Ok(()),
|
||||
});
|
||||
}
|
||||
}
|
||||
Err(e) if e.kind() != io::ErrorKind::NotFound => record(Err(e)),
|
||||
Err(_) => {}
|
||||
}
|
||||
match first_err {
|
||||
Some(e) => Err(e),
|
||||
None => Ok(()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Removes every staged generation `<shard_file>.v<N>` of one shard file.
|
||||
/// Returns true when at least one generation file was removed.
|
||||
pub fn remove_ec_shard_generations(shard_file: &str) -> io::Result<bool> {
|
||||
let path = std::path::Path::new(shard_file);
|
||||
let (Some(parent), Some(fname)) = (path.parent(), path.file_name()) else {
|
||||
return Ok(false);
|
||||
};
|
||||
let fname = fname.to_string_lossy().into_owned();
|
||||
let mut removed = false;
|
||||
match fs::read_dir(parent) {
|
||||
Ok(entries) => {
|
||||
for entry in entries {
|
||||
let entry = entry?;
|
||||
let name = entry.file_name().to_string_lossy().into_owned();
|
||||
if ec_file_generation(&name, &fname).is_some() {
|
||||
match fs::remove_file(entry.path()) {
|
||||
Err(e) if e.kind() != io::ErrorKind::NotFound => return Err(e),
|
||||
_ => removed = true,
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) if e.kind() != io::ErrorKind::NotFound => return Err(e),
|
||||
Err(_) => {}
|
||||
}
|
||||
Ok(removed)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
Reference in new issue
Block a user