mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 15:41:15 +02:00
volume: rebuild a missing .idx from the .dat (#11115)
* volume: rebuild a missing .idx from the .dat Pointing -dir.idx at a directory that holds no index aborted the whole volume server: checkIdxFile found no .idx and load() called glog.Fatalf. Every row of the index is derivable from the .dat, so walk it in append order and write the index back, which reproduces byte for byte what the server's own writes had left in the old directory. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: keep the index co-located with the data in the Rust server Go's load() drops back to the data directory when an .idx already sits beside the .dat, so naming a --dir.idx does not strand a pre-existing index. Rust had no such adjustment: it opened the new directory with create, and the volume came up on an empty index with every needle invisible. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: rebuild a missing .idx from the .dat in the Rust server Mirrors the Go side. Rust did not abort on a missing index the way checkIdxFile did; it opened the new directory with create and mounted the volume on an empty index, so every needle read as missing while the .dat still held the data. Walk the .dat in append order and write the index back, byte for byte what the server's own writes had left behind. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: stop the idx rebuild at a zero-padded .dat tail An all-zero needle header is unwritten space, not a record. Go's .dat walk keeps reading past it and would index a truncated data file's tail as millions of needle 0 rows; the Rust walk already stops there. Stop the Go rebuild at the same place. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: create the -dir.idx directory when it does not exist Rust's DiskLocation creates the index directory as it takes it; Go only resolved the path, so naming a directory that does not exist yet left every volume unable to open or rebuild its index and took the server down. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: stop the idx rebuild at a torn .dat record A crash between writing a needle's header and its body leaves a record whose declared size runs past the end of .dat. Indexing it puts a row in the .idx that points at bytes that do not exist, which fails every read of that needle and trips the past-EOF check on the next load. Stop at the first record that does not fit, in both servers. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: stop the idx rebuild at a negative-size header A corrupt header whose size field is negative makes the .dat walk advance backwards: NeedleBodyLength adds the negative size, so the next offset is lower than the current one. The Go walk then reads at a negative offset and the rebuild fails, which puts the volume server right back to exiting at startup; the Rust walk seeks past EOF and truncates the index instead. A negative size is never a record, so stop there. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: skip a volume whose index cannot be rebuilt, do not exit glog.Fatalf calls os.Exit(255), so a rebuild that could not write -- a full or read-only index directory -- put the server right back to dying at startup for one bad volume. Return the error instead: loadExistingVolume logs it and skips that volume, which is what the remote-volume branch just above already does and what the Rust loader has always done. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: create the index directory from the rebuild too The rebuild is the first thing to write into a fresh -dir.idx, and it runs before the loaders that create the directory on their way to opening .idx. Create it in both rebuilds so the ordering does not matter. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * ci: let codespell past the sme variable in the mount tests weedfs_stream_mutate_error_test.go names its *streamMutateError local sme, which codespell reads as a misspelling of same/some. It is an identifier, so exempt it beside the other variable-name entries. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ
This commit is contained in:
Chris Lu
@@ -9,6 +9,7 @@ pub mod store_ec_reconcile;
|
||||
pub mod super_block;
|
||||
pub mod types;
|
||||
pub mod volume;
|
||||
pub mod volume_idx_rebuild;
|
||||
pub mod volume_idx_repair;
|
||||
pub mod volume_open;
|
||||
pub mod volume_report;
|
||||
|
||||
@@ -801,6 +801,30 @@ impl Volume {
|
||||
}
|
||||
|
||||
if also_load_index {
|
||||
// Adjust for existing volumes with .idx together with .dat files:
|
||||
// an index already beside the data keeps serving after --dir.idx
|
||||
// named a different directory.
|
||||
if self.dir_idx != self.dir
|
||||
&& Path::new(&format!("{}.idx", self.data_file_name())).exists()
|
||||
{
|
||||
self.dir_idx = self.dir.clone();
|
||||
}
|
||||
|
||||
// A changed --dir.idx leaves the new directory without an index.
|
||||
// The .dat still holds every row, so rebuild rather than mount the
|
||||
// volume with every needle invisible.
|
||||
if !self.has_remote_file
|
||||
&& !Path::new(&self.file_name(".idx")).exists()
|
||||
&& self.current_dat_file_size()? > SUPER_BLOCK_SIZE as u64
|
||||
{
|
||||
self.rebuild_idx_file()?;
|
||||
info!(
|
||||
volume_id = self.id.0,
|
||||
idx = %self.file_name(".idx"),
|
||||
"rebuilt the index from the data file"
|
||||
);
|
||||
}
|
||||
|
||||
// Recover rows that deletes on a tiered read-only volume overwrote
|
||||
// at the front of .idx. Best effort: a volume that cannot be
|
||||
// repaired is still servable for everything the surviving rows
|
||||
@@ -3995,7 +4019,7 @@ impl Volume {
|
||||
/// Byte offset just past the needle's on-disk record. Deletion tombstones
|
||||
/// carry TombstoneFileSize (-1) in the .idx but are written with DataSize=0,
|
||||
/// so their on-disk record is sized as 0. Mirrors Go's needleDiskEnd.
|
||||
fn needle_disk_end(offset: Offset, size: Size, version: Version) -> i64 {
|
||||
pub(crate) fn needle_disk_end(offset: Offset, size: Size, version: Version) -> i64 {
|
||||
let on_disk_size = if size.is_deleted() { Size(0) } else { size };
|
||||
offset.to_actual_offset() + get_actual_size(on_disk_size, version)
|
||||
}
|
||||
@@ -4166,6 +4190,11 @@ pub fn scan_volume_file(
|
||||
if size.0 == 0 && _id.is_empty() {
|
||||
break; // end of valid data
|
||||
}
|
||||
// A negative size is a corrupt header, and body_length would advance the
|
||||
// walk backwards from it. Go's scanners stop here by returning io.EOF.
|
||||
if size.0 < 0 {
|
||||
break;
|
||||
}
|
||||
|
||||
let body_length = needle::needle_body_length(size, version);
|
||||
let total_size = NEEDLE_HEADER_SIZE as i64 + body_length;
|
||||
@@ -5239,6 +5268,71 @@ mod tests {
|
||||
assert!(Path::new(&idx_dir_idx).exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_load_keeps_index_co_located_with_the_data() {
|
||||
let root = TempDir::new().unwrap();
|
||||
let data_dir = root.path().join("data");
|
||||
let idx_dir = root.path().join("idx");
|
||||
fs::create_dir_all(&data_dir).unwrap();
|
||||
fs::create_dir_all(&idx_dir).unwrap();
|
||||
let data = data_dir.to_str().unwrap();
|
||||
let idx = idx_dir.to_str().unwrap();
|
||||
|
||||
let mut v = Volume::new(
|
||||
data,
|
||||
data,
|
||||
"",
|
||||
VolumeId(7),
|
||||
NeedleMapKind::InMemory,
|
||||
None,
|
||||
None,
|
||||
0,
|
||||
Version::current(),
|
||||
)
|
||||
.unwrap();
|
||||
let payload = b"payload-beside-the-data".to_vec();
|
||||
let mut n = Needle {
|
||||
id: NeedleId(42),
|
||||
cookie: Cookie(0x55),
|
||||
data: payload.clone(),
|
||||
data_size: payload.len() as u32,
|
||||
..Needle::default()
|
||||
};
|
||||
v.write_needle(&mut n, true, false).unwrap();
|
||||
v.sync_to_disk().unwrap();
|
||||
drop(v);
|
||||
|
||||
// --dir.idx now names an empty directory: the index already beside the
|
||||
// data keeps serving, and nothing lands in the new directory.
|
||||
let reopened = Volume::new(
|
||||
data,
|
||||
idx,
|
||||
"",
|
||||
VolumeId(7),
|
||||
NeedleMapKind::InMemory,
|
||||
None,
|
||||
None,
|
||||
0,
|
||||
Version::current(),
|
||||
)
|
||||
.unwrap();
|
||||
assert!(
|
||||
Path::new(&format!("{data}/7.idx")).exists(),
|
||||
"index stays with the data"
|
||||
);
|
||||
assert!(
|
||||
!Path::new(&format!("{idx}/7.idx")).exists(),
|
||||
"nothing written to the new idx dir"
|
||||
);
|
||||
|
||||
let mut got = Needle {
|
||||
id: NeedleId(42),
|
||||
..Needle::default()
|
||||
};
|
||||
reopened.read_needle(&mut got).unwrap();
|
||||
assert_eq!(got.data, payload);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_relocate_index_to_noop_when_already_in_place() {
|
||||
let tmp = TempDir::new().unwrap();
|
||||
|
||||
@@ -0,0 +1,376 @@
|
||||
//! Rebuild a missing .idx from the .dat it indexes. Mirrors
|
||||
//! `weed/storage/volume_idx_rebuild.go`.
|
||||
|
||||
use std::fs::{self, OpenOptions};
|
||||
use std::io::{BufWriter, Write};
|
||||
use std::path::Path;
|
||||
|
||||
use crate::storage::idx;
|
||||
use crate::storage::needle::Needle;
|
||||
use crate::storage::super_block::SuperBlock;
|
||||
use crate::storage::types::*;
|
||||
use crate::storage::volume::{
|
||||
fsync_dir, needle_disk_end, scan_volume_file, Volume, VolumeError, VolumeFileVisitor,
|
||||
};
|
||||
|
||||
/// Writes one .idx row per .dat record, in .dat append order, which is the
|
||||
/// shape the volume server's own writes leave behind.
|
||||
struct VolumeFileScanner4RebuildIdx<W: Write> {
|
||||
writer: W,
|
||||
dat_size: i64,
|
||||
version: Version,
|
||||
stopped: bool,
|
||||
}
|
||||
|
||||
impl<W: Write> VolumeFileVisitor for VolumeFileScanner4RebuildIdx<W> {
|
||||
fn visit_super_block(&mut self, _sb: &SuperBlock) -> Result<(), VolumeError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn read_needle_body(&self) -> bool {
|
||||
false
|
||||
}
|
||||
|
||||
fn visit_needle(&mut self, n: &Needle, offset: i64) -> Result<(), VolumeError> {
|
||||
// A record reaching past the end of .dat is a torn append or a corrupt
|
||||
// header: nothing beyond it is indexable, and a row pointing past EOF
|
||||
// would fail every read of that needle. The all-zero header case ends
|
||||
// the walk upstream.
|
||||
if self.stopped {
|
||||
return Ok(());
|
||||
}
|
||||
if needle_disk_end(Offset::from_actual_offset(offset), n.size, self.version) > self.dat_size
|
||||
{
|
||||
self.stopped = true;
|
||||
return Ok(());
|
||||
}
|
||||
let size = if n.size.is_valid() {
|
||||
n.size
|
||||
} else {
|
||||
TOMBSTONE_FILE_SIZE
|
||||
};
|
||||
idx::write_index_entry(
|
||||
&mut self.writer,
|
||||
n.id,
|
||||
Offset::from_actual_offset(offset),
|
||||
size,
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl Volume {
|
||||
/// Regenerate the volume's .idx from its .dat. The whole index is derivable
|
||||
/// from the data file, so a volume whose index directory has no .idx -- a
|
||||
/// --dir.idx pointed at an empty directory, or a lost index -- comes back on
|
||||
/// its own instead of mounting with every needle invisible. The rows go to a
|
||||
/// temp file that is renamed in, so an interrupted rebuild leaves no partial
|
||||
/// index behind.
|
||||
pub(crate) fn rebuild_idx_file(&self) -> Result<(), VolumeError> {
|
||||
let idx_path = self.file_name(".idx");
|
||||
let dat_path = self.file_name(".dat");
|
||||
let tmp_path = format!("{idx_path}.tmp");
|
||||
|
||||
let rebuild = || -> Result<(), VolumeError> {
|
||||
if let Some(parent) = Path::new(&idx_path).parent() {
|
||||
fs::create_dir_all(parent)?;
|
||||
}
|
||||
let tmp_file = OpenOptions::new()
|
||||
.write(true)
|
||||
.create(true)
|
||||
.truncate(true)
|
||||
.open(&tmp_path)?;
|
||||
let mut scanner = VolumeFileScanner4RebuildIdx {
|
||||
writer: BufWriter::new(&tmp_file),
|
||||
dat_size: fs::metadata(&dat_path)?.len() as i64,
|
||||
version: self.version(),
|
||||
stopped: false,
|
||||
};
|
||||
scan_volume_file(&dat_path, &mut scanner)?;
|
||||
scanner.writer.flush()?;
|
||||
drop(scanner);
|
||||
tmp_file.sync_all()?;
|
||||
fs::rename(&tmp_path, &idx_path)?;
|
||||
fsync_dir(&idx_path)?;
|
||||
Ok(())
|
||||
};
|
||||
|
||||
let result = rebuild();
|
||||
if result.is_err() {
|
||||
let _ = fs::remove_file(&tmp_path);
|
||||
}
|
||||
result
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use crate::storage::needle::crc::CRC;
|
||||
use crate::storage::needle::Needle;
|
||||
use crate::storage::needle_map::NeedleMapKind;
|
||||
use crate::storage::types::*;
|
||||
use crate::storage::volume::Volume;
|
||||
use std::fs;
|
||||
use std::path::Path;
|
||||
use tempfile::TempDir;
|
||||
|
||||
fn needle(id: u64) -> Needle {
|
||||
let data = format!("payload-{id}").into_bytes();
|
||||
Needle {
|
||||
id: NeedleId(id),
|
||||
cookie: Cookie(0x55),
|
||||
data_size: data.len() as u32,
|
||||
checksum: CRC::new(&data),
|
||||
data,
|
||||
..Needle::default()
|
||||
}
|
||||
}
|
||||
|
||||
// Pointing --dir.idx at a directory with no .idx used to mount the volume on
|
||||
// an empty index; the index is derivable from the .dat, so it must be
|
||||
// rebuilt in place instead.
|
||||
#[test]
|
||||
fn test_load_moved_idx_directory_rebuilds_idx() {
|
||||
let root = TempDir::new().unwrap();
|
||||
let data_dir = root.path().join("data");
|
||||
let old_idx_dir = root.path().join("idxA");
|
||||
let new_idx_dir = root.path().join("idxB");
|
||||
for dir in [&data_dir, &old_idx_dir, &new_idx_dir] {
|
||||
fs::create_dir_all(dir).unwrap();
|
||||
}
|
||||
let data = data_dir.to_str().unwrap();
|
||||
let old_idx = old_idx_dir.to_str().unwrap();
|
||||
let new_idx = new_idx_dir.to_str().unwrap();
|
||||
|
||||
let mut v = Volume::new(
|
||||
data,
|
||||
old_idx,
|
||||
"",
|
||||
VolumeId(1),
|
||||
NeedleMapKind::InMemory,
|
||||
None,
|
||||
None,
|
||||
0,
|
||||
Version::current(),
|
||||
)
|
||||
.unwrap();
|
||||
for id in 1..=3 {
|
||||
v.write_needle(&mut needle(id), true, false).unwrap();
|
||||
}
|
||||
v.delete_needle(&mut needle(2)).unwrap();
|
||||
v.sync_to_disk().unwrap();
|
||||
let (want_count, want_deleted) = (v.file_count(), v.deleted_count());
|
||||
drop(v);
|
||||
|
||||
let seeded = fs::read(format!("{old_idx}/1.idx")).unwrap();
|
||||
|
||||
let reopened = Volume::new(
|
||||
data,
|
||||
new_idx,
|
||||
"",
|
||||
VolumeId(1),
|
||||
NeedleMapKind::InMemory,
|
||||
None,
|
||||
None,
|
||||
0,
|
||||
Version::current(),
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert!(
|
||||
Path::new(&format!("{new_idx}/1.idx")).exists(),
|
||||
"idx not rebuilt in the new idx dir"
|
||||
);
|
||||
assert_eq!(reopened.file_count(), want_count);
|
||||
assert_eq!(reopened.deleted_count(), want_deleted);
|
||||
assert_eq!(
|
||||
fs::read(format!("{new_idx}/1.idx")).unwrap(),
|
||||
seeded,
|
||||
"rebuilt idx differs from the one the server wrote"
|
||||
);
|
||||
|
||||
for id in [1, 3] {
|
||||
let mut got = needle(id);
|
||||
got.data.clear();
|
||||
reopened.read_needle(&mut got).unwrap();
|
||||
assert_eq!(got.data, format!("payload-{id}").into_bytes());
|
||||
}
|
||||
}
|
||||
|
||||
// A .dat padded with zeros must not be indexed as needle 0 rows: the walk
|
||||
// stops where the records do.
|
||||
#[test]
|
||||
fn test_rebuild_idx_stops_at_zero_padded_dat_tail() {
|
||||
let root = TempDir::new().unwrap();
|
||||
let dir = root.path().to_str().unwrap();
|
||||
|
||||
let mut v = Volume::new(
|
||||
dir,
|
||||
dir,
|
||||
"",
|
||||
VolumeId(1),
|
||||
NeedleMapKind::InMemory,
|
||||
None,
|
||||
None,
|
||||
0,
|
||||
Version::current(),
|
||||
)
|
||||
.unwrap();
|
||||
v.write_needle(&mut needle(1), true, false).unwrap();
|
||||
v.sync_to_disk().unwrap();
|
||||
drop(v);
|
||||
|
||||
let seeded = fs::read(format!("{dir}/1.idx")).unwrap();
|
||||
let dat = fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.open(format!("{dir}/1.dat"))
|
||||
.unwrap();
|
||||
let dat_size = dat.metadata().unwrap().len();
|
||||
dat.set_len(dat_size + 4096).unwrap();
|
||||
drop(dat);
|
||||
fs::remove_file(format!("{dir}/1.idx")).unwrap();
|
||||
|
||||
let reopened = Volume::new(
|
||||
dir,
|
||||
dir,
|
||||
"",
|
||||
VolumeId(1),
|
||||
NeedleMapKind::InMemory,
|
||||
None,
|
||||
None,
|
||||
0,
|
||||
Version::current(),
|
||||
)
|
||||
.unwrap();
|
||||
drop(reopened);
|
||||
|
||||
assert_eq!(
|
||||
fs::read(format!("{dir}/1.idx")).unwrap(),
|
||||
seeded,
|
||||
"the zero-padded tail leaked into the rebuilt idx"
|
||||
);
|
||||
}
|
||||
|
||||
// A .dat whose last append was torn mid-body must not gain an index row
|
||||
// that points past the end of the file.
|
||||
#[test]
|
||||
fn test_rebuild_idx_skips_truncated_dat_tail() {
|
||||
let root = TempDir::new().unwrap();
|
||||
let dir = root.path().to_str().unwrap();
|
||||
|
||||
let mut v = Volume::new(
|
||||
dir,
|
||||
dir,
|
||||
"",
|
||||
VolumeId(1),
|
||||
NeedleMapKind::InMemory,
|
||||
None,
|
||||
None,
|
||||
0,
|
||||
Version::current(),
|
||||
)
|
||||
.unwrap();
|
||||
v.write_needle(&mut needle(1), true, false).unwrap();
|
||||
v.sync_to_disk().unwrap();
|
||||
let kept = fs::read(format!("{dir}/1.idx")).unwrap();
|
||||
v.write_needle(&mut needle(2), true, false).unwrap();
|
||||
v.sync_to_disk().unwrap();
|
||||
drop(v);
|
||||
|
||||
// Chop the second needle's body, leaving its header intact.
|
||||
let dat = fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.open(format!("{dir}/1.dat"))
|
||||
.unwrap();
|
||||
let torn_size = dat.metadata().unwrap().len() - 8;
|
||||
dat.set_len(torn_size).unwrap();
|
||||
drop(dat);
|
||||
fs::remove_file(format!("{dir}/1.idx")).unwrap();
|
||||
|
||||
let reopened = Volume::new(
|
||||
dir,
|
||||
dir,
|
||||
"",
|
||||
VolumeId(1),
|
||||
NeedleMapKind::InMemory,
|
||||
None,
|
||||
None,
|
||||
0,
|
||||
Version::current(),
|
||||
)
|
||||
.unwrap();
|
||||
drop(reopened);
|
||||
|
||||
assert_eq!(
|
||||
fs::read(format!("{dir}/1.idx")).unwrap(),
|
||||
kept,
|
||||
"the torn record leaked into the rebuilt idx"
|
||||
);
|
||||
}
|
||||
|
||||
// A corrupt header carrying a negative size advances the .dat walk
|
||||
// backwards, which cycles forever between it and the record before it.
|
||||
#[test]
|
||||
fn test_rebuild_idx_stops_at_negative_size_header() {
|
||||
let root = TempDir::new().unwrap();
|
||||
let dir = root.path().to_str().unwrap();
|
||||
|
||||
let mut v = Volume::new(
|
||||
dir,
|
||||
dir,
|
||||
"",
|
||||
VolumeId(1),
|
||||
NeedleMapKind::InMemory,
|
||||
None,
|
||||
None,
|
||||
0,
|
||||
Version::current(),
|
||||
)
|
||||
.unwrap();
|
||||
v.write_needle(&mut needle(1), true, false).unwrap();
|
||||
v.sync_to_disk().unwrap();
|
||||
let kept = fs::read(format!("{dir}/1.idx")).unwrap();
|
||||
v.write_needle(&mut needle(2), true, false).unwrap();
|
||||
v.sync_to_disk().unwrap();
|
||||
drop(v);
|
||||
|
||||
let rows = fs::read(format!("{dir}/1.idx")).unwrap();
|
||||
let (_, offset, _) = idx_entry_from_bytes(&rows[NEEDLE_MAP_ENTRY_SIZE..]);
|
||||
let corrupt_at = offset.to_actual_offset();
|
||||
|
||||
// Overwrite the second needle's size field with a negative i32.
|
||||
use std::io::{Seek, SeekFrom, Write};
|
||||
let mut dat = fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.open(format!("{dir}/1.dat"))
|
||||
.unwrap();
|
||||
dat.seek(SeekFrom::Start(
|
||||
corrupt_at as u64 + COOKIE_SIZE as u64 + NEEDLE_ID_SIZE as u64,
|
||||
))
|
||||
.unwrap();
|
||||
dat.write_all(&[0xff, 0xff, 0xf0, 0x00]).unwrap();
|
||||
drop(dat);
|
||||
fs::remove_file(format!("{dir}/1.idx")).unwrap();
|
||||
|
||||
// Pre-fix the rebuild walked backwards from here and never terminated.
|
||||
let reopened = Volume::new(
|
||||
dir,
|
||||
dir,
|
||||
"",
|
||||
VolumeId(1),
|
||||
NeedleMapKind::InMemory,
|
||||
None,
|
||||
None,
|
||||
0,
|
||||
Version::current(),
|
||||
)
|
||||
.unwrap();
|
||||
drop(reopened);
|
||||
|
||||
assert_eq!(
|
||||
fs::read(format!("{dir}/1.idx")).unwrap(),
|
||||
kept,
|
||||
"the corrupt record leaked into the rebuilt idx"
|
||||
);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user