Github-Mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-09 08:00:43 +02:00
Code Issues Packages Projects Releases Wiki Activity

rust volume: advance idx_file_offset only after redb commit succeeds

Browse Source 
put() and delete() appended to the .idx file and advanced idx_file_offset
before committing to redb. If the redb commit failed, the offset included
the orphan row that redb doesn't reflect. A later checkpoint would record
that offset as "the table reflects up to here," and the reload would skip
the orphan row entirely — the entry becomes permanently unindexed.

Move the idx_file_offset increment to after the successful redb commit.
The .idx file still has the orphan row (append-only), but idx_file_offset
stays behind it, so the next checkpoint records the smaller offset and
the reload replays the orphan row back into redb.

Co-Authored-By: Chris Lu <chris.lu@gmail.com>
This commit is contained in:
chrislusf
2026-09-05 21:00:47 -07:00
co-authored by Chris Lu
parent 46534017cc
commit 947ee285de
1 changed files with 14 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
seaweed-volume/src/storage/needle_map.rs
+14 -4
View File
@@ -768,10 +768,13 @@ impl RedbNeedleMap {
/// Insert or update an entry. Writes to idx file first, then redb.
pub fn put(&mut self, key: NeedleId, offset: Offset, size: Size) -> io::Result<()> {
// Persist to idx file BEFORE mutating redb state for crash consistency
// Persist to idx file BEFORE mutating redb state for crash consistency.
// The offset is advanced only after the redb commit succeeds: a failed
// commit leaves an orphan row in .idx that redb doesn't reflect, and
// advancing the offset here would let a later checkpoint record it as
// reflected, making the reload skip it permanently.
if let Some(ref mut idx_file) = self.idx_file {
idx::write_index_entry(idx_file, key, offset, size)?;
self.idx_file_offset += NEEDLE_MAP_ENTRY_SIZE as u64;
}
let key_u64: u64 = key.into();
@@ -792,6 +795,9 @@ impl RedbNeedleMap {
}
txn.commit()
.map_err(|e| io::Error::new(io::ErrorKind::Other, format!("redb commit: {}", e)))?;
if self.idx_file.is_some() {
self.idx_file_offset += NEEDLE_MAP_ENTRY_SIZE as u64;
}
self.writes_since_checkpoint = self.writes_since_checkpoint.saturating_add(1);
self.metric.on_put(key, old.as_ref(), size);
@@ -840,10 +846,11 @@ impl RedbNeedleMap {
if let Some(old) = self.get_internal(key_u64)? {
if old.size.is_valid() {
// Persist tombstone to idx file BEFORE mutating redb
// Persist tombstone to idx file BEFORE mutating redb. The
// offset is advanced only after the redb commit succeeds
// (see put).
if let Some(ref mut idx_file) = self.idx_file {
idx::write_index_entry(idx_file, key, offset, TOMBSTONE_FILE_SIZE)?;
self.idx_file_offset += NEEDLE_MAP_ENTRY_SIZE as u64;
}
let deleted_size = Size(-(old.size.0));
@@ -866,6 +873,9 @@ impl RedbNeedleMap {
txn.commit().map_err(|e| {
io::Error::new(io::ErrorKind::Other, format!("redb commit: {}", e))
})?;
if self.idx_file.is_some() {
self.idx_file_offset += NEEDLE_MAP_ENTRY_SIZE as u64;
}
self.writes_since_checkpoint = self.writes_since_checkpoint.saturating_add(1);
// Only now is the tombstone in the table the metrics describe.