Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-11-08 13:07:45 +01:00
Code Issues Packages Projects Releases Wiki Activity

rust: clippy-clean both crates and adopt the std APIs the 1.91 MSRV allows (#11312)

Browse Source 
* rust: apply clippy --fix to both crates

The mechanical part of a clippy sweep: `cargo clippy --all-targets --fix`
on seaweed-volume and the seaweed-worker workspace, hand-reviewed. Both
manifests declare their MSRV (1.91.1 and 1.94.1), so every suggestion
clippy applied is within it: the collapsible_if sites become let chains
(1.88, edition 2024), `% n == 0` becomes is_multiple_of (1.87),
chunks_exact with a constant becomes as_chunks (1.88), repeat().take()
becomes repeat_n (1.82), and io::Error::new(Other, ..) becomes
io::Error::other (1.74). The rest is redundant clones, borrows, casts,
closures and field names.

Nothing here changes behaviour. The three let_and_return sites in
needle_map.rs and store_ec.rs deserve a note: the `let result = ..;
result` shape was a deliberate edition-2021 workaround to drop a redb
guard before the table it borrows. Edition 2024 drops tail-expression
temporaries before locals, which is why clippy now flags it, and the
two comments that described the workaround say so instead.

Manual edits on top of the tool output: the blocks clippy rewrote are
re-indented the way rustfmt lays them out (only those blocks — the
crate is not rustfmt-clean and a whole-crate fmt would bury this diff),
the blank lines let_and_return left behind are removed, and the CRC
legacy_value test compares against a literal worked out from the
original shift formula rather than restating rotate_right.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust: clear the clippy warnings --fix cannot apply, and say why the rest stay

Hand fixes for the lints clippy only reports. Behaviour is unchanged
throughout; each rewrite is the one clippy names.

- needless_range_loop (7): index loops over shard vectors become
  iterator loops. Where the old code indexed `v[..n]` the new loop
  iterates `v[..n]` so an undersized vector still panics the same way.
- field_reassign_with_default (6): struct literals with `..Default`.
- redundant_pattern_matching (3): `if let Err(_) = guard.check()` becomes
  `.is_err()`, which also releases the read guard at the end of the
  condition instead of at the end of the block.
- manual_strip (2), manual_checked_ops, format_in_format_args,
  redundant_locals, wrong_self_convention (to_vif takes self by value,
  so it is into_vif; CompactEntry is Copy, so to_needle_value takes self).
- type_complexity (2): `OrphanShardLoad` and `RawNeedleEntry` name two
  tuples that were spelled out inline.
- new_without_default: CompactNeedleMap gets a Default that calls new().
- suspicious_open_options: a test helper spells out `.truncate(false)`,
  which is what `.create(true).write(true)` already did.

What stays, and the attribute that says so:

- too_many_arguments (10): `#[expect]` on each function. Folding 8–15
  parameters into a struct is a design change, not a lint fix.
- await_holding_lock / readonly_write_lock: one test holds the store
  write guard across a sleep on purpose, as a barrier that parks the
  copy task at the mount block. `#[expect(.., reason = ..)]` records it.
- module_inception: needle/needle.rs mirrors the Go package layout.

Two lints become crate-wide policy in `[lints.clippy]`, with the reason
next to each: result_large_err, because every RPC path returns
tonic::Status (176 bytes) and boxing it would change every handler
signature; and needless_update, because `..Default::default()` on a
protobuf message literal is what lets a proto gain a field without
touching every constructor (all 11 sites are pb messages). The worker
workspace gets the same table and its members opt in with
`lints.workspace = true`; its generated plugin.rs also allows
large_enum_variant on prost's oneof enums.

Both crates are now clean under `cargo clippy --all-targets -- -D warnings`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust volume: use the std APIs the 1.91 MSRV already pays for

The crate declares rust-version 1.91.1, so a few things the code still
worked around are plain std now. All of them come from the 1.85–1.91
release notes; nothing here needs a newer toolchain than the manifest
already requires.

- std::sync::LazyLock (1.80) replaces the lazy_static! block in
  metrics.rs, and the lazy_static dependency goes. Every use site reads
  the same through Deref, so no caller changes.
- Duration::from_mins / from_hours (1.91) replace `from_secs(v * 60)`
  and `from_secs(v * 3600)` in the option parser and the shard-location
  refresh TTLs. One difference for the parser: an absurd count that
  overflows u64 seconds now panics in release builds too, where the
  multiplication used to wrap.
- Result::flatten (1.89) replaces `.and_then(|r| r)` on the replication
  join handle.
- OsStr::display (1.87) replaces `to_string_lossy()` where the name was
  only being formatted; the output is byte-identical.
- `#[allow]` becomes `#[expect]` (1.81) on the suppressions that are
  meant to be permanent, so a suppression that stops being needed
  becomes a warning rather than lingering. Doing that found four that
  already had: dead_code on ChunkManifest, base_name and last_io_error,
  and too_many_arguments on read_from_data_shards, which is down to
  seven parameters. Those attributes are deleted. The three allows that
  depend on cfg (a unix-only mutation, a linux-only field set, a
  profiling-only parameter) stay as allow, because expect would be
  unfulfilled on the other platforms.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* ci: add a commented-out clippy step to both Rust workflows

Both crates are warning-free under `cargo clippy --all-targets
-D warnings` now. Whether that becomes a gate is a policy call, so the
step is present but commented out; uncommenting it is the whole change.
The comment points at the `[lints.clippy]` table where crate-wide
exceptions are recorded, so the gate does not become a reason to
sprinkle allows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust volume: guard parse_duration against overflow panics

Duration::from_mins/from_hours panic when the count overflows u64
seconds. Use checked_mul so an oversized CLI value falls back to the
parser default instead of crashing volume startup.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
This commit is contained in:
Eliah RusinClaude Fable 5.1Chris Lu
 authored and GitHub committed 2026-09-14 11:29:29 -07:00
1 parent 49f20489e4
commit adaf3534fa
40 files changed
+1610 -1579
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

seaweed-volume/src/storage/erasure_coding/ec_bitrot.rs
+7 -7
View File
@@ -164,16 +164,16 @@ pub fn remove_bitrot_sidecars(base: &str) -> io::Result<()> {
};
let mut first_err: Option<io::Error> = None;
let mut record = |res: io::Result<()>| {
if let Err(e) = res {
if first_err.is_none() {
first_err = Some(e);
}
if let Err(e) = res
&& first_err.is_none()
{
first_err = Some(e);
}
};
record(rm(format!("{}{}", base, BITROT_SIDECAR_EXT).into()));
let path = Path::new(base);
if let (Some(parent), Some(fname)) = (path.parent(), path.file_name()) {
let prefix = format!("{}{}.v", fname.to_string_lossy(), BITROT_SIDECAR_EXT);
let prefix = format!("{}{}.v", fname.display(), BITROT_SIDECAR_EXT);
match fs::read_dir(parent) {
Ok(entries) => {
for entry in entries.flatten() {
@@ -203,7 +203,7 @@ pub fn new_encode_uuid() -> Vec<u8> {
/// Reports whether `block_size` is a power of two in [1 MiB, MAX_BITROT_BLOCK_SIZE].
pub fn is_pow2_multiple_of_1mib(block_size: u32) -> bool {
block_size >= (1 << 20) && block_size <= MAX_BITROT_BLOCK_SIZE && block_size.count_ones() == 1
((1 << 20)..=MAX_BITROT_BLOCK_SIZE).contains(&block_size) && block_size.count_ones() == 1
}
/// Returns ceil(covered_size / block_size).
@@ -402,7 +402,7 @@ pub fn validate_manifest(
total
));
}
let mut seen = vec![false; MAX_SHARD_COUNT];
let mut seen = [false; MAX_SHARD_COUNT];
for s in &prot.shards {
if s.shard_id >= total as u32 {
return Err(format!(
seaweed-volume/src/storage/erasure_coding/ec_decoder.rs
+8 -10
View File
@@ -73,6 +73,7 @@ pub fn find_dat_file_size_with_dirs(
/// must live in `dir`. For the cross-disk reconciled layout where
/// shards are split across multiple data dirs of the same node, use
/// [`write_dat_file_from_shards_with_dirs`] instead.
#[expect(clippy::too_many_arguments)]
pub fn write_dat_file_from_shards(
dir: &str,
collection: &str,
@@ -120,7 +121,7 @@ pub fn write_dat_file_from_shards(
/// size. `large_block_size`/`small_block_size` are the volume's shard
/// block layout, e.g. `EcVolume::large_block_size()` /
/// `small_block_size()` from its .vif EC config.
#[allow(clippy::too_many_arguments)]
#[expect(clippy::too_many_arguments)]
pub fn write_dat_file_from_shards_with_dirs(
dat_dir: &str,
collection: &str,
@@ -145,7 +146,7 @@ pub fn write_dat_file_from_shards_with_dirs(
)
}
#[allow(clippy::too_many_arguments)]
#[expect(clippy::too_many_arguments)]
fn write_dat_file(
dat_dir: &str,
collection: &str,
@@ -233,10 +234,10 @@ fn write_dat_file(
// Read large blocks
while encoded_remaining >= large_row_size && remaining > 0 {
for i in 0..data_shards {
for (i, shard) in shards[..data_shards].iter().enumerate() {
let to_write = large_block_size.min(remaining as usize);
let mut buf = vec![0u8; to_write];
let n = shards[i].read_at(&mut buf, shard_offset)?;
let n = shard.read_at(&mut buf, shard_offset)?;
if n != to_write {
return Err(io::Error::new(
io::ErrorKind::UnexpectedEof,
@@ -255,10 +256,10 @@ fn write_dat_file(
// Read small blocks
while remaining > 0 {
for i in 0..data_shards {
for (i, shard) in shards[..data_shards].iter().enumerate() {
let to_write = small_block_size.min(remaining as usize);
let mut buf = vec![0u8; to_write];
let n = shards[i].read_at(&mut buf, shard_offset)?;
let n = shard.read_at(&mut buf, shard_offset)?;
if n != to_write {
return Err(io::Error::new(
io::ErrorKind::UnexpectedEof,
@@ -324,10 +325,7 @@ pub fn write_idx_file_from_ec_index(
// and treat only NotFound as "no journal": Path::exists would also
// swallow a permission/IO error and silently skip deletions, which
// would resurrect deleted needles as live.
let mut idx_file = std::fs::OpenOptions::new()
.write(true)
.append(true)
.open(&tmp_path)?;
let mut idx_file = std::fs::OpenOptions::new().append(true).open(&tmp_path)?;
match std::fs::read(&ecj_path) {
Ok(ecj_data) => {
let count = ecj_data.len() / NEEDLE_ID_SIZE;
seaweed-volume/src/storage/erasure_coding/ec_encoder.rs
+40 -51
View File
@@ -50,7 +50,7 @@ pub fn write_ec_files(
let dat_size = dat_file.metadata()?.len() as i64;
let rs = ReedSolomon::new(data_shards, parity_shards)
.map_err(|e| io::Error::new(io::ErrorKind::Other, format!("reed-solomon init: {:?}", e)))?;
.map_err(|e| io::Error::other(format!("reed-solomon init: {:?}", e)))?;
// Create shard files
let total_shards = data_shards + parity_shards;
@@ -162,7 +162,7 @@ pub fn rebuild_ec_files(
}
let rs = ReedSolomon::new(data_shards, parity_shards)
.map_err(|e| io::Error::new(io::ErrorKind::Other, format!("reed-solomon init: {:?}", e)))?;
.map_err(|e| io::Error::other(format!("reed-solomon init: {:?}", e)))?;
let total_shards = data_shards + parity_shards;
let mut shards: Vec<EcVolumeShard> = (0..total_shards as u8)
@@ -175,7 +175,7 @@ pub fn rebuild_ec_files(
let mut shard_size = 0;
for (i, shard) in shards.iter_mut().enumerate() {
if !missing_shard_ids.contains(&(i as u32)) {
if let Ok(_) = shard.open() {
if shard.open().is_ok() {
let size = shard.file_size();
if size > shard_size {
shard_size = size;
@@ -185,7 +185,7 @@ pub fn rebuild_ec_files(
let mut found = false;
for &other_dir in additional_dirs {
let mut alt = EcVolumeShard::new(other_dir, collection, volume_id, i as u8);
if let Ok(_) = alt.open() {
if alt.open().is_ok() {
let size = alt.file_size();
if size > shard_size {
shard_size = size;
@@ -251,12 +251,8 @@ pub fn rebuild_ec_files(
}
// Reconstruct missing shards
rs.reconstruct(&mut buffers).map_err(|e| {
io::Error::new(
io::ErrorKind::Other,
format!("reed-solomon reconstruct: {:?}", e),
)
})?;
rs.reconstruct(&mut buffers)
.map_err(|e| io::Error::other(format!("reed-solomon reconstruct: {:?}", e)))?;
// Write recovered data into the missing shards
for i in missing_shard_ids {
@@ -296,7 +292,7 @@ pub fn verify_ec_shards(
parity_shards: usize,
) -> io::Result<(Vec<u32>, Vec<String>)> {
let rs = ReedSolomon::new(data_shards, parity_shards)
.map_err(|e| io::Error::new(io::ErrorKind::Other, format!("reed-solomon init: {:?}", e)))?;
.map_err(|e| io::Error::other(format!("reed-solomon init: {:?}", e)))?;
let total_shards = data_shards + parity_shards;
let mut shards: Vec<Option<EcVolumeShard>> = (0..total_shards)
@@ -378,27 +374,27 @@ pub fn verify_ec_shards(
if !read_failed {
// Need to convert Vec<Vec<u8>> to &[&[u8]] for rs.verify
let slice_ptrs: Vec<&[u8]> = buffers.iter().map(|v| v.as_slice()).collect();
if let Ok(is_valid) = rs.verify(&slice_ptrs) {
if !is_valid {
// Reed-Solomon verification failed. We cannot easily pinpoint which shard
// is corrupted without recalculating parities or syndromes, so we just
// log that this batch has corruption. Wait, we can test each parity shard!
// Let's re-encode from the first `data_shards` and compare to the actual `parity_shards`.
if let Ok(is_valid) = rs.verify(&slice_ptrs)
&& !is_valid
{
// Reed-Solomon verification failed. We cannot easily pinpoint which shard
// is corrupted without recalculating parities or syndromes, so we just
// log that this batch has corruption. Wait, we can test each parity shard!
// Let's re-encode from the first `data_shards` and compare to the actual `parity_shards`.
let mut verify_buffers = buffers.clone();
// Clear the parity parts
for i in data_shards..total_shards {
verify_buffers[i].fill(0);
}
if rs.encode(&mut verify_buffers).is_ok() {
for i in 0..total_shards {
if buffers[i] != verify_buffers[i] {
broken_shards.insert(i as u32);
details.push(format!(
"parity mismatch on shard {} at offset {}",
i, offset
));
}
let mut verify_buffers = buffers.clone();
// Clear the parity parts
for buf in &mut verify_buffers[data_shards..total_shards] {
buf.fill(0);
}
if rs.encode(&mut verify_buffers).is_ok() {
for i in 0..total_shards {
if buffers[i] != verify_buffers[i] {
broken_shards.insert(i as u32);
details.push(format!(
"parity mismatch on shard {} at offset {}",
i, offset
));
}
}
}
@@ -490,7 +486,7 @@ pub fn rebuild_ecx_file(
.collect();
for (i, shard) in shards.iter_mut().enumerate() {
if let Err(_) = shard.open() {
if shard.open().is_err() {
let mut found = false;
for &other_dir in additional_dirs {
let mut alt = EcVolumeShard::new(other_dir, collection, volume_id, i as u8);
@@ -507,7 +503,7 @@ pub fn rebuild_ecx_file(
}
return Err(io::Error::new(
io::ErrorKind::NotFound,
format!("cannot open data shard for ecx rebuild"),
"cannot open data shard for ecx rebuild".to_string(),
));
}
}
@@ -515,7 +511,7 @@ pub fn rebuild_ecx_file(
// Determine total logical data size from shard sizes
let shard_size = shards.iter().map(|s| s.file_size()).max().unwrap_or(0);
let total_data_size = shard_size as i64 * data_shards as i64;
let total_data_size = shard_size * data_shards as i64;
// The volume's shard block layout: the .vif-recorded uniform block size,
// or the legacy two-tier sizes when 0. The row count comes from the shard
// length; -1 disambiguates a legacy shard that is an exact large-block
@@ -538,7 +534,7 @@ pub fn rebuild_ecx_file(
let locate_shard_size = if dat_file_size > 0 {
dat_file_size / data_shards as i64
} else {
(shard_size as i64 - 1).max(0)
(shard_size - 1).max(0)
};
// Read version from superblock (first byte of logical data)
@@ -640,7 +636,6 @@ pub fn rebuild_ecx_file(
/// Read bytes from EC data shards at a logical offset in the .dat file,
/// resolving the shard/offset through the volume's block layout via
/// locate_data — the same mapping the read path uses.
#[allow(clippy::too_many_arguments)]
fn read_from_data_shards(
shards: &[EcVolumeShard],
buf: &mut [u8],
@@ -714,7 +709,7 @@ const ENCODE_BUFFER_SIZE: usize = 256 * 1024;
/// 2. Process remaining data with small blocks
///
/// `buffer_size` must divide both block sizes.
#[allow(clippy::too_many_arguments)]
#[expect(clippy::too_many_arguments)]
pub(crate) fn encode_dat_file(
dat_file: &File,
dat_size: i64,
@@ -778,7 +773,7 @@ pub(crate) fn encode_dat_file(
/// Encode one row of blocks, streaming it in ENCODE_BUFFER_SIZE sub-batches so
/// arbitrarily large blocks never require block-sized allocations. Mirrors
/// Go's encodeData.
#[allow(clippy::too_many_arguments)]
#[expect(clippy::too_many_arguments)]
fn encode_data(
dat_file: &File,
row_offset: u64,
@@ -790,7 +785,7 @@ fn encode_data(
data_shards: usize,
) -> io::Result<()> {
let buffer_size = buffers[0].len();
if block_size % buffer_size != 0 {
if !block_size.is_multiple_of(buffer_size) {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
format!(
@@ -817,7 +812,7 @@ fn encode_data(
/// Encode one sub-batch: the same buffer-sized slice of every shard's block in
/// this row. Mirrors Go's encodeDataOneBatch.
#[allow(clippy::too_many_arguments)]
#[expect(clippy::too_many_arguments)]
fn encode_one_batch(
dat_file: &File,
offset: u64,
@@ -830,21 +825,15 @@ fn encode_one_batch(
) -> io::Result<()> {
// Read data shards from the .dat file, zero-filling past EOF — the buffers
// are reused across batches, so the tail must be cleared explicitly.
for i in 0..data_shards {
for (i, buf) in buffers[..data_shards].iter_mut().enumerate() {
let read_offset = offset + (i * block_size) as u64;
let n = read_at_most(dat_file, &mut buffers[i], read_offset)?;
for b in buffers[i][n..].iter_mut() {
*b = 0;
}
let n = read_at_most(dat_file, buf, read_offset)?;
buf[n..].fill(0);
}
// Encode parity shards
rs.encode(&mut *buffers).map_err(|e| {
io::Error::new(
io::ErrorKind::Other,
format!("reed-solomon encode: {:?}", e),
)
})?;
rs.encode(&mut *buffers)
.map_err(|e| io::Error::other(format!("reed-solomon encode: {:?}", e)))?;
// Write all shard buffers to files and feed the same bytes to each
// shard's bitrot checksum builder, keeping covered_size == on-disk length.
seaweed-volume/src/storage/erasure_coding/ec_shard.rs
+3 -3
View File
@@ -78,7 +78,7 @@ impl EcVolumeShard {
let file = self
.ecd_file
.as_ref()
.ok_or_else(|| io::Error::new(io::ErrorKind::Other, "shard file not open"))?;
.ok_or_else(|| io::Error::other("shard file not open"))?;
#[cfg(unix)]
{
@@ -102,7 +102,7 @@ impl EcVolumeShard {
let file = self
.ecd_file
.as_mut()
.ok_or_else(|| io::Error::new(io::ErrorKind::Other, "shard file not open"))?;
.ok_or_else(|| io::Error::other("shard file not open"))?;
file.write_all(data)?;
self.ecd_file_size += data.len() as i64;
Ok(())
@@ -123,7 +123,7 @@ impl EcVolumeShard {
pub fn try_clone_file(&self) -> io::Result<File> {
self.ecd_file
.as_ref()
.ok_or_else(|| io::Error::new(io::ErrorKind::Other, "shard file not open"))?
.ok_or_else(|| io::Error::other("shard file not open"))?
.try_clone()
}
seaweed-volume/src/storage/erasure_coding/ec_volume.rs
+93 -104
View File
@@ -565,32 +565,31 @@ impl EcVolume {
// A sidecar written for THIS generation that contradicts the volume's
// geometry is not "no protection" — it says the layout the volume is
// about to serve reads with is wrong. Fail the mount.
if let Ok(prot) = &loaded {
if prot.generation == generation
&& !ec_bitrot::geometry_matches(
prot,
self.data_shards as usize,
self.parity_shards as usize,
if let Ok(prot) = &loaded
&& prot.generation == generation
&& !ec_bitrot::geometry_matches(
prot,
self.data_shards as usize,
self.parity_shards as usize,
self.block_size,
)
{
let cfg = prot.ec_shard_config.as_ref();
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!(
"ec volume {} generation {}: {} records layout {}+{} block {} but the volume is mounted as {}+{} block {}; refusing to serve one of the two layouts",
self.volume_id.0,
generation,
path,
cfg.map(|c| c.data_shards).unwrap_or(0),
cfg.map(|c| c.parity_shards).unwrap_or(0),
cfg.map(|c| c.block_size).unwrap_or(0),
self.data_shards,
self.parity_shards,
self.block_size,
)
{
let cfg = prot.ec_shard_config.as_ref();
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!(
"ec volume {} generation {}: {} records layout {}+{} block {} but the volume is mounted as {}+{} block {}; refusing to serve one of the two layouts",
self.volume_id.0,
generation,
path,
cfg.map(|c| c.data_shards).unwrap_or(0),
cfg.map(|c| c.parity_shards).unwrap_or(0),
cfg.map(|c| c.block_size).unwrap_or(0),
self.data_shards,
self.parity_shards,
self.block_size,
),
));
}
),
));
}
let status = ec_bitrot::resolve_status(
&loaded,
@@ -698,7 +697,7 @@ impl EcVolume {
let mut set = self
.deleted_needles
.write()
.map_err(|_| io::Error::new(io::ErrorKind::Other, "deleted_needles lock poisoned"))?;
.map_err(|_| io::Error::other("deleted_needles lock poisoned"))?;
let mut off: i64 = 0;
while off + NEEDLE_ID_SIZE as i64 <= self.ecj_file_size {
#[cfg(unix)]
@@ -744,7 +743,6 @@ impl EcVolume {
// ---- File names ----
#[allow(dead_code)]
fn base_name(&self) -> String {
crate::storage::volume::volume_file_name(&self.dir, &self.collection, self.volume_id)
}
@@ -826,10 +824,8 @@ impl EcVolume {
/// default to the physical location's disk type.
pub fn set_disk_type(&mut self, d: DiskType) {
self.disk_type = d.clone();
for slot in self.shards.iter_mut() {
if let Some(shard) = slot {
shard.disk_type = d.clone();
}
for shard in self.shards.iter_mut().flatten() {
shard.disk_type = d.clone();
}
}
@@ -986,21 +982,21 @@ impl EcVolume {
pub fn check_read_write_error(&self, err: Option<&io::Error>) {
use std::sync::atomic::Ordering;
if let Some(e) = err {
if crate::storage::volume::is_storage_io_error(e) {
self.io_error_count.fetch_add(1, Ordering::Relaxed);
if let Ok(mut guard) = self.last_io_error.lock() {
*guard = Some(e.to_string());
}
crate::metrics::STORAGE_IO_ERROR_COUNTER.inc();
return;
if let Some(e) = err
&& crate::storage::volume::is_storage_io_error(e)
{
self.io_error_count.fetch_add(1, Ordering::Relaxed);
if let Ok(mut guard) = self.last_io_error.lock() {
*guard = Some(e.to_string());
}
crate::metrics::STORAGE_IO_ERROR_COUNTER.inc();
return;
}
self.io_error_count.store(0, Ordering::Relaxed);
if let Ok(mut guard) = self.last_io_error.lock() {
if guard.is_some() {
*guard = None;
}
if let Ok(mut guard) = self.last_io_error.lock()
&& guard.is_some()
{
*guard = None;
}
}
@@ -1033,7 +1029,7 @@ impl EcVolume {
let ecx_file = self
.ecx_file
.as_ref()
.ok_or_else(|| io::Error::new(io::ErrorKind::Other, "ecx file not open"))?;
.ok_or_else(|| io::Error::other("ecx file not open"))?;
let entry_count = self.ecx_file_size as usize / NEEDLE_MAP_ENTRY_SIZE;
if entry_count == 0 {
@@ -1253,10 +1249,8 @@ impl EcVolume {
/// Get the size of a single shard (all shards are the same size).
fn shard_file_size(&self) -> i64 {
for shard in &self.shards {
if let Some(s) = shard {
return s.file_size();
}
if let Some(s) = self.shards.iter().flatten().next() {
return s.file_size();
}
0
}
@@ -1363,13 +1357,10 @@ impl EcVolume {
/// the index (ignored by callers) and an error on IO failure.
fn tombstone_ecx_entry(&self, needle_id: NeedleId) -> io::Result<bool> {
let ecx_file = self.ecx_file.as_ref().ok_or_else(|| {
io::Error::new(
io::ErrorKind::Other,
format!(
"ec volume {} has no open .ecx file (closed or corrupt)",
self.volume_id.0
),
)
io::Error::other(format!(
"ec volume {} has no open .ecx file (closed or corrupt)",
self.volume_id.0
))
})?;
let entry_count = self.ecx_file_size as usize / NEEDLE_MAP_ENTRY_SIZE;
@@ -1416,7 +1407,7 @@ impl EcVolume {
/// `deleted_needles` instead). The rebuild is atomic with respect to
/// the journal: if any individual write fails the .ecj file is left
/// in place and the error is propagated so tombstones are not lost.
#[allow(dead_code)]
#[expect(dead_code, reason = "no caller yet; see the doc comment")]
fn rebuild_ecx_from_journal(&mut self) -> io::Result<()> {
let ecj_path = self.ecj_file_name();
if !std::path::Path::new(&ecj_path).exists() {
@@ -1510,7 +1501,7 @@ impl EcVolume {
let ecj_file = self
.ecj_file
.as_mut()
.ok_or_else(|| io::Error::new(io::ErrorKind::Other, "ecj file not open"))?;
.ok_or_else(|| io::Error::other("ecj file not open"))?;
let mut buf = [0u8; NEEDLE_ID_SIZE];
needle_id.to_bytes(&mut buf);
ecj_file.write_all(&buf).and_then(|_| ecj_file.sync_all())
@@ -1528,15 +1519,15 @@ impl EcVolume {
// write_all may have extended the file on disk before
// sync_all failed; truncate back to the known-good size so
// the on-disk journal never drifts past `deleted_needles`.
if let Some(ecj) = self.ecj_file.as_mut() {
if let Err(trunc_err) = ecj.set_len(prev_ecj_size as u64) {
tracing::error!(
volume_id = self.volume_id.0,
needle_id = needle_id.0,
truncate_error = %trunc_err,
"failed to truncate ecj after append failure"
);
}
if let Some(ecj) = self.ecj_file.as_mut()
&& let Err(trunc_err) = ecj.set_len(prev_ecj_size as u64)
{
tracing::error!(
volume_id = self.volume_id.0,
needle_id = needle_id.0,
truncate_error = %trunc_err,
"failed to truncate ecj after append failure"
);
}
Err(e)
}
@@ -1550,7 +1541,7 @@ impl EcVolume {
let ecx_file = self
.ecx_file
.as_ref()
.ok_or_else(|| io::Error::new(io::ErrorKind::Other, "ecx file not open"))?;
.ok_or_else(|| io::Error::other("ecx file not open"))?;
let entry_count = self.ecx_file_size as usize / NEEDLE_MAP_ENTRY_SIZE;
if entry_count == 0 {
return Ok(None);
@@ -1590,31 +1581,32 @@ impl EcVolume {
if cookie.0 != 0 {
// Try to read the needle's cookie from the EC shards to validate
// Look up the needle in ecx index to find its offset, then read header from shard
if let Ok(Some((offset, size))) = self.find_needle_from_ecx(needle_id) {
if !size.is_deleted() && !offset.is_zero() {
let actual_offset = offset.to_actual_offset() as u64;
// Determine which shard contains this offset and read the cookie
let shard_size = self
.shards
.iter()
.filter_map(|s| s.as_ref())
.map(|s| s.file_size())
.next()
.unwrap_or(0) as u64;
if shard_size > 0 {
let shard_id = (actual_offset / shard_size) as usize;
let shard_offset = actual_offset % shard_size;
if let Some(Some(shard)) = self.shards.get(shard_id) {
let mut header_buf = [0u8; 4]; // cookie is first 4 bytes of needle
if shard.read_at(&mut header_buf, shard_offset).is_ok() {
let needle_cookie =
crate::storage::types::Cookie(u32::from_be_bytes(header_buf));
if needle_cookie != cookie {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("unexpected cookie {:x}", cookie.0),
));
}
if let Ok(Some((offset, size))) = self.find_needle_from_ecx(needle_id)
&& !size.is_deleted()
&& !offset.is_zero()
{
let actual_offset = offset.to_actual_offset() as u64;
// Determine which shard contains this offset and read the cookie
let shard_size = self
.shards
.iter()
.filter_map(|s| s.as_ref())
.map(|s| s.file_size())
.next()
.unwrap_or(0) as u64;
if let Some(shard_id) = actual_offset.checked_div(shard_size) {
let shard_id = shard_id as usize;
let shard_offset = actual_offset % shard_size;
if let Some(Some(shard)) = self.shards.get(shard_id) {
let mut header_buf = [0u8; 4]; // cookie is first 4 bytes of needle
if shard.read_at(&mut header_buf, shard_offset).is_ok() {
let needle_cookie =
crate::storage::types::Cookie(u32::from_be_bytes(header_buf));
if needle_cookie != cookie {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("unexpected cookie {:x}", cookie.0),
));
}
}
}
@@ -3742,10 +3734,10 @@ pub(crate) fn merge_ec_runtimes<'a>(runtimes: &[&'a EcVolume]) -> Option<MergedE
let mut slots: Vec<Option<(&'a EcVolume, &'a EcVolumeShard)>> = vec![None; width];
for v in &merged {
for (id, slot) in v.shards.iter().enumerate() {
if let Some(shard) = slot.as_ref() {
if slots[id].is_none() {
slots[id] = Some((*v, shard));
}
if let Some(shard) = slot.as_ref()
&& slots[id].is_none()
{
slots[id] = Some((*v, shard));
}
}
}
@@ -4366,13 +4358,10 @@ impl EcLocalScrubPlan {
if read != want {
// Like Go, returning from the walk callback aborts the scan.
return Err(io::Error::new(
io::ErrorKind::Other,
format!(
"expected {} bytes for needle {} on volume {}, got {}",
want, id.0, volume_id.0, read
),
));
return Err(io::Error::other(format!(
"expected {} bytes for needle {} on volume {}, got {}",
want, id.0, volume_id.0, read
)));
}
// Only a fully-local needle can be reassembled and CRC-checked.
@@ -4405,7 +4394,7 @@ impl EcLocalScrubPlan {
.filter_map(|sid| shards.get(*sid as usize).and_then(|s| s.as_ref()))
.map(|s| s.info.clone())
.collect();
broken.sort_by(|a, b| a.shard_id.cmp(&b.shard_id));
broken.sort_by_key(|a| a.shard_id);
(count, broken, errs)
}