mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-19 10:54:08 +01:00
* fix(seaweed-volume): fall back to idx dir when reading .vif
EcVolume::new and read_ec_shard_config only looked for .vif at the
data dir. With the cross-disk reconcile path (where shards live on
one disk and .ecx / .ecj / .vif live on a sibling disk —
seaweedfs/seaweedfs#9212 / #9244), this would either write a stub
.vif on the shard disk and lose the real EC config + dat_file_size
or fall back to default ratios despite a perfectly good .vif being
present elsewhere on the same volume server.
Add a small `locate_vif_path` helper that prefers the data dir and
falls back to the idx dir when it differs, and thread the data dir
+ idx dir pair through `read_ec_shard_config`. Three call sites in
grpc_server.rs (VolumeEcShardsGenerate, VolumeEcShardsRebuild, scrub)
updated; the scrub path passes the same dir for both args because
`find_ec_dir` is the only locator there.
* feat(seaweed-volume): primitives for cross-disk EC shard reconcile
Adds the three small helpers the reconcile pass needs:
- DiskLocation::mount_ec_shards_with_idx_dir — mounts shards on this
disk while pointing the EcVolume at a sibling disk's idx dir for
.ecx / .ecj / .vif. Mirrors loadEcShardsWithIdxDir in
weed/storage/disk_location_ec.go. The existing mount_ec_shards is
kept as a thin wrapper over it.
- EcVolume::has_shard — `pub` accessor over the internal Vec<Option>
shard slot so the reconcile pass can skip shards that are already
registered.
- pub(crate) re-exports of parse_collection_volume_id and
parse_ec_shard_extension under names parse_collection_volume_id_pub
and is_ec_shard_extension so the reconcile module can call them
without re-implementing the parsers.
No behaviour change. Reconciliation logic in the next commit.
* feat(seaweed-volume): cross-disk EC shard reconciliation (#9212)
Closes the loader half of seaweedfs/seaweedfs#9212 on the Rust side,
mirroring the Go fix in seaweedfs/seaweedfs#9244. With the auto-load
in feat/rust-load-all-ec-shards-9212 in place, the only remaining gap
is shards that landed on a disk without their `.ecx` — for example
when ec.balance / ec.rebuild moved them onto a destination node's
second disk while leaving the index files on the disk that already
held the volume. Without this, those orphan shards stay invisible to
the master and ec.rebuild reports the volume as unrepairable.
After every DiskLocation has finished its per-disk EC scan, sweep the
store for shards that live on a disk without local index files and
load them by reaching across to a sibling disk's `.ecx` / `.ecj` /
`.vif`:
- Store::reconcile_ec_shards_across_disks walks each disk for
orphan `.ec??` files (present on disk, not yet registered to an
EcVolume) and matches them against an `(collection, vid) ->
EcxOwnerInfo` map of which disk owns each `.ecx`.
- Each matched group is mounted on its physical disk's ec_volumes
map (so heartbeat reporting carries the right disk_id per shard)
via `mount_ec_shards_with_idx_dir`, pointing the EcVolume at the
sibling's idx dir.
- `index_ecx_owners` records the directory each `.ecx` was found in
(IdxDirectory or Directory) so the loader doesn't ENOENT when the
legacy "written before -dir.idx was set" layout puts `.ecx` in
the data dir. This mirrors the PR #9244 review fix from
@gemini-code-assist / @coderabbitai (see Go commit af57cc652).
- True orphans (no `.ecx` anywhere on this server) log a warning
and stay on disk untouched — operator can restore the index later.
Wired into Store::add_location and Store::load_new_volumes so a fresh
restart and any later disk additions both pick up cross-disk shards.
Tests cover all four behaviour shapes:
- shards on dir0 + .ecx on dir1 → reconciled to dir0's ec_volumes
- .ecx in owner's data dir (legacy layout) → reconciled correctly
- self-contained disks → reconcile is a no-op
- truly-orphan shards (no .ecx anywhere) → left on disk, logged
* fix(seaweed-volume): propagate EcVolume::new errors instead of unwrap
mount_ec_shards_with_idx_dir built the missing EcVolume inside an
entry().or_insert_with() closure, which can't return a Result — so
any EcVolume::new failure (e.g. .ecx open error, .ecj create error,
malformed .vif) panicked the volume server via unwrap(). The
constructor already returns Result<>, so propagate it as
VolumeError::Io instead.
Reported in PR #9252 review by @gemini-code-assist (high) and
@coderabbitai (critical).
* perf(seaweed-volume): use DirEntry::metadata in collect_orphan_ec_shards
Replaced the extra fs::metadata(&path) lookup with ent.metadata() so
we don't pay an additional stat syscall per directory entry beyond
what read_dir already returned. Drops the now-unused std::path::Path
import alongside.
Reported in PR #9252 review by @gemini-code-assist.
* fix(seaweed-volume): scrub uses EcVolume's real dir_idx for split-disk volumes
After cross-disk reconciliation an EcVolume can legitimately have
ecv.dir != ecv.dir_idx (shards on one disk, .ecx / .ecj / .vif on a
sibling). The scrub path collapsed both args to find_ec_dir's single
answer, so read_ec_shard_config fell back to the wrong .vif location
for exactly the split-disk layout this PR loads — skewing
shard-count detection and verification results.
Use ecv.dir / ecv.dir_idx directly so scrub reads the metadata from
where the volume's index files actually live.
Reported in PR #9252 review by @coderabbitai.
* feat(seaweed-volume): primitives for split-disk EC volume operations
Reconciliation can mount the same `vid` on multiple DiskLocations
with disjoint shard subsets. The existing first-match `find_ec_volume`
isn't enough for read/unmount/delete/decode paths that need to act on
a specific shard or aggregate across the whole volume — they have to
walk every location and find the right home for each shard.
Add the small Store-level lookup primitives Go's findEcShard /
CollectEcShards already provide:
- `Store::find_ec_shard_location(vid, shard_id)` — returns the index
of the location that has `(vid, shard_id)` mounted, if any.
- `Store::find_ec_volume_with_shard(vid, shard_id)` — same idea but
returns the EcVolume directly.
- `Store::collect_ec_shard_dirs(vid, max_shard_count)` — returns
the EcVolume to use for metadata plus per-shard data dirs (None
when the shard isn't mounted on any disk). Mirrors
`Store.CollectEcShards` in `weed/storage/store_ec.go`.
And the EcVolume accessors callers need:
- `EcVolume::has_shard(shard_id)` — was already added for the cross-
disk reconcile but is now a load-bearing primitive for placement
decisions on a per-shard basis. Pulled into the dedicated commit.
- `EcVolume::ecx_actual_dir()` — exposes the directory the `.ecx`
was actually opened from. The decoder needs it for the .ecx
lookup when shards are split across data dirs and `.ecx` lives on
a sibling idx dir.
Plus a small defensive change to `DiskLocation::unmount_ec_shards`:
only decrement the per-shard gauge for shards that were actually
mounted. Without this, the upcoming `Store::unmount_ec_shards`
fan-out to every location would underflow the metric whenever a
shard is requested for unmount on a sibling disk that doesn't have
it.
No behaviour change at the call sites yet — wiring follows in the
next commits.
* fix(seaweed-volume): unmount_ec_shards visits every location with the vid
Store::unmount_ec_shards and Store::unmount_ec_shard returned after
the first DiskLocation with the volume id, even if that location did
not contain the requested shard. With reconciled split-disk volumes
(shards 0/12 on disk 0, shard 1 on disk 1 — the issue #9212 layout
this PR loads), VolumeEcShardsUnmount for a later-disk shard became a
silent no-op and Store::delete_ec_shards could remove the shard file
while leaving an in-memory shard + open file handle stale on the
later location.
Walk all locations that have the EcVolume and ask each to unmount
whatever subset of `shard_ids` it actually has — the
`DiskLocation::unmount_ec_shards` defensive guard from the previous
commit makes the fan-out safe (no metric underflow when a sibling
disk is asked to unmount a shard it doesn't hold).
* fix(seaweed-volume): VolumeEcShardRead reads from the shard's home disk
VolumeEcShardRead resolved the EcVolume via first-match
`find_ec_volume(vid)` and then looked up the requested shard on that
single EcVolume. With reconciled split-disk volumes (the layout
seaweedfs/seaweedfs#9212 produces — shards 0/12 on disk 0, shard 1
on disk 1), a request for shard 1 hit disk 0 first and returned
"shard 1 not mounted" even though it was happily mounted on disk 1.
Switch to `find_ec_volume_with_shard(vid, shard_id)` so the lookup
walks every location and returns the EcVolume whose disk actually
holds the shard. The deleted-needle check still works because every
per-disk EcVolume for the same vid points at the same `.ecx` file
(post-reconcile, both disks open the same sealed index).
* fix(seaweed-volume): VolumeEcShardsToVolume aggregates shards across disks
VolumeEcShardsToVolume resolved a single EcVolume via
`find_ec_volume(vid)` and then checked `ec_vol.shards[i]` for each
data shard. With reconciled split-disk volumes that's the wrong
view: the first-match EcVolume only carries the shards on its disk,
so the presence check would either reject the request as
"missing shard" or — if shards happened to be on the first disk —
fall through to `write_dat_file_from_shards(&dir, ...)` which only
reads from the EcVolume's single dir.
Mirror Go's CollectEcShards by aggregating per-shard data dirs
across every location with the volume:
- Add `Store::collect_ec_shard_dirs` (in the previous primitives
commit) returning the EcVolume to use for metadata + per-shard
dir slots.
- Extend `find_dat_file_size` and `write_dat_file_from_shards` with
`_with_dirs` variants that take the `.ec00` dir and per-shard
dirs separately, so a decoded volume whose shards live on
several disks can still be reconstructed. The original signatures
delegate to the new ones with the same dir for all shards, so
every existing caller keeps working unchanged.
- Rewire VolumeEcShardsToVolume through the helpers — presence
check sees the union, dat_file_size reads `.ec00` from the right
disk and `.ecx` from the EcVolume's actual idx dir, the decoder
reads each shard from its own home dir.
* test(seaweed-volume): split-disk read / unmount / delete / collect
Five tests exercising the four behaviour shapes the PR #9252 review
flagged on multi-location EC volumes. Each builds the cross-disk
split layout from issue #9212 (shards 0 and 12 on disk 0, shard 1 +
.ecx on disk 1) via the new `build_split_disk_store` helper and
asserts:
- `find_ec_shard_location` / `find_ec_volume_with_shard` route to
the disk that actually holds each shard (not first-match).
- `Store::unmount_ec_shards([1])` reaches disk 1 and removes shard 1
while leaving disk 0's unrelated shards mounted (used to be a
silent no-op).
- `Store::unmount_ec_shard(vid, 1)` ditto for the single-shard
variant.
- `Store::delete_ec_shards` removes both the on-disk file and the
in-memory mount on the right disk; previously deletion could
remove the file while the in-memory shard with its open file
handle survived on a different location.
- `collect_ec_shard_dirs` reports the right per-shard data dir for
each location and `None` for unmounted shards.
* fix(seaweed-volume): retry same-disk legacy .ecx layout in reconcile
The unconditional `owner.location == loc_idx` skip missed the layout
where `idx_directory` is configured but the owner's `.ecx` / `.ecj` /
`.vif` still live in `loc.directory` (the legacy "written before
-dir.idx was set" shape). In that case the per-disk loader's
mount_ec_shards used `loc.idx_directory` and ENOENT'd, then this
branch suppressed the only recovery path — the owner disk's own
shards stayed unloaded after startup.
Tighten the skip so it only fires when the discovered owner dir is
already `loc.idx_directory` (the loader-already-tried-and-failed
case). When `owner.idx_dir` differs (legacy data-dir layout), queue
a same-disk retry through `mount_ec_shards_with_idx_dir(...,
&owner.idx_dir)` so reconcile becomes the recovery path.
Reported in PR #9252 review by @coderabbitai.
* fix(seaweed-volume): roll back partial mounts on cross-disk reconcile failure
mount_ec_shards_with_idx_dir adds shards one at a time and
increments the `ec_shards` gauge per shard that successfully attaches.
A mid-loop failure (e.g. an EcVolumeShard::open error after the
first few shards already attached) used to leave the EcVolume
half-mounted with stale metric increments — the warn!() branch only
logged the error.
Mirror DiskLocation::handle_found_ecx_file's recovery path: drive
the cleanup through `loc.unmount_ec_shards(vid, &shard_ids)` after
a failed mount. The defensive change in #9251 makes
unmount_ec_shards only decrement the gauge for shards that were
actually mounted and drops the EcVolume when it reaches zero
shards, so the rollback is safe even though some of `shard_ids`
never attached.
Reported in PR #9252 review by @coderabbitai.
* test(seaweed-volume): cover the two reconcile fixes from PR #9252 review
Two new tests in store_ec_reconcile:
- test_reconcile_recovers_same_disk_legacy_ecx_layout — sets up the
layout where idx_directory is configured but the owner's .ecx
lives in loc.directory. The per-disk loader's mount_ec_shards
uses loc.idx_directory and fails; reconcile should retry on the
same disk with the owner's actual idx_dir and the owner's own
shards must come back online.
- test_reconcile_rolls_back_partial_mounts_on_failure — sabotages
one of the orphan shard files (replaces it with a directory of
the same name) so EcVolumeShard::open errors out partway through
mount_ec_shards_with_idx_dir. Asserts the post-condition that no
EcVolume entry retains a "shard mounted" claim that doesn't
correspond to a real shard file.
1229 lines
46 KiB
Rust
1229 lines
46 KiB
Rust
//! EcVolume: an erasure-coded volume with up to 14 shards.
|
|
//!
|
|
//! Each EcVolume has a sorted index (.ecx) and a deletion journal (.ecj).
|
|
//! Shards (.ec00-.ec13) may be distributed across multiple servers.
|
|
|
|
use std::collections::{HashMap, HashSet};
|
|
use std::fs::{self, File, OpenOptions};
|
|
use std::io::{self, Write};
|
|
use std::sync::RwLock;
|
|
use std::time::{SystemTime, UNIX_EPOCH};
|
|
|
|
use crate::pb::master_pb;
|
|
use crate::storage::erasure_coding::ec_locate;
|
|
use crate::storage::erasure_coding::ec_shard::*;
|
|
use crate::storage::needle::needle::{get_actual_size, Needle};
|
|
use crate::storage::types::*;
|
|
|
|
/// An erasure-coded volume managing its local shards and index.
|
|
pub struct EcVolume {
|
|
pub volume_id: VolumeId,
|
|
pub collection: String,
|
|
pub dir: String,
|
|
pub dir_idx: String,
|
|
pub version: Version,
|
|
pub shards: Vec<Option<EcVolumeShard>>, // indexed by ShardId (0..14)
|
|
pub dat_file_size: i64,
|
|
pub data_shards: u32,
|
|
pub parity_shards: u32,
|
|
ecx_file: Option<File>,
|
|
ecx_file_size: i64,
|
|
ecj_file: Option<File>,
|
|
/// On-disk size of the .ecj deletion journal. Used only by IO helpers
|
|
/// (seek / set_len on partial writes) — the authoritative runtime
|
|
/// delete count comes from `deleted_needles.len()`.
|
|
ecj_file_size: i64,
|
|
/// In-memory set of needle ids that have been deleted since the volume
|
|
/// was encoded. .ecx is immutable at runtime — it only stores the
|
|
/// sorted (id, offset, size) index written at encode time — and runtime
|
|
/// deletes are journaled to .ecj + tracked here. Reads consult this
|
|
/// set to mask out deleted needles on top of the sealed .ecx lookup.
|
|
/// Seeded from .ecj in `new()` and updated by `journal_delete`.
|
|
deleted_needles: RwLock<HashSet<NeedleId>>,
|
|
pub disk_type: DiskType,
|
|
/// Directory where .ecx/.ecj were actually found (may differ from dir_idx after fallback).
|
|
ecx_actual_dir: String,
|
|
/// Maps shard ID -> list of server addresses where that shard exists.
|
|
/// Used for distributed EC reads across the cluster.
|
|
pub shard_locations: HashMap<ShardId, Vec<String>>,
|
|
/// EC volume expiration time (unix epoch seconds), set during EC encode from TTL.
|
|
pub expire_at_sec: u64,
|
|
}
|
|
|
|
/// Locate the `.vif` for a (collection, vid) by preferring the data dir
|
|
/// and falling back to the idx dir when it lives there instead. The
|
|
/// fallback covers the cross-disk reconcile path: when a volume's
|
|
/// shards live on one disk but its `.ecx` / `.ecj` / `.vif` live on a
|
|
/// sibling disk (seaweedfs/seaweedfs#9212 / #9244), we want to read the
|
|
/// real `.vif` from the sibling rather than write a stub on the shard
|
|
/// disk and lose the EC config + dat file size.
|
|
fn locate_vif_path(dir: &str, dir_idx: &str, collection: &str, volume_id: VolumeId) -> String {
|
|
let data_vif = format!(
|
|
"{}.vif",
|
|
crate::storage::volume::volume_file_name(dir, collection, volume_id),
|
|
);
|
|
if dir_idx != dir && !std::path::Path::new(&data_vif).exists() {
|
|
let idx_vif = format!(
|
|
"{}.vif",
|
|
crate::storage::volume::volume_file_name(dir_idx, collection, volume_id),
|
|
);
|
|
if std::path::Path::new(&idx_vif).exists() {
|
|
return idx_vif;
|
|
}
|
|
}
|
|
data_vif
|
|
}
|
|
|
|
/// Read EC data/parity shard counts from `.vif`, defaulting to the
|
|
/// build's standard ratio when no `.vif` is present or is malformed.
|
|
/// Looks at the data dir first, then the idx dir — see [`locate_vif_path`].
|
|
pub fn read_ec_shard_config(
|
|
dir: &str,
|
|
dir_idx: &str,
|
|
collection: &str,
|
|
volume_id: VolumeId,
|
|
) -> (u32, u32) {
|
|
let mut data_shards = crate::storage::erasure_coding::ec_shard::DATA_SHARDS_COUNT as u32;
|
|
let mut parity_shards = crate::storage::erasure_coding::ec_shard::PARITY_SHARDS_COUNT as u32;
|
|
let vif_path = locate_vif_path(dir, dir_idx, collection, volume_id);
|
|
if let Ok(vif_content) = std::fs::read_to_string(&vif_path) {
|
|
if let Ok(vif_info) =
|
|
serde_json::from_str::<crate::storage::volume::VifVolumeInfo>(&vif_content)
|
|
{
|
|
if let Some(ec) = vif_info.ec_shard_config {
|
|
if ec.data_shards > 0
|
|
&& ec.parity_shards > 0
|
|
&& (ec.data_shards + ec.parity_shards) <= TOTAL_SHARDS_COUNT as u32
|
|
{
|
|
data_shards = ec.data_shards;
|
|
parity_shards = ec.parity_shards;
|
|
}
|
|
}
|
|
}
|
|
}
|
|
(data_shards, parity_shards)
|
|
}
|
|
|
|
impl EcVolume {
|
|
/// Create a new EcVolume. Loads .ecx index and .ecj journal if present.
|
|
pub fn new(
|
|
dir: &str,
|
|
dir_idx: &str,
|
|
collection: &str,
|
|
volume_id: VolumeId,
|
|
) -> io::Result<Self> {
|
|
let (data_shards, parity_shards) = read_ec_shard_config(dir, dir_idx, collection, volume_id);
|
|
|
|
let total_shards = (data_shards + parity_shards) as usize;
|
|
let mut shards = Vec::with_capacity(total_shards);
|
|
for _ in 0..total_shards {
|
|
shards.push(None);
|
|
}
|
|
|
|
// Read expire_at_sec and version from .vif if present (matches Go's MaybeLoadVolumeInfo).
|
|
// Prefer the data dir; fall back to the idx dir for the
|
|
// cross-disk reconcile case (#9212 / #9244).
|
|
let (expire_at_sec, vif_version) = {
|
|
let vif_path = locate_vif_path(dir, dir_idx, collection, volume_id);
|
|
if let Ok(vif_content) = std::fs::read_to_string(&vif_path) {
|
|
if let Ok(vif_info) =
|
|
serde_json::from_str::<crate::storage::volume::VifVolumeInfo>(&vif_content)
|
|
{
|
|
let ver = if vif_info.version > 0 {
|
|
Version(vif_info.version as u8)
|
|
} else {
|
|
Version::current()
|
|
};
|
|
(vif_info.expire_at_sec, ver)
|
|
} else {
|
|
(0, Version::current())
|
|
}
|
|
} else {
|
|
(0, Version::current())
|
|
}
|
|
};
|
|
|
|
let mut vol = EcVolume {
|
|
volume_id,
|
|
collection: collection.to_string(),
|
|
dir: dir.to_string(),
|
|
dir_idx: dir_idx.to_string(),
|
|
version: vif_version,
|
|
shards,
|
|
dat_file_size: 0,
|
|
data_shards,
|
|
parity_shards,
|
|
ecx_file: None,
|
|
ecx_file_size: 0,
|
|
ecj_file: None,
|
|
ecj_file_size: 0,
|
|
deleted_needles: RwLock::new(HashSet::new()),
|
|
disk_type: DiskType::default(),
|
|
ecx_actual_dir: dir_idx.to_string(),
|
|
shard_locations: HashMap::new(),
|
|
expire_at_sec,
|
|
};
|
|
|
|
// Open .ecx file (sorted index) in read/write mode for in-place deletion marking.
|
|
// Matches Go which opens ecx for writing via MarkNeedleDeleted.
|
|
let ecx_path = vol.ecx_file_name();
|
|
if std::path::Path::new(&ecx_path).exists() {
|
|
let file = OpenOptions::new().read(true).write(true).open(&ecx_path)?;
|
|
vol.ecx_file_size = file.metadata()?.len() as i64;
|
|
vol.ecx_file = Some(file);
|
|
} else if dir_idx != dir {
|
|
// Fall back to data directory if .ecx was created before -dir.idx was configured
|
|
let data_base = crate::storage::volume::volume_file_name(dir, collection, volume_id);
|
|
let fallback_ecx = format!("{}.ecx", data_base);
|
|
if std::path::Path::new(&fallback_ecx).exists() {
|
|
tracing::info!(
|
|
volume_id = volume_id.0,
|
|
"ecx file not found in idx dir, falling back to data dir"
|
|
);
|
|
let file = OpenOptions::new().read(true).write(true).open(&fallback_ecx)?;
|
|
vol.ecx_file_size = file.metadata()?.len() as i64;
|
|
vol.ecx_file = Some(file);
|
|
vol.ecx_actual_dir = dir.to_string();
|
|
}
|
|
}
|
|
|
|
// Open .ecj file (deletion journal) — use ecx_actual_dir for consistency.
|
|
// Note: Go does NOT replay .ecj into .ecx at volume load (RebuildEcxFile
|
|
// is only invoked from specific decode/rebuild gRPC handlers), so we
|
|
// don't either. Tombstones from prior sessions were already written
|
|
// in-place in .ecx, and the journal grows monotonically until a
|
|
// decode/rebuild operation folds it in.
|
|
let ecj_base =
|
|
crate::storage::volume::volume_file_name(&vol.ecx_actual_dir, collection, volume_id);
|
|
let ecj_path = format!("{}.ecj", ecj_base);
|
|
let ecj_file = OpenOptions::new()
|
|
.read(true)
|
|
.write(true)
|
|
.create(true)
|
|
.append(true)
|
|
.open(&ecj_path)?;
|
|
vol.ecj_file_size = ecj_file.metadata()?.len() as i64;
|
|
vol.ecj_file = Some(ecj_file);
|
|
|
|
// Seed the in-memory deleted set from the journal.
|
|
vol.load_deleted_needles_from_ecj()?;
|
|
|
|
Ok(vol)
|
|
}
|
|
|
|
/// Walk the .ecj journal and populate `deleted_needles`. Called once
|
|
/// from `new()` under exclusive ownership of the just-constructed
|
|
/// EcVolume, so locking is not strictly required — but we take the
|
|
/// write lock anyway for symmetry with later mutations.
|
|
fn load_deleted_needles_from_ecj(&mut self) -> io::Result<()> {
|
|
let ecj_file = match self.ecj_file.as_ref() {
|
|
Some(f) => f,
|
|
None => return Ok(()),
|
|
};
|
|
if self.ecj_file_size < NEEDLE_ID_SIZE as i64 {
|
|
return Ok(());
|
|
}
|
|
let mut buf = [0u8; NEEDLE_ID_SIZE];
|
|
let mut set = self
|
|
.deleted_needles
|
|
.write()
|
|
.map_err(|_| io::Error::new(io::ErrorKind::Other, "deleted_needles lock poisoned"))?;
|
|
let mut off: i64 = 0;
|
|
while off + NEEDLE_ID_SIZE as i64 <= self.ecj_file_size {
|
|
#[cfg(unix)]
|
|
{
|
|
use std::os::unix::fs::FileExt;
|
|
ecj_file.read_exact_at(&mut buf, off as u64)?;
|
|
}
|
|
set.insert(NeedleId::from_bytes(&buf));
|
|
off += NEEDLE_ID_SIZE as i64;
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
/// Returns (file_count, delete_count) for this EC volume. Mirrors Go's
|
|
/// `EcVolume.FileAndDeleteCount`:
|
|
///
|
|
/// file_count = ecx_file_size / NEEDLE_MAP_ENTRY_SIZE — total
|
|
/// entries in the sealed sorted .ecx index.
|
|
/// delete_count = deleted_needles.len() — unique
|
|
/// runtime deletes tracked in memory (seeded from
|
|
/// .ecj on load and updated by `journal_delete`).
|
|
///
|
|
/// Because each needle delete is applied on exactly one shard holder,
|
|
/// the admin aggregation sums delete_count across nodes while taking
|
|
/// file_count from a single holder (they are identical per volume).
|
|
pub fn file_and_delete_count(&self) -> (u64, u64) {
|
|
let file_count = (self.ecx_file_size as u64) / (NEEDLE_MAP_ENTRY_SIZE as u64);
|
|
let delete_count = self
|
|
.deleted_needles
|
|
.read()
|
|
.map(|s| s.len() as u64)
|
|
.unwrap_or(0);
|
|
(file_count, delete_count)
|
|
}
|
|
|
|
/// Reports whether the given needle id is in the in-memory deleted set.
|
|
pub fn is_needle_deleted(&self, needle_id: NeedleId) -> bool {
|
|
self.deleted_needles
|
|
.read()
|
|
.map(|s| s.contains(&needle_id))
|
|
.unwrap_or(false)
|
|
}
|
|
|
|
// ---- File names ----
|
|
|
|
#[allow(dead_code)]
|
|
fn base_name(&self) -> String {
|
|
crate::storage::volume::volume_file_name(&self.dir, &self.collection, self.volume_id)
|
|
}
|
|
|
|
/// Base path for the .ecx / .ecj index pair. Resolved from
|
|
/// `ecx_actual_dir` (initialized to `dir_idx` and only updated after a
|
|
/// successful idx-dir → data-dir fallback in `new()`), so every call site
|
|
/// agrees on the same file regardless of whether the fallback fired.
|
|
fn idx_base_name(&self) -> String {
|
|
crate::storage::volume::volume_file_name(
|
|
&self.ecx_actual_dir,
|
|
&self.collection,
|
|
self.volume_id,
|
|
)
|
|
}
|
|
|
|
pub fn ecx_file_name(&self) -> String {
|
|
format!("{}.ecx", self.idx_base_name())
|
|
}
|
|
|
|
pub fn ecj_file_name(&self) -> String {
|
|
format!("{}.ecj", self.idx_base_name())
|
|
}
|
|
|
|
/// Sync the EC volume's journal and index files to disk (matching Go's ecv.Sync()).
|
|
/// Go flushes both .ecj and .ecx to ensure in-place deletion marks are persisted.
|
|
pub fn sync_to_disk(&self) -> io::Result<()> {
|
|
if let Some(ref ecj_file) = self.ecj_file {
|
|
ecj_file.sync_all()?;
|
|
}
|
|
if let Some(ref ecx_file) = self.ecx_file {
|
|
ecx_file.sync_all()?;
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
// ---- Shard management ----
|
|
|
|
/// Add a shard to this volume.
|
|
pub fn add_shard(&mut self, mut shard: EcVolumeShard) -> io::Result<()> {
|
|
let id = shard.shard_id as usize;
|
|
let total_shards = (self.data_shards + self.parity_shards) as usize;
|
|
if id >= total_shards {
|
|
return Err(io::Error::new(
|
|
io::ErrorKind::InvalidInput,
|
|
format!("invalid shard id: {} (max {})", id, total_shards - 1),
|
|
));
|
|
}
|
|
shard.open()?;
|
|
self.shards[id] = Some(shard);
|
|
Ok(())
|
|
}
|
|
|
|
/// Remove and close a shard.
|
|
pub fn remove_shard(&mut self, shard_id: ShardId) {
|
|
if let Some(ref mut shard) = self.shards[shard_id as usize] {
|
|
shard.close();
|
|
}
|
|
self.shards[shard_id as usize] = None;
|
|
}
|
|
|
|
/// Get a ShardBits bitmap of locally available shards.
|
|
pub fn shard_bits(&self) -> ShardBits {
|
|
let mut bits = ShardBits::default();
|
|
for (i, shard) in self.shards.iter().enumerate() {
|
|
if shard.is_some() {
|
|
bits.add_shard_id(i as ShardId);
|
|
}
|
|
}
|
|
bits
|
|
}
|
|
|
|
/// Count of locally available shards.
|
|
pub fn shard_count(&self) -> usize {
|
|
self.shards.iter().filter(|s| s.is_some()).count()
|
|
}
|
|
|
|
/// Reports whether `shard_id` is currently registered to this
|
|
/// EcVolume (used by the cross-disk reconcile to skip already-
|
|
/// loaded shards).
|
|
pub fn has_shard(&self, shard_id: u8) -> bool {
|
|
self.shards
|
|
.get(shard_id as usize)
|
|
.map(|s| s.is_some())
|
|
.unwrap_or(false)
|
|
}
|
|
|
|
/// Directory where this EcVolume's `.ecx` was actually opened
|
|
/// (may differ from `dir_idx` when the legacy "written before
|
|
/// -dir.idx was set" fallback or the cross-disk reconcile path
|
|
/// pointed it elsewhere).
|
|
pub fn ecx_actual_dir(&self) -> &str {
|
|
&self.ecx_actual_dir
|
|
}
|
|
|
|
pub fn is_time_to_destroy(&self) -> bool {
|
|
self.expire_at_sec > 0
|
|
&& SystemTime::now()
|
|
.duration_since(UNIX_EPOCH)
|
|
.unwrap_or_default()
|
|
.as_secs()
|
|
> self.expire_at_sec
|
|
}
|
|
|
|
pub fn to_volume_ec_shard_information_messages(
|
|
&self,
|
|
disk_id: u32,
|
|
) -> Vec<master_pb::VolumeEcShardInformationMessage> {
|
|
let mut ec_index_bits: u32 = 0;
|
|
let mut shard_sizes = Vec::new();
|
|
for shard in self.shards.iter().flatten() {
|
|
ec_index_bits |= 1u32 << shard.shard_id;
|
|
shard_sizes.push(shard.file_size());
|
|
}
|
|
|
|
if ec_index_bits == 0 {
|
|
return Vec::new();
|
|
}
|
|
|
|
let (file_count, delete_count) = self.file_and_delete_count();
|
|
|
|
vec![master_pb::VolumeEcShardInformationMessage {
|
|
id: self.volume_id.0,
|
|
collection: self.collection.clone(),
|
|
ec_index_bits,
|
|
shard_sizes,
|
|
disk_type: self.disk_type.to_string(),
|
|
expire_at_sec: self.expire_at_sec,
|
|
disk_id,
|
|
file_count,
|
|
delete_count,
|
|
..Default::default()
|
|
}]
|
|
}
|
|
|
|
// ---- Shard locations (distributed tracking) ----
|
|
|
|
/// Set the list of server addresses for a given shard ID.
|
|
pub fn set_shard_locations(&mut self, shard_id: ShardId, locations: Vec<String>) {
|
|
self.shard_locations.insert(shard_id, locations);
|
|
}
|
|
|
|
/// Get the list of server addresses for a given shard ID.
|
|
pub fn get_shard_locations(&self, shard_id: ShardId) -> &[String] {
|
|
self.shard_locations
|
|
.get(&shard_id)
|
|
.map(|v| v.as_slice())
|
|
.unwrap_or(&[])
|
|
}
|
|
|
|
// ---- Index operations ----
|
|
|
|
/// Find a needle's offset and size in the sorted .ecx index via binary search.
|
|
pub fn find_needle_from_ecx(&self, needle_id: NeedleId) -> io::Result<Option<(Offset, Size)>> {
|
|
let ecx_file = self
|
|
.ecx_file
|
|
.as_ref()
|
|
.ok_or_else(|| io::Error::new(io::ErrorKind::Other, "ecx file not open"))?;
|
|
|
|
let entry_count = self.ecx_file_size as usize / NEEDLE_MAP_ENTRY_SIZE;
|
|
if entry_count == 0 {
|
|
return Ok(None);
|
|
}
|
|
|
|
// Binary search
|
|
let mut lo: usize = 0;
|
|
let mut hi: usize = entry_count;
|
|
let mut entry_buf = [0u8; NEEDLE_MAP_ENTRY_SIZE];
|
|
|
|
while lo < hi {
|
|
let mid = lo + (hi - lo) / 2;
|
|
let file_offset = (mid * NEEDLE_MAP_ENTRY_SIZE) as u64;
|
|
|
|
#[cfg(unix)]
|
|
{
|
|
use std::os::unix::fs::FileExt;
|
|
ecx_file.read_exact_at(&mut entry_buf, file_offset)?;
|
|
}
|
|
|
|
let (key, offset, size) = idx_entry_from_bytes(&entry_buf);
|
|
if key == needle_id {
|
|
// Apply runtime deletion state on top of the sealed .ecx
|
|
// lookup: a needle in the in-memory deleted set is
|
|
// reported with TOMBSTONE_FILE_SIZE even though the .ecx
|
|
// record itself is untouched.
|
|
if self.is_needle_deleted(needle_id) {
|
|
return Ok(Some((offset, TOMBSTONE_FILE_SIZE)));
|
|
}
|
|
return Ok(Some((offset, size)));
|
|
} else if key < needle_id {
|
|
lo = mid + 1;
|
|
} else {
|
|
hi = mid;
|
|
}
|
|
}
|
|
|
|
Ok(None)
|
|
}
|
|
|
|
/// Locate the EC shard intervals needed to read a needle.
|
|
pub fn locate_needle(
|
|
&self,
|
|
needle_id: NeedleId,
|
|
) -> io::Result<Option<(Offset, Size, Vec<ec_locate::Interval>)>> {
|
|
let (offset, size) = match self.find_needle_from_ecx(needle_id)? {
|
|
Some((o, s)) => (o, s),
|
|
None => return Ok(None),
|
|
};
|
|
|
|
if size.is_deleted() || offset.is_zero() {
|
|
return Ok(None);
|
|
}
|
|
|
|
// Match Go's LocateEcShardNeedleInterval: shardSize = shard.ecdFileSize - 1
|
|
// Shards are usually padded to ErasureCodingSmallBlockSize, so subtract 1
|
|
// to avoid off-by-one in large block row count calculation.
|
|
// If datFileSize is known, use datFileSize / DataShards instead.
|
|
let shard_size = if self.dat_file_size > 0 {
|
|
self.dat_file_size / self.data_shards as i64
|
|
} else {
|
|
self.shard_file_size() - 1
|
|
};
|
|
// Pass the actual on-disk size (header+body+checksum+timestamp+padding)
|
|
// to locate_data, matching Go: types.Size(needle.GetActualSize(size, version))
|
|
let actual = get_actual_size(size, self.version);
|
|
let intervals = ec_locate::locate_data(
|
|
offset.to_actual_offset(),
|
|
Size(actual as i32),
|
|
shard_size,
|
|
self.data_shards,
|
|
);
|
|
|
|
Ok(Some((offset, size, intervals)))
|
|
}
|
|
|
|
/// Read a full needle from locally available EC shards.
|
|
///
|
|
/// Locates the needle in the .ecx index, determines which shard intervals
|
|
/// contain its data, reads from local shards, and parses the result into
|
|
/// a fully populated Needle (including last_modified, checksum, ttl).
|
|
///
|
|
/// Returns `Ok(None)` if the needle is not found or is deleted.
|
|
/// Returns an error if a required shard is not available locally.
|
|
pub fn read_ec_shard_needle(&self, needle_id: NeedleId) -> io::Result<Option<Needle>> {
|
|
let (offset, size, intervals) = match self.locate_needle(needle_id)? {
|
|
Some(v) => v,
|
|
None => return Ok(None),
|
|
};
|
|
|
|
if intervals.is_empty() {
|
|
return Err(io::Error::new(
|
|
io::ErrorKind::InvalidData,
|
|
"no intervals for needle",
|
|
));
|
|
}
|
|
|
|
// Compute the total bytes we need to read (full needle on disk)
|
|
let actual_size = get_actual_size(size, self.version) as usize;
|
|
let mut bytes = Vec::with_capacity(actual_size);
|
|
|
|
for interval in &intervals {
|
|
let (shard_id, shard_offset) = interval.to_shard_id_and_offset(self.data_shards);
|
|
let shard = self
|
|
.shards
|
|
.get(shard_id as usize)
|
|
.and_then(|s| s.as_ref())
|
|
.ok_or_else(|| {
|
|
io::Error::new(
|
|
io::ErrorKind::NotFound,
|
|
format!("ec shard {} not available locally", shard_id),
|
|
)
|
|
})?;
|
|
|
|
let mut buf = vec![0u8; interval.size as usize];
|
|
shard.read_at(&mut buf, shard_offset as u64)?;
|
|
bytes.extend_from_slice(&buf);
|
|
}
|
|
|
|
// Truncate to exact actual_size (intervals may span more than needed)
|
|
bytes.truncate(actual_size);
|
|
|
|
if bytes.len() < actual_size {
|
|
return Err(io::Error::new(
|
|
io::ErrorKind::UnexpectedEof,
|
|
format!(
|
|
"read {} bytes but need {} for needle {}",
|
|
bytes.len(),
|
|
actual_size,
|
|
needle_id
|
|
),
|
|
));
|
|
}
|
|
|
|
let mut n = Needle::default();
|
|
n.read_bytes(&bytes, offset.to_actual_offset(), size, self.version)
|
|
.map_err(|e| io::Error::new(io::ErrorKind::InvalidData, format!("{}", e)))?;
|
|
|
|
Ok(Some(n))
|
|
}
|
|
|
|
/// Get the size of a single shard (all shards are the same size).
|
|
fn shard_file_size(&self) -> i64 {
|
|
for shard in &self.shards {
|
|
if let Some(s) = shard {
|
|
return s.file_size();
|
|
}
|
|
}
|
|
0
|
|
}
|
|
|
|
/// Walk the .ecx index and return (file_count, file_deleted_count, total_size).
|
|
/// total_size sums size.Raw() for all entries (including deleted), matching Go's WalkIndex.
|
|
pub fn walk_ecx_stats(&self) -> io::Result<(u64, u64, u64)> {
|
|
let ecx_file = match self.ecx_file.as_ref() {
|
|
Some(f) => f,
|
|
None => return Ok((0, 0, 0)),
|
|
};
|
|
|
|
let entry_count = self.ecx_file_size as usize / NEEDLE_MAP_ENTRY_SIZE;
|
|
let mut files: u64 = 0;
|
|
let mut files_deleted: u64 = 0;
|
|
let mut total_size: u64 = 0;
|
|
let mut entry_buf = [0u8; NEEDLE_MAP_ENTRY_SIZE];
|
|
|
|
for i in 0..entry_count {
|
|
let file_offset = (i * NEEDLE_MAP_ENTRY_SIZE) as u64;
|
|
#[cfg(unix)]
|
|
{
|
|
use std::os::unix::fs::FileExt;
|
|
ecx_file.read_exact_at(&mut entry_buf, file_offset)?;
|
|
}
|
|
let (_key, _offset, size) = idx_entry_from_bytes(&entry_buf);
|
|
// Match Go's Size.Raw(): tombstone (-1) returns 0, other negatives return abs
|
|
if !size.is_tombstone() {
|
|
total_size += size.0.unsigned_abs() as u64;
|
|
}
|
|
if size.is_deleted() {
|
|
files_deleted += 1;
|
|
} else {
|
|
files += 1;
|
|
}
|
|
}
|
|
|
|
Ok((files, files_deleted, total_size))
|
|
}
|
|
|
|
/// ScrubIndex verifies index integrity of an EC volume.
|
|
/// Matches Go's `(ev *EcVolume) ScrubIndex()` → `idx.CheckIndexFile()`.
|
|
/// Returns (entry_count, errors).
|
|
pub fn scrub_index(&self) -> (u64, Vec<String>) {
|
|
let ecx_file = match self.ecx_file.as_ref() {
|
|
Some(f) => f,
|
|
None => {
|
|
return (
|
|
0,
|
|
vec![format!(
|
|
"no ECX file associated with EC volume {}",
|
|
self.volume_id.0
|
|
)],
|
|
)
|
|
}
|
|
};
|
|
|
|
if self.ecx_file_size == 0 {
|
|
return (
|
|
0,
|
|
vec![format!(
|
|
"zero-size ECX file for EC volume {}",
|
|
self.volume_id.0
|
|
)],
|
|
);
|
|
}
|
|
|
|
let entry_count = self.ecx_file_size as usize / NEEDLE_MAP_ENTRY_SIZE;
|
|
let mut entries: Vec<(usize, NeedleId, i64, Size)> = Vec::with_capacity(entry_count);
|
|
let mut errs: Vec<String> = Vec::new();
|
|
let mut entry_buf = [0u8; NEEDLE_MAP_ENTRY_SIZE];
|
|
|
|
// Walk all entries
|
|
for i in 0..entry_count {
|
|
let file_offset = (i * NEEDLE_MAP_ENTRY_SIZE) as u64;
|
|
#[cfg(unix)]
|
|
{
|
|
use std::os::unix::fs::FileExt;
|
|
if let Err(e) = ecx_file.read_exact_at(&mut entry_buf, file_offset) {
|
|
errs.push(format!("read ecx entry {}: {}", i, e));
|
|
continue;
|
|
}
|
|
}
|
|
let (key, offset, size) = idx_entry_from_bytes(&entry_buf);
|
|
entries.push((i, key, offset.to_actual_offset(), size));
|
|
}
|
|
|
|
// Sort by offset, then size
|
|
entries.sort_by(|a, b| a.2.cmp(&b.2).then(a.3 .0.cmp(&b.3 .0)));
|
|
|
|
// Check for overlapping needles
|
|
for i in 1..entries.len() {
|
|
let (idx, id, offset, size) = entries[i];
|
|
let (_, last_id, last_offset, last_size) = entries[i - 1];
|
|
|
|
let actual_size =
|
|
crate::storage::needle::needle::get_actual_size(size, self.version);
|
|
let end = if actual_size != 0 {
|
|
offset + actual_size - 1
|
|
} else {
|
|
offset
|
|
};
|
|
|
|
let last_actual_size =
|
|
crate::storage::needle::needle::get_actual_size(last_size, self.version);
|
|
let last_end = if last_actual_size != 0 {
|
|
last_offset + last_actual_size - 1
|
|
} else {
|
|
last_offset
|
|
};
|
|
|
|
if offset <= last_end {
|
|
errs.push(format!(
|
|
"needle {} (#{}) at [{}-{}] overlaps needle {} at [{}-{}]",
|
|
id.0,
|
|
idx + 1,
|
|
offset,
|
|
end,
|
|
last_id.0,
|
|
last_offset,
|
|
last_end
|
|
));
|
|
}
|
|
}
|
|
|
|
// Verify file size matches entry count
|
|
let expected_size = entry_count as i64 * NEEDLE_MAP_ENTRY_SIZE as i64;
|
|
if expected_size != self.ecx_file_size {
|
|
errs.push(format!(
|
|
"expected an index file of size {}, got {}",
|
|
expected_size, self.ecx_file_size
|
|
));
|
|
}
|
|
|
|
(entries.len() as u64, errs)
|
|
}
|
|
|
|
// ---- Deletion ----
|
|
|
|
/// Write `TOMBSTONE_FILE_SIZE` over the Size field of an existing .ecx
|
|
/// entry, matching Go's `MarkNeedleDeleted`. Only used by the offline
|
|
/// `rebuild_ecx_from_journal` path — the runtime delete path does not
|
|
/// touch .ecx because the index is treated as an immutable sorted
|
|
/// (id, offset, size) table. Returns `false` if the needle is not in
|
|
/// the index (ignored by callers) and an error on IO failure.
|
|
fn tombstone_ecx_entry(&self, needle_id: NeedleId) -> io::Result<bool> {
|
|
let ecx_file = self.ecx_file.as_ref().ok_or_else(|| {
|
|
io::Error::new(
|
|
io::ErrorKind::Other,
|
|
format!(
|
|
"ec volume {} has no open .ecx file (closed or corrupt)",
|
|
self.volume_id.0
|
|
),
|
|
)
|
|
})?;
|
|
|
|
let entry_count = self.ecx_file_size as usize / NEEDLE_MAP_ENTRY_SIZE;
|
|
if entry_count == 0 {
|
|
return Ok(false);
|
|
}
|
|
|
|
let mut lo: usize = 0;
|
|
let mut hi: usize = entry_count;
|
|
let mut entry_buf = [0u8; NEEDLE_MAP_ENTRY_SIZE];
|
|
while lo < hi {
|
|
let mid = lo + (hi - lo) / 2;
|
|
let file_offset = (mid * NEEDLE_MAP_ENTRY_SIZE) as u64;
|
|
#[cfg(unix)]
|
|
{
|
|
use std::os::unix::fs::FileExt;
|
|
ecx_file.read_exact_at(&mut entry_buf, file_offset)?;
|
|
}
|
|
let (key, _offset, _old_size) = idx_entry_from_bytes(&entry_buf);
|
|
if key == needle_id {
|
|
let size_offset = file_offset + NEEDLE_ID_SIZE as u64 + OFFSET_SIZE as u64;
|
|
let mut size_buf = [0u8; SIZE_SIZE];
|
|
TOMBSTONE_FILE_SIZE.to_bytes(&mut size_buf);
|
|
#[cfg(unix)]
|
|
{
|
|
use std::os::unix::fs::FileExt;
|
|
ecx_file.write_all_at(&size_buf, size_offset)?;
|
|
}
|
|
return Ok(true);
|
|
} else if key < needle_id {
|
|
lo = mid + 1;
|
|
} else {
|
|
hi = mid;
|
|
}
|
|
}
|
|
Ok(false)
|
|
}
|
|
|
|
/// Replay .ecj journal entries into .ecx: for each needle id in .ecj,
|
|
/// overwrite its .ecx size field with a tombstone, then remove the
|
|
/// journal file. Mirrors Go's `RebuildEcxFile`, which is invoked from
|
|
/// specific decode / rebuild gRPC handlers — it is intentionally
|
|
/// **not** called on volume load (runtime reads consult
|
|
/// `deleted_needles` instead). The rebuild is atomic with respect to
|
|
/// the journal: if any individual write fails the .ecj file is left
|
|
/// in place and the error is propagated so tombstones are not lost.
|
|
#[allow(dead_code)]
|
|
fn rebuild_ecx_from_journal(&mut self) -> io::Result<()> {
|
|
let ecj_path = self.ecj_file_name();
|
|
if !std::path::Path::new(&ecj_path).exists() {
|
|
return Ok(());
|
|
}
|
|
|
|
let data = fs::read(&ecj_path)?;
|
|
if data.is_empty() {
|
|
return Ok(());
|
|
}
|
|
|
|
let count = data.len() / NEEDLE_ID_SIZE;
|
|
for i in 0..count {
|
|
let start = i * NEEDLE_ID_SIZE;
|
|
if start + NEEDLE_ID_SIZE > data.len() {
|
|
break;
|
|
}
|
|
let needle_id = NeedleId::from_bytes(&data[start..start + NEEDLE_ID_SIZE]);
|
|
// A needle that never made it into .ecx is fine (e.g. the
|
|
// delete raced against encode). Any other IO error aborts the
|
|
// rebuild so the journal survives to be retried later.
|
|
self.tombstone_ecx_entry(needle_id)?;
|
|
}
|
|
|
|
// Durably flush the newly-written .ecx tombstones before dropping
|
|
// the journal: the writes went through write_all_at and may still
|
|
// be in page cache.
|
|
if let Some(ref ecx_file) = self.ecx_file {
|
|
ecx_file.sync_all()?;
|
|
}
|
|
|
|
// Fold successful — drop and recreate the journal, clear the
|
|
// in-memory deleted set (all of its contents are now materialized
|
|
// in .ecx), and reset the cached size.
|
|
fs::remove_file(&ecj_path)?;
|
|
let ecj_file = OpenOptions::new()
|
|
.read(true)
|
|
.write(true)
|
|
.create(true)
|
|
.append(true)
|
|
.open(&ecj_path)?;
|
|
self.ecj_file = Some(ecj_file);
|
|
self.ecj_file_size = 0;
|
|
if let Ok(mut set) = self.deleted_needles.write() {
|
|
set.clear();
|
|
}
|
|
|
|
Ok(())
|
|
}
|
|
|
|
// ---- Deletion journal ----
|
|
|
|
/// Record a needle delete: append the id to the .ecj deletion journal
|
|
/// and insert it into the in-memory deleted set. `.ecx` is not touched
|
|
/// at runtime — it is a sealed sorted (id, offset, size) index and
|
|
/// runtime deletion state lives exclusively in .ecj + `deleted_needles`.
|
|
/// A lookup via `find_needle_from_ecx` masks the id out by returning
|
|
/// `TOMBSTONE_FILE_SIZE` on a subsequent read.
|
|
///
|
|
/// The .ecj append is the durable commit point. On any failure the
|
|
/// file is truncated back to the pre-append length so the on-disk
|
|
/// journal and in-memory state cannot drift. Only after the sync
|
|
/// succeeds is the id published into the set, so a failure leaves
|
|
/// the delete invisible to readers.
|
|
pub fn journal_delete(&mut self, needle_id: NeedleId) -> io::Result<()> {
|
|
// Look the needle up read-only. Missing is a silent no-op; a
|
|
// pre-existing .ecx tombstone (from a prior decode/rebuild) is
|
|
// mirrored into the in-memory set so delete_count stays accurate
|
|
// without needing to walk .ecx on every heartbeat.
|
|
match self.find_needle_from_ecx_raw(needle_id)? {
|
|
None => return Ok(()),
|
|
Some((_, size)) if size.is_deleted() => {
|
|
if let Ok(mut set) = self.deleted_needles.write() {
|
|
set.insert(needle_id);
|
|
}
|
|
return Ok(());
|
|
}
|
|
Some(_) => {}
|
|
}
|
|
|
|
// Idempotent fast path for repeat deletes — avoids the journal
|
|
// append entirely so the derived delete_count stays stable.
|
|
if self.is_needle_deleted(needle_id) {
|
|
return Ok(());
|
|
}
|
|
|
|
let prev_ecj_size = self.ecj_file_size;
|
|
let append_result: io::Result<()> = {
|
|
let ecj_file = self
|
|
.ecj_file
|
|
.as_mut()
|
|
.ok_or_else(|| io::Error::new(io::ErrorKind::Other, "ecj file not open"))?;
|
|
let mut buf = [0u8; NEEDLE_ID_SIZE];
|
|
needle_id.to_bytes(&mut buf);
|
|
ecj_file
|
|
.write_all(&buf)
|
|
.and_then(|_| ecj_file.sync_all())
|
|
};
|
|
|
|
match append_result {
|
|
Ok(()) => {
|
|
self.ecj_file_size += NEEDLE_ID_SIZE as i64;
|
|
if let Ok(mut set) = self.deleted_needles.write() {
|
|
set.insert(needle_id);
|
|
}
|
|
Ok(())
|
|
}
|
|
Err(e) => {
|
|
// write_all may have extended the file on disk before
|
|
// sync_all failed; truncate back to the known-good size so
|
|
// the on-disk journal never drifts past `deleted_needles`.
|
|
if let Some(ecj) = self.ecj_file.as_mut() {
|
|
if let Err(trunc_err) = ecj.set_len(prev_ecj_size as u64) {
|
|
tracing::error!(
|
|
volume_id = self.volume_id.0,
|
|
needle_id = needle_id.0,
|
|
truncate_error = %trunc_err,
|
|
"failed to truncate ecj after append failure"
|
|
);
|
|
}
|
|
}
|
|
Err(e)
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Internal: binary search .ecx without masking by `deleted_needles`.
|
|
/// Used by `journal_delete` so a repeat delete can still see the raw
|
|
/// pre-existing .ecx tombstone from a prior rebuild.
|
|
fn find_needle_from_ecx_raw(
|
|
&self,
|
|
needle_id: NeedleId,
|
|
) -> io::Result<Option<(Offset, Size)>> {
|
|
let ecx_file = self
|
|
.ecx_file
|
|
.as_ref()
|
|
.ok_or_else(|| io::Error::new(io::ErrorKind::Other, "ecx file not open"))?;
|
|
let entry_count = self.ecx_file_size as usize / NEEDLE_MAP_ENTRY_SIZE;
|
|
if entry_count == 0 {
|
|
return Ok(None);
|
|
}
|
|
let mut lo: usize = 0;
|
|
let mut hi: usize = entry_count;
|
|
let mut entry_buf = [0u8; NEEDLE_MAP_ENTRY_SIZE];
|
|
while lo < hi {
|
|
let mid = lo + (hi - lo) / 2;
|
|
let file_offset = (mid * NEEDLE_MAP_ENTRY_SIZE) as u64;
|
|
#[cfg(unix)]
|
|
{
|
|
use std::os::unix::fs::FileExt;
|
|
ecx_file.read_exact_at(&mut entry_buf, file_offset)?;
|
|
}
|
|
let (key, offset, size) = idx_entry_from_bytes(&entry_buf);
|
|
if key == needle_id {
|
|
return Ok(Some((offset, size)));
|
|
} else if key < needle_id {
|
|
lo = mid + 1;
|
|
} else {
|
|
hi = mid;
|
|
}
|
|
}
|
|
Ok(None)
|
|
}
|
|
|
|
/// Append a deleted needle ID to the .ecj journal, validating the cookie first.
|
|
/// Matches Go's DeleteEcShardNeedle which validates cookie before journaling.
|
|
/// A cookie of 0 means skip cookie check (e.g., orphan cleanup).
|
|
pub fn journal_delete_with_cookie(
|
|
&mut self,
|
|
needle_id: NeedleId,
|
|
cookie: crate::storage::types::Cookie,
|
|
) -> io::Result<()> {
|
|
// cookie == 0 indicates SkipCookieCheck was requested
|
|
if cookie.0 != 0 {
|
|
// Try to read the needle's cookie from the EC shards to validate
|
|
// Look up the needle in ecx index to find its offset, then read header from shard
|
|
if let Ok(Some((offset, size))) = self.find_needle_from_ecx(needle_id) {
|
|
if !size.is_deleted() && !offset.is_zero() {
|
|
let actual_offset = offset.to_actual_offset() as u64;
|
|
// Determine which shard contains this offset and read the cookie
|
|
let shard_size = self
|
|
.shards
|
|
.iter()
|
|
.filter_map(|s| s.as_ref())
|
|
.map(|s| s.file_size())
|
|
.next()
|
|
.unwrap_or(0) as u64;
|
|
if shard_size > 0 {
|
|
let shard_id = (actual_offset / shard_size) as usize;
|
|
let shard_offset = actual_offset % shard_size;
|
|
if let Some(Some(shard)) = self.shards.get(shard_id) {
|
|
let mut header_buf = [0u8; 4]; // cookie is first 4 bytes of needle
|
|
if shard.read_at(&mut header_buf, shard_offset).is_ok() {
|
|
let needle_cookie =
|
|
crate::storage::types::Cookie(u32::from_be_bytes(header_buf));
|
|
if needle_cookie != cookie {
|
|
return Err(io::Error::new(
|
|
io::ErrorKind::InvalidData,
|
|
format!("unexpected cookie {:x}", cookie.0),
|
|
));
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
self.journal_delete(needle_id)
|
|
}
|
|
|
|
/// Read all deleted needle IDs from the .ecj journal.
|
|
pub fn read_deleted_needles(&self) -> io::Result<Vec<NeedleId>> {
|
|
let ecj_path = self.ecj_file_name();
|
|
if !std::path::Path::new(&ecj_path).exists() {
|
|
return Ok(Vec::new());
|
|
}
|
|
|
|
let data = fs::read(&ecj_path)?;
|
|
let count = data.len() / NEEDLE_ID_SIZE;
|
|
let mut needles = Vec::with_capacity(count);
|
|
for i in 0..count {
|
|
let start = i * NEEDLE_ID_SIZE;
|
|
let id = NeedleId::from_bytes(&data[start..start + NEEDLE_ID_SIZE]);
|
|
needles.push(id);
|
|
}
|
|
Ok(needles)
|
|
}
|
|
|
|
// ---- Lifecycle ----
|
|
|
|
pub fn close(&mut self) {
|
|
for shard in &mut self.shards {
|
|
if let Some(s) = shard {
|
|
s.close();
|
|
}
|
|
*shard = None;
|
|
}
|
|
// Sync .ecx before closing to flush in-place deletion marks (matches Go's ev.ecxFile.Sync())
|
|
if let Some(ref ecx_file) = self.ecx_file {
|
|
let _ = ecx_file.sync_all();
|
|
}
|
|
self.ecx_file = None;
|
|
self.ecj_file = None;
|
|
}
|
|
|
|
pub fn destroy(&mut self) {
|
|
for shard in &mut self.shards {
|
|
if let Some(s) = shard {
|
|
s.destroy();
|
|
}
|
|
*shard = None;
|
|
}
|
|
// Remove .ecx/.ecj/.vif from ecx_actual_dir (where they were found)
|
|
// Go's Destroy() removes .ecx, .ecj, and .vif files.
|
|
let actual_base = crate::storage::volume::volume_file_name(
|
|
&self.ecx_actual_dir,
|
|
&self.collection,
|
|
self.volume_id,
|
|
);
|
|
let _ = fs::remove_file(format!("{}.ecx", actual_base));
|
|
let _ = fs::remove_file(format!("{}.ecj", actual_base));
|
|
let _ = fs::remove_file(format!("{}.vif", actual_base));
|
|
// Also sweep the originally-configured idx dir in case stale files
|
|
// exist there (ecx_file_name() / ecj_file_name() now resolve from
|
|
// ecx_actual_dir, so we have to build the idx-dir paths explicitly).
|
|
if self.ecx_actual_dir != self.dir_idx {
|
|
let idx_base = crate::storage::volume::volume_file_name(
|
|
&self.dir_idx,
|
|
&self.collection,
|
|
self.volume_id,
|
|
);
|
|
let _ = fs::remove_file(format!("{}.ecx", idx_base));
|
|
let _ = fs::remove_file(format!("{}.ecj", idx_base));
|
|
let _ = fs::remove_file(format!("{}.vif", idx_base));
|
|
}
|
|
if self.ecx_actual_dir != self.dir && self.dir_idx != self.dir {
|
|
let data_base = crate::storage::volume::volume_file_name(
|
|
&self.dir,
|
|
&self.collection,
|
|
self.volume_id,
|
|
);
|
|
let _ = fs::remove_file(format!("{}.ecx", data_base));
|
|
let _ = fs::remove_file(format!("{}.ecj", data_base));
|
|
let _ = fs::remove_file(format!("{}.vif", data_base));
|
|
}
|
|
self.ecx_file = None;
|
|
self.ecj_file = None;
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
use tempfile::TempDir;
|
|
|
|
fn write_ecx_file(
|
|
dir: &str,
|
|
collection: &str,
|
|
vid: VolumeId,
|
|
entries: &[(NeedleId, Offset, Size)],
|
|
) {
|
|
let base = crate::storage::volume::volume_file_name(dir, collection, vid);
|
|
let ecx_path = format!("{}.ecx", base);
|
|
let mut file = File::create(&ecx_path).unwrap();
|
|
|
|
// Write sorted entries
|
|
for &(key, offset, size) in entries {
|
|
let mut buf = [0u8; NEEDLE_MAP_ENTRY_SIZE];
|
|
idx_entry_to_bytes(&mut buf, key, offset, size);
|
|
file.write_all(&buf).unwrap();
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn test_ec_volume_find_needle() {
|
|
let tmp = TempDir::new().unwrap();
|
|
let dir = tmp.path().to_str().unwrap();
|
|
|
|
// Write sorted ecx entries
|
|
let entries = vec![
|
|
(NeedleId(1), Offset::from_actual_offset(8), Size(100)),
|
|
(NeedleId(5), Offset::from_actual_offset(200), Size(200)),
|
|
(NeedleId(10), Offset::from_actual_offset(500), Size(300)),
|
|
];
|
|
write_ecx_file(dir, "", VolumeId(1), &entries);
|
|
|
|
let vol = EcVolume::new(dir, dir, "", VolumeId(1)).unwrap();
|
|
|
|
// Found
|
|
let result = vol.find_needle_from_ecx(NeedleId(5)).unwrap();
|
|
assert!(result.is_some());
|
|
let (offset, size) = result.unwrap();
|
|
assert_eq!(offset.to_actual_offset(), 200);
|
|
assert_eq!(size, Size(200));
|
|
|
|
// Not found
|
|
let result = vol.find_needle_from_ecx(NeedleId(7)).unwrap();
|
|
assert!(result.is_none());
|
|
}
|
|
|
|
#[test]
|
|
fn test_ec_volume_journal() {
|
|
let tmp = TempDir::new().unwrap();
|
|
let dir = tmp.path().to_str().unwrap();
|
|
|
|
// .ecj append is gated on a live->tombstone transition in .ecx, so
|
|
// the fixture must contain the needles we are about to delete.
|
|
let entries = vec![
|
|
(NeedleId(10), Offset::from_actual_offset(8), Size(100)),
|
|
(NeedleId(20), Offset::from_actual_offset(200), Size(200)),
|
|
];
|
|
write_ecx_file(dir, "", VolumeId(1), &entries);
|
|
|
|
let mut vol = EcVolume::new(dir, dir, "", VolumeId(1)).unwrap();
|
|
let (fc0, dc0) = vol.file_and_delete_count();
|
|
assert_eq!((fc0, dc0), (2, 0));
|
|
|
|
vol.journal_delete(NeedleId(10)).unwrap();
|
|
vol.journal_delete(NeedleId(20)).unwrap();
|
|
|
|
let deleted = vol.read_deleted_needles().unwrap();
|
|
assert_eq!(deleted, vec![NeedleId(10), NeedleId(20)]);
|
|
|
|
let (fc, dc) = vol.file_and_delete_count();
|
|
assert_eq!((fc, dc), (2, 2));
|
|
|
|
// Idempotent re-delete must not bump delete_count.
|
|
vol.journal_delete(NeedleId(10)).unwrap();
|
|
// Deleting a missing needle must not bump delete_count either.
|
|
vol.journal_delete(NeedleId(999)).unwrap();
|
|
let (fc, dc) = vol.file_and_delete_count();
|
|
assert_eq!((fc, dc), (2, 2));
|
|
}
|
|
|
|
#[test]
|
|
fn test_ec_volume_shard_bits() {
|
|
let tmp = TempDir::new().unwrap();
|
|
let dir = tmp.path().to_str().unwrap();
|
|
write_ecx_file(dir, "", VolumeId(1), &[]);
|
|
|
|
let mut vol = EcVolume::new(dir, dir, "", VolumeId(1)).unwrap();
|
|
assert_eq!(vol.shard_count(), 0);
|
|
|
|
// Create a shard file so we can add it
|
|
let mut shard = EcVolumeShard::new(dir, "", VolumeId(1), 3);
|
|
shard.create().unwrap();
|
|
shard.write_all(&[0u8; 100]).unwrap();
|
|
shard.close();
|
|
|
|
vol.add_shard(EcVolumeShard::new(dir, "", VolumeId(1), 3))
|
|
.unwrap();
|
|
assert_eq!(vol.shard_count(), 1);
|
|
assert!(vol.shard_bits().has_shard_id(3));
|
|
}
|
|
|
|
#[test]
|
|
fn test_ec_volume_uses_collection_prefixed_vif_config() {
|
|
let tmp = TempDir::new().unwrap();
|
|
let dir = tmp.path().to_str().unwrap();
|
|
write_ecx_file(dir, "pics", VolumeId(1), &[]);
|
|
|
|
let vif = crate::storage::volume::VifVolumeInfo {
|
|
ec_shard_config: Some(crate::storage::volume::VifEcShardConfig {
|
|
data_shards: 6,
|
|
parity_shards: 3,
|
|
}),
|
|
..Default::default()
|
|
};
|
|
let base = crate::storage::volume::volume_file_name(dir, "pics", VolumeId(1));
|
|
std::fs::write(
|
|
format!("{}.vif", base),
|
|
serde_json::to_string_pretty(&vif).unwrap(),
|
|
)
|
|
.unwrap();
|
|
|
|
let vol = EcVolume::new(dir, dir, "pics", VolumeId(1)).unwrap();
|
|
assert_eq!(vol.data_shards, 6);
|
|
assert_eq!(vol.parity_shards, 3);
|
|
}
|
|
|
|
#[test]
|
|
fn test_ec_volume_invalid_vif_config_falls_back_to_defaults() {
|
|
let tmp = TempDir::new().unwrap();
|
|
let dir = tmp.path().to_str().unwrap();
|
|
write_ecx_file(dir, "pics", VolumeId(1), &[]);
|
|
|
|
let vif = crate::storage::volume::VifVolumeInfo {
|
|
ec_shard_config: Some(crate::storage::volume::VifEcShardConfig {
|
|
data_shards: 10,
|
|
parity_shards: 10,
|
|
}),
|
|
..Default::default()
|
|
};
|
|
let base = crate::storage::volume::volume_file_name(dir, "pics", VolumeId(1));
|
|
std::fs::write(
|
|
format!("{}.vif", base),
|
|
serde_json::to_string_pretty(&vif).unwrap(),
|
|
)
|
|
.unwrap();
|
|
|
|
let vol = EcVolume::new(dir, dir, "pics", VolumeId(1)).unwrap();
|
|
assert_eq!(vol.data_shards, DATA_SHARDS_COUNT as u32);
|
|
assert_eq!(vol.parity_shards, PARITY_SHARDS_COUNT as u32);
|
|
}
|
|
}
|