ssshr-66andChris Lu cd1e738422 [Volume] Scrub local deletion tombstones during FULL scrub (#11396) ...

* fix 11388

* fix(volume): scrub validates local deletion tombstones

TombstoneFileSize (-1) is an .idx-only sentinel; the physical record it
points at carries a zero-sized body. Normalize deleted index sizes to 0
via onDiskSize before computing disk usage and calling ReadData, so
corrupted or truncated tombstone records are detected instead of
skipped. Offset-zero entries (remote logical deletes, no .dat record)
remain skipped, and the physical needle id is checked against the index
key. Mirror the behavior in the Rust volume server.

* fix(volume): scrub preserves physical size of deleted non-tombstone entries

Size.Raw()/raw() already encodes the index-to-disk mapping: tombstone
(-1) -> 0, other negative sizes -> their absolute value (the offset then
points at the original record, per the ReadDeleted path). Use it instead
of mapping every deleted size to 0.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>

2026-09-19 18:55:27 -07:00

..

backend

Cloud/remote storage & tiering: configurable multipart upload/download concurrency (#11319)

2026-09-14 22:09:08 -07:00

erasure_coding

[Volume] Scrub every disk's EC shards for a volume id, not just the first (#11258)

2026-09-12 14:19:18 -07:00

idx

fix(scrub): don't flag offset-0 logical tombstones in volume scrub (#10148)

2026-06-30 02:01:14 -07:00

needle

fix(volume): derive needle body tail bound from the version layout (#11395)

2026-09-19 12:20:01 -07:00

needle_map

Add codespell support with configuration and typo fixes (#10393)

2026-07-22 14:38:06 -07:00

super_block

perf(weed/storage/super_block): intern the byte-encoded replica placements (#10610)

2026-08-07 00:53:02 -07:00

test_files

Have volume scrubs account for zero-sized volumes. (#9609)

2026-05-21 09:42:07 -07:00

types

volume: resolve the disk IO slow-latency threshold per disk (#10976)

2026-08-27 10:01:05 -07:00

volume_info

admin: allow setting volume read-only and read/write modes (#11217)

2026-09-07 18:40:37 -07:00

volume_replica

fix(ec): bring ec.encode worker and EC/volume helpers to parity with shell (#9599)

2026-05-21 02:16:28 -07:00

dir_scan_test.go

volume: cut the memory a server holding millions of volumes still uses (#10999)

2026-08-27 22:25:15 -07:00

dir_scan.go

volume: cut the memory a server holding millions of volumes still uses (#10999)

2026-08-27 22:25:15 -07:00

disk_location_ec_realworld_test.go

EC bitrot detection: per-shard checksum sidecars (#9761)

2026-05-31 18:52:44 -07:00

disk_location_ec_shard_size_test.go

fix(volume_server): load orphan EC shards across disks on startup (#9212) (#9244)

2026-04-27 16:01:10 -07:00

disk_location_ec_test.go

ec: refuse to mount a 0-byte shard file when the index has entries (#11030)

2026-08-29 14:19:16 -07:00

disk_location_ec.go

ec: refuse to mount a 0-byte shard file when the index has entries (#11030)

2026-08-29 14:19:16 -07:00

disk_location_scan_order_test.go

volume: cut the memory a server holding millions of volumes still uses (#10999)

2026-08-27 22:25:15 -07:00

disk_location_test.go

volume: rebuild a missing .idx from the .dat (#11115)

2026-09-03 08:43:10 -07:00

disk_location.go

Mount req with collection (#11249)

2026-09-09 12:55:47 -07:00

ec_startup_shard_safety_test.go

fix(ec): never delete recoverable EC shards on startup/reconcile (the non-empty-.dat sibling of the stub bug) (#9941)

2026-06-12 23:51:29 -07:00

io_error_other.go

fix(volume): handle faulty storage media (Go + Rust) (#11233)

2026-09-08 21:42:56 -07:00

io_error_windows.go

fix(volume): handle faulty storage media (Go + Rust) (#11233)

2026-09-08 21:42:56 -07:00

io_error.go

fix(volume): handle faulty storage media (Go + Rust) (#11233)

2026-09-08 21:42:56 -07:00

needle_map_file_pool_test.go

admin: allow setting volume read-only and read/write modes (#11217)

2026-09-07 18:40:37 -07:00

needle_map_file_pool.go

volume: open volume files with O_NOATIME (#11055)

2026-08-31 21:41:50 -07:00

needle_map_leveldb_test.go

fix(volume): persist the leveldb needle map watermark at batch boundaries (#10557)

2026-08-04 13:33:31 -07:00

needle_map_leveldb.go

storage: fix corrupted leveldb detection in DoOffsetLoading (#10650)

2026-08-08 06:00:46 -07:00

needle_map_memory.go

fix(volume): keep vacuum running past dangling .idx entries (#9115)

2026-04-16 22:01:34 -07:00

needle_map_metric_test.go

fix(volume): keep vacuum running past dangling .idx entries (#9115)

2026-04-16 22:01:34 -07:00

needle_map_metric.go

fix(volume): keep vacuum running past dangling .idx entries (#9115)

2026-04-16 22:01:34 -07:00

needle_map_sorted_file_test.go

fix(volume): seed indexFileOffset in SortedFileNeedleMap so Delete appends (#9483)

2026-05-13 10:22:01 -07:00

needle_map_sorted_file.go

volume: stop read-only volumes from pinning .idx and .sdx (#10950)

2026-08-25 10:23:00 -07:00

needle_map.go

fix(volume): keep vacuum running past dangling .idx entries (#9115)

2026-04-16 22:01:34 -07:00

remote_tier_integration_test.go

Cloud/remote storage & tiering: configurable multipart upload/download concurrency (#11319)

2026-09-14 22:09:08 -07:00

store_collection_metrics_test.go

volume: stop reporting read-only volumes that are no longer here (#10867)

2026-08-21 22:33:01 -07:00

store_consolidate_index_test.go

Mount req with collection (#11249)

2026-09-09 12:55:47 -07:00

store_delete_collection_test.go

heartbeat: keep the master current through collection churn (#10657)

2026-08-08 20:23:10 -07:00

store_delete_volume_test.go

filer: batch exact lookup RPC, authoritative volume lookup, VolumeDelete status codes (#11122)

2026-09-03 14:52:05 -07:00

store_disk_space_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_duplicate_vid_test.go

fix(storage): delete/unmount every copy of a duplicate volume id (#9954)

2026-06-14 06:36:47 -07:00

store_ec_9plus3_reboot_repro_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_ec_delete.go

ec: uniform shard block layout (#10932)

2026-08-28 20:46:59 -07:00

store_ec_disk_type_test.go

batch drain delta heartbeat messages (#9914)

2026-06-10 13:33:45 -07:00

store_ec_hybrid_repro_test.go

fix(ec): never delete recoverable EC shards on startup/reconcile (the non-empty-.dat sibling of the stub bug) (#9941)

2026-06-12 23:51:29 -07:00

store_ec_interval_read_test.go

ec: uniform shard block layout (#10932)

2026-08-28 20:46:59 -07:00

store_ec_location_cache_test.go

Re-check an EC shard map a failed read has disproved (#11023)

2026-08-28 15:43:05 -07:00

store_ec_mirror_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_ec_mirror.go

fix(ec): mirror EC sidecars onto every shard-bearing disk at startup (#9525)

2026-05-17 19:55:15 -07:00

store_ec_mount_cross_disk_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_ec_multidisk_lifecycle_test.go

test: assert EC shard identity and empty-view in multi-disk lifecycle tests (#10723)

2026-08-12 20:18:39 -07:00

store_ec_orphan_shard_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_ec_phantom_dat_test.go

fix(storage): never let an empty .dat delete healthy distributed EC shards (#9930)

2026-06-11 20:26:20 -07:00

store_ec_reconcile.go

volume: cut the memory a server holding millions of volumes still uses (#10999)

2026-08-27 22:25:15 -07:00

store_ec_reconstruct_test.go

ec: split the shard-interval recovery into a gather and a rebuild (#11005)

2026-08-28 15:55:12 -07:00

store_ec_recover_fanout_test.go

ec: uniform shard block layout (#10932)

2026-08-28 20:46:59 -07:00

store_ec_recover_test.go

ec: recover EC shards whose .ecx index lives only on a peer server (#10108)

2026-06-25 10:38:14 -07:00

store_ec_recover.go

ec: recover EC shards whose .ecx index lives only on a peer server (#10108)

2026-06-25 10:38:14 -07:00

store_ec_recovery_test.go

Update store_ec_recovery_test.go

2026-01-23 16:38:36 -08:00

store_ec_scrub_reads_test.go

ec: uniform shard block layout (#10932)

2026-08-28 20:46:59 -07:00

store_ec_scrub.go

[Volume] Scrub every disk's EC shards for a volume id, not just the first (#11258)

2026-09-12 14:19:18 -07:00

store_ec_target_location_test.go

ec: pin auto-selected shard placement to the disk that already owns the shard (#11029)

2026-08-29 12:12:41 -07:00

store_ec.go

fix(volume): handle faulty storage media (Go + Rust) (#11233)

2026-09-08 21:42:56 -07:00

store_heartbeat_bench_test.go

volume: stop reporting read-only volumes that are no longer here (#10867)

2026-08-21 22:33:01 -07:00

store_heartbeat_digest_test.go

master: keep new volumes and writes off servers in maintenance mode (#11147)

2026-09-03 23:50:12 -07:00

store_load_balancing_simple_test.go

create new volumes on less occupied disk locations (#7349)

2025-10-20 16:11:29 -07:00

store_load_balancing_test.go

volume: cut idle memory at high volume counts (#10861)

2026-08-21 13:04:56 -07:00

store_mark_readonly_can_delete_test.go

Mount req with collection (#11249)

2026-09-09 12:55:47 -07:00

store_maxvolume_deadzone_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_mount_validation_test.go

fix(volume): Harden Volume Copy Validation and Failure Handling (#11252)

2026-09-09 23:33:05 -07:00

store_state.go

chore: remove ~50k lines of unreachable dead code (#8913)

2026-04-03 16:04:27 -07:00

store_status_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

store_vacuum_test.go

fix tests

2025-06-30 13:57:28 -07:00

store_vacuum.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

store_volume_report_test.go

volume: cut the memory a server holding millions of volumes still uses (#10999)

2026-08-27 22:25:15 -07:00

store_volume_report.go

volume: cut the memory a server holding millions of volumes still uses (#10999)

2026-08-27 22:25:15 -07:00

store.go

fix(volume): Harden Volume Copy Validation and Failure Handling (#11252)

2026-09-09 23:33:05 -07:00

volume_async_worker_test.go

volume: cut idle memory at high volume counts (#10861)

2026-08-21 13:04:56 -07:00

volume_backup_test.go

refactoring

2019-04-17 22:04:49 -07:00

volume_backup.go

add version to volume proto

2025-06-16 22:05:06 -07:00

volume_checking_test.go

[Volume] Scrub local deletion tombstones during FULL scrub (#11396)

2026-09-19 18:55:27 -07:00

volume_checking.go

[Volume] Scrub local deletion tombstones during FULL scrub (#11396)

2026-09-19 18:55:27 -07:00

volume_destroy_ec_vif_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

volume_idx_rebuild_test.go

volume: rebuild a missing .idx from the .dat (#11115)

2026-09-03 08:43:10 -07:00

volume_idx_rebuild.go

volume: rebuild a missing .idx from the .dat (#11115)

2026-09-03 08:43:10 -07:00

volume_idx_repair_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

volume_idx_repair.go

volume: recover .idx rows overwritten by tiered deletes (#10474)

2026-07-28 16:48:30 -07:00

volume_info_counts_test.go

storage: count a volume's needles in uint32 (#10718)

2026-08-11 16:36:36 -07:00

volume_info_intern_test.go

storage: stop keeping the remote storage key on the master (#10672)

2026-08-09 12:43:31 -07:00

volume_info_layout_test.go

storage: order VolumeInfo by alignment (#10669)

2026-08-09 09:41:50 -07:00

volume_info_test.go

master: keep disk_id when registering volumes from incremental heartbeats (#10686)

2026-08-10 00:40:29 -07:00

volume_info.go

volume: avoid read-only replica write targets (#11195)

2026-09-07 09:23:56 -07:00

volume_io_error_test.go

EC: handle zero-sized shard files uniformly (moves, rebuilds, startup cleanup) (#10753)

2026-08-13 21:38:22 -07:00

volume_loading_corrupt_idx_test.go

admin: allow setting volume read-only and read/write modes (#11217)

2026-09-07 18:40:37 -07:00

volume_loading.go

volume: expire TTL volumes whose only traffic is deletes (#11167)

2026-09-04 23:48:40 -07:00

volume_mark_writable_test.go

admin: allow setting volume read-only and read/write modes (#11217)

2026-09-07 18:40:37 -07:00

volume_read_all.go

Include name/mime in ReadAllNeedles (#4005)

2022-11-23 15:59:38 -08:00

volume_read_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

volume_read.go

s3api: don't delete chunks when CreateEntry outcome is ambiguous (#11376)

2026-09-17 19:58:49 -07:00

volume_report_hash.go

volume: avoid read-only replica write targets (#11195)

2026-09-07 09:23:56 -07:00

volume_super_block_replication_test.go

test: pin that a .vif replication outranks the superblock (#10499)

2026-07-30 17:03:38 -07:00

volume_super_block.go

volume: open volume files with O_NOATIME (#11055)

2026-08-31 21:41:50 -07:00

volume_tier_intern_test.go

volume: cut the memory a server holding millions of volumes still uses (#10999)

2026-08-27 22:25:15 -07:00

volume_tier.go

volume: expire TTL volumes whose only traffic is deletes (#11167)

2026-09-04 23:48:40 -07:00

volume_ttl_expiry_test.go

volume: expire TTL volumes whose only traffic is deletes (#11167)

2026-09-04 23:48:40 -07:00

volume_vacuum_crash_safe_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

volume_vacuum_test.go

fix(volume): return an error instead of panicking on a corrupt needle size (#11393)

2026-09-18 23:00:06 -07:00

volume_vacuum.go

fix(vacuum): batch fsync in makeupDiff to prevent test timeout (#11289)

2026-09-13 00:05:30 -07:00

volume_write_fsync_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

volume_write_test.go

admin: allow setting volume read-only and read/write modes (#11217)

2026-09-07 18:40:37 -07:00

volume_write.go

fix(volume): handle faulty storage media (Go + Rust) (#11233)

2026-09-08 21:42:56 -07:00

volume.go

fix(volume): handle faulty storage media (Go + Rust) (#11233)

2026-09-08 21:42:56 -07:00