Chris Lu ab559bca76 storage: hash index-metric bloom filter with xxhash ...

The reverse index walk dedupes seen NeedleIds with a BoomFilters bloom
filter. Its default kernel is FNV-1, whose final step only XORs the last
input byte into the hash's low bits, so keys sharing the high 7 bytes of
a big-endian NeedleId land on correlated probe positions and false
positives arrive in bursts of hundreds instead of the ~0.001 target
rate. Each false positive hides a live-latest key and inflates the
rebuilt deletion counters, which is what made
TestFastLoadingNeedleMapMetrics flake past its +256 bound in CI
(observed DeletedCount drift of 267).

Swap the kernel for xxhash, which fully diffuses the key bytes; the
observed false-positive count in 300 local runs of the test drops to
max 2 (avg 0.23).

2026-10-19 01:47:25 -07:00

..

backend

Cloud/remote storage & tiering: configurable multipart upload/download concurrency (#11319)

2026-09-14 22:09:08 -07:00

erasure_coding

ec: re-plan balance moves when a target reports no space left (#11759)

2026-10-15 18:47:07 -07:00

idx

fix(scrub): don't flag offset-0 logical tombstones in volume scrub (#10148)

2026-06-30 02:01:14 -07:00

needle

fix(volume): validate sizes in ReadNeedleBlob and WriteNeedleBlob (#11399)

2026-09-19 21:28:37 -07:00

needle_map

[Volume] Keep DAT and index state consistent after async batch Sync failure (#11425)

2026-09-24 06:57:44 +08:00

super_block

perf(weed/storage/super_block): intern the byte-encoded replica placements (#10610)

2026-08-07 00:53:02 -07:00

test_files

Have volume scrubs account for zero-sized volumes. (#9609)

2026-05-21 09:42:07 -07:00

types

volume: resolve the disk IO slow-latency threshold per disk (#10976)

2026-08-27 10:01:05 -07:00

volume_info

admin: allow setting volume read-only and read/write modes (#11217)

2026-09-07 18:40:37 -07:00

volume_replica

ec: compare live idx NeedleId sets before encode (#11812)

2026-10-19 00:56:17 -07:00

dir_scan_test.go

volume: cut the memory a server holding millions of volumes still uses (#10999)

2026-08-27 22:25:15 -07:00

dir_scan.go

volume: cut the memory a server holding millions of volumes still uses (#10999)

2026-08-27 22:25:15 -07:00

disk_location_ec_realworld_test.go

EC bitrot detection: per-shard checksum sidecars (#9761)

2026-05-31 18:52:44 -07:00

disk_location_ec_shard_size_test.go

fix(volume_server): load orphan EC shards across disks on startup (#9212) (#9244)

2026-04-27 16:01:10 -07:00

disk_location_ec_test.go

volume server: ec.decode verifies, cleans up and compacts like Go, off the runtime (#11547)

2026-10-03 14:55:15 +08:00

disk_location_ec.go

volume server: ec.decode verifies, cleans up and compacts like Go, off the runtime (#11547)

2026-10-03 14:55:15 +08:00

disk_location_scan_order_test.go

volume: cut the memory a server holding millions of volumes still uses (#10999)

2026-08-27 22:25:15 -07:00

disk_location_test.go

volume: rebuild a missing .idx from the .dat (#11115)

2026-09-03 08:43:10 -07:00

disk_location.go

volume server: ec.decode verifies, cleans up and compacts like Go, off the runtime (#11547)

2026-10-03 14:55:15 +08:00

ec_startup_shard_safety_test.go

fix(ec): never delete recoverable EC shards on startup/reconcile (the non-empty-.dat sibling of the stub bug) (#9941)

2026-06-12 23:51:29 -07:00

io_error_other.go

fix(volume): handle faulty storage media (Go + Rust) (#11233)

2026-09-08 21:42:56 -07:00

io_error_windows.go

fix(volume): handle faulty storage media (Go + Rust) (#11233)

2026-09-08 21:42:56 -07:00

io_error.go

fix(volume): handle faulty storage media (Go + Rust) (#11233)

2026-09-08 21:42:56 -07:00

needle_map_file_pool_test.go

admin: allow setting volume read-only and read/write modes (#11217)

2026-09-07 18:40:37 -07:00

needle_map_file_pool.go

volume: open volume files with O_NOATIME (#11055)

2026-08-31 21:41:50 -07:00

needle_map_leveldb_test.go

fix(volume): persist the leveldb needle map watermark at batch boundaries (#10557)

2026-08-04 13:33:31 -07:00

needle_map_leveldb.go

storage: unify needle-map counters across index loaders and warn on copy-count drift (#11701)

2026-10-10 22:31:46 +08:00

needle_map_memory.go

storage: unify needle-map counters across index loaders and warn on copy-count drift (#11701)

2026-10-10 22:31:46 +08:00

needle_map_metric_test.go

storage: unify needle-map counters across index loaders and warn on copy-count drift (#11701)

2026-10-10 22:31:46 +08:00

needle_map_metric.go

storage: hash index-metric bloom filter with xxhash

2026-10-19 01:47:25 -07:00

needle_map_sorted_file_test.go

fix(volume): seed indexFileOffset in SortedFileNeedleMap so Delete appends (#9483)

2026-05-13 10:22:01 -07:00

needle_map_sorted_file.go

[Volume] Keep DAT and index state consistent after async batch Sync failure (#11425)

2026-09-24 06:57:44 +08:00

needle_map.go

[Volume] Keep DAT and index state consistent after async batch Sync failure (#11425)

2026-09-24 06:57:44 +08:00

remote_tier_integration_test.go

shell: keep volume.tier.compact replicas on the same remote object (#11783)

2026-10-16 19:45:17 -07:00

store_collection_metrics_test.go

volume: stop reporting read-only volumes that are no longer here (#10867)

2026-08-21 22:33:01 -07:00

store_consolidate_index_test.go

Mount req with collection (#11249)

2026-09-09 12:55:47 -07:00

store_delete_collection_test.go

heartbeat: keep the master current through collection churn (#10657)

2026-08-08 20:23:10 -07:00

store_delete_volume_test.go

shell: let volume.deleteEmpty drop volumes with no live needles (#11437)

2026-09-26 15:53:39 +08:00

store_disk_space_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_duplicate_vid_test.go

volume server: refuse the rest of a grouped run after a durable index failure (#11576)

2026-10-03 21:04:30 +08:00

store_ec_9plus3_reboot_repro_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_ec_delete.go

ec: uniform shard block layout (#10932)

2026-08-28 20:46:59 -07:00

store_ec_disk_type_test.go

batch drain delta heartbeat messages (#9914)

2026-06-10 13:33:45 -07:00

store_ec_hybrid_repro_test.go

fix(ec): never delete recoverable EC shards on startup/reconcile (the non-empty-.dat sibling of the stub bug) (#9941)

2026-06-12 23:51:29 -07:00

store_ec_interval_read_test.go

ec: uniform shard block layout (#10932)

2026-08-28 20:46:59 -07:00

store_ec_journal_test.go

volume: merge .ecj as a set union on EC shard copy + index recovery (Rust+Go) (#11554)

2026-10-02 22:46:36 +08:00

store_ec_journal.go

volume: compact an oversized .ecj at mount, safely (Rust + Go) (#11555)

2026-10-03 09:08:19 +08:00

store_ec_location_cache_test.go

Re-check an EC shard map a failed read has disproved (#11023)

2026-08-28 15:43:05 -07:00

store_ec_mirror_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_ec_mirror.go

fix(ec): mirror EC sidecars onto every shard-bearing disk at startup (#9525)

2026-05-17 19:55:15 -07:00

store_ec_mount_cross_disk_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_ec_multidisk_lifecycle_test.go

test: assert EC shard identity and empty-view in multi-disk lifecycle tests (#10723)

2026-08-12 20:18:39 -07:00

store_ec_orphan_shard_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_ec_phantom_dat_test.go

fix(storage): never let an empty .dat delete healthy distributed EC shards (#9930)

2026-06-11 20:26:20 -07:00

store_ec_reconcile.go

volume: cut the memory a server holding millions of volumes still uses (#10999)

2026-08-27 22:25:15 -07:00

store_ec_reconstruct_test.go

ec: split the shard-interval recovery into a gather and a rebuild (#11005)

2026-08-28 15:55:12 -07:00

store_ec_recover_fanout_test.go

ec: uniform shard block layout (#10932)

2026-08-28 20:46:59 -07:00

store_ec_recover_test.go

ec: recover EC shards whose .ecx index lives only on a peer server (#10108)

2026-06-25 10:38:14 -07:00

store_ec_recover.go

ec: recover EC shards whose .ecx index lives only on a peer server (#10108)

2026-06-25 10:38:14 -07:00

store_ec_recovery_test.go

Update store_ec_recovery_test.go

2026-01-23 16:38:36 -08:00

store_ec_scrub_reads_test.go

ec: uniform shard block layout (#10932)

2026-08-28 20:46:59 -07:00

store_ec_scrub.go

[Volume] Scrub every disk's EC shards for a volume id, not just the first (#11258)

2026-09-12 14:19:18 -07:00

store_ec_target_location_test.go

ec: pin auto-selected shard placement to the disk that already owns the shard (#11029)

2026-08-29 12:12:41 -07:00

store_ec.go

fix(volume): handle faulty storage media (Go + Rust) (#11233)

2026-09-08 21:42:56 -07:00

store_heartbeat_bench_test.go

volume: stop reporting read-only volumes that are no longer here (#10867)

2026-08-21 22:33:01 -07:00

store_heartbeat_digest_test.go

master: keep new volumes and writes off servers in maintenance mode (#11147)

2026-09-03 23:50:12 -07:00

store_load_balancing_simple_test.go

create new volumes on less occupied disk locations (#7349)

2025-10-20 16:11:29 -07:00

store_load_balancing_test.go

volume: cut idle memory at high volume counts (#10861)

2026-08-21 13:04:56 -07:00

store_mark_readonly_can_delete_test.go

Mount req with collection (#11249)

2026-09-09 12:55:47 -07:00

store_maxvolume_deadzone_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_mount_validation_test.go

fix(volume): Harden Volume Copy Validation and Failure Handling (#11252)

2026-09-09 23:33:05 -07:00

store_state.go

chore: remove ~50k lines of unreachable dead code (#8913)

2026-04-03 16:04:27 -07:00

store_status_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

store_vacuum_fs_unix.go

vacuum: check compaction space against live bytes, not volume size (#11524)

2026-10-03 08:31:56 +08:00

store_vacuum_fs_windows.go

vacuum: check compaction space against live bytes, not volume size (#11524)

2026-10-03 08:31:56 +08:00

store_vacuum_test.go

vacuum: check compaction space against live bytes, not volume size (#11524)

2026-10-03 08:31:56 +08:00

store_vacuum.go

volume server: ec.decode verifies, cleans up and compacts like Go, off the runtime (#11547)

2026-10-03 14:55:15 +08:00

store_volume_report_test.go

volume: cut the memory a server holding millions of volumes still uses (#10999)

2026-08-27 22:25:15 -07:00

store_volume_report.go

volume: cut the memory a server holding millions of volumes still uses (#10999)

2026-08-27 22:25:15 -07:00

store.go

ec: compare live idx NeedleId sets before encode (#11812)

2026-10-19 00:56:17 -07:00

volume_async_worker_test.go

shell: let volume.deleteEmpty drop volumes with no live needles (#11437)

2026-09-26 15:53:39 +08:00

volume_backup_test.go

refactoring

2019-04-17 22:04:49 -07:00

volume_backup.go

[Volume] Keep DAT and index state consistent after async batch Sync failure (#11425)

2026-09-24 06:57:44 +08:00

volume_checking_test.go

volume: make volume.scrub report a live needle whose stored id is damaged (#11468)

2026-09-26 16:10:14 +08:00

volume_checking.go

fix(volume): keep the TTL clock across a vacuum commit instead of rescanning (#11630)

2026-10-08 22:58:44 +08:00

volume_destroy_ec_vif_test.go

shell: let volume.deleteEmpty drop volumes with no live needles (#11437)

2026-09-26 15:53:39 +08:00

volume_idx_rebuild_test.go

volume: rebuild a missing .idx from the .dat (#11115)

2026-09-03 08:43:10 -07:00

volume_idx_rebuild.go

volume: rebuild a missing .idx from the .dat (#11115)

2026-09-03 08:43:10 -07:00

volume_idx_repair_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

volume_idx_repair.go

volume: recover .idx rows overwritten by tiered deletes (#10474)

2026-07-28 16:48:30 -07:00

volume_info_counts_test.go

storage: count a volume's needles in uint32 (#10718)

2026-08-11 16:36:36 -07:00

volume_info_intern_test.go

storage: stop keeping the remote storage key on the master (#10672)

2026-08-09 12:43:31 -07:00

volume_info_layout_test.go

storage: order VolumeInfo by alignment (#10669)

2026-08-09 09:41:50 -07:00

volume_info_test.go

master: keep disk_id when registering volumes from incremental heartbeats (#10686)

2026-08-10 00:40:29 -07:00

volume_info.go

volume: avoid read-only replica write targets (#11195)

2026-09-07 09:23:56 -07:00

volume_io_error_test.go

EC: handle zero-sized shard files uniformly (moves, rebuilds, startup cleanup) (#10753)

2026-08-13 21:38:22 -07:00

volume_loading_corrupt_idx_test.go

admin: allow setting volume read-only and read/write modes (#11217)

2026-09-07 18:40:37 -07:00

volume_loading_open_fail_test.go

volume: return error instead of panicking when .dat open fails (#11619)

2026-10-06 14:24:42 +08:00

volume_loading.go

ec: compare live idx NeedleId sets before encode (#11812)

2026-10-19 00:56:17 -07:00

volume_mark_writable_test.go

admin: allow setting volume read-only and read/write modes (#11217)

2026-09-07 18:40:37 -07:00

volume_read_all.go

[Volume] Keep DAT and index state consistent after async batch Sync failure (#11425)

2026-09-24 06:57:44 +08:00

volume_read_test.go

fix(volume): skip unnecessary needle data reads on delete (#11711)

2026-10-12 18:08:19 +08:00

volume_read.go

fix(volume): skip unnecessary needle data reads on delete (#11711)

2026-10-12 18:08:19 +08:00

volume_report_hash.go

volume: avoid read-only replica write targets (#11195)

2026-09-07 09:23:56 -07:00

volume_super_block_replication_test.go

test: pin that a .vif replication outranks the superblock (#10499)

2026-07-30 17:03:38 -07:00

volume_super_block.go

volume: open volume files with O_NOATIME (#11055)

2026-08-31 21:41:50 -07:00

volume_tier_intern_test.go

volume: cut the memory a server holding millions of volumes still uses (#10999)

2026-08-27 22:25:15 -07:00

volume_tier.go

volume: expire TTL volumes whose only traffic is deletes (#11167)

2026-09-04 23:48:40 -07:00

volume_ttl_expiry_test.go

fix(volume): keep the TTL clock across a vacuum commit instead of rescanning (#11630)

2026-10-08 22:58:44 +08:00

volume_vacuum_5byte_offset_test.go

volume: fix 5-byte index offset corruption in makeupDiff (#11411)

2026-09-21 00:59:35 -07:00

volume_vacuum_crash_safe_test.go

shell: let volume.deleteEmpty drop volumes with no live needles (#11437)

2026-09-26 15:53:39 +08:00

volume_vacuum_test.go

storage: unify needle-map counters across index loaders and warn on copy-count drift (#11701)

2026-10-10 22:31:46 +08:00

volume_vacuum.go

fix(volume): keep the TTL clock across a vacuum commit instead of rescanning (#11630)

2026-10-08 22:58:44 +08:00

volume_write_fsync_test.go

volume server: refuse the rest of a grouped run after a durable index failure (#11576)

2026-10-03 21:04:30 +08:00

volume_write_test.go

ec: compare live idx NeedleId sets before encode (#11812)

2026-10-19 00:56:17 -07:00

volume_write.go

ec: compare live idx NeedleId sets before encode (#11812)

2026-10-19 00:56:17 -07:00

volume.go

fix(volume): keep the TTL clock across a vacuum commit instead of rescanning (#11630)

2026-10-08 22:58:44 +08:00