Chris Lu 70c3adb983 volume: stop read-only volumes from pinning .idx and .sdx (#10950) ...

A read-only or cloud-tiered volume loads a SortedFileNeedleMap, which held
both its .idx and its .sdx open for the life of the process. On a server with
~600K tiered volumes that is 1.2M descriptors before a single read, enough to
exhaust the fd limit and take the listeners down. The .dat is not the problem:
a tiered volume serves it from the remote backend.

Neither index file is needed except while a lookup is in flight, so borrow them
from a bounded process-wide pool instead. An idle volume now holds zero
descriptors; a busy one keeps its handles hot rather than paying an open() per
needle. Reads borrow O_RDONLY, so a volume on a read-only mount answers lookups
that previously failed at load. Sync tracks whether a tombstone was appended,
which also drops the fsync-per-volume storm at shutdown.

2026-08-25 10:23:00 -07:00

..

backend

Add codespell support with configuration and typo fixes (#10393)

2026-07-22 14:38:06 -07:00

erasure_coding

ec: refund the cleared leftover shards' slots in the encode source health check (#10903)

2026-08-23 21:48:20 -07:00

idx

fix(scrub): don't flag offset-0 logical tombstones in volume scrub (#10148)

2026-06-30 02:01:14 -07:00

needle

EC: handle zero-sized shard files uniformly (moves, rebuilds, startup cleanup) (#10753)

2026-08-13 21:38:22 -07:00

needle_map

Add codespell support with configuration and typo fixes (#10393)

2026-07-22 14:38:06 -07:00

super_block

perf(weed/storage/super_block): intern the byte-encoded replica placements (#10610)

2026-08-07 00:53:02 -07:00

test_files

Have volume scrubs account for zero-sized volumes. (#9609)

2026-05-21 09:42:07 -07:00

types

volume.fsck: fix orphan purge against the rust volume server (#10289)

2026-07-09 11:19:31 -07:00

volume_info

storage: register tier backends at the binary composition root (#9989)

2026-06-16 11:47:32 -07:00

volume_replica

fix(ec): bring ec.encode worker and EC/volume helpers to parity with shell (#9599)

2026-05-21 02:16:28 -07:00

disk_location_ec_realworld_test.go

EC bitrot detection: per-shard checksum sidecars (#9761)

2026-05-31 18:52:44 -07:00

disk_location_ec_shard_size_test.go

fix(volume_server): load orphan EC shards across disks on startup (#9212) (#9244)

2026-04-27 16:01:10 -07:00

disk_location_ec_test.go

EC: handle zero-sized shard files uniformly (moves, rebuilds, startup cleanup) (#10753)

2026-08-13 21:38:22 -07:00

disk_location_ec.go

EC: handle zero-sized shard files uniformly (moves, rebuilds, startup cleanup) (#10753)

2026-08-13 21:38:22 -07:00

disk_location_test.go

Implement RPC skeleton for regular/EC volumes scrubbing. (#8187)

2026-02-02 17:55:04 -08:00

disk_location.go

heartbeat: keep the master current through collection churn (#10657)

2026-08-08 20:23:10 -07:00

ec_startup_shard_safety_test.go

fix(ec): never delete recoverable EC shards on startup/reconcile (the non-empty-.dat sibling of the stub bug) (#9941)

2026-06-12 23:51:29 -07:00

needle_map_file_pool_test.go

volume: stop read-only volumes from pinning .idx and .sdx (#10950)

2026-08-25 10:23:00 -07:00

needle_map_file_pool.go

volume: stop read-only volumes from pinning .idx and .sdx (#10950)

2026-08-25 10:23:00 -07:00

needle_map_leveldb_test.go

fix(volume): persist the leveldb needle map watermark at batch boundaries (#10557)

2026-08-04 13:33:31 -07:00

needle_map_leveldb.go

storage: fix corrupted leveldb detection in DoOffsetLoading (#10650)

2026-08-08 06:00:46 -07:00

needle_map_memory.go

fix(volume): keep vacuum running past dangling .idx entries (#9115)

2026-04-16 22:01:34 -07:00

needle_map_metric_test.go

fix(volume): keep vacuum running past dangling .idx entries (#9115)

2026-04-16 22:01:34 -07:00

needle_map_metric.go

fix(volume): keep vacuum running past dangling .idx entries (#9115)

2026-04-16 22:01:34 -07:00

needle_map_sorted_file_test.go

fix(volume): seed indexFileOffset in SortedFileNeedleMap so Delete appends (#9483)

2026-05-13 10:22:01 -07:00

needle_map_sorted_file.go

volume: stop read-only volumes from pinning .idx and .sdx (#10950)

2026-08-25 10:23:00 -07:00

needle_map.go

fix(volume): keep vacuum running past dangling .idx entries (#9115)

2026-04-16 22:01:34 -07:00

remote_tier_integration_test.go

volume: cut idle memory at high volume counts (#10861)

2026-08-21 13:04:56 -07:00

store_collection_metrics_test.go

volume: stop reporting read-only volumes that are no longer here (#10867)

2026-08-21 22:33:01 -07:00

store_consolidate_index_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

store_delete_collection_test.go

heartbeat: keep the master current through collection churn (#10657)

2026-08-08 20:23:10 -07:00

store_disk_space_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_duplicate_vid_test.go

fix(storage): delete/unmount every copy of a duplicate volume id (#9954)

2026-06-14 06:36:47 -07:00

store_ec_9plus3_reboot_repro_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_ec_delete.go

admin: report file and delete counts for EC volumes (#9060)

2026-04-13 21:10:36 -07:00

store_ec_disk_type_test.go

batch drain delta heartbeat messages (#9914)

2026-06-10 13:33:45 -07:00

store_ec_hybrid_repro_test.go

fix(ec): never delete recoverable EC shards on startup/reconcile (the non-empty-.dat sibling of the stub bug) (#9941)

2026-06-12 23:51:29 -07:00

store_ec_interval_read_test.go

ec: read a needle's intervals in parallel (#10911)

2026-08-24 14:03:44 -07:00

store_ec_mirror_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_ec_mirror.go

fix(ec): mirror EC sidecars onto every shard-bearing disk at startup (#9525)

2026-05-17 19:55:15 -07:00

store_ec_mount_cross_disk_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_ec_multidisk_lifecycle_test.go

test: assert EC shard identity and empty-view in multi-disk lifecycle tests (#10723)

2026-08-12 20:18:39 -07:00

store_ec_orphan_shard_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_ec_phantom_dat_test.go

fix(storage): never let an empty .dat delete healthy distributed EC shards (#9930)

2026-06-11 20:26:20 -07:00

store_ec_reconcile.go

Add codespell support with configuration and typo fixes (#10393)

2026-07-22 14:38:06 -07:00

store_ec_recover_test.go

ec: recover EC shards whose .ecx index lives only on a peer server (#10108)

2026-06-25 10:38:14 -07:00

store_ec_recover.go

ec: recover EC shards whose .ecx index lives only on a peer server (#10108)

2026-06-25 10:38:14 -07:00

store_ec_recovery_test.go

Update store_ec_recovery_test.go

2026-01-23 16:38:36 -08:00

store_ec_scrub.go

EC scrubbing: list shards for needles failing scrubs in the result output. (#10510)

2026-08-01 00:27:06 -07:00

store_ec_target_location_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_ec.go

ec: read a needle's intervals in parallel (#10911)

2026-08-24 14:03:44 -07:00

store_heartbeat_bench_test.go

volume: stop reporting read-only volumes that are no longer here (#10867)

2026-08-21 22:33:01 -07:00

store_heartbeat_digest_test.go

volume: stop reporting read-only volumes that are no longer here (#10867)

2026-08-21 22:33:01 -07:00

store_load_balancing_simple_test.go

create new volumes on less occupied disk locations (#7349)

2025-10-20 16:11:29 -07:00

store_load_balancing_test.go

volume: cut idle memory at high volume counts (#10861)

2026-08-21 13:04:56 -07:00

store_mark_readonly_can_delete_test.go

shell: multi-target fs.mergeVolumes and volume.mark -readonlyCanDelete (#10706)

2026-08-10 16:31:26 -07:00

store_maxvolume_deadzone_test.go

[CheckDisk]: implement disk health detection (#9560)

2026-06-02 09:02:05 -07:00

store_state.go

chore: remove ~50k lines of unreachable dead code (#8913)

2026-04-03 16:04:27 -07:00

store_status_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

store_vacuum_test.go

fix tests

2025-06-30 13:57:28 -07:00

store_vacuum.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

store_volume_report_test.go

volume: stop reporting read-only volumes that are no longer here (#10867)

2026-08-21 22:33:01 -07:00

store_volume_report.go

volume: cut idle memory at high volume counts (#10861)

2026-08-21 13:04:56 -07:00

store.go

ec: refund the cleared leftover shards' slots in the encode source health check (#10903)

2026-08-23 21:48:20 -07:00

volume_async_worker_test.go

volume: cut idle memory at high volume counts (#10861)

2026-08-21 13:04:56 -07:00

volume_backup_test.go

refactoring

2019-04-17 22:04:49 -07:00

volume_backup.go

add version to volume proto

2025-06-16 22:05:06 -07:00

volume_checking_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

volume_checking.go

fix(scrub): don't flag offset-0 logical tombstones in volume scrub (#10148)

2026-06-30 02:01:14 -07:00

volume_destroy_ec_vif_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

volume_idx_repair_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

volume_idx_repair.go

volume: recover .idx rows overwritten by tiered deletes (#10474)

2026-07-28 16:48:30 -07:00

volume_info_counts_test.go

storage: count a volume's needles in uint32 (#10718)

2026-08-11 16:36:36 -07:00

volume_info_intern_test.go

storage: stop keeping the remote storage key on the master (#10672)

2026-08-09 12:43:31 -07:00

volume_info_layout_test.go

storage: order VolumeInfo by alignment (#10669)

2026-08-09 09:41:50 -07:00

volume_info_test.go

master: keep disk_id when registering volumes from incremental heartbeats (#10686)

2026-08-10 00:40:29 -07:00

volume_info.go

storage: count a volume's needles in uint32 (#10718)

2026-08-11 16:36:36 -07:00

volume_io_error_test.go

EC: handle zero-sized shard files uniformly (moves, rebuilds, startup cleanup) (#10753)

2026-08-13 21:38:22 -07:00

volume_loading_corrupt_idx_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

volume_loading.go

shell: multi-target fs.mergeVolumes and volume.mark -readonlyCanDelete (#10706)

2026-08-10 16:31:26 -07:00

volume_mark_writable_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

volume_read_all.go

Include name/mime in ReadAllNeedles (#4005)

2022-11-23 15:59:38 -08:00

volume_read_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

volume_read.go

fix(volume): sticky EIO quarantine; track streamed reads (#9384)

2026-05-09 09:55:02 -07:00

volume_report_hash.go

storage: stop keeping the remote storage key on the master (#10672)

2026-08-09 12:43:31 -07:00

volume_super_block_replication_test.go

test: pin that a .vif replication outranks the superblock (#10499)

2026-07-30 17:03:38 -07:00

volume_super_block.go

Fix: Fail fast on unsupported volume versions (#8047)

2026-01-16 19:19:18 -08:00

volume_tier.go

volume: reload a remote-tiered volume without re-entering the data lock (#10266)

2026-07-08 01:27:10 -07:00

volume_vacuum_crash_safe_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

volume_vacuum_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

volume_vacuum.go

volume: skip directory fsync on Windows, report a failed makeupDiff (#10572)

2026-08-04 21:02:52 -07:00

volume_write_fsync_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

volume_write_test.go

volume: forward fsync=true to replicas in ReplicatedWrite (#10805)

2026-08-18 17:17:50 -07:00

volume_write.go

volume: cut idle memory at high volume counts (#10861)

2026-08-21 13:04:56 -07:00

volume.go

volume: stop reporting read-only volumes that are no longer here (#10867)

2026-08-21 22:33:01 -07:00