* fix(volume): keep preallocated growth above -volume.minFreeSpace
When free space fell below -volume.minFreeSpace, every volume went
read-only and the master kept growing new volumes. With preallocation on,
each new volume reserved its full size, driving the disk further below the
floor (15 Oct incident: 66 empty volumes reserved about 1980 GiB).
Two causes, both in weed/storage/store.go:
- MaybeAdjustVolumeMax sized a disk's volume slots from raw free space, so
space inside the floor was counted as available. It now counts only free
space above the floor.
- addVolume trusted the once-a-minute low-disk flag. A preallocated volume
is now refused when the disk is low now, or when its reservation would
not fit above the floor. Volumes without preallocation, including merges
that copy existing data, are not affected.
addVolume runs under a store-level lock, so two concurrent creates cannot
both pass the room check against the same free space.
* volume: mirror headroom guard in the rust store
find_free_location only trusted the once-a-probe is_disk_space_low flag,
so a burst of preallocated creates could still run the disk under
-volume.minFreeSpace between probes, and maybe_adjust_volume_max sized
slots from raw free space. Adds MinFreeSpace::is_low/available_space to
match Go and applies both checks the same way. store.write() already
serializes add_volume, so no extra lock is needed.
* volume: read rust headroom tests from a fake disk
The floor arithmetic used a live free-space reading, so unrelated disk
activity could flip a test between pass and fail. Store.disk_stats_of is
now the same seam Go uses (diskStatusOf): the growth-loop and floor tests
run against a controlled disk, mirroring the Go suite one for one.
* volume: hold back create metadata from the headroom check
An exact-fit preallocation left free space exactly at the floor, so the
.vif and .idx writes that follow a create could dip under it and get the
location flagged read-only at the next probe. Reserve room for .vif on
the data filesystem, and for .idx growth only when dir_idx shares it.
* volume: mirror the metadata allowance in the rust store
Same rule as the Go side: charge .vif against the checked filesystem and
.idx only when dir_idx shares it, and grow the exact-fit test's fake
disk by the per-create cost.
* volume: skip the reservation check where create cannot preallocate
* volume: mirror the platform preallocation gate in the rust store
* volume.copy: count the preallocation reservation in the destination space check
The copy path picked a location with room for the copied .dat, then reserved
the master's full volume size limit on it. A volume smaller than the limit
could land on a disk whose free space above -volume.minFreeSpace fit the
copy but not the reservation, driving the disk under the floor the same way
the growth loop did.
Fetch the preallocation setting before choosing the location and require
room for the reservation itself, only where CreateVolumeFile actually
reserves blocks.
* seaweed-volume: store min-free-space-adjusted available space like Go
Go's DiskLocation.AvailableSpace holds free space above the
-volume.minFreeSpace floor; the Rust probe cached raw free instead, so the
VolumeCopy location check could accept a disk already inside the floor.
Store the same adjusted figure and cover it with a test.
* seaweed-volume: reserve the copied .dat like Go's CreateVolumeFile
VolumeCopy called set_len for the reservation, which only grows a sparse
file — no blocks are held, so a disk with room for the copied bytes but
not the full reservation only fails mid-copy. preallocate_file fallocates
KEEP_SIZE on Linux like Go, reserving the blocks up front; off Linux it
is a no-op, matching PREALLOCATE_SUPPORTED.
* seaweed-volume: charge a copy's destination pick for the .dat reservation
Go's VolumeCopy now sizes its free-location check against the master's
volume size limit when the platform preallocates, not just the copied
bytes. The Rust pick only measured the copied bytes, so a destination
could accept a copy whose reservation then dips under the floor. The
config fetch moves ahead of the pick (same order as Go) and feeds both
the space check and the reservation itself.
* volume.copy: charge the picked location's free space under volumeCreateLock
The copy path picked a destination off the once-a-minute AvailableSpace
probe and released the decision long before the .dat reservation landed.
Two copies could pass the same cached headroom and together drive the disk
under -volume.minFreeSpace.
ReserveFreeLocation serializes the pick under volumeCreateLock, verifies
the disk still has live room for the reservation, and charges it against
the location's cached free space until the next probe refreshes it.
* seaweed-volume: mirror the copy pick's live-check and headroom charge
Go's VolumeCopy now picks through Store.ReserveFreeLocation: live disk
stats under the store write lock, then the reservation charged against the
location's cached free space. The Rust plan_copy_destination gains the
same store.reserve_free_location so two copies cannot spend the same
headroom before the first .dat reservation lands.
* volume.copy: saturate the free-space charge against a live-space drop
The two statfs reads bracketing a pick can race a large external write; a
plain subtraction would underflow and leave the location looking roomy
until the next probe.
* volume.copy: release the headroom reservation when a copy does not land
ReserveFreeLocation charges the picked location's cached free space, but
a failed copy never returned it, so the headroom stayed booked until the
next disk-space probe and blocked retries. ReleaseFreeLocation refunds
the charge; the copy's cleanup defer now registers with the reservation
so every failure path (existing-replica delete, .note write, stream send)
releases it, including the post-transfer mount/validate failures covered
by the outer defer.
* seaweed-volume: mirror the copy headroom release on failed copies
release_free_location refunds reserve_free_location's charge. The copy
returns it when the plan's existing-replica delete fails, when the .note
write fails, and in cleanup_failed_copy, so a failed copy stops holding
the headroom until the next disk-space probe.
* volume.copy: keep pending copy headroom out of probe readings
ReleaseFreeLocation refunded the reservation into AvailableSpace even when
a CheckDiskSpace probe had already rebased the cache on live stats — the
probe's reading still shows the in-flight copy's bytes as free, so the
refund double-counted headroom and two copies could spend the same space,
driving the disk under -volume.minFreeSpace.
Track the outstanding charge in DiskLocation.ReservedSpace instead: probes
subtract it so they cannot re-spend it, a failed copy removes only its own
charge, and a landed copy commits it — the charge and the new usage cancel
out. hasRoomForNewVolume respects pending headroom too, so a preallocating
create cannot overrun an in-flight copy.
* seaweed-volume: mirror pending copy headroom across disk probes
reserved_space tracks the in-flight copy's charge separately from the
cached free space: check_disk_space subtracts it, release removes only the
failed copy's charge, a landed copy commits it, and has_room_for_new_volume
respects it — matching the Go side.
* volume.copy: settle a copy's headroom as its bytes land on disk
The pending charge held the copy's full reservation until commit, but
written and fallocated bytes already show up in statfs, so the live-space
check counted them twice. A slow copy could then block other copies and
volume growth despite free space above -volume.minFreeSpace.
Settle the charge instead: commit the bytes CreateVolumeFile actually
reserved (a failed fallocate leaves nothing committed), then settle each
file's write progress as it lands. The failure paths now release only the
unsettled remainder; allocatedSize is exported for the .dat's st_blocks.
* seaweed-volume: mirror copy headroom settling as bytes land
reserved_space keeps only the promised bytes statfs cannot see yet:
preallocate_dat settles whatever blocks the filesystem actually reserved
(0 when fallocate cannot), each copied file's progress settles the rest as
it lands, and a failed copy releases only the unsettled remainder —
matching the Go side.
* volume.copy: release a failed copy's headroom charge only once
Both defers fired on the same in-closure error: the inner released the
unsettled charge and the outer released it again, driving ReservedSpace
negative (an atomic.Uint64 underflow) and inflating AvailableSpace —
the location would then fail every later headroom check, or worse,
pass them with fabricated space. Zero the charge once it is released;
the Rust mirror already releases via swap(0) for the same reason.
* volume.copy: write into the preallocated .dat instead of staging over it
The .dat stream staged into .copying and renamed over the file
CreateVolumeFile had just fallocated, so the copy's transient footprint
was preallocate+dat — more than the reservation the pick just checked —
and the reservation itself was discarded at rename. Writing in place
reuses the fallocated extent, keeps the peak at the reserved maximum,
and leaves the copied volume preallocated like a created one.
* seaweed-volume: mirror the in-place write into the preallocated .dat
The .dat copy opened the just-fallocated file with truncate, freeing its
blocks at open and spending the headroom again as the stream re-grew it.
Append mode preserves them, matching the Go side.
---------
Co-authored-by: chrislusf <chrislusf@gmail.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* volume server: open volume files with O_NOATIME
Nothing reads the atime of .dat, .idx, .sdx, or EC files, but every
needle read still dirtied the inode: even relatime writes atime on the
first read after each write, so an actively written volume paid a
metadata write per read/write cycle, and strictatime mounts paid one
per read. Open the serving handles with O_NOATIME, falling back to a
plain open when the file belongs to another owner (EPERM).
Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD
* seaweed-volume: mirror the O_NOATIME volume file opens
Same change as the Go volume server: serving handles for .dat, .idx,
.sdx, .ecx, .ecj, and shard files open with O_NOATIME on Linux, with a
plain-open fallback on EPERM.
Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD
* route the tier-down and recreate .dat opens through the no-atime helper
Review caught the Rust tier-down swap opening the local .dat directly.
The Go swapToLocalDatBackend and the zero-length read-only .dat
recreate in maybeWriteSuperBlock had the same gap: all three install
long-lived serving handles.
Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD
