* fix(volume): keep the TTL clock across a vacuum commit instead of rescanning
CommitCompact reloads the swapped files while holding dataFileAccessLock,
and for a vacuumed TTL volume that reload re-derived lastModifiedTsSeconds
by reading every live needle's append timestamp from the .dat: two random
reads per needle, with every read of the volume blocked behind them.
The in-memory clock is already current at that point. Every write since the
volume loaded moved it, and makeupDiff only replays writes that went through
that path. Carry it across the reload instead. This also stops an
over-budget scan from falling back to the new .dat's mtime and restarting
an expiring volume's TTL at the commit.
* volume: carry the append watermark as the TTL clock across a vacuum commit
The running append watermark is the clock the reload's recovery scan
recomputes, so the commit can keep it directly. Client-supplied needle
modified times can run ahead of or behind the append time; keeping
lastModifiedTsSeconds itself would let a forged or stale timestamp move
expiry through a vacuum, where the scan it replaces used server-side
append timestamps.
* storage: test that a vacuum commit keeps the append clock
A write's client supplied modified time can lie ahead of or behind its
append time; the commit must land the TTL clock on the append watermark,
the same value the recovery scan would have recomputed.
* volume: carry the append watermark as the TTL clock across a vacuum commit
Mirrors the Go volume server: the running append watermark is the clock
the reload's recovery scan recomputes, so the commit keeps it instead of
rescanning live needles under the write lock.
* volume: commit carries the last-write append time, not the latest append
lastAppendAtNs counts tombstone appends and is reseeded from the .dat
tail at every load, so it can sit ahead of the last write -- a delete
freshens the commit clock -- or behind it: a restarted vacuumed volume's
tail needle is not its newest write, and the commit would move the TTL
clock backward into premature expiry.
Track lastWriteAppendAtNs instead, bumped only on needle appends and
seeded by the recovery scan, so the commit lands the clock on the same
live-write maximum the rescan would have recomputed.
* volume: rescan at commit when the newest write was deleted
lastWriteAppendAtNs can hold a write the index no longer holds, so
carrying it extends the TTL clock past what recovery over the compacted
index would compute. Remember the key behind the watermark so its
tombstone or index rollback can send the reload back through
recoverLastModifiedTs, landing on the newest surviving write.
* volume: a tombstone retires the write rows beneath it in the last-write scan
A needle deleted after the compaction copy leaves its write row followed
by a tombstone in the committed index. The reverse scan skipped the
tombstone row but then counted the dead write, reseeding the watermark
and flag as if it were alive. Track keys whose latest row is a tombstone
so their earlier write rows stop counting, and exercise the
delete-inside-the-commit-window ordering in the tests.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* volume: skip TTL last-write scan when it cannot fit its budget
* Update weed/storage/volume_checking.go
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* volume: count a TTL volume's age from its last write, not the .dat mtime
A delete appends a tombstone needle and vacuum rewrites the .dat wholesale,
so the file's mtime moves without any write ever landing. The loader read
lastModifiedTsSeconds back from that mtime, so every restart of a volume
taking delete traffic re-armed expired() for another full TTL: an
overwrite-heavy collection kept growing until it hit the max-volume cap.
Recover the clock from the newest .idx entry that is not a tombstone and
read that needle's append timestamp, falling back to the mtime when no
write is recoverable. Only TTL volumes pay for the scan.
Fixes#11160
* volume: count the .vif destroy time from the last write too
ExpireAtSec is what an EC volume is reclaimed on, and it was recomputed as
now+TTL every time the .vif was written. A read-only mark, a tier upload or
an EC encode therefore handed an already expiring volume another full TTL,
the same way the .dat mtime did.
Derive it from the volume's last write, falling back to now for a volume
that has not taken one yet so a fresh volume is not born expired.
* volume: mirror the last-write TTL clock in the Rust volume server
Same recovery as the Go loader: scan the .idx backwards for the newest
entry that is not a tombstone and take that needle's append timestamp,
leaving the clock on the .dat mtime when no write is recoverable.
* volume: mirror the last-write destroy time in the Rust volume server
Both .vif writers and the EC encode computed ExpireAtSec as now+TTL, the
same way Go did, so the destroy time moved every time the sidecar was
rewritten. Route all three through the volume's last write.
* volume: report the .dat mtime in the Rust heartbeat, like Go does
The Rust server reported its TTL clock as ModifiedAtSecond while Go
reports the .dat mtime. The shell's quiet-period gates (volume.tier.move,
volume.delete_empty) read that field as "last touched", which a delete
has to count towards even though the TTL clock deliberately ignores it --
and with the clock now recovered from the last write, the two drift
further apart.
* volume: take the newest write by timestamp on a vacuumed volume
The reverse .idx scan trusted position, which holds only while the .dat is
append ordered. Vacuum rewrites it in key order, and since an overwrite
keeps its original key, the highest-key survivor is not necessarily the
newest write -- the recovered clock could land up to a TTL early and take
the volume with data still inside its TTL.
A volume that has been vacuumed (CompactionRevision > 0) now takes the
maximum append timestamp over a bounded window of write entries instead.
An append-ordered volume still answers in one read.
* volume: never guess a vacuumed volume's last write, and resolve wrapped offsets
Two holes in the reverse scan, both from review:
A vacuumed volume's writes are ordered by key, so any of them can hold the
newest timestamp. Reading a capped window sampled the highest keys, which
could still miss a recently overwritten low-key needle and expire data
inside its TTL. The scan now covers every write a vacuumed volume indexes,
and a volume too large to scan keeps the .dat mtime rather than report a
partial maximum -- late is recoverable, early is not.
A .dat past MaxPossibleVolumeSize wraps the offsets in its .idx, so reading
a timestamp at the unwrapped offset picks up an unrelated needle. Resolve
the entry against the needle header first and retry one volume size in,
the way doCheckAndFixVolumeData already does.
* volume: drop GitHub issue references from TTL comments