volume: expire TTL volumes whose only traffic is deletes (#11167)

* volume: count a TTL volume's age from its last write, not the .dat mtime

A delete appends a tombstone needle and vacuum rewrites the .dat wholesale,
so the file's mtime moves without any write ever landing. The loader read
lastModifiedTsSeconds back from that mtime, so every restart of a volume
taking delete traffic re-armed expired() for another full TTL: an
overwrite-heavy collection kept growing until it hit the max-volume cap.

Recover the clock from the newest .idx entry that is not a tombstone and
read that needle's append timestamp, falling back to the mtime when no
write is recoverable. Only TTL volumes pay for the scan.

Fixes #11160

* volume: count the .vif destroy time from the last write too

ExpireAtSec is what an EC volume is reclaimed on, and it was recomputed as
now+TTL every time the .vif was written. A read-only mark, a tier upload or
an EC encode therefore handed an already expiring volume another full TTL,
the same way the .dat mtime did.

Derive it from the volume's last write, falling back to now for a volume
that has not taken one yet so a fresh volume is not born expired.

* volume: mirror the last-write TTL clock in the Rust volume server

Same recovery as the Go loader: scan the .idx backwards for the newest
entry that is not a tombstone and take that needle's append timestamp,
leaving the clock on the .dat mtime when no write is recoverable.

* volume: mirror the last-write destroy time in the Rust volume server

Both .vif writers and the EC encode computed ExpireAtSec as now+TTL, the
same way Go did, so the destroy time moved every time the sidecar was
rewritten. Route all three through the volume's last write.

* volume: report the .dat mtime in the Rust heartbeat, like Go does

The Rust server reported its TTL clock as ModifiedAtSecond while Go
reports the .dat mtime. The shell's quiet-period gates (volume.tier.move,
volume.delete_empty) read that field as "last touched", which a delete
has to count towards even though the TTL clock deliberately ignores it --
and with the clock now recovered from the last write, the two drift
further apart.

* volume: take the newest write by timestamp on a vacuumed volume

The reverse .idx scan trusted position, which holds only while the .dat is
append ordered. Vacuum rewrites it in key order, and since an overwrite
keeps its original key, the highest-key survivor is not necessarily the
newest write -- the recovered clock could land up to a TTL early and take
the volume with data still inside its TTL.

A volume that has been vacuumed (CompactionRevision > 0) now takes the
maximum append timestamp over a bounded window of write entries instead.
An append-ordered volume still answers in one read.

* volume: never guess a vacuumed volume's last write, and resolve wrapped offsets

Two holes in the reverse scan, both from review:

A vacuumed volume's writes are ordered by key, so any of them can hold the
newest timestamp. Reading a capped window sampled the highest keys, which
could still miss a recently overwritten low-key needle and expire data
inside its TTL. The scan now covers every write a vacuumed volume indexes,
and a volume too large to scan keeps the .dat mtime rather than report a
partial maximum -- late is recoverable, early is not.

A .dat past MaxPossibleVolumeSize wraps the offsets in its .idx, so reading
a timestamp at the unwrapped offset picks up an unrelated needle. Resolve
the entry against the needle header first and retry one volume size in,
the way doCheckAndFixVolumeData already does.

* volume: drop GitHub issue references from TTL comments

weed/server/volume_grpc_erasure_coding.go

@@ -139,15 +139,8 @@ func (vs *VolumeServer) VolumeEcShardsGenerate(ctx context.Context, req *volume_
 	}
 
 	// write .vif files
-	var expireAtSec uint64
-	if v.Ttl != nil {
-		ttlSecond := v.Ttl.ToSeconds()
-		if ttlSecond > 0 {
-			expireAtSec = uint64(time.Now().Unix()) + ttlSecond //calculated expiration time
-		}
-	}
 	volumeInfo := &volume_server_pb.VolumeInfo{Version: uint32(v.Version())}
-	volumeInfo.ExpireAtSec = expireAtSec
+	volumeInfo.ExpireAtSec = v.ExpireAtSec()
 	// The size the encode actually read, not a separate stat: a replica-sync
 	// write can land between two stats of a live .dat, and the .vif would then
 	// record a DatFileSize and a BlockSize describing different files.

weed/storage/volume.go

@@ -442,6 +442,25 @@ func (v *Volume) expired(contentSize uint64, volumeSizeLimit uint64) bool {
 	return false
 }
 
+// ExpireAtSec is when this volume's data becomes garbage, counted from its last
+// write. Counting from the current time instead let every .vif rewrite -- a
+// read-only mark, a tier upload, an EC encode -- hand an already expiring volume
+// another full TTL. Zero when the volume has no TTL.
+func (v *Volume) ExpireAtSec() uint64 {
+	if v.Ttl == nil {
+		return 0
+	}
+	ttlSeconds := v.Ttl.ToSeconds()
+	if ttlSeconds == 0 {
+		return 0
+	}
+	lastWriteSec := v.lastModifiedTsSeconds
+	if lastWriteSec == 0 {
+		lastWriteSec = uint64(time.Now().Unix())
+	}
+	return lastWriteSec + ttlSeconds
+}
+
 // wait either maxDelayMinutes or 10% of ttl minutes
 func (v *Volume) expiredLongEnough(maxDelayMinutes uint32) bool {
 	if v.Ttl == nil || v.Ttl.Minutes() == 0 {

weed/storage/volume_checking.go

@@ -4,6 +4,7 @@ import (
 	"fmt"
 	"io"
 	"os"
+	"time"
 
 	"github.com/seaweedfs/seaweedfs/weed/glog"
 	"github.com/seaweedfs/seaweedfs/weed/storage/backend"
@@ -258,6 +259,124 @@ func doCheckAndFixVolumeData(v *Volume, indexFile *os.File, indexOffset int64) (
 	return lastAppendAtNs, nil
 }
 
+// recoverLastModifiedTs points the TTL clock at the newest write recorded in
+// the volume, replacing the .dat mtime the loader starts from. A delete appends
+// a tombstone and vacuum rewrites the .dat wholesale, so the mtime moves
+// without any write: every restart of a volume taking delete traffic re-armed
+// expired() for another full TTL and the volume was never reclaimed. Left on
+// the mtime when no write is recoverable.
+func (v *Volume) recoverLastModifiedTs(indexFile *os.File) {
+	if v.Ttl == nil || v.Ttl.Minutes() == 0 {
+		return
+	}
+	indexSize, err := verifyIndexFileIntegrity(indexFile)
+	if err != nil || indexSize == 0 {
+		return
+	}
+	appendAtNs, err := findLastWriteAppendAtNs(v, indexFile, indexSize)
+	if err != nil {
+		glog.Warningf("volume %d recover last write from %s: %v", v.Id, indexFile.Name(), err)
+		return
+	}
+	if appendAtNs == 0 {
+		return
+	}
+	v.lastModifiedTsSeconds = appendAtNs / uint64(time.Second)
+}
+
+// vacuumedLastWriteScanEntries bounds the work a vacuumed volume's recovery
+// does, where key order makes every write a candidate for the newest one. A
+// volume with more live needles than this keeps the .dat mtime: reading a
+// subset could recover a timestamp older than the newest write and expire data
+// still inside its TTL, so a scan that will not fit declines instead of
+// guessing. A variable so tests can exercise that path.
+var vacuumedLastWriteScanEntries = 1 << 16
+
+// findLastWriteAppendAtNs scans the .idx backwards for the newest write -- an
+// entry that is not a deletion tombstone -- and returns that needle's append
+// timestamp. The .idx and the .dat share an order, so an append-ordered volume
+// answers with the first write the scan reaches. Vacuum rewrites both in key
+// order, which tracks write order only because the master issues keys
+// increasing: an overwrite keeps its original, lower key, so a vacuumed volume
+// has to take the maximum over every write it indexes. Returns 0 when the .idx
+// holds nothing but tombstones, when a vacuumed volume holds more needles than
+// the scan budget, or for a volume older than version 3, whose needles carry no
+// append timestamp.
+func findLastWriteAppendAtNs(v *Volume, indexFile *os.File, indexSize int64) (uint64, error) {
+	version := v.Version()
+	if version != needle.Version3 {
+		return 0, nil
+	}
+	scanEveryWrite := v.SuperBlock.CompactionRevision > 0
+	entryBudget := vacuumedLastWriteScanEntries
+	var lastWriteAppendAtNs uint64
+	block := make([]byte, types.NeedleMapEntrySize*idx.RowsToRead)
+	for end := indexSize; end > 0; {
+		start := max(end-int64(len(block)), 0)
+		entries := block[:end-start]
+		readCount, err := indexFile.ReadAt(entries, start)
+		if err == io.EOF && readCount == len(entries) {
+			err = nil
+		}
+		if err != nil {
+			return 0, fmt.Errorf("read %s at %d: %v", indexFile.Name(), start, err)
+		}
+		for i := len(entries) - types.NeedleMapEntrySize; i >= 0; i -= types.NeedleMapEntrySize {
+			key, offset, size := idx.IdxFileEntry(entries[i : i+types.NeedleMapEntrySize])
+			if offset.IsZero() || size.IsDeleted() {
+				continue
+			}
+			needleOffset := findNeedleOffset(v.DataBackend, version, offset.ToActualOffset(), key, size)
+			if needleOffset < 0 {
+				continue
+			}
+			appendAtNs, err := readNeedleAppendAtNs(v.DataBackend, needleOffset, size)
+			if err != nil {
+				return 0, err
+			}
+			lastWriteAppendAtNs = max(lastWriteAppendAtNs, appendAtNs)
+			if !scanEveryWrite {
+				return lastWriteAppendAtNs, nil
+			}
+			if entryBudget--; entryBudget == 0 {
+				glog.V(0).Infof("volume %d: more than %d needles to scan for its last write, keeping the %s mtime",
+					v.Id, vacuumedLastWriteScanEntries, v.FileName(".dat"))
+				return 0, nil
+			}
+		}
+		end = start
+	}
+	return lastWriteAppendAtNs, nil
+}
+
+// findNeedleOffset returns the .dat offset holding the needle an .idx entry
+// describes, or -1 when no needle there matches it. A .dat past
+// MaxPossibleVolumeSize wraps the 4-byte offsets in its .idx, so the needle can
+// sit one volume size further in; doCheckAndFixVolumeData retries the same way.
+func findNeedleOffset(datFile backend.BackendStorageFile, version needle.Version, offset int64, key types.NeedleId, size types.Size) int64 {
+	for _, at := range []int64{offset, offset + int64(types.MaxPossibleVolumeSize)} {
+		n, _, _, err := needle.ReadNeedleHeader(datFile, version, at)
+		if err == nil && n.Id == key && n.Size == size {
+			return at
+		}
+	}
+	return -1
+}
+
+// readNeedleAppendAtNs reads the append timestamp a version 3 needle carries
+// past its checksum.
+func readNeedleAppendAtNs(datFile backend.BackendStorageFile, offset int64, size types.Size) (uint64, error) {
+	bytes := make([]byte, types.TimestampSize)
+	readCount, err := datFile.ReadAt(bytes, offset+types.NeedleHeaderSize+int64(size)+needle.NeedleChecksumSize)
+	if err == io.EOF && readCount == types.TimestampSize {
+		err = nil
+	}
+	if err != nil {
+		return 0, err
+	}
+	return util.BytesToUint64(bytes), nil
+}
+
 func verifyIndexFileIntegrity(indexFile *os.File) (indexSize int64, err error) {
 	if indexSize, err = util.GetFileSize(indexFile); err == nil {
 		if indexSize%types.NeedleMapEntrySize != 0 {
@@ -293,19 +412,13 @@ func verifyNeedleIntegrity(datFile backend.BackendStorageFile, v needle.Version,
 		return 0, ErrorSizeMismatch
 	}
 	if v == needle.Version3 {
-		bytes := make([]byte, types.TimestampSize)
-		var readCount int
-		readCount, err = datFile.ReadAt(bytes, offset+types.NeedleHeaderSize+int64(size)+needle.NeedleChecksumSize)
-		if err == io.EOF && readCount == types.TimestampSize {
-			err = nil
-		}
+		n.AppendAtNs, err = readNeedleAppendAtNs(datFile, offset, size)
 		if err == io.EOF {
 			return 0, err
 		}
 		if err != nil {
 			return 0, fmt.Errorf("verifyNeedleIntegrity check %s entry offset %d size %d: %v", datFile.Name(), offset, size, err)
 		}
-		n.AppendAtNs = util.BytesToUint64(bytes)
 		fileTailOffset := offset + needle.GetActualSize(size, v)
 		fileSize, _, err := datFile.GetStat()
 		if err != nil {

weed/storage/volume_loading.go

@@ -293,6 +293,7 @@ func (v *Volume) load(alsoLoadIndex bool, createDatIfMissing bool, needleMapKind
 				v.noWriteOrDelete = true
 				glog.V(0).Infof("volumeDataIntegrityChecking failed %v", err)
 			}
+			v.recoverLastModifiedTs(indexFile)
 		}
 
 		// The post-load structural check below uses the in-memory needle map

weed/storage/volume_tier.go

@@ -2,7 +2,6 @@ package storage
 
 import (
 	"fmt"
-	"time"
 
 	"github.com/seaweedfs/seaweedfs/weed/glog"
 	"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
@@ -109,11 +108,8 @@ func (v *Volume) loadRemoteFileLocked() error {
 func (v *Volume) SaveVolumeInfo() error {
 
 	tierFileName := v.FileName(".vif")
-	if v.Ttl != nil {
-		ttlSeconds := v.Ttl.ToSeconds()
-		if ttlSeconds > 0 {
-			v.volumeInfo.ExpireAtSec = uint64(time.Now().Unix()) + ttlSeconds //calculated destroy time from the ec volume was created
-		}
+	if expireAtSec := v.ExpireAtSec(); expireAtSec > 0 {
+		v.volumeInfo.ExpireAtSec = expireAtSec
 	}
 
 	return volume_info.SaveVolumeInfo(tierFileName, v.volumeInfo)

weed/storage/volume_ttl_expiry_test.go

@@ -0,0 +1,249 @@
+package storage
+
+import (
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/storage/needle"
+	"github.com/seaweedfs/seaweedfs/weed/storage/super_block"
+	"github.com/seaweedfs/seaweedfs/weed/storage/types"
+	"github.com/seaweedfs/seaweedfs/weed/util"
+)
+
+// TestVolumeTtlClockSurvivesDeletes reproduces the delete-traffic TTL bug:
+// deletes append a tombstone to the .dat, which moves the file's mtime, and
+// the loader read the TTL clock back from that mtime. A volume taking delete
+// traffic therefore had expired() re-armed for another full TTL on every
+// restart and was never reclaimed. The clock has to come from the newest
+// write instead.
+func TestVolumeTtlClockSurvivesDeletes(t *testing.T) {
+	dir := t.TempDir()
+	ttl, err := needle.ReadTTL("5m")
+	if err != nil {
+		t.Fatalf("read ttl: %v", err)
+	}
+
+	v, err := NewVolume(dir, dir, "", 1, NeedleMapInMemory, &super_block.ReplicaPlacement{}, ttl, 0, needle.GetCurrentVersion(), 0, 0)
+	if err != nil {
+		t.Fatalf("volume creation: %v", err)
+	}
+
+	// Backdate the writes on disk so the last one sits well outside the TTL,
+	// while the tombstones below leave the .dat mtime at now.
+	lastWriteNs := uint64(time.Now().Add(-2 * time.Hour).UnixNano())
+	for i := 1; i <= 3; i++ {
+		n := newRandomNeedle(uint64(i))
+		offset, _, _, err := v.writeNeedle2(n, true, false, false)
+		if err != nil {
+			t.Fatalf("write needle %d: %v", i, err)
+		}
+		backdateAppendAtNs(t, v, int64(offset), n.Size, lastWriteNs)
+	}
+	// More than one tombstone: the scan has to walk back over the whole run of
+	// them to reach a write.
+	for _, id := range []uint64{2, 3} {
+		if _, err := v.doDeleteRequest(newEmptyNeedle(id)); err != nil {
+			t.Fatalf("delete needle %d: %v", id, err)
+		}
+	}
+	contentSize := v.ContentSize()
+	v.Close()
+
+	reloaded, err := NewVolume(dir, dir, "", 1, NeedleMapInMemory, &super_block.ReplicaPlacement{}, ttl, 0, needle.GetCurrentVersion(), 0, 0)
+	if err != nil {
+		t.Fatalf("reload: %v", err)
+	}
+	defer reloaded.Close()
+
+	if got, want := reloaded.lastModifiedTsSeconds, lastWriteNs/uint64(time.Second); got != want {
+		t.Errorf("TTL clock recovered as %d, want the last write at %d", got, want)
+	}
+	if !reloaded.expired(contentSize, 1024*1024) {
+		t.Error("a TTL volume whose last write is 2h old must be expired after a reload")
+	}
+}
+
+// TestVolumeTtlClockKeepsMtimeWithoutRecoverableWrite covers a TTL volume whose
+// needles carry no append timestamp: the loader must stay on the .dat mtime
+// rather than treat the volume as written at the epoch and drop it on sight.
+func TestVolumeTtlClockKeepsMtimeWithoutRecoverableWrite(t *testing.T) {
+	dir := t.TempDir()
+	ttl, err := needle.ReadTTL("5m")
+	if err != nil {
+		t.Fatalf("read ttl: %v", err)
+	}
+
+	v, err := NewVolume(dir, dir, "", 1, NeedleMapInMemory, &super_block.ReplicaPlacement{}, ttl, 0, needle.Version2, 0, 0)
+	if err != nil {
+		t.Fatalf("volume creation: %v", err)
+	}
+	if _, _, _, err := v.writeNeedle2(newRandomNeedle(1), true, false, false); err != nil {
+		t.Fatalf("write needle: %v", err)
+	}
+	contentSize := v.ContentSize()
+	v.Close()
+
+	reloaded, err := NewVolume(dir, dir, "", 1, NeedleMapInMemory, &super_block.ReplicaPlacement{}, ttl, 0, needle.Version2, 0, 0)
+	if err != nil {
+		t.Fatalf("reload: %v", err)
+	}
+	defer reloaded.Close()
+
+	if reloaded.expired(contentSize, 1024*1024) {
+		t.Error("a just-written volume must not be expired after a reload")
+	}
+}
+
+// TestVolumeTtlClockAfterVacuumTakesNewestWrite covers the one layout where a
+// .dat's order does not track its write order: vacuum rewrites it by key, and
+// an overwrite keeps its original, lower key. Reading the position rather than
+// the timestamps would recover the highest-key needle's older write time and
+// expire the volume before the overwrite has lived out its TTL.
+func TestVolumeTtlClockAfterVacuumTakesNewestWrite(t *testing.T) {
+	dir := t.TempDir()
+	ttl, err := needle.ReadTTL("5m")
+	if err != nil {
+		t.Fatalf("read ttl: %v", err)
+	}
+
+	v, err := NewVolume(dir, dir, "", 1, NeedleMapInMemory, &super_block.ReplicaPlacement{}, ttl, 0, needle.GetCurrentVersion(), 0, 0)
+	if err != nil {
+		t.Fatalf("volume creation: %v", err)
+	}
+
+	// Needle 1 is overwritten last but sorts first, so vacuum leaves it at the
+	// head of the .dat with the newest timestamp of the three.
+	oldWriteNs := uint64(time.Now().Add(-2 * time.Hour).UnixNano())
+	newWriteNs := uint64(time.Now().Add(-time.Minute).UnixNano())
+	for _, w := range []struct {
+		id uint64
+		ns uint64
+	}{{2, oldWriteNs}, {3, oldWriteNs}, {1, newWriteNs}} {
+		n := newRandomNeedle(w.id)
+		offset, _, _, err := v.writeNeedle2(n, true, false, false)
+		if err != nil {
+			t.Fatalf("write needle %d: %v", w.id, err)
+		}
+		backdateAppendAtNs(t, v, int64(offset), n.Size, w.ns)
+	}
+	if err := v.CompactByIndex(nil); err != nil {
+		t.Fatalf("compact: %v", err)
+	}
+	if err := v.CommitCompact(); err != nil {
+		t.Fatalf("commit compact: %v", err)
+	}
+	contentSize := v.ContentSize()
+	v.Close()
+
+	reloaded, err := NewVolume(dir, dir, "", 1, NeedleMapInMemory, &super_block.ReplicaPlacement{}, ttl, 0, needle.GetCurrentVersion(), 0, 0)
+	if err != nil {
+		t.Fatalf("reload: %v", err)
+	}
+	defer reloaded.Close()
+
+	if got, want := reloaded.lastModifiedTsSeconds, newWriteNs/uint64(time.Second); got != want {
+		t.Errorf("TTL clock recovered as %d, want the newest write at %d", got, want)
+	}
+	if reloaded.expired(contentSize, 1024*1024) {
+		t.Error("a volume overwritten a minute ago must not be expired after a vacuum and reload")
+	}
+}
+
+// TestVolumeTtlClockDeclinesUnaffordableScan covers the budget the vacuumed
+// path runs under. Reading a subset of a key-ordered volume's writes could
+// recover a timestamp older than the newest write and expire live data, so a
+// scan that does not fit has to leave the clock on the mtime instead.
+func TestVolumeTtlClockDeclinesUnaffordableScan(t *testing.T) {
+	dir := t.TempDir()
+	ttl, err := needle.ReadTTL("5m")
+	if err != nil {
+		t.Fatalf("read ttl: %v", err)
+	}
+
+	v, err := NewVolume(dir, dir, "", 1, NeedleMapInMemory, &super_block.ReplicaPlacement{}, ttl, 0, needle.GetCurrentVersion(), 0, 0)
+	if err != nil {
+		t.Fatalf("volume creation: %v", err)
+	}
+	oldWriteNs := uint64(time.Now().Add(-2 * time.Hour).UnixNano())
+	for i := 1; i <= 3; i++ {
+		n := newRandomNeedle(uint64(i))
+		offset, _, _, err := v.writeNeedle2(n, true, false, false)
+		if err != nil {
+			t.Fatalf("write needle %d: %v", i, err)
+		}
+		backdateAppendAtNs(t, v, int64(offset), n.Size, oldWriteNs)
+	}
+	if err := v.CompactByIndex(nil); err != nil {
+		t.Fatalf("compact: %v", err)
+	}
+	if err := v.CommitCompact(); err != nil {
+		t.Fatalf("commit compact: %v", err)
+	}
+	contentSize := v.ContentSize()
+	v.Close()
+
+	defer func(budget int) { vacuumedLastWriteScanEntries = budget }(vacuumedLastWriteScanEntries)
+	vacuumedLastWriteScanEntries = 2
+
+	reloaded, err := NewVolume(dir, dir, "", 1, NeedleMapInMemory, &super_block.ReplicaPlacement{}, ttl, 0, needle.GetCurrentVersion(), 0, 0)
+	if err != nil {
+		t.Fatalf("reload: %v", err)
+	}
+	defer reloaded.Close()
+
+	if reloaded.lastModifiedTsSeconds == oldWriteNs/uint64(time.Second) {
+		t.Error("a scan that ran out of budget must not report a partial maximum as the last write")
+	}
+	if reloaded.expired(contentSize, 1024*1024) {
+		t.Error("declining the scan must leave the volume on its mtime, not expire it")
+	}
+}
+
+// TestVolumeExpireAtSecCountsFromLastWrite guards the destroy time an EC volume
+// is reclaimed on (erasure_coding.EcVolume.IsTimeToDestroy). It was recomputed
+// as now+TTL on every .vif write, so a read-only mark, a tier upload or an EC
+// encode handed an already expiring volume another full TTL.
+func TestVolumeExpireAtSecCountsFromLastWrite(t *testing.T) {
+	dir := t.TempDir()
+	ttl, err := needle.ReadTTL("5m")
+	if err != nil {
+		t.Fatalf("read ttl: %v", err)
+	}
+
+	v, err := NewVolume(dir, dir, "", 1, NeedleMapInMemory, &super_block.ReplicaPlacement{}, ttl, 0, needle.GetCurrentVersion(), 0, 0)
+	if err != nil {
+		t.Fatalf("volume creation: %v", err)
+	}
+	defer v.Close()
+
+	// A volume with nothing written yet has no last write to count from, and
+	// must not land in 1970 with its data due for destruction on sight.
+	if got := v.GetVolumeInfo().ExpireAtSec; got < uint64(time.Now().Unix()) {
+		t.Errorf("a fresh volume expires at %d, already in the past", got)
+	}
+
+	if _, _, _, err := v.writeNeedle2(newRandomNeedle(1), true, false, false); err != nil {
+		t.Fatalf("write needle: %v", err)
+	}
+	v.lastModifiedTsSeconds = uint64(time.Now().Add(-time.Hour).Unix())
+	want := v.lastModifiedTsSeconds + ttl.ToSeconds()
+
+	for i := 0; i < 2; i++ {
+		if err := v.SaveVolumeInfo(); err != nil {
+			t.Fatalf("save .vif: %v", err)
+		}
+		if got := v.GetVolumeInfo().ExpireAtSec; got != want {
+			t.Fatalf(".vif save %d put ExpireAtSec at %d, want %d counted from the last write", i, got, want)
+		}
+	}
+}
+
+func backdateAppendAtNs(t *testing.T, v *Volume, offset int64, size types.Size, appendAtNs uint64) {
+	t.Helper()
+	stamp := make([]byte, types.TimestampSize)
+	util.Uint64toBytes(stamp, appendAtNs)
+	tsOffset := offset + types.NeedleHeaderSize + int64(size) + needle.NeedleChecksumSize
+	if _, err := v.DataBackend.WriteAt(stamp, tsOffset); err != nil {
+		t.Fatalf("backdate the needle at offset %d: %v", offset, err)
+	}
+}