* volume: count a TTL volume's age from its last write, not the .dat mtime
A delete appends a tombstone needle and vacuum rewrites the .dat wholesale,
so the file's mtime moves without any write ever landing. The loader read
lastModifiedTsSeconds back from that mtime, so every restart of a volume
taking delete traffic re-armed expired() for another full TTL: an
overwrite-heavy collection kept growing until it hit the max-volume cap.
Recover the clock from the newest .idx entry that is not a tombstone and
read that needle's append timestamp, falling back to the mtime when no
write is recoverable. Only TTL volumes pay for the scan.
Fixes#11160
* volume: count the .vif destroy time from the last write too
ExpireAtSec is what an EC volume is reclaimed on, and it was recomputed as
now+TTL every time the .vif was written. A read-only mark, a tier upload or
an EC encode therefore handed an already expiring volume another full TTL,
the same way the .dat mtime did.
Derive it from the volume's last write, falling back to now for a volume
that has not taken one yet so a fresh volume is not born expired.
* volume: mirror the last-write TTL clock in the Rust volume server
Same recovery as the Go loader: scan the .idx backwards for the newest
entry that is not a tombstone and take that needle's append timestamp,
leaving the clock on the .dat mtime when no write is recoverable.
* volume: mirror the last-write destroy time in the Rust volume server
Both .vif writers and the EC encode computed ExpireAtSec as now+TTL, the
same way Go did, so the destroy time moved every time the sidecar was
rewritten. Route all three through the volume's last write.
* volume: report the .dat mtime in the Rust heartbeat, like Go does
The Rust server reported its TTL clock as ModifiedAtSecond while Go
reports the .dat mtime. The shell's quiet-period gates (volume.tier.move,
volume.delete_empty) read that field as "last touched", which a delete
has to count towards even though the TTL clock deliberately ignores it --
and with the clock now recovered from the last write, the two drift
further apart.
* volume: take the newest write by timestamp on a vacuumed volume
The reverse .idx scan trusted position, which holds only while the .dat is
append ordered. Vacuum rewrites it in key order, and since an overwrite
keeps its original key, the highest-key survivor is not necessarily the
newest write -- the recovered clock could land up to a TTL early and take
the volume with data still inside its TTL.
A volume that has been vacuumed (CompactionRevision > 0) now takes the
maximum append timestamp over a bounded window of write entries instead.
An append-ordered volume still answers in one read.
* volume: never guess a vacuumed volume's last write, and resolve wrapped offsets
Two holes in the reverse scan, both from review:
A vacuumed volume's writes are ordered by key, so any of them can hold the
newest timestamp. Reading a capped window sampled the highest keys, which
could still miss a recently overwritten low-key needle and expire data
inside its TTL. The scan now covers every write a vacuumed volume indexes,
and a volume too large to scan keeps the .dat mtime rather than report a
partial maximum -- late is recoverable, early is not.
A .dat past MaxPossibleVolumeSize wraps the offsets in its .idx, so reading
a timestamp at the unwrapped offset picks up an unrelated needle. Resolve
the entry against the needle header first and retry one volume size in,
the way doCheckAndFixVolumeData already does.
* volume: drop GitHub issue references from TTL comments
* volume: stop the .vif guard depending on which entry the scan handed over
A volume has both an .idx and a .vif, and loadExistingVolume skipped a .vif
next to an .ecx as EC shard metadata. That was only ever correct because
os.ReadDir sorted .idx ahead of .vif: an interrupted encode, where the .idx is
still there, has to reach validateEcVolume to be reclaimed. Ask for the .idx
instead of trusting the order.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* volume: walk volume directories in batches instead of listing them whole
os.ReadDir builds, and sorts, a slice of every entry before the caller sees
the first one. A disk holding millions of volumes has a .dat, .idx and .vif
per volume, so each startup scan costs hundreds of MB of peak heap that the
runtime is slow to hand back -- and there are several of them before the
first volume loads.
Walk in batches instead, and keep only the entries each scan acts on:
loadAllEcShards now sorts and stats the shard and index files alone rather
than every file on the disk.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* volume: skip the sibling-.dat scan when no EC volume is loaded
pruneIncompleteEcWithSiblingDat only ever prunes EC volumes that are loaded,
but it first walks every disk and keys a map by every .dat on the server. On
a store with no EC volumes at all that is millions of map entries built to
answer no question.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* volume: stop keeping a departure message for every volume
The report state held a VolumeShortInformationMessage per volume copy so a
departure could be named, but almost no volume ever departs. Hold a handle to
the identity instead -- volumes share very few distinct ones -- and build the
message on the way out.
Measured over a populated report state: 195 -> 83 bytes per volume.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* rust volume: stop keeping a whole volume message per volume held
The send loop kept a VolumeInformationMessage for every volume just to notice
mounts and unmounts, and rebuilt the map from scratch on every beat. Keep the
identity a delta names, which is what the Go report state keeps for the same
reason.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* rust volume: keep only the EC files the shard scan acts on
load_all_ec_shards named every file on the disk twice -- once in the dedup set
and once in the sorted vector -- before deciding it only wanted .ec?? and .ecx.
Filter while reading instead. Mirrors the same change in loadAllEcShards.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* volume: share the strings every .vif repeats
A tiered volume's .vif names its replication and its backend, and every decode
allocates a fresh copy, so a server holding millions of them holds millions of
copies of the same handful of names. Route them through the interning table
the volume info decode already uses. The remote key names one volume and is
left alone.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
LoadRemoteFile now takes dataFileAccessLock (so a live tier upload does
not race the heartbeat's DataBackend read). But load() also runs it, and
CommitCompact calls load() while already holding that lock, so reloading
a remote-tiered volume during a compaction commit re-enters the
non-reentrant lock and deadlocks.
Split the locked bodies out: swapDataBackendLocked and loadRemoteFileLocked
assume the caller holds dataFileAccessLock. load() uses loadRemoteFileLocked;
the public LoadRemoteFile keeps taking the lock for the live tier-upload
handler that does not hold it.
VolumeTierMoveDatFromRemote downloads the .dat, trims the .vif, and swaps
the data backend to the local file, but left hasRemoteFile set. The
volume.tier.download command masks this by unmounting and remounting
right after, which reloads the flag from the trimmed .vif — but in the
window before the remount the in-memory flag is wrong: doDeleteRequest
would skip appending the tombstone to the freshly local .dat, and the
phantom-.dat guard stays disabled.
Give SwapDataBackend a hasRemoteFile argument so the backend swap and the
flag move together under one lock, and route both tier directions through
it: the tier-down download passes false, LoadRemoteFile passes true. The
flag can no longer disagree with the live backend.
volume: keep tier-uploaded volume reporting to master
A live volume.tier.upload removes the local .dat and swaps the data
backend to remote, but v.hasRemoteFile was only ever set when a volume
is loaded from disk, so the running volume kept it false. The phantom
.dat guard then saw fileCount>0, !HasRemoteFile, and a missing .dat, and
stopped reporting the volume to the master. The volume vanished from the
topology even though the upload succeeded and the data was in cloud
storage.
Set hasRemoteFile in LoadRemoteFile, the single point where a volume's
backend becomes remote, so it is true both on disk-scan load and after
an in-process tier upload. Route the backend reassignment through
SwapDataBackend so it happens under dataFileAccessLock, closing the old
backend and never racing the heartbeat's concurrent DataBackend read.
Make the field atomic since the heartbeat now reads it concurrently with
the tier-upload handler that writes it.
The s3 and rclone tiered-storage backends were registered via blank imports
in weed/storage (volume_tier.go and volume_info/volume_info.go). That forced
every library consumer of weed/storage -- weed/shell, and through it external
tools -- to link aws-sdk-go and, under the rclone build tag, the full rclone
backend set and its cloud-storage SDKs, even though those consumers never tier
volumes.
Move the registrations into a new weed/storage/backend/all aggregator and
blank-import it once from weed/command, the binary's composition root. The weed
binary still registers both backends; weed/storage and its library consumers no
longer pull the backend SDKs into their dependency graph.
* Add Rclone storage backend
* Support templating the name of files stored via Rclone
* Enable Rclone accounting
* Remove redundant type conversion
* Provide progress information for Rclone download/upload operations
* Log error when Rclone can't instantiate filesystem
* Remove filename templating functionality for Rclone storage
To (maybe) be later reintroduced as a generic functionality for all
storage backends.
* Remove S3 specific check
* Move Rclone config initialisation to init() method
Chris Lu