* volume: never compact a volume tiered to remote storage
A tiered volume's data backend is the remote object, but compaction reads
the local .dat path. When tiering removed the local .dat, every vacuum of a
tiered volume with garbage fails with "no such file or directory", and the
master's sweep and the worker's vacuum task keep selecting it. When the
local .dat was kept, the compaction succeeds against the stale copy and the
commit reloads the volume to read the remote object (old layout) through the
compacted index, so live needles stop reading.
Mirror the Rust volume server (#11545):
- CompactByIndex and CompactByVolumeData refuse a volume with a remote file
before creating .cpd/.cpx ("volume N is tiered to remote storage, cannot
compact").
- CommitCompact refuses under the data-file locks when the volume became
tiered during the copy, and removes .cpd/.cpx.
- Store.CheckCompactVolume reports no garbage for a tiered volume, so the
master's vacuum and volume.vacuum do not select it.
- The worker's vacuum detection skips a volume with a remote copy, as balance
and EC detection already do.
Tests (each fails on master): TestCompactRefusesTieredVolume,
TestCommitCompactRefusesVolumeTieredDuringCopy (on master the commit
succeeds and a surviving needle no longer reads),
TestPickPrimaryReplicaSkipsTieredVolumes.
* volume: skip tiered replicas in vacuum selection, keep their real ratio
Reporting a zero garbage ratio for a tiered volume hid it from
volume.tier.compact, whose preflight reads that ratio while the volume is
still remote, and did not exclude it from a vacuum with a zero threshold.
Report the real ratio again, as the Rust volume server does, and have the
master's vacuum check skip a replica whose volume info names remote
storage, whatever the threshold or request.
When CommitCompact refuses a volume tiered during the copy, also close the
temporary needle map and remove .cpldb: nothing else would, since no later
vacuum selects the volume.
The tiered-compaction test no longer depends on random needle sizes
reaching a 0.3 garbage ratio.
* volume tier: do not attach a copy of a .dat a compaction replaced
The upload copies the .dat, then records the remote file in the .vif and
switches the backend. A compaction commit in between swaps in a compacted
.dat and index; the upload then served the old-layout remote object through
the new index, so live needles stopped reading. CommitCompact's tiered
check cannot see this: the volume is not tiered until the switch.
AttachRemoteDatFile records the remote file and switches the backend under
the data-file locks, and refuses when the backend is no longer the file
that was copied. The upload then removes the unused remote copy.
* volume tier: keep a committed .vif entry when only the dir fsync fails
AttachRemoteDatFile rolled back the in-memory remote entry on any
SaveVolumeInfo error. A NotCrashDurableError comes after the rename, so
the .vif on disk already names the copy; with the entry gone, the upload
handler deleted that object, and a reload would serve from a missing
remote file. Keep the entry and switch to the copy as the .vif says,
and still return the error so the caller keeps the local .dat.
* vacuum worker: filter tiered replicas instead of skipping the volume
A partially tiered volume is the leftover of an interrupted
volume.tier.upload; skipping the whole volume left garbage on its local
replicas unreclaimed. Per-replica filtering matches the master's
batchVacuumVolumeCheck and is safe: a tiered replica refuses both
compaction and commit.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume tier: accept deletes on a volume as soon as it is tiered
volume.tier.upload marks the volume read-only without deletes before the
upload (markVolumeReplicasWritable(..., false, false), i.e. noWriteOrDelete),
and nothing clears that until the volume server restarts, when load() gives
every tiered volume noWriteCanDelete. Every delete in between is refused with
"volume N is read only": the needle stays live in the index, and its bytes in
the remote object, until someone notices.
After the remote file is loaded, the upload handler now gives the volume the
flags a restart would (MarkTieredDeletable: writes refused, deletes accepted),
with two exceptions that keep the current behaviour:
- a replicated volume: the shell copies the uploaded replica's .idx and .vif
to the other replicas after the upload, and a tombstone written before that
copy would reach one replica only;
- a volume in I/O quarantine: that flag is not the upload's to lift.
Tests: TestTierUploadAcceptsDeletesOnASingleCopyVolume fails without the
change ("volume 1 is read only") and checks writes stay refused;
TestTierUploadKeepsRefusingDeletesOnAReplicatedVolume; TestMarkTieredDeletable
(quarantine kept).
* volume tier: check the I/O quarantine under noWriteLock
MarkTieredDeletable read the quarantine state before taking noWriteLock.
CollectHeartbeat could quarantine the volume and set noWriteOrDelete in
between, and the upload then cleared that refusal, so a quarantined volume
accepted deletes until the next heartbeat. Take noWriteLock first: the
heartbeat marks the quarantine before it takes noWriteLock, so either the
check sees it or the heartbeat's refusal lands afterwards and stays.
Also leave alone a volume past IoErrorTolerance that the next heartbeat
will quarantine, as the heartbeat itself does.
* volume: read noWriteOrDelete under noWriteLock in DeleteVolumeNeedle
DeleteVolumeNeedle checked v.noWriteOrDelete without the lock that
MarkVolumeReadonly, CollectHeartbeat and now MarkTieredDeletable hold when
they write it, so a delete racing a tier upload was a data race. Copy the
flag under the read lock; the delete path is otherwise unchanged.
* volume tier: mirror the delete acceptance rules in the Rust volume server
The Rust tier upload already promoted a freshly tiered volume to
no-write-can-delete via refresh_remote_write_mode, and did so for every
volume -- including the replicated and I/O-quarantined ones the Go
MarkTieredDeletable deliberately refuses. A live tier-up now goes
through mark_tiered_deletable (replicated volumes and quarantines keep
their upload-time mark until the next load), while load() keeps
promoting every tiered volume unconditionally, as Go does.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep a tiered volume delete-only when marked writable
MarkVolumeWritable preserved noWriteCanDelete on a remote-tiered volume
only when it was already set. A tiered volume still holding the
upload-time mark (a replicated volume, or one whose tier upload ran
before MarkTieredDeletable existed) had it clear, so marking writable
published the volume as fully writable while its .dat lives remotely --
writes then failed against a backend that cannot append, after the
master was told the volume is writable again.
load() gives every remote-tiered volume noWriteCanDelete; deriving the
same flag here keeps mark-writable consistent with a restart: "writable"
on a remote volume resolves to can-delete.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep a tiered volume delete-only when marked writable (rust)
set_writable preserved the remote delete-only mode only when the flag was
already set, so a tiered volume holding the upload-time read-only mark
(replicated, or quarantined under mark_tiered_deletable) was published
fully writable with its .dat remote. Derive no_write_can_delete from
has_remote_file() instead, matching load() and the Go MarkVolumeWritable.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume tier: serialize the quarantine check with the flag swap (rust)
A heartbeat refusal landing between should_quarantine() and the store
could be clobbered until the next heartbeat re-asserted it — the same
race the Go side closed by checking under noWriteLock. A
compare-exchange gives the same ordering: the swap fails, and the retry
sees the quarantine mark the heartbeat sets first.
* volume tier: MarkTieredDeletable refuses a non-tiered volume
A stray call on a local single-copy volume would pin it
read-only-can-delete. The only caller promotes right after
LoadRemoteFile, but the guard makes the method safe to reuse; the Rust
mark_tiered_deletable gets the same check, though its caller already
only reaches it under has_remote.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: run the durable sync outside the reader-visible lock
Reads of a volume waited on that volume's fsync because the write lock
was held across Sync()/sync_all. A new per-volume lock (Go
dataFileSyncLock, Rust data_file_sync_lock) now serializes every .dat
mutation and lifecycle op across append+publish+sync+rollback, while the
fsync itself runs with dataFileAccessLock/the io write guard dropped —
so readers proceed during a sync, and a failed sync's rollback still
cannot clobber another write or a compaction swap.
Same change in Go (volume_write.go syncWrite/processBatch, SyncToDisk)
and Rust (write_needles_grouped/write_synced_run).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: publish after sync; fix Close/CommitCompact lock order
Address review on #11863:
- Close took dataFileSyncLock before waiting on isCompactionInProgress
while CommitCompact claims the flag then takes the lock — a deadlock.
Close now claims the flag before taking the locks (flag -> sync -> access).
- Go now publishes needle-map entries only after the .dat sync, matching
Rust's ordering: appendWriteRequest/appendDeleteRequest stage the bytes,
publishWriteRequest/publishDeleteRequest index them after Sync. A failed
sync rolls back bytes no reader ever resolved — the streamed-read
offset-invalidation window is gone. processBatch splits runs at repeated
needle ids (Rust parity) so dedup/cookie checks stay sequential; the old
snapshot/restore rollback machinery is removed.
- Rust commit_compact (store-lock caller) takes data_file_sync_lock via
try_lock -> WouldBlock before the write lease; commit_compact_held and
the tier-down _held paths document/require the caller-held sync lock,
and the two vol.instance().write_lock() grpc sites take it first. The
swap can no longer land between a durable write's append and publish.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: restore write watermarks when an inline fsync fails
The syncWrite rollback truncated the .dat but left lastAppendAtNs,
lastWriteAppendAtNs, lastWriteNeedleKey, lastWriteDeleted and
lastModifiedTsSeconds describing the rejected write; recovery and
compaction could read a watermark for a record that no longer exists.
Snapshot and restore them on failure, as processBatchRun already does.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: count a TTL volume's age from its last write, not the .dat mtime
A delete appends a tombstone needle and vacuum rewrites the .dat wholesale,
so the file's mtime moves without any write ever landing. The loader read
lastModifiedTsSeconds back from that mtime, so every restart of a volume
taking delete traffic re-armed expired() for another full TTL: an
overwrite-heavy collection kept growing until it hit the max-volume cap.
Recover the clock from the newest .idx entry that is not a tombstone and
read that needle's append timestamp, falling back to the mtime when no
write is recoverable. Only TTL volumes pay for the scan.
Fixes#11160
* volume: count the .vif destroy time from the last write too
ExpireAtSec is what an EC volume is reclaimed on, and it was recomputed as
now+TTL every time the .vif was written. A read-only mark, a tier upload or
an EC encode therefore handed an already expiring volume another full TTL,
the same way the .dat mtime did.
Derive it from the volume's last write, falling back to now for a volume
that has not taken one yet so a fresh volume is not born expired.
* volume: mirror the last-write TTL clock in the Rust volume server
Same recovery as the Go loader: scan the .idx backwards for the newest
entry that is not a tombstone and take that needle's append timestamp,
leaving the clock on the .dat mtime when no write is recoverable.
* volume: mirror the last-write destroy time in the Rust volume server
Both .vif writers and the EC encode computed ExpireAtSec as now+TTL, the
same way Go did, so the destroy time moved every time the sidecar was
rewritten. Route all three through the volume's last write.
* volume: report the .dat mtime in the Rust heartbeat, like Go does
The Rust server reported its TTL clock as ModifiedAtSecond while Go
reports the .dat mtime. The shell's quiet-period gates (volume.tier.move,
volume.delete_empty) read that field as "last touched", which a delete
has to count towards even though the TTL clock deliberately ignores it --
and with the clock now recovered from the last write, the two drift
further apart.
* volume: take the newest write by timestamp on a vacuumed volume
The reverse .idx scan trusted position, which holds only while the .dat is
append ordered. Vacuum rewrites it in key order, and since an overwrite
keeps its original key, the highest-key survivor is not necessarily the
newest write -- the recovered clock could land up to a TTL early and take
the volume with data still inside its TTL.
A volume that has been vacuumed (CompactionRevision > 0) now takes the
maximum append timestamp over a bounded window of write entries instead.
An append-ordered volume still answers in one read.
* volume: never guess a vacuumed volume's last write, and resolve wrapped offsets
Two holes in the reverse scan, both from review:
A vacuumed volume's writes are ordered by key, so any of them can hold the
newest timestamp. Reading a capped window sampled the highest keys, which
could still miss a recently overwritten low-key needle and expire data
inside its TTL. The scan now covers every write a vacuumed volume indexes,
and a volume too large to scan keeps the .dat mtime rather than report a
partial maximum -- late is recoverable, early is not.
A .dat past MaxPossibleVolumeSize wraps the offsets in its .idx, so reading
a timestamp at the unwrapped offset picks up an unrelated needle. Resolve
the entry against the needle header first and retry one volume size in,
the way doCheckAndFixVolumeData already does.
* volume: drop GitHub issue references from TTL comments
* volume: stop the .vif guard depending on which entry the scan handed over
A volume has both an .idx and a .vif, and loadExistingVolume skipped a .vif
next to an .ecx as EC shard metadata. That was only ever correct because
os.ReadDir sorted .idx ahead of .vif: an interrupted encode, where the .idx is
still there, has to reach validateEcVolume to be reclaimed. Ask for the .idx
instead of trusting the order.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* volume: walk volume directories in batches instead of listing them whole
os.ReadDir builds, and sorts, a slice of every entry before the caller sees
the first one. A disk holding millions of volumes has a .dat, .idx and .vif
per volume, so each startup scan costs hundreds of MB of peak heap that the
runtime is slow to hand back -- and there are several of them before the
first volume loads.
Walk in batches instead, and keep only the entries each scan acts on:
loadAllEcShards now sorts and stats the shard and index files alone rather
than every file on the disk.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* volume: skip the sibling-.dat scan when no EC volume is loaded
pruneIncompleteEcWithSiblingDat only ever prunes EC volumes that are loaded,
but it first walks every disk and keys a map by every .dat on the server. On
a store with no EC volumes at all that is millions of map entries built to
answer no question.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* volume: stop keeping a departure message for every volume
The report state held a VolumeShortInformationMessage per volume copy so a
departure could be named, but almost no volume ever departs. Hold a handle to
the identity instead -- volumes share very few distinct ones -- and build the
message on the way out.
Measured over a populated report state: 195 -> 83 bytes per volume.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* rust volume: stop keeping a whole volume message per volume held
The send loop kept a VolumeInformationMessage for every volume just to notice
mounts and unmounts, and rebuilt the map from scratch on every beat. Keep the
identity a delta names, which is what the Go report state keeps for the same
reason.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* rust volume: keep only the EC files the shard scan acts on
load_all_ec_shards named every file on the disk twice -- once in the dedup set
and once in the sorted vector -- before deciding it only wanted .ec?? and .ecx.
Filter while reading instead. Mirrors the same change in loadAllEcShards.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
* volume: share the strings every .vif repeats
A tiered volume's .vif names its replication and its backend, and every decode
allocates a fresh copy, so a server holding millions of them holds millions of
copies of the same handful of names. Route them through the interning table
the volume info decode already uses. The remote key names one volume and is
left alone.
Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy
LoadRemoteFile now takes dataFileAccessLock (so a live tier upload does
not race the heartbeat's DataBackend read). But load() also runs it, and
CommitCompact calls load() while already holding that lock, so reloading
a remote-tiered volume during a compaction commit re-enters the
non-reentrant lock and deadlocks.
Split the locked bodies out: swapDataBackendLocked and loadRemoteFileLocked
assume the caller holds dataFileAccessLock. load() uses loadRemoteFileLocked;
the public LoadRemoteFile keeps taking the lock for the live tier-upload
handler that does not hold it.
VolumeTierMoveDatFromRemote downloads the .dat, trims the .vif, and swaps
the data backend to the local file, but left hasRemoteFile set. The
volume.tier.download command masks this by unmounting and remounting
right after, which reloads the flag from the trimmed .vif — but in the
window before the remount the in-memory flag is wrong: doDeleteRequest
would skip appending the tombstone to the freshly local .dat, and the
phantom-.dat guard stays disabled.
Give SwapDataBackend a hasRemoteFile argument so the backend swap and the
flag move together under one lock, and route both tier directions through
it: the tier-down download passes false, LoadRemoteFile passes true. The
flag can no longer disagree with the live backend.
volume: keep tier-uploaded volume reporting to master
A live volume.tier.upload removes the local .dat and swaps the data
backend to remote, but v.hasRemoteFile was only ever set when a volume
is loaded from disk, so the running volume kept it false. The phantom
.dat guard then saw fileCount>0, !HasRemoteFile, and a missing .dat, and
stopped reporting the volume to the master. The volume vanished from the
topology even though the upload succeeded and the data was in cloud
storage.
Set hasRemoteFile in LoadRemoteFile, the single point where a volume's
backend becomes remote, so it is true both on disk-scan load and after
an in-process tier upload. Route the backend reassignment through
SwapDataBackend so it happens under dataFileAccessLock, closing the old
backend and never racing the heartbeat's concurrent DataBackend read.
Make the field atomic since the heartbeat now reads it concurrently with
the tier-upload handler that writes it.
The s3 and rclone tiered-storage backends were registered via blank imports
in weed/storage (volume_tier.go and volume_info/volume_info.go). That forced
every library consumer of weed/storage -- weed/shell, and through it external
tools -- to link aws-sdk-go and, under the rclone build tag, the full rclone
backend set and its cloud-storage SDKs, even though those consumers never tier
volumes.
Move the registrations into a new weed/storage/backend/all aggregator and
blank-import it once from weed/command, the binary's composition root. The weed
binary still registers both backends; weed/storage and its library consumers no
longer pull the backend SDKs into their dependency graph.
* Add Rclone storage backend
* Support templating the name of files stored via Rclone
* Enable Rclone accounting
* Remove redundant type conversion
* Provide progress information for Rclone download/upload operations
* Log error when Rclone can't instantiate filesystem
* Remove filename templating functionality for Rclone storage
To (maybe) be later reintroduced as a generic functionality for all
storage backends.
* Remove S3 specific check
* Move Rclone config initialisation to init() method
