Commit Graph
4 Commits
Author SHA1 Message Date
Chris LuandDevin cc04c805a4 volume.copy: drop the needle-map index a failed copy leaves behind (#12252)
* volume.copy: drop the needle-map index a failed copy leaves behind

A VolumeCopy that mounts and then fails validation is only unloaded, not
deleted, so a leveldb (.ldb) or sorted-file (.sdx) index built by the
mount outlives the copy. The next copy of the same volume to this server
then trusts the stale index (generateLevelDbFile replays only past its
watermark), which is wrong once the source has compacted in between.
Mirrors the Rust server's remove_partial_files.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: clear a stale index before a copy writes

A crashed copy, or one whose cleanup failed, leaves a stale .ldb/.sdx/.rdb
next to the .idx that this copy's mount would trust (generateLevelDbFile
replays only past the stored watermark, and a same-size .idx keeps every
un-replayed offset). Abort on a removal error rather than risk a mount
reusing a poisoned index. Mirrors the Rust server's pre-write
remove_partial_files call.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: keep a mounted replica when the final response send fails

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: discard a mounted replica the caller abandoned

A send failure after MountVolume means the caller is gone, but the
replica was already announced to the master. Deleting its files under
the live mount breaks the loaded index; leaving it mounted keeps an
orphan the master routes to. Mirror the Rust copy cleanup: track the
mount, and on error delete through the store (unmount + withdraw the
announcement + remove files, keeping remote data), and check the stream
context before mounting.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: serialize copies of a volume on one server

A second VolumeCopy of the same volume could pass the existing-volume
check before the first copy mounts, then clear or overwrite the files
the first copy still owns. Mirror ecDecodesInFlight: the claim covers
the whole copy, and a duplicate fails fast so the caller can retry.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: remove only files the copy owns, and every stale index

Three cleanup gaps around which files a copy may remove:

- The failure cleanup ran on any error, so an abort before the first
  write (a failed stale-index clear or a failed .note write) could
  delete a previous replica's files the copy had not touched. File
  removal is now gated on the copy having started writing.
- A mount that finds an .idx beside the .dat builds the needle-map
  index under the data base, so leftovers there poison the next copy
  the same way the idx-dir ones do. Clear and clean up .idx/.ldb/
  .sdx/.rdb under both bases when the dirs differ.
- A regular volume and an EC volume for the same id share <base>.vif.
  Keep it while the EC index is on the disk, as the destroy path does.
- When the discard of a mounted copy is refused (e.g. it is compacting)
  the volume stays registered; unlinking its files would strand a live
  replica.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: do not unlink a previous replica on an early abort

An abort before the first write now leaves the destination files alone,
but three removal paths still leaked past that gate: a partially-written
.note, and the post-copy defer's .idx/.vif at the data base. Remove the
.note unconditionally (it only ever marks an in-flight copy) and gate the
data-file removals in the outer defer on copyOwnsFiles too.

* volume.copy: refuse a mount or configure while a copy is in flight

A copy claims the volume id, then waits on the source before it clears or
overwrites the destination files. A mount or configure landing in that
window loads the replica whose index files the copy is about to remove.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: abort when a replica registers during the source RPCs

The existing-volume snapshot runs before the copy calls the source, so a
mount that lands in between escapes it. Recheck registration just before
the destructive clear, which can share the index directory with the
mounted replica's bases.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: keep files whenever a replica of the volume stays registered

The cleanup already kept files after a refused delete; a replica a racing
mount registered during the copy needs the same protection, since its
index directory can coincide with the bases being cleaned.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: skip the index cleanup when the copy never picked bases

An early failure leaves the base names empty, so the removals would
otherwise run against ".ldb"/".sdx"/".rdb" in the process directory.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: serialize mount and configure registration under a per-volume guard

The in-flight flag check in VolumeMount and VolumeConfigure was not
atomic with volume registration, and the copy's registration recheck was
not atomic with its stale-index clearing. A mount that passed the check
just before the copy claimed the volume could register after the recheck
and then lose its index files to the clearing.

Replace the bare flag with a per-volume guard: the copy claims it for
the copy's life, mounts and configures hold it across check plus
registration, and the copy holds it across the recheck plus clearing.
Whichever side takes the guard first wins, so a mount either registers
before the recheck (and aborts the copy) or sees the claim and refuses.
Guards persist in the map so a mutex handed to a waiter is never
replaced mid-wait.

* volume.copy: refuse allocation and staged adoption under the copy guard

AllocateVolume and adoptStagedVolume still registered a volume while a
copy of it held the destination: NewVolume loads the index before
registration, and the copy then truncates or replaces files the live
replica already opened, leaving it mounted on a stale index.

Both paths now take the per-volume guard through their registration, so
they either register before the copy's recheck (and make it abort) or
see the claim and refuse with Unavailable.

* volume.copy: serialize reload loads with the per-volume guard

A SIGHUP reload ran loadExistingVolume outside the copy's guard: its
.note branch could unlink a copy's files mid-write, and its
NewVolume+SetVolume could register a replica between the copy's recheck
and its file clearing, leaving a live volume on a stale index.

loadExistingVolume now consults a per-location hook the server installs:
a vid with a copy in flight (or whose guard the copy currently holds) is
skipped for the next pass; otherwise the returned release keeps the
guard held through registration, so the copy's recheck either sees the
volume and aborts or clears first.

* volume.copy: skip compaction-state reconcile while a copy owns the volume

* volume.copy: let a guarded operation's own mount past the load guard

VolumeMount, VolumeConfigure, adoptStagedVolume hold the volume guard
locked, and VolumeCopy holds copying=true, while calling store.MountVolume.
The mount reaches loadExistingVolume where the new load guard saw the
caller's own hold (TryLock fails, or copying set) and skipped the disk:
every guarded mount failed 'volume not found on disk' — a completed copy
deleted its copied files, configure left the volume unmounted, staged
adoption failed after renaming.

Plumb ownLoad through MountVolume -> LoadVolume -> loadExistingVolume via
the new MountVolumeClaimed entry point so the owner's own load proceeds
while unrelated loads are still skipped.

* volume.copy: fix own-load test call, assert unguarded mounts still skip

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: refuse unmount and delete while a copy owns the volume

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-30 09:20:55 -07:00
Chris LuandDevin 8ad2f29e3e shell: let volume.deleteEmpty drop volumes with no live needles (#11437)
* shell: let volume.deleteEmpty drop volumes with no live needles

The candidate check only accepted a .dat at superblock size, so a volume
whose every needle was deleted still had to be vacuumed first — minutes
of compaction to rewrite bytes that were all garbage anyway. FileCount
counts every indexed entry and DeleteCount every entry made garbage by
overwrite or delete, so FileCount <= DeleteCount means nothing live
remains and the volume can be unlinked directly. The quietFor guard is
unchanged.

* volume server: add only_garbage VolumeDelete guard

VolumeDelete(only_empty) refuses every volume that ever held data, so a
volume whose needles are all deleted could only be removed after a
vacuum rewrote it. The new only_garbage flag deletes only when the byte
counters show nothing live: DeletedSize covering all of ContentSize, the
same all-garbage state vacuum measures. Byte counters are used because
the file/delete counts drift on index reload.

* rust volume: mirror only_garbage VolumeDelete guard

Same check as the Go server: a volume deletes under only_garbage when
its deleted bytes cover all content bytes. The grpc handler rejects
before the store drops the volume from its map, since destroy errors
after removal would still unmount it.

* volume delete: let either enabled check pass, keep onlyEmpty on the wire

An upgraded shell sending only_garbage to a pre-upgrade server would be
read as an unconditional delete (field ignored, only_empty false). The
request now keeps only_empty set so old servers check emptiness and
refuse, while new servers delete when either check passes.

* volume.deleteEmpty: skip remote-backed and protected read-only volumes

A remote-tiered replica shares its cloud object with the other replicas,
so keepRemoteData=false on one delete removes data they still reference.
Protected read-only volumes are quarantined or under maintenance, which
is exactly when a replica should not be dropped.

* volume delete: validate guarded copies across disks before deleting

* volume delete: hold copy locks across guarded validate-and-delete

CheckVolumeDeletable released each copy's locks before Destroy ran, so a
write landing on a later copy between the two passes refused its destroy
after earlier copies were already removed. Pin every copy's
dataFileAccessLock (and its location's volumesLock) across validation and
removal so a refused delete leaves all copies intact.

* volume delete: send deleted-volume notices after releasing locks

A blocking send on a full DeletedVolumesChan under volumesLock can stall
the heartbeat loop that drains it while it waits on the same locks.
Collect the notices under the lock span and send after release.

* pb: restore generated-file cosmetics to match the repo's protoc version

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 15:53:39 +08:00
9575032b4c volume: forward fsync=true to replicas in ReplicatedWrite (#10805)
* volume: forward fsync=true to replicas in ReplicatedWrite

When a write request carries fsync=true, only the primary volume server
flushed to disk: the replica fan-out URL in ReplicatedWrite only carried
type/ttl/ts/cm, so replicas always wrote without fsync even when the
client explicitly requested a durable write.

Forward the fsync request parameter to the replica volume servers so a
durable write means every replica has flushed to disk, not just the
primary. Replicas without fsync are untouched (zero behavior change).

* storage: flush a durable write inline while stopping

The fsync flag on the write path really selects the async batch worker,
and it was switched off once the store is stopping. So a fsync=true write
landing during the pre-stop drain got acked without ever being flushed -
and now that ReplicatedWrite forwards fsync, that covers replicas too.

Flush it inline instead of queueing it. The drain keeps accepting writes,
which is the whole point of preStopSeconds, and the ack still means the
.dat is on disk. If the fsync fails, the append comes back off the .dat
and the needle map goes back to what it pointed at before, so nothing
resolves to an offset past the truncated end.

* storage: make the store's stopping flag atomic

SetStopping runs on the signal handler goroutine while the write and
vacuum paths read the flag, so every read of it was racy. Nothing about
the shutdown ordering changes; only the flag itself is now safe to read.

* topology: check the errors the replication test was dropping

The mock replica ignored its response write and the mock master ignored
whatever Serve returned, so a broken mock would have shown up as a
confusing timeout rather than a failure. Also drops the explicit listener
close: grpc.Server.Stop already closes the listener it was given.

---------

Co-authored-by: hzsunchao <hzsunchao@corp.netease.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-08-18 17:17:50 -07:00
Chris Lu 339a597e7e fix(vacuum): crash-safe compaction commit with a durable .cpc marker, fsync-before-rename, and a reload fence (#9944)
* storage: make vacuum/compaction commit crash-safe with a durable .cpc marker

A crash mid-compaction-commit could lose or corrupt volume data. The
two-rename commit (.cpd->.dat, .cpx->.idx) was not atomic, fsync results
were discarded before renaming over a healthy .dat, a stale .ldb could
poison the needle map, and a duplicate/late commit could delete the live
.dat/.idx outright.

Introduce a durable .cpc commit marker so the swap is atomic across a
crash:

- CommitCompact writes and fsyncs the .cpc marker after makeupDiff
  fsyncs the .cpd/.cpx, then runs applyCompactSwap: an existence-guarded
  rename of .cpd->.dat and .cpx->.idx, a directory fsync, removal of the
  stale .ldb/.rdb, and finally removal of the marker.
- reconcileCompactState recovers an interrupted commit on load: roll
  forward (finish the renames) when the marker is present, roll back
  (delete the orphan .cpd/.cpx) when it is absent. It runs from a
  directory pre-pass keyed on .cpd/.cpc existence, since the per-volume
  loader is keyed on .idx/.vif and misses the marker-only and
  already-renamed-.idx states.
- applyCompactSwap verifies BOTH .cpd and .cpx exist before touching the
  live files, so a stale-state commit (including the Windows
  RemoveAll-then-rename path) errors without deleting anything.
- Error-check the fsyncs that gate the swap: the .cpd close-fsync and
  .cpx fsync in copyDataBasedOnIndexFile, the makeupDiff .idx fsync, and
  MemDb.SaveToIdx.
- generateLevelDbFile rebuilds from offset 0 when the stored watermark
  sits past the end of the .idx, instead of replaying zero entries and
  poisoning the needle map.
- removeVolumeFiles and cleanupCompact sweep the .cpc marker; cleanup
  refuses to unlink the temp files while a marker is present.

Mirror the commit-marker, fsync-before-rename, guard, and
load/reconcile logic in the Rust volume server.

* storage: don't reconcile an already-loaded volume's compaction state on reload

reconcileCompactStates runs in loadExistingVolumes, which is re-invoked at
runtime on SIGHUP (Store.LoadNewVolumes). For a volume that is already loaded
and mid-vacuum, its .cpd/.cpx are live temp files, not crash leftovers --
rolling them back would clobber the in-flight compaction (and remove a live
.ldb out from under an open handle). Skip any vid already present in the
volume map; genuine startup recovery runs before any volume is loaded, so the
map is empty then. Mirrored in the Rust volume server.

Also drop the .note keepVif change that crept into this branch; it belongs to
the replica-copy/verify workstream and is restored to master's behavior here
so the two changes don't collide.

* storage: roll a compaction commit forward per-file, not all-or-nothing

A crash after the .cpd->.dat rename but before .cpx->.idx leaves .cpd gone,
.cpx and .cpc present, and a stale .idx. The roll-forward required BOTH temp
files, so it skipped the swap and cleared the marker, pairing the fresh .dat
with the stale .idx (index corruption). Finish whichever temp file remains:
extract finishCompactSwap to rename .cpd->.dat and/or .cpx->.idx independently;
applyCompactSwap keeps the both-present guard for the normal commit. Existence
in the Rust mirror is checked robustly so a transient error never skips the swap.

* seaweed-volume: propagate directory fsync failures on the compaction commit path

fsync_dir dropped every sync_all error, so the commit could proceed with an
undurable marker or rename and a later restart could recover the wrong
generation. Return the error and check it at the commit call sites (marker write
and the swap), matching the Go fsyncDir which already propagates. Directory
fsync stays a no-op on Windows, where it is unsupported.

* storage: overflow-safe stale-watermark check when rebuilding the leveldb index

watermark*NeedleMapEntrySize can overflow uint64 for a corrupted watermark and
wrap below the file size, defeating the stale-.ldb guard. Compare in entries
(watermark > size/NeedleMapEntrySize) instead, which is equivalent and cannot
overflow. LevelDb-backed needle map is Go-only; no Rust mirror.

* storage: propagate idxFile.Close error when writing the compacted index

SaveToIdx writes the .cpx that is renamed to .idx at commit; a discarded Close
error (buffered data not flushed) could leave a partially-written index after a
crash. Surface it in the same durability gate as the fsync.
2026-06-13 20:06:24 -07:00