3 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandDevin	8ad2f29e3e	shell: let volume.deleteEmpty drop volumes with no live needles (#11437) ... * shell: let volume.deleteEmpty drop volumes with no live needles The candidate check only accepted a .dat at superblock size, so a volume whose every needle was deleted still had to be vacuumed first — minutes of compaction to rewrite bytes that were all garbage anyway. FileCount counts every indexed entry and DeleteCount every entry made garbage by overwrite or delete, so FileCount <= DeleteCount means nothing live remains and the volume can be unlinked directly. The quietFor guard is unchanged. * volume server: add only_garbage VolumeDelete guard VolumeDelete(only_empty) refuses every volume that ever held data, so a volume whose needles are all deleted could only be removed after a vacuum rewrote it. The new only_garbage flag deletes only when the byte counters show nothing live: DeletedSize covering all of ContentSize, the same all-garbage state vacuum measures. Byte counters are used because the file/delete counts drift on index reload. * rust volume: mirror only_garbage VolumeDelete guard Same check as the Go server: a volume deletes under only_garbage when its deleted bytes cover all content bytes. The grpc handler rejects before the store drops the volume from its map, since destroy errors after removal would still unmount it. * volume delete: let either enabled check pass, keep onlyEmpty on the wire An upgraded shell sending only_garbage to a pre-upgrade server would be read as an unconditional delete (field ignored, only_empty false). The request now keeps only_empty set so old servers check emptiness and refuse, while new servers delete when either check passes. * volume.deleteEmpty: skip remote-backed and protected read-only volumes A remote-tiered replica shares its cloud object with the other replicas, so keepRemoteData=false on one delete removes data they still reference. Protected read-only volumes are quarantined or under maintenance, which is exactly when a replica should not be dropped. * volume delete: validate guarded copies across disks before deleting * volume delete: hold copy locks across guarded validate-and-delete CheckVolumeDeletable released each copy's locks before Destroy ran, so a write landing on a later copy between the two passes refused its destroy after earlier copies were already removed. Pin every copy's dataFileAccessLock (and its location's volumesLock) across validation and removal so a refused delete leaves all copies intact. * volume delete: send deleted-volume notices after releasing locks A blocking send on a full DeletedVolumesChan under volumesLock can stall the heartbeat loop that drains it while it waits on the same locks. Collect the notices under the lock span and send after release. * pb: restore generated-file cosmetics to match the repo's protoc version Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-26 15:53:39 +08:00
3 people	9575032b4c	volume: forward fsync=true to replicas in ReplicatedWrite (#10805) ... * volume: forward fsync=true to replicas in ReplicatedWrite When a write request carries fsync=true, only the primary volume server flushed to disk: the replica fan-out URL in ReplicatedWrite only carried type/ttl/ts/cm, so replicas always wrote without fsync even when the client explicitly requested a durable write. Forward the fsync request parameter to the replica volume servers so a durable write means every replica has flushed to disk, not just the primary. Replicas without fsync are untouched (zero behavior change). * storage: flush a durable write inline while stopping The fsync flag on the write path really selects the async batch worker, and it was switched off once the store is stopping. So a fsync=true write landing during the pre-stop drain got acked without ever being flushed - and now that ReplicatedWrite forwards fsync, that covers replicas too. Flush it inline instead of queueing it. The drain keeps accepting writes, which is the whole point of preStopSeconds, and the ack still means the .dat is on disk. If the fsync fails, the append comes back off the .dat and the needle map goes back to what it pointed at before, so nothing resolves to an offset past the truncated end. * storage: make the store's stopping flag atomic SetStopping runs on the signal handler goroutine while the write and vacuum paths read the flag, so every read of it was racy. Nothing about the shutdown ordering changes; only the flag itself is now safe to read. * topology: check the errors the replication test was dropping The mock replica ignored its response write and the mock master ignored whatever Serve returned, so a broken mock would have shown up as a confusing timeout rather than a failure. Also drops the explicit listener close: grpc.Server.Stop already closes the listener it was given. --------- Co-authored-by: hzsunchao <hzsunchao@corp.netease.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-18 17:17:50 -07:00
Chris Lu	339a597e7e	fix(vacuum): crash-safe compaction commit with a durable .cpc marker, fsync-before-rename, and a reload fence (#9944) ... * storage: make vacuum/compaction commit crash-safe with a durable .cpc marker A crash mid-compaction-commit could lose or corrupt volume data. The two-rename commit (.cpd->.dat, .cpx->.idx) was not atomic, fsync results were discarded before renaming over a healthy .dat, a stale .ldb could poison the needle map, and a duplicate/late commit could delete the live .dat/.idx outright. Introduce a durable .cpc commit marker so the swap is atomic across a crash: - CommitCompact writes and fsyncs the .cpc marker after makeupDiff fsyncs the .cpd/.cpx, then runs applyCompactSwap: an existence-guarded rename of .cpd->.dat and .cpx->.idx, a directory fsync, removal of the stale .ldb/.rdb, and finally removal of the marker. - reconcileCompactState recovers an interrupted commit on load: roll forward (finish the renames) when the marker is present, roll back (delete the orphan .cpd/.cpx) when it is absent. It runs from a directory pre-pass keyed on .cpd/.cpc existence, since the per-volume loader is keyed on .idx/.vif and misses the marker-only and already-renamed-.idx states. - applyCompactSwap verifies BOTH .cpd and .cpx exist before touching the live files, so a stale-state commit (including the Windows RemoveAll-then-rename path) errors without deleting anything. - Error-check the fsyncs that gate the swap: the .cpd close-fsync and .cpx fsync in copyDataBasedOnIndexFile, the makeupDiff .idx fsync, and MemDb.SaveToIdx. - generateLevelDbFile rebuilds from offset 0 when the stored watermark sits past the end of the .idx, instead of replaying zero entries and poisoning the needle map. - removeVolumeFiles and cleanupCompact sweep the .cpc marker; cleanup refuses to unlink the temp files while a marker is present. Mirror the commit-marker, fsync-before-rename, guard, and load/reconcile logic in the Rust volume server. * storage: don't reconcile an already-loaded volume's compaction state on reload reconcileCompactStates runs in loadExistingVolumes, which is re-invoked at runtime on SIGHUP (Store.LoadNewVolumes). For a volume that is already loaded and mid-vacuum, its .cpd/.cpx are live temp files, not crash leftovers -- rolling them back would clobber the in-flight compaction (and remove a live .ldb out from under an open handle). Skip any vid already present in the volume map; genuine startup recovery runs before any volume is loaded, so the map is empty then. Mirrored in the Rust volume server. Also drop the .note keepVif change that crept into this branch; it belongs to the replica-copy/verify workstream and is restored to master's behavior here so the two changes don't collide. * storage: roll a compaction commit forward per-file, not all-or-nothing A crash after the .cpd->.dat rename but before .cpx->.idx leaves .cpd gone, .cpx and .cpc present, and a stale .idx. The roll-forward required BOTH temp files, so it skipped the swap and cleared the marker, pairing the fresh .dat with the stale .idx (index corruption). Finish whichever temp file remains: extract finishCompactSwap to rename .cpd->.dat and/or .cpx->.idx independently; applyCompactSwap keeps the both-present guard for the normal commit. Existence in the Rust mirror is checked robustly so a transient error never skips the swap. * seaweed-volume: propagate directory fsync failures on the compaction commit path fsync_dir dropped every sync_all error, so the commit could proceed with an undurable marker or rename and a later restart could recover the wrong generation. Return the error and check it at the commit call sites (marker write and the swap), matching the Go fsyncDir which already propagates. Directory fsync stays a no-op on Windows, where it is unsupported. * storage: overflow-safe stale-watermark check when rebuilding the leveldb index watermark*NeedleMapEntrySize can overflow uint64 for a corrupted watermark and wrap below the file size, defeating the stale-.ldb guard. Compare in entries (watermark > size/NeedleMapEntrySize) instead, which is equivalent and cannot overflow. LevelDb-backed needle map is Go-only; no Rust mirror. * storage: propagate idxFile.Close error when writing the compacted index SaveToIdx writes the .cpx that is renamed to .idx at commit; a discarded Close error (buffered data not flushed) could leave a partially-written index after a crash. Surface it in the same durability gate as the fsync.	2026-06-13 20:06:24 -07:00