mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-19 10:54:08 +01:00
7b4bdb77bbcf03a56d1f885d3fd5e8aa68265e47
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
9cd183b04f |
volume: fsync a needle delete that asks for it (#12736)
* volume: fsync a needle delete that asks for it A delete now takes fsync like a write: the tombstone is synced before the needle map drops the needle, and the .idx row before the delete is acknowledged. A failed .dat sync takes the tombstone back off the .dat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: honor fsync=true on DELETE and forward it to replicas Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): fsync a needle delete that asks for it Mirrors the Go volume server: DELETE with fsync=true flushes the tombstone before the index drops the needle and the .idx before the ack, and a failed .dat flush takes the tombstone back off the .dat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: let a durable delete through on a full volume A queued delete hit the batch worker's volume size check, which a tombstone append is exempt from inline, so fsync=true deletes failed once the volume reached its hard limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: sync a durable delete of an already-deleted needle An earlier delete without fsync may have dropped the needle with its tombstone still unsynced, so the inline path syncs before acking anyway, as the batch worker already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: refuse a batch request that races the worker stopping stopWorker can close the batch channel between startWorker handing it out and the send, which panicked. The request now falls back to the inline path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): sync a durable delete of an already-deleted needle Mirrors the Go volume server: an earlier delete without fsync may have dropped the needle with its tombstone still unsynced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): run a durable delete on a blocking thread Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: keep syncDelete's signature for existing callers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: refuse a write a delete cancelled before it appends A write cancelled while still queued behind the volume locks used to append, publish and then tombstone itself without a sync, after its delete may already have been acknowledged as durable. A crash could keep the row and lose the tombstone. Refuse it before the append. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): refuse a write a delete cancelled before it appends Mirrors the Go change: a write cancelled while still queued behind the volume locks no longer appends, publishes and tombstones itself without a sync after its delete may already be durable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): drop the write guards for a durable delete's flushes The durable delete held the write lease and the io write guard across both fsyncs, so reads of the volume waited on them. Append and publish the tombstone under the guards and flush with only the io read side, as write_synced_run and the Go path already do. do_delete_request keeps its signature. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: stop taking writes when a durable delete fails to index The needle map drops the needle before it appends the .idx row, so after a failed append a durable retry found nothing to delete and acked, and a later write could bury the unindexed tombstone so a reload brought the needle back. Fence the volume the way a failed durable write does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): stop taking writes when a durable delete fails to index Rust counterpart of the Go change: the needle map drops the needle before it appends the .idx row, so a failed append left a durable retry nothing to delete, and a later write could bury the unindexed tombstone. publish_delete_request now fences the volume for a durable delete, as publish_write does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: stop taking writes when any delete fails to index A non-durable delete whose .idx append failed left the volume writable with the needle already dropped from the map, so a later durable retry found nothing to delete and acknowledged without the missing row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): stop taking writes when any delete fails to index Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: take the fsync flag on the existing delete functions DeleteVolumeNeedle, deleteNeedle2 and syncDelete take fsync instead of gaining *WithFsync variants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): take the fsync flag on the existing delete functions delete_volume_needle and delete_needle take fsync instead of gaining *_with_fsync variants; the durable path and the deletable checks fold back into delete_needle, and the cancelled-write check mirrors Go's isCancelledWrite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ac319fe318 |
volume: never compact a volume tiered to remote storage (#11895)
* volume: never compact a volume tiered to remote storage A tiered volume's data backend is the remote object, but compaction reads the local .dat path. When tiering removed the local .dat, every vacuum of a tiered volume with garbage fails with "no such file or directory", and the master's sweep and the worker's vacuum task keep selecting it. When the local .dat was kept, the compaction succeeds against the stale copy and the commit reloads the volume to read the remote object (old layout) through the compacted index, so live needles stop reading. Mirror the Rust volume server (#11545): - CompactByIndex and CompactByVolumeData refuse a volume with a remote file before creating .cpd/.cpx ("volume N is tiered to remote storage, cannot compact"). - CommitCompact refuses under the data-file locks when the volume became tiered during the copy, and removes .cpd/.cpx. - Store.CheckCompactVolume reports no garbage for a tiered volume, so the master's vacuum and volume.vacuum do not select it. - The worker's vacuum detection skips a volume with a remote copy, as balance and EC detection already do. Tests (each fails on master): TestCompactRefusesTieredVolume, TestCommitCompactRefusesVolumeTieredDuringCopy (on master the commit succeeds and a surviving needle no longer reads), TestPickPrimaryReplicaSkipsTieredVolumes. * volume: skip tiered replicas in vacuum selection, keep their real ratio Reporting a zero garbage ratio for a tiered volume hid it from volume.tier.compact, whose preflight reads that ratio while the volume is still remote, and did not exclude it from a vacuum with a zero threshold. Report the real ratio again, as the Rust volume server does, and have the master's vacuum check skip a replica whose volume info names remote storage, whatever the threshold or request. When CommitCompact refuses a volume tiered during the copy, also close the temporary needle map and remove .cpldb: nothing else would, since no later vacuum selects the volume. The tiered-compaction test no longer depends on random needle sizes reaching a 0.3 garbage ratio. * volume tier: do not attach a copy of a .dat a compaction replaced The upload copies the .dat, then records the remote file in the .vif and switches the backend. A compaction commit in between swaps in a compacted .dat and index; the upload then served the old-layout remote object through the new index, so live needles stopped reading. CommitCompact's tiered check cannot see this: the volume is not tiered until the switch. AttachRemoteDatFile records the remote file and switches the backend under the data-file locks, and refuses when the backend is no longer the file that was copied. The upload then removes the unused remote copy. * volume tier: keep a committed .vif entry when only the dir fsync fails AttachRemoteDatFile rolled back the in-memory remote entry on any SaveVolumeInfo error. A NotCrashDurableError comes after the rename, so the .vif on disk already names the copy; with the entry gone, the upload handler deleted that object, and a reload would serve from a missing remote file. Keep the entry and switch to the copy as the .vif says, and still return the error so the caller keeps the local .dat. * vacuum worker: filter tiered replicas instead of skipping the volume A partially tiered volume is the leftover of an interrupted volume.tier.upload; skipping the whole volume left garbage on its local replicas unreclaimed. Per-replica filtering matches the master's batchVacuumVolumeCheck and is safe: a tiered replica refuses both compaction and commit. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |