Files
seaweedfs/weed/storage/volume_pending_write_test.go
T
Chris LuandClaude Opus 5.5 9cd183b04f volume: fsync a needle delete that asks for it (#12736)
* volume: fsync a needle delete that asks for it

A delete now takes fsync like a write: the tombstone is synced before the
needle map drops the needle, and the .idx row before the delete is
acknowledged. A failed .dat sync takes the tombstone back off the .dat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: honor fsync=true on DELETE and forward it to replicas

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): fsync a needle delete that asks for it

Mirrors the Go volume server: DELETE with fsync=true flushes the tombstone
before the index drops the needle and the .idx before the ack, and a failed
.dat flush takes the tombstone back off the .dat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: let a durable delete through on a full volume

A queued delete hit the batch worker's volume size check, which a
tombstone append is exempt from inline, so fsync=true deletes failed once
the volume reached its hard limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: sync a durable delete of an already-deleted needle

An earlier delete without fsync may have dropped the needle with its
tombstone still unsynced, so the inline path syncs before acking anyway,
as the batch worker already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: refuse a batch request that races the worker stopping

stopWorker can close the batch channel between startWorker handing it out
and the send, which panicked. The request now falls back to the inline path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): sync a durable delete of an already-deleted needle

Mirrors the Go volume server: an earlier delete without fsync may have
dropped the needle with its tombstone still unsynced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): run a durable delete on a blocking thread

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep syncDelete's signature for existing callers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: refuse a write a delete cancelled before it appends

A write cancelled while still queued behind the volume locks used to
append, publish and then tombstone itself without a sync, after its
delete may already have been acknowledged as durable. A crash could
keep the row and lose the tombstone. Refuse it before the append.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): refuse a write a delete cancelled before it appends

Mirrors the Go change: a write cancelled while still queued behind the
volume locks no longer appends, publishes and tombstones itself without
a sync after its delete may already be durable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): drop the write guards for a durable delete's flushes

The durable delete held the write lease and the io write guard across
both fsyncs, so reads of the volume waited on them. Append and publish
the tombstone under the guards and flush with only the io read side, as
write_synced_run and the Go path already do. do_delete_request keeps its
signature.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stop taking writes when a durable delete fails to index

The needle map drops the needle before it appends the .idx row, so after
a failed append a durable retry found nothing to delete and acked, and a
later write could bury the unindexed tombstone so a reload brought the
needle back. Fence the volume the way a failed durable write does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): stop taking writes when a durable delete fails to index

Rust counterpart of the Go change: the needle map drops the needle
before it appends the .idx row, so a failed append left a durable retry
nothing to delete, and a later write could bury the unindexed tombstone.
publish_delete_request now fences the volume for a durable delete, as
publish_write does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stop taking writes when any delete fails to index

A non-durable delete whose .idx append failed left the volume writable with
the needle already dropped from the map, so a later durable retry found
nothing to delete and acknowledged without the missing row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): stop taking writes when any delete fails to index

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the fsync flag on the existing delete functions

DeleteVolumeNeedle, deleteNeedle2 and syncDelete take fsync instead of
gaining *WithFsync variants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): take the fsync flag on the existing delete functions

delete_volume_needle and delete_needle take fsync instead of gaining
*_with_fsync variants; the durable path and the deletable checks fold
back into delete_needle, and the cancelled-write check mirrors Go's
isCancelledWrite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:18:29 -08:00

261 lines
8.1 KiB
Go

package storage
import (
"fmt"
"sync/atomic"
"testing"
"time"
"github.com/seaweedfs/seaweedfs/weed/storage/backend"
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
"github.com/seaweedfs/seaweedfs/weed/storage/super_block"
. "github.com/seaweedfs/seaweedfs/weed/storage/types"
"github.com/stretchr/testify/require"
)
func waitForPendingWrites(t *testing.T, v *Volume, id NeedleId, count int) {
t.Helper()
deadline := time.Now().Add(10 * time.Second)
for {
v.pendingWrites.Lock()
got := len(v.pendingWrites.byId[id])
v.pendingWrites.Unlock()
if got >= count {
return
}
if time.Now().After(deadline) {
t.Fatalf("%d of %d writes of needle %d registered", got, count, id)
}
time.Sleep(time.Millisecond)
}
}
func requireNeedleDeleted(t *testing.T, v *Volume, id NeedleId) {
t.Helper()
_, err := v.readNeedle(&needle.Needle{Id: id}, nil, nil)
require.ErrorIs(t, err, ErrorDeleted)
}
// A replicate delete that finds nothing can still have the write it undoes
// queued behind the volume locks. That write must not land live afterwards.
func TestDeleteOfMissingNeedleCancelsQueuedWrite(t *testing.T) {
for _, fsync := range []bool{false, true} {
t.Run(fmt.Sprintf("fsync=%v", fsync), func(t *testing.T) {
dir := t.TempDir()
v, err := NewVolume(dir, dir, "", 1, NeedleMapInMemory, &super_block.ReplicaPlacement{}, &needle.TTL{}, 0, needle.GetCurrentVersion(), 0, 0)
require.NoError(t, err)
n := newRandomNeedle(1)
v.dataFileSyncLock.Lock()
written := make(chan error, 1)
go func() {
_, _, _, err := v.writeNeedle2(n, true, fsync, false)
written <- err
}()
waitForPendingWrites(t, v, n.Id, 1)
_, err = v.readNeedle(&needle.Needle{Id: n.Id}, nil, nil)
require.ErrorIs(t, err, ErrorNotFound)
v.CancelPendingWrites(n.Id, n.Cookie)
v.dataFileSyncLock.Unlock()
require.ErrorIs(t, <-written, ErrorDeleted)
_, err = v.readNeedle(&needle.Needle{Id: n.Id}, nil, nil)
require.ErrorIs(t, err, ErrorNotFound)
require.Empty(t, v.pendingWrites.byId)
v.Close()
v, err = NewVolume(dir, dir, "", 1, NeedleMapInMemory, nil, nil, 0, needle.GetCurrentVersion(), 0, 0)
require.NoError(t, err)
defer v.Close()
_, err = v.readNeedle(&needle.Needle{Id: n.Id}, nil, nil)
require.ErrorIs(t, err, ErrorNotFound)
})
}
}
// A delete that finds a copy also cancels a retry of the same write that is
// still queued, so the retry does not bring the needle back.
func TestDeleteCancelsQueuedRewrite(t *testing.T) {
v := newWorkerTestVolume(t)
n := newRandomNeedle(1)
_, _, _, err := v.writeNeedle2(n, true, false, false)
require.NoError(t, err)
rewrite := newRandomNeedle(1)
rewrite.Cookie = n.Cookie
// holds the durable rewrite after it registers, before it reaches the worker
v.asyncWorkerLock.Lock()
written := make(chan error, 1)
go func() {
_, _, _, err := v.writeNeedle2(rewrite, true, true, false)
written <- err
}()
waitForPendingWrites(t, v, n.Id, 1)
_, err = v.deleteNeedle2(&needle.Needle{Id: n.Id, Cookie: n.Cookie}, false)
require.NoError(t, err)
requireNeedleDeleted(t, v, n.Id)
v.asyncWorkerLock.Unlock()
require.ErrorIs(t, <-written, ErrorDeleted)
requireNeedleDeleted(t, v, n.Id)
require.Empty(t, v.pendingWrites.byId)
}
// A rewrite cancelled before it takes the volume locks appends nothing, so
// no unsynced row of it can outlive a durable delete after a crash.
func TestCancelledRewriteDoesNotAppend(t *testing.T) {
v := newWorkerTestVolume(t)
n := newRandomNeedle(1)
_, _, _, err := v.writeNeedle2(n, true, false, false)
require.NoError(t, err)
before, _, err := v.DataBackend.GetStat()
require.NoError(t, err)
rewrite := newRandomNeedle(1)
rewrite.Cookie = n.Cookie
v.dataFileSyncLock.Lock()
written := make(chan error, 1)
go func() {
_, _, _, err := v.writeNeedle2(rewrite, true, false, false)
written <- err
}()
waitForPendingWrites(t, v, n.Id, 1)
v.CancelPendingWrites(n.Id, n.Cookie)
v.dataFileSyncLock.Unlock()
require.ErrorIs(t, <-written, ErrorDeleted)
after, _, err := v.DataBackend.GetStat()
require.NoError(t, err)
require.Equal(t, before, after)
_, err = v.deleteNeedle2(&needle.Needle{Id: n.Id, Cookie: n.Cookie}, true)
require.NoError(t, err)
requireNeedleDeleted(t, v, n.Id)
}
// Only writes already in flight are cancelled: writing the needle again after
// the delete works as before.
func TestWriteAfterDeleteIsNotCancelled(t *testing.T) {
v := newWorkerTestVolume(t)
n := newRandomNeedle(1)
v.CancelPendingWrites(n.Id, n.Cookie)
_, err := v.deleteNeedle2(&needle.Needle{Id: n.Id, Cookie: n.Cookie}, false)
require.NoError(t, err)
_, _, _, err = v.writeNeedle2(n, true, false, false)
require.NoError(t, err)
got := &needle.Needle{Id: n.Id}
_, err = v.readNeedle(got, nil, nil)
require.NoError(t, err)
require.Equal(t, n.Data, got.Data)
}
// A delete only cancels writes that carry its cookie.
func TestCancelWithOtherCookieKeepsWrite(t *testing.T) {
v := newWorkerTestVolume(t)
n := newRandomNeedle(1)
v.dataFileSyncLock.Lock()
written := make(chan error, 1)
go func() {
_, _, _, err := v.writeNeedle2(n, true, false, false)
written <- err
}()
waitForPendingWrites(t, v, n.Id, 1)
v.CancelPendingWrites(n.Id, n.Cookie+1)
v.dataFileSyncLock.Unlock()
require.NoError(t, <-written)
got := &needle.Needle{Id: n.Id}
_, err := v.readNeedle(got, nil, nil)
require.NoError(t, err)
require.Equal(t, n.Data, got.Data)
}
// The cancelled write is taken back before the next write of the needle runs,
// so an identical write sent after the delete appends its own copy instead of
// matching the record about to be deleted, even in the same batch.
func TestWriteAfterCancelledWriteStaysLive(t *testing.T) {
v := newWorkerTestVolume(t)
v.location = &DiskLocation{syncBatch: SyncBatchOption{
MinBytes: 1 << 30,
MinRequests: syncBatchMaxRequests,
Wait: time.Minute,
}}
armed := make(chan chan time.Time, 1)
prevAfter := syncBatchAfter
syncBatchAfter = func(time.Duration) <-chan time.Time {
window := make(chan time.Time, 1)
armed <- window
return window
}
defer func() { syncBatchAfter = prevAfter }()
queue := func(n *needle.Needle) *needle.AsyncRequest {
w := v.trackPendingWrite(n)
t.Cleanup(func() { v.untrackPendingWrite(w) })
r := needle.NewAsyncRequest(n, true)
r.ActualSize = needle.GetActualSize(Size(len(n.Data)), v.Version())
require.True(t, v.asyncRequestAppend(r))
return r
}
n := newRandomNeedle(1)
first := queue(n)
window := <-armed
v.CancelPendingWrites(n.Id, n.Cookie)
retry := queue(&needle.Needle{Id: n.Id, Cookie: n.Cookie, Data: n.Data, Checksum: n.Checksum})
require.Eventually(t, func() bool { return len(v.asyncRequestsChan) == 0 }, 10*time.Second, time.Millisecond)
window <- time.Now()
_, _, _, err := first.WaitComplete()
require.ErrorIs(t, err, ErrorDeleted)
_, _, isUnchanged, err := retry.WaitComplete()
require.NoError(t, err)
require.False(t, isUnchanged, "the retry matched the cancelled write")
got := &needle.Needle{Id: n.Id}
_, err = v.readNeedle(got, nil, nil)
require.NoError(t, err)
require.Equal(t, n.Data, got.Data)
}
type syncedSizeBackend struct {
backend.BackendStorageFile
synced atomic.Int64
onSync func()
}
func (b *syncedSizeBackend) Sync() error {
if f := b.onSync; f != nil {
b.onSync = nil
f()
}
size, _, err := b.GetStat()
if err != nil {
return err
}
b.synced.Store(size)
return b.BackendStorageFile.Sync()
}
// A durable write taken back during its fsync appends the tombstone after
// that fsync; it must be flushed before the .idx row that points at it.
func TestTakenBackTombstoneIsSynced(t *testing.T) {
for _, isStopping := range []bool{false, true} {
t.Run(fmt.Sprintf("inline=%v", isStopping), func(t *testing.T) {
v := newWorkerTestVolume(t)
synced := &syncedSizeBackend{BackendStorageFile: v.DataBackend}
v.DataBackend = synced
n := newRandomNeedle(1)
synced.onSync = func() { v.CancelPendingWrites(n.Id, n.Cookie) }
_, _, _, err := v.writeNeedle2(n, true, true, isStopping)
require.ErrorIs(t, err, ErrorDeleted)
size, _, err := v.DataBackend.GetStat()
require.NoError(t, err)
require.Equal(t, size, synced.synced.Load(), "the tombstone was not flushed")
})
}
}