Files
seaweedfs/weed/filer/meta_aggregator_replay_test.go
T
Chris LuandDevin 603a1221a7 filer: converge peer metadata replication on restart (#11698)
* filer: skip replayed peer events older than the local entry

Replay deleted and re-inserted every received event unconditionally, so
a restarted filer that rewound its subscription offset re-applied stale
peer writes over newer local state (values reverted, deleted entries
came back and pointed at already-vacuumed chunks). Compare the event's
TsNs against the local entry's Mtime and skip per-path, so a move event
can still remove its source while keeping a newer destination.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: add replay last-writer-wins regression tests

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: stop peer subscriptions and flush offsets on shutdown

Filer.Shutdown closed the store while subscription goroutines were still
mid-stream; their deferred offset save was never guaranteed to run, so a
restart rewound each peer's stream back to the last periodic flush and
replayed stale events. MetaAggregator.Shutdown now closes every peer's
stop channel and waits for the loops so the saved offset covers what was
replayed, and the reconnect sleep is interruptible by the stop channel.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: test that shutdown stops and waits for peer loops

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: order replay by persisted per-path commit timestamps

Review feedback found the mtime comparison still lets stale replay win:
a local delete removes the entry it would have been compared against, an
entry mtime is client-controllable, and the check ran unlocked so a local
write could land between it and the store mutation.

- Record the event timestamp per mutated path in the store KV on every
  emitted mutation, plus a delete record for sources of deletes and
  renames. The record survives the entry, so a replayed create older than
  a local delete no longer resurrects it; ancestor delete records cover
  children swept without per-child events.
- Replay compares the event timestamp against that record under the same
  per-path entry locks local RPC mutations hold, falling back to entry
  mtime only for entries written before records existed. Lookup errors
  now abort the event instead of being treated as an absent entry.
- MetaAggregator.Shutdown marks the aggregator stopped under peerChansLock
  so a master update cannot start a subscription that outlives the wait,
  and waits for the loops without the five-second cap now that every
  blocking stage — signature RPC, bootstrap traversal, retry sleeps, and
  per-event processing — cancels on the stop channel.

* filer: pin peer offset at failed replay, fix commit-record context and key size

A replayed event that exhausted its retries (or was interrupted by
shutdown) was still followed by advancing the peer watermark, so the
persisted offset skipped the unapplied event forever. replicateMetadataChange
now reports the outcome: errPeerSubscriptionStopped aborts the subscription
so the deferred save keeps the last completed event, and an exhausted retry
records a barrier the persisted offset and reconnect resume point cannot
pass while the stream keeps flowing.

Commit records also failed on two fronts: NotifyUpdateEvent runs after
CommitTransaction with the same context, so SQL stores wrote MpTs/MpDl
through an already-committed *sql.Tx; and metaPathKey embedded the whole
path, overflowing the 766-char name column SQL KV stores use, which the
KvPut fallback update then silently dropped. The writes now use a fresh
context and a sha256-hashed fixed-size key.

* filer: lock HTTP delete handler and replay against ancestor paths

The HTTP DeleteHandler called DeleteEntryMetaAndData without the shared
entry lock, so a peer replay could still interleave with a local delete and
its commit record. It now holds the same lock the gRPC DeleteEntry handler
does.

Replay also locked only the exact event paths, but its ordering check
consults ancestor delete records: a local recursive delete holds only the
ancestor's lock, so a stale child create could check before the tombstone
landed and then reinsert. lockReplayPaths now locks every ancestor of both
event paths in one lexicographic order — parents sort before children, as
the rename handler's two-path order already does — while never locking the
root, which cannot be deleted and would serialize all replays.

* filer: record replay commit before the replacement insert

If the same-path update DeleteEntry succeeded but InsertEntry failed, the path sat deleted with no commit record until the retry arrived, and a stale peer create could resurrect it in the gap. The commit record now lands immediately after the delete — without a delete record, since an update is not a delete — and again after the successful insert.

* filer: lock replay ancestors shared, not exclusive

Exclusive locks on every ancestor serialized all replays under a shared
parent — unrelated objects in the same bucket could no longer apply in
parallel. Ancestors only need to exclude an exclusive holder (a local
delete or rename of the ancestor), so they now take shared locks; a path
the event changes stays exclusive even when it is also the other path's
ancestor.

* filer: order peer bootstrap on the sender's commit timestamp

traversePeerMetadata compared localCommitTsNs — a commit-clock value —
against the entry's mtime, which a client can set freely: a peer change
committed recently with an old mtime was skipped, and a stale entry
mtime-stamped in the future overwrote a newer local commit.

TraverseBfsMetadataResponse now carries the sender's per-path commit
timestamp (recorded MpTs record, entry mtime as the unrecorded fallback).
The bootstrap applies an entry only when its commit is newer than the
local one and records the applied timestamp, so replayed events order
against it — closing the gap between bootstrap content and the event
stream. A peer without the field sends 0 and keeps the mtime fallback,
preserving rolling-upgrade behavior.

* filer: keep failed commit records pending until they land

notePathChange ran after the mutation committed and only logged a KvPut
failure: a delete whose MpTs/MpDl record never landed left no tombstone,
so a stale replayed create could resurrect the path while the store was
unreachable.

Failed records now stay in metaTsPending and a repair pass retries them
until they land. localCommitTsNs, deletedAfterTsNs and CommitTsNs
consult the pending record, so ordering holds in the window between the
committed mutation and the durable record. A per-path flag bounds the
repair to one goroutine, and respawns cover merges landing between its
last pass and the flag release.

* filer: keep only unwritten work in the pending commit record

The pending record merged every obligation into one timestamp and flag,
and a later successful write left it untouched: a repair pass could then
write an older timestamp over a newer one, and a merged delete inherited
the newer change's time, which the ancestor check reads as a newer
delete. The record now holds a change and a delete timestamp of its own,
both discharged under the record's lock alongside the store writes that
cover them, so a repair writes only what is still owed.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: fail traversal on a commit-record read error

CommitTsNs answered a store read failure with the entry mtime, a value
clients can set arbitrarily far ahead; a peer bootstrapping on it would
record the minted time and reject real updates below it. The lookup now
returns the error so the traversal restarts instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 17:39:34 +08:00

711 lines
24 KiB
Go

package filer
import (
"context"
"errors"
"sync"
"testing"
"time"
dto "github.com/prometheus/client_model/go"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/stats"
"github.com/seaweedfs/seaweedfs/weed/util"
)
// newReplayTestFiler builds the minimum Filer Replay needs over a stub store.
func newReplayTestFiler(store FilerStore) *Filer {
return &Filer{
Store: NewFilerStoreWrapper(store),
EntryLockTable: util.NewLockTable[util.FullPath](),
}
}
// flakyReplayStore fails InsertEntry for its first failUntil calls, then
// delegates to the embedded stubFilerStore.
type flakyReplayStore struct {
*stubFilerStore
countMu sync.Mutex
failUntil int
err error
attempts int
}
func (s *flakyReplayStore) InsertEntry(ctx context.Context, entry *Entry) error {
s.countMu.Lock()
s.attempts++
fail := s.attempts <= s.failUntil
s.countMu.Unlock()
if fail {
return s.err
}
return s.stubFilerStore.InsertEntry(ctx, entry)
}
func (s *flakyReplayStore) attemptCount() int {
s.countMu.Lock()
defer s.countMu.Unlock()
return s.attempts
}
func replayFailureCount(t *testing.T, peer pb.ServerAddress) float64 {
t.Helper()
var m dto.Metric
if err := stats.FilerMetaAggregatorReplayFailures.WithLabelValues(string(peer)).Write(&m); err != nil {
t.Fatalf("read counter: %v", err)
}
return m.GetCounter().GetValue()
}
func quotaChangeEvent(bucket string, quota int64) *filer_pb.SubscribeMetadataResponse {
return &filer_pb.SubscribeMetadataResponse{
Directory: "/buckets",
EventNotification: &filer_pb.EventNotification{
NewEntry: &filer_pb.Entry{Name: bucket, IsDirectory: true, Quota: quota},
},
TsNs: time.Now().UnixNano(),
}
}
// The old body logged a Replay error and returned, so one transient failure
// permanently diverged the entry from the peer.
func TestReplicateMetadataChangeRetriesTransientFailure(t *testing.T) {
// int64: untyped, the value defaults to int and overflows a 32-bit build
const wantQuota int64 = 131072 << 20
store := &flakyReplayStore{
stubFilerStore: newStubFilerStore(),
failUntil: 1,
err: errors.New("i/o timeout talking to store"),
}
peer := pb.ServerAddress("peer-transient:1")
f := newReplayTestFiler(store)
if err := replicateMetadataChange(f, peer, quotaChangeEvent("my-bucket", wantQuota), nil); err != nil {
t.Fatalf("replicateMetadataChange: %v", err)
}
inserted, err := store.FindEntry(context.Background(), util.NewFullPath("/buckets", "my-bucket"))
if err != nil {
t.Fatal("expected the entry to be inserted once the transient failure is retried past")
}
if inserted.Quota != wantQuota {
t.Fatalf("quota = %d, want %d", inserted.Quota, wantQuota)
}
if got := store.attemptCount(); got < 2 {
t.Fatalf("attempts = %d, want at least 2", got)
}
}
// An event that can never replay must fail fast instead of blocking the
// subscribe stream, and must be counted so the divergence is not silent.
func TestReplicateMetadataChangeGivesUpLoudlyOnPermanentFailure(t *testing.T) {
store := &flakyReplayStore{
stubFilerStore: newStubFilerStore(),
failUntil: 1 << 30,
err: errors.New("entry checksum mismatch"),
}
peer := pb.ServerAddress("peer-permanent:1")
f := newReplayTestFiler(store)
before := replayFailureCount(t, peer)
err := replicateMetadataChange(f, peer, quotaChangeEvent("poison-bucket", 65536<<20), nil)
if err == nil {
t.Fatal("expected the exhausted replay error so the caller can pin the peer offset at the failed event")
}
if got := store.attemptCount(); got != 1 {
t.Fatalf("attempts = %d, want exactly 1: a non-transient error must fail fast, not retry", got)
}
if got := replayFailureCount(t, peer); got != before+1 {
t.Fatalf("FilerMetaAggregatorReplayFailures[%s] = %v, want %v", peer, got, before+1)
}
}
func overwriteEvent(dir, name string, tsNs int64, content string) *filer_pb.SubscribeMetadataResponse {
return &filer_pb.SubscribeMetadataResponse{
Directory: dir,
EventNotification: &filer_pb.EventNotification{
OldEntry: &filer_pb.Entry{Name: name},
NewEntry: &filer_pb.Entry{Name: name, Content: []byte(content)},
},
TsNs: tsNs,
}
}
func deleteEvent(dir, name string, tsNs int64) *filer_pb.SubscribeMetadataResponse {
return &filer_pb.SubscribeMetadataResponse{
Directory: dir,
EventNotification: &filer_pb.EventNotification{
OldEntry: &filer_pb.Entry{Name: name},
},
TsNs: tsNs,
}
}
func TestReplaySkipsEventOlderThanLocalEntry(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
now := time.Now()
path := util.NewFullPath("/dir", "file")
store.entries[string(path)] = &Entry{
FullPath: path,
Attr: Attr{Mtime: now},
Content: []byte("newer local write"),
}
if err := f.Replay(overwriteEvent("/dir", "file", now.Add(-time.Minute).UnixNano(), "old replayed version")); err != nil {
t.Fatalf("Replay: %v", err)
}
entry, err := store.FindEntry(context.Background(), path)
if err != nil {
t.Fatalf("FindEntry: %v", err)
}
if string(entry.Content) != "newer local write" {
t.Fatalf("content = %q, want local write kept", entry.Content)
}
}
func TestReplayAppliesEventNewerThanLocalEntry(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
now := time.Now()
path := util.NewFullPath("/dir", "file")
store.entries[string(path)] = &Entry{
FullPath: path,
Attr: Attr{Mtime: now.Add(-time.Minute)},
Content: []byte("old local write"),
}
if err := f.Replay(overwriteEvent("/dir", "file", now.UnixNano(), "newer peer write")); err != nil {
t.Fatalf("Replay: %v", err)
}
entry, err := store.FindEntry(context.Background(), path)
if err != nil {
t.Fatalf("FindEntry: %v", err)
}
if string(entry.Content) != "newer peer write" {
t.Fatalf("content = %q, want peer write applied", entry.Content)
}
}
func TestReplaySkipsDeleteOlderThanLocalEntry(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
now := time.Now()
path := util.NewFullPath("/dir", "file")
store.entries[string(path)] = &Entry{FullPath: path, Attr: Attr{Mtime: now}}
if err := f.Replay(deleteEvent("/dir", "file", now.Add(-time.Minute).UnixNano())); err != nil {
t.Fatalf("Replay: %v", err)
}
if _, err := store.FindEntry(context.Background(), path); err != nil {
t.Fatalf("newer local entry must survive a stale delete: %v", err)
}
}
func TestReplayAppliesNewerDelete(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
now := time.Now()
path := util.NewFullPath("/dir", "file")
store.entries[string(path)] = &Entry{FullPath: path, Attr: Attr{Mtime: now.Add(-time.Minute)}}
if err := f.Replay(deleteEvent("/dir", "file", now.UnixNano())); err != nil {
t.Fatalf("Replay: %v", err)
}
if _, err := store.FindEntry(context.Background(), path); err == nil {
t.Fatal("newer delete must remove the local entry")
}
}
// A move event touches two paths; the local check applies to each side
// independently, so a newer destination is kept while the source is removed.
func TestReplayMoveChecksEachPathSeparately(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
now := time.Now()
oldPath := util.NewFullPath("/a", "file")
newPath := util.NewFullPath("/b", "file")
store.entries[string(oldPath)] = &Entry{FullPath: oldPath, Attr: Attr{Mtime: now.Add(-time.Minute)}}
store.entries[string(newPath)] = &Entry{FullPath: newPath, Attr: Attr{Mtime: now}, Content: []byte("kept")}
event := &filer_pb.SubscribeMetadataResponse{
Directory: "/a",
EventNotification: &filer_pb.EventNotification{
OldEntry: &filer_pb.Entry{Name: "file"},
NewEntry: &filer_pb.Entry{Name: "file"},
NewParentPath: "/b",
},
TsNs: now.Add(-time.Minute).UnixNano(),
}
if err := f.Replay(event); err != nil {
t.Fatalf("Replay: %v", err)
}
if _, err := store.FindEntry(context.Background(), oldPath); err == nil {
t.Fatal("old source entry should be deleted")
}
entry, err := store.FindEntry(context.Background(), newPath)
if err != nil {
t.Fatalf("FindEntry: %v", err)
}
if string(entry.Content) != "kept" {
t.Fatalf("content = %q, want newer destination kept", entry.Content)
}
}
// A local delete leaves a recorded tombstone, so a replayed create older than
// the delete must not resurrect the path even though no entry remains to
// compare against.
func TestReplaySkipsCreateOlderThanLocalDelete(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
now := time.Now()
path := util.NewFullPath("/dir", "file")
// The delete removed the entry; its commit record survives.
f.notePathChange(path, now.UnixNano(), true)
createEvent := &filer_pb.SubscribeMetadataResponse{
Directory: "/dir",
EventNotification: &filer_pb.EventNotification{
NewEntry: &filer_pb.Entry{Name: "file", Content: []byte("stale resurrect")},
},
TsNs: now.Add(-time.Minute).UnixNano(),
}
if err := f.Replay(createEvent); err != nil {
t.Fatalf("Replay: %v", err)
}
if _, err := store.FindEntry(context.Background(), path); err == nil {
t.Fatal("stale replayed create resurrected a locally deleted entry")
}
}
// A deleted directory covers its children: a bulk drop emits no per-child
// events, so the tombstone on the ancestor is what a replayed child create
// must check.
func TestReplaySkipsCreateUnderDeletedAncestor(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
now := time.Now()
f.notePathChange("/dir", now.UnixNano(), true)
createEvent := &filer_pb.SubscribeMetadataResponse{
Directory: "/dir",
EventNotification: &filer_pb.EventNotification{
NewEntry: &filer_pb.Entry{Name: "child", Content: []byte("stale")},
},
TsNs: now.Add(-time.Minute).UnixNano(),
}
if err := f.Replay(createEvent); err != nil {
t.Fatalf("Replay: %v", err)
}
if _, err := store.FindEntry(context.Background(), "/dir/child"); err == nil {
t.Fatal("stale replayed create resurrected a child of a deleted directory")
}
}
// A recorded commit timestamp orders replay, not the entry's mtime: a rename
// keeps the source's mtime on the destination, and a mounted client can set
// mtime arbitrarily, so an entry whose mtime postdates the event must still
// be overwritten when its commit record is older.
func TestReplayAppliesWhenCommitRecordIsOlderThanFutureMtime(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
now := time.Now()
path := util.NewFullPath("/b", "x")
store.entries[string(path)] = &Entry{
FullPath: path,
Attr: Attr{Mtime: now.Add(time.Hour)}, // future mtime, e.g. client-set or kept from a rename source
Content: []byte("old destination"),
}
// The last committed change predates the event.
f.notePathChange(path, now.Add(-time.Minute).UnixNano(), false)
event := &filer_pb.SubscribeMetadataResponse{
Directory: "/a",
EventNotification: &filer_pb.EventNotification{
OldEntry: &filer_pb.Entry{Name: "x"},
NewEntry: &filer_pb.Entry{Name: "x", Content: []byte("renamed in")},
NewParentPath: "/b",
},
TsNs: now.UnixNano(),
}
if err := f.Replay(event); err != nil {
t.Fatalf("Replay: %v", err)
}
entry, err := store.FindEntry(context.Background(), path)
if err != nil {
t.Fatalf("FindEntry: %v", err)
}
if string(entry.Content) != "renamed in" {
t.Fatalf("content = %q, want the rename destination applied", entry.Content)
}
}
// A store read failure is not a not-found: the replayed change must not be
// applied on the guess that no newer local entry exists.
func TestReplayPropagatesLookupError(t *testing.T) {
store := newStubFilerStore()
store.findErr = errors.New("store read failed")
f := newReplayTestFiler(store)
err := f.Replay(overwriteEvent("/dir", "file", time.Now().UnixNano(), "v"))
if err == nil {
t.Fatal("expected Replay to propagate the FindEntry failure")
}
}
// The check and the mutation share the per-path entry lock, so a local write
// holding that lock cannot be split by a replayed change.
func TestReplaySerializesWithEntryLock(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
now := time.Now()
path := util.NewFullPath("/dir", "file")
store.entries[string(path)] = &Entry{FullPath: path, Attr: Attr{Mtime: now}}
// A local mutation holding the path lock; a replay started behind it
// sees the recorded commit and skips.
held := f.EntryLockTable.AcquireLock("test local write", path, util.ExclusiveLock)
f.notePathChange(path, now.UnixNano(), false)
done := make(chan error, 1)
go func() {
done <- f.Replay(overwriteEvent("/dir", "file", now.Add(-time.Minute).UnixNano(), "stale"))
}()
select {
case <-done:
t.Fatal("replay finished while the path lock was still held")
case <-time.After(50 * time.Millisecond):
}
f.EntryLockTable.ReleaseLock(path, held)
if err := <-done; err != nil {
t.Fatalf("Replay: %v", err)
}
entry, err := store.FindEntry(context.Background(), path)
if err != nil {
t.Fatalf("FindEntry: %v", err)
}
if entry.Content != nil {
t.Fatalf("content = %q, want local entry kept", entry.Content)
}
}
// A closed stop channel must abort the retry loop with
// errPeerSubscriptionStopped so the subscription tears down and the offset
// save keeps the last completed event instead of skipping this one.
func TestReplicateMetadataChangeAbortsOnStop(t *testing.T) {
store := &flakyReplayStore{
stubFilerStore: newStubFilerStore(),
failUntil: 1 << 30,
err: errors.New("i/o timeout talking to store"),
}
peer := pb.ServerAddress("peer-stopped:1")
f := newReplayTestFiler(store)
stopChan := make(chan struct{})
close(stopChan)
err := replicateMetadataChange(f, peer, quotaChangeEvent("stuck-bucket", 1<<20), stopChan)
if !errors.Is(err, errPeerSubscriptionStopped) {
t.Fatalf("err = %v, want errPeerSubscriptionStopped", err)
}
if got := store.attemptCount(); got != 0 {
t.Fatalf("attempts = %d, want 0: a stopped subscription must not replay", got)
}
}
// Commit-record keys must stay small: SQL KV stores keep the key suffix in a
// VARCHAR(766) column, so a raw path over ~578 bytes silently drops the
// record (the insert fails and the fallback update touches no row).
func TestMetaPathKeyFixedSizeForLongPaths(t *testing.T) {
path := util.NewFullPath("/", string(make([]rune, 600)))
for _, p := range []util.FullPath{"/a", path} {
if got := len(metaPathKey(metaPathTsPrefix, p)); got != len(metaPathTsPrefix)+16 {
t.Fatalf("metaPathKey(%d-byte path) = %d bytes, want %d", len(p), got, len(metaPathTsPrefix)+16)
}
}
}
// Exact-path locking alone lets a replayed child create slip between a local
// ancestor delete's tombstone check and the sweep: replay locks every
// ancestor too, so it waits for the delete's lock and then sees the
// tombstone.
func TestReplaySerializesWithAncestorLock(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
now := time.Now()
child := util.NewFullPath("/dir", "file")
held := f.EntryLockTable.AcquireLock("test local delete", "/dir", util.ExclusiveLock)
f.notePathChange("/dir", now.UnixNano(), true)
createEvent := &filer_pb.SubscribeMetadataResponse{
Directory: "/dir",
EventNotification: &filer_pb.EventNotification{
NewEntry: &filer_pb.Entry{Name: "file", Content: []byte("stale resurrect")},
},
TsNs: now.Add(-time.Minute).UnixNano(),
}
done := make(chan error, 1)
go func() {
done <- f.Replay(createEvent)
}()
select {
case <-done:
t.Fatal("replay finished while the ancestor lock was still held")
case <-time.After(50 * time.Millisecond):
}
f.EntryLockTable.ReleaseLock("/dir", held)
if err := <-done; err != nil {
t.Fatalf("Replay: %v", err)
}
if _, err := store.FindEntry(context.Background(), child); err == nil {
t.Fatal("stale replayed create resurrected a child of a locally deleted directory")
}
}
// Bootstrap ordering uses the sender's commit timestamp: a local commit
// newer than the traversed entry keeps its state even when the peer entry's
// mtime is far in the future.
func TestBootstrapSkipsPeerEntryBehindLocalDelete(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
now := time.Now()
path := util.NewFullPath("/dir", "file")
f.notePathChange(path, now.UnixNano(), true)
entry := &Entry{
FullPath: path,
Attr: Attr{Mtime: now.Add(time.Hour)}, // client-settable, must not win
Content: []byte("stale peer entry"),
}
if err := f.applyTraversedEntry(context.Background(), entry, now.Add(-time.Minute).UnixNano()); err != nil {
t.Fatalf("applyTraversedEntry: %v", err)
}
if _, err := store.FindEntry(context.Background(), path); err == nil {
t.Fatal("bootstrap resurrected a path newer local delete covered")
}
}
// The bug this guards: a peer entry committed recently but stamped with an
// old client mtime must still apply over an absent-or-older local commit.
func TestBootstrapAppliesPeerCommitNewerThanLocal(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
now := time.Now()
path := util.NewFullPath("/dir", "file")
entry := &Entry{
FullPath: path,
Attr: Attr{Mtime: now.Add(-time.Hour)}, // old client-set mtime
Content: []byte("fresh peer write"),
}
if err := f.applyTraversedEntry(context.Background(), entry, now.UnixNano()); err != nil {
t.Fatalf("applyTraversedEntry: %v", err)
}
got, err := store.FindEntry(context.Background(), path)
if err != nil {
t.Fatalf("peer entry with newer commit must bootstrap: %v", err)
}
if string(got.Content) != "fresh peer write" {
t.Fatalf("content = %q, want peer entry applied", got.Content)
}
}
// A bootstrap-applied entry records the sender's commit timestamp, so a
// replayed event older than it — but newer than the entry's mtime — is
// still skipped.
func TestBootstrapAppliedCommitBlocksStaleReplay(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
now := time.Now()
path := util.NewFullPath("/dir", "file")
entry := &Entry{
FullPath: path,
Attr: Attr{Mtime: now.Add(-time.Hour)},
Content: []byte("fresh peer write"),
}
if err := f.applyTraversedEntry(context.Background(), entry, now.UnixNano()); err != nil {
t.Fatalf("applyTraversedEntry: %v", err)
}
if err := f.Replay(overwriteEvent("/dir", "file", now.Add(-time.Minute).UnixNano(), "stale replay")); err != nil {
t.Fatalf("Replay: %v", err)
}
got, _ := store.FindEntry(context.Background(), path)
if string(got.Content) != "fresh peer write" {
t.Fatalf("content = %q, stale replay must not overwrite the bootstrap entry", got.Content)
}
}
// An existing local entry is replaced only when the peer commit is newer —
// not when its mtime happens to be.
func TestBootstrapUpdatesExistingByCommitTime(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
now := time.Now()
path := util.NewFullPath("/dir", "file")
store.entries[string(path)] = &Entry{
FullPath: path,
Attr: Attr{Mtime: now.Add(time.Hour)}, // future mtime, e.g. client-set
Content: []byte("old local write"),
}
f.notePathChange(path, now.Add(-time.Minute).UnixNano(), false)
entry := &Entry{
FullPath: path,
Attr: Attr{Mtime: now.Add(-time.Hour)},
Content: []byte("newer peer commit"),
}
if err := f.applyTraversedEntry(context.Background(), entry, now.UnixNano()); err != nil {
t.Fatalf("applyTraversedEntry: %v", err)
}
got, _ := store.FindEntry(context.Background(), path)
if string(got.Content) != "newer peer commit" {
t.Fatalf("content = %q, newer peer commit must update despite the future local mtime", got.Content)
}
}
// A delete whose commit record failed to persist still blocks a stale
// replayed create: the pending record stands in until the retry lands it.
func TestPendingDeleteRecordBlocksStaleReplay(t *testing.T) {
store := newStubFilerStore()
store.setKvPutErr(errors.New("kv store unreachable"))
f := newReplayTestFiler(store)
f.flushCtx, f.flushCancel = context.WithCancel(context.Background())
defer f.flushCancel()
now := time.Now()
path := util.NewFullPath("/dir", "file")
// The delete already committed; only its ordering record failed.
f.notePathChange(path, now.UnixNano(), true)
createEvent := &filer_pb.SubscribeMetadataResponse{
Directory: "/dir",
EventNotification: &filer_pb.EventNotification{
NewEntry: &filer_pb.Entry{Name: "file", Content: []byte("stale resurrect")},
},
TsNs: now.Add(-time.Minute).UnixNano(),
}
if err := f.Replay(createEvent); err != nil {
t.Fatalf("Replay: %v", err)
}
if _, err := store.FindEntry(context.Background(), path); err == nil {
t.Fatal("stale create resurrected a path whose delete record is still pending")
}
}
// The failed record write retries until it lands, then leaves the pending set.
func TestPendingCommitRecordRepairs(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
f.flushCtx, f.flushCancel = context.WithCancel(context.Background())
defer f.flushCancel()
path := util.NewFullPath("/dir", "file")
store.setKvPutErr(errors.New("kv store unreachable"))
f.notePathChange(path, 100, true)
if _, loaded := f.metaTsPending.Load(string(path)); !loaded {
t.Fatal("failed record write must stay pending")
}
store.setKvPutErr(nil)
deadline := time.Now().Add(5 * time.Second)
for time.Now().Before(deadline) {
if _, loaded := f.metaTsPending.Load(string(path)); !loaded {
ts, found, err := f.kvReadTsNs(context.Background(), metaPathKey(metaPathDelPrefix, path))
if err != nil || !found || ts != 100 {
t.Fatalf("delete record after repair: ts=%v found=%v err=%v", ts, found, err)
}
return
}
time.Sleep(10 * time.Millisecond)
}
t.Fatal("pending record was not repaired")
}
// CommitTsNs reports the recorded commit time over the entry mtime, and a
// still-pending record over both.
func TestCommitTsNsPrefersRecordAndPending(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
path := util.NewFullPath("/dir", "file")
if got, err := f.CommitTsNs(context.Background(), path, 7); err != nil || got != 7 {
t.Fatalf("no record: CommitTsNs = %d, %v; want fallback 7", got, err)
}
f.notePathChange(path, 50, false)
if got, err := f.CommitTsNs(context.Background(), path, 7); err != nil || got != 50 {
t.Fatalf("recorded: CommitTsNs = %d, %v; want 50", got, err)
}
pending := &metaTsPendingRec{}
pending.changeTsNs.Store(80)
f.metaTsPending.Store(string(path), pending)
if got, err := f.CommitTsNs(context.Background(), path, 7); err != nil || got != 80 {
t.Fatalf("pending: CommitTsNs = %d, %v; want 80", got, err)
}
}
// A successful write discharges what the pending record owed before it, so a
// repair pass can never put an older timestamp back over it.
func TestPendingCommitRecordDischargedByNewerWrite(t *testing.T) {
store := newStubFilerStore()
f := newReplayTestFiler(store)
f.flushCtx, f.flushCancel = context.WithCancel(context.Background())
defer f.flushCancel()
path := util.NewFullPath("/dir", "file")
store.setKvPutErr(errors.New("kv store unreachable"))
f.notePathChange(path, 100, false)
store.setKvPutErr(nil)
f.notePathChange(path, 200, false)
deadline := time.Now().Add(5 * time.Second)
for time.Now().Before(deadline) {
if _, loaded := f.metaTsPending.Load(string(path)); !loaded {
break
}
time.Sleep(10 * time.Millisecond)
}
ts, found, err := f.kvReadTsNs(context.Background(), metaPathKey(metaPathTsPrefix, path))
if err != nil || !found || ts != 200 {
t.Fatalf("MpTs after repair = %v, %v, %v; want 200", ts, found, err)
}
}
// A failed delete record keeps its own time even when a later failed change
// merges into the pending record: the repair must not stamp the delete at
// the newer change's time.
func TestPendingDeleteRecordKeepsItsTime(t *testing.T) {
store := newStubFilerStore()
store.setKvPutErr(errors.New("kv store unreachable"))
f := newReplayTestFiler(store)
f.flushCtx, f.flushCancel = context.WithCancel(context.Background())
defer f.flushCancel()
path := util.NewFullPath("/dir", "file")
f.notePathChange(path, 100, true)
f.notePathChange(path, 200, false)
store.setKvPutErr(nil)
deadline := time.Now().Add(5 * time.Second)
for time.Now().Before(deadline) {
if _, loaded := f.metaTsPending.Load(string(path)); !loaded {
break
}
time.Sleep(10 * time.Millisecond)
}
if ts, found, err := f.kvReadTsNs(context.Background(), metaPathKey(metaPathTsPrefix, path)); err != nil || !found || ts != 200 {
t.Fatalf("MpTs = %v, %v, %v; want 200", ts, found, err)
}
if ts, found, err := f.kvReadTsNs(context.Background(), metaPathKey(metaPathDelPrefix, path)); err != nil || !found || ts != 100 {
t.Fatalf("MpDl = %v, %v, %v; want 100", ts, found, err)
}
}