mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-30 08:50:48 +01:00
Chris LuandDevin
8a2fec8610
* filer.sync: skip superseded replays before fetching any chunks On a directory under heavy overwrite, most entries the sync replays are already superseded on the source. The sink only learned that after chunk reads failed — or worse, after fetching and uploading bytes a later event replaces — so the backlog drained slowly and the log filled with 404s. CreateEntry and UpdateEntry now check sourceMovedOn before touching a chunk: one entry lookup skips a stale replay entirely. A mid-replication supersession is still caught by the existing retry gates. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: bound the source liveness probes with a timeout The supersession checks added a source entry lookup to the replication path, and every lookup ran on context.Background() with no deadline. A source that accepts the call but whose metadata store stalls would hold the sync forever — before a single chunk fetch, even when every chunk is still readable. Bound the probe at 30s; an unanswered lookup proves nothing, so replication proceeds and the existing retry gates decide. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: sync the live version a superseded replay points at Skipping a stale replay assumed a later event always re-carries the current content. It does not when the superseding write was itself replicated in from another cluster: those events are dropped before they reach a filer sink, so the destination could be left missing the file. When the source reports a live entry newer than the replay, commit that version through the normal path instead — equivalent to applying the later event, with no dependence on its delivery. A gone source entry still skips, since the destination already matches the source. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: reconcile moved entry content after a native rename AtomicRenameEntry relocates whatever version the destination held. If the preceding update replay was skipped because the source had already moved to the new name, the moved entry keeps the old bytes and — with no further writes — stays stale forever. MoveEntry now runs the same reconciliation as an in-place update against the rename's snapshot, so the moved entry picks up pending content changes. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: repair from the live snapshot a superseded replay resolves to An incremental diff against the replayed oldEntry can resolve manifests the overwrite already vacuumed, which fails every retry of the event. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
53 lines
2.4 KiB
Go
53 lines
2.4 KiB
Go
package sink
|
|
|
|
import (
|
|
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
|
"github.com/seaweedfs/seaweedfs/weed/replication/source"
|
|
"github.com/seaweedfs/seaweedfs/weed/util"
|
|
)
|
|
|
|
type ReplicationSink interface {
|
|
GetName() string
|
|
Initialize(configuration util.Configuration, prefix string) error
|
|
DeleteEntry(key string, isDirectory, deleteIncludeChunks bool, signatures []int32) error
|
|
CreateEntry(key string, entry *filer_pb.Entry, signatures []int32) error
|
|
UpdateEntry(key string, oldEntry *filer_pb.Entry, newParentPath string, newEntry *filer_pb.Entry, deleteIncludeChunks bool, signatures []int32) (foundExistingEntry bool, err error)
|
|
GetSinkToDirectory() string
|
|
// GetDestinationIdentity distinguishes this sink's write destination from
|
|
// any other destination the same sink type could write to: endpoint or
|
|
// account, bucket or container, and directory. filer.backup keys its resume
|
|
// checkpoint on it, so two configurations writing to different places must
|
|
// not share a value.
|
|
GetDestinationIdentity() string
|
|
SetSourceFiler(s *source.FilerSource)
|
|
IsIncremental() bool
|
|
}
|
|
|
|
// EntryMover is an optional capability for sinks that can relocate an entry
|
|
// natively, in one atomic step, instead of create-then-delete. Drivers prefer
|
|
// it for a rename so a failed copy can never leave the source deleted with no
|
|
// committed destination, a directory move never deletes descendants before they
|
|
// are recreated, and the entry's chunks are neither re-copied nor leaked.
|
|
type EntryMover interface {
|
|
MoveEntry(oldKey, newKey string, oldEntry, newEntry *filer_pb.Entry, signatures []int32) error
|
|
}
|
|
|
|
// MetadataOnlyDeleter is an optional capability for sinks whose own delete
|
|
// events reach remote write-back daemons: the destination filer re-emits a
|
|
// replicated delete, so a source delete that kept the remote object must stay
|
|
// metadata-only there too.
|
|
type MetadataOnlyDeleter interface {
|
|
DeleteEntryKeepingRemoteObject(key string, isDirectory, deleteIncludeChunks bool, signatures []int32) error
|
|
}
|
|
|
|
func DeleteEntry(s ReplicationSink, key string, isDirectory, deleteIncludeChunks, keepRemoteObject bool, signatures []int32) error {
|
|
if deleter, ok := s.(MetadataOnlyDeleter); ok && keepRemoteObject {
|
|
return deleter.DeleteEntryKeepingRemoteObject(key, isDirectory, deleteIncludeChunks, signatures)
|
|
}
|
|
return s.DeleteEntry(key, isDirectory, deleteIncludeChunks, signatures)
|
|
}
|
|
|
|
var (
|
|
Sinks []ReplicationSink
|
|
)
|