mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-12 09:17:44 +02:00
Peter Dodd
5f1a742726
A remote-only entry whose object was deleted from the remote storage outside the filer answered GET with 500 and stayed in the filer. The remote's not-found was lost on the way: the backends' ReadFile returned it as an untyped error, so FetchAndWriteNeedle failed with codes.Unknown and nothing downstream could tell it from any other failure. - remote_storage: GCS, S3 (NoSuchKey) and Azure (BlobNotFound) reads return ErrRemoteObjectNotFound. GCS reports a missing bucket the same way as a missing object, so it confirms the bucket with a listing. - volume server and filer: the not-found crosses gRPC as codes.NotFound carrying the sentinel's text, and the filer's cache RPC returns codes.NotFound, which the S3 gateway already maps to NoSuchKey. - s3api: the origin fallback answers NoSuchKey on a confirmed not-found. - filer: a confirmed not-found removes the stale entry, so the lazy remote-metadata cache converges on the remote. Only remote-only files outside .versions and without an active object lock are removed, only if unchanged since the fetch (checked on the object's write owner, under the lock S3 object writes take), and with a metadata-only delete: the filer skips its inline remote delete, and the delete events' entries carry a marker that makes filer.remote.sync and filer.remote.gateway skip their remote delete, while filer.sync still replicates it. The replicated DeleteEntryRequest carries keep_remote_object, so the destination's delete events are marked too. The store drops the marker from every write, so clients cannot plant it.
53 lines
2.4 KiB
Go
53 lines
2.4 KiB
Go
package sink
|
|
|
|
import (
|
|
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
|
"github.com/seaweedfs/seaweedfs/weed/replication/source"
|
|
"github.com/seaweedfs/seaweedfs/weed/util"
|
|
)
|
|
|
|
type ReplicationSink interface {
|
|
GetName() string
|
|
Initialize(configuration util.Configuration, prefix string) error
|
|
DeleteEntry(key string, isDirectory, deleteIncludeChunks bool, signatures []int32) error
|
|
CreateEntry(key string, entry *filer_pb.Entry, signatures []int32) error
|
|
UpdateEntry(key string, oldEntry *filer_pb.Entry, newParentPath string, newEntry *filer_pb.Entry, deleteIncludeChunks bool, signatures []int32) (foundExistingEntry bool, err error)
|
|
GetSinkToDirectory() string
|
|
// GetDestinationIdentity distinguishes this sink's write destination from
|
|
// any other destination the same sink type could write to: endpoint or
|
|
// account, bucket or container, and directory. filer.backup keys its resume
|
|
// checkpoint on it, so two configurations writing to different places must
|
|
// not share a value.
|
|
GetDestinationIdentity() string
|
|
SetSourceFiler(s *source.FilerSource)
|
|
IsIncremental() bool
|
|
}
|
|
|
|
// EntryMover is an optional capability for sinks that can relocate an entry
|
|
// natively, in one atomic step, instead of create-then-delete. Drivers prefer
|
|
// it for a rename so a failed copy can never leave the source deleted with no
|
|
// committed destination, a directory move never deletes descendants before they
|
|
// are recreated, and the entry's chunks are neither re-copied nor leaked.
|
|
type EntryMover interface {
|
|
MoveEntry(oldKey, newKey string, newEntry *filer_pb.Entry, signatures []int32) error
|
|
}
|
|
|
|
// MetadataOnlyDeleter is an optional capability for sinks whose own delete
|
|
// events reach remote write-back daemons: the destination filer re-emits a
|
|
// replicated delete, so a source delete that kept the remote object must stay
|
|
// metadata-only there too.
|
|
type MetadataOnlyDeleter interface {
|
|
DeleteEntryKeepingRemoteObject(key string, isDirectory, deleteIncludeChunks bool, signatures []int32) error
|
|
}
|
|
|
|
func DeleteEntry(s ReplicationSink, key string, isDirectory, deleteIncludeChunks, keepRemoteObject bool, signatures []int32) error {
|
|
if deleter, ok := s.(MetadataOnlyDeleter); ok && keepRemoteObject {
|
|
return deleter.DeleteEntryKeepingRemoteObject(key, isDirectory, deleteIncludeChunks, signatures)
|
|
}
|
|
return s.DeleteEntry(key, isDirectory, deleteIncludeChunks, signatures)
|
|
}
|
|
|
|
var (
|
|
Sinks []ReplicationSink
|
|
)
|