mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-08 04:57:50 +01:00
filer.remote.sync: -allMounts writes back every mounted directory from one process (#12476)
* filer.remote.sync: -allMounts writes back every mounted directory from one process A filer with many remote mounts needs one filer.remote.sync process per mount. -allMounts runs the existing -dir write-back once per directory in the mount table, in one process: each with its own subscription, client id (the filer keeps one subscription per client id), offset, processor and retry loop. A supervisor reads the mount table every -mountsCheckInterval (5s) and starts a write-back for each mounted directory that has none running, 10 at a time, 500 ms apart. It never stops one because the live table changed: a write-back ends itself where the -dir process exits, at the first mount-table version it receives that ends its mount (#12472), and a directory still mounted is started again from its saved offset, after a pause that grows from 1s to 30s while it keeps ending within a minute. A directory mounted while the process runs, without a saved offset, waits up to 2 minutes for one (a mount followed by a pull sets it after the pull). A write-back that ends at an unmount 5 times in a row while the table still mounts it at the same location is logged as a warning. To run under a supervisor, a follow: - returns errMountGone at the end of its mount instead of exiting. admitUntilMountEnds takes the end action: the -dir process keeps draining and exiting as before; a -allMounts follow admits nothing more (events after the end can still arrive in the same batch or before the stream closes, including remote.unmount's purge and a remount), ends its stream, drains, and saves no offset. - drains its admitted jobs before it returns, however it ended, so a successor never races a job still running for the same path. - saves its offset only over the value it read at its start or last saved (offsetGuard), both at its checkpoints and at its end. Otherwise a write-back behind in the log can overwrite the offset remote.unmount writes after its purge (or a mount after its mapping write), and its successor replays the unmount and never writes the remounted directory back. Once it sees the offset moved it saves nothing more and keeps no resume position. A value whose write reported an error counts as its own when read back. - reads the stored offset once, before the mount table, and uses it as both its start and its guard's value: read after the table, an unmount completing in between would start a follow past the unmount's table. - keeps a resume position for its next attempt only with the mount location it was earned under and the offset stored at the time; it is dropped when the directory is mounted elsewhere or someone else moved the offset. - turns a panic in a job into that job's error, so one directory's failure does not end every directory's write-back. Fixes #12471 * filer.remote.sync -allMounts: retire a failed follow whose mount is gone A follow that failed before its stream reached the unmount (or before it started) was retried every 30s forever: DetectMountInfo reports the directory not mounted, which is not errMountGone. No stream runs on the retry path, so a directory the live table no longer maps is retired there and started again by reconcile if it is mounted again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync -allMounts: drop the restart pause of an unmounted directory ended() left notBefore and ends behind for every directory ever mounted. reconcile now drops them once the directory is neither mounted nor followed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: log a -dir drain held at the mount end The -dir exit drained with processor.Drain, so a remote call hung during an unmount or move held the process silently; the follow's own drain already warns every 5 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: keep the resume position through a failed mount read A transient DetectMountInfo error dropped the only position behind the event a follow failed on. With -timeAgo and no checkpoint saved yet, the retry then started at a later now minus timeAgo and skipped that event. The next attempt that reads the table still checks the position against the mount location and the stored offset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: judge a kept resume position by the directory's remote location resumeFor compared the mount's raw location, so a -dir below a mount kept its position when the same remote was mounted one level up (or down), although its paths now map to other keys; a replay from that position skips startAfterMountOffset and writes old events to the new keys. The position now carries dirRemoteLocation and is checked against it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync -allMounts: warn about a replayed unmount only from an unmoved offset The warning counted every end at an unmount while the directory stayed mounted at one location, so write-backs that each saved progress, or unmount and remount cycles between two reconciles, also tripped it, and it told the operator to set the offset to now, skipping events. An end saves no offset, so a run of ends is one replay only while the stored offset stays put: a moved offset now starts a new run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync -allMounts tests: wait for the supervisor before moving the frozen clock A reconcile or end recorded after the clock moved would start its wait or pause at the moved time, which the frozen clock never passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * remote.mount: save the sync offset right after the mount mapping A new mount had no saved offset, so filer.remote.sync -allMounts waited the full allMountsOffsetWait before following it and then started from the directory's creation time anyway. remote.mount and remote.mount.buckets now save the time taken just before the mapping write, keeping an offset remote.unmount saved for a remount. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * remote.mount: save the directory's creation time as the sync offset The mount time skipped local files already in a -nonempty directory, which a write-back with no offset replays from the directory's creation. Saving that creation time keeps the start position and only ends the -allMounts wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * remote.mount: warn instead of failing when the sync offset is not saved The mount mapping is already saved by then, and a mount without an offset still starts from the same creation time after the -allMounts wait. Failing stopped remote.mount.buckets before the remaining buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
8 files changed
+1421
-22
No files matched your search
@@ -0,0 +1,366 @@
|
||||
package command
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"sort"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/remote_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/remote_storage"
|
||||
"github.com/seaweedfs/seaweedfs/weed/replication/source"
|
||||
)
|
||||
|
||||
// The write-backs of -allMounts start this many at a time, this far apart:
|
||||
// each one replays the filer's log from its own offset, and a process that
|
||||
// starts a hundred of them at once makes the filer replay a hundred logs
|
||||
// together.
|
||||
const (
|
||||
allMountsStartBatch = 10
|
||||
allMountsStartPause = 500 * time.Millisecond
|
||||
allMountsRetryMin = time.Second
|
||||
allMountsRetryMax = 30 * time.Second
|
||||
// A directory mounted while this process runs, with no offset saved yet,
|
||||
// waits this long for one: remote.mount saves it right after the
|
||||
// mapping, and other mount tooling may save it later.
|
||||
allMountsOffsetWait = 2 * time.Minute
|
||||
// A directory whose write-back keeps ending at an unmount this many times
|
||||
// in a row, while the live table still mounts it at the same location, is
|
||||
// warned about: its replayed log holds an unmount it stops at every time.
|
||||
allMountsGoneWarnAfter = 5
|
||||
)
|
||||
|
||||
// mountSupervisor runs one write-back per mounted directory in this process.
|
||||
// Each is the -dir write-back: its own subscription, offset, processor and
|
||||
// retry loop, under its own client id (the filer keeps one subscription per
|
||||
// client id, so a shared id would make each new follow replace the last).
|
||||
//
|
||||
// The supervisor only starts write-backs. A write-back ends itself when the
|
||||
// mount table it receives, in log order, no longer maps its directory to its
|
||||
// location, exactly where the -dir process exits: everything before that
|
||||
// point is written back, nothing after it (remote.unmount's purge) is. The
|
||||
// supervisor never stops a running write-back because the live table changed,
|
||||
// which would cut off a write-back still behind in the log. A directory whose
|
||||
// write-back ended is started again, from its saved offset, while the live
|
||||
// table still mounts it (with a growing pause if it keeps ending at once).
|
||||
type mountSupervisor struct {
|
||||
option *RemoteSyncOptions
|
||||
filerSource *source.FilerSource
|
||||
|
||||
// readMappings, follow, readOffset, now and warn are the seams the tests
|
||||
// replace.
|
||||
readMappings func() (*remote_pb.RemoteStorageMapping, error)
|
||||
follow func(option *RemoteSyncOptions, dir string, watch *mountWatch) error
|
||||
readOffset func(dir string) (int64, error)
|
||||
now func() time.Time
|
||||
warn func(format string, args ...interface{})
|
||||
|
||||
mu sync.Mutex
|
||||
followers map[string]*mountFollower
|
||||
// notBefore holds a directory whose write-back ended until its next start;
|
||||
// ends counts its consecutive quick ends, for the pause.
|
||||
notBefore map[string]time.Time
|
||||
ends map[string]int
|
||||
// firstSeen is when a mounted directory without a saved offset was first
|
||||
// read in the table (allMountsOffsetWait).
|
||||
firstSeen map[string]time.Time
|
||||
// gone counts, per directory, the write-backs in a row that ended at an
|
||||
// unmount while each was started at the same location and stored offset
|
||||
// (goneStreak).
|
||||
gone map[string]*goneStreak
|
||||
wake chan struct{}
|
||||
// clientIds hands each follow its own non-zero client id.
|
||||
clientIds atomic.Int32
|
||||
}
|
||||
|
||||
// goneStreak is a directory's run of write-backs that ended at an unmount,
|
||||
// each started at loc from the stored offset; warned is set once the run has
|
||||
// been reported.
|
||||
type goneStreak struct {
|
||||
loc *remote_pb.RemoteStorageLocation
|
||||
offset int64
|
||||
ends int
|
||||
warned bool
|
||||
}
|
||||
|
||||
type mountFollower struct {
|
||||
stop chan struct{}
|
||||
done chan struct{}
|
||||
}
|
||||
|
||||
func newMountSupervisor(option *RemoteSyncOptions, filerSource *source.FilerSource) *mountSupervisor {
|
||||
m := &mountSupervisor{
|
||||
option: option,
|
||||
filerSource: filerSource,
|
||||
followers: map[string]*mountFollower{},
|
||||
notBefore: map[string]time.Time{},
|
||||
ends: map[string]int{},
|
||||
firstSeen: map[string]time.Time{},
|
||||
gone: map[string]*goneStreak{},
|
||||
wake: make(chan struct{}, 1),
|
||||
now: time.Now,
|
||||
warn: glog.Warningf,
|
||||
}
|
||||
m.clientIds.Store(int32(time.Now().UnixNano()))
|
||||
m.readMappings = func() (*remote_pb.RemoteStorageMapping, error) { return readMountMappings(option) }
|
||||
m.follow = func(o *RemoteSyncOptions, dir string, watch *mountWatch) error {
|
||||
return followMountedDir(o, filerSource, dir, watch)
|
||||
}
|
||||
m.readOffset = func(dir string) (int64, error) {
|
||||
return remote_storage.GetSyncOffset(option.grpcDialOption, pb.ServerAddress(*option.filerAddress), dir)
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// run starts write-backs until quit closes (never, for the command), then
|
||||
// stops every write-back and waits for them.
|
||||
func (m *mountSupervisor) run(quit <-chan struct{}) {
|
||||
interval := 5 * time.Second
|
||||
if m.option.mountsCheck != nil && *m.option.mountsCheck > 0 {
|
||||
interval = *m.option.mountsCheck
|
||||
}
|
||||
ticker := time.NewTicker(interval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
if err := m.reconcile(); err != nil {
|
||||
glog.Errorf("filer.remote.sync -allMounts: read mount table: %v", err)
|
||||
}
|
||||
select {
|
||||
case <-quit:
|
||||
m.stopAll()
|
||||
return
|
||||
case <-ticker.C:
|
||||
case <-m.wake:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// reconcile starts a write-back for every mounted directory that has none
|
||||
// running, is past its pause, and has a saved offset (or waited for one).
|
||||
func (m *mountSupervisor) reconcile() error {
|
||||
mappings, err := m.readMappings()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
now := m.now()
|
||||
var starting []string
|
||||
m.mu.Lock()
|
||||
for dir := range m.firstSeen {
|
||||
if mappings == nil || mappings.Mappings[dir] == nil {
|
||||
delete(m.firstSeen, dir)
|
||||
}
|
||||
}
|
||||
for dir := range m.gone {
|
||||
if mappings == nil || mappings.Mappings[dir] == nil {
|
||||
delete(m.gone, dir)
|
||||
}
|
||||
}
|
||||
for dir := range m.notBefore {
|
||||
if _, running := m.followers[dir]; !running && (mappings == nil || mappings.Mappings[dir] == nil) {
|
||||
delete(m.notBefore, dir)
|
||||
delete(m.ends, dir)
|
||||
}
|
||||
}
|
||||
if mappings != nil {
|
||||
for dir := range mappings.Mappings {
|
||||
if _, running := m.followers[dir]; running {
|
||||
continue
|
||||
}
|
||||
if t, paused := m.notBefore[dir]; paused && now.Before(t) {
|
||||
continue
|
||||
}
|
||||
starting = append(starting, dir)
|
||||
}
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
sort.Strings(starting)
|
||||
ready := starting[:0]
|
||||
offsets := map[string]int64{}
|
||||
for _, dir := range starting {
|
||||
if settled, offset := m.offsetSettled(dir, now); settled {
|
||||
ready = append(ready, dir)
|
||||
offsets[dir] = offset
|
||||
}
|
||||
}
|
||||
for i, dir := range ready {
|
||||
if i > 0 && i%allMountsStartBatch == 0 {
|
||||
time.Sleep(allMountsStartPause)
|
||||
}
|
||||
m.noteStart(dir, mappings.Mappings[dir], offsets[dir])
|
||||
m.start(dir)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// noteStart keeps a directory's run of ends at an unmount going while it is
|
||||
// started again at the same location from the same stored offset (an end
|
||||
// saves none; a write-back that saved progress, or remote.unmount, moved
|
||||
// it), and warns once when the run reaches
|
||||
// allMountsGoneWarnAfter: a write-back that keeps replaying an unmount from
|
||||
// its saved offset ends there each time and never writes the directory back.
|
||||
// Nothing changes but the log; the operator decides (setting the directory's
|
||||
// sync offset to now skips the replay).
|
||||
func (m *mountSupervisor) noteStart(dir string, loc *remote_pb.RemoteStorageLocation, offset int64) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
streak := m.gone[dir]
|
||||
if streak == nil || !sameMountLocation(streak.loc, loc) || streak.offset != offset {
|
||||
m.gone[dir] = &goneStreak{loc: loc, offset: offset}
|
||||
return
|
||||
}
|
||||
if streak.ends >= allMountsGoneWarnAfter && !streak.warned {
|
||||
streak.warned = true
|
||||
m.warn("filer.remote.sync -allMounts: the write-back of %s has ended %d times in a row at an unmount in its replayed log while the mount table still mounts it at %+v; it is not writing the directory back. Set its sync offset to now to skip the replay", dir, streak.ends, loc)
|
||||
}
|
||||
}
|
||||
|
||||
// offsetSettled reports whether a directory may start, and its stored
|
||||
// offset: it has a saved offset, or has waited allMountsOffsetWait for one.
|
||||
// An unreadable offset waits too (the follow could not start from it
|
||||
// anyway).
|
||||
func (m *mountSupervisor) offsetSettled(dir string, now time.Time) (bool, int64) {
|
||||
offset, err := m.readOffset(dir)
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if err == nil && offset > 0 {
|
||||
delete(m.firstSeen, dir)
|
||||
return true, offset
|
||||
}
|
||||
seen, ok := m.firstSeen[dir]
|
||||
if !ok {
|
||||
m.firstSeen[dir] = now
|
||||
seen = now
|
||||
}
|
||||
if err != nil {
|
||||
glog.Errorf("filer.remote.sync -allMounts: read sync offset of %s: %v", dir, err)
|
||||
return false, 0
|
||||
}
|
||||
if now.Sub(seen) >= allMountsOffsetWait {
|
||||
glog.V(0).Infof("filer.remote.sync -allMounts: %s has no saved offset after %v; starting from its creation", dir, allMountsOffsetWait)
|
||||
delete(m.firstSeen, dir)
|
||||
return true, 0
|
||||
}
|
||||
return false, 0
|
||||
}
|
||||
|
||||
func (m *mountSupervisor) poke() {
|
||||
select {
|
||||
case m.wake <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
func (m *mountSupervisor) start(dir string) {
|
||||
f := &mountFollower{stop: make(chan struct{}), done: make(chan struct{})}
|
||||
m.mu.Lock()
|
||||
m.followers[dir] = f
|
||||
m.mu.Unlock()
|
||||
|
||||
option := *m.option
|
||||
option.clientId = m.clientIds.Add(1)
|
||||
if option.clientId == 0 { // the filer does not register client id 0
|
||||
option.clientId = m.clientIds.Add(1)
|
||||
}
|
||||
option.clientEpoch = 0
|
||||
watch := &mountWatch{stop: f.stop}
|
||||
|
||||
go func() {
|
||||
defer close(f.done)
|
||||
glog.V(0).Infof("filer.remote.sync -allMounts: synchronize %s to remote storage", dir)
|
||||
began := m.now()
|
||||
wait := allMountsRetryMin
|
||||
for {
|
||||
attempt := time.Now()
|
||||
err := m.follow(&option, dir, watch)
|
||||
if errors.Is(err, errFollowStopped) || isClosed(f.stop) {
|
||||
m.forget(dir, f)
|
||||
return
|
||||
}
|
||||
if errors.Is(err, errMountGone) {
|
||||
m.mu.Lock()
|
||||
if streak := m.gone[dir]; streak != nil {
|
||||
streak.ends++
|
||||
}
|
||||
m.mu.Unlock()
|
||||
// ended where the -dir process exits; the next reconcile
|
||||
// starts it again if the live table still mounts it
|
||||
m.ended(dir, f, m.now().Sub(began))
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
glog.Errorf("synchronize %s: %v", dir, err)
|
||||
}
|
||||
// no stream is running here: a directory the live table no longer
|
||||
// mounts is retired, and started again if it is mounted again
|
||||
if !m.stillMounted(dir) {
|
||||
m.forget(dir, f)
|
||||
return
|
||||
}
|
||||
if time.Since(attempt) > allMountsRetryMax {
|
||||
wait = allMountsRetryMin
|
||||
}
|
||||
select {
|
||||
case <-f.stop:
|
||||
m.forget(dir, f)
|
||||
return
|
||||
case <-time.After(wait):
|
||||
}
|
||||
if wait *= 2; wait > allMountsRetryMax {
|
||||
wait = allMountsRetryMax
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// ended records a write-back that ended itself: its directory waits before
|
||||
// its next start, longer each time it ends again within a minute.
|
||||
func (m *mountSupervisor) ended(dir string, f *mountFollower, ran time.Duration) {
|
||||
m.mu.Lock()
|
||||
if m.followers[dir] == f {
|
||||
delete(m.followers, dir)
|
||||
}
|
||||
if ran > time.Minute {
|
||||
m.ends[dir] = 0
|
||||
}
|
||||
pause := allMountsRetryMin << m.ends[dir]
|
||||
if pause > allMountsRetryMax || pause <= 0 {
|
||||
pause = allMountsRetryMax
|
||||
} else {
|
||||
m.ends[dir]++
|
||||
}
|
||||
m.notBefore[dir] = m.now().Add(pause)
|
||||
m.mu.Unlock()
|
||||
m.poke()
|
||||
}
|
||||
|
||||
// stillMounted reports whether the live table maps dir; an unreadable table
|
||||
// counts as mapped.
|
||||
func (m *mountSupervisor) stillMounted(dir string) bool {
|
||||
mappings, err := m.readMappings()
|
||||
return err != nil || (mappings != nil && mappings.Mappings[dir] != nil)
|
||||
}
|
||||
|
||||
func (m *mountSupervisor) forget(dir string, f *mountFollower) {
|
||||
m.mu.Lock()
|
||||
if m.followers[dir] == f {
|
||||
delete(m.followers, dir)
|
||||
}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
|
||||
func (m *mountSupervisor) stopAll() {
|
||||
m.mu.Lock()
|
||||
var all []*mountFollower
|
||||
for _, f := range m.followers {
|
||||
close(f.stop)
|
||||
all = append(all, f)
|
||||
}
|
||||
m.mu.Unlock()
|
||||
for _, f := range all {
|
||||
<-f.done
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user