mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 15:41:15 +02:00
* fix(mini): shut down admin/s3/webdav/filer before volume/master on Ctrl+C Interrupts fired grace hooks in registration order, so master (started first) shut down before its clients, producing heartbeat-canceled errors and masterClient reconnection noise during weed mini shutdown. Admin/s3/ webdav had no interrupt hooks at all and were killed at os.Exit. - grace: execute interrupt hooks in LIFO (defer-style) order so later- started services tear down first. - filer: consolidate the three separate interrupt hooks (gRPC / HTTP / DB) into one that runs in order, so filer shutdown stays correct independent of FIFO/LIFO semantics. - mini: add MiniClientsShutdownCtx (separate from test-facing MiniClusterCtx) plus an OnMiniClientsShutdown helper. Admin, S3, WebDAV and the maintenance worker observe it; runMini registers a cancel hook after startup so under LIFO it fires first and waits up to 10s on a WaitGroup for those services to drain before filer, volume, and master shut down. Resulting order on Ctrl+C: admin/s3/webdav/worker -> filer (gRPC -> HTTP -> DB) -> volume -> master. * refactor(mini): group mini-client shutdown into one state struct The first pass spread the shutdown plumbing across three globals (MiniClientsShutdownCtx, miniClientsWg, cancelMiniClients) and two ctx-derivation sites (OnMiniClientsShutdown and startMiniAdminWithWorker). Group into a private miniClientsState (ctx/cancel/wg) rebuilt per runMini invocation, and chain its ctx from MiniClusterCtx so clients only observe one signal. Tests that cancel MiniClusterCtx still trigger client shutdown via parent-child propagation. - resetMiniClients() installs fresh state at the top of runMini, so in-process test reruns don't inherit stale ctx/wg. - onMiniClientsShutdown(fn) replaces the exported OnMiniClientsShutdown and only observes one ctx. - trackMiniClient() replaces the manual wg.Add/Done dance for the admin goroutine. - miniClientsCtx() gives the admin startup a ctx without re-deriving. - triggerMiniClientsShutdown(timeout) is the interrupt hook body. No behaviour change; existing tests pass. * refactor: generalize shutdown ctx as an option, not a mini-specific helper Several service files (s3, webdav, filer, master, volume) observed the mini-specific MiniClusterCtx or called onMiniClientsShutdown directly. That leaked mini orchestration into code that also runs under weed s3, weed webdav, weed filer, weed master, and weed volume standalone. Replace with a generic `shutdownCtx context.Context` field on each service's Options struct. When non-nil, the server watches it and shuts down gracefully; when nil (standalone), the shutdown path is a no-op. Mini wires the contexts up from a single place (runMini): - miniMasterOptions/miniOptions.v/miniFilerOptions.shutdownCtx = MiniClusterCtx (drives test-triggered teardown) - miniS3Options/miniWebDavOptions.shutdownCtx = miniClientsCtx() (drives Ctrl+C teardown before filer/volume/master) All knowledge of MiniClusterCtx now lives in mini.go. * fix(mini): stop worker before clients ctx so admin shutdown isn't blocked Symptom on Ctrl+C of a clean weed mini: mini's Shutting down admin/s3/ webdav hook sat for 10s then logged "timed out". Admin had started its shutdown but was blocked inside StopWorkerGrpcServer's GracefulStop, waiting for the still-connected worker stream. That in turn left filer clients connected and cascaded into filer's own 10s gRPC graceful-stop timeout. Two causes, both fixed: 1. worker.Stop() deadlocked on clean shutdown. It sent ActionStop (which makes managerLoop `break out` and exit), then called getTaskLoad() which sends to the same unbuffered cmd channel — no receiver, hangs forever. Reorder Stop() to snapshot the admin client and drain tasks BEFORE sending ActionStop, and call Disconnect() via the local snapshot afterwards. 2. Worker's taskRequestLoop raced with Disconnect(): RequestTask reads from c.incoming, which Disconnect closes, yielding a nil response and a panic on response.Message. Handle the closed channel explicitly. 3. Mini now has a preCancel phase (beforeMiniClientsShutdown) that runs synchronously BEFORE the clients ctx is cancelled. Register worker shutdown there so admin's worker-gRPC GracefulStop finds the worker already disconnected and returns immediately, instead of waiting on a stream that is about to close anyway. Observed shutdown of a clean mini: admin/s3/webdav down in <10ms; full process exit in ~11s (the remaining 10s is a pre-existing filer gRPC graceful-stop timeout, not cascaded from the clients tier). * feat(mini): cap filer gRPC graceful stop at 1s under weed mini Full weed mini shutdown was ~11s on a clean exit, dominated by the filer's default 10s gRPC GracefulStop timeout while background SubscribeLocalMetadata streams drained. Expose the timeout as a FilerOptions.gracefulStopTimeout field (default 10s for standalone weed filer) and set it to 1s in mini. Clean weed mini shutdown now takes ~2s.
77 lines
1.6 KiB
Go
77 lines
1.6 KiB
Go
//go:build !plan9
|
|
|
|
package grace
|
|
|
|
import (
|
|
"os"
|
|
"os/signal"
|
|
"reflect"
|
|
"runtime"
|
|
"sync"
|
|
"syscall"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/glog"
|
|
)
|
|
|
|
var signalChan chan os.Signal
|
|
var interruptHooks = make([]func(), 0)
|
|
var interruptHookLock sync.RWMutex
|
|
var reloadHooks = make([]func(), 0)
|
|
var reloadHookLock sync.RWMutex
|
|
|
|
func GetFunctionName(i interface{}) string {
|
|
return runtime.FuncForPC(reflect.ValueOf(i).Pointer()).Name()
|
|
}
|
|
|
|
func init() {
|
|
signalChan = make(chan os.Signal, 1)
|
|
signal.Notify(signalChan,
|
|
os.Interrupt,
|
|
os.Kill,
|
|
syscall.SIGALRM,
|
|
syscall.SIGHUP,
|
|
syscall.SIGINT,
|
|
syscall.SIGTERM,
|
|
// syscall.SIGQUIT,
|
|
)
|
|
go func() {
|
|
for s := range signalChan {
|
|
if s.String() == syscall.SIGHUP.String() {
|
|
reloadHookLock.RLock()
|
|
for _, hook := range reloadHooks {
|
|
hook()
|
|
}
|
|
reloadHookLock.RUnlock()
|
|
} else {
|
|
interruptHookLock.RLock()
|
|
// Execute hooks in reverse registration order (LIFO/defer-style) so
|
|
// later-started services shut down before the services they depend on.
|
|
for i := len(interruptHooks) - 1; i >= 0; i-- {
|
|
hook := interruptHooks[i]
|
|
glog.V(4).Infof("exec interrupt hook func name:%s", GetFunctionName(hook))
|
|
hook()
|
|
}
|
|
interruptHookLock.RUnlock()
|
|
os.Exit(0)
|
|
}
|
|
}
|
|
}()
|
|
}
|
|
|
|
func OnReload(fn func()) {
|
|
// prevent reentry
|
|
reloadHookLock.Lock()
|
|
defer reloadHookLock.Unlock()
|
|
reloadHooks = append(reloadHooks, fn)
|
|
}
|
|
|
|
func OnInterrupt(fn func()) {
|
|
// prevent reentry
|
|
interruptHookLock.Lock()
|
|
defer interruptHookLock.Unlock()
|
|
|
|
// deal with control+c,etc
|
|
// controlling terminal close, daemon not exit
|
|
interruptHooks = append(interruptHooks, fn)
|
|
}
|