Files
seaweedfs/weed/util/grace/signal_handling.go
T
0d3f5a220c s3: keep serving for -preStopSeconds after SIGTERM (#12703)
* s3: keep serving for -preStopSeconds after SIGTERM

On SIGTERM the gateway closed its listeners at once, so requests a load
balancer or Kubernetes Service still routed to it failed until the endpoint
was removed. With -preStopSeconds set, the gateway makes /status, /healthz
and /readyz return 503, keeps serving for that many seconds, then runs the
existing graceful shutdown. 0 (the default) keeps the old behavior.

The flag is registered for weed s3 and as -s3.preStopSeconds for weed
filer, weed server and weed mini. A cancelled weed mini cluster context
ends the wait early. The Helm chart passes s3.preStopSeconds and
filer.s3.preStopSeconds through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: assert the preStopSeconds chart render under set -e, drop timing from the probe test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: wait before weed mini cancels its clients; size grace periods in the chart

weed mini's interrupt hook runs before the S3 one and cancels the S3
gateway's shutdownCtx, so the pre-stop wait never happened there. The wait
is now also registered with beforeMiniClientsShutdown, which runs before
that cancel, and a sync.OnceFunc keeps the interrupt hook from waiting a
second time. Outside weed mini the registration is a no-op.

Helm: the s3 deployment fails to render when terminationGracePeriodSeconds
is below preStopSeconds + 15. The filer statefulset and the all-in-one
deployment, whose grace period is fixed at 60s, add preStopSeconds to it.
allInOne.s3.preStopSeconds is passed through, inheriting s3.preStopSeconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: keep 64-bit counters aligned on 386; let allInOne.s3.preStopSeconds=0 disable the wait

draining sat ahead of inFlightDataSize and inFlightUploads in S3ApiServer
and shifted them off 8-byte alignment on 32-bit builds, so the atomic
upload-limit counters panicked there. It now follows them.

Helm: allInOne.s3.preStopSeconds inherits s3.preStopSeconds only when null;
default treated an explicit 0 as unset. The s3 values and the grace period
error now say a lifecycle preStop hook's time comes on top.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grace: add OnInterruptFirst for hooks that must run before any shutdown

Interrupt hooks run in reverse registration order, so a hook cannot run
ahead of a service that registers later, such as a volume server that is
still loading its volumes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: run the -preStopSeconds wait before every other interrupt hook

Under weed server the volume server registers its hook once its volumes
are loaded, often after the S3 gateway. Its hook then ran first and shut
the volume server down, so the gateway spent the whole wait serving
requests that could not read or write objects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-17 17:06:06 -08:00

94 lines
2.1 KiB
Go

//go:build !plan9
package grace
import (
"os"
"os/signal"
"reflect"
"runtime"
"sync"
"syscall"
"github.com/seaweedfs/seaweedfs/weed/glog"
)
var signalChan chan os.Signal
var interruptHooks = make([]func(), 0)
var firstInterruptHooks = make([]func(), 0)
var interruptHookLock sync.RWMutex
var reloadHooks = make([]func(), 0)
var reloadHookLock sync.RWMutex
func GetFunctionName(i interface{}) string {
return runtime.FuncForPC(reflect.ValueOf(i).Pointer()).Name()
}
func init() {
signalChan = make(chan os.Signal, 1)
signal.Notify(signalChan,
os.Interrupt,
os.Kill,
syscall.SIGALRM,
syscall.SIGHUP,
syscall.SIGINT,
syscall.SIGTERM,
// syscall.SIGQUIT,
)
go func() {
for s := range signalChan {
if s.String() == syscall.SIGHUP.String() {
reloadHookLock.RLock()
for _, hook := range reloadHooks {
hook()
}
reloadHookLock.RUnlock()
} else {
runInterruptHooks()
os.Exit(0)
}
}
}()
}
// runInterruptHooks runs the OnInterruptFirst hooks in registration order,
// then the OnInterrupt hooks in reverse registration order (LIFO/defer-style)
// so later-started services shut down before the services they depend on.
func runInterruptHooks() {
interruptHookLock.RLock()
defer interruptHookLock.RUnlock()
for _, hook := range firstInterruptHooks {
hook()
}
for i := len(interruptHooks) - 1; i >= 0; i-- {
hook := interruptHooks[i]
glog.V(4).Infof("exec interrupt hook func name:%s", GetFunctionName(hook))
hook()
}
}
func OnReload(fn func()) {
// prevent reentry
reloadHookLock.Lock()
defer reloadHookLock.Unlock()
reloadHooks = append(reloadHooks, fn)
}
func OnInterrupt(fn func()) {
// prevent reentry
interruptHookLock.Lock()
defer interruptHookLock.Unlock()
// deal with control+c,etc
// controlling terminal close, daemon not exit
interruptHooks = append(interruptHooks, fn)
}
// OnInterruptFirst registers fn to run before every OnInterrupt hook,
// whenever those were registered.
func OnInterruptFirst(fn func()) {
interruptHookLock.Lock()
defer interruptHookLock.Unlock()
firstInterruptHooks = append(firstInterruptHooks, fn)
}