Commit Graph
17 Commits
Author SHA1 Message Date
0d3f5a220c s3: keep serving for -preStopSeconds after SIGTERM (#12703)
* s3: keep serving for -preStopSeconds after SIGTERM

On SIGTERM the gateway closed its listeners at once, so requests a load
balancer or Kubernetes Service still routed to it failed until the endpoint
was removed. With -preStopSeconds set, the gateway makes /status, /healthz
and /readyz return 503, keeps serving for that many seconds, then runs the
existing graceful shutdown. 0 (the default) keeps the old behavior.

The flag is registered for weed s3 and as -s3.preStopSeconds for weed
filer, weed server and weed mini. A cancelled weed mini cluster context
ends the wait early. The Helm chart passes s3.preStopSeconds and
filer.s3.preStopSeconds through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: assert the preStopSeconds chart render under set -e, drop timing from the probe test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: wait before weed mini cancels its clients; size grace periods in the chart

weed mini's interrupt hook runs before the S3 one and cancels the S3
gateway's shutdownCtx, so the pre-stop wait never happened there. The wait
is now also registered with beforeMiniClientsShutdown, which runs before
that cancel, and a sync.OnceFunc keeps the interrupt hook from waiting a
second time. Outside weed mini the registration is a no-op.

Helm: the s3 deployment fails to render when terminationGracePeriodSeconds
is below preStopSeconds + 15. The filer statefulset and the all-in-one
deployment, whose grace period is fixed at 60s, add preStopSeconds to it.
allInOne.s3.preStopSeconds is passed through, inheriting s3.preStopSeconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: keep 64-bit counters aligned on 386; let allInOne.s3.preStopSeconds=0 disable the wait

draining sat ahead of inFlightDataSize and inFlightUploads in S3ApiServer
and shifted them off 8-byte alignment on 32-bit builds, so the atomic
upload-limit counters panicked there. It now follows them.

Helm: allInOne.s3.preStopSeconds inherits s3.preStopSeconds only when null;
default treated an explicit 0 as unset. The s3 values and the grace period
error now say a lifecycle preStop hook's time comes on top.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grace: add OnInterruptFirst for hooks that must run before any shutdown

Interrupt hooks run in reverse registration order, so a hook cannot run
ahead of a service that registers later, such as a volume server that is
still loading its volumes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: run the -preStopSeconds wait before every other interrupt hook

Under weed server the volume server registers its hook once its volumes
are loaded, often after the S3 gateway. Its hook then ran first and shut
the volume server down, so the gateway spent the whole wait serving
requests that could not read or write objects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-17 17:06:06 -08:00
Chris Lu 9d15705c16 fix(mini): shut down admin/s3/webdav/filer before volume/master on Ctrl+C (#9112)
* fix(mini): shut down admin/s3/webdav/filer before volume/master on Ctrl+C

Interrupts fired grace hooks in registration order, so master (started
first) shut down before its clients, producing heartbeat-canceled errors
and masterClient reconnection noise during weed mini shutdown. Admin/s3/
webdav had no interrupt hooks at all and were killed at os.Exit.

- grace: execute interrupt hooks in LIFO (defer-style) order so later-
  started services tear down first.
- filer: consolidate the three separate interrupt hooks (gRPC / HTTP /
  DB) into one that runs in order, so filer shutdown stays correct
  independent of FIFO/LIFO semantics.
- mini: add MiniClientsShutdownCtx (separate from test-facing
  MiniClusterCtx) plus an OnMiniClientsShutdown helper. Admin, S3,
  WebDAV and the maintenance worker observe it; runMini registers a
  cancel hook after startup so under LIFO it fires first and waits up to
  10s on a WaitGroup for those services to drain before filer, volume,
  and master shut down.

Resulting order on Ctrl+C: admin/s3/webdav/worker -> filer (gRPC -> HTTP
-> DB) -> volume -> master.

* refactor(mini): group mini-client shutdown into one state struct

The first pass spread the shutdown plumbing across three globals
(MiniClientsShutdownCtx, miniClientsWg, cancelMiniClients) and two
ctx-derivation sites (OnMiniClientsShutdown and startMiniAdminWithWorker).

Group into a private miniClientsState (ctx/cancel/wg) rebuilt per runMini
invocation, and chain its ctx from MiniClusterCtx so clients only observe
one signal. Tests that cancel MiniClusterCtx still trigger client
shutdown via parent-child propagation.

- resetMiniClients() installs fresh state at the top of runMini, so
  in-process test reruns don't inherit stale ctx/wg.
- onMiniClientsShutdown(fn) replaces the exported OnMiniClientsShutdown
  and only observes one ctx.
- trackMiniClient() replaces the manual wg.Add/Done dance for the admin
  goroutine.
- miniClientsCtx() gives the admin startup a ctx without re-deriving.
- triggerMiniClientsShutdown(timeout) is the interrupt hook body.

No behaviour change; existing tests pass.

* refactor: generalize shutdown ctx as an option, not a mini-specific helper

Several service files (s3, webdav, filer, master, volume) observed the
mini-specific MiniClusterCtx or called onMiniClientsShutdown directly.
That leaked mini orchestration into code that also runs under weed s3,
weed webdav, weed filer, weed master, and weed volume standalone.

Replace with a generic `shutdownCtx context.Context` field on each
service's Options struct. When non-nil, the server watches it and shuts
down gracefully; when nil (standalone), the shutdown path is a no-op.

Mini wires the contexts up from a single place (runMini):
 - miniMasterOptions/miniOptions.v/miniFilerOptions.shutdownCtx =
   MiniClusterCtx (drives test-triggered teardown)
 - miniS3Options/miniWebDavOptions.shutdownCtx = miniClientsCtx() (drives
   Ctrl+C teardown before filer/volume/master)

All knowledge of MiniClusterCtx now lives in mini.go.

* fix(mini): stop worker before clients ctx so admin shutdown isn't blocked

Symptom on Ctrl+C of a clean weed mini: mini's Shutting down admin/s3/
webdav hook sat for 10s then logged "timed out". Admin had started its
shutdown but was blocked inside StopWorkerGrpcServer's GracefulStop,
waiting for the still-connected worker stream. That in turn left filer
clients connected and cascaded into filer's own 10s gRPC graceful-stop
timeout.

Two causes, both fixed:

1. worker.Stop() deadlocked on clean shutdown. It sent ActionStop (which
   makes managerLoop `break out` and exit), then called getTaskLoad()
   which sends to the same unbuffered cmd channel — no receiver, hangs
   forever. Reorder Stop() to snapshot the admin client and drain tasks
   BEFORE sending ActionStop, and call Disconnect() via the local
   snapshot afterwards.

2. Worker's taskRequestLoop raced with Disconnect(): RequestTask reads
   from c.incoming, which Disconnect closes, yielding a nil response and
   a panic on response.Message. Handle the closed channel explicitly.

3. Mini now has a preCancel phase (beforeMiniClientsShutdown) that runs
   synchronously BEFORE the clients ctx is cancelled. Register worker
   shutdown there so admin's worker-gRPC GracefulStop finds the worker
   already disconnected and returns immediately, instead of waiting on
   a stream that is about to close anyway.

Observed shutdown of a clean mini: admin/s3/webdav down in <10ms; full
process exit in ~11s (the remaining 10s is a pre-existing filer gRPC
graceful-stop timeout, not cascaded from the clients tier).

* feat(mini): cap filer gRPC graceful stop at 1s under weed mini

Full weed mini shutdown was ~11s on a clean exit, dominated by the
filer's default 10s gRPC GracefulStop timeout while background
SubscribeLocalMetadata streams drained.

Expose the timeout as a FilerOptions.gracefulStopTimeout field (default
10s for standalone weed filer) and set it to 1s in mini. Clean weed mini
shutdown now takes ~2s.
2026-04-16 16:11:01 -07:00
easonandeasonysliu a04c9c7dde fix: close CPU profile file after stopping profiling (#9000)
The file handle from os.Create(cpuProfile) was passed to
pprof.StartCPUProfile but never closed in the OnInterrupt handler.
The block and mutex profile files are correctly closed, but the
main CPU profile file was leaked.

Add f.Close() after pprof.StopCPUProfile() to prevent the file
descriptor leak.

Co-authored-by: easonysliu <easonysliu@tencent.com>
2026-04-08 22:13:02 -07:00
Chris Lu f6df7126b6 feat(admin): add profiling options for debugging high memory/CPU usage (#8923)
* feat(admin): add profiling options for debugging high memory/CPU usage

Add -debug, -debug.port, -cpuprofile, and -memprofile flags to the admin
command, matching the profiling support already available in master, volume,
and other server commands. This enables investigation of resource usage
issues like #8919.

* refactor(admin): move profiling flags into AdminOptions struct

Move cpuprofile and memprofile flags from global variables into the
AdminOptions struct and init() function for consistency with other flags.

* fix(debug): bind pprof server to localhost only and document profiling flags

StartDebugServer was binding to all interfaces (0.0.0.0), exposing
runtime profiling data to the network. Restrict to 127.0.0.1 since
this is a development/debugging tool.

Also add a "Debugging and Profiling" section to the admin command's
help text documenting the new flags.
2026-04-04 10:05:19 -07:00
Chris Lu e4b70c2521 go fix 2026-02-20 18:42:00 -08:00
promalertandChris Lu 9012069bd7 chore: execute goimports to format the code (#7983)
* chore: execute goimports to format the code

Signed-off-by: promalert <promalert@outlook.com>

* goimports -w .

---------

Signed-off-by: promalert <promalert@outlook.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-01-07 13:06:08 -08:00
Chris Luandgemini-code-assist[bot] ed1da07665 Add consistent -debug and -debug.port flags to commands (#7816)
* Add consistent -debug and -debug.port flags to commands

Add -debug and -debug.port flags to weed master, weed volume, weed s3,
weed mq.broker, and weed filer.sync commands for consistency with
weed filer.

When -debug is enabled, an HTTP server starts on the specified port
(default 6060) serving runtime profiling data at /debug/pprof/.

For mq.broker, replaced the older -port.pprof flag with the new
-debug and -debug.port pattern for consistency.

* Update weed/util/grace/pprof.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-12-18 17:44:36 -08:00
Konstantin Lebedev 401315f337 master fix interruption through ctrl+c (#3834) 2022-10-12 07:18:40 -07:00
Konstantin Lebedev 301b678147 [volume] Add new volumes to HUP(reload) signal (#3755)
Add new volumes to HUP(reload) signal
2022-09-28 12:44:13 -07:00
Konstantin Lebedev 853880bd83 [filer] DATA RACE on signal_handling fixed (#3575) 2022-09-02 06:20:19 -07:00
chrislu d1bb23645e Revert "avoid data race on grace.hooks (#3572)"
This reverts commit c37d6fc01a.
2022-09-01 13:08:34 -07:00
Konstantin Lebedev c37d6fc01a avoid data race on grace.hooks (#3572)
https://github.com/seaweedfs/seaweedfs/issues/3564
2022-09-01 10:34:06 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Chris Lu 7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
Chris Lu b624090398 go fmt 2021-07-01 01:21:14 -07:00
Chris Lu 7225cb4ac5 add block and mutex profiling 2021-06-13 16:15:54 -07:00
wuyuxiang 6850d28d6b refacotr: move signal handling and pprof to grace package 2020-04-28 14:10:23 +08:00