Commit Graph
7 Commits
Author SHA1 Message Date
0d3f5a220c s3: keep serving for -preStopSeconds after SIGTERM (#12703)
* s3: keep serving for -preStopSeconds after SIGTERM

On SIGTERM the gateway closed its listeners at once, so requests a load
balancer or Kubernetes Service still routed to it failed until the endpoint
was removed. With -preStopSeconds set, the gateway makes /status, /healthz
and /readyz return 503, keeps serving for that many seconds, then runs the
existing graceful shutdown. 0 (the default) keeps the old behavior.

The flag is registered for weed s3 and as -s3.preStopSeconds for weed
filer, weed server and weed mini. A cancelled weed mini cluster context
ends the wait early. The Helm chart passes s3.preStopSeconds and
filer.s3.preStopSeconds through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: assert the preStopSeconds chart render under set -e, drop timing from the probe test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: wait before weed mini cancels its clients; size grace periods in the chart

weed mini's interrupt hook runs before the S3 one and cancels the S3
gateway's shutdownCtx, so the pre-stop wait never happened there. The wait
is now also registered with beforeMiniClientsShutdown, which runs before
that cancel, and a sync.OnceFunc keeps the interrupt hook from waiting a
second time. Outside weed mini the registration is a no-op.

Helm: the s3 deployment fails to render when terminationGracePeriodSeconds
is below preStopSeconds + 15. The filer statefulset and the all-in-one
deployment, whose grace period is fixed at 60s, add preStopSeconds to it.
allInOne.s3.preStopSeconds is passed through, inheriting s3.preStopSeconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: keep 64-bit counters aligned on 386; let allInOne.s3.preStopSeconds=0 disable the wait

draining sat ahead of inFlightDataSize and inFlightUploads in S3ApiServer
and shifted them off 8-byte alignment on 32-bit builds, so the atomic
upload-limit counters panicked there. It now follows them.

Helm: allInOne.s3.preStopSeconds inherits s3.preStopSeconds only when null;
default treated an explicit 0 as unset. The s3 values and the grace period
error now say a lifecycle preStop hook's time comes on top.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grace: add OnInterruptFirst for hooks that must run before any shutdown

Interrupt hooks run in reverse registration order, so a hook cannot run
ahead of a service that registers later, such as a volume server that is
still loading its volumes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: run the -preStopSeconds wait before every other interrupt hook

Under weed server the volume server registers its hook once its volumes
are loaded, often after the S3 gateway. Its hook then ran first and shut
the volume server down, so the gateway spent the whole wait serving
requests that could not read or write objects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-17 17:06:06 -08:00
Chris Lu 8236c8d136 vector: integrate engine health, admin management, and local deployment (#12517)
* vector: expose request metrics and registration-aware health endpoints

* vector: report immutable read cache and storage fetch metrics

* docs: describe vector engine health, metrics, and maintenance

* vector: count canceled requests when their handler is dropped

* vector: account for uncached large immutable storage reads

* vector: reliably release the stalled credential reader in tests

* vector: observe large reads as their streamed body is consumed

* vector: bound heartbeats on established master registration streams

* test: exercise hardened vector maintenance on Go and Rust volumes

* vector: register engines in their configured filer group

* vector: make retried index creation preserve compatible datasets

* vector: allow registration to advertise a separate address

* test: release stalled credential readers without blocking FIFO cleanup

* test: allow hardened vector coverage in a named filer group

* vector: discover engines in the same filer group as their gateway

* vector: fall back when the advertised address is empty

* admin: expose vector readiness, maintenance, and management tools (#12520)

* s3vectors: share vector discovery with trusted management clients

* admin: show vector engine readiness and recent maintenance activity

* admin: suggest S3 Vectors IAM actions from the supported operations

* admin: manage vector indexes, policies, tags, encryption, and query data

* admin: validate SSE-S3 support before creating encrypted vector buckets

* admin: discover vector engines in their registration group

* admin: probe vector engines concurrently with independent time budgets

* admin: verify encryption capability on the remote S3 gateway

* admin: defer discovery until after the HTTP server can start

* admin: distinguish unknown vector health services from failed engines

* admin: count workers that can execute vector index maintenance

* admin: allow read-only users to query vectors and inspect tags

* vector: scope gateway and admin discovery to the configured filer group

* admin: refresh policy state after mutations and serialize policy controls

* vector: drain active RPCs before closing replaced discovery connections

* vector: respect request cancellation while waiting for discovery

* vector: prefer another registered engine after a transport failure

* admin: check all masters before reporting no registered vector engines

* iam: honor attached vector policies without broadening legacy grants

* vector: allow short-lived discovery clients to close their connections

* vector: refresh cached master addresses for each discovery lookup

* iam: enforce vector bucket tag conditions in attached policies

* iam: authorize vector bucket creation with the requested tags

* test: exercise vector IAM grants and tag denies on Go and Rust volumes

* admin: reject authenticated vector mutations without an admin role

* test: run vector IAM assertions before the shared engine is stopped

* vector: add native mini startup, shell tools, and a runnable Compose example (#12522)

* s3vectors: share vector discovery with trusted management clients

* admin: show vector engine readiness and recent maintenance activity

* admin: suggest S3 Vectors IAM actions from the supported operations

* admin: manage vector indexes, policies, tags, encryption, and query data

* admin: validate SSE-S3 support before creating encrypted vector buckets

* admin: discover vector engines in their registration group

* admin: probe vector engines concurrently with independent time budgets

* admin: verify encryption capability on the remote S3 gateway

* admin: defer discovery until after the HTTP server can start

* admin: distinguish unknown vector health services from failed engines

* admin: count workers that can execute vector index maintenance

* admin: allow read-only users to query vectors and inspect tags

* vector: scope gateway and admin discovery to the configured filer group

* admin: refresh policy state after mutations and serialize policy controls

* vector: drain active RPCs before closing replaced discovery connections

* vector: respect request cancellation while waiting for discovery

* vector: prefer another registered engine after a transport failure

* admin: check all masters before reporting no registered vector engines

* iam: honor attached vector policies without broadening legacy grants

* vector: allow short-lived discovery clients to close their connections

* vector: refresh cached master addresses for each discovery lookup

* iam: enforce vector bucket tag conditions in attached policies

* iam: authorize vector bucket creation with the requested tags

* test: exercise vector IAM grants and tag denies on Go and Rust volumes

* admin: reject authenticated vector mutations without an admin role

* test: run vector IAM assertions before the shared engine is stopped

* mini: optionally start and supervise the vector engine

* mini: bootstrap vector buckets with existing Lance catalog helpers

* shell: manage vector buckets, indexes, and vector data

* docs: provide a runnable vector and maintenance worker Compose stack

* shell: scope vector discovery to the session and close its connections

* mini: honor the bind address for vector and metrics listeners

* mini: reserve vector ports before assigning gRPC and compatibility listeners

* mini: wait for vector storage setup and master registration

* mini: create requested vector buckets without requiring the Lance HTTP listener

* shell: refresh master addresses during vector discovery

* mini: resolve sibling vector executables with platform extensions

* vector: preserve uncertain creates and fix mini listener allocation (#12525)

* vector: retain catalog records when index creation has an unknown outcome

* mini: keep chosen listener ports reserved throughout allocation

* mini: resolve hostname binds before starting vector listeners

* vector: fix integration test imports and recover interrupted index creation (#12533)

* shell: register vector operations from the command package

* s3vectors: resume interrupted index initialization safely

* mini: exclude disabled WebDAV from listener allocation

* s3vectors: retain uncertain creates across refused retries

* s3vectors: claim pending indexes before retrying initialization

* s3vectors: compare resolved distance metrics during recovery

* s3vectors: preserve metadata keys when retrying initialization

* s3vectors: verify ownership before initializing new catalog records
2026-11-10 04:21:18 -08:00
promalertandChris Lu 9012069bd7 chore: execute goimports to format the code (#7983)
* chore: execute goimports to format the code

Signed-off-by: promalert <promalert@outlook.com>

* goimports -w .

---------

Signed-off-by: promalert <promalert@outlook.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-01-07 13:06:08 -08:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Konstantin Lebedev 969f513265 disable audit log for s3 statusHandler 2021-12-13 16:08:19 +05:00
Chris Lu 006b4ec0f8 refactoring 2021-10-31 18:02:08 -07:00
Bl1tz23 e6e57db530 Add liveness\readiness probe for s3 api handler on /status path 2021-08-10 13:42:46 +03:00