s3api: retry a transient failure when listing multipart uploads/parts
A blip on the way to the filer failed the whole ListMultipartUploads or
ListParts request. Both reported failure points sit inside one streaming
listing: the ListEntries call that opens the stream, and the stream.Recv
calls that drain it. Neither retried, so a single Unavailable answer from
a filer that was restarting turned into a 500 for the S3 client.
Replay the listing instead, bounded to three attempts with a 100ms
backoff that doubles. Only a transient failure is replayed. A not-found
answer stays authoritative so the empty-list branch still works, and
every other error still reaches the client on the first attempt.
This is scoped to (*S3ApiServer).list rather than added inside
DoSeaweedListWithSnapshot, which mount, the shell and the other object
listings share, and where a retry after a partial stream would
re-deliver entries the callback had already seen. Within one call to
list, a replay is safe: it collects into a fresh slice each time, so it
can neither duplicate nor drop entries.
That guarantee does not extend past this function. withFilerClientFailover
already re-runs its callback against the next filer on any non-NotFound
error without resetting the caller's accumulator, so on a multi-filer
gateway a mid-listing failover can itself produce a duplicated result
with err == nil, independent of this change and not fixed by it. Noted
in the PR rather than silently left for someone to rediscover.
Fixes#7221
References #7235
* admin: count plugin-runtime workers in worker metrics
The admin server keeps two worker registries: the legacy maintenance-worker
map, filled by workers registering over the worker gRPC stream, and the plugin
worker registry, filled by workers started as `weed worker`. Both the
SeaweedFS_admin_workers_connected / SeaweedFS_admin_worker_slots gauges and the
dashboard's Workers card read only the legacy map, so a cluster that runs the
admin and its workers as separate components reported 0 workers even while its
workers showed up on the plugin pages and ran scheduled jobs.
Aggregate both registries instead. The two are merged by worker ID: `weed mini`
starts both runtimes out of one working directory, so they share the persisted
worker ID and must not be counted twice. For such a worker the slot numbers
still come from the legacy registry, which keeps mini's existing readings.
Plugin workers report their slots in the heartbeat, so detection and execution
slots are summed from there; a worker that has connected but not yet sent a
heartbeat counts as connected with zero slots.
Fixes#10525
* admin: clamp negative worker-reported slot values in metrics merge
A plugin worker's self-reported heartbeat slot counts are untrusted
input; clamp them to 0 before summing so a stale or misbehaving
worker can't drive the aggregate gauge negative, matching the same
defensiveness already used in registry.go's own slot arithmetic.
Junker der Provinz