admin: make the cluster lock's fairness window configurable (#12426)

* admin: make the cluster lock's fairness window configurable

Scheduled jobs hold the cluster admin lock for their whole run. Once the
lock had been held for a minute, every new job waited until all running
jobs had released it, so with hour-long EC jobs dispatch ran in waves and
most workers sat idle between them.

-adminLock.fairnessWindow keeps the one-minute default; 0 lets new jobs
join the shared hold, and shell clients get the lock when the scheduler
next drains.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin: wait for the blocked acquire before disabling the fairness window

The wake-up test slept 50 ms before SetFairnessWindow(0), so a late
acquire that started after the change joined without exercising the
Broadcast. Wait until it is registered as a waiter instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chris LuandClaude Opus 5.5 authored and GitHub committed 2026-11-05 05:02:41 -08:00
1 parent 8c8d0f1cdf
commit c971252871
4 files changed
+110 -6

No files matched your search

+6
View File
@@ -65,6 +65,8 @@ type AdminOptions struct {
cpuProfile *string
memProfile *string
uploadMaxSizeMB *int
// nil for callers other than runAdmin; the lock keeps its default then
lockFairnessWindow *time.Duration
// workerGrpcListener, when set, is a listener already bound to grpcPort by
// the caller. `weed mini` reserves the port this way because the admin
@@ -127,6 +129,7 @@ func init() {
a.cpuProfile = cmdAdmin.Flag.String("cpuprofile", "", "cpu profile output file")
a.memProfile = cmdAdmin.Flag.String("memprofile", "", "memory profile output file")
a.uploadMaxSizeMB = cmdAdmin.Flag.Int("uploadMaxSizeMB", dash.DefaultUploadMaxSizeMB, "maximum total size in MB of one file browser upload (0 for no limit)")
a.lockFairnessWindow = cmdAdmin.Flag.Duration("adminLock.fairnessWindow", dash.DefaultAdminLockFairnessWindow, "once scheduled jobs have held the cluster admin lock this long, new jobs wait until every holder releases it so shell clients get a turn; 0 lets jobs keep joining the hold, and shell clients then get the lock when the scheduler next drains")
}
var cmdAdmin = &Command{
@@ -505,6 +508,9 @@ func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool,
if options.uploadMaxSizeMB != nil {
adminServer.SetUploadMaxSize(int64(*options.uploadMaxSizeMB) << 20)
}
if options.lockFairnessWindow != nil {
adminServer.SetAdminLockFairnessWindow(*options.lockFairnessWindow)
}
if err := adminServer.ApplyPluginConfigFromToml(util.GetViper()); err != nil {
return fmt.Errorf("apply admin.toml to plugin config: %w", err)