admin: count executor slots per job type and stop lanes waiting on capacity they cannot free (#12384)

* admin: count executor slots per job type, deduped against the heartbeat

per_worker_execution_concurrency is a per-job-type setting, but the
scheduler compared it against the worker's execution_slots_used, which
counts every job type. One s3_lifecycle job a worker reported running
after a reconnect made admin_script (per-worker 1) undispatchable on a
worker with three free slots. A reservation and the heartbeat entry for
the same job were also charged as two slots, so a per-worker limit of N
let N-1 jobs run.

Reservations now carry the job id and job type. The per-type limit is
checked against that type's work alone, the worker's slot total against
work of every type, and a heartbeat entry covered by a reservation is
counted once. Per-lane reservation pools go away: the per-type count is
what kept lanes from starving each other.

* admin: stop a lane waiting on executor capacity it cannot free

A queued job no executor could take was retried every backoff until
job_type_max_runtime ran out, up to 30 minutes, while the default lane
sat in executing and every other job type in it idled.

The dispatch loop now asks what holds the capacity. Slots held by the
lane's own jobs release when they complete, so it keeps waiting. Slots
held by other lanes or by work this admin never dispatched clear on a
schedule the lane has no view of, so the queued jobs are canceled at
once with a skipped_no_executor activity naming the worker, its slot
count and the job types holding it, and the next detection proposes
them again. A missing worker is waited for only the orphan grace.

* admin: cancel a job on its worker when the admin expires it

Expiring a job, by the stale sweep or by the operator's expire endpoint,
only failed the tracked record. The worker never heard about it and kept
running the job with its slot taken, so a job stuck in "starting" held a
slot for over a day and, with the old slot accounting, blocked a whole
lane.

Expiring a job now also sends the worker a cancel for it, so the slot
comes back on the next heartbeat.

* admin: count a completed job's heartbeat echo as its lane's own slot

The worker sends JobCompleted before the heartbeat that frees the slot, so
right after a dispatch goroutine releases its reservation the job still
shows in the worker's running work. That entry was counted as foreign, and
a lane with per-worker 1 gave up and canceled the rest of its queue after
the first job. A released reservation now stays for completedJobEchoGrace
and claims that heartbeat entry as the lane's own load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: let the admin's own jobs claim slots a worker does not itemize

The Rust worker reports only a slot count, every 10s, and the Go worker
counts a slot one heartbeat before it lists the job. Those slots were
charged on top of the reservations behind them, and once a job completed
its stale count read as foreign work, so the lane gave up its queue.
Unitemized slots now go first to this admin's unlisted reservations and
recent completions; the rest are of unknown type and count against every
type's per-worker limit, not only the slot total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: give up pinned jobs whose busy home a free worker cannot relieve

A job homed on a replica-host worker stays on that home while its queue is
shorter than LocalQueuePin, so a free remote worker cannot take it. The
outlook treated that free worker as a reason to wait, and a home held by
foreign work parked the lane for its whole window. A free worker now only
holds the wait for the orphan grace, after which the held homes decide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: let a redispatched job's reservation replace its released one

A job the worker turns away for capacity is released, kept as an echo, and
queued again under the same id. Once the new attempt ran, both the echo and
the new reservation matched its heartbeat entry and it counted as two
slots. Reserving a job now drops what an earlier reservation of it left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: credit a count-only slot to a completed job only until the next heartbeat

A released reservation claimed an unitemized slot for its full 30s grace,
so on a count-only heartbeat other work of another type was charged to the
completed job's type. A heartbeat that arrives after the release already
reflects the freed slot; from then on the slot is of unknown type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: name unitemized slots in the no-capacity reason

holders() listed only the job types the worker itemized, so a worker with
named and count-only work read as "2/2 slots used by <type>".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep the blocked clock running until a reservation succeeds

A popped job whose reservation then failed reset the clock before the
failure pushed it back, so pop-then-fail cycles never aged the block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: resend an expiry cancel the worker did not receive

Expiring a job sends its worker one cancel. When the worker is
disconnected, or its outgoing buffer stays full past the send timeout,
that cancel is dropped. The record is already failed, so later sweeps
and expire calls skip it, and a worker that kept the job running (a
detached job across a reconnect, or a jammed stream) holds the slot
until the job ends on its own.

The admin now holds an undelivered expiry cancel and sends it again on
the worker's next heartbeat that still lists the job. It is forgotten
when that heartbeat no longer lists the job, and skipped when the job is
active again, since a new attempt reuses the job id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: cancel an expired attempt by its request id

The expiry cancel named the job id, and a retry reuses it. The worker
keys its cancel funcs by both ids, so a cancel arriving after a retry
started on the same worker stopped the retry. The held resend also gave
up on any job that was active again, and the expired attempt's own late
progress marks the job running, so a worker that buffered progress
while disconnected kept the expired job.

The cancel now names the attempt's request id, as the other execution
cancels do, and the held resend no longer depends on the tracked state.
Only a cancel that can name nothing but the job id still skips a job
that is active again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: ignore late reports from an attempt the admin already ended

Expiring a job now cancels it on its worker, and the worker answers
with its own JobCompleted for that attempt. That overwrote the expiry's
failure with the cancel error and logged a second run record under the
same run id. Late progress from the expired attempt likewise reopened
the record as running.

A progress update or completion whose request id names an attempt the
record already closed is now dropped after the dispatcher's waiter is
told. A retry runs under a new request id, so its reports still apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

weed/admin/plugin/plugin_monitor_test.go

@@ -629,3 +629,275 @@ func TestBuildJobDetailLoadsFromDiskWhenMemoryCleared(t *testing.T) {
 		t.Fatalf("unexpected disk detail payload: %q", detail.Job.Detail)
 	}
 }
+
+func expectCancelFor(t *testing.T, session *streamSession, target string) {
+	t.Helper()
+	select {
+	case msg := <-session.outgoing:
+		cancel := msg.GetCancelRequest()
+		if cancel == nil || cancel.TargetId != target || cancel.TargetKind != plugin_pb.WorkKind_WORK_KIND_EXECUTION {
+			t.Fatalf("expected a cancel for %s, got %+v", target, msg)
+		}
+	case <-time.After(time.Second):
+		t.Fatalf("expected the worker to be told to stop %s", target)
+	}
+}
+
+// Expiring a job reaches the worker too: writing a job off in the admin
+// while the worker keeps running it would leave the slot taken.
+func TestExpireJobCancelsTheJobOnItsWorker(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	const workerID = "worker-1"
+	const jobID = "job-operator-expired"
+	session := &streamSession{workerID: workerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 8), done: make(chan struct{})}
+	pluginSvc.putSession(session)
+	pluginSvc.trackExecutionStart("req-operator-expired", workerID, &plugin_pb.JobSpec{JobId: jobID, JobType: "s3_lifecycle"}, 1)
+
+	updated, expired, err := pluginSvc.ExpireJob(jobID, "")
+	if err != nil || !expired {
+		t.Fatalf("ExpireJob: expired=%v err=%v", expired, err)
+	}
+	if updated == nil || updated.State != StateFailed {
+		t.Fatalf("expected the job to be failed, got %+v", updated)
+	}
+	expectCancelFor(t, session, "req-operator-expired")
+}
+
+// The stale sweep sends the same cancel, so the worker's slot comes back
+// on its next heartbeat rather than staying taken by a job the admin
+// already wrote off.
+func TestExpireStaleJobsCancelsTheJobOnItsWorker(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	const workerID = "worker-1"
+	const jobID = "job-stale"
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{WorkerId: workerID})
+	pluginSvc.registry.UpdateHeartbeat(workerID, &plugin_pb.WorkerHeartbeat{WorkerId: workerID, ExecutionSlotsTotal: 4})
+	session := &streamSession{workerID: workerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 8), done: make(chan struct{})}
+	pluginSvc.putSession(session)
+	pluginSvc.trackExecutionStart("req-stale", workerID, &plugin_pb.JobSpec{JobId: jobID, JobType: "s3_lifecycle"}, 1)
+
+	startedAt := time.Now().UTC()
+	if n := pluginSvc.expireStaleJobs(startedAt.Add(defaultStaleActiveJobTimeout - time.Minute)); n != 0 {
+		t.Fatalf("expired %d jobs before the stale limit", n)
+	}
+	if n := pluginSvc.expireStaleJobs(startedAt.Add(defaultStaleActiveJobTimeout + time.Minute)); n != 1 {
+		t.Fatalf("expired %d jobs past the stale limit, want 1", n)
+	}
+	got, _ := pluginSvc.GetTrackedJob(jobID)
+	if got == nil || got.State != StateFailed {
+		t.Fatalf("expected the stale job to be failed, got %+v", got)
+	}
+	expectCancelFor(t, session, "req-stale")
+}
+
+func expectNoCancel(t *testing.T, session *streamSession) {
+	t.Helper()
+	select {
+	case msg := <-session.outgoing:
+		t.Fatalf("expected no message, got %+v", msg)
+	case <-time.After(100 * time.Millisecond):
+	}
+}
+
+func heartbeatListing(workerID string, jobIDs ...string) *plugin_pb.WorkerToAdminMessage {
+	heartbeat := &plugin_pb.WorkerHeartbeat{WorkerId: workerID, ExecutionSlotsTotal: 4}
+	for _, jobID := range jobIDs {
+		heartbeat.RunningWork = append(heartbeat.RunningWork, &plugin_pb.RunningWork{
+			WorkId:  jobID,
+			Kind:    plugin_pb.WorkKind_WORK_KIND_EXECUTION,
+			JobType: "s3_lifecycle",
+			State:   plugin_pb.JobState_JOB_STATE_RUNNING,
+		})
+	}
+	return &plugin_pb.WorkerToAdminMessage{Body: &plugin_pb.WorkerToAdminMessage_Heartbeat{Heartbeat: heartbeat}}
+}
+
+// A cancel the worker could not receive when its job expired goes out with
+// the worker's next heartbeat that still lists the job, once.
+func TestExpireJobResendsAnUndeliveredCancelOnHeartbeat(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	const workerID = "worker-1"
+	const jobID = "job-expired-while-away"
+	pluginSvc.trackExecutionStart("req-away", workerID, &plugin_pb.JobSpec{JobId: jobID, JobType: "s3_lifecycle"}, 1)
+	if _, expired, err := pluginSvc.ExpireJob(jobID, ""); err != nil || !expired {
+		t.Fatalf("ExpireJob: expired=%v err=%v", expired, err)
+	}
+
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{WorkerId: workerID})
+	session := &streamSession{workerID: workerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 8), done: make(chan struct{})}
+	pluginSvc.putSession(session)
+
+	pluginSvc.handleWorkerMessage(session, heartbeatListing(workerID, jobID))
+	expectCancelFor(t, session, "req-away")
+
+	pluginSvc.handleWorkerMessage(session, heartbeatListing(workerID, jobID))
+	expectNoCancel(t, session)
+}
+
+// A held cancel names the expired attempt, so it neither stops a retry
+// that reuses the job id nor gets lost when the expired attempt's late
+// progress marks the job running again.
+func TestUndeliveredCancelNamesTheExpiredAttempt(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	const workerID = "worker-1"
+	const retriedJobID = "job-retried"
+	const revivedJobID = "job-revived"
+	const stoppedJobID = "job-stopped"
+	pluginSvc.trackExecutionStart("req-retried-1", workerID, &plugin_pb.JobSpec{JobId: retriedJobID, JobType: "s3_lifecycle"}, 1)
+	pluginSvc.trackExecutionStart("req-revived", workerID, &plugin_pb.JobSpec{JobId: revivedJobID, JobType: "s3_lifecycle"}, 1)
+	pluginSvc.trackExecutionStart("req-stopped", workerID, &plugin_pb.JobSpec{JobId: stoppedJobID, JobType: "s3_lifecycle"}, 1)
+	for _, jobID := range []string{retriedJobID, revivedJobID, stoppedJobID} {
+		if _, expired, err := pluginSvc.ExpireJob(jobID, ""); err != nil || !expired {
+			t.Fatalf("ExpireJob(%s): expired=%v err=%v", jobID, expired, err)
+		}
+	}
+	pluginSvc.trackExecutionStart("req-retried-2", workerID, &plugin_pb.JobSpec{JobId: retriedJobID, JobType: "s3_lifecycle"}, 2)
+	pluginSvc.handleJobProgressUpdate(workerID, &plugin_pb.JobProgressUpdate{
+		JobId:     revivedJobID,
+		JobType:   "s3_lifecycle",
+		RequestId: "req-revived",
+		State:     plugin_pb.JobState_JOB_STATE_RUNNING,
+	})
+
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{WorkerId: workerID})
+	session := &streamSession{workerID: workerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 8), done: make(chan struct{})}
+	pluginSvc.putSession(session)
+
+	pluginSvc.handleWorkerMessage(session, heartbeatListing(workerID, retriedJobID, revivedJobID))
+	targets := map[string]bool{}
+	for range 2 {
+		select {
+		case msg := <-session.outgoing:
+			targets[msg.GetCancelRequest().GetTargetId()] = true
+		case <-time.After(time.Second):
+			t.Fatalf("expected two cancels, got %v", targets)
+		}
+	}
+	if !targets["req-retried-1"] || !targets["req-revived"] {
+		t.Fatalf("expected cancels for the expired attempts, got %v", targets)
+	}
+
+	pluginSvc.handleWorkerMessage(session, heartbeatListing(workerID, stoppedJobID))
+	expectNoCancel(t, session)
+}
+
+// Without a request id the cancel can only name the job, so it is held
+// back once the job is active again rather than stopping the new attempt.
+func TestUndeliveredCancelByJobIDSkipsAnActiveJob(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	const workerID = "worker-1"
+	const jobID = "job-without-request"
+	pluginSvc.trackExecutionStart("", workerID, &plugin_pb.JobSpec{JobId: jobID, JobType: "s3_lifecycle"}, 1)
+	if _, expired, err := pluginSvc.ExpireJob(jobID, ""); err != nil || !expired {
+		t.Fatalf("ExpireJob: expired=%v err=%v", expired, err)
+	}
+	pluginSvc.trackExecutionStart("", workerID, &plugin_pb.JobSpec{JobId: jobID, JobType: "s3_lifecycle"}, 2)
+
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{WorkerId: workerID})
+	session := &streamSession{workerID: workerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 8), done: make(chan struct{})}
+	pluginSvc.putSession(session)
+
+	pluginSvc.handleWorkerMessage(session, heartbeatListing(workerID, jobID))
+	expectNoCancel(t, session)
+}
+
+// Once the admin expires an attempt, that attempt's late progress and its
+// completion from the cancel neither reopen the record nor log a second run,
+// while a retry under a new request id is still recorded.
+func TestExpiredAttemptIgnoresItsLateReports(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	const workerID = "worker-1"
+	const jobID = "job-late-reports"
+	const jobType = "s3_lifecycle"
+	pluginSvc.trackExecutionStart("req-late-1", workerID, &plugin_pb.JobSpec{JobId: jobID, JobType: jobType}, 1)
+	if _, expired, err := pluginSvc.ExpireJob(jobID, "stuck"); err != nil || !expired {
+		t.Fatalf("ExpireJob: expired=%v err=%v", expired, err)
+	}
+
+	pluginSvc.handleJobProgressUpdate(workerID, &plugin_pb.JobProgressUpdate{
+		JobId:     jobID,
+		JobType:   jobType,
+		RequestId: "req-late-1",
+		State:     plugin_pb.JobState_JOB_STATE_RUNNING,
+		Stage:     "copying",
+	})
+	pluginSvc.handleJobCompleted(&plugin_pb.JobCompleted{
+		JobId:        jobID,
+		JobType:      jobType,
+		RequestId:    "req-late-1",
+		ErrorMessage: "context canceled",
+	})
+
+	got, _ := pluginSvc.GetTrackedJob(jobID)
+	if got == nil || got.State != StateFailed || got.ErrorMessage != "stuck" {
+		t.Fatalf("expected the expiry to stand, got %+v", got)
+	}
+	history, err := pluginSvc.LoadRunHistory(jobType)
+	if err != nil {
+		t.Fatalf("LoadRunHistory: %v", err)
+	}
+	if len(history.ErrorRuns) != 1 || len(history.SuccessfulRuns) != 0 {
+		t.Fatalf("expected the expiry's run record only, got %+v", history)
+	}
+
+	pluginSvc.trackExecutionStart("req-late-2", workerID, &plugin_pb.JobSpec{JobId: jobID, JobType: jobType}, 2)
+	pluginSvc.handleJobCompleted(&plugin_pb.JobCompleted{
+		JobId:     jobID,
+		JobType:   jobType,
+		RequestId: "req-late-2",
+		Success:   true,
+	})
+	got, _ = pluginSvc.GetTrackedJob(jobID)
+	if got == nil || got.State != StateSucceeded {
+		t.Fatalf("expected the retry to complete, got %+v", got)
+	}
+	history, err = pluginSvc.LoadRunHistory(jobType)
+	if err != nil {
+		t.Fatalf("LoadRunHistory: %v", err)
+	}
+	if len(history.ErrorRuns) != 1 || len(history.SuccessfulRuns) != 1 {
+		t.Fatalf("expected the retry's run to be recorded, got %+v", history)
+	}
+}