admin: count executor slots per job type and stop lanes waiting on capacity they cannot free (#12384)

* admin: count executor slots per job type, deduped against the heartbeat

per_worker_execution_concurrency is a per-job-type setting, but the
scheduler compared it against the worker's execution_slots_used, which
counts every job type. One s3_lifecycle job a worker reported running
after a reconnect made admin_script (per-worker 1) undispatchable on a
worker with three free slots. A reservation and the heartbeat entry for
the same job were also charged as two slots, so a per-worker limit of N
let N-1 jobs run.

Reservations now carry the job id and job type. The per-type limit is
checked against that type's work alone, the worker's slot total against
work of every type, and a heartbeat entry covered by a reservation is
counted once. Per-lane reservation pools go away: the per-type count is
what kept lanes from starving each other.

* admin: stop a lane waiting on executor capacity it cannot free

A queued job no executor could take was retried every backoff until
job_type_max_runtime ran out, up to 30 minutes, while the default lane
sat in executing and every other job type in it idled.

The dispatch loop now asks what holds the capacity. Slots held by the
lane's own jobs release when they complete, so it keeps waiting. Slots
held by other lanes or by work this admin never dispatched clear on a
schedule the lane has no view of, so the queued jobs are canceled at
once with a skipped_no_executor activity naming the worker, its slot
count and the job types holding it, and the next detection proposes
them again. A missing worker is waited for only the orphan grace.

* admin: cancel a job on its worker when the admin expires it

Expiring a job, by the stale sweep or by the operator's expire endpoint,
only failed the tracked record. The worker never heard about it and kept
running the job with its slot taken, so a job stuck in "starting" held a
slot for over a day and, with the old slot accounting, blocked a whole
lane.

Expiring a job now also sends the worker a cancel for it, so the slot
comes back on the next heartbeat.

* admin: count a completed job's heartbeat echo as its lane's own slot

The worker sends JobCompleted before the heartbeat that frees the slot, so
right after a dispatch goroutine releases its reservation the job still
shows in the worker's running work. That entry was counted as foreign, and
a lane with per-worker 1 gave up and canceled the rest of its queue after
the first job. A released reservation now stays for completedJobEchoGrace
and claims that heartbeat entry as the lane's own load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: let the admin's own jobs claim slots a worker does not itemize

The Rust worker reports only a slot count, every 10s, and the Go worker
counts a slot one heartbeat before it lists the job. Those slots were
charged on top of the reservations behind them, and once a job completed
its stale count read as foreign work, so the lane gave up its queue.
Unitemized slots now go first to this admin's unlisted reservations and
recent completions; the rest are of unknown type and count against every
type's per-worker limit, not only the slot total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: give up pinned jobs whose busy home a free worker cannot relieve

A job homed on a replica-host worker stays on that home while its queue is
shorter than LocalQueuePin, so a free remote worker cannot take it. The
outlook treated that free worker as a reason to wait, and a home held by
foreign work parked the lane for its whole window. A free worker now only
holds the wait for the orphan grace, after which the held homes decide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: let a redispatched job's reservation replace its released one

A job the worker turns away for capacity is released, kept as an echo, and
queued again under the same id. Once the new attempt ran, both the echo and
the new reservation matched its heartbeat entry and it counted as two
slots. Reserving a job now drops what an earlier reservation of it left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: credit a count-only slot to a completed job only until the next heartbeat

A released reservation claimed an unitemized slot for its full 30s grace,
so on a count-only heartbeat other work of another type was charged to the
completed job's type. A heartbeat that arrives after the release already
reflects the freed slot; from then on the slot is of unknown type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: name unitemized slots in the no-capacity reason

holders() listed only the job types the worker itemized, so a worker with
named and count-only work read as "2/2 slots used by <type>".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep the blocked clock running until a reservation succeeds

A popped job whose reservation then failed reset the clock before the
failure pushed it back, so pop-then-fail cycles never aged the block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: resend an expiry cancel the worker did not receive

Expiring a job sends its worker one cancel. When the worker is
disconnected, or its outgoing buffer stays full past the send timeout,
that cancel is dropped. The record is already failed, so later sweeps
and expire calls skip it, and a worker that kept the job running (a
detached job across a reconnect, or a jammed stream) holds the slot
until the job ends on its own.

The admin now holds an undelivered expiry cancel and sends it again on
the worker's next heartbeat that still lists the job. It is forgotten
when that heartbeat no longer lists the job, and skipped when the job is
active again, since a new attempt reuses the job id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: cancel an expired attempt by its request id

The expiry cancel named the job id, and a retry reuses it. The worker
keys its cancel funcs by both ids, so a cancel arriving after a retry
started on the same worker stopped the retry. The held resend also gave
up on any job that was active again, and the expired attempt's own late
progress marks the job running, so a worker that buffered progress
while disconnected kept the expired job.

The cancel now names the attempt's request id, as the other execution
cancels do, and the held resend no longer depends on the tracked state.
Only a cancel that can name nothing but the job id still skips a job
that is active again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: ignore late reports from an attempt the admin already ended

Expiring a job now cancels it on its worker, and the worker answers
with its own JobCompleted for that attempt. That overwrote the expiry's
failure with the cancel error and logged a second run record under the
same run id. Late progress from the expired attempt likewise reopened
the record as running.

A progress update or completion whose request id names an attempt the
record already closed is now dropped after the dispatcher's waiter is
told. A retry runs under a new request id, so its reports still apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

weed/admin/plugin/plugin_monitor.go

@@ -2,6 +2,7 @@ package plugin
 
 import (
 	"encoding/json"
+	"errors"
 	"fmt"
 	"sort"
 	"strings"
@@ -143,12 +144,14 @@ func (r *Plugin) ExpireJob(jobID, reason string) (*TrackedJob, bool, error) {
 
 	var jobType string
 	var requestID string
+	var workerID string
 	active := false
 
 	r.jobsMu.RLock()
 	if tracked := r.jobs[normalizedJobID]; tracked != nil {
 		jobType = tracked.JobType
 		requestID = tracked.RequestID
+		workerID = tracked.WorkerID
 		active = isActiveTrackedJobState(tracked.State)
 	}
 	r.jobsMu.RUnlock()
@@ -163,6 +166,9 @@ func (r *Plugin) ExpireJob(jobID, reason string) (*TrackedJob, bool, error) {
 			if requestID == "" {
 				requestID = detail.RequestID
 			}
+			if workerID == "" {
+				workerID = detail.WorkerID
+			}
 			if !active && isActiveTrackedJobState(detail.State) {
 				active = true
 			}
@@ -185,14 +191,12 @@ func (r *Plugin) ExpireJob(jobID, reason string) (*TrackedJob, bool, error) {
 	}
 
 	now := time.Now().UTC()
-	r.handleJobCompleted(&plugin_pb.JobCompleted{
-		JobId:        normalizedJobID,
-		JobType:      jobType,
-		RequestId:    requestID,
-		Success:      false,
-		ErrorMessage: reason,
-		CompletedAt:  timestamppb.New(now),
-	})
+	r.expireActiveJob(activeJobSnapshot{
+		jobID:     normalizedJobID,
+		jobType:   jobType,
+		workerID:  workerID,
+		requestID: requestID,
+	}, reason, now)
 	r.appendActivity(JobActivity{
 		JobID:      normalizedJobID,
 		JobType:    jobType,
@@ -235,20 +239,108 @@ func (r *Plugin) expireStaleJobs(now time.Time) int {
 			continue
 		}
 
-		r.handleJobCompleted(&plugin_pb.JobCompleted{
-			JobId:        snap.jobID,
-			JobType:      snap.jobType,
-			RequestId:    snap.requestID,
-			Success:      false,
-			ErrorMessage: reason,
-			CompletedAt:  timestamppb.New(now),
-		})
+		r.expireActiveJob(snap, reason, now)
 		expired++
 	}
 
 	return expired
 }
 
+// expireActiveJob fails the tracked job and tells its worker to stop it.
+// Failing the record alone leaves the worker running the job and its slot
+// taken, which is what kept a stuck job's slot out of reach for days.
+func (r *Plugin) expireActiveJob(snap activeJobSnapshot, reason string, now time.Time) {
+	r.handleJobCompleted(&plugin_pb.JobCompleted{
+		JobId:        snap.jobID,
+		JobType:      snap.jobType,
+		RequestId:    snap.requestID,
+		Success:      false,
+		ErrorMessage: reason,
+		CompletedAt:  timestamppb.New(now),
+	})
+	// A retry reuses the job id, so the request id is what names this attempt.
+	target := snap.requestID
+	if target == "" {
+		target = snap.jobID
+	}
+	if err := r.sendCancel(snap.workerID, target, plugin_pb.WorkKind_WORK_KIND_EXECUTION, errors.New(reason)); err != nil {
+		r.holdUndeliveredCancel(snap.jobID, undeliveredCancel{workerID: snap.workerID, target: target, reason: reason})
+	}
+}
+
+// undeliveredCancel is an expiry cancel its worker never received: it was
+// disconnected, or its outgoing buffer stayed full past the send timeout.
+type undeliveredCancel struct {
+	workerID string
+	target   string
+	reason   string
+}
+
+// holdUndeliveredCancel keeps the cancel for the worker's next heartbeat.
+// Entries whose job is no longer tracked are dropped here, so a worker that
+// never comes back does not keep them forever.
+func (r *Plugin) holdUndeliveredCancel(jobID string, pending undeliveredCancel) {
+	r.undeliveredCancelsMu.Lock()
+	defer r.undeliveredCancelsMu.Unlock()
+	if r.undeliveredCancels == nil {
+		r.undeliveredCancels = make(map[string]undeliveredCancel)
+	}
+	r.jobsMu.RLock()
+	for id := range r.undeliveredCancels {
+		if _, tracked := r.jobs[id]; !tracked {
+			delete(r.undeliveredCancels, id)
+		}
+	}
+	r.jobsMu.RUnlock()
+	r.undeliveredCancels[jobID] = pending
+}
+
+// resendUndeliveredCancels sends a held cancel again while the worker's
+// heartbeat still lists the job, and forgets it otherwise. A cancel that
+// can only name the job id is skipped once the job is active again, since
+// it would stop a new attempt too.
+func (r *Plugin) resendUndeliveredCancels(workerID string, heartbeat *plugin_pb.WorkerHeartbeat) {
+	r.undeliveredCancelsMu.Lock()
+	var held map[string]undeliveredCancel
+	for jobID, pending := range r.undeliveredCancels {
+		if pending.workerID != workerID {
+			continue
+		}
+		delete(r.undeliveredCancels, jobID)
+		if held == nil {
+			held = make(map[string]undeliveredCancel)
+		}
+		held[jobID] = pending
+	}
+	r.undeliveredCancelsMu.Unlock()
+	if len(held) == 0 {
+		return
+	}
+
+	for _, work := range heartbeat.GetRunningWork() {
+		jobID := strings.TrimSpace(work.GetWorkId())
+		pending, ok := held[jobID]
+		if !ok || work.GetKind() != plugin_pb.WorkKind_WORK_KIND_EXECUTION {
+			continue
+		}
+		delete(held, jobID)
+		if pending.target == jobID {
+			r.jobsMu.RLock()
+			tracked := r.jobs[jobID]
+			active := tracked == nil || isActiveTrackedJobState(tracked.State)
+			r.jobsMu.RUnlock()
+			if active {
+				continue
+			}
+		}
+		go func() {
+			if err := r.sendCancel(workerID, pending.target, plugin_pb.WorkKind_WORK_KIND_EXECUTION, errors.New(pending.reason)); err != nil {
+				r.holdUndeliveredCancel(jobID, pending)
+			}
+		}()
+	}
+}
+
 func (r *Plugin) snapshotActiveJobs() []activeJobSnapshot {
 	r.jobsMu.RLock()
 	defer r.jobsMu.RUnlock()
@@ -276,6 +368,13 @@ func (r *Plugin) snapshotActiveJobs() []activeJobSnapshot {
 	return out
 }
 
+// attemptEnded reports whether the job's record already closed the attempt
+// requestID names, as an expiry does while the worker is still running it.
+// That attempt's late reports must not reopen the record or log a second run.
+func attemptEnded(job *TrackedJob, requestID string) bool {
+	return job != nil && requestID != "" && job.RequestID == requestID && !isActiveTrackedJobState(job.State)
+}
+
 func jobLastUpdated(job *TrackedJob) time.Time {
 	if job == nil {
 		return time.Time{}
@@ -692,6 +791,10 @@ func (r *Plugin) handleJobProgressUpdate(workerID string, update *plugin_pb.JobP
 	if strings.TrimSpace(update.JobId) != "" {
 		r.jobsMu.Lock()
 		job := r.jobs[update.JobId]
+		if attemptEnded(job, update.RequestId) {
+			r.jobsMu.Unlock()
+			return
+		}
 		if job == nil {
 			job = &TrackedJob{
 				JobID:     update.JobId,