admin: count executor slots per job type and stop lanes waiting on capacity they cannot free (#12384)

* admin: count executor slots per job type, deduped against the heartbeat

per_worker_execution_concurrency is a per-job-type setting, but the
scheduler compared it against the worker's execution_slots_used, which
counts every job type. One s3_lifecycle job a worker reported running
after a reconnect made admin_script (per-worker 1) undispatchable on a
worker with three free slots. A reservation and the heartbeat entry for
the same job were also charged as two slots, so a per-worker limit of N
let N-1 jobs run.

Reservations now carry the job id and job type. The per-type limit is
checked against that type's work alone, the worker's slot total against
work of every type, and a heartbeat entry covered by a reservation is
counted once. Per-lane reservation pools go away: the per-type count is
what kept lanes from starving each other.

* admin: stop a lane waiting on executor capacity it cannot free

A queued job no executor could take was retried every backoff until
job_type_max_runtime ran out, up to 30 minutes, while the default lane
sat in executing and every other job type in it idled.

The dispatch loop now asks what holds the capacity. Slots held by the
lane's own jobs release when they complete, so it keeps waiting. Slots
held by other lanes or by work this admin never dispatched clear on a
schedule the lane has no view of, so the queued jobs are canceled at
once with a skipped_no_executor activity naming the worker, its slot
count and the job types holding it, and the next detection proposes
them again. A missing worker is waited for only the orphan grace.

* admin: cancel a job on its worker when the admin expires it

Expiring a job, by the stale sweep or by the operator's expire endpoint,
only failed the tracked record. The worker never heard about it and kept
running the job with its slot taken, so a job stuck in "starting" held a
slot for over a day and, with the old slot accounting, blocked a whole
lane.

Expiring a job now also sends the worker a cancel for it, so the slot
comes back on the next heartbeat.

* admin: count a completed job's heartbeat echo as its lane's own slot

The worker sends JobCompleted before the heartbeat that frees the slot, so
right after a dispatch goroutine releases its reservation the job still
shows in the worker's running work. That entry was counted as foreign, and
a lane with per-worker 1 gave up and canceled the rest of its queue after
the first job. A released reservation now stays for completedJobEchoGrace
and claims that heartbeat entry as the lane's own load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: let the admin's own jobs claim slots a worker does not itemize

The Rust worker reports only a slot count, every 10s, and the Go worker
counts a slot one heartbeat before it lists the job. Those slots were
charged on top of the reservations behind them, and once a job completed
its stale count read as foreign work, so the lane gave up its queue.
Unitemized slots now go first to this admin's unlisted reservations and
recent completions; the rest are of unknown type and count against every
type's per-worker limit, not only the slot total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: give up pinned jobs whose busy home a free worker cannot relieve

A job homed on a replica-host worker stays on that home while its queue is
shorter than LocalQueuePin, so a free remote worker cannot take it. The
outlook treated that free worker as a reason to wait, and a home held by
foreign work parked the lane for its whole window. A free worker now only
holds the wait for the orphan grace, after which the held homes decide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: let a redispatched job's reservation replace its released one

A job the worker turns away for capacity is released, kept as an echo, and
queued again under the same id. Once the new attempt ran, both the echo and
the new reservation matched its heartbeat entry and it counted as two
slots. Reserving a job now drops what an earlier reservation of it left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: credit a count-only slot to a completed job only until the next heartbeat

A released reservation claimed an unitemized slot for its full 30s grace,
so on a count-only heartbeat other work of another type was charged to the
completed job's type. A heartbeat that arrives after the release already
reflects the freed slot; from then on the slot is of unknown type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: name unitemized slots in the no-capacity reason

holders() listed only the job types the worker itemized, so a worker with
named and count-only work read as "2/2 slots used by <type>".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep the blocked clock running until a reservation succeeds

A popped job whose reservation then failed reset the clock before the
failure pushed it back, so pop-then-fail cycles never aged the block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: resend an expiry cancel the worker did not receive

Expiring a job sends its worker one cancel. When the worker is
disconnected, or its outgoing buffer stays full past the send timeout,
that cancel is dropped. The record is already failed, so later sweeps
and expire calls skip it, and a worker that kept the job running (a
detached job across a reconnect, or a jammed stream) holds the slot
until the job ends on its own.

The admin now holds an undelivered expiry cancel and sends it again on
the worker's next heartbeat that still lists the job. It is forgotten
when that heartbeat no longer lists the job, and skipped when the job is
active again, since a new attempt reuses the job id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: cancel an expired attempt by its request id

The expiry cancel named the job id, and a retry reuses it. The worker
keys its cancel funcs by both ids, so a cancel arriving after a retry
started on the same worker stopped the retry. The held resend also gave
up on any job that was active again, and the expired attempt's own late
progress marks the job running, so a worker that buffered progress
while disconnected kept the expired job.

The cancel now names the attempt's request id, as the other execution
cancels do, and the held resend no longer depends on the tracked state.
Only a cancel that can name nothing but the job id still skips a job
that is active again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: ignore late reports from an attempt the admin already ended

Expiring a job now cancels it on its worker, and the worker answers
with its own JobCompleted for that attempt. That overwrote the expiry's
failure with the cancel error and logged a second run record under the
same run id. Late progress from the expired attempt likewise reopened
the record as running.

A progress update or completion whose request id names an attempt the
record already closed is now dropped after the dispatcher's waiter is
told. A retry runs under a new request id, so its reports still apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

weed/admin/plugin/plugin.go

@@ -79,16 +79,19 @@ type Plugin struct {
 
 	lanes map[SchedulerLane]*schedulerLaneState
 
-	schedulerExecMu           sync.Mutex
-	schedulerExecReservations map[string]int
-	adminScriptRunMu          sync.RWMutex
-	schedulerDetectionMu      sync.Mutex
-	schedulerDetection        map[string]*schedulerDetectionInfo
-	schedulerRunMu            sync.Mutex
-	schedulerRun              map[string]*schedulerRunInfo
-	schedulerLoopMu           sync.Mutex
-	schedulerLoopState        schedulerLoopState
-	schedulerWakeCh           chan struct{}
+	// Execution slots this admin holds on workers, one entry per job in
+	// flight; see executorLoadLocked for how they meet the heartbeat.
+	execResMu            sync.Mutex
+	execRes              map[uint64]execReservation
+	execResSeq           uint64
+	adminScriptRunMu     sync.RWMutex
+	schedulerDetectionMu sync.Mutex
+	schedulerDetection   map[string]*schedulerDetectionInfo
+	schedulerRunMu       sync.Mutex
+	schedulerRun         map[string]*schedulerRunInfo
+	schedulerLoopMu      sync.Mutex
+	schedulerLoopState   schedulerLoopState
+	schedulerWakeCh      chan struct{}
 
 	dedupeMu           sync.Mutex
 	recentDedupeByType map[string]map[string]time.Time
@@ -115,6 +118,9 @@ type Plugin struct {
 	// serialize stale job cleanup to avoid duplicate expirations
 	staleJobsMu sync.Mutex
 
+	undeliveredCancelsMu sync.Mutex
+	undeliveredCancels   map[string]undeliveredCancel
+
 	jobDetailsMu sync.Mutex
 
 	activitiesMu sync.RWMutex
@@ -216,7 +222,7 @@ func New(options Options) (*Plugin, error) {
 		nextDetectionAt:           make(map[string]time.Time),
 		detectionInFlight:         make(map[string]bool),
 		detectorLeases:            make(map[string]string),
-		schedulerExecReservations: make(map[string]int),
+		execRes:                   make(map[uint64]execReservation),
 		schedulerDetection:        make(map[string]*schedulerDetectionInfo),
 		schedulerRun:              make(map[string]*schedulerRunInfo),
 		recentDedupeByType:        make(map[string]map[string]time.Time),
@@ -1034,9 +1040,9 @@ func (r *Plugin) clearDetectorLease(jobType string, workerID string) {
 	delete(r.detectorLeases, jobType)
 }
 
-func (r *Plugin) sendCancel(workerID, targetID string, kind plugin_pb.WorkKind, cause error) {
+func (r *Plugin) sendCancel(workerID, targetID string, kind plugin_pb.WorkKind, cause error) error {
 	if strings.TrimSpace(workerID) == "" || strings.TrimSpace(targetID) == "" {
-		return
+		return nil
 	}
 
 	requestID, err := newRequestID("cancel")
@@ -1059,9 +1065,11 @@ func (r *Plugin) sendCancel(workerID, targetID string, kind plugin_pb.WorkKind,
 			},
 		},
 	}
-	if err := r.sendToWorker(workerID, message); err != nil {
+	err = r.sendToWorker(workerID, message)
+	if err != nil {
 		glog.V(1).Infof("Plugin failed to send cancel request to worker=%s target=%s: %v", workerID, targetID, err)
 	}
+	return err
 }
 
 func (r *Plugin) sendAdminHello(session *streamSession) error {
@@ -1148,6 +1156,7 @@ func (r *Plugin) handleWorkerMessage(session *streamSession, message *plugin_pb.
 		if !r.updateRegistryFromSession(session, func() { r.registry.UpdateHeartbeat(workerID, body.Heartbeat) }) {
 			return
 		}
+		r.resendUndeliveredCancels(workerID, body.Heartbeat)
 	case *plugin_pb.WorkerToAdminMessage_ConfigSchemaResponse:
 		r.handleConfigSchemaResponse(body.ConfigSchemaResponse)
 	case *plugin_pb.WorkerToAdminMessage_DetectionProposals:
@@ -1440,6 +1449,12 @@ func (r *Plugin) handleJobCompleted(completed *plugin_pb.JobCompleted) {
 	if completed.RequestId != "" {
 		r.safeSendJobCompleted(completed.RequestId, completed)
 	}
+	r.jobsMu.RLock()
+	ended := attemptEnded(r.jobs[completed.JobId], completed.RequestId)
+	r.jobsMu.RUnlock()
+	if ended {
+		return
+	}
 
 	tracked := r.trackExecutionCompletion(completed)
 	workerID := ""