admin/plugin: fail a dispatch its restarted worker no longer lists (#12402)

* admin/plugin: fail a dispatch its restarted worker no longer lists

A job dispatched to a worker that then restarts dies with the old
process. The worker re-registers under the same id on a new stream and
its heartbeats stop listing the job, but nothing ended the pending
execution: the dispatcher kept waiting and held the executor slot until
the attempt's execution timeout.

Each pending execution now records the stream it went out on. An
itemized heartbeat from a newer stream of that worker that leaves the
job out twice in a row fails it through the expire path, so the
dispatch returns, the slot is released and the retry machinery takes
over. Jobs on the heartbeat's own stream, jobs the new stream still
lists, and heartbeats that count slots without itemizing them are left
alone.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: let a partly itemized listing clear a dispatch's misses

A heartbeat that counted more slots than it itemized returned before the
listed jobs had their misses reset, so an omission, a partial heartbeat
listing the job, and another omission failed a job the worker was still
running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: look up the worker's stream before tracking the attempt

A dispatch to a worker with no stream returned after the job was already
recorded as assigned, and that record kept new proposals with the same
dedupe key out until the stale-job sweep expired it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: fail the attempt whose request could not be sent

The dispatch now sends on the stream it looked up before tracking the
attempt, so a worker that reconnects in between makes the send fail.
That returned with the job still tracked as assigned, and with the
worker back nothing closed the record for the 24h stale timeout while
its dedupe key kept new proposals out. The attempt is now completed as
failed, after it is unregistered so the completion does not wait on a
channel nobody reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: leave a send failure to the expiry that already claimed it

The send-error path and an orphan expiry could both complete the same
attempt and record two failed runs for it. The send-error path now
completes the attempt only if no expiry has marked it failed, decided
under the same lock that unregisters it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: record a send failure whose entry Shutdown already removed

Shutdown closes shutdownCh before it unregisters pending dispatches, so a
blocked send returns while its map entry is gone. Read the expiry claim
from the dispatch's own state instead of the map lookup, so the attempt
is still failed and does not stay assigned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: match a padded job id against its trimmed listing

Heartbeat work ids are trimmed before the orphan check, so compare the
dispatch's job id trimmed too; otherwise a job id with surrounding
spaces never matches its own listing and the running job is failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: skip orphan misses once a completion was handed over

The buffer length only shows a completion the dispatcher has not taken
yet; after it takes one, the attempt stays registered until the dispatch
returns. Mark the hand-off under the pending lock and check that flag
instead, so heartbeats in that gap cannot fail a finished attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

weed/admin/plugin/plugin.go

@@ -106,7 +106,7 @@ type Plugin struct {
 	pendingDetection   map[string]*pendingDetectionState
 
 	pendingExecutionMu sync.Mutex
-	pendingExecution   map[string]chan *plugin_pb.JobCompleted
+	pendingExecution   map[string]*pendingExecutionState
 
 	pendingPreviewMu sync.Mutex
 	pendingPreview   map[string]chan *plugin_pb.ObjectPreviewResponse
@@ -216,7 +216,7 @@ func New(options Options) (*Plugin, error) {
 		sessions:                  make(map[string]*streamSession),
 		pendingSchema:             make(map[string]chan *plugin_pb.ConfigSchemaResponse),
 		pendingDetection:          make(map[string]*pendingDetectionState),
-		pendingExecution:          make(map[string]chan *plugin_pb.JobCompleted),
+		pendingExecution:          make(map[string]*pendingExecutionState),
 		pendingPreview:            make(map[string]chan *plugin_pb.ObjectPreviewResponse),
 		observations:              NewObservationStore(),
 		nextDetectionAt:           make(map[string]time.Time),
@@ -289,8 +289,8 @@ func (r *Plugin) Shutdown() {
 	r.pendingDetectionMu.Unlock()
 
 	r.pendingExecutionMu.Lock()
-	for requestID, ch := range r.pendingExecution {
-		close(ch)
+	for requestID, pending := range r.pendingExecution {
+		close(pending.completed)
 		delete(r.pendingExecution, requestID)
 	}
 	r.pendingExecutionMu.Unlock()
@@ -815,9 +815,24 @@ func (r *Plugin) executeJobWithExecutor(
 	// the single active worker, not 1/N of it.
 	clusterContext = r.decorateClusterContextForJob(clusterContext, job.JobType, adminConfigValues, int(adminRuntime.GetMaxJobsPerDetection()))
 
+	// The request goes out on this stream and no other, so a newer stream
+	// of the worker can tell it never received it.
+	session, ok := r.getSession(executor.WorkerID)
+	if !ok {
+		return nil, fmt.Errorf("worker %s is not connected", executor.WorkerID)
+	}
+
+	r.trackExecutionStart(requestID, executor.WorkerID, job, attempt)
+
 	completedCh := make(chan *plugin_pb.JobCompleted, 1)
+	pending := &pendingExecutionState{
+		completed: completedCh,
+		jobID:     job.JobId,
+		jobType:   job.JobType,
+		session:   session,
+	}
 	r.pendingExecutionMu.Lock()
-	r.pendingExecution[requestID] = completedCh
+	r.pendingExecution[requestID] = pending
 	r.pendingExecutionMu.Unlock()
 	defer func() {
 		r.pendingExecutionMu.Lock()
@@ -825,8 +840,6 @@ func (r *Plugin) executeJobWithExecutor(
 		r.pendingExecutionMu.Unlock()
 	}()
 
-	r.trackExecutionStart(requestID, executor.WorkerID, job, attempt)
-
 	message := &plugin_pb.AdminToWorkerMessage{
 		RequestId: requestID,
 		SentAt:    timestamppb.Now(),
@@ -843,7 +856,23 @@ func (r *Plugin) executeJobWithExecutor(
 		},
 	}
 
-	if err := r.sendToWorker(executor.WorkerID, message); err != nil {
+	if err := r.sendToSession(session, message); err != nil {
+		// The worker never got the request, so no report will close the
+		// attempt unless an expiry already claimed it. Unregister first:
+		// nobody reads completedCh any more. Shutdown may have done so.
+		r.pendingExecutionMu.Lock()
+		delete(r.pendingExecution, requestID)
+		claimed := pending.failed
+		r.pendingExecutionMu.Unlock()
+		if !claimed {
+			r.handleJobCompleted(&plugin_pb.JobCompleted{
+				JobId:        job.JobId,
+				JobType:      job.JobType,
+				RequestId:    requestID,
+				ErrorMessage: err.Error(),
+				CompletedAt:  timestamppb.Now(),
+			})
+		}
 		return nil, err
 	}
 
@@ -1157,6 +1186,7 @@ func (r *Plugin) handleWorkerMessage(session *streamSession, message *plugin_pb.
 			return
 		}
 		r.resendUndeliveredCancels(workerID, body.Heartbeat)
+		r.failExecutionsMissingFromHeartbeat(session, body.Heartbeat)
 	case *plugin_pb.WorkerToAdminMessage_ConfigSchemaResponse:
 		r.handleConfigSchemaResponse(body.ConfigSchemaResponse)
 	case *plugin_pb.WorkerToAdminMessage_DetectionProposals:
@@ -1501,8 +1531,12 @@ func (r *Plugin) handleJobCompleted(completed *plugin_pb.JobCompleted) {
 }
 
 func (r *Plugin) safeSendJobCompleted(requestID string, completed *plugin_pb.JobCompleted) {
+	var ch chan *plugin_pb.JobCompleted
 	r.pendingExecutionMu.Lock()
-	ch := r.pendingExecution[requestID]
+	if pending := r.pendingExecution[requestID]; pending != nil {
+		ch = pending.completed
+		pending.completionSent = true
+	}
 	r.pendingExecutionMu.Unlock()
 	safeSendCh(ch, completed, r.shutdownCh)
 }