2 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandClaude Opus 5.5	5dfda3645e	admin: count executor slots per job type and stop lanes waiting on capacity they cannot free (#12384) ... * admin: count executor slots per job type, deduped against the heartbeat per_worker_execution_concurrency is a per-job-type setting, but the scheduler compared it against the worker's execution_slots_used, which counts every job type. One s3_lifecycle job a worker reported running after a reconnect made admin_script (per-worker 1) undispatchable on a worker with three free slots. A reservation and the heartbeat entry for the same job were also charged as two slots, so a per-worker limit of N let N-1 jobs run. Reservations now carry the job id and job type. The per-type limit is checked against that type's work alone, the worker's slot total against work of every type, and a heartbeat entry covered by a reservation is counted once. Per-lane reservation pools go away: the per-type count is what kept lanes from starving each other. * admin: stop a lane waiting on executor capacity it cannot free A queued job no executor could take was retried every backoff until job_type_max_runtime ran out, up to 30 minutes, while the default lane sat in executing and every other job type in it idled. The dispatch loop now asks what holds the capacity. Slots held by the lane's own jobs release when they complete, so it keeps waiting. Slots held by other lanes or by work this admin never dispatched clear on a schedule the lane has no view of, so the queued jobs are canceled at once with a skipped_no_executor activity naming the worker, its slot count and the job types holding it, and the next detection proposes them again. A missing worker is waited for only the orphan grace. * admin: cancel a job on its worker when the admin expires it Expiring a job, by the stale sweep or by the operator's expire endpoint, only failed the tracked record. The worker never heard about it and kept running the job with its slot taken, so a job stuck in "starting" held a slot for over a day and, with the old slot accounting, blocked a whole lane. Expiring a job now also sends the worker a cancel for it, so the slot comes back on the next heartbeat. * admin: count a completed job's heartbeat echo as its lane's own slot The worker sends JobCompleted before the heartbeat that frees the slot, so right after a dispatch goroutine releases its reservation the job still shows in the worker's running work. That entry was counted as foreign, and a lane with per-worker 1 gave up and canceled the rest of its queue after the first job. A released reservation now stays for completedJobEchoGrace and claims that heartbeat entry as the lane's own load. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: let the admin's own jobs claim slots a worker does not itemize The Rust worker reports only a slot count, every 10s, and the Go worker counts a slot one heartbeat before it lists the job. Those slots were charged on top of the reservations behind them, and once a job completed its stale count read as foreign work, so the lane gave up its queue. Unitemized slots now go first to this admin's unlisted reservations and recent completions; the rest are of unknown type and count against every type's per-worker limit, not only the slot total. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: give up pinned jobs whose busy home a free worker cannot relieve A job homed on a replica-host worker stays on that home while its queue is shorter than LocalQueuePin, so a free remote worker cannot take it. The outlook treated that free worker as a reason to wait, and a home held by foreign work parked the lane for its whole window. A free worker now only holds the wait for the orphan grace, after which the held homes decide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: let a redispatched job's reservation replace its released one A job the worker turns away for capacity is released, kept as an echo, and queued again under the same id. Once the new attempt ran, both the echo and the new reservation matched its heartbeat entry and it counted as two slots. Reserving a job now drops what an earlier reservation of it left. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: credit a count-only slot to a completed job only until the next heartbeat A released reservation claimed an unitemized slot for its full 30s grace, so on a count-only heartbeat other work of another type was charged to the completed job's type. A heartbeat that arrives after the release already reflects the freed slot; from then on the slot is of unknown type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: name unitemized slots in the no-capacity reason holders() listed only the job types the worker itemized, so a worker with named and count-only work read as "2/2 slots used by <type>". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep the blocked clock running until a reservation succeeds A popped job whose reservation then failed reset the clock before the failure pushed it back, so pop-then-fail cycles never aged the block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: resend an expiry cancel the worker did not receive Expiring a job sends its worker one cancel. When the worker is disconnected, or its outgoing buffer stays full past the send timeout, that cancel is dropped. The record is already failed, so later sweeps and expire calls skip it, and a worker that kept the job running (a detached job across a reconnect, or a jammed stream) holds the slot until the job ends on its own. The admin now holds an undelivered expiry cancel and sends it again on the worker's next heartbeat that still lists the job. It is forgotten when that heartbeat no longer lists the job, and skipped when the job is active again, since a new attempt reuses the job id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: cancel an expired attempt by its request id The expiry cancel named the job id, and a retry reuses it. The worker keys its cancel funcs by both ids, so a cancel arriving after a retry started on the same worker stopped the retry. The held resend also gave up on any job that was active again, and the expired attempt's own late progress marks the job running, so a worker that buffered progress while disconnected kept the expired job. The cancel now names the attempt's request id, as the other execution cancels do, and the held resend no longer depends on the tracked state. Only a cancel that can name nothing but the job id still skips a job that is active again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: ignore late reports from an attempt the admin already ended Expiring a job now cancels it on its worker, and the worker answers with its own JobCompleted for that attempt. That overwrote the expiry's failure with the cancel error and logged a second run record under the same run id. Late progress from the expired attempt likewise reopened the record as running. A progress update or completion whose request id names an attempt the record already closed is now dropped after the dispatcher's waiter is told. A retry runs under a new request id, so its reports still apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 15:10:26 -08:00
Чепурнов СемёнandChepurnov Simon	624a13ecc6	EC worker: pin jobs to replica-host workers, late-bind local source (#11718) ... * feat: prefer same-host worker and local EC replica Admin executor pick ranked idle workers only, so a job whose volume lived on full-03 could still run on full-01 and pull the whole .dat across the LAN. Rank workers whose Hello host matches any replica first (fallback to others if the local worker is full). Detection named Sources[0] / the lowest server id; execute then SyncAndSelectBestReplica and could switch the copy source to another rack. After sync, if a replica on this machine has a complete file count, copy from that URL so affinity and pull stay on one host. * feat: pin scheduler jobs to replica-host workers until load K Reserve prefers the least-loaded replica-host worker and waits while that load is below 5 instead of spilling to a free remote worker. K=0 restores immediate spill. Waterfill uses reservation load so four idle preferred workers each get a job. * feat: park scheduled jobs on per-worker admin queues Assign a detection batch to the least-loaded replica-host worker in one pass, then let ExecutionConcurrency goroutines pop a home-runnable job or steal when that home queue reaches K. Dispatch no longer holds a job in reserve-backoff, so a free worker is not blocked by another home's pin. Tracked jobs stay pending/queued and record the home worker for the admin UI. * fix: rehome queued jobs when the home worker dies A live home still pins below K. A home missing from live executors is stolen immediately onto a free worker so a short orphaned queue does not sit until the dispatch window ends. * feat: wait before stealing a dead home worker queue Orphaned queues are held for 30s so a worker reconnect blink does not dump jobs onto other machines. After the grace, steal without K as before. Zero disables the wait. A home that comes back during grace keeps its queue. * fix: late-bind EC from to worker hostname After replicas are consistent (equal FileCount), rebind encode source to a volume server on the same hostname as worker -address. Any port on that host is local. Diverged FileCount still uses the max replica. * feat: assign homeless jobs to global FIFO Jobs whose replica hostname has no registered worker skip pin and orphan grace. Free workers take own home, then steal at K/orphan, then leftover FIFO. * fix: take homeless FIFO before steal for no-home workers Workers without a replica-host match take leftover first; steal (K/orphan) only when FIFO is empty. Home workers keep own-queue, steal, then leftover. --------- Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>	2026-10-13 05:31:34 +08:00