mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-04 11:07:35 +01:00
admin: count executor slots per job type and stop lanes waiting on capacity they cannot free (#12384)
* admin: count executor slots per job type, deduped against the heartbeat per_worker_execution_concurrency is a per-job-type setting, but the scheduler compared it against the worker's execution_slots_used, which counts every job type. One s3_lifecycle job a worker reported running after a reconnect made admin_script (per-worker 1) undispatchable on a worker with three free slots. A reservation and the heartbeat entry for the same job were also charged as two slots, so a per-worker limit of N let N-1 jobs run. Reservations now carry the job id and job type. The per-type limit is checked against that type's work alone, the worker's slot total against work of every type, and a heartbeat entry covered by a reservation is counted once. Per-lane reservation pools go away: the per-type count is what kept lanes from starving each other. * admin: stop a lane waiting on executor capacity it cannot free A queued job no executor could take was retried every backoff until job_type_max_runtime ran out, up to 30 minutes, while the default lane sat in executing and every other job type in it idled. The dispatch loop now asks what holds the capacity. Slots held by the lane's own jobs release when they complete, so it keeps waiting. Slots held by other lanes or by work this admin never dispatched clear on a schedule the lane has no view of, so the queued jobs are canceled at once with a skipped_no_executor activity naming the worker, its slot count and the job types holding it, and the next detection proposes them again. A missing worker is waited for only the orphan grace. * admin: cancel a job on its worker when the admin expires it Expiring a job, by the stale sweep or by the operator's expire endpoint, only failed the tracked record. The worker never heard about it and kept running the job with its slot taken, so a job stuck in "starting" held a slot for over a day and, with the old slot accounting, blocked a whole lane. Expiring a job now also sends the worker a cancel for it, so the slot comes back on the next heartbeat. * admin: count a completed job's heartbeat echo as its lane's own slot The worker sends JobCompleted before the heartbeat that frees the slot, so right after a dispatch goroutine releases its reservation the job still shows in the worker's running work. That entry was counted as foreign, and a lane with per-worker 1 gave up and canceled the rest of its queue after the first job. A released reservation now stays for completedJobEchoGrace and claims that heartbeat entry as the lane's own load. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: let the admin's own jobs claim slots a worker does not itemize The Rust worker reports only a slot count, every 10s, and the Go worker counts a slot one heartbeat before it lists the job. Those slots were charged on top of the reservations behind them, and once a job completed its stale count read as foreign work, so the lane gave up its queue. Unitemized slots now go first to this admin's unlisted reservations and recent completions; the rest are of unknown type and count against every type's per-worker limit, not only the slot total. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: give up pinned jobs whose busy home a free worker cannot relieve A job homed on a replica-host worker stays on that home while its queue is shorter than LocalQueuePin, so a free remote worker cannot take it. The outlook treated that free worker as a reason to wait, and a home held by foreign work parked the lane for its whole window. A free worker now only holds the wait for the orphan grace, after which the held homes decide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: let a redispatched job's reservation replace its released one A job the worker turns away for capacity is released, kept as an echo, and queued again under the same id. Once the new attempt ran, both the echo and the new reservation matched its heartbeat entry and it counted as two slots. Reserving a job now drops what an earlier reservation of it left. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: credit a count-only slot to a completed job only until the next heartbeat A released reservation claimed an unitemized slot for its full 30s grace, so on a count-only heartbeat other work of another type was charged to the completed job's type. A heartbeat that arrives after the release already reflects the freed slot; from then on the slot is of unknown type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: name unitemized slots in the no-capacity reason holders() listed only the job types the worker itemized, so a worker with named and count-only work read as "2/2 slots used by <type>". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep the blocked clock running until a reservation succeeds A popped job whose reservation then failed reset the clock before the failure pushed it back, so pop-then-fail cycles never aged the block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: resend an expiry cancel the worker did not receive Expiring a job sends its worker one cancel. When the worker is disconnected, or its outgoing buffer stays full past the send timeout, that cancel is dropped. The record is already failed, so later sweeps and expire calls skip it, and a worker that kept the job running (a detached job across a reconnect, or a jammed stream) holds the slot until the job ends on its own. The admin now holds an undelivered expiry cancel and sends it again on the worker's next heartbeat that still lists the job. It is forgotten when that heartbeat no longer lists the job, and skipped when the job is active again, since a new attempt reuses the job id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: cancel an expired attempt by its request id The expiry cancel named the job id, and a retry reuses it. The worker keys its cancel funcs by both ids, so a cancel arriving after a retry started on the same worker stopped the retry. The held resend also gave up on any job that was active again, and the expired attempt's own late progress marks the job running, so a worker that buffered progress while disconnected kept the expired job. The cancel now names the attempt's request id, as the other execution cancels do, and the held resend no longer depends on the tracked state. Only a cancel that can name nothing but the job id still skips a job that is active again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: ignore late reports from an attempt the admin already ended Expiring a job now cancels it on its worker, and the worker answers with its own JobCompleted for that attempt. That overwrote the expiry's failure with the cancel error and logged a second run record under the same run id. Late progress from the expired attempt likewise reopened the record as running. A progress update or completion whose request id names an attempt the record already closed is now dropped after the dispatcher's waiter is told. A retry runs under a new request id, so its reports still apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chris LuandClaude Opus 5.5 authored and GitHub committed
1 parent
ca15fdbff6
commit
5dfda3645e
8 files changed
+1407
-176
No files matched your search
@@ -1048,3 +1048,642 @@ func TestReserveWaterfillsPreferredHosts(t *testing.T) {
|
||||
t.Fatalf("want one job per worker, got %v", seen)
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- executor capacity ----------
|
||||
|
||||
// registerBusyWorker connects one worker that can run every job type in
|
||||
// caps with four slots, one of which the worker reports taken by running
|
||||
// work of busyType that this admin never dispatched.
|
||||
func registerBusyWorker(t *testing.T, pluginSvc *Plugin, workerID, busyType, busyJobID string, caps ...string) {
|
||||
t.Helper()
|
||||
hello := &plugin_pb.WorkerHello{WorkerId: workerID}
|
||||
for _, jobType := range caps {
|
||||
hello.Capabilities = append(hello.Capabilities, &plugin_pb.JobTypeCapability{
|
||||
JobType: jobType, CanExecute: true, MaxExecutionConcurrency: 4,
|
||||
})
|
||||
}
|
||||
pluginSvc.registry.UpsertFromHello(hello)
|
||||
heartbeat := &plugin_pb.WorkerHeartbeat{
|
||||
WorkerId: workerID,
|
||||
ExecutionSlotsUsed: 1,
|
||||
ExecutionSlotsTotal: 4,
|
||||
RunningWork: []*plugin_pb.RunningWork{{
|
||||
WorkId: busyJobID,
|
||||
Kind: plugin_pb.WorkKind_WORK_KIND_EXECUTION,
|
||||
JobType: busyType,
|
||||
State: plugin_pb.JobState_JOB_STATE_RUNNING,
|
||||
Stage: "starting",
|
||||
}},
|
||||
}
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, heartbeat)
|
||||
}
|
||||
|
||||
// The field case: the only admin_script-capable worker has 4 slots, one held
|
||||
// by an s3_lifecycle job the worker reports running that this admin never
|
||||
// dispatched, and admin_script runs at global 1 / per-worker 1.
|
||||
// per_worker_execution_concurrency bounds admin_script work, so the
|
||||
// lifecycle job must not make admin_script undispatchable; the limit still
|
||||
// holds within admin_script itself and against foreign work of the same type.
|
||||
func TestExecutorCapacityCountsOnlyTheJobTypeBeingDispatched(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
pluginSvc, err := New(Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer pluginSvc.Shutdown()
|
||||
|
||||
const workerID = "w-penso--7aa8"
|
||||
registerBusyWorker(t, pluginSvc, workerID, "s3_lifecycle", "s3_lifecycle-scheduled-1793477386659262943-0",
|
||||
"admin_script", "s3_lifecycle", "vacuum", "ec_balance")
|
||||
policy := schedulerPolicy{ExecutionConcurrency: 1, PerWorkerConcurrency: 1, LocalQueuePin: 5}
|
||||
|
||||
job := &plugin_pb.JobSpec{JobId: "admin_script-scheduled-1793738609215134308-0", JobType: "admin_script", DedupeKey: "admin_script"}
|
||||
qs := pluginSvc.assignJobsToWorkerQueues("admin_script", []*plugin_pb.JobSpec{job}, policy)
|
||||
item, target := pluginSvc.takeQueuedJob(qs, "admin_script", policy)
|
||||
if item == nil || target != workerID {
|
||||
t.Fatalf("admin_script should dispatch to %s past the foreign s3_lifecycle job, got item=%v target=%q", workerID, item, target)
|
||||
}
|
||||
executor, release, err := pluginSvc.reserveExecutorByID("admin_script", policy, target, job.JobId)
|
||||
if err != nil {
|
||||
t.Fatalf("reserve admin_script: %v", err)
|
||||
}
|
||||
defer release()
|
||||
|
||||
// The per-type limit of 1 is now spent on admin_script's own job.
|
||||
if pluginSvc.executorHasCapacity(executor, "admin_script", policy) {
|
||||
t.Fatalf("a second admin_script job must wait for the first")
|
||||
}
|
||||
// Unrelated types on the same worker still have their own per-type budget.
|
||||
if !pluginSvc.executorHasCapacity(executor, "vacuum", policy) {
|
||||
t.Fatalf("vacuum should still fit: 2 of 4 slots used, none by vacuum")
|
||||
}
|
||||
// Foreign work of the same type counts against that type's limit.
|
||||
if pluginSvc.executorHasCapacity(executor, "s3_lifecycle", policy) {
|
||||
t.Fatalf("a second s3_lifecycle job must not fit beside the foreign one at per-worker 1")
|
||||
}
|
||||
}
|
||||
|
||||
// Per-type budgets never add up to more than the worker's slot total: a
|
||||
// worker whose slots are all taken by other job types takes nothing, whether
|
||||
// it itemized that work or only reported the count.
|
||||
func TestExecutorCapacityHonorsWorkerSlotTotalAcrossJobTypes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
pluginSvc, err := New(Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer pluginSvc.Shutdown()
|
||||
|
||||
const workerID = "worker-a"
|
||||
pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
|
||||
WorkerId: workerID,
|
||||
Capabilities: []*plugin_pb.JobTypeCapability{
|
||||
{JobType: "admin_script", CanExecute: true, MaxExecutionConcurrency: 4},
|
||||
{JobType: "s3_lifecycle", CanExecute: true, MaxExecutionConcurrency: 4},
|
||||
},
|
||||
})
|
||||
policy := schedulerPolicy{ExecutionConcurrency: 1, PerWorkerConcurrency: 1}
|
||||
executor := func() *WorkerSession {
|
||||
session, ok := pluginSvc.registry.Get(workerID)
|
||||
if !ok {
|
||||
t.Fatalf("worker missing")
|
||||
}
|
||||
return session
|
||||
}
|
||||
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, &plugin_pb.WorkerHeartbeat{
|
||||
WorkerId: workerID, ExecutionSlotsUsed: 1, ExecutionSlotsTotal: 1,
|
||||
RunningWork: []*plugin_pb.RunningWork{{
|
||||
WorkId: "lc-1", Kind: plugin_pb.WorkKind_WORK_KIND_EXECUTION, JobType: "s3_lifecycle",
|
||||
}},
|
||||
})
|
||||
if pluginSvc.executorHasCapacity(executor(), "admin_script", policy) {
|
||||
t.Fatalf("the worker's only slot is taken by s3_lifecycle")
|
||||
}
|
||||
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, &plugin_pb.WorkerHeartbeat{
|
||||
WorkerId: workerID, ExecutionSlotsUsed: 1, ExecutionSlotsTotal: 1,
|
||||
})
|
||||
if pluginSvc.executorHasCapacity(executor(), "admin_script", policy) {
|
||||
t.Fatalf("a slot the worker counts as used is used even when not itemized")
|
||||
}
|
||||
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, &plugin_pb.WorkerHeartbeat{
|
||||
WorkerId: workerID, ExecutionSlotsUsed: 0, ExecutionSlotsTotal: 1,
|
||||
})
|
||||
if !pluginSvc.executorHasCapacity(executor(), "admin_script", policy) {
|
||||
t.Fatalf("a free worker should take the job")
|
||||
}
|
||||
}
|
||||
|
||||
// A job this admin dispatched shows up twice, as its reservation and in the
|
||||
// worker's heartbeat. It is one slot, and must not be charged as two.
|
||||
func TestExecutorCapacityDoesNotDoubleCountReservedJobsEchoedByHeartbeat(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
pluginSvc, err := New(Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer pluginSvc.Shutdown()
|
||||
|
||||
const workerID = "worker-a"
|
||||
pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
|
||||
WorkerId: workerID,
|
||||
Capabilities: []*plugin_pb.JobTypeCapability{
|
||||
{JobType: "vacuum", CanExecute: true, MaxExecutionConcurrency: 4},
|
||||
},
|
||||
})
|
||||
policy := schedulerPolicy{ExecutionConcurrency: 2, PerWorkerConcurrency: 2}
|
||||
|
||||
_, release, err := pluginSvc.reserveExecutorByID("vacuum", policy, workerID, "vacuum-1")
|
||||
if err != nil {
|
||||
t.Fatalf("reserve vacuum-1: %v", err)
|
||||
}
|
||||
defer release()
|
||||
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, &plugin_pb.WorkerHeartbeat{
|
||||
WorkerId: workerID, ExecutionSlotsUsed: 1, ExecutionSlotsTotal: 4,
|
||||
RunningWork: []*plugin_pb.RunningWork{{
|
||||
WorkId: "vacuum-1", Kind: plugin_pb.WorkKind_WORK_KIND_EXECUTION, JobType: "vacuum",
|
||||
}},
|
||||
})
|
||||
executor, _ := pluginSvc.registry.Get(workerID)
|
||||
if got := pluginSvc.executorWorkerLoad(executor); got != 1 {
|
||||
t.Fatalf("worker load = %d, want 1: the reservation and its heartbeat echo are one slot", got)
|
||||
}
|
||||
if !pluginSvc.executorHasCapacity(executor, "vacuum", policy) {
|
||||
t.Fatalf("one vacuum job in flight leaves room for a second at per-worker 2")
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- undispatchable jobs do not hold the lane ----------
|
||||
|
||||
// A worker whose slots are all held by work this admin never dispatched
|
||||
// will not free them on any schedule the lane can see. The run gives the
|
||||
// job up at once, says why, and leaves the window for the lane's other
|
||||
// job types instead of idling until job_type_max_runtime.
|
||||
func TestDispatchGivesUpWhenCapacityIsHeldByForeignWork(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
pluginSvc, err := New(Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer pluginSvc.Shutdown()
|
||||
|
||||
const workerID = "worker-a"
|
||||
pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
|
||||
WorkerId: workerID,
|
||||
Capabilities: []*plugin_pb.JobTypeCapability{
|
||||
{JobType: "admin_script", CanExecute: true, MaxExecutionConcurrency: 1},
|
||||
},
|
||||
})
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, &plugin_pb.WorkerHeartbeat{
|
||||
WorkerId: workerID, ExecutionSlotsUsed: 1, ExecutionSlotsTotal: 1,
|
||||
RunningWork: []*plugin_pb.RunningWork{{
|
||||
WorkId: "lc-1", Kind: plugin_pb.WorkKind_WORK_KIND_EXECUTION, JobType: "s3_lifecycle",
|
||||
}},
|
||||
})
|
||||
|
||||
window, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
policy := schedulerPolicy{
|
||||
ExecutionConcurrency: 1,
|
||||
PerWorkerConcurrency: 1,
|
||||
ExecutionTimeout: time.Second,
|
||||
ExecutorReserveBackoff: time.Millisecond,
|
||||
OrphanRehomeGrace: time.Second,
|
||||
}
|
||||
proposals := []*plugin_pb.JobProposal{{ProposalId: "p1", JobType: "admin_script", DedupeKey: "k1"}}
|
||||
|
||||
started := time.Now()
|
||||
success, errCount, canceled := pluginSvc.dispatchScheduledProposals(window, "admin_script", proposals, &plugin_pb.ClusterContext{}, policy)
|
||||
if elapsed := time.Since(started); elapsed > 5*time.Second {
|
||||
t.Fatalf("dispatch held the lane for %v with no executor able to take the job", elapsed)
|
||||
}
|
||||
if success != 0 || errCount != 0 || canceled != 1 {
|
||||
t.Fatalf("unexpected dispatch counts: success=%d errors=%d canceled=%d", success, errCount, canceled)
|
||||
}
|
||||
|
||||
activity := findActivity(pluginSvc.ListActivities("admin_script", 100), "admin_scheduler", "skipped_no_executor")
|
||||
if activity == nil {
|
||||
t.Fatalf("expected a skipped_no_executor activity, got %+v", pluginSvc.ListActivities("admin_script", 100))
|
||||
}
|
||||
want := "no executor capacity for admin_script: worker worker-a has 1/1 slots used by s3_lifecycle this admin did not dispatch"
|
||||
if activity.Message != want {
|
||||
t.Fatalf("activity message = %q, want %q", activity.Message, want)
|
||||
}
|
||||
for _, job := range pluginSvc.ListTrackedJobs("admin_script", "", 10) {
|
||||
if job.State != StateCanceled || job.Message != want {
|
||||
t.Fatalf("queued job should be canceled with the reason, got %+v", job)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Capacity held by the lane's own jobs is a different matter: they finish
|
||||
// and release, so the run waits for them as before.
|
||||
func TestDispatchWaitsForItsOwnLaneToReleaseCapacity(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
pluginSvc, err := New(Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer pluginSvc.Shutdown()
|
||||
|
||||
const workerID = "worker-a"
|
||||
pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
|
||||
WorkerId: workerID,
|
||||
Capabilities: []*plugin_pb.JobTypeCapability{
|
||||
{JobType: "admin_script", CanExecute: true, MaxExecutionConcurrency: 1},
|
||||
{JobType: "vacuum", CanExecute: true, MaxExecutionConcurrency: 1},
|
||||
},
|
||||
})
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, &plugin_pb.WorkerHeartbeat{
|
||||
WorkerId: workerID, ExecutionSlotsUsed: 0, ExecutionSlotsTotal: 1,
|
||||
})
|
||||
session := &streamSession{workerID: workerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 8), done: make(chan struct{})}
|
||||
pluginSvc.putSession(session)
|
||||
|
||||
policy := schedulerPolicy{
|
||||
ExecutionConcurrency: 1,
|
||||
PerWorkerConcurrency: 1,
|
||||
ExecutionTimeout: time.Hour,
|
||||
ExecutorReserveBackoff: time.Millisecond,
|
||||
OrphanRehomeGrace: time.Second,
|
||||
}
|
||||
// A vacuum job of the same (default) lane holds the worker's only slot.
|
||||
_, releaseVacuum, err := pluginSvc.reserveExecutorByID("vacuum", policy, workerID, "vacuum-1")
|
||||
if err != nil {
|
||||
t.Fatalf("reserve vacuum: %v", err)
|
||||
}
|
||||
|
||||
resultCh := make(chan [3]int, 1)
|
||||
go func() {
|
||||
success, errCount, canceled := pluginSvc.dispatchScheduledProposals(
|
||||
context.Background(), "admin_script", []*plugin_pb.JobProposal{{ProposalId: "p1", JobType: "admin_script", DedupeKey: "k1"}},
|
||||
&plugin_pb.ClusterContext{}, policy)
|
||||
resultCh <- [3]int{success, errCount, canceled}
|
||||
}()
|
||||
|
||||
select {
|
||||
case result := <-resultCh:
|
||||
t.Fatalf("dispatch gave up on capacity its own lane holds: %v", result)
|
||||
case <-time.After(50 * time.Millisecond):
|
||||
}
|
||||
|
||||
releaseVacuum()
|
||||
first := <-session.outgoing
|
||||
execReq := first.GetExecuteJobRequest()
|
||||
if execReq == nil {
|
||||
t.Fatalf("expected execute_job_request after the vacuum slot was released, got %+v", first)
|
||||
}
|
||||
pluginSvc.handleJobCompleted(&plugin_pb.JobCompleted{
|
||||
RequestId: first.RequestId, JobId: execReq.Job.JobId, JobType: "admin_script", Success: true, CompletedAt: timestamppb.Now(),
|
||||
})
|
||||
if result := <-resultCh; result != [3]int{1, 0, 0} {
|
||||
t.Fatalf("unexpected dispatch counts: success=%d errors=%d canceled=%d", result[0], result[1], result[2])
|
||||
}
|
||||
}
|
||||
|
||||
// With no capable worker connected the run waits out the reconnect grace,
|
||||
// then gives up rather than holding the lane for the whole window.
|
||||
func TestDispatchGivesUpWhenNoExecutorReturnsWithinGrace(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
pluginSvc, err := New(Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer pluginSvc.Shutdown()
|
||||
|
||||
window, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
policy := schedulerPolicy{
|
||||
ExecutionConcurrency: 1,
|
||||
PerWorkerConcurrency: 1,
|
||||
ExecutorReserveBackoff: time.Millisecond,
|
||||
OrphanRehomeGrace: 20 * time.Millisecond,
|
||||
}
|
||||
|
||||
started := time.Now()
|
||||
_, _, canceled := pluginSvc.dispatchScheduledProposals(window, "admin_script",
|
||||
[]*plugin_pb.JobProposal{{ProposalId: "p1", JobType: "admin_script", DedupeKey: "k1"}}, &plugin_pb.ClusterContext{}, policy)
|
||||
if elapsed := time.Since(started); elapsed < policy.OrphanRehomeGrace || elapsed > 5*time.Second {
|
||||
t.Fatalf("dispatch took %v, want about the %v grace", elapsed, policy.OrphanRehomeGrace)
|
||||
}
|
||||
if canceled != 1 {
|
||||
t.Fatalf("canceled = %d, want 1", canceled)
|
||||
}
|
||||
activity := findActivity(pluginSvc.ListActivities("admin_script", 100), "admin_scheduler", "skipped_no_executor")
|
||||
if activity == nil || activity.Message != "no executor worker available for admin_script" {
|
||||
t.Fatalf("expected the no-worker reason, got %+v", activity)
|
||||
}
|
||||
}
|
||||
|
||||
// The worker reports a job's completion before the heartbeat that frees its
|
||||
// slot, so for a moment the lane's own finished job still shows as running,
|
||||
// itemized by the Go worker and only as a slot count by the Rust one. That
|
||||
// echo is the lane's slot coming back, not foreign work.
|
||||
func TestDispatchWaitsForTheEchoOfItsOwnCompletedJob(t *testing.T) {
|
||||
t.Parallel()
|
||||
for _, itemized := range []bool{true, false} {
|
||||
itemized := itemized
|
||||
t.Run(fmt.Sprintf("itemized=%v", itemized), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
pluginSvc, err := New(Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer pluginSvc.Shutdown()
|
||||
|
||||
const workerID = "worker-a"
|
||||
pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
|
||||
WorkerId: workerID,
|
||||
Capabilities: []*plugin_pb.JobTypeCapability{
|
||||
{JobType: "vacuum", CanExecute: true, MaxExecutionConcurrency: 1},
|
||||
},
|
||||
})
|
||||
idle := &plugin_pb.WorkerHeartbeat{WorkerId: workerID, ExecutionSlotsUsed: 0, ExecutionSlotsTotal: 1}
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, idle)
|
||||
session := &streamSession{workerID: workerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 8), done: make(chan struct{})}
|
||||
pluginSvc.putSession(session)
|
||||
|
||||
policy := schedulerPolicy{
|
||||
ExecutionConcurrency: 1,
|
||||
PerWorkerConcurrency: 1,
|
||||
ExecutionTimeout: time.Hour,
|
||||
ExecutorReserveBackoff: time.Millisecond,
|
||||
OrphanRehomeGrace: time.Second,
|
||||
}
|
||||
proposals := []*plugin_pb.JobProposal{
|
||||
{ProposalId: "p1", JobType: "vacuum", DedupeKey: "k1"},
|
||||
{ProposalId: "p2", JobType: "vacuum", DedupeKey: "k2"},
|
||||
}
|
||||
resultCh := make(chan [3]int, 1)
|
||||
go func() {
|
||||
success, errCount, canceled := pluginSvc.dispatchScheduledProposals(
|
||||
context.Background(), "vacuum", proposals, &plugin_pb.ClusterContext{}, policy)
|
||||
resultCh <- [3]int{success, errCount, canceled}
|
||||
}()
|
||||
|
||||
complete := func(message *plugin_pb.AdminToWorkerMessage) {
|
||||
t.Helper()
|
||||
execReq := message.GetExecuteJobRequest()
|
||||
if execReq == nil {
|
||||
t.Fatalf("expected execute_job_request, got %+v", message)
|
||||
}
|
||||
running := &plugin_pb.WorkerHeartbeat{WorkerId: workerID, ExecutionSlotsUsed: 1, ExecutionSlotsTotal: 1}
|
||||
if itemized {
|
||||
running.RunningWork = []*plugin_pb.RunningWork{{
|
||||
WorkId: execReq.Job.JobId, Kind: plugin_pb.WorkKind_WORK_KIND_EXECUTION, JobType: "vacuum",
|
||||
}}
|
||||
}
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, running)
|
||||
pluginSvc.handleJobCompleted(&plugin_pb.JobCompleted{
|
||||
RequestId: message.RequestId, JobId: execReq.Job.JobId, JobType: "vacuum", Success: true, CompletedAt: timestamppb.Now(),
|
||||
})
|
||||
}
|
||||
|
||||
complete(<-session.outgoing)
|
||||
select {
|
||||
case result := <-resultCh:
|
||||
t.Fatalf("dispatch gave up on the echo of its own completed job: %v", result)
|
||||
case <-time.After(50 * time.Millisecond):
|
||||
}
|
||||
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, idle)
|
||||
complete(<-session.outgoing)
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, idle)
|
||||
if result := <-resultCh; result != [3]int{2, 0, 0} {
|
||||
t.Fatalf("unexpected dispatch counts: success=%d errors=%d canceled=%d", result[0], result[1], result[2])
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A worker may count a slot before it itemizes the work in it (the Go worker
|
||||
// for a moment after taking a job, the Rust worker always). The admin's own
|
||||
// jobs claim such slots first, so they are not charged twice; what is left
|
||||
// is of unknown type and holds every type's per-worker limit.
|
||||
func TestExecutorCapacityCountsUnitemizedSlots(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
pluginSvc, err := New(Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer pluginSvc.Shutdown()
|
||||
|
||||
const workerID = "worker-a"
|
||||
pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
|
||||
WorkerId: workerID,
|
||||
Capabilities: []*plugin_pb.JobTypeCapability{
|
||||
{JobType: "vacuum", CanExecute: true, MaxExecutionConcurrency: 4},
|
||||
},
|
||||
})
|
||||
executor := func() *WorkerSession {
|
||||
session, ok := pluginSvc.registry.Get(workerID)
|
||||
if !ok {
|
||||
t.Fatalf("worker missing")
|
||||
}
|
||||
return session
|
||||
}
|
||||
oneUsed := &plugin_pb.WorkerHeartbeat{WorkerId: workerID, ExecutionSlotsUsed: 1, ExecutionSlotsTotal: 4}
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, oneUsed)
|
||||
if pluginSvc.executorHasCapacity(executor(), "vacuum", schedulerPolicy{PerWorkerConcurrency: 1}) {
|
||||
t.Fatalf("a slot the worker counts but has not itemized may hold vacuum work")
|
||||
}
|
||||
|
||||
policy := schedulerPolicy{PerWorkerConcurrency: 2}
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, &plugin_pb.WorkerHeartbeat{WorkerId: workerID, ExecutionSlotsTotal: 4})
|
||||
_, release, err := pluginSvc.reserveExecutorByID("vacuum", policy, workerID, "vacuum-1")
|
||||
if err != nil {
|
||||
t.Fatalf("reserve vacuum-1: %v", err)
|
||||
}
|
||||
defer release()
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, oneUsed)
|
||||
if got := pluginSvc.executorWorkerLoad(executor()); got != 1 {
|
||||
t.Fatalf("worker load = %d, want 1: the unitemized slot is the reserved job", got)
|
||||
}
|
||||
if !pluginSvc.executorHasCapacity(executor(), "vacuum", policy) {
|
||||
t.Fatalf("one vacuum job in flight leaves room for a second at per-worker 2")
|
||||
}
|
||||
}
|
||||
|
||||
// A job homed on a replica-host worker stays pinned there while its home
|
||||
// queue is short, so a free remote worker cannot take it. When the home is
|
||||
// held by work the lane cannot free, the run gives the job up after the
|
||||
// orphan grace instead of waiting out the window beside the free worker.
|
||||
func TestDispatchGivesUpPinnedJobsWhoseHomeIsHeldByForeignWork(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
pluginSvc, err := New(Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer pluginSvc.Shutdown()
|
||||
|
||||
registerECWorkers(t, pluginSvc, map[string]string{
|
||||
"worker-local": "10.0.0.1:9334",
|
||||
"worker-remote": "10.0.0.2:9334",
|
||||
})
|
||||
pluginSvc.registry.UpdateHeartbeat("worker-local", &plugin_pb.WorkerHeartbeat{
|
||||
WorkerId: "worker-local", ExecutionSlotsUsed: 1, ExecutionSlotsTotal: 1,
|
||||
RunningWork: []*plugin_pb.RunningWork{{
|
||||
WorkId: "vac-1", Kind: plugin_pb.WorkKind_WORK_KIND_EXECUTION, JobType: "vacuum",
|
||||
}},
|
||||
})
|
||||
pluginSvc.registry.UpdateHeartbeat("worker-remote", &plugin_pb.WorkerHeartbeat{
|
||||
WorkerId: "worker-remote", ExecutionSlotsUsed: 0, ExecutionSlotsTotal: 1,
|
||||
})
|
||||
|
||||
window, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
policy := schedulerPolicy{
|
||||
ExecutionConcurrency: 1,
|
||||
PerWorkerConcurrency: 1,
|
||||
LocalQueuePin: 5,
|
||||
ExecutionTimeout: time.Hour,
|
||||
ExecutorReserveBackoff: time.Millisecond,
|
||||
OrphanRehomeGrace: 50 * time.Millisecond,
|
||||
}
|
||||
proposals := []*plugin_pb.JobProposal{{
|
||||
ProposalId: "p1", JobType: "erasure_coding", DedupeKey: "k1",
|
||||
Parameters: replicaJob("j1", "10.0.0.1:8080").Parameters,
|
||||
}}
|
||||
|
||||
started := time.Now()
|
||||
success, errCount, canceled := pluginSvc.dispatchScheduledProposals(window, "erasure_coding", proposals, &plugin_pb.ClusterContext{}, policy)
|
||||
if elapsed := time.Since(started); elapsed < policy.OrphanRehomeGrace || elapsed > 4*time.Second {
|
||||
t.Fatalf("dispatch took %v, want about the %v grace", elapsed, policy.OrphanRehomeGrace)
|
||||
}
|
||||
if success != 0 || errCount != 0 || canceled != 1 {
|
||||
t.Fatalf("unexpected dispatch counts: success=%d errors=%d canceled=%d", success, errCount, canceled)
|
||||
}
|
||||
activity := findActivity(pluginSvc.ListActivities("erasure_coding", 100), "admin_scheduler", "skipped_no_executor")
|
||||
want := "no executor capacity for erasure_coding: worker worker-local has 1/1 slots used by vacuum this admin did not dispatch"
|
||||
if activity == nil || activity.Message != want {
|
||||
t.Fatalf("expected %q, got %+v", want, activity)
|
||||
}
|
||||
}
|
||||
|
||||
// A job the worker turned away is released and queued again under the same
|
||||
// id; its new reservation replaces what the first one left behind, so the
|
||||
// running job is one slot.
|
||||
func TestExecutorCapacityCountsARedispatchedJobOnce(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
pluginSvc, err := New(Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer pluginSvc.Shutdown()
|
||||
|
||||
const workerID = "worker-a"
|
||||
pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
|
||||
WorkerId: workerID,
|
||||
Capabilities: []*plugin_pb.JobTypeCapability{
|
||||
{JobType: "vacuum", CanExecute: true, MaxExecutionConcurrency: 4},
|
||||
},
|
||||
})
|
||||
policy := schedulerPolicy{PerWorkerConcurrency: 2}
|
||||
_, rejected, err := pluginSvc.reserveExecutorByID("vacuum", policy, workerID, "vacuum-1")
|
||||
if err != nil {
|
||||
t.Fatalf("reserve vacuum-1: %v", err)
|
||||
}
|
||||
rejected()
|
||||
_, release, err := pluginSvc.reserveExecutorByID("vacuum", policy, workerID, "vacuum-1")
|
||||
if err != nil {
|
||||
t.Fatalf("reserve vacuum-1 again: %v", err)
|
||||
}
|
||||
defer release()
|
||||
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, &plugin_pb.WorkerHeartbeat{
|
||||
WorkerId: workerID, ExecutionSlotsUsed: 1, ExecutionSlotsTotal: 4,
|
||||
RunningWork: []*plugin_pb.RunningWork{{
|
||||
WorkId: "vacuum-1", Kind: plugin_pb.WorkKind_WORK_KIND_EXECUTION, JobType: "vacuum",
|
||||
}},
|
||||
})
|
||||
executor, _ := pluginSvc.registry.Get(workerID)
|
||||
if got := pluginSvc.executorWorkerLoad(executor); got != 1 {
|
||||
t.Fatalf("worker load = %d, want 1", got)
|
||||
}
|
||||
if !pluginSvc.executorHasCapacity(executor, "vacuum", policy) {
|
||||
t.Fatalf("one vacuum job in flight leaves room for a second at per-worker 2")
|
||||
}
|
||||
}
|
||||
|
||||
// A count-only heartbeat sent after a job completed no longer holds that
|
||||
// job: a slot it still counts belongs to other work, of unknown type.
|
||||
func TestExecutorCapacityStopsCreditingACompletedJobAfterTheNextHeartbeat(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
pluginSvc, err := New(Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer pluginSvc.Shutdown()
|
||||
|
||||
const workerID = "worker-a"
|
||||
pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
|
||||
WorkerId: workerID,
|
||||
Capabilities: []*plugin_pb.JobTypeCapability{
|
||||
{JobType: "vacuum", CanExecute: true, MaxExecutionConcurrency: 4},
|
||||
{JobType: "balance", CanExecute: true, MaxExecutionConcurrency: 4},
|
||||
},
|
||||
})
|
||||
policy := schedulerPolicy{PerWorkerConcurrency: 1}
|
||||
oneUsed := &plugin_pb.WorkerHeartbeat{WorkerId: workerID, ExecutionSlotsUsed: 1, ExecutionSlotsTotal: 4}
|
||||
_, release, err := pluginSvc.reserveExecutorByID("vacuum", policy, workerID, "vacuum-1")
|
||||
if err != nil {
|
||||
t.Fatalf("reserve vacuum-1: %v", err)
|
||||
}
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, oneUsed)
|
||||
release()
|
||||
|
||||
executor, _ := pluginSvc.registry.Get(workerID)
|
||||
if !pluginSvc.executorHasCapacity(executor, "balance", policy) {
|
||||
t.Fatalf("the counted slot is still vacuum-1's echo, not balance work")
|
||||
}
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, oneUsed)
|
||||
executor, _ = pluginSvc.registry.Get(workerID)
|
||||
if pluginSvc.executorHasCapacity(executor, "balance", policy) {
|
||||
t.Fatalf("a slot counted after vacuum-1 completed may hold balance work")
|
||||
}
|
||||
}
|
||||
|
||||
// The reason a dispatch gives up names every holder of the worker's slots,
|
||||
// the unitemized ones included.
|
||||
func TestCapacityReasonNamesUnitemizedSlots(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
pluginSvc, err := New(Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer pluginSvc.Shutdown()
|
||||
|
||||
const workerID = "worker-a"
|
||||
pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
|
||||
WorkerId: workerID,
|
||||
Capabilities: []*plugin_pb.JobTypeCapability{
|
||||
{JobType: "admin_script", CanExecute: true, MaxExecutionConcurrency: 2},
|
||||
},
|
||||
})
|
||||
pluginSvc.registry.UpdateHeartbeat(workerID, &plugin_pb.WorkerHeartbeat{
|
||||
WorkerId: workerID, ExecutionSlotsUsed: 2, ExecutionSlotsTotal: 2,
|
||||
RunningWork: []*plugin_pb.RunningWork{{
|
||||
WorkId: "lc-1", Kind: plugin_pb.WorkKind_WORK_KIND_EXECUTION, JobType: "s3_lifecycle",
|
||||
}},
|
||||
})
|
||||
|
||||
wait, reason := pluginSvc.executorCapacityOutlook("admin_script", schedulerPolicy{PerWorkerConcurrency: 2}, 0)
|
||||
want := "no executor capacity for admin_script: worker worker-a has 2/2 slots used by s3_lifecycle, unitemized work this admin did not dispatch"
|
||||
if wait || reason != want {
|
||||
t.Fatalf("outlook = (%v, %q), want (false, %q)", wait, reason, want)
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user