mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-04 11:07:35 +01:00
329ccb8bc0f16d2203531d0657104222bcfbf650
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
cd82e663ad |
admin: credit an unitemized slot to a pending job only after a later heartbeat (#12385)
* admin: credit an unitemized slot to a pending job only after a later heartbeat A pending reservation the worker does not list took one of the slots the worker counts without itemizing, even when that heartbeat came before the reservation and could not have counted the job. The slot belonged to unknown work, so the admin undercounted it and could admit one job too many, past a job type's per-worker limit, until the next heartbeat. A reservation now records when it was made, and claims an unitemized slot only from a heartbeat received at or after that time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: time unitemized-slot credit by the heartbeat, not the last hello A reconnect's hello moves LastSeenAt but keeps the old heartbeat, so a pending reservation could claim a slot counted before it existed, and a completed job's echo could end without a new heartbeat. Record when the heartbeat itself arrived and compare reservations against that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: a completed job claims no slot from a heartbeat before its reservation A job that completes before any heartbeat counts it has no echo, so the slot the last heartbeat counted is other work. Crediting it to the job's type let another job type pass its per-worker limit until the next heartbeat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
5dfda3645e |
admin: count executor slots per job type and stop lanes waiting on capacity they cannot free (#12384)
* admin: count executor slots per job type, deduped against the heartbeat per_worker_execution_concurrency is a per-job-type setting, but the scheduler compared it against the worker's execution_slots_used, which counts every job type. One s3_lifecycle job a worker reported running after a reconnect made admin_script (per-worker 1) undispatchable on a worker with three free slots. A reservation and the heartbeat entry for the same job were also charged as two slots, so a per-worker limit of N let N-1 jobs run. Reservations now carry the job id and job type. The per-type limit is checked against that type's work alone, the worker's slot total against work of every type, and a heartbeat entry covered by a reservation is counted once. Per-lane reservation pools go away: the per-type count is what kept lanes from starving each other. * admin: stop a lane waiting on executor capacity it cannot free A queued job no executor could take was retried every backoff until job_type_max_runtime ran out, up to 30 minutes, while the default lane sat in executing and every other job type in it idled. The dispatch loop now asks what holds the capacity. Slots held by the lane's own jobs release when they complete, so it keeps waiting. Slots held by other lanes or by work this admin never dispatched clear on a schedule the lane has no view of, so the queued jobs are canceled at once with a skipped_no_executor activity naming the worker, its slot count and the job types holding it, and the next detection proposes them again. A missing worker is waited for only the orphan grace. * admin: cancel a job on its worker when the admin expires it Expiring a job, by the stale sweep or by the operator's expire endpoint, only failed the tracked record. The worker never heard about it and kept running the job with its slot taken, so a job stuck in "starting" held a slot for over a day and, with the old slot accounting, blocked a whole lane. Expiring a job now also sends the worker a cancel for it, so the slot comes back on the next heartbeat. * admin: count a completed job's heartbeat echo as its lane's own slot The worker sends JobCompleted before the heartbeat that frees the slot, so right after a dispatch goroutine releases its reservation the job still shows in the worker's running work. That entry was counted as foreign, and a lane with per-worker 1 gave up and canceled the rest of its queue after the first job. A released reservation now stays for completedJobEchoGrace and claims that heartbeat entry as the lane's own load. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: let the admin's own jobs claim slots a worker does not itemize The Rust worker reports only a slot count, every 10s, and the Go worker counts a slot one heartbeat before it lists the job. Those slots were charged on top of the reservations behind them, and once a job completed its stale count read as foreign work, so the lane gave up its queue. Unitemized slots now go first to this admin's unlisted reservations and recent completions; the rest are of unknown type and count against every type's per-worker limit, not only the slot total. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: give up pinned jobs whose busy home a free worker cannot relieve A job homed on a replica-host worker stays on that home while its queue is shorter than LocalQueuePin, so a free remote worker cannot take it. The outlook treated that free worker as a reason to wait, and a home held by foreign work parked the lane for its whole window. A free worker now only holds the wait for the orphan grace, after which the held homes decide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: let a redispatched job's reservation replace its released one A job the worker turns away for capacity is released, kept as an echo, and queued again under the same id. Once the new attempt ran, both the echo and the new reservation matched its heartbeat entry and it counted as two slots. Reserving a job now drops what an earlier reservation of it left. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: credit a count-only slot to a completed job only until the next heartbeat A released reservation claimed an unitemized slot for its full 30s grace, so on a count-only heartbeat other work of another type was charged to the completed job's type. A heartbeat that arrives after the release already reflects the freed slot; from then on the slot is of unknown type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: name unitemized slots in the no-capacity reason holders() listed only the job types the worker itemized, so a worker with named and count-only work read as "2/2 slots used by <type>". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep the blocked clock running until a reservation succeeds A popped job whose reservation then failed reset the clock before the failure pushed it back, so pop-then-fail cycles never aged the block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: resend an expiry cancel the worker did not receive Expiring a job sends its worker one cancel. When the worker is disconnected, or its outgoing buffer stays full past the send timeout, that cancel is dropped. The record is already failed, so later sweeps and expire calls skip it, and a worker that kept the job running (a detached job across a reconnect, or a jammed stream) holds the slot until the job ends on its own. The admin now holds an undelivered expiry cancel and sends it again on the worker's next heartbeat that still lists the job. It is forgotten when that heartbeat no longer lists the job, and skipped when the job is active again, since a new attempt reuses the job id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: cancel an expired attempt by its request id The expiry cancel named the job id, and a retry reuses it. The worker keys its cancel funcs by both ids, so a cancel arriving after a retry started on the same worker stopped the retry. The held resend also gave up on any job that was active again, and the expired attempt's own late progress marks the job running, so a worker that buffered progress while disconnected kept the expired job. The cancel now names the attempt's request id, as the other execution cancels do, and the held resend no longer depends on the tracked state. Only a cancel that can name nothing but the job id still skips a job that is active again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: ignore late reports from an attempt the admin already ended Expiring a job now cancels it on its worker, and the worker answers with its own JobCompleted for that attempt. That overwrote the expiry's failure with the cancel error and logged a second run record under the same run id. Late progress from the expired attempt likewise reopened the record as running. A progress update or completion whose request id names an attempt the record already closed is now dropped after the dispatcher's waiter is told. A retry runs under a new request id, so its reports still apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris Lu
|
30f0364256 |
admin: fail an in-flight detection when its worker disconnects (#12376)
* admin: keep a reconnected plugin worker registered when its old stream unwinds cleanupSession removed whatever session was registered under the worker ID, so a worker that reconnected while its previous stream was still unwinding lost the new session and its registry entry to the old stream's deferred cleanup. Tie cleanup to the stream that is ending and publish the new stream before the registry upsert so a lingering predecessor sees itself superseded. * admin: fail an in-flight detection when its worker disconnects RunDetectionWithReport waited only on the caller's context and the DetectionComplete channel, so a detection whose worker stream ended mid-request could only finish by timeout. On a customer cluster a planned restart of ec-vacuum-1 during an ec_vacuum detection left the default lane in 'detecting' with every execution slot idle until the 3600s detection timeout, blocking every job type in that lane. Bind the request to the stream it was sent on and fail the detection as soon as that stream closes, whether by disconnect or by a reconnect that replaced the session. A liveness tick also fails it once the registry no longer considers the worker available (heartbeat stale), sending a cancel in that case because the stream may still be open. The detector lease is dropped so the next pick starts fresh, and a 'disconnected' detector activity is recorded. WorkerDisconnectForTest now goes through the real session cleanup. * admin: retry a scheduled detection once when its detector drops When a detection fails because its worker disconnected, pick a detector again and retry once within the same iteration while the job type's runtime budget allows, instead of giving the whole cycle up. Any other failure, or a failed retry, ends the run as before and the lane moves on; the next cycle re-detects anyway. * admin: register a plugin worker's hello and stream as one step WorkerStream published the stream under sessionsMu but ran the registry upsert after releasing it, so two concurrent streams for one worker ID could publish their hellos in the opposite order from their sessions: the registry then described the older hello's capabilities while only the newer stream was sendable, and detector picks routed on them. registerSession upserts the hello and installs the stream under one sessionsMu critical section, and a hello or heartbeat arriving on a stream that is no longer the registered one is dropped instead of rewriting the live stream's entry. * admin: steer a detection retry off the stream that dropped The retry picked while the lost worker could still be in the registry: cleanupSession closed done before taking the registry entry down, and a stream closed by a reconnect keeps its entry on purpose. The lost worker ranks first on free slots, so the single retry went back to it and failed on send while a healthy worker sat idle. The disconnect error now carries the stream it came from and the retry picks around it; pickDetector only returns workers whose current stream is open, excluding registry entries whose stream is closed; and cleanupSession removes the registry entry before closing done, so a waiter woken by the closure never sees the dead worker as a candidate. |
||
|
Чепурнов СемёнandChepurnov Simon
|
624a13ecc6 |
EC worker: pin jobs to replica-host workers, late-bind local source (#11718)
* feat: prefer same-host worker and local EC replica Admin executor pick ranked idle workers only, so a job whose volume lived on full-03 could still run on full-01 and pull the whole .dat across the LAN. Rank workers whose Hello host matches any replica first (fallback to others if the local worker is full). Detection named Sources[0] / the lowest server id; execute then SyncAndSelectBestReplica and could switch the copy source to another rack. After sync, if a replica on this machine has a complete file count, copy from that URL so affinity and pull stay on one host. * feat: pin scheduler jobs to replica-host workers until load K Reserve prefers the least-loaded replica-host worker and waits while that load is below 5 instead of spilling to a free remote worker. K=0 restores immediate spill. Waterfill uses reservation load so four idle preferred workers each get a job. * feat: park scheduled jobs on per-worker admin queues Assign a detection batch to the least-loaded replica-host worker in one pass, then let ExecutionConcurrency goroutines pop a home-runnable job or steal when that home queue reaches K. Dispatch no longer holds a job in reserve-backoff, so a free worker is not blocked by another home's pin. Tracked jobs stay pending/queued and record the home worker for the admin UI. * fix: rehome queued jobs when the home worker dies A live home still pins below K. A home missing from live executors is stolen immediately onto a free worker so a short orphaned queue does not sit until the dispatch window ends. * feat: wait before stealing a dead home worker queue Orphaned queues are held for 30s so a worker reconnect blink does not dump jobs onto other machines. After the grace, steal without K as before. Zero disables the wait. A home that comes back during grace keeps its queue. * fix: late-bind EC from to worker hostname After replicas are consistent (equal FileCount), rebind encode source to a volume server on the same hostname as worker -address. Any port on that host is local. Diverged FileCount still uses the max replica. * feat: assign homeless jobs to global FIFO Jobs whose replica hostname has no registered worker skip pin and orphan grace. Free workers take own home, then steal at K/orphan, then leftover FIFO. * fix: take homeless FIFO before steal for no-home workers Workers without a replica-host match take leftover first; steal (K/orphan) only when FIFO is empty. Home workers keep own-queue, steal, then leftover. --------- Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com> |
||
|
Chris Lu
|
78e7e04377 |
plugin scheduler: drain started jobs past the window close instead of killing them (#10728)
* plugin scheduler: drain started jobs past the window close instead of killing them * plugin scheduler: never drain-cap an attempt below its declared estimated runtime * plugin scheduler: cap estimated_runtime_seconds before the Duration conversion |
||
|
Chris Lu
|
cba2e5150c |
plugin: fix flaky scheduler lock test (#10432)
plugin: stop the scheduler lock test racing its own background loops TestRunLaneSchedulerIterationLockBehavior constructed the plugin with a cluster-context provider, which makes New start a background scheduler loop per lane. Those loops call runLaneSchedulerIteration on the same lane the test then drives by hand, so a loop could consume the due job — running detection and pushing the next-detection time forward — before the manual call observed the lock. The Default case then saw the lock acquired zero times and failed intermittently. Construct without the provider so no loops start, and set the provider afterward so the manual iteration can still detect. This is the pattern scheduler_status_test.go already uses for the same reason. Reproduced under -race -count=100 -p 4 before, green after. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris Lu
|
9a7cfaf371 |
admin: stop holding the shell admin lock across whole scheduler batches (#10290)
* admin: let waiting shell clients win the admin lock The admin lock manager re-acquired the cluster shell lock immediately after releasing it, while a waiting weed shell client only polls the master once a second, so an operator could lose the race indefinitely. Leave the lock free for slightly more than one poll interval before re-acquiring, and once overlapping reference-counted holds have kept the lock continuously held for over a minute, make new admin-side acquires wait for a full release before piggybacking. * admin/plugin: take the shell lock per detection and per job, not per batch The default-lane scheduler acquired the cluster shell lock once and held it across the whole pass: every due job type's detection plus all of its dispatched jobs, each drained to completion. A manual weed shell operation sharing that lock could stall behind the batch for up to the extended execution window. Hold the lock only while it protects something: around each detection scan and around each dispatched job. Manual shell operations now wait for at most one in-flight job. The admin UI detect+execute path stops wrapping dispatch in an outer hold, since a nested acquire would deadlock against the lock manager's fairness window; detection and per-job dispatch take the lock themselves. * admin/plugin: stop extending the dispatch window to the largest job estimate When any proposal's estimated runtime exceeded the remaining JobTypeMaxRuntime, the whole dispatch context was replaced with a fresh one capped at eight hours, so a single balance backlog could hold the default lane (and with it erasure_coding and vacuum detection) for that long. Keep the dispatch window at JobTypeMaxRuntime and instead detach each started attempt onto its own estimated-runtime deadline. Large jobs still get their full time once started; jobs not yet started when the window closes are canceled and re-proposed by a later detection, so sibling job types get a turn every window. * worker/balance: re-check each planned move against the master before executing A balance plan is computed at detection time, but the admin lock is released between detection and execution, so a manual shell operation can rearrange the volume in the gap. The task's own guards catch a vanished source, but a target that gained a replica in the meantime would be silently overwritten by VolumeCopy and the source delete would then reduce the volume to a single copy. Before executing each move, ask the master for the volume's current locations (uncached) and skip the move if the volume has left the source or the target already holds a replica. Skipped moves fail with a stale-move error and the next detection replans them. Without master addresses in the cluster context the check is skipped, preserving the old behavior with older admins. * admin: block re-acquire while the final lock release is in flight Release dropped the manager mutex before calling ReleaseLock, so a concurrent Acquire could see hold count zero and call RequestLock while the locker still considered itself locked. That request no-ops, leaving a hold with no live master lease. Track the in-flight release and make Acquire wait for it. Also normalize a nil release function from lock manager implementations, and make the yield and fairness windows per-instance fields so tests stop mutating globals. |
||
|
ahalaun
|
bcd2c958e1 | fix(admin): make scheduler pruning lane-aware (#9790) | ||
|
Chris Lu
|
5d43f84df7 |
refactor(plugin): rename detection_interval_seconds → detection_interval_minutes (#9366)
Minutes is the natural granularity for detection cadence — every production handler already set the seconds field to a 60-multiple (17*60, 30*60, 3600, 24*60*60). Switching to minutes drops the *60 arithmetic and matches the unit conventions used elsewhere in the plugin worker forms. - Proto: AdminRuntimeDefaults + AdminRuntimeConfig.detection_interval_* field renamed. - Helpers: durationFromMinutes / minutesFromDuration alongside the existing seconds variants in plugin_scheduler.go. - Handlers: vacuum, ec_balance, balance, erasure_coding, iceberg, admin_script, s3_lifecycle now declare DetectionIntervalMinutes. - Admin: scheduler_status + types + UI templ + plugin_api.go pass through the new field; UI label and table cells switch to "min". |
||
|
Chris Lu
|
e648c76bcf | go fmt | ||
|
Chris Lu
|
a95b8396e4 |
plugin scheduler: run iceberg and lifecycle lanes concurrently (#8821)
* plugin scheduler: run iceberg and lifecycle lanes concurrently The default lane serialises job types under a single admin lock because volume-management operations share global state. Iceberg and lifecycle lanes have no such constraint, so run each of their job types independently in separate goroutines. * Fix concurrent lane scheduler status * plugin scheduler: address review feedback - Extract collectDueJobTypes helper to deduplicate policy loading between locked and concurrent iteration paths. - Use atomic.Bool instead of sync.Mutex for hadJobs in the concurrent path. - Set lane loop state to "busy" before launching concurrent goroutines so the lane is not reported as idle while work runs. - Convert TestLaneRequiresLock to table-driven style. - Add TestRunLaneSchedulerIterationLockBehavior to verify the scheduler acquires the admin lock only for lanes that require it. - Fix flaky TestGetLaneSchedulerStatusShowsActiveConcurrentLaneWork by not starting background scheduler goroutines that race with the direct runJobTypeIteration call. |
||
|
Chris LuandCopilot
|
18ccc9b773 |
Plugin scheduler: sequential iterations with max runtime (#8496)
* pb: add job type max runtime setting * plugin: default job type max runtime * plugin: redesign scheduler loop * admin ui: update scheduler settings * plugin: fix scheduler loop state name * plugin scheduler: restore backlog skip * plugin scheduler: drop legacy detection helper * admin api: require scheduler config body * admin ui: preserve detection interval on save * plugin scheduler: use job context and drain cancels * plugin scheduler: respect detection intervals * plugin scheduler: gate runs and drain queue * ec test: reuse req/resp vars * ec test: add scheduler debug logs * Adjust scheduler idle sleep and initial run delay * Clear pending job queue before scheduler runs * Log next detection time in EC integration test * Improve plugin scheduler debug logging in EC test * Expose scheduler next detection time * Log scheduler next detection time in EC test * Wake scheduler on config or worker updates * Expose scheduler sleep interval in UI * Fix scheduler sleep save value selection * Set scheduler idle sleep default to 613s * Show scheduler next run time in plugin UI --------- Co-authored-by: Copilot <copilot@github.com> |
||
|
Chris LuandCopilot Autofix powered by AI
|
8ec9ff4a12 |
Refactor plugin system and migrate worker runtime (#8369)
* admin: add plugin runtime UI page and route wiring * pb: add plugin gRPC contract and generated bindings * admin/plugin: implement worker registry, runtime, monitoring, and config store * admin/dash: wire plugin runtime and expose plugin workflow APIs * command: add flags to enable plugin runtime * admin: rename remaining plugin v2 wording to plugin * admin/plugin: add detectable job type registry helper * admin/plugin: add scheduled detection and dispatch orchestration * admin/plugin: prefetch job type descriptors when workers connect * admin/plugin: add known job type discovery API and UI * admin/plugin: refresh design doc to match current implementation * admin/plugin: enforce per-worker scheduler concurrency limits * admin/plugin: use descriptor runtime defaults for scheduler policy * admin/ui: auto-load first known plugin job type on page open * admin/plugin: bootstrap persisted config from descriptor defaults * admin/plugin: dedupe scheduled proposals by dedupe key * admin/ui: add job type and state filters for plugin monitoring * admin/ui: add per-job-type plugin activity summary * admin/plugin: split descriptor read API from schema refresh * admin/ui: keep plugin summary metrics global while tables are filtered * admin/plugin: retry executor reservation before timing out * admin/plugin: expose scheduler states for monitoring * admin/ui: show per-job-type scheduler states in plugin monitor * pb/plugin: rename protobuf package to plugin * admin/plugin: rename pluginRuntime wiring to plugin * admin/plugin: remove runtime naming from plugin APIs and UI * admin/plugin: rename runtime files to plugin naming * admin/plugin: persist jobs and activities for monitor recovery * admin/plugin: lease one detector worker per job type * admin/ui: show worker load from plugin heartbeats * admin/plugin: skip stale workers for detector and executor picks * plugin/worker: add plugin worker command and stream runtime scaffold * plugin/worker: implement vacuum detect and execute handlers * admin/plugin: document external vacuum plugin worker starter * command: update plugin.worker help to reflect implemented flow * command/admin: drop legacy Plugin V2 label * plugin/worker: validate vacuum job type and respect min interval * plugin/worker: test no-op detect when min interval not elapsed * command/admin: document plugin.worker external process * plugin/worker: advertise configured concurrency in hello * command/plugin.worker: add jobType handler selection * command/plugin.worker: test handler selection by job type * command/plugin.worker: persist worker id in workingDir * admin/plugin: document plugin.worker jobType and workingDir flags * plugin/worker: support cancel request for in-flight work * plugin/worker: test cancel request acknowledgements * command/plugin.worker: document workingDir and jobType behavior * plugin/worker: emit executor activity events for monitor * plugin/worker: test executor activity builder * admin/plugin: send last successful run in detection request * admin/plugin: send cancel request when detect or execute context ends * admin/plugin: document worker cancel request responsibility * admin/handlers: expose plugin scheduler states API in no-auth mode * admin/handlers: test plugin scheduler states route registration * admin/plugin: keep worker id on worker-generated activity records * admin/plugin: test worker id propagation in monitor activities * admin/dash: always initialize plugin service * command/admin: remove plugin enable flags and default to enabled * admin/dash: drop pluginEnabled constructor parameter * admin/plugin UI: stop checking plugin enabled state * admin/plugin: remove docs for plugin enable flags * admin/dash: remove unused plugin enabled check method * admin/dash: fallback to in-memory plugin init when dataDir fails * admin/plugin API: expose worker gRPC port in status * command/plugin.worker: resolve admin gRPC port via plugin status * split plugin UI into overview/configuration/monitoring pages * Update layout_templ.go * add volume_balance plugin worker handler * wire plugin.worker CLI for volume_balance job type * add erasure_coding plugin worker handler * wire plugin.worker CLI for erasure_coding job type * support multi-job handlers in plugin worker runtime * allow plugin.worker jobType as comma-separated list * admin/plugin UI: rename to Workers and simplify config view * plugin worker: queue detection requests instead of capacity reject * Update plugin_worker.go * plugin volume_balance: remove force_move/timeout from worker config UI * plugin erasure_coding: enforce local working dir and cleanup * admin/plugin UI: rename admin settings to job scheduling * admin/plugin UI: persist and robustly render detection results * admin/plugin: record and return detection trace metadata * admin/plugin UI: show detection process and decision trace * plugin: surface detector decision trace as activities * mini: start a plugin worker by default * admin/plugin UI: split monitoring into detection and execution tabs * plugin worker: emit detection decision trace for EC and balance * admin workers UI: split monitoring into detection and execution pages * plugin scheduler: skip proposals for active assigned/running jobs * admin workers UI: add job queue tab * plugin worker: add dummy stress detector and executor job type * admin workers UI: reorder tabs to detection queue execution * admin workers UI: regenerate plugin template * plugin defaults: include dummy stress and add stress tests * plugin dummy stress: rotate detection selections across runs * plugin scheduler: remove cross-run proposal dedupe * plugin queue: track pending scheduled jobs * plugin scheduler: wait for executor capacity before dispatch * plugin scheduler: skip detection when waiting backlog is high * plugin: add disk-backed job detail API and persistence * admin ui: show plugin job detail modal from job id links * plugin: generate unique job ids instead of reusing proposal ids * plugin worker: emit heartbeats on work state changes * plugin registry: round-robin tied executor and detector picks * add temporary EC overnight stress runner * plugin job details: persist and render EC execution plans * ec volume details: color data and parity shard badges * shard labels: keep parity ids numeric and color-only distinction * admin: remove legacy maintenance UI routes and templates * admin: remove dead maintenance endpoint helpers * Update layout_templ.go * remove dummy_stress worker and command support * refactor plugin UI to job-type top tabs and sub-tabs * migrate weed worker command to plugin runtime * remove plugin.worker command and keep worker runtime with metrics * update helm worker args for jobType and execution flags * set plugin scheduling defaults to global 16 and per-worker 4 * stress: fix RPC context reuse and remove redundant variables in ec_stress_runner * admin/plugin: fix lifecycle races, safe channel operations, and terminal state constants * admin/dash: randomize job IDs and fix priority zero-value overwrite in plugin API * admin/handlers: implement buffered rendering to prevent response corruption * admin/plugin: implement debounced persistence flusher and optimize BuildJobDetail memory lookups * admin/plugin: fix priority overwrite and implement bounded wait in scheduler reserve * admin/plugin: implement atomic file writes and fix run record side effects * admin/plugin: use P prefix for parity shard labels in execution plans * admin/plugin: enable parallel execution for cancellation tests * admin: refactor time.Time fields to pointers for better JSON omitempty support * admin/plugin: implement pointer-safe time assignments and comparisons in plugin core * admin/plugin: fix time assignment and sorting logic in plugin monitor after pointer refactor * admin/plugin: update scheduler activity tracking to use time pointers * admin/plugin: fix time-based run history trimming after pointer refactor * admin/dash: fix JobSpec struct literal in plugin API after pointer refactor * admin/view: add D/P prefixes to EC shard badges for UI consistency * admin/plugin: use lifecycle-aware context for schema prefetching * Update ec_volume_details_templ.go * admin/stress: fix proposal sorting and log volume cleanup errors * stress: refine ec stress runner with math/rand and collection name - Added Collection field to VolumeEcShardsDeleteRequest for correct filename construction. - Replaced crypto/rand with seeded math/rand PRNG for bulk payloads. - Added documentation for EcMinAge zero-value behavior. - Added logging for ignored errors in volume/shard deletion. * admin: return internal server error for plugin store failures Changed error status code from 400 Bad Request to 500 Internal Server Error for failures in GetPluginJobDetail to correctly reflect server-side errors. * admin: implement safe channel sends and graceful shutdown sync - Added sync.WaitGroup to Plugin struct to manage background goroutines. - Implemented safeSendCh helper using recover() to prevent panics on closed channels. - Ensured Shutdown() waits for all background operations to complete. * admin: robustify plugin monitor with nil-safe time and record init - Standardized nil-safe assignment for *time.Time pointers (CreatedAt, UpdatedAt, CompletedAt). - Ensured persistJobDetailSnapshot initializes new records correctly if they don't exist on disk. - Fixed debounced persistence to trigger immediate write on job completion. * admin: improve scheduler shutdown behavior and logic guards - Replaced brittle error string matching with explicit r.shutdownCh selection for shutdown detection. - Removed redundant nil guard in buildScheduledJobSpec. - Standardized WaitGroup usage for schedulerLoop. * admin: implement deep copy for job parameters and atomic write fixes - Implemented deepCopyGenericValue and used it in cloneTrackedJob to prevent shared state. - Ensured atomicWriteFile creates parent directories before writing. * admin: remove unreachable branch in shard classification Removed an unreachable 'totalShards <= 0' check in classifyShardID as dataShards and parityShards are already guarded. * admin: secure UI links and use canonical shard constants - Added rel="noopener noreferrer" to external links for security. - Replaced magic number 14 with erasure_coding.TotalShardsCount. - Used renderEcShardBadge for missing shard list consistency. * admin: stabilize plugin tests and fix regressions - Composed a robust plugin_monitor_test.go to handle asynchronous persistence. - Updated all time.Time literals to use timeToPtr helper. - Added explicit Shutdown() calls in tests to synchronize with debounced writes. - Fixed syntax errors and orphaned struct literals in tests. * Potential fix for code scanning alert no. 278: Slice memory allocation with excessive size value Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Potential fix for code scanning alert no. 283: Uncontrolled data used in path expression Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * admin: finalize refinements for error handling, scheduler, and race fixes - Standardized HTTP 500 status codes for store failures in plugin_api.go. - Tracked scheduled detection goroutines with sync.WaitGroup for safe shutdown. - Fixed race condition in safeSendDetectionComplete by extracting channel under lock. - Implemented deep copy for JobActivity details. - Used defaultDirPerm constant in atomicWriteFile. * test(ec): migrate admin dockertest to plugin APIs * admin/plugin_api: fix RunPluginJobTypeAPI to return 500 for server-side detection/filter errors * admin/plugin_api: fix ExecutePluginJobAPI to return 500 for job execution failures * admin/plugin_api: limit parseProtoJSONBody request body to 1MB to prevent unbounded memory usage * admin/plugin: consolidate regex to package-level validJobTypePattern; add char validation to sanitizeJobID * admin/plugin: fix racy Shutdown channel close with sync.Once * admin/plugin: track sendLoop and recv goroutines in WorkerStream with r.wg * admin/plugin: document writeProtoFiles atomicity — .pb is source of truth, .json is human-readable only * admin/plugin: extract activityLess helper to deduplicate nil-safe OccurredAt sort comparators * test/ec: check http.NewRequest errors to prevent nil req panics * test/ec: replace deprecated ioutil/math/rand, fix stale step comment 5.1→3.1 * plugin(ec): raise default detection and scheduling throughput limits * topology: include empty disks in volume list and EC capacity fallback * topology: remove hard 10-task cap for detection planning * Update ec_volume_details_templ.go * adjust default * fix tests --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> |