6 Commits

Include renames

Author	SHA1	Message	Date
Junker der Provinz	8d2c0273bd	admin: stop the maintenance scanner pinning itself to one scan per second after a transient failure (#10887) ... * admin: honour persisted task configs when building the maintenance policy buildPolicyFromTaskConfigs passed a literal nil to vacuum, erasure_coding and balance LoadConfigFromPersistence. Those functions look for their LoadXTaskPolicy() accessor via a type assertion, which a nil interface can never satisfy, so every call fell through to NewDefaultConfig() and the policy came back with the compiled-in defaults - Enabled: true among them. A task disabled on disk was therefore still scheduled, and the only trace was a glog.V(1) "Using default ... configuration" line. Thread the real ConfigPersistence through instead. There are two copies of this function: the one in weed/admin/dash builds config.Policy on the normal admin startup path and can simply take cp as its receiver, and the one in weed/admin/maintenance is the fallback used when the config carries no policy yet, which now receives the store from NewMaintenanceManager. weed/admin/dash already imports weed/admin/maintenance, so the maintenance side has to keep the duck-typed interface{} parameter that the task loaders already use rather than importing the concrete type back. The store is only handed over when a data directory is configured: an unconfigured one has nothing to read, and a typed nil pointer would pass the loaders' type assertion and then panic on first use. Fixes #10874 * admin: restore the maintenance scan cadence after an error backoff scanLoop shortens its ticker to the error backoff delay after a failed scan, but it decided whether to replace the ticker by comparing the target interval against the configured scan interval instead of against the interval the ticker was actually running at. Once the errors stopped, getScanInterval returned the configured interval again, the comparison came out false, and the ticker was left at the backoff delay - so a single transient scan failure pinned the scanner to one scan per second for the rest of the process lifetime. That is the ~1/second cadence in issue #10874: 658 KB/s of "Cancelled N stale pending balance tasks before re-detection" and 193k orphaned task files over two days. Track the interval the ticker is running at and compare against that, so both entering the backoff and returning to the normal cadence replace the ticker. While in here: - defer ticker.Stop() bound the ticker that was current when the defer was registered, so every replacement ticker leaked on return. Wrap it in a closure. - running was written by Start/Stop and read by all three background loops without synchronisation. Guard it with the existing mutex, fold the running check in triggerScanInternal into the lock it already takes, and make Stop a no-op when not running so a second call cannot close the stop channel twice. Refs #10874 * admin: make the maintenance policy actually reach the task detectors Loading the persisted task configs into the maintenance policy only matters if something reads that policy, and nothing did. MaintenanceIntegration pushes the policy into every registered detector and scheduler through interface{ SetEnabled(bool) } and interface{ SetMaxConcurrent(int) } type assertions. Every task registered through base.RegisterTask is backed by base.GenericDetector and base.GenericScheduler, and neither implemented either method, so all four assertions failed silently for every task on every startup. The policy's enabled flag reached nothing: ScanWithTaskDetectors gates on detector.IsEnabled(), and the queue's policy lookups for max concurrent and repeat interval are fallbacks that only fire when the scheduler reports zero, which the generic scheduler never does. Add the setters, delegating to the TaskConfig.SetEnabled the interface already declares and to TaskDefinition.MaxConcurrent, which is what GetMaxConcurrent returns. Applying the policy required three more fixes, because with the assertions working the policy could now do damage as well as good: - IsTaskEnabled reports false for a task type the policy has no entry for, so applying it unconditionally would have disabled every task the policy does not list. Skip task types with no policy entry: no entry means no opinion, not disabled. - ec_balance was exactly such a task. It is registered like the other three but had no entry in the policy builder and no accessor on ConfigPersistence at all, so its configuration could never be persisted. Add SaveEcBalanceTaskPolicy/LoadEcBalanceTaskPolicy, the task_ec_balance.pb file, the SaveTaskPolicy dispatcher case, and the policy entry. - InitMaintenanceManager ran before loadTaskConfigurationsFromPersistence, which replaces each task's whole config object, so the policy was applied and then immediately thrown away. Swap the order. Both read the same files, so the policy is now the last writer and stays authoritative. MaintenanceManager.UpdateConfig also updated the queue's and the scanner's policy but not the integration's, so a policy changed at runtime never reached the detectors. Add MaintenanceIntegration.SetPolicy and call it. While building the policy, stop hand-copying each task's fields and use the task's own ToTaskPolicy(). The hand-written version was a second definition of every task's policy and had already lost the erasure coding preferred tags and replica placement and the balance IO rate limit. For the same reason, the "nothing persisted yet" branches of LoadVacuumTaskPolicy, LoadErasureCodingTaskPolicy and LoadBalanceTaskPolicy now derive from each task's NewDefaultConfig() instead of a third hand-written copy. Those copies had drifted, so with a data directory but no config file on disk the effective defaults differed from what the task and the admin UI schema both advertise: vacuum scan interval 24h -> 2h balance scan interval 6h -> 30m balance imbalance 0.1 -> 0.2 erasure coding scan interval 168h -> 1h erasure coding fullness 0.90 -> 0.95 erasure coding min volume 1024MB -> 30MB Finally, weed/admin/dash and weed/admin/maintenance each carried a copy of the policy builder and they had already diverged. Export the maintenance one as BuildPolicyFromTaskConfigs and have dash call it. Refs #10874 * worker: warn when a config store cannot supply a task's persisted config LoadConfigFromPersistence logged a single glog.V(1) "Using default X configuration" for every way of not loading anything, so the bug in issue #10874 - a store handed in that the type assertion rejects, leaving a task running on compiled-in defaults - looked exactly like the normal "no data directory configured" case. The reporter had to read the source to work out why their disabled task kept running, and asked for this specifically. Separate the cases. A non-nil store that does not provide the accessor is always a wiring bug and is now logged at warning level, naming the type and the missing method. A read error or a policy that will not apply is also a warning. No persistence configured, and a store with nothing saved yet, stay at V(1): those are normal. Refs #10874 * admin: stop GetTaskPolicy panicking on a maintenance policy that is nil GetTaskPolicy dereferenced its MaintenancePolicy argument to look at TaskPolicies, so IsTaskEnabled, GetMaxConcurrent and GetRepeatInterval all took the admin process down when handed a nil policy. A nil policy is not a programming error here: MaintenanceConfig.Policy is unset until something builds one, DefaultMaintenanceConfig returns a config with no policy at all, and UpdateConfig installs whatever config it is given. Found by calling IsTaskEnabled with the policy from a freshly defaulted MaintenanceConfig. Treat a nil policy as "no entry": no task enabled, the safe concurrency default of 1, and a repeat interval of 0 so callers fall back to their own default instead of reading DefaultRepeatIntervalSeconds off nil. Also add the startup test this was found with. It walks the admin server's startup sequence over a data directory that has balance saved as disabled and checks the state that decides whether issue #10874 happens: the balance detector reports disabled, vacuum stays enabled, and tasks whose config was never saved keep their compiled-in default. Refs #10874 * admin: document the synchronisation SetPolicy would need beyond startup ConfigureTasksFromPolicy now really writes TaskDefinition.Config and TaskDefinition.MaxConcurrent, which the scan loop reads through detector.IsEnabled() with nothing synchronising the two. Every caller runs during admin server startup today, before the scan loop exists, so there is no live race - but the next caller has to add the locking, and the same already applies to UpdateAllConfigs replacing the whole config object. Write it down at the seam instead of leaving it to be rediscovered. Refs #10874	2026-08-22 23:41:45 -07:00
Chris Lu	8a4fdf06c0	admin/maintenance: reload in-flight tasks on startup instead of discarding them (#9857) ... * admin/maintenance: reload in-flight tasks on startup instead of discarding LoadTasksFromPersistence deleted all persisted task files on startup and relied on the scanner to re-detect, so saved task state was never consumed — the persistence was effectively write-only. Reload non-terminal tasks (pending/assigned/in_progress) into the queue, resetting in-flight ones to pending since their worker is gone after a restart (maintenance tasks are idempotent). Terminal task files are dropped; the scanner still backfills anything not persisted. * address review: nil-guard reloaded tasks and SyncTask to ActiveTopology - skip nil entries from LoadAllTaskStates (corrupted state) - re-sync restored tasks with MaintenanceIntegration so ActiveTopology (in-memory, empty on startup) knows about them; otherwise GetNextTask's AssignTask rejects them as unknown and they never get assigned	2026-06-07 22:45:38 -07:00
Chris Lu	076d504044	fix(admin): reduce memory usage and verbose logging for large clusters (#8927) ... * fix(admin): reduce memory usage and verbose logging for large clusters (#8919) The admin server used excessive memory and produced thousands of log lines on clusters with many volumes (e.g., 33k volumes). Three root causes: 1. Scanner duplicated all volume metrics: getVolumeHealthMetrics() created VolumeHealthMetrics objects, then convertToTaskMetrics() copied them all into identical types.VolumeHealthMetrics. Now uses the task-system type directly, eliminating the duplicate allocation and removing convertToTaskMetrics. 2. All previous task states loaded at startup: LoadTasksFromPersistence read and deserialized every .pb file from disk, logging each one. With thousands of balance tasks persisted, this caused massive startup I/O, memory usage, and log noise (including unguarded DEBUG glog.Infof per task). Now starts with an empty queue — the scanner re-detects current needs from live cluster state. Terminal tasks are purged from memory and disk when new scan results arrive. 3. Verbose per-volume/per-node logging: V(2) and V(3) logs produced thousands of lines per scan. Per-volume logs bumped to V(4), per-node/rack/disk logs bumped to V(3). Topology summary now logs counts instead of full node ID arrays. Also removes lastTopologyInfo field from MaintenanceScanner — the raw protobuf topology is returned as a local value and not retained between 30-minute scans. * fix(admin): delete stale task files at startup, add DeleteAllTaskStates Old task .pb files from previous runs were left on disk. The periodic CleanupCompletedTasks still loads all files to find completed ones — the same expensive 4GB path from the pprof profile. Now at startup, DeleteAllTaskStates removes all .pb files by scanning the directory without reading or deserializing them. The scanner will re-detect any tasks still needed from live cluster state. * fix(admin): don't persist terminal tasks to disk CompleteTask was saving failed/completed tasks to disk where they'd accumulate. The periodic cleanup only triggered for completed tasks, not failed ones. Now terminal tasks are deleted from disk immediately and only kept in memory for the current session's UI. * fix(admin): cap in-memory tasks to 100 per job type Without a limit, the task map grows unbounded — balance could create thousands of pending tasks for a cluster with many imbalanced volumes. Now AddTask rejects new tasks when a job type already has 100 in the queue. The scanner will re-detect skipped volumes on the next scan. * fix(admin): address PR review - memory-only purge, active-only capacity - purgeTerminalTasks now only cleans in-memory map (terminal tasks are already deleted from disk by CompleteTask) - Per-type capacity limit counts only active tasks (pending/assigned/ in_progress), not terminal ones - When at capacity, purge terminal tasks first before rejecting * fix(admin): fix orphaned comment, add TaskStatusCancelled to terminal switch - Move hasQueuedOrActiveTaskForVolume comment to its function definition - Add TaskStatusCancelled to the terminal state switch in CompleteTask so cancelled task files are deleted from disk	2026-04-04 18:45:57 -07:00
Chris Lu	b9e560dcf1	Prevent overlapping maintenance tasks per volume (#8463) ... * Prevent concurrent maintenance tasks per volume * fix panic	2026-02-27 13:14:52 -08:00
Chris Luandgemini-code-assist[bot]	19c18d827a	admin: fix capacity leak in maintenance system by preserving Task IDs (#8214) ... * admin: fix capacity leak in maintenance system by preserving Task IDs Preserve the original TaskID generated during detection and sync task states (Assign/Complete/Retry) with ActiveTopology. This ensures that capacity reserved during task assignment is properly released when a task completes or fails, preventing 'need 9, have 0' capacity exhaustion. Fixes https://github.com/seaweedfs/seaweedfs/issues/8202 * Update weed/admin/maintenance/maintenance_queue.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update weed/admin/maintenance/maintenance_queue.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * test: rename ActiveTopologySync to TaskIDPreservation Rename the test case to more accurately reflect its scope, as suggested by the code review bot. * Add TestMaintenanceQueue_ActiveTopologySync to verify task state synchronization and capacity management * Implement task assignment rollback and add verification test * Enhance ActiveTopology.CompleteTask to support pending tasks * Populate storage impact in MaintenanceIntegration.SyncTask * Release capacity in RemoveStaleWorkers when worker becomes unavailable * Release capacity in MaintenanceManager.CancelTask when pending task is cancelled * Sync reloaded tasks with ActiveTopology in LoadTasksFromPersistence * Add verification tests for consistent capacity management lifecycle * Add TestMaintenanceQueue_RetryCapacitySync to verify capacity tracking during retries --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	2026-02-04 20:39:34 -08:00
Chris Luandgemini-code-assist[bot]	891a2fb6eb	Admin: misc improvements on admin server and workers. EC now works. (#7055) ... * initial design * added simulation as tests * reorganized the codebase to move the simulation framework and tests into their own dedicated package * integration test. ec worker task * remove "enhanced" reference * start master, volume servers, filer Current Status ✅ Master: Healthy and running (port 9333) ✅ Filer: Healthy and running (port 8888) ✅ Volume Servers: All 6 servers running (ports 8080-8085) 🔄 Admin/Workers: Will start when dependencies are ready * generate write load * tasks are assigned * admin start wtih grpc port. worker has its own working directory * Update .gitignore * working worker and admin. Task detection is not working yet. * compiles, detection uses volumeSizeLimitMB from master * compiles * worker retries connecting to admin * build and restart * rendering pending tasks * skip task ID column * sticky worker id * test canScheduleTaskNow * worker reconnect to admin * clean up logs * worker register itself first * worker can run ec work and report status but: 1. one volume should not be repeatedly worked on. 2. ec shards needs to be distributed and source data should be deleted. * move ec task logic * listing ec shards * local copy, ec. Need to distribute. * ec is mostly working now * distribution of ec shards needs improvement * need configuration to enable ec * show ec volumes * interval field UI component * rename * integration test with vauuming * garbage percentage threshold * fix warning * display ec shard sizes * fix ec volumes list * Update ui.go * show default values * ensure correct default value * MaintenanceConfig use ConfigField * use schema defined defaults * config * reduce duplication * refactor to use BaseUIProvider * each task register its schema * checkECEncodingCandidate use ecDetector * use vacuumDetector * use volumeSizeLimitMB * remove remove * remove unused * refactor * use new framework * remove v2 reference * refactor * left menu can scroll now * The maintenance manager was not being initialized when no data directory was configured for persistent storage. * saving config * Update task_config_schema_templ.go * enable/disable tasks * protobuf encoded task configurations * fix system settings * use ui component * remove logs * interface{} Reduction * reduce interface{} * reduce interface{} * avoid from/to map * reduce interface{} * refactor * keep it DRY * added logging * debug messages * debug level * debug * show the log caller line * use configured task policy * log level * handle admin heartbeat response * Update worker.go * fix EC rack and dc count * Report task status to admin server * fix task logging, simplify interface checking, use erasure_coding constants * factor in empty volume server during task planning * volume.list adds disk id * track disk id also * fix locking scheduled and manual scanning * add active topology * simplify task detector * ec task completed, but shards are not showing up * implement ec in ec_typed.go * adjust log level * dedup * implementing ec copying shards and only ecx files * use disk id when distributing ec shards 🎯 Planning: ActiveTopology creates DestinationPlan with specific TargetDisk 📦 Task Creation: maintenance_integration.go creates ECDestination with DiskId 🚀 Task Execution: EC task passes DiskId in VolumeEcShardsCopyRequest 💾 Volume Server: Receives disk_id and stores shards on specific disk (vs.store.Locations[req.DiskId]) 📂 File System: EC shards and metadata land in the exact disk directory planned * Delete original volume from all locations * clean up existing shard locations * local encoding and distributing * Update docker/admin_integration/EC-TESTING-README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * check volume id range * simplify * fix tests * fix types * clean up logs and tests --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	2025-07-30 12:38:03 -07:00