* topology: list collections without nesting the collection map read lock
Items() already takes the read lock and copies the values. Holding it
around the call made ListCollections take the read lock twice, so a
DeleteCollection waiting for the write lock in between blocked the
second read lock and wedged the master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: test ListCollections against a concurrent collection delete
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: undo the stored record when a delete report arrives
Deleting a volume decremented the active count from the delete report's
ReadOnly flag, which the volume server never sets, so a volume the master
had already stopped counting as writable was uncounted a second time.
Every volume.move does this to its source, and the node, rack, DC and
topology counts drifted negative; the full-list path already used the
stored record. Undo exactly what was counted for the record held.
* volume: say whether a deleted volume was read-only
Explicit deletes, unmounts, collection deletes and the departures named
in a delta heartbeat all left ReadOnly unset, so an older master undid
the writable count for a volume it was no longer counting. Carry the
flags, as the Rust server already does.
* topology: keep the oversized verdict after the location list is gone
The per-replica oversized marks live in the volume's location list, which
is deleted when its last location leaves and recreated flagless on the
next registration. Between that recreation and the next heartbeat that
re-measures the volume, ensureCorrectWritables saw no mark and
re-admitted an oversized volume to the writable list; two other paths
(quarantine release, mark-writable) never consulted the marks at all.
VolumeLayout now remembers the oversized verdict per vid, written only
by heartbeat sizes and forgotten only when a report measures the volume
under the limit. The writable admission gate itself refuses a marked
volume, and the vacuum commit path drops a verdict its live size check
has just disproved.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: mark oversized at registration when the report carries a size
RegisterVolumeLayout ran EnsureCorrectWritables while the recreated
location list still held no per-replica marks. When the registering
report knows the volume's size, refresh the oversized state before the
writable check instead of waiting for the next heartbeat pass.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: test that oversized volumes stay out of writables across re-registration
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: remember the oversized verdict per replica
A shared per-volume verdict was cleared by any single replica's
under-limit report, so a still-oversized replica whose location entry
had died could re-register from a size-less report and put the volume
back on the writable list. Keep one verdict per reporting replica: an
under-limit report clears only its own, and a replica that never
returns cannot pin its volume out of writables forever because the
verdict ages out. The vacuum commit now feeds the node's fresh answer
through the same per-replica bookkeeping instead of erasing the whole
verdict.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: count a volume at the size limit as full on vacuum commit
The layout's oversized test is >= the limit, but the vacuum commit and
status checks used >, so a replica sitting exactly at the limit passed
as not-full and the commit path could re-admit it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: cover per-replica oversized verdicts in tests
One replica shrinking must not clear a still-oversized replica's
remembered verdict; cover that plus a size-less re-registration before
the second replica is re-measured. The vacuum-commit case now shrinks
the stored record first, since the commit re-derives the mark from the
node's own answer.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: judge the vacuum commit by the size it just measured
SetVolumeAvailable rebuilt the oversized verdict from the cached
VolumeInfo, which still carries the pre-compaction size until the next
heartbeat. A commit that shrank a volume under the limit re-marked the
replica oversized and never restored writes. The commit path already
asks every replica for its fresh size; pass it through per replica and
record it on the mark, the verdict, and the size tracking seed.
* topology: hold an expired oversized verdict while its replica is back
The TTL forgives a replica that never returns, but a remount announces
the volume without a size and registration skips the oversized check.
Deleting the verdict on expiry then re-admitted a still-oversized
volume until the next size-bearing heartbeat. Keep counting an expired
verdict while its replica sits in the live location list, and keep
sweeping verdicts whose replicas stayed gone.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chrislusf@gmail.com>
* topology: serialize quarantine records with their layout marks
A quarantine publishes a (node, vid) record and marks the replica read-only at the layout level; release and heartbeat flag updates must observe that pair as one state. Store-then-mark let a concurrent release delete the record before the mark landed, stranding the surviving replica read-only; mark-then-store let a heartbeat clear the mark before the record was visible, returning a conflicting replica to the writable list.
quarantineLock now covers every record<->mark pair: flagDuplicateVolumeId's quarantine, releaseResolvedQuarantine's undo, the isQuarantined-gated marks on registration, and the quarantine-folded read-only flag writes on heartbeat paths. Remote server pushes stay outside the lock; a push that lands after the record was released is undone, and a failed release push re-adds the record for a later retry.
The release test now waits for in-flight async flags to settle instead of asserting on a single read.
* topology: fold quarantine into RegisterVolume's flag rebuild
RegisterVolume rewrites every replica's location read-only flag from what
its node last reported, and reported flags deliberately exclude quarantine.
Any registration of a shared vid -- a reconnected replica, a new copy, the
self-heal and delta-heartbeat re-register paths -- therefore cleared the
quarantine mark on every replica in the location, while the gate afterwards
only re-marked the registering node. Two call sites had no gate at all.
RegisterVolume now takes an isQuarantined predicate and evaluates it under
accessLock for every replica it touches, so the mark is never dropped in
the first place: a release unmarks after the rebuild, and a quarantine
mark re-applies on the next rebuild, with no window between.
* topology: keep quarantine when a vacuum commit re-adds a replica
SetVolumeAvailable re-adds the replica into the location list; a re-added
slot starts with a cleared read-only flag, so a quarantined replica came
back flag-free and the next ensureCorrectWritables could return a
conflicting volume to the writable list. Pass the quarantine state in and
set the flag back under the same accessLock, with the caller holding
quarantineLock so release cannot interleave between the check and the mark.
* topology: merge quarantine records instead of overwriting them
enforceQuarantine wrote back the state copy it loaded before its remote
push, so a concurrent writer's enforced=true was lost: the server stayed
read-only while the record said unenforced, and a later release skipped
the writable push it owed. The write-back now reloads the record under
quarantineLock and merges -- enforced wins from either side, reported is
taken from the newer pre-enforcement observation only while no push has
landed yet.
All three compensating writable pushes had the same hole in reverse: a
failed un-quarantine left the server read-only with no record to retry
from, and release's restore blindly overwrote any record a fresh flag
pass had created in between. They now go through restoreQuarantineRecord,
which adds the record only when absent and otherwise sets enforced on
the current one, preserving its reported flag.
* topology: hold quarantineLock over registration's quarantine check-and-mark
RegisterVolume reads the quarantine record and marks the replica
read-only under accessLock, but the pair ran without quarantineLock.
A release could push writable, watch the record get re-read, and then
delete it: the replica stayed read-only with no record left to clear,
and unchanged writable heartbeats skip SetReplicaReadOnlyFlag so the
mark never repaired. Holding quarantineLock around the check-and-mark
in all three production callers serializes it against the release
clear-and-delete.
* topology: serialize remote quarantine pushes behind a record recheck
releaseResolvedQuarantine deletes the record under quarantineLock and
sends the writable request after unlock. A conflict flagged while that
request is in flight creates a fresh record and marks the replica
read-only again, and the older writable request can still land last,
leaving the server writable under an active quarantine. The reverse
flap strands a stale read-only mark the same way.
Route every QuarantineVolumeFn call through pushQuarantine, which
holds quarantinePushLock across the remote call and re-checks the
record under quarantineLock right before sending: a read-only push
requires the pair to exist, a writable push requires it to be gone.
A skipped push means the decision already moved on, so the final
server state always follows the newest record.
* topology: serialize quarantine pushes per replica, not globally
A single quarantinePushLock held across the remote mark call let one slow
volume server (up to the 10s client timeout) delay pushes for unrelated
replicas. Keying the lock by (node, vid) keeps pushes to one replica
ordered in record-decision order while pushes to others run in parallel.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: restore the layout read-only mark with a released record
A failed writable push re-armed the quarantine record but left the
replica marked writable in the layout, so PickForWrite could route to a
replica the server still holds read-only.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: wait out async duplicate checks in the release-retry test
TestFailedReleasePushRestoresQuarantineMark asserts layout marks around a
release pass, but the two RegisterVolumeLayout calls each start a
background flagDuplicateVolumeId that can still be mid-flight. A late
mark could land after the final release and fail the writable assertion
even though the code behaved correctly. Track the async passes in a
WaitGroup and drain them before the explicit flag call so the rest of
the test is deterministic.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: keep quarantine when a replica's layout key is unreadable
* topology: keep a failed writable push's restore ordered under the push lock
* topology: retry the quarantine release on every duplicate check
* ci: isolate Java and S3 test servers on persistent runners
* ci: drop the duplicated -ip flags in the lifecycle test server
The bind flags were already on the command; the isolation pass added a
second pair.
* topology: retry a failed quarantine writable push on the next report
* topology: retry pending quarantine releases off the leader pulse
A failed writable push re-arms the quarantine record with a retry that only
enforceQuarantine fires, and that runs only for volumes a heartbeat carries.
Once the server has reported the replica read-only it drops out of the delta
reports, and restoreQuarantineRecord keeps the stored flag matching the
server's so the digests agree and no full list is requested - the pending
release then never retries and the idle volume stays read-only forever. Run
the same release pass from the leader's pulse loop so a pending retry no
longer waits on a report.
* topology: exercise the pulse-driven quarantine release retry
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A replica's read-only flag in the volume layout only moved on registration
and on volume.mark. A change that arrived in the regular heartbeat updated
the node's record, which the writable list follows, but not the layout
flag, which the vacuum sweep reads. So the sweep kept trying volumes on a
disk that had gone read-only while the server ran, and after a restart it
skipped volumes that had since become writable again until the next
restart (issue #11516).
Apply the reported state to the flag for every changed volume. Only the
flag: the writable list stays with EnsureCorrectWritables and its
capacity guards.
* master: delete replica_placement_mismatch labels when volumes leave topology
Fixes#10804. Setting the gauge to 0 left stale Prometheus time series
that grew unbounded with volume churn; remove the label set on unregister
instead.
* master: delete replica_placement_mismatch only after last placement leaves
Unconditional DeleteLabelValues on UnRegisterVolumeLayout dropped the series
while other data nodes still held the volume, hiding under-replication until
the next collect cycle. Delete only when Lookup is empty, and cover the
two-copy case in a regression test.
* topology: refresh oversized mark on every heartbeat
The oversized flag on a volume location was only set when the volume was
registered (RegisterVolume). A volume that later grew past the size limit
kept its stale "not oversized" mark, so the heartbeat path
(ensureCorrectWritables) kept re-adding it to the writable list while
RecordAssign removed it on every assign - a writable/unwritable flip loop
that let writes continue past the limit and made vacuum race in-flight
writes.
Refresh the mark from each heartbeat's reported size in both heartbeat
paths (ApplyVolumeChanges and SyncDataNodeRegistration), mirroring what
RegisterVolume already did at registration time. A volume that grew past
the limit now stays unwritable, and one that shrank back clears the mark
and can recover.
* topology: order heartbeat writable correction after decay and honor cooldown
Review feedback (Greptile, CodeRabbit) on the oversized-mark refresh:
1. Greptile: clearing the oversized mark before EnsureCorrectWritables let
the delay-unaware helper re-add a just-compacted volume to writables,
bypassing capacityRecoveryDelay. ensureCorrectWritables now checks
fullSince and skips the re-add while the cooldown is pending, so a
volume removed for capacity only recovers through UpdateVolumeSize's
heartbeat recovery path.
2. CodeRabbit: in the full-heartbeat path the mark was refreshed after
the writable correction, so a newly oversized volume stayed writable
for an extra heartbeat cycle. The standalone changedVolumes loop is
merged into the volumeInfos loop and EnsureCorrectWritables now runs
after UpdateOversizedState + UpdateVolumeSize in both heartbeat paths,
using the freshly refreshed mark.
3. TestHandlingVolumeServerHeartbeat used a size (254320) that is past
the test's volumeSizeLimit (32768); it only passed because the stale
mark hid the oversized state. Sized down to 30000 to keep testing the
add/remove flow, and added TestEnsureCorrectWritablesHonorsRecoveryCooldown
covering the cooldown window and the recovery after it.
* topology: do not restore a still-crowded volume after the cooldown
Greptile review: after capacityRecoveryDelay elapses, ensureCorrectWritables
could restore a volume whose effective size is still past the crowded
threshold. UpdateVolumeSize refuses the recovery (effectiveSize > crowded
threshold -> setVolumeCrowded + return false), but the cooldown check in
ensureCorrectWritables only looked at fullSince, so once the delay passed
it re-added the volume even though capacity tracking still considers it
crowded.
Check the crowded mark before re-adding: a volume UpdateVolumeSize just
marked crowded must not be restored here, otherwise assignments resume
while the volume is still flagged for growth.
Adds TestEnsureCorrectWritablesDoesNotRestoreCrowdedVolume: effectiveSize
decays to 10500 (past the 9000 crowded threshold) after a report of 8000,
and ensureCorrectWritables keeps the volume unwritable past the cooldown.
* ci: trigger re-run of flaky FUSE jobs
* topology: gate the writable restore on the limit, not on crowded
A crowded volume is above the growth threshold, not full, and is normally
writable. Refusing to restore one locks it out for good: nothing writes to
a volume that is not writable, so its size can never fall back under the
threshold. Gate on the same size the assign path uses to remove it.
* topology: let only the heartbeat refresh set the oversized mark
Registration also set it, from whatever VolumeInfo it was handed. The
incremental path builds that from a short heartbeat message, which carries
no size, so every arrival announcement cleared the mark and handed the
volume back to the writable list until the next full report.
* topology: use the re-resolved layout after a dropped one is replaced
A layout dropped with its collection makes RegisterVolume refuse, and the
full heartbeat then re-registered against a fresh layout but kept applying
the size, oversized and writable updates to the dropped one.
---------
Co-authored-by: hzsunchao <hzsunchao@corp.netease.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* master: let VolumeList ask for the volumes it wants
The request carried nothing, so every caller was answered with the whole
cluster. A dashboard opening one volume's page, or a capacity probe adding up
one bucket, was served all 800k of them and threw away the rest -- and the
master built every one of those messages first.
The topology, its disks and their counters are still reported in full: a caller
reading free space or replica placement needs the cluster whichever volumes it
asked about. Only what is listed under a disk is selected, ec shards included.
An empty collection and a zero volume id take everything, the way volume.list
already reads its own -collectionPattern and -volumeId, so a caller that
forgets to narrow is answered too much rather than answered wrongly. That
leaves the default collection unnameable, since it is the one the empty string
names, so it gets a field of its own.
An older client sends none of it and is answered exactly as before.
* admin: ask the master for the volume the page is showing
A volume's detail page was pulling every volume in the cluster to find one and
its replicas, and discarding the rest.
* admin: ask the master for the ec volume the page is showing
Same as the volume detail page: one volume's shards were found by pulling every
ec shard in the cluster.
* s3: ask the master for the bucket's own collection
The SOSAPI capacity probe summed one collection's volumes out of a listing of
every volume in the cluster. Cluster capacity still comes out the same: it is
read from the disk counters, which a filtered listing reports in full.
* topology: read the disk usage counters atomically
They are written with atomic.AddInt64 from heartbeats but were read plainly by
the two listings and by FreeSpace, and the map they sit in was iterated without
the lock its neighbour takes. Under -race a listing concurrent with a heartbeat
trips on both.
A master decides nothing from it. Every caller that read it was asking whether
a volume is remote, which the backend name answers, and the value itself is
reported on demand by the server holding the volume, through the volume info in
ReadVolumeFileStatus.
It is also the one string here that cannot be shared: unique per volume, so
unlike the collection and backend names it carries its own characters for every
volume a master tracks.
VolumeInfo goes from 136 bytes to 120. 800k volumes registered from a heartbeat
that has been over the wire go from 214 to 163 B/volume when tiered.
The volume server's own status page keeps showing the key, now read from the
volume it holds rather than relayed through a master, which is also where the
other volume server implementation reads it.
The heartbeat digest drops it on the same grounds: a change to something the
master does not hold cannot make its copy stale. Both implementations and their
shared vectors move together, and the field-coverage test now names what is
deliberately not retained rather than being loosened.
* master: re-create a vacuum-committed volume the index has lost
SetVolumeAvailable dereferenced vid2location[vid] with no nil check. A
disconnect during a long vacuum can drop a single-replica volume from the
lookup index while it stays on the node; the commit then panics the master
instead of re-registering it. Re-create the entry, and seed its size tracking
so assigns are counted right away rather than after the next heartbeat.
* master: re-register a vacuumed volume the index lost on mark-writable
The maintenance worker re-enables a volume by marking it writable after the
vacuum commit, but VolumeMarkReadonly only updated nodes the lookup index
already knew. If a disconnect race dropped the volume from the index during the
vacuum, that path was a no-op and the volume stayed "not found" until the next
full heartbeat healed it. Re-register it from the node that still holds it.
A disconnect/reconnect race could drop a volume from vid2location while it stayed in the data node's disk map, so it showed in volume.list and the admin UI but LookupVolume returned "volume id not found" and never self-healed (the full heartbeat only registered volumes new to the disk map). The full heartbeat now re-registers any reported volume missing from the lookup index, reusing the already-resolved VolumeLayout.
Ping previously dialled whatever host:port the caller asked for. Gate
each server's Ping handler on cluster membership: masters check the
topology, registered cluster nodes, and configured master peers; volume
servers only accept their seed/current masters; filers accept tracked
peer filers, the master-learned volume server set, and configured
masters.
Use address-indexed peer lookups to keep Ping target validation O(1):
- topology maintains a pb.ServerAddress -> *DataNode index alongside
the dc/rack/node tree, kept in sync from doLinkChildNode and
UnlinkChildNode plus the ip/port-rewrite branch in
GetOrCreateDataNode. GetTopology now returns nil on a detached
subtree instead of panicking, so the linkage hooks can no-op safely.
- vid_map tracks a refcount per volume-server address so
hasVolumeServer answers without scanning every vid location. The
add path skips empty-address entries the same way the delete path
already does, so a zero-value Location cannot leak a permanent
serverRefCount[""] bucket.
- masters reuse a cached master-address set from MasterClient instead
of walking the configured peer slice on every request.
- volume servers compare against a pre-built seed-master set and
protect currentMaster reads/writes with an RWMutex, fixing the
data race with the heartbeat goroutine. The seed slice is copied
on construction so external mutation cannot desync it from the
frozen lookup set.
- cluster.check drops the direct volume-to-volume sweep; volume
servers no longer carry a peer-volume list, and the note next to
the dropped probe is reworded to make clear that direct
volume-to-volume reachability is intentionally not validated by
this command.
Update the volume-server integration tests that drove Ping through the
new admission gate: success-path coverage now targets the master peer
(the only type a volume server tracks), and the unknown/unreachable
path asserts the InvalidArgument the gate now returns instead of the
old downstream dial error.
Mirror the same admission gate in the Rust volume server crate: a
seed-master HashSet built once at startup plus a tokio RwLock over the
heartbeat-tracked current master, both consulted in is_known_ping_target
on every Ping, with InvalidArgument returned for any target that isn't
a recognised master.
* admin: add plugin runtime UI page and route wiring
* pb: add plugin gRPC contract and generated bindings
* admin/plugin: implement worker registry, runtime, monitoring, and config store
* admin/dash: wire plugin runtime and expose plugin workflow APIs
* command: add flags to enable plugin runtime
* admin: rename remaining plugin v2 wording to plugin
* admin/plugin: add detectable job type registry helper
* admin/plugin: add scheduled detection and dispatch orchestration
* admin/plugin: prefetch job type descriptors when workers connect
* admin/plugin: add known job type discovery API and UI
* admin/plugin: refresh design doc to match current implementation
* admin/plugin: enforce per-worker scheduler concurrency limits
* admin/plugin: use descriptor runtime defaults for scheduler policy
* admin/ui: auto-load first known plugin job type on page open
* admin/plugin: bootstrap persisted config from descriptor defaults
* admin/plugin: dedupe scheduled proposals by dedupe key
* admin/ui: add job type and state filters for plugin monitoring
* admin/ui: add per-job-type plugin activity summary
* admin/plugin: split descriptor read API from schema refresh
* admin/ui: keep plugin summary metrics global while tables are filtered
* admin/plugin: retry executor reservation before timing out
* admin/plugin: expose scheduler states for monitoring
* admin/ui: show per-job-type scheduler states in plugin monitor
* pb/plugin: rename protobuf package to plugin
* admin/plugin: rename pluginRuntime wiring to plugin
* admin/plugin: remove runtime naming from plugin APIs and UI
* admin/plugin: rename runtime files to plugin naming
* admin/plugin: persist jobs and activities for monitor recovery
* admin/plugin: lease one detector worker per job type
* admin/ui: show worker load from plugin heartbeats
* admin/plugin: skip stale workers for detector and executor picks
* plugin/worker: add plugin worker command and stream runtime scaffold
* plugin/worker: implement vacuum detect and execute handlers
* admin/plugin: document external vacuum plugin worker starter
* command: update plugin.worker help to reflect implemented flow
* command/admin: drop legacy Plugin V2 label
* plugin/worker: validate vacuum job type and respect min interval
* plugin/worker: test no-op detect when min interval not elapsed
* command/admin: document plugin.worker external process
* plugin/worker: advertise configured concurrency in hello
* command/plugin.worker: add jobType handler selection
* command/plugin.worker: test handler selection by job type
* command/plugin.worker: persist worker id in workingDir
* admin/plugin: document plugin.worker jobType and workingDir flags
* plugin/worker: support cancel request for in-flight work
* plugin/worker: test cancel request acknowledgements
* command/plugin.worker: document workingDir and jobType behavior
* plugin/worker: emit executor activity events for monitor
* plugin/worker: test executor activity builder
* admin/plugin: send last successful run in detection request
* admin/plugin: send cancel request when detect or execute context ends
* admin/plugin: document worker cancel request responsibility
* admin/handlers: expose plugin scheduler states API in no-auth mode
* admin/handlers: test plugin scheduler states route registration
* admin/plugin: keep worker id on worker-generated activity records
* admin/plugin: test worker id propagation in monitor activities
* admin/dash: always initialize plugin service
* command/admin: remove plugin enable flags and default to enabled
* admin/dash: drop pluginEnabled constructor parameter
* admin/plugin UI: stop checking plugin enabled state
* admin/plugin: remove docs for plugin enable flags
* admin/dash: remove unused plugin enabled check method
* admin/dash: fallback to in-memory plugin init when dataDir fails
* admin/plugin API: expose worker gRPC port in status
* command/plugin.worker: resolve admin gRPC port via plugin status
* split plugin UI into overview/configuration/monitoring pages
* Update layout_templ.go
* add volume_balance plugin worker handler
* wire plugin.worker CLI for volume_balance job type
* add erasure_coding plugin worker handler
* wire plugin.worker CLI for erasure_coding job type
* support multi-job handlers in plugin worker runtime
* allow plugin.worker jobType as comma-separated list
* admin/plugin UI: rename to Workers and simplify config view
* plugin worker: queue detection requests instead of capacity reject
* Update plugin_worker.go
* plugin volume_balance: remove force_move/timeout from worker config UI
* plugin erasure_coding: enforce local working dir and cleanup
* admin/plugin UI: rename admin settings to job scheduling
* admin/plugin UI: persist and robustly render detection results
* admin/plugin: record and return detection trace metadata
* admin/plugin UI: show detection process and decision trace
* plugin: surface detector decision trace as activities
* mini: start a plugin worker by default
* admin/plugin UI: split monitoring into detection and execution tabs
* plugin worker: emit detection decision trace for EC and balance
* admin workers UI: split monitoring into detection and execution pages
* plugin scheduler: skip proposals for active assigned/running jobs
* admin workers UI: add job queue tab
* plugin worker: add dummy stress detector and executor job type
* admin workers UI: reorder tabs to detection queue execution
* admin workers UI: regenerate plugin template
* plugin defaults: include dummy stress and add stress tests
* plugin dummy stress: rotate detection selections across runs
* plugin scheduler: remove cross-run proposal dedupe
* plugin queue: track pending scheduled jobs
* plugin scheduler: wait for executor capacity before dispatch
* plugin scheduler: skip detection when waiting backlog is high
* plugin: add disk-backed job detail API and persistence
* admin ui: show plugin job detail modal from job id links
* plugin: generate unique job ids instead of reusing proposal ids
* plugin worker: emit heartbeats on work state changes
* plugin registry: round-robin tied executor and detector picks
* add temporary EC overnight stress runner
* plugin job details: persist and render EC execution plans
* ec volume details: color data and parity shard badges
* shard labels: keep parity ids numeric and color-only distinction
* admin: remove legacy maintenance UI routes and templates
* admin: remove dead maintenance endpoint helpers
* Update layout_templ.go
* remove dummy_stress worker and command support
* refactor plugin UI to job-type top tabs and sub-tabs
* migrate weed worker command to plugin runtime
* remove plugin.worker command and keep worker runtime with metrics
* update helm worker args for jobType and execution flags
* set plugin scheduling defaults to global 16 and per-worker 4
* stress: fix RPC context reuse and remove redundant variables in ec_stress_runner
* admin/plugin: fix lifecycle races, safe channel operations, and terminal state constants
* admin/dash: randomize job IDs and fix priority zero-value overwrite in plugin API
* admin/handlers: implement buffered rendering to prevent response corruption
* admin/plugin: implement debounced persistence flusher and optimize BuildJobDetail memory lookups
* admin/plugin: fix priority overwrite and implement bounded wait in scheduler reserve
* admin/plugin: implement atomic file writes and fix run record side effects
* admin/plugin: use P prefix for parity shard labels in execution plans
* admin/plugin: enable parallel execution for cancellation tests
* admin: refactor time.Time fields to pointers for better JSON omitempty support
* admin/plugin: implement pointer-safe time assignments and comparisons in plugin core
* admin/plugin: fix time assignment and sorting logic in plugin monitor after pointer refactor
* admin/plugin: update scheduler activity tracking to use time pointers
* admin/plugin: fix time-based run history trimming after pointer refactor
* admin/dash: fix JobSpec struct literal in plugin API after pointer refactor
* admin/view: add D/P prefixes to EC shard badges for UI consistency
* admin/plugin: use lifecycle-aware context for schema prefetching
* Update ec_volume_details_templ.go
* admin/stress: fix proposal sorting and log volume cleanup errors
* stress: refine ec stress runner with math/rand and collection name
- Added Collection field to VolumeEcShardsDeleteRequest for correct filename construction.
- Replaced crypto/rand with seeded math/rand PRNG for bulk payloads.
- Added documentation for EcMinAge zero-value behavior.
- Added logging for ignored errors in volume/shard deletion.
* admin: return internal server error for plugin store failures
Changed error status code from 400 Bad Request to 500 Internal Server Error for failures in GetPluginJobDetail to correctly reflect server-side errors.
* admin: implement safe channel sends and graceful shutdown sync
- Added sync.WaitGroup to Plugin struct to manage background goroutines.
- Implemented safeSendCh helper using recover() to prevent panics on closed channels.
- Ensured Shutdown() waits for all background operations to complete.
* admin: robustify plugin monitor with nil-safe time and record init
- Standardized nil-safe assignment for *time.Time pointers (CreatedAt, UpdatedAt, CompletedAt).
- Ensured persistJobDetailSnapshot initializes new records correctly if they don't exist on disk.
- Fixed debounced persistence to trigger immediate write on job completion.
* admin: improve scheduler shutdown behavior and logic guards
- Replaced brittle error string matching with explicit r.shutdownCh selection for shutdown detection.
- Removed redundant nil guard in buildScheduledJobSpec.
- Standardized WaitGroup usage for schedulerLoop.
* admin: implement deep copy for job parameters and atomic write fixes
- Implemented deepCopyGenericValue and used it in cloneTrackedJob to prevent shared state.
- Ensured atomicWriteFile creates parent directories before writing.
* admin: remove unreachable branch in shard classification
Removed an unreachable 'totalShards <= 0' check in classifyShardID as dataShards and parityShards are already guarded.
* admin: secure UI links and use canonical shard constants
- Added rel="noopener noreferrer" to external links for security.
- Replaced magic number 14 with erasure_coding.TotalShardsCount.
- Used renderEcShardBadge for missing shard list consistency.
* admin: stabilize plugin tests and fix regressions
- Composed a robust plugin_monitor_test.go to handle asynchronous persistence.
- Updated all time.Time literals to use timeToPtr helper.
- Added explicit Shutdown() calls in tests to synchronize with debounced writes.
- Fixed syntax errors and orphaned struct literals in tests.
* Potential fix for code scanning alert no. 278: Slice memory allocation with excessive size value
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Potential fix for code scanning alert no. 283: Uncontrolled data used in path expression
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* admin: finalize refinements for error handling, scheduler, and race fixes
- Standardized HTTP 500 status codes for store failures in plugin_api.go.
- Tracked scheduled detection goroutines with sync.WaitGroup for safe shutdown.
- Fixed race condition in safeSendDetectionComplete by extracting channel under lock.
- Implemented deep copy for JobActivity details.
- Used defaultDirPerm constant in atomicWriteFile.
* test(ec): migrate admin dockertest to plugin APIs
* admin/plugin_api: fix RunPluginJobTypeAPI to return 500 for server-side detection/filter errors
* admin/plugin_api: fix ExecutePluginJobAPI to return 500 for job execution failures
* admin/plugin_api: limit parseProtoJSONBody request body to 1MB to prevent unbounded memory usage
* admin/plugin: consolidate regex to package-level validJobTypePattern; add char validation to sanitizeJobID
* admin/plugin: fix racy Shutdown channel close with sync.Once
* admin/plugin: track sendLoop and recv goroutines in WorkerStream with r.wg
* admin/plugin: document writeProtoFiles atomicity — .pb is source of truth, .json is human-readable only
* admin/plugin: extract activityLess helper to deduplicate nil-safe OccurredAt sort comparators
* test/ec: check http.NewRequest errors to prevent nil req panics
* test/ec: replace deprecated ioutil/math/rand, fix stale step comment 5.1→3.1
* plugin(ec): raise default detection and scheduling throughput limits
* topology: include empty disks in volume list and EC capacity fallback
* topology: remove hard 10-task cap for detection planning
* Update ec_volume_details_templ.go
* adjust default
* fix tests
---------
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* pb: add id field to Heartbeat message for stable volume server identification
This adds an 'id' field to the Heartbeat protobuf message that allows
volume servers to identify themselves independently of their IP:port address.
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* storage: add Id field to Store struct
Add Id field to Store struct and include it in CollectHeartbeat().
The Id field provides a stable volume server identity independent of IP:port.
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* topology: support id-based DataNode identification
Update GetOrCreateDataNode to accept an id parameter for stable node
identification. When id is provided, the DataNode can maintain its identity
even when its IP address changes (e.g., in Kubernetes pod reschedules).
For backward compatibility:
- If id is provided, use it as the node ID
- If id is empty, fall back to ip:port
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* volume: add -id flag for stable volume server identity
Add -id command line flag to volume server that allows specifying a stable
identifier independent of the IP address. This is useful for Kubernetes
deployments with hostPath volumes where pods can be rescheduled to different
nodes while the persisted data remains on the original node.
Usage: weed volume -id=node-1 -ip=10.0.0.1 ...
If -id is not specified, it defaults to ip:port for backward compatibility.
Fixes https://github.com/seaweedfs/seaweedfs/issues/7487
* server: add -volume.id flag to weed server command
Support the -volume.id flag in the all-in-one 'weed server' command,
consistent with the standalone 'weed volume' command.
Usage: weed server -volume.id=node-1 ...
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* topology: add test for id-based DataNode identification
Test the key scenarios:
1. Create DataNode with explicit id
2. Same id with different IP returns same DataNode (K8s reschedule)
3. IP/PublicUrl are updated when node reconnects with new address
4. Different id creates new DataNode
5. Empty id falls back to ip:port (backward compatibility)
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* pb: add address field to DataNodeInfo for proper node addressing
Previously, DataNodeInfo.Id was used as the node address, which worked
when Id was always ip:port. Now that Id can be an explicit string,
we need a separate Address field for connection purposes.
Changes:
- Add 'address' field to DataNodeInfo protobuf message
- Update ToDataNodeInfo() to populate the address field
- Update NewServerAddressFromDataNode() to use Address (with Id fallback)
- Fix LookupEcVolume to use dn.Url() instead of dn.Id()
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* fix: trim whitespace from volume server id and fix test
- Trim whitespace from -id flag to treat ' ' as empty
- Fix store_load_balancing_test.go to include id parameter in NewStore call
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* refactor: extract GetVolumeServerId to util package
Move the volume server ID determination logic to a shared utility function
to avoid code duplication between volume.go and rack.go.
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* fix: improve transition logic for legacy nodes
- Use exact ip:port match instead of net.SplitHostPort heuristic
- Update GrpcPort and PublicUrl during transition for consistency
- Remove unused net import
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* fix: add id normalization and address change logging
- Normalize id parameter at function boundary (trim whitespace)
- Log when DataNode IP:Port changes (helps debug K8s pod rescheduling)
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487