mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-04 11:07:35 +01:00
995084f71ef4bffb96d03da31b929845e92a6aac
342
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
5dfda3645e |
admin: count executor slots per job type and stop lanes waiting on capacity they cannot free (#12384)
* admin: count executor slots per job type, deduped against the heartbeat per_worker_execution_concurrency is a per-job-type setting, but the scheduler compared it against the worker's execution_slots_used, which counts every job type. One s3_lifecycle job a worker reported running after a reconnect made admin_script (per-worker 1) undispatchable on a worker with three free slots. A reservation and the heartbeat entry for the same job were also charged as two slots, so a per-worker limit of N let N-1 jobs run. Reservations now carry the job id and job type. The per-type limit is checked against that type's work alone, the worker's slot total against work of every type, and a heartbeat entry covered by a reservation is counted once. Per-lane reservation pools go away: the per-type count is what kept lanes from starving each other. * admin: stop a lane waiting on executor capacity it cannot free A queued job no executor could take was retried every backoff until job_type_max_runtime ran out, up to 30 minutes, while the default lane sat in executing and every other job type in it idled. The dispatch loop now asks what holds the capacity. Slots held by the lane's own jobs release when they complete, so it keeps waiting. Slots held by other lanes or by work this admin never dispatched clear on a schedule the lane has no view of, so the queued jobs are canceled at once with a skipped_no_executor activity naming the worker, its slot count and the job types holding it, and the next detection proposes them again. A missing worker is waited for only the orphan grace. * admin: cancel a job on its worker when the admin expires it Expiring a job, by the stale sweep or by the operator's expire endpoint, only failed the tracked record. The worker never heard about it and kept running the job with its slot taken, so a job stuck in "starting" held a slot for over a day and, with the old slot accounting, blocked a whole lane. Expiring a job now also sends the worker a cancel for it, so the slot comes back on the next heartbeat. * admin: count a completed job's heartbeat echo as its lane's own slot The worker sends JobCompleted before the heartbeat that frees the slot, so right after a dispatch goroutine releases its reservation the job still shows in the worker's running work. That entry was counted as foreign, and a lane with per-worker 1 gave up and canceled the rest of its queue after the first job. A released reservation now stays for completedJobEchoGrace and claims that heartbeat entry as the lane's own load. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: let the admin's own jobs claim slots a worker does not itemize The Rust worker reports only a slot count, every 10s, and the Go worker counts a slot one heartbeat before it lists the job. Those slots were charged on top of the reservations behind them, and once a job completed its stale count read as foreign work, so the lane gave up its queue. Unitemized slots now go first to this admin's unlisted reservations and recent completions; the rest are of unknown type and count against every type's per-worker limit, not only the slot total. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: give up pinned jobs whose busy home a free worker cannot relieve A job homed on a replica-host worker stays on that home while its queue is shorter than LocalQueuePin, so a free remote worker cannot take it. The outlook treated that free worker as a reason to wait, and a home held by foreign work parked the lane for its whole window. A free worker now only holds the wait for the orphan grace, after which the held homes decide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: let a redispatched job's reservation replace its released one A job the worker turns away for capacity is released, kept as an echo, and queued again under the same id. Once the new attempt ran, both the echo and the new reservation matched its heartbeat entry and it counted as two slots. Reserving a job now drops what an earlier reservation of it left. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: credit a count-only slot to a completed job only until the next heartbeat A released reservation claimed an unitemized slot for its full 30s grace, so on a count-only heartbeat other work of another type was charged to the completed job's type. A heartbeat that arrives after the release already reflects the freed slot; from then on the slot is of unknown type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: name unitemized slots in the no-capacity reason holders() listed only the job types the worker itemized, so a worker with named and count-only work read as "2/2 slots used by <type>". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep the blocked clock running until a reservation succeeds A popped job whose reservation then failed reset the clock before the failure pushed it back, so pop-then-fail cycles never aged the block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: resend an expiry cancel the worker did not receive Expiring a job sends its worker one cancel. When the worker is disconnected, or its outgoing buffer stays full past the send timeout, that cancel is dropped. The record is already failed, so later sweeps and expire calls skip it, and a worker that kept the job running (a detached job across a reconnect, or a jammed stream) holds the slot until the job ends on its own. The admin now holds an undelivered expiry cancel and sends it again on the worker's next heartbeat that still lists the job. It is forgotten when that heartbeat no longer lists the job, and skipped when the job is active again, since a new attempt reuses the job id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: cancel an expired attempt by its request id The expiry cancel named the job id, and a retry reuses it. The worker keys its cancel funcs by both ids, so a cancel arriving after a retry started on the same worker stopped the retry. The held resend also gave up on any job that was active again, and the expired attempt's own late progress marks the job running, so a worker that buffered progress while disconnected kept the expired job. The cancel now names the attempt's request id, as the other execution cancels do, and the held resend no longer depends on the tracked state. Only a cancel that can name nothing but the job id still skips a job that is active again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: ignore late reports from an attempt the admin already ended Expiring a job now cancels it on its worker, and the worker answers with its own JobCompleted for that attempt. That overwrote the expiry's failure with the cancel error and logged a second run record under the same run id. Late progress from the expired attempt likewise reopened the record as running. A progress update or completion whose request id names an attempt the record already closed is now dropped after the dispatcher's waiter is told. A retry runs under a new request id, so its reports still apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris Lu
|
ab6aa9b645 |
admin: let a buffered detection completion win over the stream closing (#12379)
The detection waiter selects on the request's completion channel and on the stream's done channel. A worker that finishes a scan and then drops its stream (a restart right after a scan is the usual shape) buffers the completion into the request before the receive loop sees the EOF, and the stream's cleanup then closes done. A waiter that reaches its select after both are ready can take the disconnect branch, and a completed detection, proposals included, is failed as "detection worker disconnected" and retried or recorded as an error. When the stream closes or the liveness tick finds the worker gone, the waiter now takes a completion that is already queued and returns it as the success it is; the detector lease still goes, since the worker that answered is unreachable. A ctx that ended keeps its meaning: the caller stopped wanting the answer. The wait loop moves into awaitDetectionOutcome so a test can hand it both channels already ready; before this change that test fails on about half its runs. A completion that lands after the waiter gave up stays unmatched, since the request's pending entry is gone and the next request has a new id. |
||
|
Chris Lu
|
30f0364256 |
admin: fail an in-flight detection when its worker disconnects (#12376)
* admin: keep a reconnected plugin worker registered when its old stream unwinds cleanupSession removed whatever session was registered under the worker ID, so a worker that reconnected while its previous stream was still unwinding lost the new session and its registry entry to the old stream's deferred cleanup. Tie cleanup to the stream that is ending and publish the new stream before the registry upsert so a lingering predecessor sees itself superseded. * admin: fail an in-flight detection when its worker disconnects RunDetectionWithReport waited only on the caller's context and the DetectionComplete channel, so a detection whose worker stream ended mid-request could only finish by timeout. On a customer cluster a planned restart of ec-vacuum-1 during an ec_vacuum detection left the default lane in 'detecting' with every execution slot idle until the 3600s detection timeout, blocking every job type in that lane. Bind the request to the stream it was sent on and fail the detection as soon as that stream closes, whether by disconnect or by a reconnect that replaced the session. A liveness tick also fails it once the registry no longer considers the worker available (heartbeat stale), sending a cancel in that case because the stream may still be open. The detector lease is dropped so the next pick starts fresh, and a 'disconnected' detector activity is recorded. WorkerDisconnectForTest now goes through the real session cleanup. * admin: retry a scheduled detection once when its detector drops When a detection fails because its worker disconnected, pick a detector again and retry once within the same iteration while the job type's runtime budget allows, instead of giving the whole cycle up. Any other failure, or a failed retry, ends the run as before and the lane moves on; the next cycle re-detects anyway. * admin: register a plugin worker's hello and stream as one step WorkerStream published the stream under sessionsMu but ran the registry upsert after releasing it, so two concurrent streams for one worker ID could publish their hellos in the opposite order from their sessions: the registry then described the older hello's capabilities while only the newer stream was sendable, and detector picks routed on them. registerSession upserts the hello and installs the stream under one sessionsMu critical section, and a hello or heartbeat arriving on a stream that is no longer the registered one is dropped instead of rewriting the live stream's entry. * admin: steer a detection retry off the stream that dropped The retry picked while the lost worker could still be in the registry: cleanupSession closed done before taking the registry entry down, and a stream closed by a reconnect keeps its entry on purpose. The lost worker ranks first on free slots, so the single retry went back to it and failed on send while a healthy worker sat idle. The disconnect error now carries the stream it came from and the retry picks around it; pickDetector only returns workers whose current stream is open, excluding registry entries whose stream is closed; and cleanupSession removes the registry entry before closing done, so a waiter woken by the closure never sees the dead worker as a candidate. |
||
|
Chris LuandClaude Opus 5.5
|
2ec0623073 |
filer UI, admin: upload a folder (#12371)
* admin: place uploaded files by their path inside an uploaded folder The multipart file name keeps only the base name, so the upload API takes an optional relativePaths field listing each file's path under the target directory. Paths that are empty, absolute or contain . or .. components are rejected per file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: upload a folder from the file browser The upload modal gets a folder picker, and a folder dropped on the listing is walked into, so its files upload with their subfolder paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer UI: upload a folder An Upload Folder button picks a whole folder, and a folder dropped on the listing is walked into. Each file is posted to its subfolder under the current directory, which the filer creates on the first upload into it. Progress rows are keyed by that path, so equal names in different subfolders no longer share a row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: reject an empty relativePaths field instead of ignoring it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: upload a large selection in batches below the multipart part limit Go's multipart parser rejects a form of more than 1000 parts, so a folder of about a thousand files failed as a whole. The upload modal now sends 500 files per request and sums the results; progress covers all batches, and the button stays disabled until the last one finishes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: escape file names in upload error messages A failed upload reports the client-supplied file or folder path, and the toast renders its message as HTML. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: list failed uploads in the modal and reload the listing when it closes A partial upload logged its failures only to the console before the page reloaded, and a batch that failed after earlier ones succeeded left the listing stale. The failures now stay in the open modal, and closing it reloads the listing once any file was uploaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: cancel the upload when its modal closes Closing the modal, including with its Cancel button, left the batch loop running, so the remaining batches were still written. The batch in flight is now aborted, which also stops the loop, and the listing reloads since the aborted request may already have stored files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: count upload progress by files when the selection has no bytes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep the newest upload selection when a folder drop finishes late Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: cap the upload preview at a fixed number of rows A folder of many thousands of files rendered one preview row per file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer UI: upload a few files at a time Every selected file started its own request at once, so a large folder opened thousands of requests together. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: regenerate the gzipped admin.js Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: reload the listing after a batch fails without a result A batch that fails mid-request may still have stored some of its files, so closing the modal reloads the listing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: stop reading a folder drop once a newer selection replaces it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: skip the read error of a folder drop a newer selection replaced Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer UI: share the four-upload limit across selections Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: regenerate the gzipped admin.js Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: set up the file browser handlers once admin.js already runs setupFileManagerEventHandlers on DOMContentLoaded, so the file browser page registered every listener twice and each drop was read twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
e8b284fef5 |
admin: scope plugin job and activity fetches to the selected job type (#12360)
* admin: scope plugin job and activity fetches to the selected job type The plugin page fetched the newest 500 jobs and 500 activities cluster-wide and filtered them by job type in the browser. On a busy cluster that window can hold none of the selected type (433 erasure_coding + 64 ec_balance jobs, 497 ec_balance activities), so /plugin/lanes/default/execution?job=ec_vacuum showed "No execution jobs tracked yet" while the server had seven recent ec_vacuum jobs. When a job-type tab is active, fetch the queue/detection/execution lists with job_type=<selected> so the server filters before applying the limit, and refetch when the selected job type changes. The unscoped lists are still fetched for the status cards and the overview per-type summary. * admin: drop a plugin monitor refresh whose job type is no longer selected A refresh started on one job-type tab could finish after the user moved to another tab and that tab's refresh had already landed. Its rows, all of the earlier job type, then replaced the current tab's and the renderers filtered them down to empty tables until the next refresh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: show the lane-wide rows when a job-type scoped fetch fails On a job-type tab a failed scoped jobs or activities request threw before anything was stored, so the status cards, scheduler state and tables all stayed stale. Fall back to the unscoped rows, which the renderers already filter by job type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: filter plugin jobs and activities by lane before the limit /api/plugin/jobs and /api/plugin/activities took the newest N rows across every lane and then dropped the other lanes' rows, so on a busy default lane a quieter lane's page could come back empty. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep a failed plugin monitor fetch's last rows and report it Falling back to the lane-wide rows when a scoped fetch failed could still blank the tables, since that window may hold none of the selected type, and the failure went unreported. A lane-wide or scheduler-state failure also threw before the scoped rows that did arrive were stored. Each request now keeps its previous value on failure: the scoped rows only while the tab is unchanged, otherwise the lane-wide rows. Whatever arrived is stored and rendered, then the first failure is rethrown so the caller reports it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: report a monitor refresh failure apart from the action it follows expireJob and runWorkflow refreshed the monitor inside the action's try block, so a failed refresh after a successful expire or workflow run was reported as a failed action, and the expired job's detail was not reloaded. Catch the refresh separately, as runDetection already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: show a repeated plugin page refresh failure once refreshAll runs every 3 seconds and notify opens a modal, so a request that kept failing reopened the same alert on every tick. Report an error only when it differs from the last one, and forget it once a refresh succeeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
222cfa8e2e |
admin: summarize the EC delete_count accounting warning per refresh (#12359)
* admin: summarize the EC delete_count accounting warning per refresh collectCollectionStats runs on every admin collection, bucket and topology refresh, and it logged a Warning for each EC volume whose delete_count summed across shard holders exceeds its file_count. That sum is an expected accounting artefact (a tombstone can be recorded on more than one holder), it recurs identically on every pass, and on one cluster with hundreds of such volumes it grew the admin log to 20 GB, more than 99% of it this one line. Log one V(1) summary per pass with the affected volume count and keep the per-volume detail at V(2). The object count is still skipped for those volumes. * admin: test that an EC volume with deletes above file_count is skipped Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b7b2a5947f |
admin: configurable file browser upload size limit (#12348)
* admin: carry a file browser upload size limit The admin server keeps the total size limit of one file browser upload, 500 MB by default as the UI enforced before, and hands it to the file browser page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: reject file browser uploads over the size limit The 500 MB limit was only checked in the browser, so any client could post larger uploads and fill the temp dir with spooled parts. Cap the request body at the limit plus room for the multipart framing, and answer 413 when the files add up to more than the limit. Parse the form before reading the path field. FormValue used to parse it first with the default 32 MB memory, which made the later 1 GB ParseMultipartForm a no-op; keep 32 MB explicitly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: show and check the configured upload limit in the file browser The upload dialog reads the limit from the page instead of a hard-coded 500 MB, lists it under the file picker, and skips the check when uploads are unlimited. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: add -uploadMaxSizeMB to set the file browser upload limit weed admin -uploadMaxSizeMB and weed mini -admin.uploadMaxSizeMB set the total size limit of one file browser upload; 0 removes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: note why the upload framing allowance is enough Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: test an upload whose files only exceed the limit together Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: remove spooled upload parts when the upload handler returns Parts over 32 MB spool to temp files. The router passes the handler a copy of the request, so net/http never sees the parsed form and never removes those files. Every large upload, and every one rejected for size after parsing, left them on disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a074538c8f |
admin: manage S3 vector buckets (#12334)
* s3vectors: run an operation for trusted local tooling Like s3tables.Manager.Execute, Server.Execute runs one S3 Vectors operation through its handler as a given identity, without a signed request. The admin console can then manage vector buckets through the same validation, authorization and catalog steps as the API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * admin: list, inspect, create and delete vector buckets over the API /api/s3vectors runs the S3 Vectors operations through Server.Execute as the admin account: list vector buckets with their encryption, read one bucket with its policy and each index's data type, dimension, distance metric and non-filterable keys, create and delete a bucket, and delete an index. Going through the operations keeps the API's rules, such as ICEBERG buckets not being vector buckets and the implicit index namespace not blocking a delete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: export the rule for which table buckets are vector buckets IsVectorBucketFormat names the check ListVectorBuckets and isVectorBucket already make, so the admin console can tell the same buckets apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * admin: vector bucket pages Object Store > Vector Buckets lists the vector buckets with their ARN, encryption and creation time, creates and deletes them, and opens one bucket with its indexes (data type, dimension, distance metric, non-filterable keys) and its policy, where an index can be deleted. A LANCE row on the Table Buckets page links to the same bucket's vector page, since both APIs serve it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * admin: stop the vector bucket and index lists at the list limit A page cut short by filtered ICEBERG buckets let the next page carry the list past s3TablesAdminListLimit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: say when a vector bucket or index list is cut at the limit The lists stop at s3TablesAdminListLimit while more remain, so mark them truncated and point to an S3 Vectors client for the rest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: indent the vector bucket policy A policy stored as one line ran off the right edge of the details page. Policies that are not valid JSON are shown as stored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: show a deeply nested vector bucket policy as stored Indenting multiplies the size of a deeply nested document, so a policy past 32 levels of nesting is shown without indenting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Mohd Quamar Tyagi
|
5f4d05b103 |
admin: measure volume server usage like its capacity (#12306)
The volume server views summed the logical size of the volumes into DiskUsage while DiskCapacity came from the filesystem statfs size. On a compressing filesystem the logical sum exceeds the physical capacity, so the server details popup derived a negative available space and rendered it as "NaN undefined". Pair each disk's capacity with the usage measured the same way, using the statfs numbers the volume server already heartbeats, as CollectTierStats does. Volume servers without statfs numbers keep the previous slot-based capacity and logical bytes, and EC shard bytes are only counted there instead of on top of the filesystem usage. |
||
|
Chris Lu
|
8f6db360da |
admin: warn on dashboard when cluster nodes run mixed versions (#12196)
* admin: warn on dashboard when cluster nodes run mixed versions
A partially-finished upgrade leaves servers on different releases, and
nothing surfaces that drift today. The dashboard now compares the
binary version reported by every node:
- volume servers are asked directly via VolumeServerStatus (the
heartbeat topology carries no version), fanned out with bounded
concurrency and a 60s cache since versions only change on restart
- filers, S3 servers, and message brokers already report their version
over KeepConnected; ListClusterNodes returns it, the dashboard just
never read the field
- masters are excluded: raft membership does not carry versions
When every node matches, the Cluster panel shows the shared version.
When they differ, a warning banner calls out the mixed cluster and the
panel lists each node type with its count grouped by version, e.g.
"31x 4.39, 24x 4.44". Version columns on the volume server, filer, and
S3 tables (and the dedicated volume servers page) identify which nodes
are stale. Versions are compared without the volume-size-limit prefix
("8000GB 4.44 abc123" -> "4.44 abc123") so capacity config does not
masquerade as version drift; nodes reporting nothing count as
"unknown".
* admin: run the volume version queries concurrently, not under the map lock
The per-server VolumeServerStatus call ran while holding the versions
mutex, so the semaphore-bounded fan-out was fully serialized: every
server's RPC waited for the previous one, up to the 3s timeout each.
* admin: export the volume server table's real columns
The new Version cell shifted every later column, so the CSV read the
version as Rack and dropped Max Volumes, EC Shards and Usage a slot
early. Map the cells the table actually renders and add Version to the
export.
* admin: warn on mixed versions only when nodes report versions
An empty or master-only cluster produced a non-uniform summary with an
empty node list, which rendered the "Mixed cluster versions detected"
banner and warning icon over nothing. Mixed() requires reported nodes,
and the version card shows a dash when none answered.
|
||
|
Chris LuandDevin
|
4ed9595a58 |
admin: explain detection cooldowns and skipped scans (#12185)
* plugin: describe detection cooldowns and report skipped scans * admin: expose cooldown eligibility and preserve skipped detection outcomes * admin: show cooldown eligibility alongside maintenance lock waits * plugin: retain cooldown activity for older admin servers * admin: keep manual detection skips out of scheduler run state * plugin: refresh descriptors on reconnect without downgrading saved schemas * plugin: allow valid schema refreshes to repair corrupt descriptors * admin: store eligible_after activity detail as RFC3339 * worker: report admin_script interval skips as skipped detection * worker: report admin_script interval skips after the last completed job * admin: show the selected job's cooldown while scheduling is disabled * admin: report an unsigned interval skip instead of a completed scan * worker: populate new DetectionComplete fields in Rust senders Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
e7c1713457 |
admin: show maintenance lock blockers in scheduler UI (#12184)
* admin: expose maintenance lock waiters and current owner * admin: display maintenance lock waits across scheduler pages * admin: preserve the active lock message while acquisitions wait * admin: synchronize lock fairness assertions with waiter registration * admin: display lock wait start times alongside elapsed duration |
||
|
Nguyễn Lộc Phúcandchrislusf
|
b62bbb99fc |
fix(iceberg): serve a registered table's metadata on load and commit (#12159)
* iceberg: serve a registered table's metadata on load and commit RegisterTable saved only the metadata location, and loads and commits never read the file there. They served a placeholder with no schema, no snapshots and a new random table UUID on every load, so a registered table read as empty, an INSERT failed against the empty schema, and every other commit failed its assert-table-uuid requirement. Metadata uploaded through the S3 gateway, as a migrated table's is, is stored in chunks that registration did not read, so it registered an empty table. Read the file following its chunks, reject one that is not Iceberg metadata with a 400, and keep its content and table UUID in the record the way CreateTable does. Records registered before this change hold only the location. Loads and commits now read the file there, and the first commit stores it in the record under the file's table UUID. * iceberg: transactions read a location-only registration's metadata file * iceberg: cover transactions on a location-only registration * iceberg: refresh the default name mapping on transaction commits * iceberg: cover name-mapping refresh in transactions * iceberg: roll a transaction back to a registration's metadata file A transaction that failed after flipping a table registered before the record kept its metadata could not undo the flip. The rollback restored the record's prior metadata, which for such a table is only a location, and UpdateTable keeps the full metadata a flip stored when the restore sends none. Loads went on serving the change the client was told had failed, and a client retrying the transaction could apply it twice. Restore the metadata loads served from the file instead, under the file's table UUID. * iceberg: register oversized metadata by its location Registration sends the metadata file's content to the S3 Tables manager, which refuses a request over its 10MB body limit, so a table whose metadata is larger could no longer be registered at all. Register such a file by its location and table UUID alone; loads read it from that location, as they do for a record that never kept its metadata. Export the limit as s3tables.MaxRequestBodySize so the catalog can check against it. A commit to such a table still sends the whole metadata to UpdateTable and stays over the limit. * iceberg: commit oversized metadata by its location * iceberg: let maintenance follow a location-only table record * admin: read a table's metadata file when the record keeps none * iceberg: bound the metadata file a location-only record reads * iceberg: bound only chunked metadata reads in the worker Inline content is what the catalog itself wrote and the REST load path accepts it at any size, so rejecting it in loadMetadata stranded tables over 64 MiB from every maintenance scan. Cap only the chunked read, where the recorded location can name any object in the bucket. --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
68565fa883 |
admin: bump a-h/templ to v0.3.1100 to match @latest codegen (#12100)
CI installs the templ CLI with `@latest`, which now generates code calling templruntime.NewAttributeExpression — a symbol added after the v0.3.1070 runtime pinned in go.mod, breaking `make install` builds on clean runners. Bump the runtime to v0.3.1100 and regenerate. |
||
|
Chris LuandDevin
|
6f8f8e7079 |
admin: install the go.mod-pinned templ, not @latest (#12090)
* admin: install the go.mod-pinned templ, not @latest install-templ fetched templ@latest, which now emits templruntime.NewAttributeExpression - a symbol absent from the runtime pinned in go.mod - so every fresh 'make' regenerated uncompilable _templ.go files. Install the go.mod version and reinstall when the binary on PATH reports a different one. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * admin: invoke templ from Go's install dir, not PATH A templ binary earlier on PATH than GOBIN still shadowed the pinned compiler after install-templ ran, so generate kept emitting code for the wrong runtime. Resolve GOBIN (or GOPATH/bin) and call the pinned binary by absolute path for install checks, generate, watch, and fmt. * admin: regenerate templ output with the pinned v0.3.1070 The checked-in _templ.go files were stamped v0.3.1020, older than the go.mod pin. Regenerate so the committed code matches the compiler the runtime links against. * admin: resolve templ under the first GOPATH entry go install drops binaries in the first GOPATH element's bin, so a multi-element GOPATH left TEMPL pointing at a path that does not exist. Split on the OS list separator so Windows GOPATHs resolve too. * admin: install pinned templ before fmt and pattern-rule generates fmt and a direct %_templ.go rebuild invoked TEMPL without ensuring the pinned binary exists; an order-only prerequisite keeps the check cheap without forcing regeneration. * admin: keep spaced GOPATH/templ paths intact in the Makefile --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
fefac028ec |
feat: send EC shards in parallel per destination disk (#12050)
* feat: send EC shards in parallel per destination disk One stream per destination disk. max_shard_transfers unset means no global cap, 0 sends shards sequentially, and N is a global cap that still keeps one stream per disk. The value is persisted through TaskPolicy. The maintenance form is a checkbox plus Max transfers because unset and 0 differ. The key is intentionally omitted from GetConfigSpec. * ec shard distribution: treat negative max_shard_transfers as sequential The config layer (normalizedMaxShardTransfers) clamps negative values to 0, which selects sequential distribution. shardTransferCap diverged: a negative cap that reached ShardDistributionOptions directly would have selected uncapped parallel mode, the most aggressive setting. Align the options layer with the same normalization so invalid input degrades to the conservative mode. * ec: serialize shard transfers to a server when a target leaves disk choice to the receiver disk_id 0 is auto-placement on the receiving volume server, so an auto-placed stream can land on a disk another queued stream is pinned to. Grouping by (destNode, disk_id) alone let both run at once; collapse all of that server's transfers under the destNode key instead. * worker: clamp max_shard_transfers to the int32 wire range ReadIntConfig returned the 0 fallback for values above MaxInt32, turning a large positive cap into sequential mode. Read int64 and clamp into range instead. * worker: drop unused shard-transfer form helper The maintenance form renders and collects max_shard_transfers in plugin.templ JavaScript; the Go copy had no callers. * worker: bound max_shard_transfers conversion under an in-range guard --------- Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
39bdbc18db |
admin: keep read-only sessions out of the filer /etc tree (#12057)
* admin: keep read-only sessions out of the filer /etc tree The file browser read routes are open to any authenticated session, so a read-only login could download or view arbitrary filer paths. The filer stores the credential and cluster configuration under /etc (identity JSON with plaintext S3 secret keys, SSE-S3 keys, remote storage config), which let a view-only session recover live secrets that the user API redacts. Gate the read paths on the session role: read-only sessions get 403 for /api/files/download, /view, /properties, /metadata on /etc paths, the browser page redirects away, and recursive metadata exports skip the /etc subtree for read-only sessions. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * admin: only reject real .. path components in file browser paths path.Clean already resolves interior parent references, so any '..' surviving in the cleaned path can only be a leading escape from a relative input or part of a valid name. Matching on the substring rejected legitimate entries like /buckets/reports/report..csv. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * admin: cover the read-only /etc export exclusion with a test Pull the excluded-prefix selection into exportExcludedPrefixes so the role-dependent /etc skip is exercised directly — the cleanReadPath tests alone would still pass if the exclusion were dropped. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
18f9d9a0b6 |
feat: Free Volume Deadband (M) for EC shard placement (#12029)
* feat: Free Volume Deadband (M) for EC Place nodes and racks |ΔfreeSlots| ≤ M is a tie; remainder is liveShards. M=0 stays strict freeSlots. M is free volume slots (× dataShards). pickNodeInRackEligible and pickRackEligible share the rule. Worker plugin config free_slots_deadband, default 0. Not on the write path. * fix: anchor EC freeSlots deadband on class maxFree Pairwise |Δfree|≤M vs the current winner is not transitive: 120→110→100 at M=10 slots picked 100. After evenness, rank by maxFree, keep |free-maxFree|≤M, then min liveShards. Shared by node and rack. M=0 is still only maxFree then live. * worker: add free_slots_deadband to ErasureCodingTaskConfig * erasure_coding: persist free_slots_deadband through TaskPolicy ApplyMaintenanceConfigFromToml accepted the key but the admin save path round-trips config through ToTaskPolicy, so the value was dropped on reload; serialize it like the other EC settings. * erasure_coding: bound free_slots_deadband at its declared maximum The TOML loader and the plugin worker config only clamped negatives, so a value above the spec's 100000 could reach the int32 TaskPolicy field and wrap; cap at 100000 on both entry paths and declare MaxValue on the plugin field. * erasure_coding: document free_slots_deadband in scaffold admin.toml --------- Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
f2d5c27f65 |
operation: renew the write JWT when a fan-out rollback delete is rejected (#11970)
* operation: renew the cleanup JWT when a fan-out rollback delete is rejected A stalled holder can out-wait the assignment JWT's 10s expiry, so the type=replicate delete that rolls back landed copies comes back 401 and the needle leaks. Retry the delete once with a fresh token minted through the new LookupJwtFunc hook when a holder answers unauthorized. * s3api: wire a cleanup-JWT renewer into chunked uploads Keep the master client on the server and use it to mint fresh volume JWTs for rollback deletes that run past the assignment token's expiry. * admin: wire a cleanup-JWT renewer into file-browser uploads * operation: bound the cleanup-JWT renewal with a per-attempt deadline A rollback delete that hits 401 renewed the token through a background context, so a master outage could park the wait for a connection and the lookup RPC inside the failed upload's cleanup path. Thread a context through LookupJwtFunc and LookupJwt, and bound each renewal with a 10s deadline covering both the master wait and the lookup call. * operation: let every holder's 401 retry with a renewed cleanup JWT A single renewed flag meant only the first rejecting holder refreshed the token; a holder that stalled past the renewed JWT's lifetime left later copies stranded with a 401 they could not resolve. The retry limit is now per holder — each delete is attempted once with the current token and once more after a bounded renewal — so a slow middle holder cannot strand the rest. * operation: log the volume server's error body on a failed cleanup delete Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * operation: count test upload attempts atomically The chunk upload loop fans AssignFunc out to parallel goroutines, so the plain int counter in the partial-results test raced under -race. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
00e64441c6 |
fix: clone protobuf snapshots without copying internal locks (#11968)
* fix: clone protobuf snapshots without copying internal locks * fix: reuse cloned snapshot fields when filtering |
||
|
GAURIandGAURI
|
da0d423bd9 |
Add OLake to README and Admin UI; fix catalog_olake path-style docs (#11858)
* README: mention OLake in the Iceberg lakehouse section * catalog_olake: fix path-style prose; admin UI: list OLake as a REST catalog client * admin UI: restore generated file import block, keep only the string change --------- Co-authored-by: GAURI <174253719+gauriiiiiiiiiiii@users.noreply.github.com> |
||
|
Mohd Quamar TyagiandMohd Quamar Tyagi
|
e20640acb4 |
fix(admin): improve Total Volumes count contrast (#11817)
fix(admin): improve Total Volumes count contrast on dashboard Co-authored-by: Mohd Quamar Tyagi <Tyagiquamar@users.noreply.github.com> |
||
|
Chris Lu
|
8ac5133abe |
admin: default to port 12535, keep 23646 as a best-effort compat listener (#11816)
* admin: default to port 12535, keep 23646 as a best-effort compat listener The default admin HTTP port 23646 derives a worker gRPC port of 33646 (port + GrpcPortOffset), inside the Linux default ephemeral range (32768+). One of our own startup dials can claim 33646 before the admin binds it, killing startup. This is the same failure #10209 fixed for the test port allocator, still reachable via the mini default (#10838) and via standalone `weed admin` defaults. Move the default admin port to 12535 (gRPC 22535) everywhere, keeping the http+10000 convention so worker -admin derivation, the Rust worker and host:http.grpc addressing keep working: - weed admin -port, weed mini -admin.port - weed worker -admin, the plugin worker fallback, Rust worker and weed-lance-worker defaults - help text, README, helm values, docker compose, tests So old workers, bookmarks and scripts keep working, admin also serves the same handler and the worker gRPC server on the previous default (-port.legacy / -admin.port.legacy, default 23646, 0 disables). The legacy gRPC port binds best-effort: it lives in the ephemeral range, so a failed bind only warns instead of reintroducing the startup failure. weed mini reserves it early like the primary port and keeps it out of its own port relocation. grpc.Server tracks every listener handed to Serve, so the compat listener stops with the primary. Also warn when an explicitly configured admin port puts the derived gRPC port at or above the ephemeral floor, the other fix suggested in the issue. Supersedes the pinned-grpc approach in #11814: pinning -admin.port.grpc off the +10000 convention breaks external workers that derive the admin gRPC port from the HTTP address. ClickHouse/ClickHouse#115512 pins -admin.port on their side; this makes that pin unnecessary. Fixes #10838 * admin: let services win the legacy ports, close compat listener on failure Review findings on the compat listeners: - Do not put the legacy ports in mini's reservedPorts: an explicit -s3.port=23646 or -master.port.grpc=33646 must be honored, not rejected or stolen by the early reservation. Mini now checks the finalized port set and skips only the conflicting compat listener, so HTTP and gRPC compat degrade independently and the service keeps its port. - Also skip the legacy HTTP listener when a mini service owns the port: a wildcard-bound service can still leave loopback bindable, which would let the compat listener shadow it for loopback clients. - Close legacyServer's raw listener in the deferred cleanup: Serve may not have registered it yet when startAdminServer returns early. - Helm: add admin.legacyPort (default 23646) and expose the legacy http/grpc ports through the admin Service, container ports and NetworkPolicy so older workers resolved via the Service keep working. * helm: only pass -port.legacy when it differs from the binary default ct install deploys the released image with the new chart, and 4.48 does not know -port.legacy, so the admin pod crash-loops. Omitting the flag at the default value is behavior-identical on new binaries (their default is also 23646) and keeps older releases installable; legacyPort: 0 or a custom port still renders the flag. |
||
|
Chris Lu
|
1155e65fc5 |
admin: regenerate the gzipped static mirror
admin.js changed in
|
||
|
joma1022
|
161310d834 |
admin: fix Create Bucket sending POST /api/s3/buckets twice (#11790)
admin: fix Create Bucket submitting twice admin.js and the Buckets page template both bound a submit listener to #createBucketForm, so one click sent POST /api/s3/buckets twice. Keep only the page handler. |
||
|
|
3c6700c310 |
volume balance: stop planning moves onto disk types a server does not have (#11762)
* fix(topology): capacity queries no longer create empty disk-type entries AvailableSpaceFor and CapacityFor called getOrCreateDisk, so asking a node whether it had room for a disk type it lacks created a zeroed DiskUsageCounts entry for that type. Volume growth walks every data center, rack and data node, and ToDataNodeInfo publishes diskUsages, so every node ended up advertising an empty disk (MaxVolumeCount 0, DiskId 0) of every type ever grown: HDD-only servers showed an "ssd" disk and SSD-only servers an "hdd" disk. Read the counters through a non-creating getDisk; a missing entry answers 0, the same numbers as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * fix(admin): don't let an empty disk-type entry shadow a node's real disk UpdateTopology keys disks by node and disk id. The master published an empty entry (no capacity, nothing on it, DiskId 0) for every disk type a capacity check had asked about, and it collided with the node's real disk 0; map iteration order decided which survived. An SSD-only node intermittently showed an empty HDD disk, and an HDD node lost its disk -- and with it its volumes from the volume index, hence "Could not find volume N on source server" in balance detection. IsShadowedEmptyDiskEntry matches an empty entry whose disk id another DiskInfo of the node, one with capacity or data, already uses. UpdateTopology and CountTopologyResources skip it. A lone empty entry is still an empty disk of unknown capacity and is kept, as GetDisksWithEffectiveCapacity expects. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * fix(balance): don't seed balance candidates from shadowed empty disk entries detectForDiskType registered every node with a DiskInfos entry of the type as a zero-volume candidate. An empty entry the master made up for a type the node lacks made that node the least-loaded destination, and its zero max switched the whole disk type to raw-count balancing. The volume server then refused every move: "no space left hdd" / "no space left ssd". Skip entries IsShadowedEmptyDiskEntry matches when seeding candidates. A server that holds volumes of the type but reports no max still enters through the metrics and still triggers the raw-count fallback. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * fix(admin,balance): ignore made-up disk-type entries at any disk id The made-up entry is published with DiskId 0, but it need not collide with a real disk: when location 0 is unavailable (not reported) the node's usable disk is location 1. IsShadowedEmptyDiskEntry required a collision, so such an entry survived both UpdateTopology -- where the unknown-capacity fallback offers it to planners -- and balance detection, which planned moves onto it again. Its disk id cannot tell it apart: a DiskInfo with no volumes, no EC shards and no per-disk max is published with DiskId 0 whichever disk it stands for. IsPhantomDiskTypeEntry (renamed, since nothing needs to be shadowed now) matches an empty entry on any node that has a disk of another type with capacity or data. A real disk looks the same only while it offers no volume slot, when it can take no volume either. A node whose entries are all empty keeps them as disks of unknown capacity, as before. Reported by review on #11762. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * test(balance): assert where the real disks land in the empty-entry test Check through GetNodeDisks that each server's real disk sits at the subtest's disk id before planning. The id-1 case already landed on physical disk 1 (SplitByPhysicalDisk folds all-zero record ids onto the entry's DiskId), so this only makes the setup explicit; on the first revision of the fix it shows the real disk at 1 beside the made-up entry at 0. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * fix(topology,balance,ec): don't publish or trust usage-only disk-type entries ToDataNodeInfo started from the raw disk-usage counters, so a node that was asked for capacity of a disk type it lacks kept advertising an empty DiskInfo for that type even though no such disk was ever reported. Skip usage entries that have no disk child and hold all-zero counters. The same stale entries still misled planners that read DiskInfos directly: the balance plugin's move precondition treated a phantom entry as proof a target has that disk type, and the EC balancer counted it as a matching disk. Filter them with IsPhantomDiskTypeEntry. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
6da432b8bf |
filer: make a retried ObjectTransaction replay its recorded outcome (#11788)
* filer_pb: add request_id to ObjectTransactionRequest Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: replay recorded outcome for a retried ObjectTransaction A failover replay carries the same request_id; under the same path lock the owner replays the recorded outcome instead of re-applying the mutations over whatever superseded them. Only application-level outcomes are recorded — a lookup error leaves the request free to re-run. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: stamp routed object transactions with a request id One id per logical transaction so failover retries replay the recorded outcome instead of re-applying the mutations. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: scope transaction replay tokens per path and keep them for their TTL The dedup key includes the lock path so a token reused on a different object cannot replay this outcome. Records are never evicted early — an early eviction would let a replay re-apply — so the expiry sweep is amortized and the map is bounded by the insert rate over the TTL. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: bound the transaction replay cache at capacity * filer: do not record indeterminate transaction outcomes A mutation error can be transient or follow a partial apply; recording it replays that failure to a retry for the whole TTL instead of letting the same idempotent mutations converge. Only successful transactions and precondition failures are definitive outcomes worth replaying. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: replay recorded mutation failures instead of re-applying --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
da3b7feeae |
admin: move dashboard progress labels beside the bar (#11781)
* s3: keep suspended-versioning conditional writes off the routed condition A suspended PUT writes the null-version slot at the regular path while the entry the conditional headers evaluate lives under .versions/. The routed WriteCondition checked the request's etag against that empty slot, so a correct If-Match failed with 412. Passing uniqueWritePath sends conditional requests through the lock path, whose precondition resolves the same entry the read path serves; unconditional writes still route. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * admin: keep the volumes progress label readable at low fill The dashboard's Volumes and tier-usage cells rendered their text inside the progress bar, so a low fill left white label text on the light track and the count was unreadable. Render the label beside the bar instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * admin: keep a minimum width on dashboard progress tracks A nonwrapping label beside a flex track could squeeze it to zero in a narrow table, hiding the bar entirely. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Чепурнов СемёнandChepurnov Simon
|
624a13ecc6 |
EC worker: pin jobs to replica-host workers, late-bind local source (#11718)
* feat: prefer same-host worker and local EC replica Admin executor pick ranked idle workers only, so a job whose volume lived on full-03 could still run on full-01 and pull the whole .dat across the LAN. Rank workers whose Hello host matches any replica first (fallback to others if the local worker is full). Detection named Sources[0] / the lowest server id; execute then SyncAndSelectBestReplica and could switch the copy source to another rack. After sync, if a replica on this machine has a complete file count, copy from that URL so affinity and pull stay on one host. * feat: pin scheduler jobs to replica-host workers until load K Reserve prefers the least-loaded replica-host worker and waits while that load is below 5 instead of spilling to a free remote worker. K=0 restores immediate spill. Waterfill uses reservation load so four idle preferred workers each get a job. * feat: park scheduled jobs on per-worker admin queues Assign a detection batch to the least-loaded replica-host worker in one pass, then let ExecutionConcurrency goroutines pop a home-runnable job or steal when that home queue reaches K. Dispatch no longer holds a job in reserve-backoff, so a free worker is not blocked by another home's pin. Tracked jobs stay pending/queued and record the home worker for the admin UI. * fix: rehome queued jobs when the home worker dies A live home still pins below K. A home missing from live executors is stolen immediately onto a free worker so a short orphaned queue does not sit until the dispatch window ends. * feat: wait before stealing a dead home worker queue Orphaned queues are held for 30s so a worker reconnect blink does not dump jobs onto other machines. After the grace, steal without K as before. Zero disables the wait. A home that comes back during grace keeps its queue. * fix: late-bind EC from to worker hostname After replicas are consistent (equal FileCount), rebind encode source to a volume server on the same hostname as worker -address. Any port on that host is local. Diverged FileCount still uses the max replica. * feat: assign homeless jobs to global FIFO Jobs whose replica hostname has no registered worker skip pin and orphan grace. Free workers take own home, then steal at K/orphan, then leftover FIFO. * fix: take homeless FIFO before steal for no-home workers Workers without a replica-host match take leftover first; steal (K/orphan) only when FIFO is empty. Home workers keep own-queue, steal, then leftover. --------- Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com> |
||
|
Chris Lu
|
55ac46f7a9 |
admin: authenticate worker and plugin gRPC streams
Worker registration previously ran unauthenticated on any interface the listener bound. The stream now requires a verified mTLS client certificate or a Bearer token signed by jwt.filer_signing.key, and falls back to local clients only when no key is configured. |
||
|
Chris LuandDevin
|
8f80dac30f |
ec: strict_placement option so encode only runs while guarantees hold (#11656)
* ec: strict_placement option so encode only runs while guarantees hold Shard placement during encode was best-effort (PlaceDurabilityFirst): when the cluster could not satisfy the per-disk caps, anti-affinity, replica-placement or per-rack caps, the constraints were relaxed and the volume was encoded anyway, weaker than configured. A strict_placement option on the erasure coding task switches planning to PlaceStrict so the volume's planning fails instead, and the encode is retried when capacity allows the guarantee. Also documents the resilience rule in ec.encode help: a volume survives losing any nodes or racks holding at most parity-shards shards between them, and how -shardReplicaPlacement's rack and node digits bound that loss. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec: expose strict_placement through the plugin form and persisted task policy The admin UI, the admin.toml maintenance mapping, and the TaskPolicy serialization all dropped the new flag; add the bool field to ErasureCodingTaskConfig, the worker config form, and both conversion directions. * shell: describe shardReplicaPlacement as requested limits, not guarantees ec.encode places shards best-effort, so the configured rack/node caps only bound shard loss when the final placement actually satisfies them. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
7e809c9991 |
admin: stop leaking cancelled maintenance task files in -dataDir/tasks (#11597)
* admin: delete persisted state when scan cancels pending tasks Each detection cycle cancels every pending task of a type before re-detecting it, and the cancel path saved the cancelled task back to disk. Nothing ever removed those files, so -dataDir/tasks gained one orphaned .pb per candidate volume per scan cycle. Cancelled is terminal, so drop the file the same way CompleteTask does for completed/failed tasks. The cancelled entry stays in memory for the UI until the next purge. Refs #11595 * admin: delete persisted state when CancelTask cancels a pending task The manual cancel path only updated memory, leaving the pending .pb on disk where a restart would resurrect the cancelled task as pending and the file would linger until then. Delete it like the scan-cycle cancel path now does. * admin: count cancelled tasks toward task retention cleanup CleanupOldTasks and ConfigPersistence.CleanupCompletedTasks only filtered completed/failed tasks, so cancelled entries were exempt from retention in both memory and on disk. Treat all terminal states alike; nil CompletedAt entries also count and sort last, so they are pruned first. * admin: run task file retention in the periodic cleanup loop cleanupCompletedTasks had no callers, so the on-disk retention bound never ran during uptime. Invoke it from performCleanup alongside the in-memory CleanupOldTasks sweep. * admin: guard task state writes against stale saves and failed deletes saveTaskState runs after mq.mutex is released, so the task may have gone terminal in between; a delayed pending save could then recreate the file a cancel just deleted and resurrect the task on restart. Skip saving non-terminal snapshots once the live task is terminal or gone. If a cancel file removal fails, fall back to writing the cancelled snapshot so the file is terminal rather than pending. deleteTaskState now returns its error, and CancelTask captures task.Status while still holding the queue lock. * admin: serialize task file check+write against cancel deletes The saveTaskState guard still had a check-then-write window: a pending snapshot could pass the terminal check before a cancel deleted the file, then write it back after. A persistMu on the queue now covers the check+save and the cancel paths' delete (with its terminal-state fallback), so the two cannot interleave for the same task. |
||
|
Chris Lu
|
37bf1cd91d |
volume: validate copy/tail source addresses before dialing (#11390)
* pb: stop exiting the process on malformed server addresses ServerToGrpcAddress and GrpcAddressToServerAddress called glog.Fatalf when hostAndPort could not parse the port, which os.Exit(255)ed the whole process. A caller-supplied copy or tail source address reached this path synchronously in the serving goroutine, so one anonymous VolumeCopy with a non-numeric port terminated the volume server. Log the parse error and return the input unchanged instead: the dial or request that consumes the address then fails as an ordinary error. * volume: validate copy and tail source addresses before dialing VolumeCopy, VolumeEcShardsCopy and VolumeTailReceiver dial a caller-supplied source address (SourceDataNode / SourceVolumeServer) with no endpoint validation, so an anonymous caller could aim the volume server at loopback, link-local (cloud metadata) or other unintended destinations and read dial behavior back as a connectivity oracle. Apply the same peer-target deny list FetchAndWriteNeedle uses for replica targets: the source must be a bare host:port whose host is not loopback, link-local or unspecified; cluster peers stay reachable on private networks, and -volume.allowUntrustedRemoteEndpoints opts out. The loopback-using copy tests set the flag to keep exercising the copy path in process. * rust volume: validate copy and tail source addresses before dialing Mirror the Go guard on the Rust volume server: volume_copy, volume_ec_shards_copy and volume_tail_receiver dial a caller-supplied source address, so run it through validate_replica_target first (bare host:port; no loopback, link-local or unspecified hosts; private peers stay allowed). --volume.allowUntrustedRemoteEndpoints opts out; the test fixture and the Rust test-cluster launcher set it so loopback sources in tests keep working. * volume: pin validated copy/tail source addresses at dial time validateReplicaTarget resolves the source hostname once, but the gRPC client resolved it again at connect, leaving a DNS-rebinding window for hostname sources. The copy and tail source dials now run through the same guardedDialerPolicy the remote-storage path uses, so every resolved address is re-checked against the replica deny list (private peers allowed) immediately before the TCP connect. guardedDialerPolicy also moves to util.OutboundDialContext so the guarded path keeps the -ip.bind source binding the default gRPC dialer had. The Rust volume server mirrors this with connect_guarded, a tonic connector that resolves, re-checks each address, and connects to the first passing IP; handlers use it whenever the untrusted-endpoint opt-out is off. A handler-level test now exercises the enabled validation branches for all three source-taking RPCs. * pb: return empty server address for malformed grpc addresses GrpcAddressToServerAddress used to return the unparseable input on a hostAndPort failure, so a malformed raft address (e.g. "host:abc") flowed into admin dashboard master maps unchanged. Return an empty string instead, skip empty conversions at the two raft-cluster merge sites, and drop the now-stale comment about the fatal exit the earlier commit removed. * test: opt erasure-coding loopback clusters out of the remote endpoint guard The erasure-coding suites drive VolumeEcShardsCopy / VolumeCopy between volume servers bound to 127.0.0.1, which the copy/tail source guard now rejects by default. Pass -volume.allowUntrustedRemoteEndpoints to the test volume launches, matching what the volume_server framework harnesses already do. * admin: only claim fallback master leadership on an empty raft response A nonempty RaftListClusterServers response whose entries were all rejected left masterMap empty, so the fallback marked the reachable current master as leader the same way a genuinely empty (non-raft) response does. Track whether the successful response returned zero servers and only promote the fallback master then. |
||
|
Chris Lu
|
994e1f7d64 |
admin: replace Font Awesome with MIT-licensed icons (#11364)
admin: replace Font Awesome with MIT icons |
||
|
Chris Lu
|
cf38c01978 |
admin: bind worker gRPC listener to -ip instead of wildcard (#11300)
* admin: bind worker gRPC listener to -ip instead of wildcard
The worker/plugin gRPC control plane called net.Listen("tcp", ":port")
directly, so it wildcard-bound every interface and ignored the -ip setting.
A cluster bound to loopback still exposed the unauthenticated
WorkerService/PluginControlService streams on 0.0.0.0. Bind through
util.JoinHostPort(bindIp, port) so the listener honors -ip like the
master, filer, and volume gRPC listeners.
* admin: warn when worker gRPC is exposed off loopback without mTLS
The worker gRPC stream has no password auth, so grpc.admin mTLS is the
only effective control once the listener leaves loopback. An operator who
sets -adminPassword and binds -ip=0.0.0.0 authenticates the HTTP API but
still exposes the unauthenticated worker control plane. Log a startup
warning naming the port and the mTLS knobs so the exposure is not silent.
* admin: address review on worker gRPC bind fix
- mini: reserve the admin gRPC port with util.JoinHostPort so an IPv6
bindIp (e.g. ::1) does not form an invalid unbracketed address and
lose the reservation.
- worker gRPC: track whether grpc.admin mTLS credentials actually loaded
rather than only whether they were configured, and gate the
non-loopback exposure warning on that. A cert/key that fails to load
now still warns instead of silently suppressing.
|
||
|
Chris Lu
|
8782749f26 |
admin: reject IAM policy deletion while still attached to a user/group (#11230)
* admin: add IsPolicyAttached helper to detect user/group attachments Introduces AdminServer.IsPolicyAttached, which lists the users and groups that still have a managed policy attached, reusing the existing credential manager ListUsers / ListAttachedUserPolicies / ListGroups / GetGroup methods. This is the building block for rejecting policy deletion while a policy is still referenced, so deleted policy names stop lingering in a user attached policy names list (issue #11225). * admin: reject IAM policy deletion while still attached Guards AdminServer.DeletePolicy with the new IsPolicyAttached check and returns the typed ErrPolicyStillAttached error when the policy is still referenced by a user or group. This matches AWS IAM and the existing IAM API handler behavior, fixing the stale reference where a deleted policy name kept showing up in a user attached policy names list (issue #11225). * admin: return 409 Conflict when deleting an attached IAM policy The admin UI DeletePolicy handler now maps ErrPolicyStillAttached to HTTP 409 Conflict instead of 500, so the dashboard can surface the attachment conflict to the user rather than reporting a generic server error. * admin: skip vanished groups when checking policy attachments IsPolicyAttached now treats a group that disappears between ListGroups and GetGroup (credential.ErrGroupNotFound) as no longer attached instead of failing the whole deletion with HTTP 500, matching the IAM API handler which skips vanished groups. * test: assert policy state after deletion paths Strengthen GetPolicy assertions in the policy deletion tests to check the returned policy is non-nil after a rejected deletion and nil after a successful one, not just that no lookup error occurred (GetPolicy returns nil, nil when a policy is absent). |
||
|
Chris Lu
|
75ec5ec193 |
admin: allow setting volume read-only and read/write modes (#11217)
* admin: support setting volume read-only and read/write modes
* admin: address PR review on volume access-mode persistence
Reject trailing JSON values in the SetVolumeReadOnly handler so
requests like {"read_only":true}{} no longer pass validation, and add
a trailing-value case to the invalid-request test.
Propagate .vif persistence failures through the access-mode chain.
PersistReadOnly now returns the SaveVolumeInfo error and rolls back
the in-memory volumeInfo on failure; Store.MarkVolumeReadonly and
Store.MarkVolumeWritable propagate that error and roll back their
noWrite flags, so the API reports failure instead of success while
restart would revert the mode.
* admin: make .vif persistence atomic and preserve error chain
SaveVolumeInfo now writes to a .vif.tmp file, syncs it, renames it
over the target, and fsyncs the directory. A write/sync/close failure
leaves the existing .vif intact, so the PersistReadOnly in-memory
rollback matches the durable state instead of diverging from a
partially written file that restart would apply.
Switch the error wrappers in PersistReadOnly, MarkVolumeReadonly, and
MarkVolumeWritable from %v to %w so callers can use errors.Is and
errors.As to classify persistence failures.
* admin: treat post-rename dir fsync failure as a warning
After os.Rename commits the new .vif, the on-disk file already holds
the requested mode. A directory fsync failure only risks losing the
rename across a crash; returning an error here would make
PersistReadOnly roll back in-memory state while the durable file keeps
the new mode, splitting the replica. Log the failure as a warning
instead, matching the best-effort nature of FsyncDir (already skipped
on Windows).
* admin: distinguish post-rename durability failures and use unique temp files
SaveVolumeInfo now uses os.CreateTemp for the staging file, preventing
concurrent saves for the same volume from colliding on a shared .tmp
path.
A directory fsync failure after os.Rename returns a
NotCrashDurableError instead of being silently swallowed. The rename
already committed the new metadata to disk, so PersistReadOnly,
MarkVolumeReadonly, and MarkVolumeWritable skip the in-memory rollback
for this error type (keeping state aligned with the durable file) while
still propagating the failure to the API. Pre-commit failures continue
to roll back as before.
* admin: continue post-commit work after NotCrashDurableError
MarkVolumeWritable now clears the EIO quarantine and the gRPC handlers
(makeVolumeReadonly step 3, makeVolumeWritable master notification)
proceed with their post-commit work when SaveVolumeInfo returns a
NotCrashDurableError, instead of aborting and leaving the volume
unavailable or the master unaware of the mode change. The durability
warning is still propagated to the API caller. Pre-commit failures
continue to abort early as before.
* admin: handle NotCrashDurableError in tier and EC callers
VolumeTierMoveDatFromRemote and VolumeEcShardsGenerate now check for
NotCrashDurableError from SaveVolumeInfo. When the rename has already
committed the new .vif, they continue with their post-commit work
(backend switch, remote deletion, keeping generated EC shards) instead
of aborting and leaving the on-disk metadata inconsistent with the
file layout. The durability warning is logged for the operator.
|
||
|
Chris Lu
|
e6f2386a0f |
admin: redact S3 secret keys for read-only sessions (#11189)
The Admin UI documents its read-only account as view-only and blocks its
write requests, but the authenticated read routes returned object-store
users with plaintext access and secret keys. A read-only admin user could
retrieve another user's live S3 credential pair from GET /api/users and
GET /api/users/{username} and use it directly against the S3 endpoint,
converting view-only access into the victim identity's object-store
authority.
Redact the reusable secret_key in GetUsers, GetUserDetails, and the
rendered users page whenever the requesting session has the read-only
role. The public access_key identifier is retained so identities remain
browsable; only the reusable secret is stripped. Admin and no-auth
sessions are unaffected.
|
||
|
Guang Jiong LouandChris Lu
|
93666c90e9 |
filter by volume ids (#10983)
* filter by volume ids * master: carry the volume ids VolumeList asks about in one repeated field One id and a list of them ask the same question, so field 2 holds the list rather than standing beside a second field that supersedes it. Claude-Session: https://claude.ai/code/session_011qAmAdhrYvnzGkw7A9N4mP --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
9e06e1d0f9 |
Report a delete the filer rejected instead of answering success (#11003)
* s3tables: report a delete the filer rejected deleteDirectory discarded DeleteEntryResponse and checked only the transport error, so DeleteTable, DeleteNamespace, DeleteView and DeleteTableBucket answered 200 for a delete the filer refused. Call filer_pb.DoRemove, which reads resp.Error and still treats a missing entry as success. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * admin: report a delete the filer rejected The bucket delete, the file browser handlers and the topic retention purger all discarded DeleteEntryResponse, so a delete the filer refused came back as success. Call filer_pb.DoRemove, which reads resp.Error. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * credential: report a delete the filer rejected DeleteUser, DeletePolicy and the full-sync cleanup loops discarded DeleteEntryResponse, so a rejected delete answered success and left the credential file in place. The service account path in the same store already read resp.Error; the rest now do too, via filer_pb.DoRemove where not-found is already tolerated. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * shell: report a delete the filer rejected remote.configure -delete, remote.cache and the remote metadata sync discarded DeleteEntryResponse, so a rejected delete printed as removed. Call filer_pb.DoRemove, which reads resp.Error. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * mq: report a delete the filer rejected The consumer offset group purge and the coordinator assignment delete discarded DeleteEntryResponse. Call filer_pb.DoRemove, which reads resp.Error. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * iam: count only the revocation entries the filer actually deleted The expiry sweep discarded DeleteEntryResponse, so a rejected delete was counted as purged and the entry stayed. Call filer_pb.DoRemove, which reads resp.Error, matching the role and provider stores beside it. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * mount: fail rmdir when the unary fallback delete was rejected The streaming branch turns DeleteEntryResponse.Error into an error, the unary fallback dropped it, so rmdir of a non-empty directory answered OK off the stream and ENOTEMPTY on it. Surface it in both. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * s3tables: fail DeleteTableBucket when the directory delete is refused The handler only failed when both the leaf entry and the directory delete failed, so a refused bucket directory delete still answered 200 with the bucket in place. The directory is the bucket, so it decides; the leaf entry stays best-effort. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU |
||
|
Chris Lu
|
902a12fd6f |
wdclient: bound the wait for a master leader by the caller's context (#11002)
* wdclient: bound the wait for a master leader by the caller's context WithClient waited on GetMaster with context.Background(), so a caller that arrived while no master leader was known parked in a 200ms poll loop until one appeared, whatever deadline it had already set on the RPC. Each retry above it then left another goroutine in the same wait. Take the context in WithClient and WithClientCustomGetMaster and hand it to GetMaster, and stop the retry loop once it is done. The dial keeps context.Background(): fn brings its own RPC context, so a cancellation seen here cannot be attributed to the shared connection. Call sites pass whatever they hold: the request context in the filer's CollectionList, DeleteCollection and Statistics handlers and in the credential store's propagation, the operation context in the shell's s3.bucket.delete and the kafka gateway's broker and filer discovery, and context.Background() where there is none - the shell commands, the admin dashboard wrapper, and the exclusive locker's initial lease. The locker's release keeps its own uncancelled context so a slow unlock cannot turn into a ghost lock. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * wdclient: test that WithClient gives up with the caller's context Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * wdclient: cut the master retry backoff short when the caller gives up util.Retry sleeps unconditionally between attempts, so a transient error arriving just before the caller's deadline still cost it a full backoff step. Use the context-aware util.RetryWithBackoff, the same helper the volume lookup in this file already uses. Two call sites went with it: the shell's lock-holder lookup builds its three second bound before WithClient so it also covers finding the leader, as its comment already promised, and the filer's post-delete collection cleanup goes back to an uncancelled context - the entry is already gone, so a caller that hung up must not leave the collection behind. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * wdclient: test that a cancel during backoff ends the retry Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU |
||
|
Chris Lu
|
4a2879abad |
admin: show a copyable S3 object URL in the bucket file browser (#10933)
* admin: offer copyable S3 object URLs in the bucket file browser * admin: hide object urls when the bucket type lookup fails * admin: ignore an s3.public_endpoint that is not an absolute http url * mini: build the seeded s3 endpoint with JoinHostPort for ipv6 * admin: reject a query or fragment in s3.public_endpoint * mini: drop the seeded s3 endpoint when a later run disables s3 * admin: reject userinfo and bare delimiters in s3.public_endpoint, redact the warning * mini: pass its s3 endpoint as an admin option instead of mutating viper * admin: keep the rejected s3.public_endpoint value out of the log |
||
|
Chris Lu
|
46ce2c45a2 |
mini: reserve the admin gRPC port instead of binding it late (#10928)
* mini: reserve the admin gRPC port instead of binding it late Port selection probes every port with a throwaway listener and closes it. Master, filer, volume and S3 bind a moment later, but the admin waits for all of them first and only then binds its worker gRPC port, roughly two seconds in. That port defaults to the admin http port + 10000, which lands inside the Linux ephemeral range, so one of the cluster's own outgoing gRPC dials can take it during the gap and the admin dies on bind, taking the worker with it. Keep the listener from the availability check and hand it to the admin. * mini: clear the admin gRPC reservation before retaking it A rerun inside one process would otherwise inherit the closed listener of the previous run whenever the reservation fails, and the admin would accept it and only find out inside Serve. * mini: snapshot the admin options for the startup goroutine The cleanup path read the package-level options long after the goroutine started, so a later in-process run could have its reserved listener closed by the previous run. |
||
|
Chris Lu
|
69cc2869ad |
Fixes from the review of the admin bucket policy UI (#10907)
* admin: treat a missing S3 Tables policy as an empty load, not an error
The bucket/table policy GET relayed the backend's 404 NoSuchPolicy to the
dialog, whose loader treats any non-OK response as a load failure and
keeps Save and Delete blocked. A bucket or table without a policy could
never be given one. Return policy null instead, the same contract
ShowBucketPolicy uses for classic buckets.
* admin: reject policy documents the structured editor would misread
A top-level JSON array passed the object guard (typeof [] is 'object')
and loaded as a zero-statement policy, which the next commit would
rewrite to an empty document. Object elements in Action/Resource were
coerced to '[object Object]' and saved that way on the s3tables surface,
which stores policies verbatim. Both now throw, which routes the
document to the JSON tab like other unrepresentable shapes.
* admin: let the JSON tab save documents the structured editor can't model
Save with the JSON tab active required a round-trip through
policyDocToEditorState, so exactly the documents the dialogs shunt to
'JSON tab only' mode (unrepresentable Effect, Resource+NotResource, and
the like) could never be saved - Delete was the only mutation left.
Invalid JSON still blocks; an unrepresentable document now saves and the
editor state stays marked unparsed.
* admin: pin the policy editor to what each consumer's backend supports
The s3tables evaluator has no NotResource/NotPrincipal fields - it
silently drops them, turning Allow+NotResource into allow-everything and
making Deny+NotPrincipal inert - and it only matches s3tables: actions
against s3tables ARNs, while the editor suggested s3: actions and
arn:aws:s3::: resources. New registerPolicyEditor knobs: allowNegation
hides the Not* modes and routes documents using them to the JSON tab;
resourceSuggestions pins the Resource autocomplete to the open
resource's ARN; the S3 Tables dialogs get an s3tables-only action
datalist. requirePrincipal now also hides NotPrincipal, which
policy_engine.ValidateBucketPolicy always rejects, and the client-side
check requires Principal specifically to match that server rule.
* admin: save S3 Tables policies from a button, not form submission
The multi-input structured editor sits inside a form whose Save button
was type=submit, so Enter in any single-line editor input - accepting an
autocomplete suggestion, say - implicitly submitted whatever half-built
statement the editor held, and the backend stores the document verbatim.
A lone statement with no Principal matches nobody, locking out every
non-owner. Save is now an ordinary button and the form ignores
submission.
* admin: block zero-statement policy saves
Committing the active tab before the emptiness check made 'Policy JSON
is required' dead code: an empty editor serializes to {"Statement":[]},
which the s3tables backend stores verbatim - evaluated default-deny for
every non-owner, while the statement-count column keeps showing 'Not
configured'. All three policy dialogs now refuse a save with no
statements and point at Delete instead. The classic bucket modal only
gained a clearer message; the server already rejected the document.
* admin: guard S3 Tables policy mutations against stale and overlapping requests
The save/delete completions ran against whatever resource the shared
modal happened to show by then: a slow PUT for one bucket would hide the
modal mid-edit of another and misattribute its alerts, a late DELETE
cleared the shared textarea over the newly opened resource with its
loaded flag set, and nothing stopped a double-click from firing two
overlapping mutations. Ported the classic modal's pattern: capture the
target on start, flag the mutation in flight with the buttons disabled,
and only touch the UI when the completion still matches the open
resource. Success now reloads the page, which also keeps the Policy
column's statement count honest.
* admin: confirm before deleting an S3 Tables policy
Delete Policy sat next to Save and fired on a single click; with
default-allow enabled one stray click silently dropped the resource
policy and left the bucket open to every principal. Same confirmation
the classic bucket modal already has.
* admin: let a corrupt stored bucket policy be shown, fixed, and deleted
A stored document the decoder rejects made the policy GET 500, and with
the loaded flag never set the modal blocked both Save and Delete - the
one policy an operator most needs to remove was the one they couldn't,
even though the delete path never reads the document. The GET now
returns the raw bytes alongside a null policy; the dialog hands them to
the JSON tab and unblocks the buttons.
* admin: url-encode the bucket name in the policy API calls
The filer lists any directory under the buckets path, names S3 would
never allow included; one carrying '#' or '%' broke the fetch URL or
addressed a different name than the modal shows.
* admin: drop stale edit-policy responses on the IAM policies page
The same race the bucket and S3 Tables dialogs already guard against:
open one policy's editor while its GET stalls, open another, and the
late response populates the editor under the second policy's name -
Update then saves the first policy's statements over the second.
* admin: warn before a bucket policy save drops unsupported fields
The editor tracks unmodeled top-level keys precisely so
confirmPolicyFieldDiscard can warn before the server's Version+Statement
decode discards them, but only the IAM page called it; the bucket modal
saved a pasted document with e.g. a console-generated Id without a word
while the editor kept displaying the field.
* s3: enforce the bucket policy size cap on both surfaces
The 20KB cap lived only in the admin UI, so a larger policy stored via
the S3 API displayed there but could never be re-saved, desyncing the
two writers the cap comment claimed could not desync. The constant now
lives in policy_engine next to the shared validator and PutBucketPolicy
rejects oversized documents with PolicyTooLarge, matching AWS.
* admin: ship the policy editor's fieldset styles with the editor
The .policy-stmt-* rules that undo Bootstrap's full-width legend reset
stayed behind in policies.templ when the editor markup moved to the
shared script, so the bucket and S3 Tables dialogs rendered Actions/
Resource/Principal as full-width jumbo headings. PolicyDatalists is the
component every consumer already renders once; the styles live there
now.
* s3: mirror bucket policy changes into the IAM store from the metadata subscription
The advanced-IAM path appends the bucket-policy:<bucket> document to
every STS/session evaluation, but only this gateway's own PutBucketPolicy
maintained that mirror - a policy tightened or created through the admin
UI (or another gateway) never reached it, so revoked access stayed live
indefinitely, and the delete side was an unimplemented TODO in any case.
The metadata subscription now diffs the stored policy on every bucket
entry change and updates or removes the mirror, covering all writers and
deletion with one mechanism; IAMManager gains the missing
RemoveBucketPolicy.
* admin: deduplicate the bucket policy write path
Set and Delete carried line-for-line identical filer closures;
bucketPolicyMutation already treats nil as clear-the-key. The shared
helper sits below Set's validation, since ValidatePolicy cannot take the
nil document Delete passes.
* s3: drop ValidateBucketPolicy's re-checks of ValidatePolicy rules
Both callers run ValidatePolicy first, which already enforces the
version and at-least-one-statement rules; the duplicates were dead code
with drifted error text.
* admin: seed a new statement's Resource from the pinned suggestions
A fresh statement on the S3 Tables dialogs started with no resource row
at all; seed it with the broadest pinned ARN the same way cfg.bucket
already seeds the classic modal.
* admin: refuse to save Not* fields the backend would silently drop
Hiding the NotResource/NotPrincipal modes was not enough where negation
is disallowed: the JSON tab accepts any valid document (that is its
job), and a statement's Advanced-fields box can reintroduce the keys, so
an s3tables save could still store fields the evaluator drops - turning
Allow+NotResource into allow-everything. commitPolicyActiveTab now runs
a final document-level check over what would actually be saved; Delete
stays available for cleanup.
* s3: move the IAM bucket policy mirror on a bucket rename
A same-directory rename delivers one event carrying both entries, and
the byte-equality short-circuit skipped the new name's mirror when the
policy was unchanged - while the replayed delete for the old name
removed its mirror, leaving the renamed bucket unmirrored. The mirror
decision is now a pure function that removes the old name and writes the
new one regardless of byte equality, with the rename cases unit tested.
* s3: backfill the IAM bucket policy mirror on lazy bucket loads
The metadata subscription only mirrors changes, so a policy that
predates the IAM integration never reached the bucket-policy:<bucket>
mirror and its grants did not bind on the IAM path until the policy was
next modified. The gateway is deliberately lazy at startup (nothing
lists all buckets), so the backfill hooks the same place a bucket's
policy first becomes known: the cold bucket-config load. EnsureBucketPolicy
writes only when no mirror is stored, so repeat loads cost one cached
read.
* s3: reconcile the bucket policy backfill against concurrent changes
The backfill's check-then-write could race an event-driven mirror update
or removal and re-store bytes that were already stale, with no later
event to heal it. EnsureBucketPolicy now reports whether it wrote, and a
write is reconciled against a fresh authoritative entry read: a changed
policy is re-mirrored, a removed one is removed. Anything changing after
that read fires its own event, which finds the backfill's write already
present and supersedes it. The backfill also carries the entry's raw
bytes rather than a re-marshaled document, so the reconcile can
byte-compare.
* s3: prime the bucket policy mirror before advanced-IAM authorization
The backfill ran from the lazy bucket-config load, but IAM authorization
evaluates the bucket-policy:<bucket> mirror before any handler runs - a
grant carried only by a not-yet-mirrored policy denied forever, and the
denied request never reached the code that would have loaded the bucket.
authorizeWithIAM now primes the bucket config first (an in-memory cache
hit once warm), and the backfill runs synchronously on the cold load so
the very first authorization already sees the mirror.
|
||
|
Chris Lu
|
68ec8ca655 |
admin: honor a persisted or admin.toml maintenance enabled=false (#10909)
* admin: honor a persisted or admin.toml maintenance enabled=false The startup path discarded an operator's enabled=false twice over: ApplyDefaultsToProtobuf treated the bool zero value as unset and applied the schema default of true, and a force-enable migration block flipped any survivor. With the legacy /maintenance UI routes gone, nothing could write the config either, so the maintenance system ran unconditionally. Keep the persisted enabled flag across schema-default application in LoadMaintenanceConfig, drop the force-enable block, and add a top-level [maintenance] enabled key to admin.toml as the config surface, persisted through SaveMaintenanceConfig like the per-task settings. Absent config still defaults to enabled. * admin: track presence on the maintenance enabled flag A plain proto3 bool cannot distinguish an operator's persisted false from a legacy file that simply omits the field, so honoring false would have silently switched maintenance off for configs written before the toggle could be persisted. Make the field optional: files that predate presence tracking keep the enabled default, while a file that explicitly persists the toggle is honored either way. |
||
|
Mathieu Arnold
|
e931cccc7b |
Manage bucket policies via the admin ui (#10895)
* admin: manage S3 bucket policies from the admin UI
Bucket policies were only manageable through the S3 PutBucketPolicy API;
the admin UI had no equivalent to the quota/owner/lifecycle editors it
already offers. Add GET/PUT/DELETE for a bucket's policy, sharing the
exact validation the S3 gateway uses.
- Extract validateBucketPolicy/validateResourceForBucket out of
s3api_bucket_policy_handlers.go into policy_engine.ValidateBucketPolicy /
ResourceMatchesBucket so both the S3 API and the admin UI enforce
identical rules.
- weed/admin/dash/bucket_policy.go: Get/Set/DeleteBucketPolicy, writing
through ObjectTransaction + PATCH_EXTENDED (the lifecycle pattern) so a
concurrent owner/quota/lifecycle change on the same bucket entry isn't
clobbered. Propagation to every S3 gateway is automatic via the existing
filer metadata log subscription. The S3 gateway's IAM policy mirror is
deliberately not replicated here (its delete path is already an
unimplemented TODO on the S3 side).
- New GET/PUT/DELETE /api/s3/buckets/{bucket}/policy routes, CSRF-guarded
on writes.
- Bucket list and details modal now show a statement-count badge, read
from the entry already fetched (no extra RPC).
- UI: a JSON-textarea policy editor modal, matching the lifecycle modal's
structure.
* admin: reuse the visual policy editor for bucket policies
Extract the structured policy editor (add/remove statement, action/
resource/principal rows with autocomplete, JSON tab kept in sync) out of
policies.templ's inline script into a shared
weed/admin/static/js/policy_editor.js, and wire the bucket policy modal
in s3_buckets.templ up to it instead of a bare JSON textarea.
- registerPolicyEditor(which, config) replaces the hardcoded create/edit
id derivation with a per-instance config (textarea/tab/body ids,
datalist ids, requirePrincipal, bucket). The IAM policies page keeps its
exact pre-extraction ids via two registerPolicyEditor calls, so its
markup is unchanged.
- New policy_datalists.templ exposes the three shared <datalist>s
(actions/resources/principals) as @PolicyDatalists(), now rendered by
both policies.templ and s3_buckets.templ.
- requirePrincipal seeds new bucket-policy statements with Principal: "*"
and adds a client-side check before save (the server, via
policy_engine.ValidateBucketPolicy, remains the actual authority); the
bucket config pins the Resource autocomplete to the open bucket instead
of fetching every bucket in the cluster.
- layout.templ loads policy_editor.js globally, after admin.js/
modal-alerts.js (basePath/escapeHtml/showAlert) which it depends on.
3a (the extraction) is a byte-preserving move verified against the
unchanged policies.templ behavior before layering 3b's parameterization
and the bucket-policy wiring on top.
* admin: migrate S3 Tables bucket/table policy editors to the shared editor
Third consumer of the shared visual policy editor: the S3 Tables bucket
and table policy modals (a bare JSON textarea each) now get the same
structured Editor/JSON tabs as the bucket policy and IAM policy pages,
via registerPolicyEditor('s3tablesBucketPolicy'/'s3tablesTablePolicy',
{ textareaId: ... }). Storage and validation are untouched - S3 Tables
policies still go through their own s3tables.PolicyDocument type and the
s3tables.policy extended attribute, unrelated to policy_engine and
s3-bucket-policy; only the editor UI is shared.
Fix a real bug surfaced by adding this second load path: the bucket
policy modal (and the naive first draft of this s3tables port) called
commitPolicyTextareaToEditor() right after a GET and then force-switched
to the Editor tab. commitPolicyTextareaToEditor() is designed to leave
the current tab in place and the editor state untouched when a document
fails to parse (so an in-progress edit survives a bad tab switch), so
forcing the Editor tab afterwards could show empty/stale editor state
that a careless Save would then serialize over a perfectly valid but
structurally-unusual stored policy. Add
loadPolicyTextareaIntoEditor(which) to policy_editor.js, which has no
"current tab" to defer to and instead falls back to the JSON tab with an
alert on a document the structured editor can't represent - the same
safety editPolicy already had in policies.templ - and use it at all three
"populate the editor right after a GET" call sites (bucket policy,
S3 Tables bucket policy, S3 Tables table policy).
* admin: show policy statement count on the S3 Tables buckets page
Mirrors the "Policy" column already added to the classic S3 buckets
list: a clickable badge with the statement count when the table bucket
has a resource policy, "Not configured" otherwise. S3 Tables policies
are a separate mechanism (s3tables.PolicyDocument under the
s3tables.policy extended attribute) from the S3 bucket policy work
elsewhere in this branch (policy_engine.PolicyDocument /
s3-bucket-policy), so this is a parallel implementation of the same
pattern rather than shared code.
- S3TablesBucketSummary gains PolicyStatementCount, populated in
GetS3TablesBucketsData from entry.Entry.Extended[s3tables.ExtendedKeyPolicy]
via the new extractS3TablesPolicyStatementCountFromEntry - no extra RPC,
the entry is already fetched for ExtendedKeyMetadata.
- The badge reuses the existing .s3tables-bucket-policy-btn class, so it
opens the same policy modal as the row's action button with no JS
changes.
* admin: don't let a failed policy GET open the door to an empty overwrite
loadS3TablesBucketPolicy/loadS3TablesTablePolicy cleared the textarea,
then unconditionally called loadPolicyTextareaIntoEditor() regardless of
whether the GET actually succeeded - including when fetch() rejected or
the response was not ok, silently logged to console only. That leaves
the structured editor holding a legitimate-looking empty policy
({version, statements: []}), with the Editor tab active by default.
If Save is then clicked, commitPolicyActiveTab() serializes that empty
state into the textarea as `{"Version":"2012-10-17","Statement":[]}` -
a non-empty string - before the "Policy JSON is required" guard ever
sees it, so the guard passes and the transient load failure gets
written over whatever policy was actually stored.
Add s3tablesBucketPolicyLoaded/s3tablesTablePolicyLoaded, set true only
once a GET has actually completed (ok, including a genuinely empty
policy) and false on any failure path (fetch rejection or a non-ok
response, which previously fell through silently). Both submit handlers
now check the flag before touching the editor at all, and a failed load
surfaces via alert() instead of only a console.error - the user
previously had no visible indication the load had failed.
Verified with a jsdom simulation driving the real rendered page against
a stubbed fetch: a failed GET followed by Save now sends no PUT at all
(previously it sent Statement: []); a successful GET followed by Save
still PUTs the loaded policy unchanged.
* admin: address code review findings on the policy editor
1. policy_editor.js: policyEditors is only pre-populated for 'create'/
'edit'; every other `which` (bucket, s3tablesBucket, s3tablesTable)
stays undefined until its first successful async load. Nothing in
this file enforces that a page hide its Editor/JSON tabs and
Add-statement button until that load completes - the S3 Tables policy
modals don't - so a click in that window (e.g. Add statement, or
switching to the JSON tab) threw "Cannot read properties of undefined
(reading 'unparsed')". Add policyEditorState(which), which lazily
initializes a default state, and route addPolicyStatement, the
jsonTabBtn 'show.bs.tab' handler, commitPolicyActiveTab, and
renderPolicyEditor through it. Verified with a jsdom simulation
against a never-resolving fetch: the exact click threw on the
pre-fix code and no longer does.
2. s3_buckets.templ: the bucket-policy Save handler checked the
textarea for emptiness before calling commitPolicyActiveTab(), which
is what actually serializes the structured Editor tab's fields into
that textarea. A policy entered entirely through the Editor tab (the
primary path - never touching the JSON tab) left the textarea at
whatever it was at load time, so creating a new policy this way hit
"Enter a policy document" and Save silently did nothing. Move the
commit before the emptiness check, preserving the existing alert and
early-return. Verified with a jsdom simulation: Add-statement then
Save (no tab switch) now PUTs the entered statement; before the fix
the same sequence never reached fetch().
3. s3tables_buckets.templ / s3tables_tables.templ: the policy Editor/
JSON nav-tabs were missing the ARIA roles Bootstrap's own tab pattern
expects (role="tab"/"tabpanel", aria-selected, aria-controls,
aria-labelledby) - screen readers had no way to tell these were tabs
or which pane went with which button. Added the standard Bootstrap 5
tab markup to both.
* admin: guard policy load/save flows against overlapping requests
1. s3tables.js: loadS3TablesBucketPolicy/loadS3TablesTablePolicy had no
protection against overlapping loads. Opening one bucket's (or
table's) policy dialog and then another's before the first GET
resolved let the late response write its document into the shared
textarea and mark the dialog "loaded" while it was now targeting the
second resource - a subsequent Save would then push the first
resource's policy onto the second. Add a per-load monotonic sequence
number (s3tablesBucketPolicyRequestSeq / s3tablesTablePolicyRequestSeq,
the same pattern already used for the classic bucket-policy load in
s3_buckets.templ); a response is only applied - textarea, loaded flag,
editor state - if its captured sequence still matches the latest one
issued.
Verified with a jsdom simulation: bucket A's policy load (artificially
slow) followed immediately by bucket B's (fast) previously left A's
policy in the textarea once A's late response landed; it now correctly
keeps B's.
2. s3_buckets.templ: the bucket-policy Save button lives outside the
(initially hidden) editor wrapper, so it stays clickable while a load
is still in flight - the existing policyRequestSeq guard only protects
the *load* from a stale response, not Save from firing before any
load for the current bucket has completed. Add bucketPolicyLoaded,
reset before each GET and set only once the matching response lands,
and check it at the top of the Save handler.
Verified with a jsdom simulation: clicking Save immediately after
opening the dialog, before a (deliberately never-resolving) GET
settles, now sends no PUT; a normal load-then-save sequence still
PUTs the loaded policy unchanged.
* admin: address further code review findings on the policy editor
1. s3tables.js: loadS3TablesBucketPolicy/loadS3TablesTablePolicy only
reset the JSON textarea when a new load starts; the structured editor
kept showing the previously loaded resource's statements (Editor tab
is the default active one) until the new fetch resolved. Call
loadPolicyTextareaIntoEditor() against the now-cleared textarea
immediately, so switching resources visibly resets the editor right
away instead of only once its own load completes. Verified with jsdom:
opening bucket A (loads fully) then bucket B (GET never resolves) no
longer leaves A's statements visible in B's editor.
2. s3tables.js: deleteS3TablesBucketPolicy/deleteS3TablesTablePolicy had
no loaded-state check, so a failed GET (which already blocks Save)
left Delete fully able to remove the resource's stored policy sight
unseen. Add the same s3tablesBucketPolicyLoaded/s3tablesTablePolicyLoaded
guard Save already uses. Verified with jsdom: delete after a failed
load now sends no DELETE; delete after a successful load is unaffected.
3. s3_buckets.templ: the bucket-policy Editor/JSON nav-tabs were missing
the same ARIA roles already added to the S3 Tables policy tabs in an
earlier round (role="tab"/"tabpanel", aria-selected, aria-controls,
aria-labelledby) - this instance was out of scope for that review
comment but is the same gap. Bootstrap's own tab.js already manages
aria-selected on tab switch once the attribute exists, so no extra JS
was needed.
4. s3_buckets.templ: neither the bucket-policy Save nor Delete handler
guarded against a double-click, or against firing while the other was
still in flight - two overlapping PUT/DELETE requests for the same
bucket could land in either order. Add a shared
bucketPolicyMutationInFlight flag: set (and both buttons disabled)
before each fetch, cleared (and buttons re-enabled) on failure so the
user can retry, left set through the existing success hide-and-reload
path, and also reset when a new bucket's dialog opens so an abandoned
in-flight request from a closed dialog can't leave the buttons stuck
disabled. Verified with jsdom: double-clicking Save now sends exactly
one PUT, and a Delete click while that PUT is still pending sends no
DELETE.
* admin: scope bucket-policy mutation completions to the bucket that started them
1. The previous round's fix reset bucketPolicyMutationInFlight whenever a
new bucket's policy dialog opened, to avoid leaving Save/Delete stuck
disabled if the modal was closed mid-request. That traded one bug for
a worse one: if bucket A's PUT/DELETE was still in flight when the
user opened bucket B's dialog, the reset let B's Save/Delete fire
immediately, and A's completion handler - unaware anything had
changed - would still hide the (now B's) modal and reload the page
out from under whatever the user was doing with B, on success, or
alert a message with no bucket context, on failure.
Stop resetting on reopen, so a pending mutation for a previous bucket
keeps this bucket's Save/Delete blocked until it settles (matches the
"preventing overlapping mutations" the review comment describes).
Instead, capture policyEditorBucket as targetBucket right before each
fetch and compare it against policyEditorBucket again in the
completion handler: the in-flight flag is always released so the
buttons never get stuck, but the modal-hide/reload/alert only fire if
this bucket is still the one showing; a stale completion for an
abandoned bucket just logs to the console instead.
Verified with a jsdom simulation: opening bucket B while bucket A's
Save is still pending leaves B's Save button disabled and a click on
it a no-op; once A's PUT resolves, B's button re-enables but no
modal.hide()/reload() fires (previously both fired unconditionally).
2. bucketPolicyDeleteBtn had no bucketPolicyLoaded check, unlike Save -
a failed GET blocked Save but left Delete free to remove a policy the
client never actually saw (the same gap already fixed for the S3
Tables policy modals in an earlier round). Added the same guard,
ahead of the confirm() dialog. Verified with jsdom: Delete after a
failed load now sends no DELETE request.
* admin: fix spelling mistake
|
||
|
孙超
|
c80664ec21 |
s3: propagate storage rule fsync to volume server uploads (#10906)
The storage rule's fsync decision was computed by the filer (detectStorageOption -> rule.Fsync) and applied on the filer's own HTTP write path, but was never carried onto the chunk uploads S3 issues: the AssignVolumeResponse had no fsync field, so the s3api client could not learn the decision, and the chunked upload URL was hardcoded without it. Every S3 write to a path with fsync configured went to the volume server as a non-fsync write. Carry the decision through the assign response: - filer.proto: AssignVolumeResponse gains bool fsync, filled from the storage option the assign resolved. - operation.AssignResult gains Fsync, so uploadChunk can append ?fsync=true to the volume server upload URL (single and replica fan-out paths). - The S3 PUT/UploadPart assignFunc, the S3 copy path, the admin file browser upload, and the Iceberg worker assign functions all forward the response field. Adds TestUploadReaderInChunksAppendsFsyncWhenAssigned. |
||
|
Chris Lu
|
3b8931c2f6 | admin: address review feedback on the maintenance scanner fix | ||
|
Junker der Provinz
|
8d2c0273bd |
admin: stop the maintenance scanner pinning itself to one scan per second after a transient failure (#10887)
* admin: honour persisted task configs when building the maintenance policy
buildPolicyFromTaskConfigs passed a literal nil to vacuum, erasure_coding
and balance LoadConfigFromPersistence. Those functions look for their
LoadXTaskPolicy() accessor via a type assertion, which a nil interface can
never satisfy, so every call fell through to NewDefaultConfig() and the
policy came back with the compiled-in defaults - Enabled: true among them.
A task disabled on disk was therefore still scheduled, and the only trace
was a glog.V(1) "Using default ... configuration" line.
Thread the real ConfigPersistence through instead. There are two copies of
this function: the one in weed/admin/dash builds config.Policy on the
normal admin startup path and can simply take cp as its receiver, and the
one in weed/admin/maintenance is the fallback used when the config carries
no policy yet, which now receives the store from NewMaintenanceManager.
weed/admin/dash already imports weed/admin/maintenance, so the maintenance
side has to keep the duck-typed interface{} parameter that the task
loaders already use rather than importing the concrete type back.
The store is only handed over when a data directory is configured: an
unconfigured one has nothing to read, and a typed nil pointer would pass
the loaders' type assertion and then panic on first use.
Fixes #10874
* admin: restore the maintenance scan cadence after an error backoff
scanLoop shortens its ticker to the error backoff delay after a failed
scan, but it decided whether to replace the ticker by comparing the
target interval against the configured scan interval instead of against
the interval the ticker was actually running at. Once the errors stopped,
getScanInterval returned the configured interval again, the comparison
came out false, and the ticker was left at the backoff delay - so a
single transient scan failure pinned the scanner to one scan per second
for the rest of the process lifetime. That is the ~1/second cadence in
issue #10874: 658 KB/s of "Cancelled N stale pending balance tasks
before re-detection" and 193k orphaned task files over two days.
Track the interval the ticker is running at and compare against that, so
both entering the backoff and returning to the normal cadence replace the
ticker.
While in here:
- defer ticker.Stop() bound the ticker that was current when the defer
was registered, so every replacement ticker leaked on return. Wrap it
in a closure.
- running was written by Start/Stop and read by all three background
loops without synchronisation. Guard it with the existing mutex, fold
the running check in triggerScanInternal into the lock it already
takes, and make Stop a no-op when not running so a second call cannot
close the stop channel twice.
Refs #10874
* admin: make the maintenance policy actually reach the task detectors
Loading the persisted task configs into the maintenance policy only
matters if something reads that policy, and nothing did.
MaintenanceIntegration pushes the policy into every registered detector
and scheduler through interface{ SetEnabled(bool) } and
interface{ SetMaxConcurrent(int) } type assertions. Every task registered
through base.RegisterTask is backed by base.GenericDetector and
base.GenericScheduler, and neither implemented either method, so all four
assertions failed silently for every task on every startup. The policy's
enabled flag reached nothing: ScanWithTaskDetectors gates on
detector.IsEnabled(), and the queue's policy lookups for max concurrent
and repeat interval are fallbacks that only fire when the scheduler
reports zero, which the generic scheduler never does.
Add the setters, delegating to the TaskConfig.SetEnabled the interface
already declares and to TaskDefinition.MaxConcurrent, which is what
GetMaxConcurrent returns.
Applying the policy required three more fixes, because with the
assertions working the policy could now do damage as well as good:
- IsTaskEnabled reports false for a task type the policy has no entry
for, so applying it unconditionally would have disabled every task the
policy does not list. Skip task types with no policy entry: no entry
means no opinion, not disabled.
- ec_balance was exactly such a task. It is registered like the other
three but had no entry in the policy builder and no accessor on
ConfigPersistence at all, so its configuration could never be
persisted. Add SaveEcBalanceTaskPolicy/LoadEcBalanceTaskPolicy, the
task_ec_balance.pb file, the SaveTaskPolicy dispatcher case, and the
policy entry.
- InitMaintenanceManager ran before loadTaskConfigurationsFromPersistence,
which replaces each task's whole config object, so the policy was
applied and then immediately thrown away. Swap the order. Both read the
same files, so the policy is now the last writer and stays
authoritative.
MaintenanceManager.UpdateConfig also updated the queue's and the
scanner's policy but not the integration's, so a policy changed at
runtime never reached the detectors. Add MaintenanceIntegration.SetPolicy
and call it.
While building the policy, stop hand-copying each task's fields and use
the task's own ToTaskPolicy(). The hand-written version was a second
definition of every task's policy and had already lost the erasure coding
preferred tags and replica placement and the balance IO rate limit. For
the same reason, the "nothing persisted yet" branches of
LoadVacuumTaskPolicy, LoadErasureCodingTaskPolicy and
LoadBalanceTaskPolicy now derive from each task's NewDefaultConfig()
instead of a third hand-written copy. Those copies had drifted, so with a
data directory but no config file on disk the effective defaults differed
from what the task and the admin UI schema both advertise:
vacuum scan interval 24h -> 2h
balance scan interval 6h -> 30m
balance imbalance 0.1 -> 0.2
erasure coding scan interval 168h -> 1h
erasure coding fullness 0.90 -> 0.95
erasure coding min volume 1024MB -> 30MB
Finally, weed/admin/dash and weed/admin/maintenance each carried a copy
of the policy builder and they had already diverged. Export the
maintenance one as BuildPolicyFromTaskConfigs and have dash call it.
Refs #10874
* worker: warn when a config store cannot supply a task's persisted config
LoadConfigFromPersistence logged a single glog.V(1) "Using default X
configuration" for every way of not loading anything, so the bug in
issue #10874 - a store handed in that the type assertion rejects, leaving
a task running on compiled-in defaults - looked exactly like the normal
"no data directory configured" case. The reporter had to read the source
to work out why their disabled task kept running, and asked for this
specifically.
Separate the cases. A non-nil store that does not provide the accessor is
always a wiring bug and is now logged at warning level, naming the type
and the missing method. A read error or a policy that will not apply is
also a warning. No persistence configured, and a store with nothing saved
yet, stay at V(1): those are normal.
Refs #10874
* admin: stop GetTaskPolicy panicking on a maintenance policy that is nil
GetTaskPolicy dereferenced its MaintenancePolicy argument to look at
TaskPolicies, so IsTaskEnabled, GetMaxConcurrent and GetRepeatInterval
all took the admin process down when handed a nil policy. A nil policy is
not a programming error here: MaintenanceConfig.Policy is unset until
something builds one, DefaultMaintenanceConfig returns a config with no
policy at all, and UpdateConfig installs whatever config it is given.
Found by calling IsTaskEnabled with the policy from a freshly defaulted
MaintenanceConfig.
Treat a nil policy as "no entry": no task enabled, the safe concurrency
default of 1, and a repeat interval of 0 so callers fall back to their
own default instead of reading DefaultRepeatIntervalSeconds off nil.
Also add the startup test this was found with. It walks the admin
server's startup sequence over a data directory that has balance saved as
disabled and checks the state that decides whether issue #10874 happens:
the balance detector reports disabled, vacuum stays enabled, and tasks
whose config was never saved keep their compiled-in default.
Refs #10874
* admin: document the synchronisation SetPolicy would need beyond startup
ConfigureTasksFromPolicy now really writes TaskDefinition.Config and
TaskDefinition.MaxConcurrent, which the scan loop reads through
detector.IsEnabled() with nothing synchronising the two. Every caller
runs during admin server startup today, before the scan loop exists, so
there is no live race - but the next caller has to add the locking, and
the same already applies to UpdateAllConfigs replacing the whole config
object. Write it down at the seam instead of leaving it to be
rediscovered.
Refs #10874
|
||
|
Junker der Provinz
|
f3caf6e7da |
admin: count plugin-runtime workers in worker metrics (#10884)
* admin: count plugin-runtime workers in worker metrics The admin server keeps two worker registries: the legacy maintenance-worker map, filled by workers registering over the worker gRPC stream, and the plugin worker registry, filled by workers started as `weed worker`. Both the SeaweedFS_admin_workers_connected / SeaweedFS_admin_worker_slots gauges and the dashboard's Workers card read only the legacy map, so a cluster that runs the admin and its workers as separate components reported 0 workers even while its workers showed up on the plugin pages and ran scheduled jobs. Aggregate both registries instead. The two are merged by worker ID: `weed mini` starts both runtimes out of one working directory, so they share the persisted worker ID and must not be counted twice. For such a worker the slot numbers still come from the legacy registry, which keeps mini's existing readings. Plugin workers report their slots in the heartbeat, so detection and execution slots are summed from there; a worker that has connected but not yet sent a heartbeat counts as connected with zero slots. Fixes #10525 * admin: clamp negative worker-reported slot values in metrics merge A plugin worker's self-reported heartbeat slot counts are untrusted input; clamp them to 0 before summing so a stale or misbehaving worker can't drive the aggregate gauge negative, matching the same defensiveness already used in registry.go's own slot arithmetic. |