16013 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Chris Lu	30f0364256	admin: fail an in-flight detection when its worker disconnects (#12376) ... * admin: keep a reconnected plugin worker registered when its old stream unwinds cleanupSession removed whatever session was registered under the worker ID, so a worker that reconnected while its previous stream was still unwinding lost the new session and its registry entry to the old stream's deferred cleanup. Tie cleanup to the stream that is ending and publish the new stream before the registry upsert so a lingering predecessor sees itself superseded. * admin: fail an in-flight detection when its worker disconnects RunDetectionWithReport waited only on the caller's context and the DetectionComplete channel, so a detection whose worker stream ended mid-request could only finish by timeout. On a customer cluster a planned restart of ec-vacuum-1 during an ec_vacuum detection left the default lane in 'detecting' with every execution slot idle until the 3600s detection timeout, blocking every job type in that lane. Bind the request to the stream it was sent on and fail the detection as soon as that stream closes, whether by disconnect or by a reconnect that replaced the session. A liveness tick also fails it once the registry no longer considers the worker available (heartbeat stale), sending a cancel in that case because the stream may still be open. The detector lease is dropped so the next pick starts fresh, and a 'disconnected' detector activity is recorded. WorkerDisconnectForTest now goes through the real session cleanup. * admin: retry a scheduled detection once when its detector drops When a detection fails because its worker disconnected, pick a detector again and retry once within the same iteration while the job type's runtime budget allows, instead of giving the whole cycle up. Any other failure, or a failed retry, ends the run as before and the lane moves on; the next cycle re-detects anyway. * admin: register a plugin worker's hello and stream as one step WorkerStream published the stream under sessionsMu but ran the registry upsert after releasing it, so two concurrent streams for one worker ID could publish their hellos in the opposite order from their sessions: the registry then described the older hello's capabilities while only the newer stream was sendable, and detector picks routed on them. registerSession upserts the hello and installs the stream under one sessionsMu critical section, and a hello or heartbeat arriving on a stream that is no longer the registered one is dropped instead of rewriting the live stream's entry. * admin: steer a detection retry off the stream that dropped The retry picked while the lost worker could still be in the registry: cleanupSession closed done before taking the registry entry down, and a stream closed by a reconnect keeps its entry on purpose. The lost worker ranks first on free slots, so the single retry went back to it and failed on send while a healthy worker sat idle. The disconnect error now carries the stream it came from and the retry picks around it; pickDetector only returns workers whose current stream is open, excluding registry entries whose stream is closed; and cleanupSession removes the registry entry before closing done, so a waiter woken by the closure never sees the dead worker as a candidate.	2026-11-03 12:28:39 -08:00
Chris LuandClaude Opus 5.5	eeb8aefe70	weed-vector: gRPC TLS to the masters and for the gateway (#12368) ... * weed-vector: read the cluster's security.toml The engine finds security.toml where weed processes do (., ~/.seaweedfs, /usr/local/etc/seaweedfs, /etc/seaweedfs) with WEED_* environment variables over it, and builds gRPC client and server credentials by the rules of security.LoadClientTLS and LoadServerTLS: a dedicated client certificate is preferred, both halves of a pair must be set, and grpc.ca is required once either is. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: gRPC TLS to the masters and from the gateway The masters are dialed with [grpc.client]'s credentials when the cluster has them, the master's certificate checked against the host dialed or --tls-server-name. The engine's own listener takes its identity from [grpc.vector] and then requires a client certificate chaining to grpc.ca, which is how the gateway, dialing with [grpc.client], reaches it. The rustls provider is pinned before the first TLS use, as the other binaries linking both aws-lc-rs and ring do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * scaffold: document [grpc.vector] Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: put every gRPC hop under mutual TLS in the hardened leg VECTOR_TLS=1 now also writes [grpc.master], [grpc.volume], [grpc.filer], [grpc.s3], [grpc.client] and [grpc.vector], and probes the master's and the engine's gRPC ports: plaintext HTTP/2 and a client without a certificate are refused, the client certificate gets a SETTINGS frame. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: give the test volume server 64 volume slots Every collection grows 7 volumes and test_vector_bucket_encryption opens two buckets while the previous test's bucket is still being torn down, so 16 slots were marginal: the Go legs failed CreateIndex now and then with "failed to find writable volumes" and master.log showing "create 7 volume, created 4: Not enough data nodes found!". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: reload the listener's certificates and enforce allowed_commonNames The listener now runs the way weed servers run theirs. The certificate, key and CA files are re-read every WEED_TLS_CERT_REFRESH_INTERVAL (weed's default of 5h) and on SIGHUP, and new handshakes use the changed files without a restart; a file that no longer parses is logged and the working configuration stays. [grpc.vector] allowed_commonNames and grpc.allowed_wildcard_domain restrict which client certificates are served, checked against the subject common name as security.Authenticator and the Rust volume server's CommonNameVerifier do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * scaffold: allowed_commonNames and the refresh cadence for [grpc.vector] Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the engine's allowlist keeps an outsider certificate out The hardened leg sets [grpc.vector] allowed_commonNames = "localhost" and probes the engine with a second certificate the same CA signed for CN=outsider: the handshake must be refused, while the gateway's certificate still gets an HTTP/2 SETTINGS frame. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: take the master's TLS name from the parsed endpoint A master given without a port (https://master.example) had the whole URL checked against its certificate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: bound the master TLS handshake connect_timeout covers only the TCP connect, so a master that accepted the connection and never answered the handshake held up the other masters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: look up security.toml keys the way viper does An empty WEED_* variable no longer hides the file's value, and keys match regardless of case, as for weed processes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: refuse master allowlists it cannot enforce weed checks a dialed server against allowed_server_commonNames and allowed_server_wildcard_domain on top of grpc.ca. The engine's master dial accepted every certificate the CA signed even when one was set; it now fails at startup and names the key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: read the whole frame header in the gRPC TLS probe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: probe the volume, filer and S3 gRPC listeners too Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read the master dial credentials only when given masters A standalone engine never dials with grpc.client, so its settings no longer stop it from starting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: run each listener TLS handshake on its own task The accept loop awaited every handshake before accepting the next connection, so a peer that connected and sent nothing stopped every gateway connection after it. Handshakes now run concurrently, each under a 10s timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read the master dial's TLS files at every dial The client certificate, key and CA were read once at startup, so after the files rotated a reconnect still presented the old pair. Each dial now reads them, as the listener re-reads its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read every unit Go durations accept in the refresh interval WEED_TLS_CERT_REFRESH_INTERVAL values in ns, us or a leading + fell back to five hours in the engine while weed servers honored them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: fall back on a refresh interval too long to hold An oversized WEED_TLS_CERT_REFRESH_INTERVAL panicked the engine at startup; weed servers reject it and keep the default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: refuse allowed_commonNames without the listener's certificate With [grpc.vector] allowed_commonNames set but no cert and key, the listener served plaintext and checked no client at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read the master dial's credentials inside its connect timeout The CA, certificate and key were read synchronously before the timeout started, so a stalled credential path held up every other master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: let a set key win over an empty spelling of it With allowed_server_commonNames = "" beside allowed_server_commonnames = "master", the lookup stopped at the empty exact match and the allowlist went unnoticed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-11-03 11:42:49 -08:00
Chris LuandClaude Opus 5.5	2ec0623073	filer UI, admin: upload a folder (#12371) ... * admin: place uploaded files by their path inside an uploaded folder The multipart file name keeps only the base name, so the upload API takes an optional relativePaths field listing each file's path under the target directory. Paths that are empty, absolute or contain . or .. components are rejected per file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: upload a folder from the file browser The upload modal gets a folder picker, and a folder dropped on the listing is walked into, so its files upload with their subfolder paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer UI: upload a folder An Upload Folder button picks a whole folder, and a folder dropped on the listing is walked into. Each file is posted to its subfolder under the current directory, which the filer creates on the first upload into it. Progress rows are keyed by that path, so equal names in different subfolders no longer share a row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: reject an empty relativePaths field instead of ignoring it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: upload a large selection in batches below the multipart part limit Go's multipart parser rejects a form of more than 1000 parts, so a folder of about a thousand files failed as a whole. The upload modal now sends 500 files per request and sums the results; progress covers all batches, and the button stays disabled until the last one finishes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: escape file names in upload error messages A failed upload reports the client-supplied file or folder path, and the toast renders its message as HTML. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: list failed uploads in the modal and reload the listing when it closes A partial upload logged its failures only to the console before the page reloaded, and a batch that failed after earlier ones succeeded left the listing stale. The failures now stay in the open modal, and closing it reloads the listing once any file was uploaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: cancel the upload when its modal closes Closing the modal, including with its Cancel button, left the batch loop running, so the remaining batches were still written. The batch in flight is now aborted, which also stops the loop, and the listing reloads since the aborted request may already have stored files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: count upload progress by files when the selection has no bytes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep the newest upload selection when a folder drop finishes late Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: cap the upload preview at a fixed number of rows A folder of many thousands of files rendered one preview row per file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer UI: upload a few files at a time Every selected file started its own request at once, so a large folder opened thousands of requests together. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: regenerate the gzipped admin.js Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: reload the listing after a batch fails without a result A batch that fails mid-request may still have stored some of its files, so closing the modal reloads the listing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: stop reading a folder drop once a newer selection replaces it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: skip the read error of a folder drop a newer selection replaced Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer UI: share the four-upload limit across selections Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: regenerate the gzipped admin.js Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: set up the file browser handlers once admin.js already runs setupFileManagerEventHandlers on DOMContentLoaded, so the file browser page registered every listener twice and each drop was read twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 11:35:32 -08:00
Chris LuandClaude Opus 5.5	d5e359d0e2	rust volume: run each gRPC TLS handshake on its own task (#12372) ... * rust volume: run each gRPC TLS handshake on its own task The incoming stream awaited every handshake before accepting the next connection, so a peer that connected to a TLS gRPC port and sent nothing stopped every gRPC connection after it. Handshakes now run concurrently, each bounded by grpc-go's default 120s connection timeout, as a Go server's are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rust volume: retry a failed gRPC accept instead of closing the listener grpc-go backs off and retries temporary accept errors such as EMFILE. The TLS incoming stream ended on the first accept error, which stopped the gRPC port for the rest of the process lifetime. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rust volume: test that the gRPC handshake timeout closes a silent peer The handshake timeout becomes a parameter so a test can use a short one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 11:23:07 -08:00
Chris LuandClaude Opus 5.5	3b7706bad8	rust volume: exit cleanly on SIGTERM, and make the harness notice when a server does not (#12370) ... * rust volume: exit cleanly on SIGTERM and SIGINT The shutdown path raced the HTTP and gRPC tasks with select!, then awaited both handles again; tokio panics when a JoinHandle is polled after it completed, so every signal ended in "JoinHandle polled after completion" with exit status 101 and the volumes never closed. Waiting for both tasks with join! is enough: both stop on the shutdown signal, and a gRPC serve error sends it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/volume_server: fail when a stopped server does not exit cleanly The harness ignored the exit status after SIGINT, which is how the Rust volume server's shutdown panic went unseen. Every server it stops must now exit 0 within 10s. The Go volume servers get -preStopSeconds=0, which the Rust ones already had, so a stop does not spend the default 10s grace inside that window. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/volume_server: let assertArgPair find flags after a bare flag rustVolumeArgs carries --volume.allowUntrustedRemoteEndpoints without a value, so pairs no longer sit at even indexes and TestRustVolumeArgsIncludeReadMode reported --readMode missing although it was there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * rust volume: stop both servers when either one exits A gRPC task that panicked left the HTTP server and heartbeat running until a signal arrived. Whichever server task ends first now sends the shutdown and the other is awaited once, matching Go where a serve failure is fatal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rust volume: report an HTTP task panic as a server error Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-11-03 10:44:01 -08:00
Chris LuandClaude Opus 5.5	cadb614630	filer.sync: stop chunk retries with the same supersession check the skip uses (#12367) ... * filer.sync: pass the replayed entry down to the chunk retry gates The fetch, manifest-resolve, and manifest-upload retry gates only received the replayed version's mtime, which is not enough to order it against the live source entry. Pass the entry itself; the gates still compare mtime only, so behavior is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.sync: stop chunk retries with the same supersession check the skip uses The fetch, manifest-resolve, and manifest-upload retry gates stopped early only for a gone or strictly newer source entry. A transient error therefore kept retrying a version the source had rewritten under an older mtime until the read recovered, while a live lookup that lags the replay could stop the retries on its newer mtime alone. The error handler that runs once a gate stops already decides with sourceMovedOn, which orders by commit timestamp and compares chunk sets; the gates now use it too, and hasSourceNewerVersion has no callers left. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.sync: a lagging live lookup cannot prove a chunk dropped A peer filer's event reaches subscribers before the source store applies it, so the live lookup can return the version the replay replaces. That entry never listed the replay's new chunk, so sourceDroppedChunk read a 404 on it as an obsolete version and cut the missing-chunk wait short. Like sourceVersion, it now ignores a live entry committed before the replayed version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.sync: test that a replicate failure on a superseded version is skipped Covers onReplicateChunkError with a transient error for a deleted source entry, a later commit with an older mtime, and an unchanged source, and points the unconfirmed-supersession test comment at it instead of the helper test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.sync: test the manifest gate against a missing volume and a lagging lookup Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 10:43:22 -08:00
Chris LuandClaude Opus 5.5	8461ae2fe9	filer.sync: skip a rewritten source version regardless of its mtime order (#12366) ... * filer.sync: a chunk the live source entry dropped is gone whatever its mtime says sourceDroppedChunk refused to judge a 404'd chunk when the live source entry carried an older mtime than the replayed version, treating that as a rollback. Timestamp-preserving writers (cp -p, rsync -a, tar extracts) produce exactly that order on every rewrite, so the vacuumed chunk sat in the retry loop for the full missingSourceChunkGrace before being written off. Whether the live entry still lists the chunk is independent of mtime order. * filer.sync: treat a rewritten source entry as moved on regardless of its mtime sourceVersion only compared chunk sets when the live entry was at least as new as the replayed one, so a live entry stamped with an older mtime but holding different chunks was never recognized as the current version. The sink then fetched the replayed chunks, hit 404 on each vacuumed one, and the same mtime guard in onReplicateChunkError kept the event from being skipped. Whether the source rewrote the file is a chunk-set question, not an mtime one; the live version is synced in its place as before. * filer.sync: apply a substituted live snapshot even under a newer destination mtime After sourceVersion swapped in the live entry, UpdateEntry re-ran chooseUpdateAction on it and a destination whose mtime was above the live entry's read as a late update, so the current source content was never written. The live snapshot is the source's current state, not a replay, so mtime order says nothing about it; repair from it unless the destination already holds it. * filer.sync: a live entry committed before the replay does not supersede it Dropping the mtime guard let any chunk-set difference read as a rewrite, including a live lookup that lags the event: a peer filer's event reaches subscribers before the local store applies it, so the lookup can still return the version the replay replaces. sourceVersion then synced that older version over the destination and consumed the event. The meta-log commit stamps order the two versions where mtime cannot, so a live entry committed before the replay proves nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.sync: check commit order before the mtime supersession A lagging live entry can carry a newer mtime than the replay, as when the replayed version was stamped back to an older time, so sourceSupersedes took it as newer and the older version was synced in place of the replay. Rule out a live entry committed before the replay first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 09:36:04 -08:00
Chris LuandClaude Opus 5.5	5c99744e00	rust volume: count sorted-map files and deletes the way Go does (#12365) ... * rust volume: count sorted-map files and deletes the way Go does The Rust sorted needle map still used the counter rule Go dropped when the index loaders were unified: every .idx row, tombstones included, counted as a file, and each delete added one file and two deletions. Go counts put rows as files and a delete as one deletion of the live row's bytes, so the Go and Rust copies of one volume reported different FileCount and DeleteCount for the same index. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rust volume: test canDelete volume counters across restarts Mirrors the Go store test: a canDelete volume reloaded from .sdx must count its deletes, and the counters it reports must match what the next reload computes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 08:13:18 -08:00
3 people	5df7faf0e3	fix(storage): update deletion metrics in SortedFileNeedleMap.Delete (#12363) (#12364) ... * fix(storage): update deletion metrics in SortedFileNeedleMap.Delete (#12363) * storage: test canDelete volume counters across restarts A canDelete volume reloaded from .sdx must count its deletes so volume.deleteEmpty and vacuum see it drain, and the counters it reports must match what the next reload computes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 08:05:45 -08:00
Chris LuandClaude Opus 5.5	d13d975b6e	shell: keep fs.mergeVolumes and fs.distributeChunks from reverting entries written while they run (#12362) ... * shell: keep fs.mergeVolumes from reverting entries written during the merge The walk reads a page of entries and moves their chunks one at a time, then wrote each entry back unconditionally. A file overwritten in between got its old version back, and the filer deleted the new version's chunks as garbage, so an acknowledged write was lost without any error. The update now carries IF_ENTRY_EQUAL over the entry as the walk read it. When the filer refuses because the entry changed or was deleted, the copies made on the target volumes are deleted, their plan room is released, and the entry is skipped; the source needles stay where the filer still points. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep fs.distributeChunks from reverting a file overwritten while it runs The command looks the entry up once, moves its chunks, and then wrote that entry back unconditionally, so an overwrite during the redistribution was reverted and the filer deleted the new version's chunks as garbage. The update now carries IF_ENTRY_EQUAL over the looked-up entry. When the filer refuses because the file changed or was deleted, the moved copies and any new manifest chunks are deleted and the command reports it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep s3.versions.audit -heal from clearing a pointer that just moved The heal cleared the latest-version pointer on the directory entry the walk read, unconditionally. A version written between the audit and the heal has already pointed the directory at a live file, and clearing that pointer hid it. The heal now carries IF_ENTRY_EQUAL over the audited entry and reports a refusal as a failed heal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep fs.meta.changeVolumeId from reverting entries written meanwhile The walk rewrote each entry's volume ids and wrote back the copy it had read, unconditionally, so an overwrite between the read and the write was reverted and its chunks were deleted as garbage. The update now carries IF_ENTRY_EQUAL over the entry as read, and a refused entry is skipped and reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep the target copies another fs.mergeVolumes run committed moveChunk keeps the key and cookie, so two overlapping merges of the same source into the same target write the same fid. When one run commits, the other's update is refused, and deleting its copies deleted the needle the committed entry now points at. A refused update now re-reads the entry, resolving its manifests, and deletes only the copies it no longer references. If the entry cannot be read, the copies are left for volume.fsck. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: let a later hard-link sibling retry a merge the filer refused The first sibling of a HardLinkId claims the move for all of them. When its update is refused, the claim made every later sibling skip, so the shared chunks stayed on the source volume for the rest of the run. A refused update now releases the claim, and a sibling listed after the change retries the move against the entry as it is now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: delete the manifest blobs a failed fold in fs.distributeChunks saved When re-manifestizing failed partway, the blobs already uploaded were only counted in a log line and stayed on their volumes with nothing referencing them. They have freshly assigned fids, so they are now deleted with the cookie-checked delete the other orphan cleanups use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: check the heal clears the pointer file name too Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: require the shell lock for fs.mergeVolumes -apply moveChunk keeps the key and cookie, so two overlapping runs that move the same chunk into the same volume write the same fid. Re-reading the entry before deleting a refused run's copies covers the run that already committed, but not one that commits after that read. With the lock, two applying runs can no longer overlap, as volume.fsck and volume.merge already require. A dry run still works without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: check unreferencedTargets reads the refused entry itself Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: check unreferencedTargets keeps a copy a manifest references Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: check unreferencedTargets keeps the manifest copy and skips an unreadable manifest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 07:59:18 -08:00
4 people	76198d15cf	fix(fsck): inspect references to deleted volumes (#12357) ... * fix(fsck): inspect references to deleted volumes * volume.fsck: treat every match-all collection filter as global ALL_COLLECTIONS and EACH_COLLECTION compile to the same match-all matcher as an empty filter or "*", but the string check treated them as a restricted scope: a selected missing volume was rejected and references to master-absent volumes were reported without being purged. Use the compiled matcher instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume.fsck: purge an entry whose manifest is on a missing volume A top-level chunk on a missing volume already proves the entry is unreadable, whether or not its manifests resolve. Skipping such entries kept the very case where the manifest needle sat on the retired volume, and the remaining unresolved manifest then blocked every later orphan purge. The missing-needle check on live volumes never had this gate, so both paths now treat these entries the same. The warning now says only orphan purging is disabled, which is the only thing the counter gates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume.fsck: report a failed Filer delete A rejected purge DELETE printed nothing unless -v was set, so the run looked successful after "deleting path" while the entry stayed. The filer answers 204 for an entry that is already gone, so only real failures print. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume.fsck: tighten help text, comments and missing-volume tests The help now wraps like the rest of it and says that a purge removes the whole entry, including its chunks on other volumes. The live test header no longer calls the run report-only, and the reverse-scan test checks the report line for each missing chunk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume.fsck: drop the missing-volume design notes The notes said no destructive change was approved while the change adds a confirmed purge, still described the dropped manifest gate, and mostly recorded local runs and a reverse index that is not part of this change. The operator contract lives in help volume.fsck. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume.fsck: allow a report-only missing-volume scan with -collection The collection check exists to stop a purge that cannot be scoped, but it also rejected a report-only run that deletes nothing. Apply it only with -reallyDeleteFilerEntries, and limit the -confirmMissingVolumeDeletion description to the -volumeId case it guards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume.fsck: count only a 2xx answer as a purged entry A redirect the client could not follow left the entry in place but was recorded as deleted, which also queued its parent for empty-directory cleanup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume.fsck: say that a filtered purge of a missing -volumeId is refused The help and the early warning both said a purge with -collection only reports references to unlisted volumes, but a -volumeId naming one is rejected before the scan. The help now says so, and the warning prints after that check, so a refused run no longer claims it will report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: MZ01-CHOIMINSU <mz01-juwannam@MZ01-CHOIMINSU.local> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 07:57:43 -08:00
Chris LuandClaude Opus 5.5	e8b284fef5	admin: scope plugin job and activity fetches to the selected job type (#12360) ... * admin: scope plugin job and activity fetches to the selected job type The plugin page fetched the newest 500 jobs and 500 activities cluster-wide and filtered them by job type in the browser. On a busy cluster that window can hold none of the selected type (433 erasure_coding + 64 ec_balance jobs, 497 ec_balance activities), so /plugin/lanes/default/execution?job=ec_vacuum showed "No execution jobs tracked yet" while the server had seven recent ec_vacuum jobs. When a job-type tab is active, fetch the queue/detection/execution lists with job_type=<selected> so the server filters before applying the limit, and refetch when the selected job type changes. The unscoped lists are still fetched for the status cards and the overview per-type summary. * admin: drop a plugin monitor refresh whose job type is no longer selected A refresh started on one job-type tab could finish after the user moved to another tab and that tab's refresh had already landed. Its rows, all of the earlier job type, then replaced the current tab's and the renderers filtered them down to empty tables until the next refresh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: show the lane-wide rows when a job-type scoped fetch fails On a job-type tab a failed scoped jobs or activities request threw before anything was stored, so the status cards, scheduler state and tables all stayed stale. Fall back to the unscoped rows, which the renderers already filter by job type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: filter plugin jobs and activities by lane before the limit /api/plugin/jobs and /api/plugin/activities took the newest N rows across every lane and then dropped the other lanes' rows, so on a busy default lane a quieter lane's page could come back empty. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep a failed plugin monitor fetch's last rows and report it Falling back to the lane-wide rows when a scoped fetch failed could still blank the tables, since that window may hold none of the selected type, and the failure went unreported. A lane-wide or scheduler-state failure also threw before the scoped rows that did arrive were stored. Each request now keeps its previous value on failure: the scoped rows only while the tab is unchanged, otherwise the lane-wide rows. Whatever arrived is stored and rendered, then the first failure is rethrown so the caller reports it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: report a monitor refresh failure apart from the action it follows expireJob and runWorkflow refreshed the monitor inside the action's try block, so a failed refresh after a successful expire or workflow run was reported as a failed action, and the expired job's detail was not reloaded. Catch the refresh separately, as runDetection already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: show a repeated plugin page refresh failure once refreshAll runs every 3 seconds and notify opens a modal, so a request that kept failing reopened the same alert on every tick. Report an error only when it differs from the last one, and forget it once a refresh succeeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 06:32:49 -08:00
Chris LuandClaude Opus 5.5	222cfa8e2e	admin: summarize the EC delete_count accounting warning per refresh (#12359) ... * admin: summarize the EC delete_count accounting warning per refresh collectCollectionStats runs on every admin collection, bucket and topology refresh, and it logged a Warning for each EC volume whose delete_count summed across shard holders exceeds its file_count. That sum is an expected accounting artefact (a tombstone can be recorded on more than one holder), it recurs identically on every pass, and on one cluster with hundreds of such volumes it grew the admin log to 20 GB, more than 99% of it this one line. Log one V(1) summary per pass with the affected volume count and keep the per-volume detail at V(2). The object count is still skipped for those volumes. * admin: test that an EC volume with deletes above file_count is skipped Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 06:31:15 -08:00
Chris LuandClaude Opus 5.5	e5bc658eb7	test/s3vectors: a hardened cluster with TLS volume servers, read JWTs and encrypted chunks (#12356) ... * test/s3vectors: a hardened cluster, with TLS volume servers, read JWTs and encrypted chunks Three switches on the test cluster: VECTOR_TLS=1 puts the volume servers behind HTTPS requiring client certificates and enables https.client for every process; VECTOR_READ_JWT=1 gives every process a jwt.signing.read.key, so volume reads need a token; VECTOR_ENCRYPT_VOLUMES=1 starts the gateway with -encryptVolumeData. The integration matrix gains a hardened leg per volume server with all three on. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the hardened cluster proves its enforcement Once the gateway is up, the harness checks that the hardening holds, not only that it is configured: a TLS volume server refuses plain HTTP (Go with HTTP 400, Rust with a TLS alert) and a connection without the client certificate, and answers with it; under a read key an unsigned read of a real chunk, written through the gateway and located through the filer, gets 401. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the hardened cluster proves its chunks are encrypted With VECTOR_ENCRYPT_VOLUMES the probe reads the stored chunk straight from the volume server, signing the read when a read key is set, and requires a recorded cipher key and bytes that differ from the plaintext. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: the signed probe read must return the chunk An empty 200 passed both branches; an unencrypted read now has to return the bytes written, and an encrypted one has to return something else. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: a read key in the caller's environment does not leak into the cluster Only VECTOR_READ_JWT turns read JWTs on; an inherited WEED_JWT_SIGNING_READ_KEY made the volume servers refuse the probe's unsigned read in an encrypt-only run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: quote certificate paths as TOML strings in security.toml Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: keep non-ASCII certificate paths as UTF-8 in security.toml TOML has no surrogate-pair escapes, and the Rust volume server reads the quoted value without decoding escapes at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: an encrypted probe chunk must not be a readable gzip stream The gateway may gzip the body before encrypting it, so a chunk uploaded compressed but unencrypted would differ from the body and still pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-11-03 04:56:15 -08:00
Chris LuandClaude Opus 5.5	af45ab8f2a	s3api: known-answer fixture generator for SSE-S3 and filer-encrypted chunks (#12355) ... * s3api: a known-answer fixture generator for SSE-S3 and filer-encrypted chunks TestGenerateDirectReadFixture writes, from this package's own encryption, the pieces a reader outside the gateway must handle: SSE-S3 chunks with their per-chunk metadata, the object-level key, a single-stream ciphertext, the KEK derived from s3.sse.key and wrapped under a passphrase in both on-disk formats, and chunks encrypted by the filer, plain and gzipped. It is skipped unless DIRECT_READ_FIXTURE names the output file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3api: the fixture generator restores the key manager and checks the write The test set the global key manager's KEK and left it; a later test in the same run then found a signing key where it expects none. The previous key is restored with t.Cleanup. The fixture file's close error is checked too, so a delayed write failure fails the test instead of leaving a short fixture. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3api: the fixture generator unwraps both KEK formats before writing The legacy v1 wrap is built by hand in the test; reading v1 and v2 back through unwrapKEK ties both to the format the key manager accepts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: pin a legacy v1 wrapped KEK made outside the package Generating the v1 sample through legacyKEKWrappingSalt let a change to the salt pass on both sides; a vector wrapped elsewhere under the historical salt must still unwrap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-11-03 04:53:57 -08:00
3 people	199b2e2dd9	fix(s3api): reject an invalid storage class before uploading the body (#12354) ... * fix(s3api): reject an invalid storage class before uploading the body * fix(s3api): reject an invalid storage class on CreateMultipartUpload * fix(s3api): validate storage class and tags before reading the PUT body * fix(s3api): validate storage class and tags with the other PUT headers Checking them next to ValidateRequestEncryption keeps the check ahead of any later hook that may answer the PUT without reaching putToFiler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(s3api): reuse validatePutHeaders in putToFiler Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(s3api): store storage class and tags on directory-marker PUTs The marker path validated x-amz-storage-class and x-amz-tagging but its mkdir callback never wrote them, so a valid value was accepted and then dropped and GetObjectTagging on the marker came back empty. Store both the way putToFiler does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(s3api): read directory-marker tags back through GetObjectTagging Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 22:08:21 -08:00
Chris LuandClaude Opus 5.5	c2ca83c0bf	filer: say jwt.filer_signing.key gates only the IAM gRPC service (#12353) ... * filer: say jwt.filer_signing.key gates only the IAM gRPC service The key never covered the rest of the filer gRPC port, which can read and write the IAM store under /etc/iam. Point operators at [grpc.filer] mTLS for a port reachable beyond a trusted network instead of the key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: note the IAM gRPC OIDC provider and role RPCs are refused without the key Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 18:00:59 -08:00
github-actions[bot]	cc71cc9585	docs: regenerate star history chart	2026-11-03 01:08:52 +00:00
Chris LuandClaude Opus 5.5	9b45aae2eb	operation: pad the marginal gzip test past Go 1.26's stored block (#12352) ... At BestSpeed, Go 1.26's flate writes 4000 random bytes plus a 300-byte zero run as a stored block, so the gzip output was 28 bytes larger than the input and the test failed its own precondition on the toolchain go.mod pins. With 450 zero bytes the saving is 5.7% on Go 1.26 and 8.4% on Go 1.27, inside the marginal range on both. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 17:01:40 -08:00
dependabot[bot]	794cebe7b0	build(deps): bump cloud.google.com/go/kms from 1.35.0 to 1.36.1 (#12342) ... Bumps [cloud.google.com/go/kms](https://github.com/googleapis/google-cloud-go) from 1.35.0 to 1.36.1. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/documentai/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/kms/v1.35.0...dlp/v1.36.1) --- updated-dependencies: - dependency-name: cloud.google.com/go/kms dependency-version: 1.36.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-11-02 13:01:04 -08:00
Chris LuandClaude Opus 5.5	8125a1549a	s3vectors: support tag conditions in vector bucket policies (#12351) ... * s3tables: resolve s3vectors:VectorBucketTag like s3tables:TableBucketTag Both name a tag on the bucket that holds the target, whichever resource the request acts on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: give vector policy conditions the bucket, index and request tags A vector bucket policy is evaluated with these condition keys: - aws:ResourceTag/<key>: the tags of the bucket, or of the index when the request names one. - s3vectors:VectorBucketTag/<key>: the bucket's tags, for bucket and index requests alike. - aws:RequestTag/<key> and aws:TagKeys: the tags TagResource and CreateIndex set, and the keys UntagResource removes. An index's tags are read only when the policy has an aws:ResourceTag condition. The bucket's come from the lookup that already reads its policy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: accept tag conditions in vector bucket policies PutVectorBucketPolicy rejected every Condition, since no condition key was supplied. A condition is now accepted when its operator is one the evaluator knows and each key is aws:ResourceTag/<key>, aws:RequestTag/<key>, aws:TagKeys or s3vectors:VectorBucketTag/<key>. Any other key would never match, so it is still rejected, as is a condition field not spelled Condition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: cover tag conditions in vector bucket policies Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: reject condition values the evaluator cannot compare A null condition value was accepted and then panicked the request that matched its statement, a Deny reaching even the bucket owner. A value must now be a string, number or boolean, or a non-empty list of them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 12:43:12 -08:00
dependabot[bot]	97e03f5a9f	build(deps): bump actions/setup-go from 5 to 7 (#12347) ... Bumps [actions/setup-go](https://github.com/actions/setup-go) from 5 to 7. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/v5...v7) --- updated-dependencies: - dependency-name: actions/setup-go dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-11-02 12:35:26 -08:00
dependabot[bot]	b91de44615	build(deps): bump github.com/peterh/liner from 1.2.2 to 1.2.3 (#12341) ... Bumps [github.com/peterh/liner](https://github.com/peterh/liner) from 1.2.2 to 1.2.3. - [Commits](https://github.com/peterh/liner/compare/v1.2.2...v1.2.3) --- updated-dependencies: - dependency-name: github.com/peterh/liner dependency-version: 1.2.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-11-02 12:01:10 -08:00
dependabot[bot]	20c8876a52	build(deps): bump github.com/aws/aws-sdk-go-v2/service/sts from 1.51.1 to 1.51.6 (#12346) ... build(deps): bump github.com/aws/aws-sdk-go-v2/service/sts Bumps [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) from 1.51.1 to 1.51.6. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.51.1...service/sts/v1.51.6) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/sts dependency-version: 1.51.6 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-11-02 12:00:58 -08:00
dependabot[bot]	a670553813	build(deps): bump google.golang.org/api from 0.297.0 to 0.310.0 (#12343) ... Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.297.0 to 0.310.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.297.0...v0.310.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-version: 0.310.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-11-02 12:00:36 -08:00
dependabot[bot]	9ae31fa425	build(deps): bump actions/setup-python from 5 to 7 (#12345) ... Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5 to 7. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v5...v7) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-11-02 12:00:20 -08:00
dependabot[bot]	e166629f9b	build(deps): bump go.etcd.io/etcd/client/pkg/v3 from 3.7.1 to 3.7.3 (#12344) ... Bumps [go.etcd.io/etcd/client/pkg/v3](https://github.com/etcd-io/etcd) from 3.7.1 to 3.7.3. - [Release notes](https://github.com/etcd-io/etcd/releases) - [Commits](https://github.com/etcd-io/etcd/compare/v3.7.1...v3.7.3) --- updated-dependencies: - dependency-name: go.etcd.io/etcd/client/pkg/v3 dependency-version: 3.7.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-11-02 11:59:59 -08:00
Chris LuandClaude Opus 5.5	b7b2a5947f	admin: configurable file browser upload size limit (#12348) ... * admin: carry a file browser upload size limit The admin server keeps the total size limit of one file browser upload, 500 MB by default as the UI enforced before, and hands it to the file browser page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: reject file browser uploads over the size limit The 500 MB limit was only checked in the browser, so any client could post larger uploads and fill the temp dir with spooled parts. Cap the request body at the limit plus room for the multipart framing, and answer 413 when the files add up to more than the limit. Parse the form before reading the path field. FormValue used to parse it first with the default 32 MB memory, which made the later 1 GB ParseMultipartForm a no-op; keep 32 MB explicitly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: show and check the configured upload limit in the file browser The upload dialog reads the limit from the page instead of a hard-coded 500 MB, lists it under the file picker, and skips the check when uploads are unlimited. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: add -uploadMaxSizeMB to set the file browser upload limit weed admin -uploadMaxSizeMB and weed mini -admin.uploadMaxSizeMB set the total size limit of one file browser upload; 0 removes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: note why the upload framing allowance is enough Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: test an upload whose files only exceed the limit together Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: remove spooled upload parts when the upload handler returns Parts over 32 MB spool to temp files. The router passes the handler a copy of the request, so net/http never sees the parsed form and never removes those files. Every large upload, and every one rejected for size after parsing, left them on disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 11:32:03 -08:00
Chris LuandClaude Opus 5.5	a074538c8f	admin: manage S3 vector buckets (#12334) ... * s3vectors: run an operation for trusted local tooling Like s3tables.Manager.Execute, Server.Execute runs one S3 Vectors operation through its handler as a given identity, without a signed request. The admin console can then manage vector buckets through the same validation, authorization and catalog steps as the API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * admin: list, inspect, create and delete vector buckets over the API /api/s3vectors runs the S3 Vectors operations through Server.Execute as the admin account: list vector buckets with their encryption, read one bucket with its policy and each index's data type, dimension, distance metric and non-filterable keys, create and delete a bucket, and delete an index. Going through the operations keeps the API's rules, such as ICEBERG buckets not being vector buckets and the implicit index namespace not blocking a delete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: export the rule for which table buckets are vector buckets IsVectorBucketFormat names the check ListVectorBuckets and isVectorBucket already make, so the admin console can tell the same buckets apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * admin: vector bucket pages Object Store > Vector Buckets lists the vector buckets with their ARN, encryption and creation time, creates and deletes them, and opens one bucket with its indexes (data type, dimension, distance metric, non-filterable keys) and its policy, where an index can be deleted. A LANCE row on the Table Buckets page links to the same bucket's vector page, since both APIs serve it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * admin: stop the vector bucket and index lists at the list limit A page cut short by filtered ICEBERG buckets let the next page carry the list past s3TablesAdminListLimit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: say when a vector bucket or index list is cut at the limit The lists stop at s3TablesAdminListLimit while more remain, so mark them truncated and point to an S3 Vectors client for the rest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: indent the vector bucket policy A policy stored as one line ran off the right edge of the details page. Policies that are not valid JSON are shown as stored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: show a deeply nested vector bucket policy as stored Indenting multiplies the size of a deeply nested document, so a policy past 32 levels of nesting is shown without indenting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 11:25:24 -08:00
Chris LuandClaude Opus 5.5	0176231ea4	volume: replicas keep the primary's compression and metadata (#12350) ... * volume(rust): replicate the stored needle instead of the request body The primary forwarded the original request body and headers, so a replica re-derived compression from the raw bytes. Send what Go's ReplicatedWrite sends: a one-part multipart upload of the stored bytes, gzip-marked when compressed, with the name, mime, pairs and Content-MD5, and carry ts, ttl, cm and fsync in the query so form-field values reach the replica too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): keep the primary's compression on replica writes A type=replicate upload now stores the bytes as received, like Go's ParseUpload, so a replica no longer compresses data the primary stored uncompressed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): replicas keep the primary's name and mime Forward the primary's filename and mime even past the 255-byte needle fields, so a replica applies the same limit and sets the same flags, and skip the filename MIME guess on type=replicate, so an untyped upload stays untyped on every copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): keep the multipart part Content-Type as sent Parsing the part type as a mime value lowercased it and dropped values it could not parse, so a replica stored a different mime than the primary. Read the header verbatim, as Go's part.Header.Get does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: replicas keep the primary's mime as forwarded A Go replica guessed the mime from the file extension, so an upload the primary stored as application/octet-stream, or with no mime at all, got a different type on the replica. On type=replicate keep the part's Content-Type as sent, and have the replication sender leave an empty mime empty instead of filling it in from the extension. This matches what the Rust replica does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): pick the JPEG orientation fix by file name, as Go does Go checks the mime and the upload's file name, Rust checked the mime and the URL path, so the same upload could be rotated on one server and not on the other. With the name and mime forwarded as stored, a replica now reaches the primary's decision, and a rotated image has no EXIF tag left to act on, so the fix stays a no-op there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 11:24:51 -08:00
Chris LuandClaude Opus 5.5	ec968aa016	s3vectors: support TagResource, UntagResource and ListTagsForResource (#12335) ... * s3vectors: tag vector buckets and indexes TagResource, UntagResource and ListTagsForResource answer on /tags/{resourceArn} (POST, DELETE, GET) and by name. Each authorizes its own s3vectors action on the bucket or index ARN, then reads or writes the catalog record's tags. A resource holds at most 50 tags. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: apply the tags passed to CreateVectorBucket and CreateIndex The tags field was accepted and dropped. It is now stored on the new bucket or index, and creating with tags needs s3vectors:TagResource in addition to the create action. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: cover resource tagging end to end Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: keep tag operations to a bound table bucket TagResource, UntagResource and ListTagsForResource run through Execute with WithTableBucketCreatedAt now answer not found when the bucket name holds another bucket, as the bound creates and deletes already do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: bind tagging to the authorized bucket and to vector indexes The catalog tag calls carry the creation time of the bucket authorizeVector approved, so a bucket created again under the name is not touched. An index ARN must name a Lance table: another table in the vector namespace of a format-less bucket is refused, as DeleteIndex does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: match a TagResource grant scoped to the bucket being created Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: merge tag writes into the tags the record holds at the write TagResource and UntagResource merged into the tags they read before the compare-and-swap and wrote that map on every attempt, so a tag write that landed in between was undone. The merge now runs inside the mutation. WithMaxTags lets an in-process caller cap the merged set there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: check the 50-tag limit inside the catalog's tag write Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: check a tag operation's bound bucket after its reads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: validate each tag key UntagResource removes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: validate untag keys one at a time UntagResource sized a map by the number of submitted tag keys before validating them, so a large body of repeated keys allocated far more than the request itself, before authorization ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: authorize UntagResource before validating its keys Validating a near-32 MiB body of tag keys costs about 13x its JSON decode, so an unauthorized caller could make the gateway spend that work on a request it then refuses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 10:33:26 -08:00
Chris LuandClaude Opus 5.5	690aabda0e	filer: reuse IsEqualOrUnder for the rename subdirectory check (#12338) ... * filer: test the rename subdirectory check through CanRename Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: reuse IsEqualOrUnder for the rename subdirectory check A trailing slash on the old name made the source path end in a slash, so a child of the source was no longer seen as under it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 10:28:48 -08:00
4 people	d650abe0e7	compression: compress JSON uploads, and match file types by extension (#12339) ... * compression: compress JSON uploads, and match file types by extension IsCompressableFileType did not recognise JSON media types, so S3 uploads of application/json (which carry no file name) were never compressed. Its callers also passed filepath.Base of the file name where it expects the extension, so the .json/.js/.html/... extension rules never matched a real file name. Both callers now pass filepath.Ext; the function lower-cases the extension and media type, ignores media-type parameters ("application/json; charset=utf-8"), and treats */json and *+json types as compressible. * compression: treat application/json-seq as JSON Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): compress JSON uploads and match media types without parameters Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * operation: keep uploader gzip only when it saves at least 10% Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Pat White <5015863+patwhite@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 10:15:01 -08:00
Jiri PucandClaude Opus 5.5	1637941368	filer: allow renaming into a sibling that shares the name prefix (#12337) ... CanRename rejected a move whenever the new parent directory's path started with the source path as a plain string, so moving /dir/a into /dir/a.d (or anything under /dir/ab) failed with "can not move directory to a subdirectory of itself". Check the path boundary instead: reject only when the new parent is the source itself or lies under it. Claude-Session: https://claude.ai/code/session_01VncHKTehVroSAzRMUHykEX Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 10:09:22 -08:00
Chris LuandClaude Opus 5.5	7298b4ed37	fix(s3tables,s3vectors): bind DeleteNamespace and the vector bucket cleanup to the authorized records (#12331) ... * s3tables: bind DeleteNamespace to the authorized namespace The delete claimed whatever finished namespace held the name at claim time, so one deleted and created again by another owner after the authorization read was removed under the first one's authorization. The claim now requires the creation time that was authorized. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: bind DeleteVectorBucket's namespace cleanup to the authorized bucket After the bound DeleteTableBucket reported the bucket not empty, the index namespace cleanup ran unbound, so a bucket deleted and created again in between lost its empty index namespace under the first bucket's authorization. DeleteNamespace now honors the bucket binding, checking the bucket after reading the namespace, and the cleanup carries it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 09:55:09 -08:00
github-actions[bot]	984eac6a79	4.52 4.52	2026-11-02 11:24:06 +00:00
Chris LuandClaude Opus 5.5	aa2d94cc50	ci: cancel a PR's runs when it is merged or closed (#12330) ... A merged or closed PR kept every pull_request run going: concurrency groups cancel a run only when a newer push arrives. After #12329 merged, 16 of its runs were still queued or running. On pull_request_target closed, cancel the queued and running pull_request runs on the PR's head branch whose commit belongs to the PR. Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 03:21:26 -08:00
Chris LuandClaude Opus 5.5	66a6598fdd	fix(s3tables,s3vectors): bind table and vector bucket writes to the bucket they were authorized against (#12329) ... * s3tables: bind table bucket policy writes to the authorized bucket PutTableBucketPolicy compared only the policy bytes, and DeleteTableBucketPolicy removed the policy unconditionally. A bucket deleted and created again under the same name between authorization and the write took a policy change nobody authorized for it. Both writes now require the bucket's CreatedAt and the policy they were authorized against, answer 409 otherwise and 404 for a bucket that is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: bind bucket tag writes to the authorized bucket TagResource and UntagResource checked the creation time only for a table ARN. On a bucket ARN the merged tags landed on a bucket deleted and created again since authorization. extractResourceOwnerAndBucket now returns the resource's CreatedAt for both kinds, and the write requires it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: bind bucket maintenance writes to the authorized bucket PutTableBucketMaintenanceConfiguration wrote onto whatever bucket held the name at write time. authorizeMaintenanceBucket now returns the bucket's CreatedAt like authorizeMaintenanceTable, and putMaintenanceConfiguration takes the check and creation time from both callers instead of an optional closure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: bind vector bucket policy writes to the authorized bucket Put/DeleteVectorBucketPolicy authorized inside the compare-and-swap with the owner from an earlier bucket read, so a bucket deleted and created again by another account in between had its policy rewritten by the old owner. The mutation now requires the entry's createdAt to match the record it authorized against. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: bind DeleteVectorBucket to the authorized bucket DeleteVectorBucket authorized one bucket record, then ran the trusted catalog DeleteTableBucket on whatever bucket held the name by then, so an empty bucket deleted and created again by another account in between was deleted by the old owner. The catalog delete now accepts the authorized bucket's createdAt through an in-process context value and treats any other bucket as gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: bind CreateIndex and DeleteIndex to the authorized bucket Both authorized the bucket, then ran trusted CreateNamespace, CreateTable or DeleteTable on whatever bucket held the name by then, so an index could be created in, or deleted from, a bucket another account created again in between. Those catalog operations now honor the bucket binding DeleteTableBucket already takes, and the vector handlers pass the authorized bucket's createdAt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 03:00:30 -08:00
3 people	4850def831	fix(s3tables): bind table and view writes to the record they were authorized against (#12328) ... * s3tables: bind table and view writes to the record they were authorized against * s3tables: bind UpdateNamespace to the namespace it was authorized against UpdateNamespace writes back the whole record it read, owner and creation time included. A namespace deleted and created again before the write took the first one's owner and CreatedAt, which also undoes the CreatedAt binding creates now rely on. Refuse the write with 409 when the namespace at the path has another CreatedAt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 02:50:45 -08:00
Chris LuandClaude Opus 5.5	ce292a4e5e	s3: require Admin to put or delete a bucket's lifecycle configuration (#12327) ... PUT and DELETE ?lifecycle were authorized as Write, so an identity granted only object writes on a bucket could add expiration rules or remove the ones the bucket administrator set. They now need Admin on the bucket, like the encryption, public access block and ownership controls writes. Identities authorized by IAM policies are unaffected: they are still checked against s3:PutLifecycleConfiguration. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 01:26:16 -08:00
Chris LuandClaude Opus 5.5	d431c1b35a	s3api: let the vectors route, not the credential scope, raise the body limit (#12326) ... SigV4 verification picked the 32 MiB S3 Vectors body limit from the credential scope, which is not verified until the body is hashed. A request claiming the s3vectors scope on any other route could make the gateway buffer 32 MiB before authentication instead of 10 MiB. The vectors handler now sets the limit on the request context, and every other route keeps 10 MiB. Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 01:15:42 -08:00
Chris LuandClaude Opus 5.5	02d22800aa	s3vectors: accept bodies up to the 32 MiB limit, report larger ones as a size error, log engine failures (#12320) ... * s3api: hash an s3vectors body up to the Vectors API limit SigV4 verification of a non-S3 request hashed at most 10 MiB of the body and silently dropped the rest, so any S3 Vectors request over 10 MiB failed its signature check with 403, although the API accepts 32 MiB. The limit now follows the signing service, and a body over it is reported as ErrEntityTooLarge instead of being truncated; the embedded IAM answers that with InvalidInput, as it already does for a body over its own limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: answer an oversized request with a ValidationException A body over 32 MiB was refused as "403 AccessDenied: authentication failed". A declared Content-Length over the limit is now refused before authentication, and a streamed body over it, whether the signature check or the handler reads it, gets the same "request body exceeds 32 MiB" ValidationException. The limit is the one SigV4 verification uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: log engine and catalog failures at the default level An engine error reached the client as a bare "internal error", and the cause was logged only with -v=1, so a misconfigured engine left nothing in a default log. Engine and catalog failures are now logged as errors, and a reported unavailability as a warning, with the operation, bucket and index. The response still leaves the engine's message out, since it can name internal hosts and paths, and points at the gateway log instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3api: hash the signed body through readRequestBody streamHashRequestBody repeated readRequestBody's read-past-the-limit check; it now reads the body with readRequestBody and hashes the bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 00:18:44 -08:00
Chris LuandClaude Opus 5.5	2eb219df61	s3vectors: encrypt the objects of a vector bucket declared AES256 (#12324) ... * s3api: encrypt the objects of a vector bucket declared AES256 CreateVectorBucket recorded encryptionConfiguration sseType AES256 and returned it, but nothing on the write path read it: the index's data files, manifests and transactions were stored without SSE. A LANCE table bucket's record now supplies the bucket's default encryption whenever the bucket has none of its own, so PutObject, CreateMultipartUpload and CopyObject apply SSE-S3 to every object the engine writes. The S3 bucket metadata itself is unchanged. ICEBERG table buckets keep recording the declaration only: their catalog reads the manifests clients upload straight from the filer, which would see ciphertext. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3tables: refuse an AES256 vector bucket on a gateway without an SSE-S3 key A vector bucket's AES256 is now applied to every index write, and SSE-S3 fails each write when the gateway has no KEK. Creating the bucket there succeeded and the first CreateIndex came back as InternalFailure. CreateTableBucket now refuses a LANCE bucket declaring AES256 when the gateway reports no SSE-S3 key, naming s3.sse.kek and s3.sse.key; CreateVectorBucket returns it as ValidationException. ICEBERG buckets, whose declaration is only recorded, and tooling that sets no checker are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3api: refuse DeleteBucketEncryption on a bucket's declared encryption GetBucketEncryption reports a vector bucket's declared AES256, but DeleteBucketEncryption answered NoSuchBucketEncryptionConfiguration and the declaration kept applying. The declaration is fixed when the bucket is created, so the delete now returns MethodNotAllowed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): keep an inherited WEED_S3_SSE_KEK out of the test cluster The gateway refuses to start SSE-S3 with both s3.sse.kek and s3.sse.key set, so a developer shell exporting WEED_S3_SSE_KEK made the AES256 bucket test fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-01 23:57:40 -08:00
Chris LuandClaude Opus 5.5	d72f761d83	seaweed-vector: work on its defaults, and without identities or credentials (#12323) ... * seaweed-vector: default --namespace to the gateway's Lance port The old default, http://localhost:8888/lance, is a path nothing serves: the Lance Namespace listens on its own port, 9101 by default on weed s3, weed server, weed filer and weed mini. With it, the filer answered the engine's catalog calls with a 500 and every CreateIndex failed with InternalFailure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: default --listen to a port no SeaweedFS server takes 127.0.0.1:8080 is the volume server's default port, so weed-vector next to weed server exited with "Address already in use (os error 98)". 8090 is not among the defaults of master, volume, filer, S3 and its Iceberg and Lance catalogs, IAM, WebDAV, SFTP, the message queue broker and agent, the admin server or weed mini, nor any of their gRPC ports (HTTP + 10000). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance: take any key while the S3 server runs without identities Without identities the S3 port accepts any signature and treats the caller as the admin. The Lance catalog still checked an x-api-key or Bearer token against identities that did not exist, so weed-vector started with --storage-option credentials on such a server got 401 on every catalog call. The catalog now skips both checks when S3 auth is off and leaves the caller to the S3 authenticator, which admits it as the admin, as it already did for a request with no key. With identities configured nothing changes: an unknown key is still 401. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance: tell clients to skip signing while S3 auth is off A client with no credentials of its own could not reach a table's data on a server without identities, though the S3 port takes unsigned requests there: lance fell back to the AWS default credential chain and failed with CredentialsNotLoaded. weed-vector without --storage-option failed every CreateIndex that way. The catalog now adds aws_skip_signature to the storage options it hands out while S3 auth is off, and leaves it out once identities exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: bind the listener before registering with the master The engine spawned its master registration and then bound --listen, so an engine whose port was taken could reach the master before it exited: on weed server with 18080 taken the master logged "vector node 127.0.0.1:18080 added" and removed it in the same millisecond, and the gateway noticed it. Bind first, and name the address in the error instead of "transport error". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-worker: keep an unsigned store off the AWS default chain A store the catalog says takes unsigned requests needs no credentials, but lance builds the AWS default credential chain for any store without them, and the chain probes instance metadata on every dataset open. With no credentials of its own, weed-vector spent a second on each query (1030-1066 ms against 32-46 ms with --storage-option ones). The merge of the operator's fallback and the vended options, which the engine and the Lance worker each did in their own copy, moves into table_options, and gives such a store empty credentials. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): serve an index with no identities and no engine credentials A second cluster runs weed s3 without -config and weed-vector without --storage-option, with no AWS credentials in either environment: weed s3 would turn them into an admin identity, and the engine would find them. On master its CreateIndex fails with InternalFailure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-01 23:52:30 -08:00
Chris LuandClaude Opus 5.5	ff7045a4e5	s3vectors: report the Euclidean distance, not its square (#12318) ... A euclidean index answered QueryVectors with Lance's L2 distance, which is the squared Euclidean distance: [3, 4] was 25 from the origin, not 5. AWS defines the metric as the straight-line distance between the vectors. The gateway now returns its square root, so the single engine and the enterprise coordinator, which both hand back the raw Lance value, report the same distance. Ordering is unchanged. Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-01 23:50:46 -08:00
Chris LuandClaude Opus 5.5	d6c0d728e4	s3vectors: resolve the caller like table buckets and honor bucket-scoped identity actions (#12319) ... * s3vectors: resolve the caller the way table buckets do The vector checks used the identity name as principal, while the catalog records a bucket's owner through getAccountID: the account id, or the issuer-scoped user id of a federated identity. An identity with an account or a federated identity did not own the vector bucket it created, a Deny naming its account did not bind it, and an Admin identity in the admin account did not get the admin bypass it has on table buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: honor identity actions scoped to a vector bucket An identity action such as s3vectors:QueryVectors:team-a granted nothing, because the vector policy context never named the bucket. Table buckets match <action>:<bucket> through the same context field, so pass the bucket the check is about. Creating a bucket stays unscoped, as for table buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-01 23:50:19 -08:00
Chris LuandClaude Opus 5.5	54777d7fb9	vector: build weed-vector without a system protoc (#12317) ... * vector: build without a system protoc lance's build scripts run $PROTOC and fail without one. Point it at the protoc that protoc-bin-vendored ships, at the version in Cargo.lock, so building inside seaweed-vector/ needs only the Rust toolchain. An explicit PROTOC still wins, and a registry without the crate falls back to PATH. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): build weed-vector inside its workspace Cargo reads seaweed-vector/.cargo/config.toml only when it runs there, so build from that directory to pick up the vendored protoc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * ci(vector): drop the PROTOC step the workspace config replaces Builds run inside seaweed-vector/, where the cargo config already points lance at the vendored protoc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): resolve CARGO_TARGET_DIR before changing directory A relative CARGO_TARGET_DIR given to make would point cargo, which now runs in seaweed-vector/, and the cp after it at different directories. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: pick the vendored protoc on every Linux host it ships for protoc-bin-vendored also carries ppc64le, s390x and x86_32 binaries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-01 18:56:09 -08:00
Chris LuandClaude Opus 5.5	80e0bf97dd	shell: dial volume servers on their registered gRPC port in volume.scrub, ec.scrub, volume.server.state (#12315) ... * shell: dial scrubbed volume servers on their registered gRPC port volume.scrub, ec.scrub and volume.server.state built each data node's address from its HTTP address alone, so a volume server whose gRPC port is not the HTTP port + 10000 was dialed on the wrong port. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: resolve -node volume servers to their registered gRPC port An explicit -node/-nodes list was dialed on HTTP port + 10000 too. Match each entry against the topology and take the data node's gRPC port; an address the master does not know is used as given. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep a gRPC port given in -node A node given as host:port.grpcPort was rewritten to the registered gRPC port; only a bare host:port takes it from the topology. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: dial -node volume servers when the topology is unavailable An explicit node list did not need the master before; keep using it as given if the volume list cannot be fetched. Selecting every node still requires the topology. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: bound the topology lookup for an explicit -node list With the master down, volumeServer.state -nodes waited on the master forever instead of reaching the given volume servers. Give the lookup a 30s deadline and fall back to the addresses as given. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: resolve volume server addresses from a topology-only listing Only the data nodes are needed to find each server's gRPC port, so ask the master for the topology without its volumes and ec shards. A large cluster no longer risks running past the -node lookup deadline and falling back to the HTTP port + 10000. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: skip the topology lookup when every -node names its gRPC port A host:port.grpcPort entry is dialed as given, so a list made only of those does not need the master and no longer waits on it during an outage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-01 18:50:48 -08:00
Chris LuandClaude Opus 5.5	b9bc42c217	s3vectors: vend write credentials to index data only to the vector engine and Lance worker (#12316) ... * security: share the filer admin token check CheckGrpcAdminPeer verifies an admin Bearer token inline. Move the check into IsFilerAdminToken so an HTTP endpoint can accept the same token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: recognize the cluster's own components by a signed token A request whose X-Seaweedfs-Cluster-Token header carries an admin token signed with jwt.filer_signing.key comes from a component that holds the cluster key: the vector engine or the Lance maintenance worker. Mark such a request as trusted. It still authenticates as an identity, which stays the principal for vended credentials. A dedicated header leaves the Authorization header to Lance OAuth clients. A gateway with no key trusts no one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: vend write credentials to a vector bucket's indexes only to trusted callers An index in a vector bucket's vector namespace is also a Lance table, so a caller granted s3tables:UpdateTable could describe it, or declare it again, and get credentials that write the index data without passing the vector bucket policy. In that namespace both now vend read-only credentials unless the caller is trusted, and a declare from anyone else no longer writes the dataset markers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance worker: prove the cluster key to the Lance namespace The gateway now vends write credentials to a vector bucket's indexes only to callers that send a token signed with jwt.filer_signing.key. The worker already takes that key for the admin stream; its namespace client now mints a token from it for every catalog request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: describe the index the gateway created and prove the cluster key CreateIndex declared a table the gateway had just created, which only worked because declare is idempotent. Describe it instead. The Lance namespace vends write credentials to a vector bucket's indexes only to callers that send a token signed with jwt.filer_signing.key, so take the key with --filer-signing-key, the same flag the Lance worker has. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance worker: keep the cluster key out of Debug output Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: answer a declare of an existing table with its own location CreateTable succeeds for a name that already exists without looking at the location requested, and declare kept using that location for the dataset markers and the vended credentials. Declaring an existing table with another table's location, such as a vector index, got credentials to that table's data. Use the location the catalog holds, which CreateTable now also reports for a new table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: sign cluster tokens with a key derived for the catalog A cluster token was a filer admin token signed with jwt.filer_signing.key and, by default, never expired. Sent to the Lance namespace on every request, often over plain HTTP, a captured one would also authorize the filer's IAM service. Sign it with a key derived from the cluster key for this one use, which the admin services do not accept, and mint it with a five minute lifetime. weed-vector drops its token lifetime flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: follow a reloaded jwt.filer_signing.key The Lance namespace copied the cluster key at startup. A SIGHUP reload refreshes the gateway's filer guard but not that copy, so after a key rotation the restarted vector engine and worker were no longer trusted. Read the key from the guard on every check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: keep a vector bucket's indexes read-only An index in a vector bucket is also a catalog table the Iceberg REST catalog loads, and LoadTable vended credentials that could write it to any caller granted s3tables:UpdateTable. A commit staged its metadata file in the index directory on the same grant. Apply the vector namespace rule to both, after the caller's own UpdateTable check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: confine a created table's location to its namespace before writing Stage-create skips the registration that confines a location to the table's namespace, yet vends write credentials for that location. A caller allowed to create a table in any namespace of a bucket could name another namespace's prefix, a vector index included, and get write credentials to it. The non-staged create also wrote its first metadata file there before the registration refused the location. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: a declare that finds an existing table answers like describe Creating a name that already exists is idempotent, so a caller allowed only to create tables could declare someone else's table and get write credentials to it, and clear its deregistered marker. Describe would have vended that caller read-only credentials. A declare now looks the name up first; when it only finds the table, writing needs UpdateTable, as it does for describe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-01 18:49:01 -08:00
Chris LuandClaude Opus 5.5	06b9d01423	s3vectors: match a filter value against the elements of an array (#12321) ... * vector: compare json_array_contains elements as typed JSON values Lance's json_array_contains compares each element's text with the value, or with the value wrapped in quotes, so 1 matches "1", true matches "true", and 1.0 misses a stored 1. The filter swap now replaces it like json_extract and json_exists: the value is parsed as JSON once per batch and compared with jsonb's typed equality, and only literal arguments are swapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: match a filter value against the elements of an array A filter never looked inside an array: {"tags": "space"} and {"tags": {"$in": ["space"]}} both missed "tags": ["space", "classic"]. AWS documents that $eq, and so a bare value, matches an array metadata value when any element equals the input. $eq, a bare value and each $in member now also match through json_array_contains; $ne and $nin remain their negations. Range operators, $exists and nested objects are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-01 18:44:49 -08:00