mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-19 02:15:02 +01:00
94de65b3b5d79b85e1112321199e81ccc1273cc1
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
94de65b3b5 |
s3api: answer a malformed Cache-Control on PutObject with InvalidRequest (#12733)
PutObject rejected a Cache-Control value it could not parse with InvalidDigest, whose message is about Content-MD5, even when the request sent no Content-MD5. CreateMultipartUpload already answers the same parse error with InvalidRequest; PutObject now does too. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ecbf7af1ca |
mount: compare permissions against local ids without mapping them again (#12726)
* mount: compare permissions against local ids without mapping them again Access, open and create (without -defaultPermissions) ran FilerToLocal on entries that maybeReadEntry and maybeLoadEntry already return in local ids. Under a uid/gid map where a local id is also a mapped filer id, the owner was mapped twice and the check refused the file's own owner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: cache a deferred create's placeholder in filer ids The meta cache stores filer uid/gid and maps them to local ids on every read, but createRegularFile cached the deferred or writeback placeholder with the caller's local ids. The second mapping in the permission checks used to cancel that out under a swapping id map; without it the owner of a placeholder was refused. Cache a mapped clone and keep the handle's entry in local ids. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test that a deferred create returns its entry in local ids Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
862913d199 |
volume, ec_balance: clean up failed EC shard copies; stop re-proposing empty-index volumes (#12718)
* volume: remove files a failed VolumeEcShardsCopy wrote The shard files were copied before the .ecx, so a source holding a 0-byte index failed the request only after its shards had landed, and nothing removed them: every retried move stranded another unmounted shard on a new disk. Copy the .ecx first, staged beside its target, so a bad index fails before any shard lands; rename it into place only after the shards and sidecars have copied, so a failed copy never truncates or replaces an .ecx the destination already has. On any error, remove exactly the files this call created; files already on the disk (an existing holder's shards, index or journal) are left alone. Go and Rust volume servers behave the same. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * ec_balance: skip EC volumes whose every holder reports an empty .ecx A shard copy refuses a 0-byte .ecx, so detection proposed the same failing move of such a volume every cycle. Leave volumes with no holder reporting a needle out of planning; their shards still count toward disk occupancy. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * volume: serialize VolumeEcShardsCopy per volume Overlapping copies of one vid shared the staged .ecx, and a failed copy's cleanup removed shards that a concurrent copy had already committed and mounted. Hold a per-vid EC lifecycle lock across the whole copy and its cleanup, named as in the enterprise build (Store.EcLifecycleLock / ec_lifecycle_lock). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: overwrite an existing .ecx in place when committing a copied index A mounted EcVolume reads its index through an open handle, so renaming the staged copy over it left the reader on the replaced inode: tombstones the source had folded into its .ecx never reached the mounted holder. An index already on the disk is now overwritten in place, as before the staging change; a fresh destination still gets the rename. Go and Rust. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: stage replacements of shards a VolumeEcShardsCopy finds on disk The shard transfer opened an existing destination shard with O_TRUNC and removed it on a stream or write error, so a failed retry deleted a shard this disk already held. A shard already present is now copied to <shard>.copying and renamed into place only after every other file has copied; a mounted copy is reopened so it serves the new file. Fresh shards still write in place and are removed on failure. Go and Rust. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: stage the .vif and .ecsum a VolumeEcShardsCopy brings until commit The sidecars landed as soon as they transferred, ahead of the deferred .ecx commit, so a failure after them left a new .vif beside the index the failed copy kept. They are now staged as .copying and renamed into place with the shards and index once every transfer has succeeded; a source without one still stages nothing. Go and Rust. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: run a Rust VolumeEcShardsCopy to completion when its caller goes away tonic drops a cancelled handler future at its current await, which released the EC lifecycle lock while file operations the copy had started (a cleanup remove_file on the blocking pool, a create or rename) could still land; a retry taking the lock could then lose its files to them. The copy now runs in a spawned task that owns the lock, as Go's handler runs to completion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: fail the sidecar copy tests with an unreadable source .ecsum A directory at the source path reads as "not found" on volume servers that skip non-regular EC files, which turned the failure into a no-op. A mode-000 file fails the transfer on every implementation; root and Windows, which can read it, skip the test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: copy each shard a VolumeEcShardsCopy lists once A shard listed twice and already held here was staged twice; the first commit rename consumed the staged file, so the second failed the request after the shard was replaced, and every retry failed the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: wait out an in-flight VolumeEcShardsCopy before mounting its shards A shard the copy had already landed could be mounted while the copy was still running; the copy's failure cleanup then removed the file but left the shard registered and announced. MountEcShards takes the per-volume lifecycle lock, as the enterprise build already does, and the Rust mount handler takes the same lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: fail a VolumeEcShardsCopy whose replaced mounted shard cannot be reopened The copy reported success while the mounted shard kept serving the replaced file. The error lets the caller retry the copy instead of treating the shard as delivered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: check which transfer fails in the EC copy cleanup tests An earlier failure, before any shard or sidecar landed, would leave the destination unchanged and pass every assertion without exercising the cleanup. The sidecar test also skips on Windows, where a mode-000 file stays readable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: fail the sidecar transfer in the EC copy test by injection A mode-000 file is still readable to root and on Windows, so the test skipped there. The source now returns an error for the .ecsum request only, and the test checks that it served .vif before that failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: take the EC lifecycle lock for a blanket full teardown A teardown acknowledged while a detached copy of the same volume was still waiting on its source let that copy recreate the index and shards afterwards. Go and Rust now wait for the copy, as the enterprise build does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: keep the request id on a Rust EC shard copy The copy now runs in a spawned task, outside the gRPC layer's request id scope, so its CopyFile calls to the source went out with a new id. Carry the caller's id into the task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: unmount a replaced EC shard whose reopen fails The failed copy left the mount reading the replaced file's inode, so the destination kept serving the old shard. Unmount it, which also reports the removal to the master, so a retry copies and mounts it again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: make the Rust EC copy tests fail for the right reason The sidecar test now serves a directory as the source .ecsum, which root cannot read either, so it no longer returns early there. The cancel test holds the store until the handler's join reports the cancellation, so a copy that finished before the abort can no longer pass it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: stage every shard a VolumeEcShardsCopy transfers A shard missing here was written in place with a cleanup guard armed before its transfer started, so a failed copy removed a shard that ReceiveFile wrote meanwhile. Every shard is now staged, and only one that did not exist when it is moved into place is removed on failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: take the EC lifecycle lock for VolumeEcShardsRebuild A rebuild folds the deletion journal into .ecx. Run during a copy of the same volume, the copy's index then replaced the folded tombstones. Go and Rust take the lock the enterprise build already takes here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: check the staged .vif bytes in the EC sidecar failure test A source without the optional .vif also returns success, so the served extension list could not prove the transfer. The test now reads the staged .vif when the .ecsum request fails and compares it with the source's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: unmount a failed reopen's shard on the copy's disk only UnmountEcShards walks every disk, so a reopen failure on the destination also unmounted a healthy copy of the same shard on a sibling disk. The copy now unloads the shard from its own disk and reports that disk's removal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: take the EC lifecycle lock for an ordinary EC shard delete Shards a copy has staged are invisible to the delete's last-shard check, so deleting the only canonical shard removed the index the copy was about to commit its shards against. Go and Rust deletes now wait for an in-flight copy of the same volume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: leave a mounted shard's copied file in place on copy failure A shard mounted on the copy's disk whose file was missing got the copied file reopened into its mount, then that file was unlinked by a later failure's cleanup while the mount kept serving it. Only an unmounted shard's new file is removed now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: skip repeated shard ids with a seen set Scanning the request prefix for each id made a request with many repeats quadratic while the copy held the volume's lifecycle lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: make the Go EC copy and lock tests observe their effects The fake source counts requests per extension, so the duplicate-id test proves .ec05 is fetched once, and returns fixture errors instead of failing the test from its handler goroutine. The staged .vif reaches the test through a channel. The mount and teardown lock tests seed real files and check that nothing happened while the lock was held and that the operation took effect after it was released. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: poll the handler directly in the Rust EC lifecycle lock tests A spawned handler that had not run when the 200 ms timer fired passed the blocking check without ever reaching the lock. The tests now poll the pinned handler future themselves, which guarantees it is waiting on the lock, and cover the ordinary shard delete as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: observe the lifecycle-lock waiter in EC copy lock tests The mount, teardown, shard-delete and rebuild tests slept 200 ms while holding the lock, so a handler scheduled only after the unlock passed without ever taking it. Wait until the handler's goroutine is blocked on the mutex instead; each test now fails within its timeout if the handler skips the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: check the shard is staged before the failure-path write in a copy test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: recover a missing EC index under the volume's lifecycle lock Missing-index recovery fetched the index and then ran a store-wide mount of every orphan shard, without the per-volume lifecycle lock. It could mount shards an in-flight VolumeEcShardsCopy of another volume had just renamed in, and that copy's failure cleanup then unlinked a mounted shard. Each recovered volume now holds its lifecycle lock across a re-check, the fetch and a mount restricted to that volume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): recover a missing EC index under the volume's lifecycle lock Mirror of the Go change: recover_missing_ec_indexes takes the volume's ec_lifecycle_lock, re-checks that the index is still missing, fetches it, and mounts only that volume's shards through mount_recovered_ec_shards_only, so an in-flight VolumeEcShardsCopy of another volume never has its renamed-in shards mounted from under it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: mount a volume whose index a copy installed during recovery Recovery re-checks under the lifecycle lock that the index is still missing. When a copy installed it in the meantime, the re-check skipped the mount too, and the volume's other shards stayed unmounted because later scans no longer list it. Skip only the fetch and still mount the volume's orphan shards under the lock, in Go and Rust. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: fetch a recovered EC index into the disk the locked re-check found The re-check under the lifecycle lock only kept a yes/no answer, so the fetch still wrote to the destination from the first scan, which may be a disk that has since gone away. Fetch into the record the re-check returned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: mount a copy-installed EC index even when no peer is known Recovery skipped the locked step when the master was unreachable, the lookup failed or no peer was listed. A copy that installed the index after discovery then left the volume's other shards unmounted, and later scans no longer list it. Always run the locked re-check, and give up for lack of peers only when the index is still missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: grow the stack buffer before scanning for a lock waiter A full buffer can cut off the handler goroutine's stack, so the test would report a handler that is waiting as one that never waited. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): fetch a recovered EC index into the disk the locked re-check found Mirror of the Go change: fetch into the record the re-check under the lifecycle lock returned instead of the destination from the first scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): mount a copy-installed EC index even when no peer is known Mirror of the Go change: a failed lookup or an empty peer list no longer skips the locked re-check, so a volume whose index a copy installed after discovery still gets its orphan shards mounted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): run missing-index recovery to completion when the mount RPC is cancelled Dropping the mount handler mid-recovery released the volume's lifecycle lock while the .ecj merge it had started on a blocking thread was still landing, so a teardown could remove the volume before that late write. Run recovery on its own task, as the copy handler does, so the lock is held until it finishes; Go's handler already runs it to completion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: skip other volumes before stat in a single-volume EC scan Recovery re-checks and mounts each volume with its own scans, so a request with many orphans repeated whole-store discovery once per volume. The index-owner and orphan-shard scans now take the same volume filter and drop other volumes' files by name, before the per-file stat, and the locked re-check scans only its own volume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): skip other volumes before stat in a single-volume EC scan Mirror of the Go change: the index-owner and orphan-shard scans take the volume filter and drop other volumes' files by name before the per-file stat, so recovery's per-volume re-check and mount no longer repeat whole-store discovery under the store lock for every orphan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): announce shards recovered after the mount RPC was cancelled Recovery now outlives a cancelled mount RPC, but the heartbeat nudge stayed in the handler, so shards it mounted after the caller left waited for the next full EC heartbeat. Nudge it from the recovery task, as Go's loads announce each recovered shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: match the collection as well as the volume id in the recovery re-check Discovery keys orphans by (collection, vid), but the locked re-check matched the vid only. Two collections with orphans of the same vid could both resolve to the first record, so the other's index was never fetched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): match the collection as well as the volume id in the recovery re-check Mirror of the Go change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8c6d09f498 |
s3api: answer NoSuchBucket from GetBucketRequestPayment for a missing bucket (#12716)
* s3api: answer NoSuchBucket from GetBucketRequestPayment for a missing bucket The handler wrote the static BucketOwner answer without resolving the bucket, so a bucket that does not exist looked like one that does. Check the bucket first, as PutBucketRequestPayment and the other static configuration getters already do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test GetBucketRequestPayment on existing and missing buckets Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
194e6ede9f |
seaweed-volume: reject FetchAndWriteNeedle reads that miss the requested size (#12717)
The Go volume server fails the RPC when the remote read returns a different length than req.Size; the Rust server cached whatever came back as a full-size chunk. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c4e1802e3e |
s3: route ';' queries as authorized, and report an empty s3:prefix to policies (#12702)
* s3: escape raw semicolons before the router matches a route registerRouter applied EscapeSemicolonsInQuery with router.Use, but mux runs router middleware only after it has picked the route. Queries() matchers split pairs on ';', so GET /bucket?uploads;x=1 reached the multipart listing handler while the handler and the IAM resolver read uploads%3Bx=1 and saw no uploads subresource. Wrap the S3 router with the escape so routing and authorization read the same query. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: report a listing without prefix as s3:prefix "" in policy conditions The advanced IAM path already sets s3:prefix to "" when a bucket listing names no prefix, as AWS does, but bucket policies and attached policies read conditions from ExtractConditionValuesFromRequest, which leaves the key out. The AWS home-directory policy that grants s3:prefix ["", "home/"] then denied the root listing, and a Deny keyed on the empty prefix never applied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test the served handler composition for the semicolon escape Move the escape and the Vectors split into s3Handler so a command test can send ';' queries through the handler the server actually serves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: keep the isVectorsRequest doc comment on its function Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
dbb2fdb93b |
pb: recover when a local gRPC socket file is deleted (#12694)
* pb: fall back to TCP when the local gRPC socket cannot be dialed A temp cleaner can delete a local gRPC socket file while its server keeps running. Every later dial to that server then failed with "no such file or directory", so the master could no longer reach its own volume server and volume growth stopped. Dial the server's TCP address when the socket dial fails, as the S3 gateway already does for the filer's HTTP socket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * pb: recreate a local gRPC socket whose file was removed Once its socket file was deleted, a server kept accepting on an unreachable listener until restart, so same-host RPCs stayed on the TCP fallback. Check the path every minute and, when the file is gone, close the old listener and bind the socket again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * pb: retry a failed local socket rebind on the next check A failed rebind ended the watcher, so the socket stayed missing after the cause cleared. Bind the replacement before closing the old listener: a failed bind keeps the old listener serving and is retried on the next check, and the old Serve still reports server shutdown. The old listener no longer unlinks the path on close, since the replacement now owns it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * pb: set the local socket unlink policy before serving the listener Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
95f46c8fff |
admin: manage filer webhook targets from the UI with live reload (#12690)
* webhook: share the default target settings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * webhook: managed webhook targets with a drainable queue Targets kept as JSON (name, endpoint, token, filters, retry and worker settings) build their own queue; Close waits for queued and in-flight deliveries before stopping it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: run webhook targets stored in /etc/seaweedfs when notification.webhook.managed is set The targets in /etc/seaweedfs/notification_webhooks.json start with the filer and are rebuilt from the store whenever the file changes on any filer, alongside notification.toml. The previous queue drains before it stops. Events for the file itself, which holds bearer tokens, are not sent to notification queues. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: manage filer webhook targets A Webhooks page under Management lists, adds, edits and deletes the targets kept in /etc/seaweedfs/notification_webhooks.json. Bearer tokens are write-only: the API reports whether one is set, and an empty token on edit keeps the stored one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: keep the webhook queue when the stored targets did not change Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: keep a replaced webhook queue open until its senders hand over Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: read chunk-backed webhook targets before editing them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep the stored webhook targets out of fs.meta.notify Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * webhook: reject negative managed target settings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * webhook: no signal handler on queues that reloads replace Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: swap the webhook queue without waiting behind blocked senders The read lock held across SendMessage made a reload wait for any sender blocked on a saturated target, and the waiting writer stalled every other filer write with it, which is when an operator disables that target. Close now refuses new sends and waits for the senders already inside SendMessage, and a send that hits the closing queue retries on its replacement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * webhook: release blocked test handlers before the server shuts down Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: always run the webhook targets managed in the admin UI Drop the notification.webhook.managed switch: every filer loads and live reloads the stored targets. Without stored targets no queue is created. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * scaffold: place the admin-managed webhook note above enabled Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * scaffold: separate the admin-managed webhook note Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * scaffold: note that admin-managed webhooks follow the notification queue Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: retry loading webhook targets after a failed read Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: retire the admin-managed webhook queue on shutdown Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6342876392 |
ci: run the Python 3.9 jobs on 3.10 (#12691)
* ci: run the Python 3.9 jobs on 3.10 ubuntu-latest now resolves to Ubuntu 26.04, which has no Python 3.9 build for actions/setup-python, so the Ceph S3 tests and the PyArrow Parquet 3.9 job fail during setup. 3.10 is the oldest version built for 26.04. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: list the Python versions the parquet CI runs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
40182a68bc |
admin: keep the dark theme from flashing light on reload (#12689)
* admin: apply the saved theme inline at the top of the page head The theme script was an external file, so a reload had to fetch or revalidate it before the root got its theme, and the page could paint with the light default in the meantime. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: set color-scheme on the root together with the theme Until the stylesheets load, the browser paints the page canvas from the root's color-scheme, which only the stylesheets set, so a dark theme showed a white canvas for the duration of every reload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b449301a4e |
ec: mount shards through the encode's .vif when a source volume's .vif shares the disk (#12676)
* ec: mount shards against the index .vif when the local one has no EC config A worker encode can send shards to a disk that still holds the source volume's .vif while the encode's .vif lands with the index on a sibling disk. The shard disk mounted through the source's config-free .vif, so it reported encode time 0 and fell back to the sidecar layout, and reads carrying the encode time were rejected as a different encode run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): mount shards against the index .vif when the local one has no EC config Mirrors the Go mount: a data-dir .vif that records no EC config yields to the idx dir's when that one records it, and load_vif_info_across_dirs reports the directory the .vif actually came from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: do not mirror EC sidecars onto a disk whose .vif has no EC config Startup mirroring copied .ecx/.ecj next to a source volume's leftover config-free .vif and kept that .vif, so after a restart the shard disk mounted self-contained with encode time 0. Leave such a disk on the cross-disk mount, which reads the owner's .vif. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): do not mirror EC sidecars onto a disk whose .vif has no EC config Mirrors the Go startup mirror: a shard disk holding a source volume's config-free .vif stays on the cross-disk mount, which reads the owner's .vif, instead of mounting self-contained with encode time 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: take the encode .vif from a sibling data dir when the index dir is shared With a separate -dir.idx the .ecx is shared by every disk, but the encode's .vif lands in the data dir of the disk that received it. A disk still holding a source volume's config-free .vif mounted through it at generation 0. NewEcVolume now accepts the server's data dirs and takes a sibling's EC .vif when the index is not local and every sibling EC .vif names the same run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: give EC mounts the data dir of every disk on the server NewStore records every data dir on each DiskLocation before loading, so the startup scan, cross-disk reconcile and MountEcShards all pass them to NewEcVolume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): take the encode .vif from a sibling data dir when the index dir is shared Mirrors the Go mount: EcVolume::new_with_vif_dirs takes a sibling's EC .vif when this disk's own records no EC config, the index is not local, and every sibling EC .vif names the same run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): give EC mounts the data dir of every disk on the server main sets Store.store_dirs before the first add_location, so a disk loaded before its siblings still sees them; each DiskLocation passes them to EcVolume::new_with_vif_dirs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: size startup EC validation by the sibling encode .vif With a shared -dir.idx, a disk that still holds the source .dat and its config-free .vif read the default data-shard count, judged custom-ratio shards an interrupted encode, and removed them along with the shared .ecx before the mount could take the sibling's .vif. The startup lookup now falls back to the same sibling .vif the mount uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): size startup EC validation by the sibling encode .vif Mirrors the Go change: validate_ec_volume falls back to the sibling data dir's encode .vif before judging shards beside a source .dat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: keep EC shards when sibling .vif files disagree on the encode SiblingEcConfigVif reports when a sibling .vif cannot be read or the siblings name different encode runs. The startup data-shard lookup then returns 0 instead of the default ratio, and validateEcVolume keeps the files rather than sizing custom-ratio shards by a ratio they do not use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): keep EC shards when sibling .vif files disagree on the encode Mirrors the Go change: sibling_ec_config_vif returns Err for an unreadable sibling .vif or disagreeing runs, ec_data_shards_from_vif returns 0 then, and validate_ec_volume keeps the files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: release the opened EC shard when its volume cannot be built Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
9e5357782e |
volume: keep reads over the download limit local when readMode is local (#12679)
* volume: keep reads over the download limit local when readMode is local With -readMode=local the server never sends a read for a volume it does not hold elsewhere, but a read over -concurrentDownloadLimitMB on a replicated volume was still redirected to another replica. Skip the replica fallback in local mode, so the read waits for a download slot and answers 429 when the wait times out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): keep reads over the download limit local when readMode is local Mirror the Go volume server: with readMode local, a read over the download limit waits for a slot instead of being redirected to a replica. proxy_or_redirect_to_target no longer needs the local-mode redirect switch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: check download capacity under the wait lock waitForDownloadSlot read the in-flight size outside the cond lock and the read handler broadcast without it, so a download finishing between the check and the wait was missed and the read waited out the timeout for a 429. Check capacity with util.WaitUntil under the lock and broadcast under it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): register for the download notification before checking capacity The wait loop created its Notified future only after the capacity check, so a body dropped in between had its notify_waiters() lost and the read waited out the deadline for a 429. Create the future first, then check, as Go now checks under the cond lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1e31c6dbf5 |
test(pjdfstest): check the sudo'd mount with ps, not kill -0 (#12673)
sudo switches its real uid to root while it starts, so kill -0 from the invoking user fails with EPERM even though the mount process is alive, and the wait loop reported "weed mount exited early" a few milliseconds after starting it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d5b4e1352e |
helm: drop the 4.53 vector metrics render guard (#12672)
* helm: pin the 4.53 tag in the vector metrics rejection test The 4.54 release made 4.54 the chart's default image, and that image serves weed-vector metrics, so rendering vector.metricsPort with the default tag no longer fails. Pin image.tag=4.53 for the rejection case and check that the default image renders the metrics port. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * helm: describe vector metrics support from 4.54 on in the chart README Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * helm: say the vector metrics render guard skips image overrides Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * helm: drop the 4.53 vector metrics render guard The chart now defaults to 4.54, whose weed-vector serves --metrics-port, so the check keyed on one old tag only blocked valid installs. The test asserts the default image renders the metrics flag and HTTP probes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * helm: describe vector metrics without the render guard in the chart README Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8e15110b4f |
filer: wake metadata subscribers only for directories they can match (#12669)
* log_buffer: let a subscriber skip wake-ups for unwanted partition keys Every append woke every registered subscriber. SetSubscriberWakeFilter lets a reader sleep through appends whose partition key it would filter out; it still reads them on its next poll. Flushes and keyless appends wake everyone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer_pb: decide subscription matches from an event's directories MetadataDirectoryMayMatchSubscription rules out events whose directory no child path of can match the subscription, and ScanMetadataEventDirectories reads an event's directory and new parent path without allocating. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: wake subscribers of a move's destination directory Appends are keyed by the source directory, so a filtered subscriber of the destination would otherwise only see the move on its next poll. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: wake metadata subscribers only for directories they can match Every metadata event woke and was scanned by every subscriber, so filer CPU grew with the number of subscribers watching other paths. Readers now register a directory wake filter, and the per-entry prefilter rules entries out from their directories before the skeleton scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test that moves into a subscribed prefix pass the prefilter Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f13b763552 |
docs: explain the 3-filer recommendation in the production setup notes (#12671)
* docs: explain the 3-filer recommendation in the production setup notes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: note that a filer load balancer must forward the gRPC port Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
df30c883fe |
s3api: keep a multipart upload's state on the object's owner filer (#12668)
* s3api: create a multipart upload's directory on the object's owner filer CreateMultipartUpload wrote .uploads/<id> to the gateway's own filer while the upload's completion commits on the object's write owner. With independent filer stores, a part or completion reaching another gateway before peer metadata replication caught up could not see the upload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: look up a multipart upload on the object's owner filer UploadPart, UploadPartCopy, ListParts, CompleteMultipartUpload, AbortMultipartUpload and the bucket policy SSE lookup read .uploads/<id> from the gateway's own filer, so a request on another gateway shortly after CreateMultipartUpload answered NoSuchUpload until replication caught up. Read it preferring the object's write owner, like object reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: write multipart parts to the upload's owner filer UploadPart and UploadPartCopy stored part entries on the gateway's own filer, so ListParts and CompleteMultipartUpload on another gateway could miss a part that replication had not delivered yet. Store them next to the upload directory on the object's write owner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: list and clean up a multipart upload on its owner filer ListParts, CompleteMultipartUpload and AbortMultipartUpload listed and removed part entries on the gateway's own filer. Do it where the upload and its parts now live, so completion sees every part and cleanup does not miss entries replication has not delivered yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test a multipart upload against a lagging gateway filer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: check the upload's owner before freeing a failed part's chunks A part write that the owner stored but answered with an error went to recovery with no object, so the owner was never among the filers checked. When the owner is outside the gateway's filer list, the others' not-found answers freed chunks the stored part still references. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: list an upload's parts on the prior owner after a ring change The upload record lookup already falls back to the key's prior owner while a ring change is in its cooling-off window, but the part listing read only the new owner. ListParts then reported no parts for an open upload. Fall back the same way when the new owner has nothing yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: find the upload record on the prior owner past a bare directory A part routed to a new owner that has not received the upload yet makes that filer create .uploads/<id> without the upload's key. The post-write check then read it as an aborted upload and removed the part. While the ring change is in its cooling-off window, a record on the prior owner now wins over such a directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: merge the prior owner's parts into an upload listing During a ring change's cooling-off window the new owner may have received only some of an upload's parts. Listing returned the new owner's parts alone once it had any, so ListParts dropped the rest and a completion naming them failed with InvalidPart. Both owners' pages are now merged, the owner's entry winning a shared name, and cut to the page limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: fail an upload listing the prior owner could not answer When the new owner has none of an upload's parts yet and the prior owner's listing fails, the parts may all be on the prior owner. That listing read as empty, so CompleteMultipartUpload answered NoSuchUpload for an upload that still exists; it now returns the error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: fail an upload lookup the prior owner could not answer During a ring change's cooling-off window an upload the new owner has not received yet, or holds only as a bare directory, read as NoSuchUpload when the prior owner's lookup failed. That failure is now returned, so the client sees a retryable error instead of an upload that is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5f7fefadae |
s3 lifecycle: write the suspended expiration marker before deleting the null version (#12664)
* s3 test: evaluate mutation conditions and serve DeleteEntry in the fake filer The fake ObjectTransaction now checks each mutation's own condition before applying it, as the filer does, and reports the failed mutation. ETag clauses read the stored ETag attribute first, like the filer. DeleteEntry removes the entry so unrouted delete paths can run against the fake. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: write the suspended expiration marker before deleting the null version A suspended-versioning expiration deleted the null version first and then wrote the delete marker. When the pointer read or the marker write failed in between, the retry found no null version, resolved as NOT_FOUND or VERSION_NOT_LATEST, and never wrote the marker, so an older version became the latest. The marker now goes first. The latest pointer outranks the null object, so the key reads as deleted once the marker lands, and a null delete that fails after it leaves a noncurrent null version rather than exposing an older one. The null delete is pinned to the entry read before the marker: on the owner it carries IF_ENTRY_EQUAL on that read, and off the ring the entry is re-read and compared by identity first. A null object written after the read is a newer write and is no longer deleted. When the null version is the one that passed the expiration checks, a missing or changed entry at that read resolves the request as STALE_IDENTITY before anything is written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: retry a failed routed null delete instead of deleting on the default filer A routed null delete that failed fell through to the unrouted path, which reads and deletes on the default filer. That filer may not have the owner's newer null object yet, so the fallback could delete it without the owner-side IF_ENTRY_EQUAL guard. The failure now returns RETRY_LATER; the marker is already written, so the key keeps reading as deleted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: recheck and delete an unrouted null version under the object write lock Without a ring owner the identity recheck and the null delete were separate calls, so a suspended PUT landing between them was deleted. Off the ring writers take the object write lock, so the recheck and the delete now run under it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 test: strip quotes from the stored ETag in the fake filer The filer normalizes both sides of an ETag comparison; the fake now strips quotes from the stored ETag attribute as well as from the candidate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: retry the null delete when the object lock lookup fails on a routed key Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: check the suspended marker against the validated version A suspended expiration that validated a named version wrote its marker without comparing the pointer, so a null PUT landing after the checks was hidden, and deleted too when it landed before the null read. The marker now compares the latest the way the enabled path does and resolves the request as VERSION_NOT_LATEST. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 test: count the marker's object write lock in the unrouted null delete Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: recheck the attempt deadline after reading the null version Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c09009a74f |
s3: accept the governance bypass header case-insensitively (#12667)
* s3: accept the governance bypass header case-insensitively botocore and the AWS CLI send x-amz-bypass-governance-retention: True, so the exact "true" match ignored the bypass and governance-locked deletes and retention changes failed with AccessDenied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test governance bypass with SDK-style header values Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c0509cf499 |
s3api: evaluate each policy condition against its own values (#12665)
* s3api: normalize a nil policy value to no values Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: evaluate each policy condition against its own values The process-wide normalized value cache keyed entries by fmt.Sprint of the value, so ["a", "b"] and ["a b"] (or [] and [""]) shared one entry and a condition in one policy could be compared against another policy's values. Normalizing is cheaper than building that key, so drop the cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test that look-alike condition values stay distinct Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
63806b7073 |
s3 lifecycle: bound each LifecycleDelete attempt with its own deadline (#12646)
* s3 lifecycle: bound each LifecycleDelete attempt with its own deadline Replay and walker deletes used the job context with no request deadline, so one stalled unary RPC could hold a shard for the whole job. Each attempt now runs under DeleteTimeout (default 1m), set after the limiter wait. A timeout while the job is still live is retried like a transport error and then halts the shard or walk without advancing; parent cancellation still stops both the attempt and the backoff. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: admin setting for the delete RPC timeout Adds delete_timeout_seconds to the S3 Lifecycle admin form and passes it to both the replay drain and the walker. 0 keeps the 60s default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: describe replay and walker timeouts separately in the help text Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: fail the pass when a replay delete exhausts its attempts A transport error or attempt timeout that outlived the retries halted the shard with a nil error, so the job reported success. processMatches now returns the error (the cursor still holds at the event), and runShard treats a drain error as a clean stop only when its own context ended, since an attempt timeout also wraps DeadlineExceeded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test that a replayed LifecycleDelete resolves after a lost reply Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: skip the expiration write once the attempt deadline passed A worker retries a LifecycleDelete whose attempt timed out, while the first attempt can still be running on the S3 server. An attempt that stalled before its marker write would then stack a second delete marker after the retry wrote one. The gRPC deadline reaches the handler context, so check it right before the expiration writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: recheck the attempt deadline after the suspended null delete The null-version delete can block on the owner filer past the attempt deadline; the worker may have retried and written a marker by then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: finish the suspended marker once the null version is deleted Skipping the marker after a successful null delete left an older version as the latest, and the retry then resolved as a no-op without hiding it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: let a routed delete marker commit only while its latest guard holds createDeleteMarkerIf passes an optional condition, evaluated on the .versions directory entry under the object's lock on the owner filer, and reports a refused commit as errLatestVersionChanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: write the expiration marker as a compare-and-set on the latest pointer A timed-out attempt can still be running on the S3 server while the worker's retry writes the marker. The marker commit is now conditioned on the .versions pointer as read just before it and skipped when the latest is already a delete marker, so whichever attempt commits second writes nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: resolve the marker as stale when another version became latest The marker's compare-and-set only covered changes after its own pointer read, so a version promoted after the expiration checks passed could be hidden by the marker. The enabled path now passes the validated version and a conflict retry that finds a different named latest resolves the request as VERSION_NOT_LATEST. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: resolve a null expiration as stale once a version takes the pointer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: read the marker's latest version the way getLatestObjectVersion does A pointer names a version only with its file name; an id alone falls back to the null object, and the null-is-latest signal makes the null version current. An expiration that validated a named version no longer hides a null written by a suspended PUT, and a current null under an id-only pointer now expires instead of resolving as stale. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: hold the object write lock for an unrouted expiration marker Off the ring createDeleteMarkerIf cannot carry the pointer guard, so a PUT committing between the pointer read and the marker write was hidden. The read and the write now run under the object write lock that unrouted writers take. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: share transient-error retries between walker and replay deletes (#12649) * s3 lifecycle: retry only transient LifecycleDelete failures dispatchWithRetry retried every RPC error except parent cancellation, so an authentication, permission or argument error spent the whole retry budget before surfacing. Retry only attempt timeouts and the failures util.IsTransientError recognizes (Unavailable, ResourceExhausted, network resets); everything else returns on the first attempt. A nil response is now an error instead of a nil dereference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: take a delete-rate token for every LifecycleDelete attempt processMatches waited on the shared limiter once per match, so the retries behind it ran outside the cluster-wide deletes/sec budget. The retry loop now waits on the limiter before each attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: give walker deletes the replay retry policy WalkerDispatcher sent each LifecycleDelete once, so one dropped connection aborted the bucket's daily walk while the same failure on the replay path was retried. The walker now goes through deleteWithRetry and shares its transient-error retries, per-attempt limiter wait and nil response handling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: retry a DeadlineExceeded the server sends back A server or proxy can end the RPC with DeadlineExceeded while the attempt's own deadline is still running. util.IsTransientError rejects that status, so the delete halted with retries left. It is a timed-out attempt like the client-side one, so retry it the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: say walker deletes get 3 attempts in the timeout help Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
504de9b562 |
security: refuse to start when per-target gRPC server allowlists are not applied (#12662)
* security: refuse to start when per-target server allowlists are not applied A component that dials every server with one option ignored <target>_allowed_server_* keys, so a config that only set those accepted any CA-signed server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * scaffold: say which components scope server allowlists per target Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * security: also catch per-target server allowlists set through WEED_ env vars Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d80e905fc7 |
security: optional dialed-host check for gRPC server certificates (#12663)
* security: optionally check the dialed host against gRPC server certificates verify_server_host under [grpc] or a [grpc.<component>] section switches the client to certificate-and-host verification. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * scaffold: document verify_server_host Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
799fd10342 |
s3 lifecycle: fail over between S3 fronts during a pass (#12647)
* s3 lifecycle: add a LifecycleDelete client that fails over between S3 fronts When the current front returns Unavailable, the client closes it, marks it failed for the rest of the pass, picks the next front that passes checkLifecycleReady and resends the same request there. Callers cut off by that switch closing their connection resend on the new front. When no front is left, every later call returns one error naming the fronts tried. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: serve the pass through the failover client Execute hands the replay drain and the walker one failover client built on the selected front, reports each switch as job progress, and closes the current connection when the pass ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: keep failing over until a front serves the resent delete The resend after a switch went straight to the new front, so a second lost front ended the call even with a healthy third one, and the walker has no retry of its own. Resend through the failover loop instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: do not recheck fronts skipped when the pass started The failover client started with no failed fronts, so losing the selected front checked the fronts selectS3Front had already skipped again, holding every delete for another readiness timeout per front. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a91b333658 |
s3 lifecycle: persist daily walk checkpoints and resume them safely (#12650)
* s3 lifecycle: walk checkpoints take each entry's resume position Versions of one object share a Path, so a checkpoint set from Path after the first version made a resume skip the rest. The ListFunc now gives each entry the start value that continues after it, and leaves it empty on all but an object's last version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: resume a bucket listing where the walk stopped The filer listing skipped entries with key <= start, but its order is not key order: each directory emits its .versions groups before its files, and a directory's children come before a sibling like dir.txt. Positions are now one (pass, name) frame per directory level, so a resume seeds each listing and skips exactly what was finished. A version group is one unit, and a skipped group still keeps its bare null version out of the second pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: content hash of a walk view Hashes every non-disabled action of a view and the versioning of their buckets, so a saved walk position can be tied to the walk it came from: another rule set, another walk mode, or a bucket that turned versioned gives another hash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: persist daily walk progress and resume it A walk that failed or was cut short started over on the next pass, listing again every bucket it had already finished. WalkBuckets now keeps the finished buckets and each unfinished bucket's position in a checkpoint file next to the shard cursors, one per shard set, and the next walk of the same view and shards resumes from it. The record is saved every 30s and when the walk stops, and cleared when a walk completes, so the next walk is full again and sees keys added behind the position; a clear that fails fails the walk rather than leave a record that would skip buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a stuck resumed walk starts over A bucket that fails at the same place on every pass kept the walk from completing, so the buckets it had finished were never walked again and their due objects waited forever. A resumed walk that fails again without finishing anything clears its checkpoint, and the next walk is full. A canceled walk keeps it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: do not resume a walk checkpoint left for a week A checkpoint is cleared only by a later walk of the same shard set, so a set that stopped walking, such as a one-off per-shard walk, left its record behind, and a walk of that set much later would skip buckets finished long before. Records now carry their save time and are not resumed after seven days; a walk that keeps retrying saves on every attempt and stays fresh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a failed walk without progress leaves its checkpoint alone A pass canceled while it read the checkpoint failed the read, then saved its own empty progress over the record it never loaded, so the next pass started every bucket again. A failed walk that finished nothing has nothing new to record and no longer writes; the stuck resume reset still clears. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: space out walk checkpoint saves after a failed one The save time advanced only on success, so once the filer refused a periodic save every following entry tried again and logged another warning. The interval now counts from the last attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a walk checkpoint save that errors may still be on disk A save whose reply was lost left onDisk false, so a walk that then completed skipped the clear and the next walk resumed from the stale record instead of walking everything. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: keep a stalled walk checkpoint for a day A resumed walk that failed again without progress dropped its record right away, so an outage spanning two attempts restarted the whole walk. Treat the walk as stuck only once its record has not advanced for a day. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a resumed walk counts its earlier buckets as walked Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: replay a resumed recovery walk from the walk that began it A recovery or cold-start pass rewinds replay to runNow - maxTTL. When the walk resumed a checkpoint, its finished buckets were listed in an earlier attempt, so an object there that was not yet due then and whose write is now older than maxTTL was neither expired by the walk nor replayed. The checkpoint now keeps the start of the walk that began it, WalkBuckets returns that time (its own start for a fresh walk), and runShard floors the recovery rewind and the cold-start replay at it. A record without a start is not resumed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a cold start that resumed a walk replays on the next pass The pass's shared subscription starts at runNow - maxTTL for a shard without a cursor, before its walk reports an earlier start, so flooring the cold-start drain at the walk start could not deliver the older events. Such a shard now saves its cursor at the walk start minus maxTTL and returns, like the recovery branch, and the next pass replays the window from that cursor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test that a cursor below the floor opens the next subscription Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test the cold-start replay across two Run passes A cold start whose walk resumed an earlier attempt sees an older event only on the next pass. The test runs Run twice against a filer that serves events past the requested floor, with a later in-window event that would otherwise move the cursor past the older one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
cdb01f8150 |
install.sh: add --enterprise (#12660)
* install.sh: offer the vector engine on linux only Releases build weed-vector for linux amd64/arm64 only, so --component vector on darwin or windows downloaded a missing asset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * install.sh: add --enterprise Installs from seaweedfs/artifactory: weed-enterprise-<os>_<arch>[_large_disk], weed-volume-enterprise[_large_disk], weed-worker-enterprise and weed-vector-enterprise. The latest tag skips vfs-* releases, and the Rust volume server is linux-only there, so --component all skips it elsewhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * Revert "install.sh: offer the vector engine on linux only" Releases after 4.53 publish weed-vector for darwin and windows too (rust-vector-platforms.yml). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * install.sh: Enterprise vector engine is linux-only Enterprise releases publish weed-vector-enterprise for linux amd64/arm64; open-source releases keep their darwin and windows engines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test: installer --enterprise asset names, platforms and latest tag Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * install.sh: Enterprise latest is the newest plain version tag artifactory also releases previews and a dev build, and a prerelease such as kvcache-filer-native-v0.1.0-preview.1 sits in the release list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * README: show passing --enterprise through bash -s -- Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * install.sh: the open-source weed for windows is a .zip too Releases publish windows_amd64[_large_disk].zip, so asking for a .tar.gz 404s on every Windows host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * install.sh: install the windows weed as weed.exe weed-volume and weed-vector keep their .exe on Windows; weed lost it, so cmd and PowerShell could not run the installed binary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * install.sh: suggest weed-volume only when it was installed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
fd4603d657 |
s3 lifecycle: count filer entries a walk reads but skips as progress (#12658)
* dailyrun: count entries read from the filer, skipped ones included A versioned bucket's pre-pass and a resumed walk read entries without emitting them, so a walk can read for hours with no visited entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: count listed entries as progress work Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
53377bcb5d |
mount: keep a handle dirty when a write lands during its flush (#12652)
A flush uploads the dirty pages without the handle lock, then commits the entry under it. A write in between grows the entry's size, but its pages are not in the committed chunks. Clearing dirtyMetadata then made the next flush skip the commit, leaving a file with the new size and no chunk for the write, which reads back as zeros. Count page writes on the handle and clear dirtyMetadata only when none landed since the flush started uploading. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
512f898600 |
s3 lifecycle: report work-based job progress during a pass (#12644)
* s3 lifecycle: report work-based job progress during a pass The daily pass sent progress only while starting, so a healthy scan that ran for many hours could reach the admin's 24-hour no-progress guard. Worker heartbeats prove the worker is alive, not that this job moves. While dailyrun.Run is in progress the handler now sends a progress update every 5 minutes, and only when the pass's counters (entries visited, replay events, RPC outcomes including errors) moved since the last delivered update. An idle tick sends nothing, so a stalled pass still expires. A failed send is retried on the next tick. The reporter is stopped and joined before the completion is sent, so no progress can follow the terminal result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: test that progress, not heartbeats, keeps an old job active Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: label progress phase and bucket as the last started Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
858f8cbad8 |
s3 lifecycle: check internal S3 authentication before starting a pass (#12645)
* s3 lifecycle: add a LifecycleReady RPC that only authorizes the caller LifecycleReady runs the same checkAdminAuth as LifecycleDelete and reads no bucket or object metadata, so a lifecycle worker can confirm that an S3 front accepts its internal credentials before it starts a pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: check S3 fronts with LifecycleReady before the pass Execute used to dial the first registered S3 front and start loading policies and walking buckets, so a missing or mismatched jwt.filer_signing.key surfaced only as repeated per-bucket delete failures. It now checks each front in order with LifecycleReady, using the credentials LifecycleDelete sends, under a 10s bound that also follows parent cancellation. The first front that passes serves the pass; fronts that fail are reported in the job's progress and the log. When none passes, the job fails before any listing with an error that names the front and says whether this worker has the key set, without key or token material. Fronts that predate LifecycleReady are checked with an empty LifecycleDelete, which they authorize and then reject before any lookup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c819e3fbea |
s3 lifecycle: per-pass run results and last full-pass status (#12643)
* s3 lifecycle: collect per-pass bucket results and work counters WalkBuckets kept only the first bucket error and logged the rest. A RunStats shared by the walker, the walker dispatcher and the replay drain now records every bucket's walk result (walked, failed, skipped on cancellation), entry and object visits, replay events, and LifecycleDelete outcomes under the dispatch_total outcome labels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: persist the last fully successful pass Shard cursors and walk timestamps advance on partial passes as well, so neither says when lifecycle last covered every policy. Keep that in its own record next to the cursors, plus a worker gauge for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: report each pass's result in the job record The job ended with a fixed success summary, or with only the first error, and malformed lifecycle policies were V(1) logs. The completion now carries the pass status (complete, partial, failed, canceled), bucket totals and failures, invalid policies, entry, object and event counts, LifecycleDelete outcomes, the last phase and bucket, the elapsed time and the last full pass, on failed passes as well. Only a complete pass advances the full-pass record; a malformed policy makes the pass partial. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep a failed job's result in its record Completions with Success=false dropped Result.Summary and Result.OutputValues, so a handler could not report what a failed run had done, which buckets failed, or how far it got. Keep them; the error message still drives the job's message and run record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: chart the time since the last full lifecycle pass Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a replay halted on an unresolved delete is a partial pass BLOCKED, RETRY_LATER and exhausted transport retries stop a shard's replay without an error, leaving its due events for a later pass. Count those shards and report the pass as partial, so it no longer advances the full-pass record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a pass with throttled walks is not a full pass WalkerInterval can skip the walk for walk-only rules (ExpirationDate, ExpiredDeleteMarker, NewerNoncurrent) with no error. Count those shards and keep the full-pass record for passes that evaluated every rule; the job reports full_pass and walks_deferred. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: keep the saved full-pass time when the save fails Publishing a timestamp that never reached the filer let the gauge move back on the next job's reload. Keep the previous value and report the save error in the job result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: drop a failed attempt's summary when the retry reports none Retries reuse the job record, and failed completions now keep their result summary, so a retry that completes without a Result showed the failed attempt's summary, and a successful one used it as its message. Clear the summary when a completion carries no Result, as the output values already are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6ad4b07598 |
mount: scope and order the final-release writeback fence (#12651)
* mount: fence a dropped handle only when it holds unpublished content dirtyMetadata is also set by chmod, xattr and fallocate, so a failed metadata-only close refused every later write of the inode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: fence the inode before its last handle leaves the handle map A reopen between the map removal and the fence could publish at the stale kernel EOF. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: fence a failed metadata flush only when the handle holds content A refused synchronous flush or an abandoned writeback metadata flush of a handle with only chmod, xattr or fallocate changes fenced the inode, so later writes were refused until FORGET. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: count a size change on an open handle as unpublished content A truncate or extend through an open handle changes the entry's size and chunks, but it set only dirtyMetadata. A failed close then skipped the final-release fence, so a reopen before FORGET could publish against the untruncated entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: mark copied content dirty before each page of a streaming copy A copy that failed or was cancelled after earlier pages were accepted left dirtyContent false, so the final-release fence skipped that content. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: mark copied content dirty only after a page is accepted A copy whose first page is rejected leaves the destination unchanged and the kernel EOF where it was, so fencing the inode on release only blocked later writes until FORGET. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: skip the server-side copy while the destination holds unpublished content A whole-file server copy clears dirtyContent when it commits, so a page a cancelled streaming copy left behind would lose its release fence. Fall back to the streaming copy instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: skip the server-side copy while the source holds unpublished content The filer copies the stored source, so pages still pending on the source handle would be missing from the destination. The streaming copy reads them through the dirty pages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: check the publication fence before a server-side whole-file copy The filer copy publishes the destination without going through the flush-time guard, so a fenced writeback inode could still take new content before FORGET. Fall back to the streaming copy, whose flush is guarded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0a38738076 |
filer: keep the listing snapshot below later and unfinished event stamps (#12642)
filer: take the listing snapshot as a claim below later and unfinished stamps A ListEntries snapshot tells the client that every event at or below it is already in the listing, so the mount drops those events from a directory build and fences them with the directory floor. The snapshot was a raw time.Now(), which an event stamp can tie on a coarse clock: on Windows a mkdir right after a listing of its parent was stamped with the snapshot's own timestamp, the mount took it as listed, and the new directory stayed missing from a parent marked cached. A create stamped before the snapshot but not yet in the store was dropped the same way. Take the snapshot through the same in-flight claim the idle heartbeat uses: it stays below every unfinished stamp, and every later stamp lands above it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e0890391ac |
mount: keep a refused content create refused when the stored-entry read stalls (#12629)
The read of the stored entry ran in the create's filer callback, so a read that hit the call deadline turned the refusal into a filer timeout, and both retry layers resent the refused create. Read it in its own call, bounded to one filer call. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6eb2fb4842 |
filer: stop full-chunk reads at the chunk size after a mount truncate (#12632)
* util/http: stop a full-chunk read at the chunk's recorded size weed mount shrinks a chunk's Size in place on ftruncate but leaves its needle untouched. A view covering the whole shortened chunk is fetched without a Range header, so the reader returned every stored byte. The filer GET then wrote past the declared Content-Length and sent an empty body, and the stream reader served the stale tail. ReadUrlAsStream and the cipher path now deliver at most size bytes; size 0 still reads the whole needle for manifest and log chunks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: limit a plain chunk read to the requested size A full-chunk read sends no Range header, so a chunk shortened by a mount truncate streamed its whole needle into the S3 response. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * util/http: still report a failure past the chunk size The volume server aborts a full-needle response when the needle fails its checksum, and only at the end. Stopping at the chunk size missed that, so a damaged copy read as healthy. Read the rest of the body and return its error so the caller tries another copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
63c99183d1 |
s3vectors: name weed-vector processes servers (#12627)
* s3vectors: call the registered weed-vector processes servers The gateway's discovery named them workers, the name of the admin plugin workers. Also drops two comments left without a function. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: build the Enterprise engine as weed-vector In an Enterprise checkout, make ./weed-vector builds the distributed engine, which also serves VectorService, instead of a separate weed-vector-worker. An open-source checkout builds weed-vector-server as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: distributed mode starts two more weed-vector engines It started weed-vector-worker, which no target builds now, and waited for a gateway log line that no longer exists. The Enterprise weed-vector's engines all join the placement controller. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a8436a621a |
admin: vector servers on the dashboard (#12606)
* admin: vector servers on the dashboard A Vector servers count in the Cluster card and a Vector Servers table after S3 Servers, from the engines the dashboard already discovers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * admin: drop the dashboard's vector server kind Every vector server is a weed-vector engine, so the column said the same thing on every row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
58d25998d0 |
admin: Cluster → Vector Servers page (#12603)
Lists the weed-vector engines registered with the master, with version and health, next to the other server lists. Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4fc41872e3 |
mount: keep an open file publishing when a peer unlinks one of its hard link names (#12613)
* mount: record a hard link's looked-up names on its inode Lookup mapped a second name of a hard-linked file to the existing inode without adding it to the inode's names, so GetAllPaths missed it and the final forget left it mapped to a released inode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep an open file publishing when a peer unlinks one of its names A delete event from another client marked the inode's open handle deleted even when other names still held the hard-linked file, so every later fsync reported success without saving metadata. The delete event's entry carries the link count the filer decided on, so an alias delete now drops only that name, keeps the handle on a name this mount knows, and has the next flush read the survivor's stored entry before publishing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test peer unlinks of a hard link name held open Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: leave an unlinked file out of the periodic metadata flush flushFileMetadata did not check the handle's delete mark. A file unlinked while open before its deferred create reached the filer was created again by the next periodic flush, which expects no entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: read a pending link change before a periodic metadata flush Only flushMetadataToFiler applied linkRefresh. After a peer unlinked one name, the periodic flush kept publishing over the old link count, so the filer refused every one until a regular flush ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test the periodic flush after a peer unlinks a hard link name Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
08f9dcb6de |
admin: vector buckets link maintenance jobs to the Lance lane (#12633)
The Vector Engines and Maintenance card linked to the general plugin monitoring page; the vector index jobs run in the Lance lane. Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8a756af348 |
vector: key shared leases by manifest commit time too (#12626)
On Windows Lance commits local tables by rename and leaves the manifest ETag empty, so a table dropped and written again at its location reused the old table's lease for the same version number. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
68b6a2ed08 |
mount: model the refused async flush as a replaced file in recovery tests (#12628)
A peer's chunk append now merges into the flush, so these tests no longer saw the refusal they exercise. A replaced file is still refused. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
45cd6033a0 |
s3: make conditional DeleteObject atomic on versioned buckets (#12624)
* s3: re-check a versioned DeleteObject If-Match under the object lock A versioned or suspended DeleteObject without a versionId routed to the owner filer after checking If-Match outside any lock, so concurrent conditional deletes could all pass the check and each add a delete marker. Keep conditional deletes on the lock path, which re-checks the precondition under the object write lock, as conditional versioned PUTs already do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: guard a conditional version delete on the owner filer DeleteObject with a versionId checked If-Match before routing the delete to the owner filer, so concurrent conditional deletes of one version could all report success. Carry the If-Match clause into the owner's transaction as a guard on the version (the null entry for versionId=null), evaluated under the object's path lock before anything applies, and map that guard failing to 412 instead of the object-lock 403. An If-Match the filer cannot evaluate stays on the lock path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: carry an If-Match ETag list into the routed delete A DeleteObject whose If-Match listed several ETags fell back to the gateway's object write lock, while a single-ETag delete of the same key ran under the owner filer's path lock. The two locks do not exclude each other, so a list delete and a single-ETag delete could both report success. Send every candidate to the filer, whose strong comparison matches checkDeleteIfMatch, so all conditional deletes with an owner are decided under the owner's lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: fail a conditional null delete on an owner transport error The owner may have committed the delete before the error, and the gateway lock path it fell back to does not exclude routed deletes, so a second If-Match delete could pass there too. Answer 500 as the specific-version path does; unconditional deletes still fall back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: fail a conditional null delete on any owner error but the nested refusal Only the owner's non-empty-folder refusal is known to apply nothing. After any other owner error the delete may have been partly applied, and a lock-path re-check does not exclude a concurrent routed delete of the same version, so a conditional null-version delete answers 500 instead of falling back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: strip a nested conditional null delete on the owner A conditional versionId=null delete that the owner refuses because other keys are nested under the entry used to fall back to the gateway lock and re-check If-Match there. That lock does not exclude routed deletes: once the last nested key is removed, a concurrent routed delete can pass the same check on the owner, and the fallback then treats the missing entry as deleted, so both requests answer 204. Demote the entry on the owner instead, in a transaction guarded by the If-Match clauses and by the entry it was read as, with the object-lock guards on the request. A missing or already demoted entry answers 412. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: add an IF_OBJECT_EXISTS write condition IF_EXISTS also holds for a plain directory, which is what is left of a key once the object on it is stripped while other keys stay nested under it. IF_OBJECT_EXISTS holds only for an entry that carries an object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: require an object for a conditional null delete with If-Match: * Stripping the null version off an entry other keys are nested under leaves a plain directory. Once the last nested key is gone, a routed delete with If-Match: * still found an entry there and removed the empty directory, so a second conditional delete could also answer 204. The owner now checks that the entry carries an object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: settle a wildcard null delete on the entry when the filer predates IF_OBJECT_EXISTS A filer older than IF_OBJECT_EXISTS fails it closed, so with the gateway upgraded first every If-Match: * delete of a null version answered 412. When the narrowed guard fails, the gateway now reads the entry on the owner: absent or a plain directory is a 412, otherwise the file is deleted, or the directory stripped, under IF_EXISTS and IF_ENTRY_EQUAL on the entry it read. The demote keeps the caller's If-Match clauses, since the entry guard already requires an object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
813ac74d79 |
s3tables: keep legacy-subject owners and policy principals for federated STS sessions (#12615)
* s3tables: re-anchor legacy owners for federated STS sessions A web-identity role session was keyed on its bare OIDC subject until owners became issuer-scoped. Its sessions now carry the scoped aws:userid but no asserted issuers, so the legacy subject check never applied and the session lost the table buckets and namespaces it had created. Read the configured issuer set for such sessions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: keep token claims from asserting trusted issuers on STS sessions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: give the catalog's table handler the local accounts and issuer set Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: test a federated session on a bucket its legacy subject owns Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sts: drop a token-supplied aws:userid when a session has no parent user Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: take a session's aws:userid from its signed parent user Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test that session claims keep only the server-asserted aws:userid Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: match legacy subject patterns in table bucket policy principals A policy written before issuer scoping can name role sessions by a pattern over their bare OIDC subjects. Match such a pattern against the caller's subject claim once that subject resolves to the session's scoped principal under the single configured issuer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: test a federated session against a legacy subject pattern principal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d685709832 |
s3api: refuse writes to a stored version's physical path (#12622)
* s3api: refuse a PUT to a stored version's physical path In a Suspended bucket a PUT to <key>.versions/v_<id> replaced the stored version's body in place. Look the entry up when the key has the version file shape and deny the write when it is a version file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: refuse copy, multipart, POST and rename writes to a version file Apply the same version file check to the CopyObject, CreateMultipartUpload, CompleteMultipartUpload, POST Object and RenameObject destinations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: read a version file write target on its logical owner The guard resolved the key as an object, which on a ring reads the physical path's owner and, in a versioned bucket, may return a nested version under <key>.versions/v_<id>.versions instead of the stored version. Look up the physical entry directly, routed by the logical key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: confirm a version file miss on a skipped owner When the owner was recently unreachable the routed lookup answers from a replica, whose miss does not show the version is absent. Ask the owner directly in that case and refuse the write when it cannot answer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: confirm every routed version file miss on the owner Whether the routed lookup skipped the owner can change between the lookup and a second unreachable check, letting a replica's miss through. Ask the owner after any miss instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
737ef68a11 |
mount: keep publishing an open file through its surviving hard link (#12609)
* mount: keep flushing an open file whose other hard link survives an unlink Unlink marks the inode's open handle deleted so no flush recreates the removed name. When the entry was one of several hard links the inode lives on under another name, yet the mark stayed and every later fsync and close returned OK without publishing metadata. Lift the unlink's own mark once the name is gone and the inode still has a path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: carry a link's stored entry into the file's open handle Link rewrites the source entry with the new hard link id, count and times, but an open handle kept its old base and entry. Its next conditional content flush was refused (EIO), and an unconditional one would have dropped the hard link id. Move the handle onto the stored entry when only the link fields and times changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: carry an alias unlink's stored entry into the open handle Removing one of several hard links lowers the stored link count and stamps a new ctime, and the handle may still hold the removed name. Look up the entry at the name the handle flushes through and adopt it, so the next content flush matches the stored entry, writes the right count, and attribute saves address the surviving name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test an open handle across link and alias unlink Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: read an alias unlink's lost entry again at the next flush When the read of the surviving entry right after an alias unlink fails, the handle kept a base with the old link count and ctime, and every later flush was refused even after the filer recovered. Leave the hard link id on the handle and have the next flush read the stored entry and adopt it before publishing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: publish a queued close before unlinking a name another link keeps With writeback caching, close only queues the flush. Unlink marked the handle deleted before waiting for that flush, so the flush skipped its metadata and the surviving link kept the content from before the close. When another link keeps the file, wait for the queued flush first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: let a queued close reach the filer before linking the file With writeback caching a new file is created on the filer by its queued close, and a listed directory serves the source from the cache, so a link right after close updated a name the filer did not have yet and failed with EIO. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: skip a link change no newer than the handle's entry A link change carried into the handle could replace a base already moved past it by a later acknowledgment. Use the same clock-domain check as installAckedEntry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep a link change's ctime in the dirty entry The flush does not restamp times, so a dirty entry carrying an earlier write's ctime moved the filer's ctime back from the link or unlink. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: flush through the file's current name when the resolved one is gone doFlush resolves the name before uploading the data. An unlink of that alias, or a rename, in the meantime sent the publish to a name the file no longer has. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep the pending link read when its name is already gone A flush could read the removed alias name, find nothing, and drop the pending read of the surviving link, leaving the stale base in place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test link and alias unlink across queued closes, flush races, versions and ctime Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: pick a flush's name again once its locks are held An unlink or rename that lands between the name pick and the locks would otherwise send the flush to the removed name, where the create guard fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test a flush waiting on its lock through an alias unlink Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: check the flush target after every lock acquisition A flush whose name kept changing published to the name chosen before its last lock wait. Check after each acquisition and return a retryable error once the attempts run out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test a flush whose name changes during every lock wait Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c248ab7865 |
credential: keep IAM changes made during the legacy-file migration (#12614)
* credential: migrate the legacy IAM file without overwriting newer entries The migration wrote every identity, service account and group from the snapshot the load read, over whatever the per-entry files held by then, so a revoke or delete landing in between was undone for good. It now migrates only the legacy file's own entries, creates each per-entry file only when none exists, and waits for the next load when the legacy file changed after the snapshot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * credential: test changes landing between the IAM load and its migration Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * credential: keep a per-entry file deleted after the IAM load deleted An entry the load read from its own file is already migrated. When that file is removed before the migration runs, the legacy copy must not recreate it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * credential: test a deleted per-entry file the legacy file also holds Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
12cb707e58 |
seaweed-volume: copy master.proto with metrics_port (#12605)
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
aa901b7ac7 |
s3tables, iceberg: create missing directories exclusively so a concurrent create keeps its record (#12600)
* s3tables: create a missing directory exclusively in ensureDirectory ensureDirectory looked the path up and, when it was missing, created it with a plain CreateEntry. A table create that lost the lookup race then replaced the directory another create had made, claimed and confirmed since, erasing that table's record, so a third create could claim the name and win as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: create missing metadata parent directories exclusively saveMetadataBlob created each missing directory on the way to the metadata directory with a plain CreateEntry after a lookup. A table directory another create had made and written its record onto since the lookup was replaced by a bare directory, erasing that record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: create missing stage-create marker directories exclusively The marker path starts at the table bucket directory, which carries the bucket record; a plain create after a missed lookup replaces a directory another writer made in between, with whatever it carries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e0beb59ef0 |
topology: treat a nil ttl as the empty ttl when looking for duplicate volume ids (#12601)
Volume layouts are keyed by ttl.String(), so a nil ttl and the empty ttl share one layout. ttlEqual told them apart, so a replica reported with a nil ttl into a layout created with the empty ttl matched its own layout as a duplicate, and every copy was quarantined. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
fe14880f4d |
shell: tier download replicates the downloaded volume before deleting the remote object (#12599)
* shell: replicate the downloaded volume before deleting its remote object volume.tier.download downloaded every replica from the remote tier and let the last one in the master's location list delete the shared object, so a replica missing from that list kept reading a deleted object. Download onto one replica, copy it over every other remote-backed replica, re-read the topology so replicas that show up meanwhile are replaced too, and only then delete the last remote-backed replica together with the object before copying the local volume onto it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: test tier download replicates before deleting the remote object Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: copy the downloaded volume onto each replica's own disk type Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: test tier download keeps each replica's disk type Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: name the hdd disk type when copying a downloaded volume VolumeCopy treats an empty disk type as the source's, so an HDD replica copied from an SSD source was placed on SSD or failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: block deletes on the tier download copy source A readonly-can-delete source took deletes after VolumeCopy read its index, so the replaced replica lost them and served the files again. Hard-mark the source readonly while replicas are replaced, then restore the writable mark or reload the persisted state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep the remote object while replicas are unlisted Tier download deleted the shared object once the listed replicas were local, even when the replica placement expected more copies than the topology listed. Keep it in that case so a rerun can finish later. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: fail tier download when no replica is listed Also point the final-replica delete error at volume.fix.replication. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep the remote object when a server lists the volume twice A stale twin on another disk of a replaced server still references the object but is filtered out with its server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: restore the tier download source when its readonly mark fails VolumeMarkReadonly blocks local writes and deletes before its second master notification, so a failed notification returns an error with the source already frozen. Restore it whenever the status was read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: report a failed tier download source restore A restore that leaves the source unmounted or frozen was only printed, so the command still returned success. Join the restore error into the result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: wrap the tier download restore error Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
94c3869057 |
volume_move: keep the live-move source frozen across volume reloads (#12597)
* volume_move: persist the move's source freeze across reloads A live move froze the source in memory only, so an unmount/mount or a volume server restart mid-move reopened it for writes that the move then deletes with the source. Persist the freeze when the move froze a writable source, and reopen the target right after the copy since the copied .vif carries the freeze. An already-readonly source keeps an in-memory mark, and a failed move still restores the source writable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: live move keeps the source frozen across a reload, on Go and Rust volume servers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: require 409 from the reloaded frozen source Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume_move: reopen an interrupted move's source only without a target copy Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
652648d9c4 |
mount: merge concurrent content changes from another mount instead of failing or dropping them (#12596)
* mount: merge a refused content flush onto the stored entry A content flush conditioned on the handle's base failed with EIO whenever a peer had changed the file since, so two mounts appending to one file could not both close it cleanly. On a refused condition, reread the entry and replay this handle's changes since its base onto it: the chunks it added or dropped, and the attributes, extended keys and inline content it changed. A local truncation keeps its size and clips the peer's chunks to it; otherwise the larger size wins. The merged entry is published against the stored one, up to three times, and installed in the handle so its next flush keeps the peer's chunks. A replaced or deleted file is still refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: merge a peer's content change into a dirty handle A subscription event carrying a peer's content change replaced a dirty handle's entry and discarded its unflushed pages, so the handle's writes were lost and its close still returned OK. Fold the peer's entry into the handle with the same merge a refused flush uses, keep the dirty pages, and take the peer's entry as the new base. A replaced file still invalidates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: stop a flush merge once a filer accepts the write withFilerClient can call the callback on several filers. A refusal from one left its stored entry behind, so a later filer's success went on to another merge round. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: refuse a merge that puts inline content next to chunks A read inside the inline bytes returns them without looking at chunks, so one side's inline content merged with the other side's chunks hid data after a successful flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: refuse to merge a whole-file rewrite over a peer's chunk change maybeMergeChunks re-uploads the handle's whole view as unstamped chunks. Merged after a peer's change, they tie with a peer's own rewrite or bring back a range the peer truncated, so that flush fails as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: clip peer chunks at the lowest truncate, not the final size A handle that truncated and then wrote past the old end ends up larger than its base, so the merge did not see the truncate and kept the peer's chunks inside the range the truncate cleared. SetAttr now records the lowest size truncated to over the current base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep a peer's truncate when the handle left the size alone The merge took the larger of the handle's and the stored size, so a peer truncate came back as a zero-filled tail when the handle only overwrote bytes inside it. The stored size now stands unless the handle changed its size or wrote past it, and rebasing keeps only the growth writes made after the flush was taken. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: retry a read view whose merged manifests did not resolve A failed SetChunks leaves the chunk group answering every read with the resolution error, and nothing re-ran it while the entry stayed the same. The handle now marks such a view (on rebase and on SetEntry) and the next read or lseek resolves the entry's chunks again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep a dirty handle's writes when a peer change cannot merge The new merge refusals sent a peer event on the same file down the old path, which discarded the dirty pages and adopted the peer's entry, so close returned OK without the writes. The handle now keeps its writes and base, and the flush meets the conflict and fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep an unpublished truncate across a peer's base change The truncate mark was tied to the base pointer, so a peer event that moved a dirty handle's base (a merged content change or a metadata-only change) dropped it, and a later merge on a regrown file kept peer chunks inside the truncated range. Those base moves now carry the mark; a flush ack or a discarded handle state still clears it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: wait for the flush before merging a shrink over buffered writes A write inside the file leaves its bytes in dirty pages without moving the size. Merging a peer's truncate then adopted the smaller size, and the chunk uploaded later sat past it, unreadable through the handle. The event path and the periodic metadata flush now refuse a merge that shrinks the handle; the close flush uploads the pages first and merges then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: tell a shortened base chunk from a whole-file rewrite A truncate shortens a base chunk in place, so it gets a new reference key, and an old chunk can carry no timestamp. The rewrite refusal now applies only to unstamped chunks whose file id is new to the base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: check the creation time when the base has no inode An entry without an inode or hard link id matched any replacement at the same path, so a stale handle could merge into a different file. Without an inode the merge now needs the same creation time. The filer keeps it across updates; only a synthesized base has neither and still merges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep a dirty handle's writes when a peer replaces the file A peer entry with another inode at the handle's path went down the fallback that destroys the dirty pages and installs the replacement, so close returned OK without the writes. The handle now keeps its writes and base like it does for an unmergeable change, and its flush refuses to update the replacement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: build the content flush tests on freebsd The invalidation tests, which every platform compiles, now use the content flush helpers, so the freebsd cross-compile vet failed with undefined: flushConditionEntry. Nothing in the file is freebsd-specific. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep a dirty handle's base when a same-content file replaces it A replacement with the base's chunks but another inode counted as a metadata-only change, so the dirty handle adopted it as its base and the next conditional flush overwrote the new file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: bring a peer's metadata change into a dirty handle A metadata-only event moved a dirty handle's base but left its live entry, so the next flush read the peer's chmod as a local edit and published the old mode back. Replay the handle's own changes onto the peer's entry instead, as for a content change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: wait for the flush before merging peer inline content over buffered writes A buffered write is not a chunk yet, so a merge could install a peer's inline content that later reads return ahead of the write's chunk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: track a whole-file chunk rewrite on the handle The merge told a rewrite by its unstamped chunks, which a manifest fold hides: manifests are unstamped either way. Record the rewrite against the base it was made from, carried across peer base changes like a pending truncate, and refuse the merge from that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: refuse a merge that shortens a manifest at the truncate Clipping a manifest reference leaves its members whole, so peer bytes past the handle's truncate could reappear once the file grew again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: flush a dirty handle after a peer replaces its file The replacement test checked only the retained handle state. It now flushes against a filer holding the replacement, including one with the base's content, and checks the flush fails and leaves it unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test an append fsync after a peer's attribute-only change A second mount's acknowledged chmod, chown, utimes or setxattr leaves the content alone, so the writer's conditional content flush must still publish its append and keep that change. On master the refused flush surfaces as EIO from Fsync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a576398662 |
weed-vector: advertise its metrics port when registering (#12604)
* master.proto: KeepConnectedRequest carries the node's metrics port Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: advertise --metricsPort when registering with the master Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * master.proto: number metrics_port 9 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
baaa60b24a |
admin: an arrow icon for vector buckets (#12602)
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
917cf534db |
s3api: sign every operation subresource in signature V2 (#12575)
* s3api: sign every operation subresource in signature V2 The V2 canonical resource only kept the subresources inherited from the original allowlist, so tagging, retention, legal-hold, cors, encryption, object-lock and the other keys the router dispatches on were left out of the signature. A V2 signature issued for one request therefore also verified that request with one of those keys appended, running a different operation under the signer's identity. Add every operation-selecting subresource to the signed list. Clients that already sign tagging, cors, object-lock or replication in V2 now verify as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: sign every value of a repeated signature V2 subresource canonicalizedResourceV2 kept only the last value of a repeated query key, while handlers read the first. A request signed for versionId=A could be sent as versionId=B&versionId=A and act on version B. Sign every value in order, as V2 clients do, so a second value no longer verifies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a2b9b37ee5 |
volume(rust): untrack queued writes before replying (#12595)
* volume(rust): untrack queued writes before replying
The durable write worker sent each write's result and only then dropped
the batch's pending-write trackers, so a caller could see its write
finish while the write was still tracked as pending. Go untracks in the
caller before writeNeedle2 returns. Drop the trackers before sending the
results so the Rust queue path matches.
test_replicate_delete_cancels_a_queued_write asserted the count was zero
right after the fsync write returned and failed about 1 run in 100 under
load (Rust Unit Tests on master
|
||
|
|
a2ffdce2ec |
volume: skip VolumeDelete when its RPC was cancelled before the delete (#12586)
* volume: skip VolumeDelete when its RPC was cancelled before the delete A caller that cancels (a lost tiering lease, an expired deadline) still had its admitted delete run in the Go server, including one that waited behind the copy guard. The Rust server drops a cancelled handler, so return the context error before the delete starts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: wait until the queued delete blocks on the guard before cancelling Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6388dfca9b |
s3api: verify signed request bodies against the signed payload hash (#12577)
* s3api: verify signed request bodies against the signed payload hash A SigV4 signature covers the declared x-amz-content-sha256, but only getRequestDataReader (PutObject, UploadPart) compared the body with it. Every other handler parsed the body as received, so a signed request, DeleteObjects for example, could be replayed with a different body. After a V4 signature verifies, wrap the body so it is handed out only once all of it has been read and matched against the hash the signature covered; a decoder that stops reading early never sees unverified bytes. getRequestDataReader unwraps it and keeps verifying uploads as they stream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: answer XAmzContentSHA256Mismatch for refused DeleteObjects bodies DeleteObjects answered a body that failed its payload-hash check with InternalError and CompleteMultipartUpload with MalformedXML. Name the check that failed instead, as PutObject does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: check Content-MD5 on signed request bodies Signature V2 signs Content-MD5 rather than a payload hash, and a V4 request sent with UNSIGNED-PAYLOAD may still sign it, but bodies read outside the upload path never compared it with the body. Check it with the payload hash once a V2 or V4 signature verifies: BadDigest on a mismatch, InvalidDigest when it is malformed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: verify presigned uploads against a signed payload hash A presigned V4 URL can sign X-Amz-Content-Sha256 as a query parameter, but getRequestDataReader compared upload bodies only with the header of that name, so the signed hash went unchecked. Verify uploads against the hash the signature covered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: check Content-MD5 on bodies buffered for their payload hash IAM and other non-S3 requests signed without x-amz-content-sha256 have their body buffered and hashed during signature verification. That buffer counted as already bound, so their Content-MD5 was never checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: check Content-MD5 on directory marker bodies A PUT of a key ending in "/" stores its body on the directory entry without going through putToFiler, so its Content-MD5 was never compared with the body. Answer BadDigest on a mismatch, as putToFiler does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: wrap and unwrap signed bodies through helpers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a3b68aa37c |
shell: confirm trim survivors still serve the volume before deleting (#12590)
* shell: confirm trim survivors still serve the volume before deleting volume.fix.replication compared the doomed replica's index with every survivor and then deleted it unconditionally. A survivor that went away while the indexes were read still let the delete through, leaving fewer copies than the check had counted. Re-read each survivor's VolumeStatus after the index checks and skip the trim if any no longer has the volume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: test a trim whose survivor goes away after its index read Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4cba4eb7bf |
s3: ignore unsigned object headers on POST Object (#12585)
* s3: ignore unsigned object headers on POST Object The POST policy signs form fields only, but the upload read tagging, user metadata, storage class, object lock and the stored content headers from the request headers as well. Drop them before the form fields are applied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test POST Object request headers against signed form fields Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
249f04e63a |
s3api: NoSuchKey for object lock calls on missing or deleted keys (#12592)
* s3api: answer PutObjectLegalHold on a missing key with NoSuchKey In a versioned bucket a key with no latest version resolves to ErrLatestVersionNotFound, which the legal hold handler did not map, so the request failed with 500 InternalError. PutObjectRetention already maps it to NoSuchKey. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: keep failed reads distinct when resolving an object lock target getObjectEntryForUpdate turned every lookup error into a not-found sentinel, so a failed filer read made PutObjectRetention and PutObjectLegalHold answer NoSuchKey for an object that exists. Only a genuine miss becomes the sentinel now, matching getObjectEntry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: answer object lock calls on a delete marker with NoSuchKey PutObjectLegalHold and PutObjectRetention on a key whose current version is a delete marker returned 200 and wrote the lock onto the marker, and the matching GETs read it back. A key whose current version is a delete marker does not exist, and delete markers take no lock, so these calls now answer NoSuchKey like the tagging and ACL handlers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: let a lock recorded on a delete marker not block its removal Delete markers are not WORM-protected, but a legal hold or retention written onto one (possible before the previous change) made both the gateway check and the routed delete's filer-side WORM guard refuse to remove the marker, so the object could not be restored and lifecycle could not clean the expired marker up. Lock enforcement now ignores a lock on a delete marker, and the routed version deletes in DeleteObject and lifecycle drop the WORM guard when the version they read is a marker. A version id never changes kind, so the marker read is the one deleted; the null version keeps its guard because a PUT can replace it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4ee82977af |
docker: pass entrypoint arguments to weed without splitting them (#12589)
The master, volume, volume-rust, server, mini, filer and s3 cases and the fallback forwarded the user arguments as an unquoted $@, so the shell split each one at whitespace and globbed it against /data. An -admin.password with a space reached weed as two words, and weed ignored every flag after the stray word. Quote "$@" as the other cases already do; $ARGS stays unquoted since the script builds it from its own words. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bef1b13b3d |
replication: keep a local sink path the source recreated after a delete (#12581)
* replication: let the local sink check a delete against the live source LocalSink removed the destination path for any delete it was handed. A delete that reaches the sink after the source recreated the path (a peer filer's event delivered late, or a historical event replayed) removed the newer file, and no later event brings it back. DeleteEntryVersion receives the deleted entry. Before removing an existing destination, the local sink looks up the source path and keeps the destination when the source holds an entry of the same kind committed after the deleted version (commit_ts_ns, or mtime for unstamped entries). A live entry that is the deleted version itself or older means the source store lags the event, so the delete proceeds. A failed lookup returns the error so the event is retried instead of deleting a file that may be live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.sync, filer.replicate: hand the deleted version to the sink Pure deletes, renames without a native move and moves out of the watched directory now go through sink.DeleteEntryVersion with the old entry, so a sink that checks the live source (the local sink) can keep a path the source recreated after the deleted version. Other sinks fall back to DeleteEntry unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * replication: order the local sink check by the delete's mutation time A source filer can hold a version written concurrently with the delete: newer than the deleted entry but older than the delete itself. The source's own replay applies the delete over it, so keeping the backup copy because it outranks the deleted entry left a file the source no longer has. The sink now gets the delete's mutation time (MutationTsNs, else TsNs) and keeps the destination only when the live entry was committed after it, the order Filer.Replay keeps. That also covers a stamped replacement of an unstamped entry whose mtime did not move. filer.replicate events carry no time, so they keep the deleted entry's stamp as the bound, and mtime only when nothing is stamped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * replication: count an unstamped live entry from its mtime A source filer that predates commit stamps stores entries with no CommitTsNs, so comparing the live stamp with the delete's time never kept a replacement there. An unstamped live entry now counts from its mtime against the delete's time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * replication: keep a stamped live entry when the deleted one is unstamped and the delete has no time filer.replicate gets no delete time from the notification queue. When the deleted entry has no commit stamp either, a stamped live entry was written after it, so its mtime no longer decides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c2625ddf00 |
seaweed-volume: read security.toml values as TOML strings (#12582)
The line parser only trimmed quotes, so `key = "abc\n"` gave the Rust volume server the bytes `abc\n` with a literal backslash while the Go servers sign with a newline, and every JWT failed. Literal strings kept their quotes and a trailing comment stayed part of the value. Decode each value with the TOML parser; text it rejects keeps the old quote trimming. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b7d6b48921 |
operation: probe octet-stream uploads for gzip like untyped ones (#12584)
* operation: probe octet-stream uploads for gzip like untyped ones The 128-byte gzip probe ran only when the caller passed no MimeType, so an upload labelled application/octet-stream or binary/octet-stream was never compressed. S3 PutObject and UploadPart pass the request Content-Type (or the octet-stream that mimeDetect fills in), so every binary S3 upload skipped compression that the same bytes get through the filer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * operation: check the probed upload keeps the caller's content type Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
02b9747bd8 |
wdclient: keep a shared volume lookup alive when its first caller leaves (#12579)
* wdclient: keep a shared volume lookup alive when its first caller leaves Concurrent misses for the same volumes share one provider lookup, but it ran on the first caller's context, so that caller's cancellation failed every request waiting on it. The lookup now runs on a context no single caller can cancel, and each caller stops waiting when its own context ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * wdclient: wait for the second caller to join before canceling the first A fixed sleep did not prove the second caller had joined the shared lookup: a late caller could hit the filled cache and pass without waiting. The test now waits until the caller reads its own Done channel, which happens only after DoChan registered it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8581151376 |
filer: gzip metadata log pieces before uploading them (#12580)
* filer: let the fake-master test helper serve any master Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: gzip metadata log pieces before uploading them Every metadata event carries the old and new entry with their whole chunk lists, so a file grown by small fsynced appends logs the same chunk records again and again: 2000 appends of 256 bytes through weed mount wrote 34.8 MB of log. The uploader only compresses when the first 128 bytes compress, and a log piece starts with fresh file ids and ETags, so it stored the log as is. The filer now gzips each piece itself and keeps the compressed bytes when they save at least 10%. After the same workload the volumes hold 3.05 MB instead of 35.2 MB, file data included. Readers already honor the chunk's IsCompressed flag on every path (filer-side decode, log file refs read straight from volume servers), and the assign request sizes the bytes actually stored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e0f8438284 |
mount: retry a writeback metadata flush until the filer answers (#12578)
* mount: keep the filer status in the flush create error flushMetadataToFiler formatted the CreateEntry error with %v, so callers classifying it saw only the message, which includes the file path. Wrap it so the gRPC status the filer sent stays reachable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: retry a writeback metadata flush until the filer answers With -writebackCache, close() returns before the chunk references reach the filer. The deferred flush gave up after four attempts, about a minute once each attempt's own filer retries are counted, then released the handle: a longer filer outage dropped the file even though write() and close() had succeeded, and its uploaded chunks became orphans. Keep retrying for as long as the filer is unreachable, as the writeback chunk uploads already do. A filer that answers with an error still ends the flush after the bounded attempts, and an unlink of the draining file still ends it on the next attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: retry only an unanswered filer call without a bound The unbounded phase used util.IsTransientError, which falls back to matching the error text. A refusal the filer sends in its response (read-only, name too long, a plain error string) has no gRPC status, and the flush error carries the file path, so a file named like "unavailable-transport.log" kept retrying against a filer that answered. Retry without a bound only when the call failed with a gRPC Unavailable status: the dial, the connection or the mount's call deadline gave out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a6a3bdef8d |
s3api: deny signed reads of a version file through its physical path (#12574)
* s3api: deny signed reads that name a version file by its physical path A stored version lives at <key>.versions/v_<id>, and that path is also a valid object key. Anonymous reads already refuse it; signed GET, HEAD, copy sources and object subresources did not, so a caller without s3:GetObjectVersion, or under a Deny on the logical key, could read any version through it. Apply the same check in the shared entry recheck. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: recheck a version file key before answering a conditional read A conditional GET or HEAD answered 304 or 412 with the ETag before the handler's entry recheck, so the physical version path still confirmed a version's ETag to signed callers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: look up version file keys in the lookup-based policy recheck Retention, legal hold, tagging writes and deletes only fetched the entry for the recheck when the bucket had a policy, so without one they still acted on a version file named by its physical path. Fetch it whenever the key has the version file shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: refuse a directory-marker delete that names a version file A trailing slash kept "<key>.versions/v_<id>/" out of the version file shape check, and the lookup recheck misses the entry for that form, so a DELETE reached deleteDirectoryMarker and removed a chunkless version or delete marker without s3:DeleteObjectVersion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
317d907334 |
filer: route tests forward to a peer port the test holds (#12576)
The ring tests used 127.0.0.1:18889 as an unreachable peer, but its gRPC port 28889 is a common filer port: on a shared CI host a live filer there accepted the forwarded exclusive create, and TestCreateEntryExclusiveLeavesNonOwner failed. The peer now listens on a port the test owns and drops every connection. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
13334a59b9 |
vector: reuse the Lance session, version leases and opened tables across reads (#12573)
* vector: share one Lance session across reads Each request opened its table in a session of its own, so every query loaded and decoded the index again. Get, list and query now open in one session the server keeps, sized by -index-cache-mb and -metadata-cache-mb; writes keep a session of their own. Lance keys some of its cached entries, such as a version's index list, by location and version alone, and an index dropped and created again at its location numbers its versions from 1 again: a shared session answered a query on the new table with the old table's index. CreateIndex now marks each table with an id in its schema metadata, and the session starts over when a location turns up holding another table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: share a version lease among the reads of that version Every get, list and query tagged the version it read and deleted the tag afterwards: two filer writes and a manifest read per request. Reads of one version now share one tag, which goes 10 s after the last read took it; a tag is reused for at most 5 minutes, so a busy version never reaches the reapers' TTLs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: keep opened tables for the next read A read built the table's store and read its latest manifest each time. Reads now keep the opened table by location and encryption, up to 1024, and move it to the latest version, which lists the versions and takes the manifest from the session. A handle opened in an earlier session is not reused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: key shared leases by manifest, and take one per version at once A table dropped and created again within the linger reused, for a version number the old table had, a lease whose tag the drop had deleted. Leases are now keyed by the manifest's ETag as well. Reads that arrive before a version has a lease wait for the first to take it, rather than each writing a tag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: drop kept tables when the shared session starts over A kept table holds the session it was opened in, so a replaced session and its caches lived on in the handles of other tables until they were evicted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: prune shared lease slots only when they double Every lease lookup scanned all slots under the map's lock, a cost that grew with the versions read in the last linger. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: shrink the lease slot map after a prune After a burst the map kept its peak allocation, which every later prune scanned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: keep a lease slot a read has taken through a prune A read that took a slot and had not yet locked it could see the slot pruned, and then pin through it while the next read pinned through a new slot, writing a second tag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b924ffabf1 |
admin: calm console restyle with light and dark themes (#12568)
* admin: restyle the UI as a calm console Grey canvas with white sheets, a docked sidebar that marks the current page with an indigo tint, ink primary buttons, tinted status pills, dense tables with tabular numbers and spectrum capacity meters. The styles are design tokens layered over Bootstrap's own variables, with light and dark values, so the page templates stay unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: add a light, dark and automatic theme switch The choice is kept in localStorage and applied before the page paints, on the sign-in page too; automatic follows the system setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: drop hard-coded colors from page templates The volume id link, utilization track and plugin form borders used fixed light-theme colors; sparklines now use the data palette. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: theme the plugin job detail panel background Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep the selected radio dot visible in dark mode Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: cycle the theme from memory when storage is blocked Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep success progress bars green Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep keyboard focus visible on the toolbar togglers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: use contrasting text on danger and success buttons in dark mode Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: lead the dashboard with cluster health and one row of tiles The header now says in one line whether the cluster is healthy and how many masters, volume servers and filers report. The two rows of stat tiles become one: the volume size limit and EC volume count move into footnotes, the mount client card joins the cluster counts, and the counts link to their pages instead of sitting in nested cards. The duplicate version list and the bucket shortcut in the header are gone, and the section menus become plain View all links. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: export the user columns the table actually has The users table has username, email and access key columns, but the export waited for six cells per row and so always produced only the header line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: drop table titles that repeat the page title List pages titled their only table with the page title again. The table now sits right under the summary tiles, and the export actions that were hidden in the table header menu move to the page header as an Export button, matching the cluster pages. The policies menu linked to nothing and is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: drop clock and duplicate summary tiles Buckets, users, policies and service accounts spent a summary tile on the refresh time, which the page footer already shows (policies gains that footer). The users page also showed its user count twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: show unset bucket settings as a dash Rows of No owner, No quota and Not configured made the bucket table hard to scan. Unset cells now show a quiet dash that keeps the wording as a tooltip and for screen readers, and the name and creation date no longer wrap (the full timestamp is in the date's tooltip). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: export buckets through the page's API exporter admin.js loads after the page, so its copy of exportBucketList replaced the buckets page's own one. That copy scraped the visible rows, so it exported only the current page, without CSV quoting, and picked up the display text of the cells. The page's exporter reads every bucket from the API with full timestamps and escaped fields. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: escape quotes in the user export Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: download the bucket export as a Blob The exporter put the CSV into a data URL with encodeURI, which leaves # alone, so a bucket owner containing # ended the download there. Use the shared downloadCSV helper instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: count masters as configured, not reporting The master list comes from raft membership, which keeps a follower that is down, so it does not show which masters are reporting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: link the Prometheus metrics endpoint from the sidebar Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: drop the sidebar metrics link Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1618d613a3 |
s3: resolve a role session's aws:username from its user, not its session name (#12572)
* s3api: record the caller on AssumeRole sessions The session name is the caller's choice; the session now carries the caller itself as the user it was issued to, as the web identity, LDAP and credential paths already do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: take a role session's aws:username from its user, not its name An assumed-role ARN ends in the session name the caller picked, so naming a session after another user resolved ${aws:username} and ${aws:userid} to that user in bucket and attached policies. Resolve them from the user the session was issued to instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iam: take a role session's aws:username from its user, not its name IsActionAllowed set aws:username to the last segment of an assumed-role ARN, the caller-chosen session name, for role and bucket policies. Use the user the session was issued to, and drop a forwarded value when the session names none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: pass a role session caller's user on to the session it assumes A role session that calls AssumeRole is named by its session ID, so the new session recorded that ID as its user. An STS session caller now passes on its own recorded user, or none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: size the session claims map from the request context Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3f557b73c8 |
admin: show the right master ports when -port.grpc is not port+10000 (#12570)
* admin: take master HTTP ports from raft member names The masters page and dashboard turned each raft member's gRPC address into an HTTP address by subtracting 10000 from the port, so masters with a custom -port.grpc showed wrong or negative ports. Raft members are named after their master address, which carries the HTTP port; use it and keep the old derivation only for names that are not addresses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: test raft member address mapping Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a0f95e16bc |
s3api: fix TestRoutedMultipartFinalize flake from a reused fake filer port (#12571)
* s3api: drop cached gRPC channels when a fake filer takes a port A test that dials a dead owner leaves its failed channel in the process-wide gRPC connection cache. When a later test's fake filer is given the same port, its first RPC fails at once with the cached connection refused, which is how TestRoutedMultipartFinalize failed on the 32-bit job. The fake filer helpers now share fakeFilerAddress, which invalidates any cached channel for the new address. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test a fake filer on a dead owner's reused port Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: drop cached channels for dead test filer addresses too closedFilerAddress can be handed a port whose cached channel belonged to a fake filer that just stopped and may still look ready; then the dead owner's transaction is sent on it instead of failing over. Build its address with fakeFilerAddress as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6c1eec8703 |
s3: store the POST Object acl on the uploaded object (#12567)
* s3api: split upload ACL assembly out of preparePutObjectACL Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: store POST Object acl and grant fields on the object The form acl and x-amz-grant-* fields were authorized but never assembled, so the object kept no ACL: acl=private on a public-read bucket stayed anonymously readable and invalid values were accepted. Build them with the same rules as PutObject, which also covers the Block Public Access check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test POST Object stores its upload ACL Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: take the POST Object ACL from the signed form only ACL request headers are not covered by the POST policy, so a holder of a signed form could add one and have it stored with the upload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: reject a POST acl field the policy does not cover The acl form field now sets the stored object ACL, so like the x-amz-* fields it needs a policy condition, as AWS requires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test unsigned ACL headers and uncovered acl fields on POST Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: evaluate POST upload policies on the canonical ACL Policy conditions on the grant headers saw the raw form value, so an equivalent spelling of a denied grantee passed. Use the PutObject request shape that the PUT path evaluates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: check bucket policy for POST ACLs with auth disabled PutObject applies the bucket policy to an explicit upload ACL when IAM is off; POST now does the same before storing it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: reject differing acl and x-amz-acl POST fields Both promote to the canned ACL header, so map order picked the stored ACL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test POST grant canonicalization, auth-off ACL policy, acl conflicts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: check the auth-off POST ACL policy after loading the bucket Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: reject an empty acl or x-amz-acl POST field beside a differing one Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test the auth-off POST ACL policy on a cold bucket cache and empty acl aliases Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: load the bucket before per-key object authorization Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test the POST ACL policy on a cold bucket cache with auth on Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: stop per-key object authorization when the bucket fails to load Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test the cold-bucket POST ACL allow path and a failed bucket load Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
eb159f3181 |
lance: deregister and re-register need UpdateTable (#12552)
* lance: require UpdateTable to deregister a table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: require UpdateTable to register over an existing table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: test that GetTable alone cannot deregister or re-register a table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: a register that finds a table the caller cannot read needs UpdateTable to bring it back Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: decide declare write access by whether the table existed, not by what the caller can read Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: test register and declare by a caller that cannot read the table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: register clears the marker on the create branch only after checking it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: keep a register whose table was created when the marker check fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4cb52d7705 |
filer: reopen a leveldb store wedged by a transient write error (#12565)
* filer/leveldb_handle: reopen a database whose journal latched a write error goleveldb keeps the first journal write error and returns it for every later write until the database is reopened, so one transient ENOSPC or EIO fails all writes until a restart. The handle offers the leveldb.DB methods the stores use; a write that fails on an OS error probes the storage and, once it accepts writes and no call holds the database, closes and reopens it and retries the write once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: open the leveldb stores through leveldb_handle leveldb, leveldb2 and leveldb3 now recover from a transient storage write error without a restart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: leveldb stores return iterator errors from listings and deletes A listing on a database that cannot be read reported an empty directory, and DeleteFolderChildren wrote an empty batch, because the iterator error was never checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/leveldb_handle: test a reopen that fails after the probe passes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/leveldb_handle: probe storage before draining readers A probe that stalls in Create, Write or Sync held mu exclusively and blocked every read on the handle. Reopen attempts are now serialized by their own mutex, and the probe runs before the drain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/leveldb_handle: hold new calls back while a reopen drains TryLock succeeds only at an instant with no reader, so overlapping listings could keep a reopen from ever draining while each write spun for the full drain timeout. The drain now holds new calls back for at most drainTimeout; a call nested inside a running one waits that out instead of deadlocking. A drain that times out holds off the next attempt for busyReopenInterval, so new calls are not held back over and over. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/leveldb_handle: bound the wait for a running reopen A write that hit the latched error waited on the reopen mutex for as long as another caller's storage probe ran, so a stalled probe queued every writer. Callers now wait at most reopenWait and return their error, and still retry once a concurrent reopen succeeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/leveldb_handle: tests wait for the drain and readers explicitly The nested-read test polled for a drain that could start and end between polls and then spun forever; it now syncs on the probe, widens the drain and fails if the put returns first. The overlapping-readers test waits until every reader holds an iterator before it writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5150168368 |
s3: authorize an empty versionId as the current-object action (#12566)
* s3: authorize an empty versionId as the current-object action GET, HEAD and DELETE treat an empty versionId as the current object, but the action resolver keyed on the parameter's presence and authorized it as GetObjectVersion/DeleteObjectVersion, so a Deny on GetObject or DeleteObject did not apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test an empty versionId against a version-only policy Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: cover a bare versionId on HEAD Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0c6c6868de |
volume: refuse a tier upload whose .dat was written during the copy (#12562)
* volume: refuse a tier upload whose .dat grew during the copy AttachRemoteDatFile only checked that the data backend was still the one copied, so a needle write acknowledged after the copy read the .dat was missing from the remote object, and the upload then switched reads to it and removed the local .dat. Compare the .dat size with the copy under the same data-file locks; on a mismatch the handler deletes the copy and keeps the local file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a tier upload whose .dat grew during the copy Mirror the Go AttachRemoteDatFile check. The tier-up attach took the write lease only inside each step, so a needle write acknowledged after the upload read the .dat was missing from the object the volume then switched to, and the local .dat holding it was removed. Hold one write lease across the instance, revision and size checks, the remote-file update, the .vif save and the backend swap, as tier-down does; a size mismatch aborts, deleting the object and keeping the local .dat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: compare the physical .dat size before a tier attach DiskFile pads its size past an unaligned tail, which a read-only load keeps, while the upload copies the physical bytes. Stat the file itself so such a volume is not refused as written during the copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
fcfca3fee3 |
s3, filer: bound the wait for an upload slot (#12559)
* util: add WaitUntil, a cond wait bounded by a context and a timeout The caller holds the cond lock. Cancellation and the timeout wake the waiter through a broadcast taken under the same lock, so no wakeup is lost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: bound the wait for an upload slot A PUT that arrived while the concurrent upload limits were exhausted waited on the condition variable with no bound. The handler reads nothing from the body while it waits, so a client that gave up and reset its connection went unnoticed: the socket and goroutine stayed pinned until a slot freed, and a stalled volume server grew the gateway's fd count by one per aborted PUT. The wait now ends on InflightUploadDataTimeout with 503 SlowDown, which S3 SDKs retry, and on request cancellation. An admitted upload takes its slot under the lock, and a finished upload broadcasts, so freed bytes admit every waiter that fits instead of one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: bound the wait for an upload slot The filer's PUT/POST upload limits queued requests the same unbounded way. Reuse the bounded wait: reject with 429 after InflightUploadDataTimeout, as the volume server does, and stop on request cancellation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * command: add -inflightUploadDataTimeout for the S3 gateway and the filer weed s3 -inflightUploadDataTimeout, weed filer -inflightUploadDataTimeout and -s3.inflightUploadDataTimeout, weed server and weed mini -filer.inflightUploadDataTimeout and -s3.inflightUploadDataTimeout. The default of 60s matches the volume server's flag of the same name; 0 keeps the old unbounded wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: reject a queued upload without reading its body WriteErrorResponse drains the request body before answering. Behind the stats recorder the drain has no read deadline, and reading the body sends an Expect: 100-continue client the go-ahead to upload what is about to be discarded. Answer the rejection with an empty-body copy of the request so the drain is skipped; net/http closes the connection when a large unread body remains. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3, filer: release an upload slot under the cond lock A waiter checks the counters under the cond lock and then waits. A release that decremented and broadcast without the lock could land between the two, so the waiter slept with a slot free until the next release or its timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3, filer: close the HTTP/1 connection when rejecting a queued upload net/http reads up to 256 KiB of an unread request body before sending a keep-alive reply, so a client that stalls its body held back the 503 or 429 until the connection idle timeout. Connection: close skips that read. HTTP/2 is left alone: there it would shut down the whole connection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d659228338 |
shell: s3.clean.uploads frees an open upload under the object write lock (#12558)
* shell: s3.clean.uploads deletes uploads over filer gRPC Same DeleteEntry the HTTP handler runs, without minting a filer JWT, and the gRPC path is the one a locked, routed delete needs next. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads frees an open upload under the object write lock The cleaner deleted a stale upload recursively with no lock, so a CompleteMultipartUpload could pass its upload-exists check after the part entries and their chunks were gone, and publish an object over freed chunks. Do what AbortMultipartUpload does: hold the object's distributed write lock, and free an open upload in an ObjectTransaction routed to the object's owner filer, conditioned on the object not carrying the upload; if it does, delete the metadata only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads keeps the object write lock for the whole cleanup A large upload's recursive delete can outlast one 15 second lease, and a completion on the distributed-lock path could then take the lock and commit over chunks being freed. Renew the lock in the background and skip the delete once it is lost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads locks the path a completion commits under For a trailing-slash key a non-versioned completion writes and locks the entry inside the directory the key names, not the object path, so the cleaner's transaction did not serialize with it. Lock and check the same path the completion uses for the bucket's versioning state; the route key stays the object's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: abort locks the path a completion commits under A routed abort locked and checked the object path, but for a trailing-slash key a non-versioned completion commits under the entry inside the directory the key names. Use the same path the completion locks for the bucket's versioning state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: end s3.clean.uploads bucket failure line with a newline Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cluster: NewBlockingLongLivedLockWithin bounds the wait on a held lock Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads skips an upload whose object lock stays held Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cluster: share the contended lock filer setup across lock tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads reads the bucket's versioning state per upload The commit path a routed completion locks depends on versioning for a trailing-slash key. Reading it once per bucket before the run left a window as long as the run; read it just before the delete, as the gateway's abort does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b11af30cb1 |
mount: remove the local control socket when the mount exits (#12560)
* mount: remove the local control socket when the mount exits weed mount listened on /tmp/seaweedfs-mount-<hash>.sock and removed it only when a later mount of the same directory started, so every unmount or SIGTERM left one socket file behind. Remove it on the normal return path and in the interrupt hook, unless a later mount of the same directory has already replaced it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: remove the control socket before a failed mount exits glog.Fatalf and glog.Exitf call os.Exit, which skips the deferred removal, so a mount that failed after creating its socket (mount point not available, fusermount failure) still left it behind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: register the socket cleanup hook before mount setup A SIGTERM that arrived while the mount was still setting up, for example waiting on the filer to create the mount root, exited through grace without running the deferred cleanup. Registering the hook right after the listener is created makes it run last, after the unmount and filesystem hooks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
7f18576798 |
filer/etcd: serve volume.fsck from one etcd revision (#12555)
* filer/etcd: scan entries from one revision volume.fsck walked etcd one directory at a time, so a file moved from a directory not yet listed into one already listed was seen under neither name and its chunks were reported as orphans. The etcd store now implements filer.EntryScanner: every page is read at the revision of the first read, and a compacted revision fails the scan instead of falling back to the walk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/etcd: read hard links at the scan revision Hard-link records resolved during a scan were read at the current revision, so a link updated meanwhile mixed two instants. Reads made with the scan's context now use its revision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: list etcd among the stores volume.fsck reads from a snapshot Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/etcd: skip hard-link records in a scan Hard-link records live beside the entries under their random ids, so a root scan could read one as an entry with a path made of random bytes, which the traversal stream cannot send. Skip hard-link ids and keys whose directory is not valid UTF-8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/etcd: recognize a hard-link record by the id it stores A hard-link id can be valid UTF-8, so the key shape alone let some records through a root scan as phantom files. The record's value is the shared entry, whose HardLinkId is the record's own key; no entry key can equal its own hard-link id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/etcd: scan entries under an invalid-UTF-8 directory The filer sanitizes names but stores directories as given, so a path written over HTTP can leave entries the listing never reaches. Skipping them let volume.fsck count their chunks as orphans. Send them under the sanitized directory instead, the substitution the filer uses for names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/etcd: fail a scan whose sanitized path names another entry An entry under an invalid-UTF-8 directory is reported under the sanitized directory. When that path is also another stored entry at the scan revision, or another sanitized entry in the same scan, acting on the reported path (volume.fsck deleting it) would reach the other file, so the scan fails and names the stored path instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
641ab6204b |
shell: fs.mergeVolumes leaves in-progress multipart parts in place (#12556)
* shell: fs.mergeVolumes leaves in-progress multipart parts in place CompleteMultipartUpload reads the part entries and later commits the object with their fids, guarded only by the upload directory. A merge that moved a part chunk in between deleted the source needle the new object pointed at. Parts stay where they are until the upload completes or is cleaned, and their chunks still count as referenced sources. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: note in fs.mergeVolumes help that upload parts stay in place Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: match upload parts when the buckets folder is the root Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: count chunks behind a part's manifest as referenced sources A skipped part whose chunks on a merge source sit behind a manifest was not counted, so the merge printed no skip line and warned that nothing references the source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep a part's direct source references when its manifest is unreadable Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: count chunks resolved before an unreadable part manifest ResolveChunkManifest returns the data chunks of earlier manifests along with a later manifest's error; record them before returning so the unreferenced-source warning does not fire for a source they reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: test the partial manifest result without a timing delay The unreadable manifest is now nested under a readable part manifest, so the first level resolves fully before the failing lookup runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e9606c8ecc |
filer.meta.backup: resume an interrupted metadata copy and add -entriesPerSecond (#12549)
* filer_pb: add a depth-first traversal that can resume after a path TraverseDfs walks a tree in listing order and reports each finished path, so a caller can save one path and continue from it later. Pages are read whole before visiting, and the entries of a page are visited by up to five workers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.meta.backup: resume an interrupted metadata copy The walk saves the last finished path every few seconds, and a walk that continues an unfinished one starts after it. A fresh walk clears the saved path first, and a finished walk clears it with the walk start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.meta.backup: add -entriesPerSecond to throttle the metadata copy Each entry the full copy reads waits on a rate limiter, and pages are listed only after the previous page is copied, so the flag also bounds how fast the copy lists the source filer. 0 keeps it unlimited. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer_pb: report each file of a depth-first walk once all before it are done A throttled walk only reported progress after a whole listing page, so a slow copy could go many minutes without a resumable path. Visit entries in runs that end at a directory and report each file as soon as every entry before it is done. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0ef0260844 |
volume: fail a vacuum step that finds the volume already compacting (#12553)
* volume: fail a vacuum step that finds a compaction already running Compact, CompactByIndex and CommitCompact returned success when another compaction held the volume. A second, uncoordinated vacuum (an expired admin job still running, a shell volume.vacuum next to the worker) then went on to commit or clean up the first run's .cpd/.cpx. Return a "volume N is compacting" error instead, and refuse cleanup while a copy is in flight, as the Rust volume server already did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: check the compaction files before a commit closes the volume CommitCompact closed the needle map and data file and wrote the .cpc marker before it found .cpd/.cpx missing. A duplicate commit, such as the second of two overlapping vacuums, then left the volume registered with no open files: it dropped out of the heartbeat and looked lost to volume.list and volume.fix.replication until a restart. Check for the files first so the refused commit leaves the volume as it was. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: run makeupDiff before a commit closes the volume CommitCompact closed the needle map and data file before makeupDiff, so a makeupDiff failure (a compaction revision another commit moved, an I/O error) returned with them closed: the volume stayed registered but missing from the heartbeat until a restart. makeupDiff reads the live files through its own handles while the commit holds the write lock, so close them only once it has succeeded, as the Rust volume server does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: log a volume the heartbeat skips because its files are closed A volume left without its needle map or data file vanished from the heartbeat with no log line, so the master saw a replica disappear while its files were still on disk. Warn on each skip, as the deleted-.dat case next to it already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: claim the compaction flag while cleanup removes the temp files Cleanup only read the flag, so a compaction could start between the check and the removal and lose its .cpd/.cpx. Claim it for the cleanup instead, in Go and Rust. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a4a68c885a |
filer.sync: move the watermark over jobs that finished out of order (#12554)
* filer.sync: move the watermark over jobs that finished ahead of an older one The watermark only took the finishing job's timestamp when it was the oldest in flight, so jobs that completed before an older one were never recorded and a quiet stream kept the offset behind finished events. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.sync: keep one finished timestamp per gap behind a running job A long job kept every later completion in the done heap. Only the newest one between two running or failed timestamps can become the watermark. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.sync: test done-heap compaction across several running and failed jobs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ea6e9d670a |
volume: forward no Content-MD5 for a JPEG the orientation fix rewrote (#12557)
* volume: forward no Content-MD5 for a JPEG the orientation fix rewrote With -images.fix.orientation the primary stores and forwards the rotated bytes, but sent replicas the client's digest of the original upload. The replica checks Content-MD5 against what it receives and rejects the write, so replicated JPEG uploads with a digest failed. The Rust volume server already drops the digest once the body is rewritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: test the forwarded Content-MD5 through PostHandler and a replica The test now posts each upload to a primary whose fake master names a real replica volume server, so it covers the JPEG gate, ReplicatedWrite and the replica's own Content-MD5 check, and reads the stored bytes back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: fail the replica forwarding test instead of hanging when nothing is forwarded Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: compare the primary and replica bytes in the replica forwarding test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
fb5ffb815f |
filer: validate bucket names before the SQL DDL hooks (#12547)
* filer: validate bucket names before the SQL DDL hooks The SQL store builds CREATE TABLE and DROP TABLE statements by formatting the bucket name into the query. The DML paths already reject names that fail VerifyS3BucketName, but the bucket-creation and bucket-deletion hooks ran the DDL without that check, so a name reaching them through the filer or WebDAV directory path (which skips the check for dot-prefixed directories) could inject stacked statements. Apply the same validation the DML paths use. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * filer: check that a valid bucket runs both DDL statements Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
d350a0ad03 |
weed-db: end startup when password authentication fails (#12550)
* weed-db: end startup when password authentication fails The failure branches returned the result of writing the error response, which is nil on success, so the handshake carried on to ReadyForQuery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-db: test startup with wrong and unknown credentials Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-db: check the auth error code and fail on stray read errors in the startup test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f3150c3495 |
admin: escape file properties before rendering the modal (#12548)
* admin: escape file properties before rendering the modal The Properties modal concatenated the entry name, path, MIME type and extended attributes into HTML, so object metadata written over S3 could inject markup into the admin page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep every property key when escaping Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8d96985131 |
security: scoped filer tokens are not admin credentials (#12551)
* security: do not accept a path- or method-scoped filer token as admin Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * security: test that scoped filer tokens fail the admin token check Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d485d2c9cd |
filer.sync: shorten retries for confirmed missing chunks (#12546)
* filer.sync: shorten retries for confirmed missing chunks * filer.sync: clarify missing chunk grace period names |
||
|
|
6c473433c5 |
vector: bound request buffering before authentication and document installation (#12537)
* s3vectors: bound body buffering before authentication * docs: include vector engine installation and source builds * s3vectors: prevent MIME sniffing of JSON error responses * docs: load vector Cargo configuration when building from source * docs: note the native protoc requirement for Windows vector builds * docs: make vector source checkout and working directory explicit * ci: handle deb822 sources during Ubuntu mirror failover * ci: bound Ubuntu mirror attempts and reject partial index updates |
||
|
|
8236c8d136 |
vector: integrate engine health, admin management, and local deployment (#12517)
* vector: expose request metrics and registration-aware health endpoints * vector: report immutable read cache and storage fetch metrics * docs: describe vector engine health, metrics, and maintenance * vector: count canceled requests when their handler is dropped * vector: account for uncached large immutable storage reads * vector: reliably release the stalled credential reader in tests * vector: observe large reads as their streamed body is consumed * vector: bound heartbeats on established master registration streams * test: exercise hardened vector maintenance on Go and Rust volumes * vector: register engines in their configured filer group * vector: make retried index creation preserve compatible datasets * vector: allow registration to advertise a separate address * test: release stalled credential readers without blocking FIFO cleanup * test: allow hardened vector coverage in a named filer group * vector: discover engines in the same filer group as their gateway * vector: fall back when the advertised address is empty * admin: expose vector readiness, maintenance, and management tools (#12520) * s3vectors: share vector discovery with trusted management clients * admin: show vector engine readiness and recent maintenance activity * admin: suggest S3 Vectors IAM actions from the supported operations * admin: manage vector indexes, policies, tags, encryption, and query data * admin: validate SSE-S3 support before creating encrypted vector buckets * admin: discover vector engines in their registration group * admin: probe vector engines concurrently with independent time budgets * admin: verify encryption capability on the remote S3 gateway * admin: defer discovery until after the HTTP server can start * admin: distinguish unknown vector health services from failed engines * admin: count workers that can execute vector index maintenance * admin: allow read-only users to query vectors and inspect tags * vector: scope gateway and admin discovery to the configured filer group * admin: refresh policy state after mutations and serialize policy controls * vector: drain active RPCs before closing replaced discovery connections * vector: respect request cancellation while waiting for discovery * vector: prefer another registered engine after a transport failure * admin: check all masters before reporting no registered vector engines * iam: honor attached vector policies without broadening legacy grants * vector: allow short-lived discovery clients to close their connections * vector: refresh cached master addresses for each discovery lookup * iam: enforce vector bucket tag conditions in attached policies * iam: authorize vector bucket creation with the requested tags * test: exercise vector IAM grants and tag denies on Go and Rust volumes * admin: reject authenticated vector mutations without an admin role * test: run vector IAM assertions before the shared engine is stopped * vector: add native mini startup, shell tools, and a runnable Compose example (#12522) * s3vectors: share vector discovery with trusted management clients * admin: show vector engine readiness and recent maintenance activity * admin: suggest S3 Vectors IAM actions from the supported operations * admin: manage vector indexes, policies, tags, encryption, and query data * admin: validate SSE-S3 support before creating encrypted vector buckets * admin: discover vector engines in their registration group * admin: probe vector engines concurrently with independent time budgets * admin: verify encryption capability on the remote S3 gateway * admin: defer discovery until after the HTTP server can start * admin: distinguish unknown vector health services from failed engines * admin: count workers that can execute vector index maintenance * admin: allow read-only users to query vectors and inspect tags * vector: scope gateway and admin discovery to the configured filer group * admin: refresh policy state after mutations and serialize policy controls * vector: drain active RPCs before closing replaced discovery connections * vector: respect request cancellation while waiting for discovery * vector: prefer another registered engine after a transport failure * admin: check all masters before reporting no registered vector engines * iam: honor attached vector policies without broadening legacy grants * vector: allow short-lived discovery clients to close their connections * vector: refresh cached master addresses for each discovery lookup * iam: enforce vector bucket tag conditions in attached policies * iam: authorize vector bucket creation with the requested tags * test: exercise vector IAM grants and tag denies on Go and Rust volumes * admin: reject authenticated vector mutations without an admin role * test: run vector IAM assertions before the shared engine is stopped * mini: optionally start and supervise the vector engine * mini: bootstrap vector buckets with existing Lance catalog helpers * shell: manage vector buckets, indexes, and vector data * docs: provide a runnable vector and maintenance worker Compose stack * shell: scope vector discovery to the session and close its connections * mini: honor the bind address for vector and metrics listeners * mini: reserve vector ports before assigning gRPC and compatibility listeners * mini: wait for vector storage setup and master registration * mini: create requested vector buckets without requiring the Lance HTTP listener * shell: refresh master addresses during vector discovery * mini: resolve sibling vector executables with platform extensions * vector: preserve uncertain creates and fix mini listener allocation (#12525) * vector: retain catalog records when index creation has an unknown outcome * mini: keep chosen listener ports reserved throughout allocation * mini: resolve hostname binds before starting vector listeners * vector: fix integration test imports and recover interrupted index creation (#12533) * shell: register vector operations from the command package * s3vectors: resume interrupted index initialization safely * mini: exclude disabled WebDAV from listener allocation * s3vectors: retain uncertain creates across refused retries * s3vectors: claim pending indexes before retrying initialization * s3vectors: compare resolved distance metrics during recovery * s3vectors: preserve metadata keys when retrying initialization * s3vectors: verify ownership before initializing new catalog records |
||
|
|
5abb18e678 |
helm: manage vector engines and provision vector buckets (#12518)
* helm: deploy vector engines with TLS, probes, and monitoring * helm: provision vector buckets through the S3 Vectors API * helm: bracket IPv6 pod addresses for vector listeners * helm: discover vector engines through the all-in-one master * helm: accept vector component NetworkPolicy overrides * helm: bind vector metrics to the pod address on both IP families * helm: provision vector buckets through embedded and all-in-one gateways * docs: use the full Helm name for external vector certificates * helm: render vector instance labels as strings * helm: allow vector bucket hooks to use restricted security contexts * helm: honor vector security context enable switches * helm: prevent duplicate vector monitoring labels * helm: reject metrics flags with the default pre-metrics vector image * test: skip vector chart compatibility checks when Helm is unavailable |
||
|
|
c1afb1b1cb |
ci: finish native vector checks and fix Windows installer tests (#12526)
* test: pass installer scripts through stdin on Windows * ci: allow cold Intel macOS vector builds to finish * ci: retain completed Rust builds when later platform checks fail |
||
|
|
89d3016d5e |
vector: publish native macOS and Windows engines and installer support (#12523)
* ci: build, test, and publish vector engines for macOS and Windows * installer: install native vector archives on macOS and Windows * ci: run desktop vector checks for shared Rust and protobuf changes * ci: publish vector releases from a separate job with write permissions * ci: build tagged vector releases without restoring caches * ci: propagate Windows bash path conversion failures |