15920 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
3 people	b7bfbaf9eb	helm: add namespaceOverride to deploy into another namespace (#12256) ... * helm: add namespaceOverride to deploy into another namespace When the chart is a subchart of an umbrella chart, every object lands in the umbrella's release namespace, with no way to give SeaweedFS its own. namespaceOverride, empty by default, now sets the namespace of every object through a new seaweedfs.namespace helper, which falls back to the release namespace. All uses of .Release.Namespace go through it, so the in-cluster addresses, certificate SANs, ClusterRoleBinding subjects, HTTPRoute backendRefs and lookups follow the override too. The self-signed Issuer, the bucket hook Job and the volume resize hook objects, which had no metadata.namespace, now set it. With namespaceOverride unset the default render is unchanged. Adds a helm_ci check that renders with an override and asserts nothing is left in the release namespace. * ci: cover the bucket hook Job in the all-in-one namespace test The all-in-one render set s3.createBuckets, which bucketHookEnabled ignores in allInOne mode, so the Job never rendered and the namespace check could not see it. Use allInOne.s3.createBuckets and assert the Job is present. * helm: fail on an invalid namespaceOverride instead of reshaping it Truncating and trimming could silently map an override onto a different existing namespace and deploy into it. Validate against the DNS-1123 namespace rules and fail the render instead. * helm: bound the master data volume name at 63 characters data-<namespace> can exceed the Kubernetes name limit when the release or override namespace is near 63 characters; truncate the composed name and trim a trailing hyphen like the other name helpers. * helm: quote the rendered namespace in YAML fields * helm: warn that changing namespaceOverride strands claims and data * helm: say namespaceOverride only moves namespaced objects * helm: retrigger the PR head sync (no changes) * helm: warn that secrets and hook resources do not follow namespaceOverride * helm: warn that hostPath data does not follow namespaceOverride --------- Co-authored-by: jeffWild <20067853+jeffWild@users.noreply.github.com> Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>	2026-10-30 11:24:36 -07:00
Chris LuandDevin	816f9356b4	volume.copy: reject only on file-count drift, warn on deleted-count (#12253) ... * volume.copy: reject only on file-count drift, warn on deleted-count Go warned and kept the copy on any count mismatch. The file count is exact on both sides of a stable source, so a mismatch means a genuinely bad copy and should reject; only the deleted count is a rebuild-time estimate that legitimately drifts. Match the Rust check_copy_counts, which warns solely on the counter that can drift. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.copy: stage the copy and validate before dropping the old replica The destination replica was deleted before the copy even started, so a copy that failed its checks — now including the file-count validation — left the server with no replica at all. The copy now writes under a copying/ staging dir the volume loader never scans, the file-size and count checks run on the staged .idx (LoadCompactNeedleMap gives the same counters the mount would), and only then is the existing replica deleted and the staged files renamed into place. A rejected or failed copy leaves the previous replica untouched, and the .note marker is no longer needed since partial staged files can never be loaded. * volume.copy: remove only the files a failed copy moved into place * volume.copy: keep the .note in-progress marker through the renames A crash between the staged-file renames left a loadable volume without its .vif (writable where the source was read-only) or a partial set the load scan could not find. Write the .note before the renames and land the .vif first so any partial state is discovered and swept, then drop the note before the mount. * volume.copy: serialize copies of the same volume Two copies of the same volume write the same staged copying/ file names and can pass the existing-volume check before the first one mounts. Fail a concurrent second copy as retryable, matching the Rust volume server's volume_copies_in_flight. * volume.copy: restore the replaced replica when the swap fails Once the staged copy validates, the outgoing replica's files are hard-linked into copying/ before DeleteVolume unlinks them. If a later rename or the mount then fails, the preserved files are renamed back and remounted, so a failed swap no longer leaves the node with no replica. The links are removed on success. * volume.copy: clear stale needle-map files around the swap A leftover .ldb/.sdx/.rdb at the destination base is trusted over the fresh .idx the mount loads, and a mount that fails leaves the backends it built. Clear them before the staged files land and again when a failed copy cleans up, gated on no live replica owning them. Also keep <base>.vif on cleanup while an .ecx index shares the basename. * volume.copy: skip the swap when the caller has gone A caller that left during the copy cannot use the result; bail before the swap deletes the replica it would replace. * volume.copy: discard a mounted copy when the result send fails A mounted copy was already announced to the master, so a later failure deletes it through the store to withdraw the announcement before the files go away, rather than unlinking files out from under a registered replica. * volume.copy: keep a vif-less copy discoverable on a split index dir With -dir.idx the staged .idx lands outside the data dir, so a copy whose source has no .vif leaves nothing the volume scan can find and the mount fails after the replica it replaced was already deleted. An empty .vif marks the basename; the loader treats it as absent and writes a real one on mount. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.copy: preserve the replaced replica on its own disk * volume.copy: restore the replaced replica's .vif past a kept incoming one * test: cover the rollback link/restore helpers * volume.copy: preserve the mounted replica's own files for swap rollback The swap backup linked files at the copy's target paths, which are wrong when the replica sits on another disk or keeps its .idx beside the .dat; a failed swap then restored nothing. Link the mounted volume's real paths and require every present file to link before the delete, since a partial backup cannot roll back. Re-check the store at swap time so a replica that mounted mid-copy is replaced rather than overwritten, and drop a .kept incoming .vif that occupies the path the replica's own .vif restores to. * volume.copy: clear a stale .idx a mount would prefer over the copy A load switches dirIdx to the data dir when an .idx sits beside the .dat, so a leftover <data>.idx shadows the fresh index the copy lands in the idx dir, and a surviving <idx>.idx fills in when the source had no index at all. Clearing only the derived backends still let a stale needle map reach the mount. * volume.copy: leave the shared copying/ dir for racing copies Copies of different volumes serialize per volume id but share the disk's copying/ dir; a finishing copy could rmdir it between another copy's MkdirAll and its first staged write, failing that copy with ENOENT. * volume.copy: fail replica preservation on stat errors other than not-found * volume.copy: refuse a mount while that volume is being copied A copy claims the volume id, then waits on the source with the replacement staged. A mount in that window loads the replica whose files the swap then clears; refuse it with Unavailable, retryable once the copy settles. Mirrors the same refusal on the Rust volume server. * volume.copy: refuse configure while that volume is being copied Configure unmounts and remounts; during a copy that already claimed the volume id the remount loads files the swap then unlinks. Refuse with Unavailable, retryable once the copy settles. Mirrors the Rust volume server's same refusal. * volume.copy: leave a still-standing replica's index on a rejected copy * volume.copy: preserve an unmounted replica's files for swap rollback * volume.copy: claim the in-flight id in mount and configure, not just check it * volume.copy: keep an EC-shared .vif when displacing an unmounted replica * volume.copy: restore replaced files over a surviving partial restoreReplacedVolumeFiles deleted a preserved link whenever the base name was still taken, on the assumption a landed file there belonged to the copy. When a moved-file removal fails, that left a surviving partial paired with the restored set's other files — mixed generations in one replica. A leftover at the basename now comes off first so the backup goes back, and the link is kept whenever it cannot go back. * volume.copy: link a repeated replaced path once so rollback cannot delete it * volume.copy: refuse allocation and staged adoption under the copy claim AllocateVolume and adoptStagedVolume still registered a volume while a copy of it held the destination: NewVolume loads the index before registration, and the copy then truncates or replaces files the live replica already opened, leaving it mounted on a stale index. Both paths now claim the volume id like VolumeMount and VolumeConfigure, so registration either wins the claim before the copy starts or sees it and refuses with Unavailable. * volume.copy: serialize reload loads with the copy claim A SIGHUP reload ran loadExistingVolume outside the copy's claim: its .note branch could unlink a copy's files mid-write, and its NewVolume+SetVolume could register a replica between the copy's recheck and its swap, leaving a live volume on a stale index. loadExistingVolume now consults a per-location hook the server installs: a vid whose claim is held is skipped for the next pass; otherwise the load takes the claim and its release drops it after registration, so whichever side claims the id first owns the files for the operation. * volume.copy: let a claim holder's own mount past the load guard VolumeMount, VolumeConfigure, VolumeCopy, and adoptStagedVolume all hold the per-volume claim while calling store.MountVolume, which reaches loadExistingVolume where the load guard saw the caller's own claim and skipped the disk: every mount failed with 'volume not found on disk'. Plumb claimHeld through MountVolume -> LoadVolume -> loadExistingVolume via the new MountVolumeClaimed entry point so an owner's own load proceeds while loads without the claim are still skipped. * volume.copy: keep the strict count check when only deletes drift Deletes append tombstone rows without moving the file count, so a copy whose source only deleted mid-flight still has an exactly comparable file count. Gate the strict check on file-count stability alone; the deleted-count drift is already a warning inside checkCopyCounts. * volume.copy: abort the swap when stale index files cannot be removed A surviving .ldb/.sdx/.idx is trusted over the copied .idx at mount, so a failed removal must stop the copy before the staged files land, not be swallowed. Mirrors the Rust side's remove_partial_files abort. * volume.copy: clear staged leftovers before starting a copy The staging names are fixed per volume, so a crashed copy can leave files under copying/; a retry whose new source lacks an optional file (.vif, or .dat on a remote volume) would rename the previous run's bytes into place. Clear them up front and refuse the copy when they cannot be removed. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-30 11:22:40 -07:00
5 people	80aad76986	rust volume: remove the redb index when a failed VolumeCopy cleans up (#12250) ... * rust volume: remove the redb index when a failed VolumeCopy cleans up A copy that mounts and then fails validation is unmounted, not deleted. With a redb needle map the mount has already built <idx base>.rdb, and cleanup_failed_copy removed .dat/.idx/.vif/.note but not the .rdb, so the index database outlived the copy. A later copy of the same volume to that server reuses it (RedbNeedleMap::try_reuse_rdb keeps an .rdb whose recorded idx size is not larger than the new .idx and replays only the tail), which is wrong once the source has been compacted in between. Move the partial-file removal into CopyDestination::remove_partial_files and include the .rdb, next to the .idx where Volume::file_name puts it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * rust volume: clear a stale index before a copy writes, and report cleanup failures A crashed copy, or one whose partial-file removal failed, leaves a stale .rdb/.sdx next to the .idx; the next copy's mount would reuse it via try_reuse_rdb's tail replay. remove_partial_files now also covers .sdx and returns the first non-NotFound error: callers that can still abort (the pre-copy clear) do so, while the post-failure cleanup warns so a surviving index is visible instead of silently poisoning the next copy. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: restore the pre-copy leftover clear A leftover index check-in accidentally gated remove_partial_files behind a constant false, so a stale .rdb/.sdx at the destination survived the copy that was meant to clear it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(volume): expect 409 when a read-only upload is refused x-seaweed-write-outcome: refused-before-append. The HTTP integration test still expected 500, so Rust unit tests failed on Linux and Windows. Deletes stay 500. * rust volume: keep a shared .vif, and clear a stale index beside the .dat remove_partial_files deleted <data>.vif even when a sealed .ecx still shares it, undoing the keep in Volume::destroy. It also cleared the index only under the index base, so a leftover .idx beside the .dat still made load trust that directory's .rdb. Skip .vif while a non-empty .ecx is present, and clear both bases when the index directory differs from the data directory. * rust volume: do not delete .dat files in the index directory The index directory can be another location's data directory. Clearing .dat and .note there removes that replica. Those files come off the data base only; .idx, .rdb, and .sdx are still cleared on both. * rust volume: serialize copies of a volume on one server A second VolumeCopy of the same volume could pass the existing-volume snapshot before the first copy mounts, then clear or overwrite the files the first copy still owns. Hold a per-volume claim for the whole copy, like ec_decodes_in_flight does for decodes, and fail a duplicate fast so the caller can retry. Mirrors the Go volumeCopiesInFlight guard. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: keep files when the mounted-copy delete is refused When a copy mounted and then failed, deleting through the store can be refused (e.g. the volume is compacting), leaving it registered. The cleanup then unlinked its files anyway, stranding a live replica. Keep the files while the volume is still in the store. Mirrors the Go copy cleanup. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: init volume_copies_in_flight in the http test state Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: refuse a mount while that volume is being copied volume_copy claims the vid, snapshots the store, then waits on the source. A VolumeMount in that window can load the replica, and the copy's pre-write cleanup then unlinks its files. Check the claim under the store write lock and return Unavailable. * rust volume: refuse configure while that volume is being copied volume_configure unmounts and remounts. A copy that already snapshotted the volume as absent can then unlink the files that remount loaded. Check the copy claim under the store write lock and return Unavailable. * rust volume: preserve files a copy displaces so a failed copy restores them * rust volume: keep an interrupted copy's .note with the files it marks displace_existing_files skipped the .note, so a crashed copy's leftovers were saved without the marker the load scan rejects them by, and remove_partial_files then deleted it. A retry that failed restored the partial set marker-free: after a restart the server could load an unfinished copy as a normal replica and serve wrong data. Displace the .note like every other file at the basename so the restore puts back the exact state the copy found. * rust volume: restore displaced files over a surviving partial restore_displaced_files dropped the backup whenever the basename was still occupied. When the partial cleanup could not remove every file the copy landed, an existing path just meant the removal failed — deleting the saved file destroyed the old replica and let the surviving partial pair with restored siblings of another generation. Clear the leftover first so the displaced file goes back, and keep the backup when it cannot: the operator can recover from copying/ rather than lose the replica outright. * rust volume: back up an EC-shared .vif the copy truncates in place A .vif a sealed .ecx shares was left out of the displaced set so the EC volume keeps it — but the incoming copy opens the same path with truncate, so a failed copy left the shared .vif overwritten by the rejected copy's settings with no way back. Go keeps the original via a hard link because its staged file swaps in by rename; truncating the shared inode makes a link useless here. Copy the content into copying/replaced_ like the other backups instead: the file stays in place for the EC volume during the copy, and the restore puts the original settings back on failure. * rust volume: serialize reload loads with the copy claim A SIGHUP reload scans every volume basename and sweeps .note-marked files as interrupted copies — including one a live VolumeCopy is still writing. Mirror Go's loadGuard: the copy-claim set is shared with each disk location, the scan skips a claimed vid outright and claims any other through its load, and reconcile_compact_states takes the same skip before rolling a leftover commit over the copy's files. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: hold the load claim through open_volumes The scan released each claim when its iteration ended, before the queued volume's index read ran in open_volumes. A VolumeCopy taking the vid in that gap could displace or remove files out from under Volume::new — the register the guard exists to prevent. Queued claims now live until the open pass returns; a second collection of an already-claimed vid merges into the entry instead of re-claiming. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: refuse allocation while that volume is being copied AllocateVolume registered a volume while a copy of it held the destination: create_volume loads the index before registration, and the copy then clears or rewrites files the live replica already opened, leaving it mounted on a stale index — the window VolumeMount and VolumeConfigure already refuse. Claim the volume id like the Go AllocateVolume does, so registration either wins the claim before the copy starts or sees it and refuses with Unavailable. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: refuse staged adoption while that volume is being copied The from_staged branch of volume_ec_shards_to_volume renames .copying files into place and mounts, all while a copy of the same vid could be rewriting the destination — the same registration window VolumeMount refuses. Hold the copy claim for the whole adoption, as Go's adoptStagedVolume does. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: drop a partial .note when its write aborts the copy A write_note failure after displace_existing_files restores the saved replica but leaves the empty or partial marker it just made; the next load scan then deletes the restored replica as an unfinished copy. Remove the marker before restoring, as the Go side's error defer does. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chris.lu@gmail.com>	2026-10-30 11:21:38 -07:00
3 people	43a312e309	rust volume: keep a copied replica on deleted-count drift, as Go does (#12251) ... * rust volume: keep a copied replica on deleted-count drift, as Go does check_copy_counts rejected a VolumeCopy whenever the target's deleted count differed from the source's. On a redb needle map that count is an estimate: metrics_from_idx rebuilds it at load through a bloom filter (0.1% false positives per entry, each counted as an overwrite), and the source holds the estimate from its own last load plus exact runtime updates. A source written since its load and a fresh load of the same files disagree by a few, so good copies were refused with "target deleted count [N+1] is not same as origin deleted count [N]". An index of unique keys 1..=128 already reports one deletion. Go fixed the same rejection in #11701 (issue #11678) by warning instead of failing. Do the same for the deleted count. The file count stays strict: it is exact on both sides (every non-tombstone .idx row is one file), and the .dat/.idx size checks still gate integrity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * test(volume): expect 409 when a read-only upload is refused x-seaweed-write-outcome: refused-before-append. The HTTP integration test still expected 500, so Rust unit tests failed on Linux and Windows. Deletes stay 500. * rust volume: keep the strict count check when only deletes drift Deletes append tombstone rows without moving the file count, so a copy whose source only deleted mid-flight still has an exactly comparable file count. Gate the strict check on file-count stability alone; the deleted-count drift is already a warning inside check_copy_counts. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com>	2026-10-30 11:20:33 -07:00
Chris LuandDevin	cc04c805a4	volume.copy: drop the needle-map index a failed copy leaves behind (#12252) ... * volume.copy: drop the needle-map index a failed copy leaves behind A VolumeCopy that mounts and then fails validation is only unloaded, not deleted, so a leveldb (.ldb) or sorted-file (.sdx) index built by the mount outlives the copy. The next copy of the same volume to this server then trusts the stale index (generateLevelDbFile replays only past its watermark), which is wrong once the source has compacted in between. Mirrors the Rust server's remove_partial_files. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.copy: clear a stale index before a copy writes A crashed copy, or one whose cleanup failed, leaves a stale .ldb/.sdx/.rdb next to the .idx that this copy's mount would trust (generateLevelDbFile replays only past the stored watermark, and a same-size .idx keeps every un-replayed offset). Abort on a removal error rather than risk a mount reusing a poisoned index. Mirrors the Rust server's pre-write remove_partial_files call. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.copy: keep a mounted replica when the final response send fails Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.copy: discard a mounted replica the caller abandoned A send failure after MountVolume means the caller is gone, but the replica was already announced to the master. Deleting its files under the live mount breaks the loaded index; leaving it mounted keeps an orphan the master routes to. Mirror the Rust copy cleanup: track the mount, and on error delete through the store (unmount + withdraw the announcement + remove files, keeping remote data), and check the stream context before mounting. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.copy: serialize copies of a volume on one server A second VolumeCopy of the same volume could pass the existing-volume check before the first copy mounts, then clear or overwrite the files the first copy still owns. Mirror ecDecodesInFlight: the claim covers the whole copy, and a duplicate fails fast so the caller can retry. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.copy: remove only files the copy owns, and every stale index Three cleanup gaps around which files a copy may remove: - The failure cleanup ran on any error, so an abort before the first write (a failed stale-index clear or a failed .note write) could delete a previous replica's files the copy had not touched. File removal is now gated on the copy having started writing. - A mount that finds an .idx beside the .dat builds the needle-map index under the data base, so leftovers there poison the next copy the same way the idx-dir ones do. Clear and clean up .idx/.ldb/ .sdx/.rdb under both bases when the dirs differ. - A regular volume and an EC volume for the same id share <base>.vif. Keep it while the EC index is on the disk, as the destroy path does. - When the discard of a mounted copy is refused (e.g. it is compacting) the volume stays registered; unlinking its files would strand a live replica. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.copy: do not unlink a previous replica on an early abort An abort before the first write now leaves the destination files alone, but three removal paths still leaked past that gate: a partially-written .note, and the post-copy defer's .idx/.vif at the data base. Remove the .note unconditionally (it only ever marks an in-flight copy) and gate the data-file removals in the outer defer on copyOwnsFiles too. * volume.copy: refuse a mount or configure while a copy is in flight A copy claims the volume id, then waits on the source before it clears or overwrites the destination files. A mount or configure landing in that window loads the replica whose index files the copy is about to remove. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.copy: abort when a replica registers during the source RPCs The existing-volume snapshot runs before the copy calls the source, so a mount that lands in between escapes it. Recheck registration just before the destructive clear, which can share the index directory with the mounted replica's bases. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.copy: keep files whenever a replica of the volume stays registered The cleanup already kept files after a refused delete; a replica a racing mount registered during the copy needs the same protection, since its index directory can coincide with the bases being cleaned. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.copy: skip the index cleanup when the copy never picked bases An early failure leaves the base names empty, so the removals would otherwise run against ".ldb"/".sdx"/".rdb" in the process directory. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.copy: serialize mount and configure registration under a per-volume guard The in-flight flag check in VolumeMount and VolumeConfigure was not atomic with volume registration, and the copy's registration recheck was not atomic with its stale-index clearing. A mount that passed the check just before the copy claimed the volume could register after the recheck and then lose its index files to the clearing. Replace the bare flag with a per-volume guard: the copy claims it for the copy's life, mounts and configures hold it across check plus registration, and the copy holds it across the recheck plus clearing. Whichever side takes the guard first wins, so a mount either registers before the recheck (and aborts the copy) or sees the claim and refuses. Guards persist in the map so a mutex handed to a waiter is never replaced mid-wait. * volume.copy: refuse allocation and staged adoption under the copy guard AllocateVolume and adoptStagedVolume still registered a volume while a copy of it held the destination: NewVolume loads the index before registration, and the copy then truncates or replaces files the live replica already opened, leaving it mounted on a stale index. Both paths now take the per-volume guard through their registration, so they either register before the copy's recheck (and make it abort) or see the claim and refuse with Unavailable. * volume.copy: serialize reload loads with the per-volume guard A SIGHUP reload ran loadExistingVolume outside the copy's guard: its .note branch could unlink a copy's files mid-write, and its NewVolume+SetVolume could register a replica between the copy's recheck and its file clearing, leaving a live volume on a stale index. loadExistingVolume now consults a per-location hook the server installs: a vid with a copy in flight (or whose guard the copy currently holds) is skipped for the next pass; otherwise the returned release keeps the guard held through registration, so the copy's recheck either sees the volume and aborts or clears first. * volume.copy: skip compaction-state reconcile while a copy owns the volume * volume.copy: let a guarded operation's own mount past the load guard VolumeMount, VolumeConfigure, adoptStagedVolume hold the volume guard locked, and VolumeCopy holds copying=true, while calling store.MountVolume. The mount reaches loadExistingVolume where the new load guard saw the caller's own hold (TryLock fails, or copying set) and skipped the disk: every guarded mount failed 'volume not found on disk' — a completed copy deleted its copied files, configure left the volume unmounted, staged adoption failed after renaming. Plumb ownLoad through MountVolume -> LoadVolume -> loadExistingVolume via the new MountVolumeClaimed entry point so the owner's own load proceeds while unrelated loads are still skipped. * volume.copy: fix own-load test call, assert unguarded mounts still skip Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.copy: refuse unmount and delete while a copy owns the volume --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-30 09:20:55 -07:00
Chris LuandDevin	ad6257ebbc	filer.backup: add -s3.flattenVersions to mirror the current S3 view of versioned buckets (#12255) ... * filer.backup: add -s3FlattenVersions to mirror the current S3 view Versioned buckets are stored as <key>.versions/v_<versionId> files plus a latest-version pointer on the .versions directory, so a backup mirror held internal version files and could never be restored through the S3 API. With -s3FlattenVersions the backup maps events on .versions directories onto the flattened object key: a new latest version uploads to <key>, a latest delete marker or a cleared pointer deletes <key> when -doDeleteFiles is set, and the suspended-versioning null object resolves to the regular path. Version files themselves are never mirrored. The initial snapshot seeds only the version the pointer names. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: serialize flattened writes per object key Same-path non-barrier dir updates run concurrently in the metadata processor, so two pointer updates for one object -- or a pointer update and a regular-path write to it -- could overlap and let the older upload finish last. Lock on the flattened key: versions events lock the object key, every other event its own path, via a striped mutex. * filer.backup: apply exclusions to the flattened object key File-name and path excludes ran against the .versions entry, so an excluded object was still copied flat and its deletion still removed the destination copy. Match the rules against the flattened key on both the apply and the versions-dir-removal paths. * filer.backup: re-resolve the pointer on version-file events A tag or attribute update on the current version touches the v_* file, not the .versions pointer, so dropping every version-file event left the mirror with stale tags until the next pointer flip. Re-resolve the pointer on each version-file event; noncurrent versions still stay out. * filer.backup: tolerate vanished versions dirs in flattened snapshot The flattened seed path returned source errors before the existing missing-source policy ran, so one version deleted mid-walk could force a full re-walk. Apply the same 404/superseded skip as the generic seed branch, and cover the review fixes with tests. * filer.backup: lock both flattened keys an event touches A rename or delete can retire one flattened key while writing another, so serialize on the old- and new-side keys together. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: refresh the flattened key only for the still-current child A v_* event on a non-pointed version cannot move the flattened key, so skip the re-upload; tag and attribute updates on the current version still flow through. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: let resolved-latest writes bypass the source staleness check A filer sink skips a CreateEntry whose source path no longer holds the entry, but a flattened write lands on a key whose source path is not the one it was read from, so the check dropped every flattened upload. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: reclaim destination chunks on flattened deletes Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: route regular-path events for versioned keys through the pointer A null-object write or delete at the regular path can coexist with a .versions pointer that names a live version; the pointer decides the flattened key, so deleting the null object must not remove it. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: make the checkpoint identity cover the flattening mode Resuming a checkpoint taken in the other mode would skip the flattened re-seed and leave mirrored version files behind; a mode change now starts a fresh snapshot. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover flattened regular-path routing and per-mode checkpoints Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: resolve both sides of a flattened rename * test: cover flattened renames with one versioned side * replication: let resolved-latest writes overwrite a newer destination The stale-destination skip trusted a newer destination mtime, but a resolved-latest write can legitimately install an older version over the copy of a since-deleted latest. Skip only on provably identical content (matching chunk provenance, or equal ETag with full byte coverage). * filer.backup: rescan .versions when the latest null object is gone Deleting ?versionId=null removes the regular-path object without touching the null-is-latest pointer, so the flattened key kept its stale copy. Fall back to the newest remaining version (or a flat delete), the same rescan the S3 read path applies. * replication: compare inline and remote content in the destination-holds checks A chunk-less entry keeps its bytes in Content and a remote-only entry in the object RemoteEntry points at, but the identical-content shortcuts only compared chunk lists and chunk-derived ETags. Two different inline files looked identical, so a flattened re-copy or a same-mtime rewrite kept the previous version's bytes. Compare Content bytes and remote object identity wherever the checks claim the destination already holds the reference's content. * filer.backup: paginate the null-latest .versions rescan A limit=0 listing is capped at DirListingLimit server-side, and old-format version names sort oldest-first, so a history longer than one page left the flattened copy on an older version after the null object was removed. Page through the directory like the S3 read path does. * replication: require matching writable metadata for resolved-latest skips A skip keeps the destination's stored metadata, so identical bytes were not enough: stale attributes or extended entries (e.g. a superseded version id) survived under equal content. Compare the fields a rewrite controls — attributes minus filer-assigned inode/crtime, with the store's atime/ctime zero-defaults applied — plus the extended map, in both the provenance and ETag fast paths. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: require the extended key to exist when comparing metadata * replication: ignore a stored rule TTL when the resolved write carries none * filer.backup: rename -s3FlattenVersions to -s3.flattenVersions Match the -s3.* flag naming convention used by the other s3 options. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-30 09:09:05 -07:00
Chris LuandDevin	1a62b1d09f	filer: buffer recently fetched chunks so parallel readers do not refetch (#12254) ... * s3sink: upload through a concurrent-safe ReaderAt so parts do not thrash s3manager splits a seekable body into parallel section readers, and every part's ReadAt landed on the single chunk buffer in ChunkStreamReader: parts at different offsets kept evicting each other's chunk and refetching it whole, which produced massive read amplification against the volume servers and collapsed throughput. Build the upload body from filer.ChunkReadAt instead, the same concurrent-safe reader the mount and query paths use: each part's range is served from a shared reader cache with parallel chunk fetches and range-reads for clipped views, so every byte is fetched once per pass. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: buffer recently fetched chunks for concurrent readers * Revert "s3sink: upload through a concurrent-safe ReaderAt so parts do not thrash" This reverts commit be1ec2d7c7. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-30 08:58:48 -07:00
Chris LuandDevin	8b0bd48755	seaweed-volume: pin the read-only write status to 409 in the test (#12244) ... * seaweed-volume: pin the read-only write status to 409 in the test PR #12236 moved read-only upload refusals from 500 to CONFLICT with the refused-before-append proof; the test still asserted Go's 500 for the write. Deletes on a read-only volume keep surfacing 500. * seaweed-volume: correct the read-only status comment in the test Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-30 08:55:53 -07:00
Chris Lu	a1f2886647	vector: accept Go-style single-dash long flags (#12247)	2026-10-30 03:44:25 -07:00
Chris Lu	452306776f	Delete test/s3vectors/README.md	2026-10-30 03:24:39 -07:00
Nguyễn Lộc Phúcandchrislusf	eba2cb926d	fix(s3tables): refuse a create that a namespace delete can lose (#12239) ... * s3tables: refuse a create that a namespace delete can lose DeleteNamespace checks the namespace for children and then deletes its directory recursively in a second filer call. The filer locks each path on its own, so a table created under the namespace between the two is not seen by the check and is removed by the delete, while both requests answer 200. The mirror case loses the table too: a create that passed its namespace check before the delete finished re-created the namespace directory through the filer's parent creation and recorded a table in a namespace that no longer exists. Each side now writes before it reads. DeleteNamespace claims the namespace record with a fresh token before it scans, deletes only while the record still holds that claim, and lifts its own claim when it stops. CreateTable, RegisterTable and CreateView re-read the namespace after writing their record and back out when it is claimed or gone: they remove what they wrote and answer 409 or 404 instead of reporting an entry that does not survive, or 500 when the record could not be removed and may remain. An entry the scan missed was written after the claim, so its create sees the claim; a create and a delete can both fail, but no longer both succeed. A create is also refused up front while a claim is held, but only after its authorization check, so a caller that may not create there does not learn of the delete. A create writes its record marked "create" and lifts the mark once the namespace is confirmed open. Until then GetTable, GetView, UpdateTable and a second create treat the record as absent, so no request is told about a table or view that the first create may still back out. * s3tables: keep a create's unfinished record out of reads and listings * s3tables: refuse catalog writes on a record an operation claimed * s3tables: answer 404 when a create's write path loses the namespace * s3tables: confirm the rename destination's namespace like a create * s3tables: cover unfinished-record reads, claimed-record writes, rename claim * s3tables: lift a leftover claim when a no-token delete clears the record * s3tables: answer 409 when a claimed record turns away a write UpdateView, TagResource, and UntagResource surfaced ErrConcurrentUpdate as a 500, so an expected conflict looked like a server fault to the caller. Map it to Conflict the way UpdateTable and the policy handlers already do. * s3tables: name the lost source when a rename finds it deleted * s3tables: report an entry deleted before its update as not found The filer answers an UpdateEntry on an entry removed since the lookup with codes.NotFound. mutateEntryExtended returned that error unchanged instead of filer_pb.ErrNotFound, so a rename whose source was deleted mid-claim answered 500 instead of NoSuchTable or NoSuchView. MemFiler now answers a missing entry the same way, so the rename tests for tables and views take the real filer's path. * s3tables: unpublish the destination when a rename's source clear fails * s3tables: refuse to delete a record another operation claimed * s3tables: release catalog claims on an uncanceled cleanup context * s3tables: answer 404 when a racing delete already removed the table * s3tables: keep the rename destination marked until the source clears * s3tables: guard the rename rollback against a recreated source and an unfinished destination * s3tables: drop the unfinished destination only once the source is restored * s3tables: re-resolve the purge target on the record a delete claims * s3tables: cover a delete that meets a moved metadata location * s3tables: answer 409 when a create's record changed under it * s3tables: adopt a record still marked by its unfinished create * s3tables: skip unfinished creates in the namespace empty check * s3tables: delete only the record the caller was authorized on DeleteView and DeleteTable check permissions on the record they read first, then read the name again to claim it. A record that took the name in between, such as a table or another owner's view created after a concurrent delete, was claimed and removed on the first record's authorization, and DeleteView never compared the caller's version token with it. The second read now repeats the first one's checks: a record of the other kind or an unfinished create answers 404, and a token that no longer matches answers 409. A delete without a token also compares the record's version token with the authorized one and answers 409 when they differ. * s3tables: re-check the parent after a create lifts its mark A create that passed checkParentOpen and paused before lifting its mark could still confirm while a namespace delete held the claim: the delete scans the still-marked record, skips it, and removes it under a create that already answered success. confirmParent now re-reads the same parent record after the mark clears and backs the confirmed record out when the namespace moved on. * s3tables: mark the rename destination with a claim of its own claimCatalogEntry adopted every create-marked record, including a rename's destination: a create landing after the source cleared could replace the destination before the rename published it, and with the source name reused the restore could leave the table on neither name. The destination now marks with the rename's own claim, which readers still treat as unpublished but a create must refuse, and the commit re-checks the destination namespace after the record goes live. The source keeps its rename claim until the commit lands so a namespace delete scanning the cleared record still refuses. * s3tables: compare the parent's createdAt, not its editable metadata A namespace update rewrites ExtendedKeyMetadata in place, so a create or rename whose second parent read landed across an UpdateNamespace saw a changed blob and reported the still-live namespace as gone: creates backed their record out and answered 404, renames undid their destination. createdAt survives the edit but changes on a delete+recreate, so it still identifies the record. * s3tables: retry lifting the rename's claim off the cleared source The release's only failure mode is a transient filer error — nothing else can write the cleared record — yet a single loss left the claim behind: the old name stayed refused to creates and the namespace counted the record as a child. Retry the release before warning. * s3tables: page the namespace scan before deleting it A page of skippable entries — bare directories or unfinished creates — ended the scan with hasChildren false, so a table on a later page was removed with the namespace. Keep listing with StartFromFileName until a child shows or the listing ends. --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>	2026-10-30 03:18:52 -07:00
Chris Lu	e80430db8e	test: add s3 integration tests for SQLite Cloud Backed Storage (#12243) ... * test: add s3 integration tests for SQLite Cloud Backed Storage (#12235) SQLite CBS stores a database as named objects in a bucket and relies on conditional PUT — put-if-absent for new blocks and ETag compare-and-swap on the manifest — so concurrent clients cannot corrupt the database. The suite runs a weed mini cluster (conditions are evaluated under the object's write lock on the owner filer via the routed-write path) and covers: - the CBS storage primitives: create/put/fetch/list/delete/destroy - If-None-Match: * put-if-absent, success and 412 rejection - If-Match ETag compare-and-swap, including stale-ETag rejection and the re-read/retry loop CBS clients run on 412, on plain and versioned buckets - racing put-if-absent and manifest CAS writers: exactly one commit wins, every update survives - conditional CompleteMultipartUpload, sequential and raced - an end-to-end bcv-style upload, download-and-verify, and checkpoint A matching job is added to the S3 mutation regression workflow. * test/s3: drop issue references from the sqlite cbs suite comments * test/s3: abort every losing upload after the conditional multipart race * test/s3: assert the winning writer's body after the conditional write races * test/s3: spell out the get-returned etag in the cbs suite * test/s3: assert the completed multipart ETag the cbs suite feeds back The conditional-completion checks pass completeOut.ETag into If-Match; if the response carried no ETag the check would silently run unconditional. * test/s3: drop the unused generation parameter from the cbs block body	2026-10-30 03:17:16 -07:00
Chris Lu	12dac68d66	seaweed-volume: replace deprecated AtomicU64::fetch_update with try_update	2026-10-30 01:37:42 -07:00
Chris Lu	56c081f90e	test/s3vectors: data-plane lifecycle integration test (#12242) ... * test/s3vectors: data-plane lifecycle integration test TestVectorDataPlaneLifecycle spawns a real weed-vector against the lance listener, mirrors the pytest cluster's flags, waits for the gateway to discover the registered engine, and walks the full lifecycle: create bucket and index, put/get/list vectors, a filtered query that excludes the nearer non-matching vector, delete vectors, index, and bucket. The test skips when no engine binary resolves (VECTOR_BINARY, ./weed-vector, PATH). * test/s3vectors: fail fast when the engine dies before listening * test/s3vectors: harden engine launch and tighten lifecycle assertions * test/s3vectors: wait on the engine's exit once through a broadcast channel	2026-10-30 01:34:02 -07:00
Chris Lu	ee7e4a64d2	s3vectors: record request metrics on the dispatch path (#12241) ... * s3vectors: record request metrics on the dispatch path The S3 API tracks every operation through S3RequestHistogram, S3RequestCounter, and S3InFlightRequestsGauge; the vectors dispatch rode the same listener with no instrumentation. The op name labels each series under s3vectors.<Operation>. * s3vectors: count auth rejects and keep bucketless metrics alive	2026-10-30 01:27:23 -07:00
Akshat Vikram Singhandchrislusf	e9f0d3ad2c	helm: set resources on the worker-lance container (#12240) ... * helm: set resources on the worker-lance container Signed-off-by: AkshatIIITD <akshatvikramsingh@gmail.com> * helm: Updated worker-deployment according to bot input Signed-off-by: AkshatIIITD <akshatvikramsingh@gmail.com> * helm: drop trailing whitespace in values.yaml --------- Signed-off-by: AkshatIIITD <akshatvikramsingh@gmail.com> Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>	2026-10-30 01:15:36 -07:00
Chris LuandDevin	72ad975803	s3vectors: bound GetVectors and DeleteVectors key lists (#12238) ... * s3vectors: bound GetVectors and DeleteVectors key lists Both forwarded req.Keys to the engine unbounded; AWS caps these at 500, the same window a PutVectors batch shares. * s3vectors: cap GetVectors at the AWS 100-key limit GetVectors allows at most 100 keys per request while DeleteVectors allows 500, so give vectorKeysBounds the per-call maximum instead of sharing the put-batch window. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: assert out-of-window key lists never reach the engine Record get/delete calls on the fake engine and cover both handlers rejecting empty and oversized lists with 400 ValidationException. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-30 00:50:55 -07:00
Chris Lu	a2496ad7ae	ci(vectors): split e2e into per-volume legs, build only needed binaries	2026-10-29 23:39:54 -07:00
Chris Lu	94e9b8a49a	fix(volume): return conflict and explicit proof for read-only upload refusals (#12236) ... * fix(volume): report read-only HTTP refusals in Rust * fix(volume): mirror HTTP write refusal proof in Go * test(volume): cover HTTP read-only refusal parity * test(volume): cover local refusal after remote append * fix(volume): honor decoded replica controls before retry proof * fix(operation): reassign proven HTTP conflict refusals * fix(operation): stop same-target retries on HTTP conflict * fix(volume): replace replica marker when forwarding requests * fix(java): gate conflict reassignment on refusal proof * fix(volume): preserve authentication query encoding in replication * fix(operation): stop unproven conflicts in forever retries	2026-10-29 23:33:56 -07:00
3 people	665659097d	helm: mount existingClaim data and logs volumes for filer and master (#12234) ... filer.data, filer.logs and master.logs accept type existingClaim and render the pod volume for it, but the volumeMount (and, for logs, -logdir) only covered hostPath, persistentVolumeClaim and emptyDir. With filer.data.type=existingClaim the claim was attached to the pod but never mounted, so the filer kept its metadata store on the container's own filesystem and lost it on every restart. admin and worker already include existingClaim in the same conditions. Adds a helm_ci check that renders both StatefulSets with existingClaim and asserts the mounts and -logdir. Co-authored-by: Pat White <5015863+patwhite@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 23:30:42 -07:00
3 people	9ee8604ff3	webdav: answer 204 to a PUT that replaces a file (#12233) ... * webdav: answer 204 to a PUT that replaces a file * webdav: decide the 204 from what OpenFile actually replaced A Stat before the write left a window where a delete or create in between made the status wrong, and it cost a second filer lookup. The OpenFile create path already knows whether it removed an existing entry, so it now marks the request context and the writer maps 201 to 204 only on a real replace. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test/webdav: cover OpenFile's own put-outcome flag for both create paths Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * webdav: take the put outcome from the create's OldEntry A stat-then-create still races: a second writer landing between the check and the write makes the reported status describe a state the write never saw. The filer decides create-vs-replace under its per-path lock and reports the previous entry in the response's metadata event, so flag the PUT replace from OldEntry instead of a pre-write stat. The in-place update keeps the replaced entry's crtime and inode and reuses the filer's own chunk cleanup, so the separate removeAll goes away, and an exclusive create now defers to the filer's atomic O_EXCL instead of a racy local check. * webdav: send no body with the 204 a replacing PUT answers --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 23:14:07 -07:00
Chris LuandDevin	cba409cae7	backup: carry index-only tombstones through incremental copies (#12231) ... * volume server: add idx_content to VolumeIncrementalCopyResponse A remote-tier delete is an index-only tombstone row with no .dat record, so it cannot reach an incremental copy through the .dat stream. Carrying the .idx tail verbatim lets the backup reproduce those rows at their exact positions. * backup: carry index-only tombstones through incremental copies A remote-tier delete leaves an offset-0 tombstone in .idx with no .dat record. The copy only streamed .dat bytes and regenerated .idx from them, so the tombstone never reached the backup: the deleted needle stayed live and every later incremental diverged and rebuilt again. backupCursor and ResumeByLastEntry now accept an index-only tombstone as a boundary, matched by key and size since there is no record to read a cookie from. VolumeIncrementalCopy appends the .idx tail after the .dat tail, and IncrementalBackup writes it at the index boundary, truncates, and loads the new rows — so tombstones land at the exact positions the replica has them. * seaweed-volume: mirror index-only tombstone handling in incremental copy resume_by_last_entry accepts an offset-0 tombstone boundary matched on key and size, skips its missing cookie check, and resumes .dat streaming at the next entry that has a record — index-only rows ride the new idx_content stream instead of reporting divergence. The server captures the .idx bound before the .dat one (a row is published only after its bytes) and streams the tail after the data. * backup: keep incremental copy bounds and streams consistent Review fixes: * Bound capture moved after the resume reads and reads .dat/.idx sizes together (Go FileStat, Rust file_stat under one lock), so every row resume saw is inside the index bound and each row's record is always inside the data bound. * A compaction commit changes CompactionRevision: the server now rejects the copy when the revision moved between resume, file opens, or the end of the stream, so a backup never mixes revisions. The .idx is opened up front to pin the inode. * An index tail that ends before the captured bound is an error, not a silent success. * IncrementalBackup rolls the .idx tail and .dat bytes back to the pre-copy boundary when the stream fails, so a partial row cannot wedge backupCursor and unapplied rows cannot pass for current ones. * The index tail is replayed through a small indexTailReceiver interface, covering the leveldb needle maps too, not just the in-memory map. * backup: bound the resume scan to the captured index size The resume and the stream bounds are now captured in one order: revision, then the .dat/.idx pair, then the scan. ResumeByLastEntry and resume_by_last_entry take the index bound and scan no further, so a row appended while the copy is being set up can never enter the index stream without its .dat bytes. The index-tail replay stays on the in-memory map only: the leveldb map's DoOffsetLoading writes to the .cpldb compaction database, not the live one, so it cannot serve a live tail replay; other maps keep regenerating the index from .dat as before. * seaweed-volume: guard incremental copy total against a located offset past .dat A corrupt index can place the resume offset beyond the captured .dat size; has_data already refuses the reader, but total still evaluates dat_size - s and underflows. * backup: bound the incremental .dat tail by index-covered records A record staged but not yet indexed can sit inside the captured .dat bound while its row stays outside the index bound — the fsync gap between staging and publishing is not covered by one stat. Streaming those bytes leaves them unindexed on the backup, and the next run appends the record again. Bound the .dat tail by the last record the streamed rows cover on both producers, and cover the bound walk with tests. * backup: cap the .dat stream at the captured bound and roll back replay failures sendFileContent could read past stopOffset on its last chunk, so a needle appended after FileStat arrived without its index row and the next copy appended at the wrong position. Each read is now limited to the remaining captured bytes. A failed truncateIndex or DoOffsetLoading after a successful stream left the index tail on disk unapplied, which a later backupCursor could mistake for applied rows; both now roll the tail back to the pre-copy boundary like the stream-error path. * backup: bound the incremental .dat tail by the maximum covered offset A key-sorted .idx ends on the highest-key row, which need not be the record physically last in .dat, so the last-row bound could cut off live records. Take the end of the highest-offset row inside the bound instead, and cover the case with a sorted-idx test in Go and Rust. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * backup: restore .dat and the needle map when index replay fails A failed DoOffsetLoading left the received .dat bytes in place and could leave partial mappings for rows the index rollback just removed, so the next copy appended the same records again or read stale entries. Truncate .dat back to the copy boundary and restore the touched mappings and metrics from a pre-replay snapshot. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * backup: cover no index rows with the superblock bound When no row inside the index bound names a record — an empty index or a bound of index-only tombstones — the covered end is the superblock, not the captured .dat size, so staged bytes are never streamed without a covering row. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * seaweed-volume: apply the covered bound even when the index streams no rows The bound was gated on idx_end > idx_start, so an empty captured index skipped it entirely and a staged .dat tail still shipped. Gate on the stream-index predicate instead, matching Go's streamIndex check. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * backup: propagate index coverage read errors * backup: rebuild the needle map from the truncated index on replay failure * seaweed-volume: propagate index coverage read errors --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 22:37:31 -07:00
3 people	0a4668fcfa	webdav: overwrite on MOVE when the Overwrite header is missing (#12232) ... * webdav: overwrite on MOVE when the Overwrite header is missing * webdav: defer a MOVE destination's delete into the rename x/net removes an existing destination before Rename, so a refused or failed move — a CanRename rejection, a vanished parent, a transient filer error — loses the target entirely. For an overwriting MOVE onto an existing destination, run the request against a filesystem that skips the destination RemoveAll and lets the filer's rename replace the target inside one transaction instead; a file vs. collection clash still takes the delete + rename path. Rename itself now tolerates an existing target, which the atomic filer rename already handles. * webdav: only default Overwrite when the header has no values An Overwrite header sent with an empty first value makes Header.Get return "", which the defaulting treated as absent and rewrote to T — overriding a client's explicit F in a later value. Check the value list instead so a present-but-empty header is left alone. * webdav: move a collection onto a non-empty one via delete + rename deferredDeleteFS skips the destination RemoveAll so the filer rename can replace the target in one transaction, but the filer only replaces an empty collection: moveSelfEntry deletes the target non-recursively and refuses ErrNonEmptyFolder. Detect a non-empty collection destination and take the depth-infinity delete + rename path moveFiles would have run. * webdav: park a clashing MOVE target aside until the rename lands A refused rename (cross-collection move, bucket target) after deleting the destination lost its contents; the target now renames to a hidden sibling first and is restored when the winning rename fails. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * webdav: restore a parked MOVE target under a detached context A client disconnect cancels the request context mid-move, which also canceled the restore and stranded the destination under its hidden sibling name; the restore and parked-entry removal now run under context.WithoutCancel with a bounded deadline. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * webdav: bound the parked MOVE target's restore after the move The detached restore deadline started before the source rename, so a long move could exhaust it; each detached context now starts where its work begins. A failed parked-entry removal retries briefly and its warning names the parked sibling for recovery. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * webdav: restore a parked MOVE target without replacing a new claimant A client can create the destination name while the old target is parked, so the restore rename must fail instead of replacing that new entry. * webdav: propagate RemoveAll's context to the filer delete The parked-target cleanup runs under a bounded detached context, but the delete RPC discarded it; passing ctx through makes the deadline effective. --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 22:24:43 -07:00
Chris Lu	74f727d987	test: move the s3vectors pytest suite under test/s3vectors (#12230) ... * test: move the s3vectors pytest suite under test/s3vectors The vectors API tests lived in test/vectors next to the new s3vectors integration home; the rust-vector-tests workflow's paths and working directory move with it. * gitignore: cover the binaries the s3vectors Makefile builds	2026-10-29 19:34:48 -07:00
Chris LuandDevin	665315d75e	test/s3vectors: integration test design and catalog-surface suite (#12229) ... * test/s3vectors: integration test design and catalog-surface suite The design doc lays out a two-tier suite: catalog/policy ops that need no vector engine run against a real weed server over HTTP, while the data plane skips itself when no engine is reachable. The first test brings up the server and exercises the bucket/policy lifecycle, encryption-config round-trip, and pre-engine validation boundaries. * test/s3vectors: sign API calls so requests reach the vectors router The gateway selects the vectors router off the SigV4 credential scope — unsigned requests with only X-Amz-Target fall through to the plain S3 router, so every call in the suite failed before dispatch. Sign each request with the s3vectors service name and point the spawned server at a written identity file so the signature is actually verified. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test/s3vectors: bound the spawned server's ports, listeners, and clients weed server derives each gRPC listener as HTTP+10000, so ephemeral allocation could hand back a port whose pair is invalid or taken; the -s3 defaults also bind the Iceberg and Lance listeners. Allocate the HTTP/gRPC pairs together, pin -ip/-ip.bind, disable the extra listeners, and give the readiness probe and API client deadlines. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test/s3vectors: verify policy round-trip and gate the engine skip on 503 The stored policy check accepted any JSON body, and a delete that left the document behind could not be caught — compare the returned document verbatim and assert a follow-up Get returns 404. The engine probe never created its bucket, so a healthy engine still produced a skip; create it first, skip only on a confirmed 503, and fail on any other error, then exercise the catalog-side index ops when an engine answers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 19:29:33 -07:00
Chris Lu	ef597cdad3	test: use an s3vectors action in the vector policy round-trip (#12226)	2026-10-29 19:13:17 -07:00
Ping QiuandClaude Opus 5.5	2a1afadcd0	s3api: return NoSuchUpload when aborting an upload that was already completed (#12227) ... s3: AbortMultipartUpload answers NoSuchUpload for an upload that no longer exists Complete and abort both remove .uploads/<id>. abortMultipartUpload answered 204 whenever that record was missing, so aborting an upload that had already completed reported success while the object stayed in place, and a client could read it as the object being discarded. AWS returns 404 NoSuchUpload ("The specified multipart upload does not exist") for that case, and ListParts here already does. Answer ErrNoSuchUpload when the record is not found. checkUploadStillOpen reuses abort for cleanup after the upload is gone; NoSuchUpload is not a failed cleanup there. The leftover-directory path from #11385 is unchanged: a .uploads/<id> that outlived its completed object is still removed metadata-only and answered. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 19:02:23 -07:00
Denis Kieftanduser.email	a2f595b207	backup: don't Close a volume after Destroy (#12225) ... Co-authored-by: user.email <chrislusf@users.noreply.github.com>	2026-10-29 19:00:15 -07:00
github-actions[bot]	cd3e2c8b2c	docs: regenerate star history chart	2026-10-30 01:57:47 +00:00
Chris Lu	349576b63e	backup: resume incremental copies by index position instead of appendAtNs (#12228) ... * volume server: add last-entry cursor fields to VolumeIncrementalCopy * backup: resume incremental copies by index position instead of appendAtNs appendAtNs is stamped by whichever replica received the write, so the same needle carries a different timestamp on every replica. A backup that resumes from a different replica than last time can append the last needle again or skip needles written inside the clock gap. Send the local copy's index entry count and last entry identity (key, size, cookie) with VolumeIncrementalCopy. The replica verifies the same record sits at that position and streams from the next index entry, so the resume no longer depends on any replica's clock. When the boundary record does not match, or the local copy is ahead, the replica returns FailedPrecondition and the backup rebuilds the local copy from scratch. Old servers ignore the new fields and keep the timestamp behavior. * rust volume server: mirror positional resume in VolumeIncrementalCopy * volume server: add entries_hash prefix checksum to VolumeIncrementalCopy * backup: verify the whole index prefix, not just the boundary entry * rust volume server: verify the index prefix hash in resume_by_last_entry * backup: keep the source's compaction revision on rebuilt copies * backup: diverge on a zero-offset boundary entry A remote-tier tombstone lives in the .idx with no .dat record, so readNeedleAt(0) parses the superblock and its error bypassed the ErrIncrementalCopyDiverged mapping; the backup got a generic failure instead of FailedPrecondition and never rebuilt. * rust volume server: mirror zero-offset boundary divergence * rust volume server: gate the empty-dat early return on entry_count With a positional resume the Go handler still reaches ResumeByLastEntry, so an empty replica reports divergence; the Rust early return answered a clean empty stream and weed backup kept a stale copy. * backup: hash the index entry's size field, not trailing offset bytes In a 5BytesOffset build the offset occupies bytes 8-12 and the size starts at byte 13, so the hash folded in one byte of the offset. Matching records at different offsets then hashed differently and a valid backup was reported diverged and rebuilt. * backup: read the index in whole-entry chunks for the prefix hash When entries are 17 bytes a 1 MiB read buffer does not end on an entry boundary: the partial tail entry was skipped by the loop but still counted as consumed, so every later chunk was hashed at a misaligned position. * rust volume server: hash the index entry's size field after the offset Mirrors the Go hashIndexEntries change: with the 5bytes feature the offset occupies bytes 8-12, so starting the size slice at byte 12 folded one offset byte into the hash and matching records at different offsets compared diverged. * rust volume server: read the index in whole-entry chunks for the prefix hash Mirrors the Go change. With 17-byte entries a 1 MiB buffer does not end on an entry boundary, so slicing the partial tail entry panicked once a backup held 61681 entries. * rust volume server: default the new incremental copy request fields in tests	2026-10-29 18:56:25 -07:00
Chris LuandDevin	d0441bbdbb	s3vectors: validate index ARNs and expand live integration coverage (#12222) ... * test: align vector bucket policy coverage with vector actions * s3vectors: validate ARN service and resource shape before resolving indexes * test: exercise signed vector bucket policies and catalog lifecycle * ci: require the expanded vector integration matrix and all gateway tests * test: reject index ARNs in bucket-level API requests * s3vectors: scope ARN resolution to the gateway namespace * s3vectors: reject name fields that conflict with resolved ARNs bucketARNs and resolveIndexARNs already reject an indexArn that names a different bucket than vectorBucketArn, but a vectorBucketName or indexName disagreeing with the ARNs was silently dropped. Validate the plain names against the resolved ARN values the same way. * test: cover conflicting vector names and ARNs in live requests --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 17:57:11 -07:00
Chris Lu	cc5912b8b8	s3vectors: e2e conformance for vector bucket policy evaluation (#12221) ... * s3vectors: reject empty boolean-operator lists and sub-objects in filters An empty $and/$or list or a nested empty object rendered '()'' — invalid SQL surfaced as an engine error instead of a clean ValidationException. * s3vectors: e2e conformance for vector bucket policy evaluation The handler-level test covers statements; this runs the whole stack a client sees — HTTP dispatch, authentication, policy evaluation, catalog, engine — and walks the full lifecycle: a non-owner is denied without a grant, an Allow on the bucket ARN grants every index under it, a Deny vetoes an Allow on the same action, and deleting the policy restores owner-only access. * s3vectors: prove deny precedence and grant removal in policy e2e * s3vectors: reject field operator objects with no operators * test: use an s3vectors action in the vector policy round-trip	2026-10-29 16:19:22 -07:00
Chris Lu	9d5929a73f	s3vectors: reject empty boolean-operator lists and sub-objects in filters (#12220) ... * s3vectors: reject empty boolean-operator lists and sub-objects in filters An empty $and/$or list or a nested empty object rendered '()'' — invalid SQL surfaced as an engine error instead of a clean ValidationException. * s3vectors: reject field operator objects with no operators	2026-10-29 16:16:34 -07:00
Ping QiuandClaude Opus 5.5	3f954e34dc	s3api: label bucket-level requests correctly in the access log (#12224) ... * s3api: label bucket-level requests correctly in the access log getResourceType treated a request as bucket-level only when the object key was "/". NormalizeObjectKey strips the leading slash, so bucket routes arrive with an empty key and every bucket operation was logged as an object operation: ListObjectsV2 as REST.GET.OBJECT, DeleteObjects as REST.POST.OBJECT, and the lifecycle, ACL and policy calls likewise. Treat an empty object key as bucket-level too. The new test covers five bucket-level and two object-level request shapes; it fails without the fix. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * s3api: label only POST ?delete as a batch delete in the access log With bucket-level requests now reaching the bucket branch, any method on /bucket?delete was labelled BATCH.DELETE.OBJECT. Only the POST route is DeleteObjects; label other methods as the plain bucket operation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-29 16:10:36 -07:00
Chris Lu	fd493f2547	mount: refuse to attach opens to a draining file handle (#12219) ... * mount: refuse to attach opens to a draining file handle With the writeback cache, releasing the last open of a file leaves the shared FileHandle in the map at counter zero while an async flush drains it; the worker removes the map entries once the flush commits. AcquireHandle waits for a pending flush and then attaches, but the two steps are not atomic: a release can slip in between, drop the counter to zero, and queue the drain while the opener goes on to mint a new file-handle id on the draining handle. When the drain completes, RemoveFileHandle deletes every fh id for the inode -- including the one just minted. The kernel still holds that id, so its next FUSE_READ gets ENOENT from GetHandle; the failed read leaves the folio not uptodate and the kernel surfaces EIO to userspace. This was the flaky TestWritebackCacheConcurrentMixedOps failure. AcquireFileHandle now refuses to attach when the shared handle has reached a zero counter and reports it as draining. AcquireHandle and the Create path retry the whole wait-resolve-attach sequence; the next pass blocks on the pending flush (flag registration and the counter drop share pendingAsyncFlushMu) and builds a fresh handle from the post-flush entry. The DLM-failure release in AcquireHandle also goes through ReleaseHandle so it cannot strand a counter-zero handle outside the drain bookkeeping. Removes the refreshEntryForRead retry from #12206, which worked around the symptom by refetching the entry after the fh id was already gone. Test: TestAcquireFileHandleRefusesDrainingHandle covers the refusal and fresh-handle path; TestWritebackCacheConcurrentMixedOps and an amplified concurrent-overwrite repro run clean. * mount: keep refresh-on-error retry for stale-chunk reads	2026-10-29 16:06:57 -07:00
Chris Lu	ac7b4f4cf6	s3vectors: bound ListVectorBuckets and ListIndexes maxResults (#12218) ... * s3vectors: bound ListVectorBuckets and ListIndexes maxResults Both pass maxResults to the catalog unbounded; AWS caps these list pages at 100, the same way ListVectors is bounded and topK is bounded. * s3vectors: raise ListVectorBuckets/ListIndexes maxResults cap to 500 AWS documents a valid range of 1-500 for both list APIs; the shared limit of 100 rejected legitimate page sizes. * s3vectors: reject explicit maxResults of zero on list calls AWS requires maxResults >= 1, but a plain int field cannot tell an omitted field from an explicit zero. Making MaxResults a pointer keeps the catalog default for omitted fields while a supplied zero fails validation. * s3vectors: test max ListIndexes page against an existing bucket	2026-10-29 15:02:30 -07:00
Chris LuandDevin	f122f313e1	s3tables: keep the admin bypass off an identified caller's empty action list (#12217) ... Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 13:41:26 -07:00
Chris Lu	229ad84e2d	s3tables: reject s3vectors resources in every ARN partition (#12215) ... * s3tables: reject s3vectors resources in bucket policies An arn:aws:s3vectors: Resource is as dead as an s3vectors action: the evaluator compares against the generated s3tables ARN, so the rule the writer believes enforced never matches. * s3tables: reject s3vectors resources in every ARN partition * s3tables: test wildcard-partition and mixed-list s3vectors resources	2026-10-29 13:38:53 -07:00
Chris LuandDevin	8a2fec8610	filer.sync: skip superseded replays before fetching any chunks (#12212) ... * filer.sync: skip superseded replays before fetching any chunks On a directory under heavy overwrite, most entries the sync replays are already superseded on the source. The sink only learned that after chunk reads failed — or worse, after fetching and uploading bytes a later event replaces — so the backlog drained slowly and the log filled with 404s. CreateEntry and UpdateEntry now check sourceMovedOn before touching a chunk: one entry lookup skips a stale replay entirely. A mid-replication supersession is still caught by the existing retry gates. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: bound the source liveness probes with a timeout The supersession checks added a source entry lookup to the replication path, and every lookup ran on context.Background() with no deadline. A source that accepts the call but whose metadata store stalls would hold the sync forever — before a single chunk fetch, even when every chunk is still readable. Bound the probe at 30s; an unanswered lookup proves nothing, so replication proceeds and the existing retry gates decide. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: sync the live version a superseded replay points at Skipping a stale replay assumed a later event always re-carries the current content. It does not when the superseding write was itself replicated in from another cluster: those events are dropped before they reach a filer sink, so the destination could be left missing the file. When the source reports a live entry newer than the replay, commit that version through the normal path instead — equivalent to applying the later event, with no dependence on its delivery. A gone source entry still skips, since the destination already matches the source. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: reconcile moved entry content after a native rename AtomicRenameEntry relocates whatever version the destination held. If the preceding update replay was skipped because the source had already moved to the new name, the moved entry keeps the old bytes and — with no further writes — stays stale forever. MoveEntry now runs the same reconciliation as an in-place update against the rename's snapshot, so the moved entry picks up pending content changes. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: repair from the live snapshot a superseded replay resolves to An incremental diff against the replayed oldEntry can resolve manifests the overwrite already vacuumed, which fails every retry of the event. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 13:37:27 -07:00
Chris LuandDevin	98fe70a247	s3vectors: keep bucket policies in vector form and evaluate them in the gateway (#12214) ... * s3vectors: translate vector bucket policies to the catalog form The document is stored verbatim on the shared table bucket and evaluated against s3tables actions and ARNs, so a policy written for the vector API was a dead rule. Actions now map to the catalog operation the request authorizes against, and an arn:aws:s3vectors: resource becomes the bucket/.../table/vector/... ARN the index's catalog entry answers to. * s3tables: expose PrincipalFor and PolicyContext for façade evaluation A façade authorizing against its own action namespace needs the same identity resolution and evaluation context the catalog uses internally. * s3vectors: keep bucket policies in vector form and evaluate them in the gateway The earlier approach translated the document to catalog actions and ARNs, which lost the caller's original policy for GetVectorBucketPolicy and could not cover data-plane calls that never touch the catalog. The document now stays in s3vectors form under a dedicated bucket attribute and every handler evaluates it through authorizeVector against the action and ARN the request names; a dedicated key also keeps a vector grant from ever authorizing a catalog call. * s3vectors: close policy-mutation races and keep vector grants visible - Authorize Put/DeleteVectorBucketPolicy inside the policy slot's compare-and-swap, so a grant revoked between reads cannot authorize the pending write. - Assert the policy attributes stay absent in the update precondition, so a concurrent policy or tag write trips the retry instead of being silently reverted. - Migrate a raw vector document out of the shared policy slot on replace or delete, keeping its catalog rules. - Pass the unresolved identity to PolicyContext so open-by-default deployments still work. - Filter ListVectorBuckets on the vector policy itself, so buckets a caller was granted under s3vectors actions stay listed. * s3vectors: reuse SnapshotExtended for the vector-policy CAS precondition * s3vectors: skip buckets that vanish mid-list instead of failing the page * s3vectors: reject non-vector bucket formats inside authorizeVector * s3vectors: check bucket format after authorization in authorizeVector An unauthorized caller probing an ICEBERG bucket got NoSuchBucket,\nleaking that the bucket exists. The format check now runs after the\npermission check so denied callers always see AccessDenied. * s3vectors: cover legacy shared-slot policy removal on delete * s3tables: resolve the forwarded identity name like getAccountID in PrincipalFor A synthesized manager request resolves its caller through the identity- name branch of getAccountID, which normalizes the name (ARNs collapse to their trailing segment, whitespace trims) before falling back to the handler's account ID. PrincipalFor returned the raw name, so ARN-shaped identities never matched their own grants. * s3tables: assert the vector policy key in update preconditions s3vectors-policy is stored on table bucket entries but was missing from s3tablesExtendedKeys, so a catalog update that read the entry before the first vector policy was saved could overwrite it without tripping the compare-and-swap. Asserting the key makes such a race fail instead of silently erasing the policy. * s3vectors: reject rules the evaluator cannot honor in vector bucket policies The shared Statement shape has no negated fields, the vector evaluation supplies no condition keys, and a document that fails the typed decode is discarded wholesale at evaluation — including denies the writer believed active. Reject NotPrincipal, Condition, foreign-service actions and resource ARNs, and require the typed PolicyDocument decode. Also cover that an admin-named identity still reaches its own buckets. * s3vectors: read vector policy fields the way the evaluator decodes them Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: match ARN policy principals through caller normalization Caller principals are normalized ids, so a statement naming a full IAM ARN could never match: an ARN-scoped deny was silently dead and an ARN-scoped allow unreachable. Compare arn: specs against the caller under the same normalizePrincipalID the ownership path uses, while an ARN caller keeps verbatim wildcard matching. * s3tables: grant the admin bypass on the caller's actions, not the name A principal that only normalizes to the admin id skipped every policy deny. When the request carries an identity, the bypass now requires the identity's own admin action — the same gate getAccountID applies — while name-only transports with no action list keep it. PolicyContext takes the request so it can carry IdentityActions, and the vector context helpers route through the shared identity keys so the resolved name and actions agree. * s3tables: match ARN policy principals only for deny statements Normalization drops the ARN's account and path, so an allow matched on the trailing name could grant a same-named caller the statement never verified. Denies keep the normalized match so the rule cannot be ducked through the same collapse. * s3tables: scope service wildcards to their own service s3tables:* counted as blanket admin inside the shared permission check, so an identity holding only catalog grants administered s3vectors operations too. hasAdminActionFor now requires the wildcard to name the operation's service; the global admin markers and s3:* stay service-agnostic. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 13:35:34 -07:00
Chris Lu	4ef123d764	s3vectors: fix shared API behavior and test Go/Rust volume parity (#12216) ... * s3vectors: reuse shared authenticated identity context * s3vectors: encode empty results as arrays * test: cover S3 Vectors with Go and Rust volume servers * ci: require both volume implementations in vector integration tests * s3vectors: keep identityOf as a shim over the shared identity context Concurrent branches add identityOf call sites; deleting the helper would break the build when they merge in either order. Route it through GetIdentityNameFromContext so every caller resolves the same identity. * test: wait for the vector engine to register before running The gateway resolves weed-vector lazily through the master cluster list, so a listening port does not mean the engine is reachable yet; the first vector write or query could race KeepConnected and fail. Restore the cluster.ps registration wait the previous harness had, and ignore the pytest cache directory the suite creates. * test: wait for master registration across OSS and enterprise Use the exact engine registration event after AddClusterNode completes. The current enterprise cluster.ps does not list vector nodes, so waiting on its output prevents the shared fixture from starting there. * s3vectors: keep contextWithIdentity as a shim alongside identityOf In-flight callers still write the caller name through it; route the write through SetIdentityNameInContext so both helpers agree on the shared key whichever lands first.	2026-10-29 13:35:05 -07:00
Chris Lu	88565df885	s3tables: reject s3vectors resources in bucket policies (#12213) ... An arn:aws:s3vectors: Resource is as dead as an s3vectors action: the evaluator compares against the generated s3tables ARN, so the rule the writer believes enforced never matches.	2026-10-29 11:59:07 -07:00
Chris Lu	15066c8a9f	vector: implement drop_index and have DeleteIndex drop the data (#12204) ... * vector: implement drop_index in the engine The RPC was a stub; it now resolves the table's vended location and removes every object under it through the same credential path open uses. A table the catalog can no longer describe is already dropped, so the call stays idempotent. * s3vectors: drop index data after the catalog delete DeleteTable carries the delete authorization, so it runs first; the engine DropIndex then removes the data under the index's location. NotFound and Unimplemented are tolerated so an index whose data went away with the catalog entry, or an older engine, still answers 200. * s3vectors: let the catalog delete carry the index data removal A vector index is colocated: the catalog entry sits at the dataset root, so DeleteTable's recursive delete already removes entry and data. The engine DropIndex call afterwards could only ever describe a missing or recreated record — a no-op in the common case, and a delete of a replacement index's files when the name was reused in between.	2026-10-29 11:57:23 -07:00
Chris Lu	2dbf9c9108	s3: evaluate all conditional write headers under the object lock (#12211) ... * filer_pb,filer: per-mutation preconditions on ObjectTransaction A transaction's request-level condition evaluates one entry before any mutation runs, so a commit guarding two paths — the object it publishes and the upload directory it assembles — could not be expressed. Give each ObjectMutation an optional condition (evaluated against condition_key, else the mutation's target, under the same lock just before the mutation applies) and report which mutation failed through failed_mutation on the response, so callers can tell a mutation guard's rejection apart from the request-level precondition's. * s3: evaluate multipart completion conditions under the object lock CompleteMultipartUpload checked conditional headers gateway-side before committing, and the routed commit transaction guarded only the upload directory — nothing made If-None-Match: * atomic with the publish, so concurrent conditional completions of the same absent key could all report success. The request's reducible write condition (If-None-Match/If-Match, star or single strong ETag) now rides the commit as the transaction's request condition on the owner filer: on the object path for unversioned buckets, falling back to the .versions directory for suspended and enabled versioning. The upload-exists guard moves onto the PUT mutation, keeping its check-before-publish ordering. Conditions the reducer cannot express stay on the distributed write lock, where check and commit already serialize. * filer,s3: resolve the latest version first for versioned conditional writes * filer: keep directory entries holding content out of the absent-slot rule * s3: keep a gateway-side upload-exists check for filers without mutation conditions * filer: honor null-version-is-latest resolving prefer-alt conditions * drop scratch multipart-race repro programs * s3: note the write lock does not cover routed writes on non-reducible conditions * filer: mirror the gateway's remote-etag precedence in condition evaluation storedEntryETag only knew the extended attribute and the chunk fallback, so a synced remote entry compared chunk-derived ETags while the gateway compares the remote ETag. Follow the same precedence: extended attribute, remote ETag while the entry still holds the remote bytes, Md5, then the chunk hash — and no ETag for an entry holding none. * s3: reduce all conditional write headers into owner-lock clauses Time-based headers, combined conditions, ETag lists, and weak ETags stayed on the distributed write lock because buildWriteCondition could not express them, leaving their gateway check racing owner-routed writes. Reduce every parseable header form to WriteCondition clauses — multiple clauses compose AND, ETag candidates carry the raw list for the filer's canonicalization, and RFC 7232 precedence keeps If-Unmodified-Since hidden by If-Match and If-Modified-Since hidden by If-None-Match — so the owner filer evaluates them under the same object-path lock as the commit. Only unparseable headers still fall back, where the lock path reports the same parse error.	2026-10-29 11:49:48 -07:00
Chris Lu	2737315e9d	s3: evaluate multipart completion conditions under the object lock (#12209) ... * filer_pb,filer: per-mutation preconditions on ObjectTransaction A transaction's request-level condition evaluates one entry before any mutation runs, so a commit guarding two paths — the object it publishes and the upload directory it assembles — could not be expressed. Give each ObjectMutation an optional condition (evaluated against condition_key, else the mutation's target, under the same lock just before the mutation applies) and report which mutation failed through failed_mutation on the response, so callers can tell a mutation guard's rejection apart from the request-level precondition's. * s3: evaluate multipart completion conditions under the object lock CompleteMultipartUpload checked conditional headers gateway-side before committing, and the routed commit transaction guarded only the upload directory — nothing made If-None-Match: * atomic with the publish, so concurrent conditional completions of the same absent key could all report success. The request's reducible write condition (If-None-Match/If-Match, star or single strong ETag) now rides the commit as the transaction's request condition on the owner filer: on the object path for unversioned buckets, falling back to the .versions directory for suspended and enabled versioning. The upload-exists guard moves onto the PUT mutation, keeping its check-before-publish ordering. Conditions the reducer cannot express stay on the distributed write lock, where check and commit already serialize. * filer,s3: resolve the latest version first for versioned conditional writes * filer: keep directory entries holding content out of the absent-slot rule * s3: keep a gateway-side upload-exists check for filers without mutation conditions * filer: honor null-version-is-latest resolving prefer-alt conditions * drop scratch multipart-race repro programs * s3: note the write lock does not cover routed writes on non-reducible conditions	2026-10-29 11:46:25 -07:00
3 people	3955e7218f	mount: keep the shared directory build out of the requester's context (#12206) ... * mount: keep the shared directory build out of the requester's context The shared directory build behind EnsureVisited is a singleflight call: the first requester for a path owns the build, and every later requester for the same path waits on that same build. The build ran on the owner's request context, so a FUSE request that returned -- or that the kernel interrupted -- canceled the build for everybody waiting on it. A waiter whose own context was still alive then received somebody else's cancellation. weed/mount/weedfs_dir_read.go classified a canceled read as EINTR only when the local context was done, so such a waiter fell through to fuse.EIO: the kernel handed EIO to readdirent, kubelet's applyFSGroup failed, and the volume never mounted. Build the shared directory on a context that keeps the request's values but not its cancellation, and report a canceled read as EINTR even when the local context is still alive. Waiters still abandon on their own context, so a canceled request stops waiting immediately. This follows up #11717, which made the wait abandonable and mapped the local cancellation to EINTR, but left the shared build itself on the owner's context. The surrounding build lifecycle calls already use context.Background() for the same reason. Test: TestEnsureVisitedSharedBuildSurvivesOwnerRequestCancel reproduces the cancellation with a fake filer and fails without this change. * mount: classify gRPC-canceled directory listings as EINTR * mount: log only real directory read failures, not interruptions * mount: keep shared section refresh out of the requester's context * mount: let section-refresh waiters leave on their own context The shared refresh flight now outlives the requester that started it, but callers still blocked in singleflight.Do until it finished — an interrupted directory read could not stop waiting, and the caller's own deadline never applied to the wait. DoChan + select on the caller's context restores early abandon while the detached refresh still completes for the remaining waiters. * ci: re-run FUSE integration after a flaky writeback-cache failure * mount: refresh a stale entry once before reporting a read error A read resolves chunk views from the handle's entry. A concurrent overwrite commits a new entry whose replaced file ids the filer queues for deletion; the open handle only learns the new chunk list when the metadata invalidation lands. A fetch that lands between the volume delete and the invalidation names data that is already gone and the read reports EIO. On a chunk read error, wait out any in-flight async flush, re-read the entry from the filer, install it when it outranks the handle, and retry once. An unchanged version means the failure is genuine. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: fence the read-error entry refresh by file identity Two guards on refreshEntryForRead's install path: the unlocked dirtyMetadata read raced with Write under the handle lock — drop it, installAckedEntry already checks under that lock, and it now fences isDeleted handles the same way. And a saved path can name a different file after unlink/reuse, so refuse the refresh when the fetched inode disagrees with the one the handle holds. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 11:32:57 -07:00
Chris Lu	c7d77eb9a8	s3vectors: accept and return encryptionConfiguration on vector buckets (#12203) ... * s3tables: carry encryptionConfiguration in table bucket metadata * s3vectors: accept and return encryptionConfiguration on vector buckets The configuration is validated (sseType AES256 or aws:kms) and stored on the bucket record, so the API round-trips what the caller declares. * s3vectors: reject caller-chosen encryption keys on vector buckets * s3tables: reject caller-chosen encryption keys on table buckets * s3tables: reject unknown sseType in bucket encryptionConfiguration The bucket check rejected aws:kms and kmsKeyArn but stored any other sseType — a made-up or mistyped value would round-trip through GetTableBucket as an encryption claim the storage does not keep. The validator now admits only the AES256 declaration, shared so the vector layer can drop its copy. * s3vectors: delegate encryption validation to the shared catalog check * s3tables: accept the sseAlgorithm name in bucket encryptionConfiguration S3 Tables declares the algorithm as sseAlgorithm while S3 Vectors uses sseType; the shared check now accepts either spelling, still admits only AES256, and still refuses any caller-chosen key. * s3vectors: call the shared encryption validator directly * s3vectors: require each API's own encryption field name The shared validator accepted sseAlgorithm on the vector path, so a CreateVectorBucket carrying the s3tables field stored a configuration GetVectorBucket could never echo back in the vectors shape (and vice versa). ValidateEncryptionConfiguration now takes the wire name its caller expects; the vectors create path normalizes to sseAlgorithm for the catalog and GetVectorBucket renames it back to sseType.	2026-10-29 11:29:00 -07:00
Chris Lu	fc9be77dbb	s3vectors: bound QueryVectors topK at the AWS limit (#12202) ... * s3vectors: bound QueryVectors topK at the AWS limit of 30 topK passed through unbounded; AWS caps a query at 30 results, so a larger request is a ValidationException before it reaches the engine. * s3vectors: raise QueryVectors topK bound to the AWS limit of 10,000 * s3vectors: validate QueryVectors topK before the catalog lookup * s3vectors: cover the QueryVectors topK bounds	2026-10-29 11:22:48 -07:00
3 people	eecc461b39	s3vectors: put, get and delete vector bucket policies (#12201) ... * util: do not bind outbound dials to a loopback -ip.bind `weed server -ip.bind=127.0.0.1` keeps every listener local, but the same address became the source of outbound connections, so the filer dialled its remote mount from 127.0.0.1 and failed with "connect: invalid argument". A loopback source cannot reach off the host; leave the source to the OS, as for a wildcard bind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: put, get and delete vector bucket policies The AWS S3 Vectors policy operations had no handlers; the policy document is stored on the shared table bucket, so the handlers delegate to the catalog's PutTableBucketPolicy, GetTableBucketPolicy and DeleteTableBucketPolicy and keep its guard. * s3vectors: treat ICEBERG buckets as absent in policy handlers GetVectorBucketPolicy/PutVectorBucketPolicy/DeleteVectorBucketPolicy delegated straight to the table bucket, so a bucket created for ICEBERG tables — invisible to every other vectors operation — could still have its policy read and written through the vectors API. Gate all three on lanceBucket, the same format check DeleteVectorBucket and CreateIndex already apply. * test/vectors: cover vector bucket policy put/get/delete Round-trips a policy document through the new ops and asserts a get on the deleted policy surfaces NotFoundException, so the boto3 conformance run exercises the added handlers end to end. * s3vectors: validate the bucket ARN and policy document Bucket-level operations now reject an index ARN and an ARN minted for another service, and PutVectorBucketPolicy refuses a document carrying s3vectors actions — the evaluator only honors s3tables actions, so such a rule would be a dead grant. * s3vectors: validate the bucket ARN on the remaining bucket-level operations * s3vectors: cover bucket ARN and policy document validation TestBucketARNs pins the accepted forms and the index/foreign-service rejections; TestValidatePolicyDocument covers a valid document, a malformed one, and the dead s3vectors action namespace. * s3vectors test: use an enforceable action in the bucket policy test The catalog refuses s3vectors actions in a stored policy, so the put call rejected the document before the get and delete assertions ran. --------- Co-authored-by: Ólafur Bjarki Bogason <olafur@genkiinstruments.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 11:15:59 -07:00
Chris LuandDevin	8e44178dc6	s3vectors: cap PutVectors at 500 items and check data against the index dimension (#12200) ... * s3vectors: cap PutVectors at 500 items and check data against the index dimension A batch could carry any number of vectors and a vector of the wrong dimensionality reached the engine: AWS caps a PutVectors call at 500 items and requires each vector to match the index's declared dimension. * s3vectors: cover the PutVectors batch and dimension checks * s3vectors: reject query vectors that do not match the index dimension Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: cover the PutVectors batch size bounds Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: cover the catalog-backed PutVectors handler path Run PutVectors through the real handlers over the in-memory catalog: a matching vector reaches the engine, a mismatching one is rejected with ValidationException before PutVectors is called, and a full vectorsMaxPutBatch batch is accepted. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 11:14:30 -07:00