mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-27 07:20:44 +01:00
7388d0d4fc7e5161cbc33f8bf61633efed062669
15781
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandDevin
|
7388d0d4fc |
vector: seaweed-vector workspace + flat/indexed top-K engine (P0 phase 1) (#12096)
* vector: scaffold the seaweed-vector workspace Standalone cargo workspace for the vector query engine, mirroring the seaweed-worker layout: own lockfile, same lint policy, path dependencies across the two trees since there is no common root manifest. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: resolve a table's vector layout VectorTable::resolve finds the vector column (explicit name, else the column named vector, else the sole fixed-size float list), the optional key column, and the metadata columns a query can project. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: flat and indexed top-k, key lookup query::flat runs an exact scan (use_index(false)); query::ann lets Lance split the query between index segments and a flat pass over fragments no index covers yet, so rows written after the last index build stay eligible. Both honour a metadata filter and return the raw _distance in the query's metric. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: reject a non-utf8 or non-identifier key column An explicit --key-column naming a non-UTF8 field made hits() silently report row ids as keys, and get() interpolated the name into a SQL filter verbatim. Require Utf8 at resolve and in hits(), and require a plain identifier before building the IN predicate. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: fall back to row id on a null key keys.value() on a null slot fabricates an empty-string key; a row with no key reports its row id instead, matching the no-key-column case. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: quote the key column in the IN filter A Utf8 key column named e.g. document-key made the predicate parse as subtraction, and an identifier interpolated unquoted is injectable. Backtick-quote it and double its inner backticks; double quotes are string literals in the filter dialect, not identifiers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: project only the columns a hit needs The knn projection pulled every metadata column — including other vector columns — although Hit returns none of them. Filters still evaluate off the scan path, so restrict the output to the key and the scoring columns. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
6f8f8e7079 |
admin: install the go.mod-pinned templ, not @latest (#12090)
* admin: install the go.mod-pinned templ, not @latest install-templ fetched templ@latest, which now emits templruntime.NewAttributeExpression - a symbol absent from the runtime pinned in go.mod - so every fresh 'make' regenerated uncompilable _templ.go files. Install the go.mod version and reinstall when the binary on PATH reports a different one. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * admin: invoke templ from Go's install dir, not PATH A templ binary earlier on PATH than GOBIN still shadowed the pinned compiler after install-templ ran, so generate kept emitting code for the wrong runtime. Resolve GOBIN (or GOPATH/bin) and call the pinned binary by absolute path for install checks, generate, watch, and fmt. * admin: regenerate templ output with the pinned v0.3.1070 The checked-in _templ.go files were stamped v0.3.1020, older than the go.mod pin. Regenerate so the committed code matches the compiler the runtime links against. * admin: resolve templ under the first GOPATH entry go install drops binaries in the first GOPATH element's bin, so a multi-element GOPATH left TEMPL pointing at a path that does not exist. Split on the OS list separator so Windows GOPATHs resolve too. * admin: install pinned templ before fmt and pattern-rule generates fmt and a direct %_templ.go rebuild invoked TEMPL without ensuring the pinned binary exists; an order-only prerequisite keeps the check cheap without forcing regeneration. * admin: keep spaced GOPATH/templ paths intact in the Makefile --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
6ebc055a94 |
master, volume, filer: start on a read-only mounted directory and serve existing data read-only (#12081)
* master, volume, filer: start on a read-only mounted directory and serve existing data read-only A read-only mount was treated as writable (permission bits only), so every service failed on its first write: the master panicked writing raft conf.tmp, the filer exited opening the leveldb LOCK, and the volume server skipped every volume while still offering the directory as free capacity. - util.IsReadOnlyMount: access(W_OK) == EROFS. - volume: on a read-only -dir take the existing O_RDONLY/noWriteOrDelete load, index .idx in memory when no .sdx exists, offer no free slots (same branches as isDiskSpaceLow), skip the vid stamp there. - master: on a read-only -mdir run raft from a private copy and refuse new volume ids. - filer: leveldb2/leveldb3 set their existing ReadOnly option. * volume: skip read-only mounts when picking EC shard copy targets A shard copy landing on a read-only mount fails at write time; offer only writable locations like the other placement paths do. * volume: mirror the read-only mount handling in the Rust volume server The .dat open only fell back to read-only on EACCES, so a real read-only mount (EROFS) still failed the load. The index side already degrades once no_write_or_delete is set. Add is_read_only_mount via access(W_OK) == EROFS, mark DiskLocation.read_only_mount, and mirror the Go checks: no free slots, heartbeat max = used, no vid-stamp writes, no EC shard copy targets, and a read-only index dir forces the volume read-only. * volume: write vid stamps while only the index dir is read-only * filer: skip the write-bit check when a leveldb store opens read-only * filer: serve reads of buckets with no leveldb3 dir on a read-only store * volume: derive a stable dir uuid when a read-only mount cannot store one * master: keep the read-only raft copy across restarts so votes survive * volume: index a missing .idx in memory on a read-only mount * volume: release the .idx handle after an in-memory read-only load * volume: mount EC shards read-only when their index dir is on a read-only mount * volume: mark the index dir read-only before the tail repair opens .idx * volume: write vid stamps while only the index dir is read-only (Rust) * volume: derive a stable dir uuid and report no free slots on a read-only mount (Rust) * volume: index a missing .idx in memory on a read-only mount (Rust) * volume: refuse an EC copy whose shard owner is on a read-only disk * master: key the read-only raft copy by the absolute -mdir Two masters sharing the same relative -mdir text from different working directories would have collided on one temp copy and shared raft state. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: mix device and host into the read-only dir uuid Deriving the uuid from the path alone let independent servers at the same mount point report one uuid, which RegisterUuids rejects. Hash in the device id and hostname so restarts stay stable but disks differ. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: vic6996-web <320442181+vic6996-web@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
0214f9b34e |
build(deps): bump github.com/pkg/sftp from 1.13.11 to 1.13.12 (#12086)
Bumps [github.com/pkg/sftp](https://github.com/pkg/sftp) from 1.13.11 to 1.13.12. - [Release notes](https://github.com/pkg/sftp/releases) - [Commits](https://github.com/pkg/sftp/compare/v1.13.11...v1.13.12) --- updated-dependencies: - dependency-name: github.com/pkg/sftp dependency-version: 1.13.12 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
b7d3c8a5a6 |
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.23.2 to 1.23.3 (#12087)
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.23.2 to 1.23.3. - [Release notes](https://github.com/Azure/azure-sdk-for-go/releases) - [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.23.2...sdk/azcore/v1.23.3) --- updated-dependencies: - dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore dependency-version: 1.23.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
74161edb7a |
build(deps): bump github.com/getsentry/sentry-go from 0.49.0 to 0.50.0 (#12089)
Bumps [github.com/getsentry/sentry-go](https://github.com/getsentry/sentry-go) from 0.49.0 to 0.50.0. - [Release notes](https://github.com/getsentry/sentry-go/releases) - [Changelog](https://github.com/getsentry/sentry-go/blob/master/CHANGELOG.md) - [Commits](https://github.com/getsentry/sentry-go/compare/v0.49.0...v0.50.0) --- updated-dependencies: - dependency-name: github.com/getsentry/sentry-go dependency-version: 0.50.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
07cfe2d5e2 |
build(deps): bump github.com/redis/go-redis/v9 from 9.22.0 to 9.23.0 (#12088)
Bumps [github.com/redis/go-redis/v9](https://github.com/redis/go-redis) from 9.22.0 to 9.23.0. - [Release notes](https://github.com/redis/go-redis/releases) - [Changelog](https://github.com/redis/go-redis/blob/master/RELEASE-NOTES.md) - [Commits](https://github.com/redis/go-redis/compare/v9.22.0...v9.23.0) --- updated-dependencies: - dependency-name: github.com/redis/go-redis/v9 dependency-version: 9.23.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
692423d641 |
build(deps): bump github.com/ThreeDotsLabs/watermill from 1.5.2 to 1.5.3 (#12091)
Bumps [github.com/ThreeDotsLabs/watermill](https://github.com/ThreeDotsLabs/watermill) from 1.5.2 to 1.5.3. - [Release notes](https://github.com/ThreeDotsLabs/watermill/releases) - [Changelog](https://github.com/ThreeDotsLabs/watermill/blob/master/RELEASE-PROCEDURE.md) - [Commits](https://github.com/ThreeDotsLabs/watermill/compare/v1.5.2...v1.5.3) --- updated-dependencies: - dependency-name: github.com/ThreeDotsLabs/watermill dependency-version: 1.5.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
2ffe61189f |
build(deps): bump actions/github-script from 7 to 9 (#12095)
Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
41c33bdb21 |
build(deps): bump github/codeql-action from 4.39.0 to 4.39.1 (#12092)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.39.0 to 4.39.1. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v4.39.0...v4.39.1) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.39.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
d010664634 |
build(deps): bump docker/login-action from 4.6.0 to 4.7.0 (#12094)
* build(deps): bump docker/login-action from 4.6.0 to 4.7.0 Bumps [docker/login-action](https://github.com/docker/login-action) from 4.6.0 to 4.7.0. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](https://github.com/docker/login-action/compare/v4.6.0...v4.7.0) --- updated-dependencies: - dependency-name: docker/login-action dependency-version: 4.7.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * ci: correct stale version comments on SHA-pinned docker actions Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
66e8211821 |
build(deps): bump actions/download-artifact from 7 to 8 (#12093)
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 7 to 8. - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](https://github.com/actions/download-artifact/compare/v7...v8) --- updated-dependencies: - dependency-name: actions/download-artifact dependency-version: '8' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
a235473105 |
[Volume] Preserve request semantics in replica proxy reads (#12075)
* fix 12074 * volume: shuffle a copy of cached replica locations for proxy reads LookupVolumeId serves cached location slices shared between lookups; shuffling in place races with concurrent proxied reads of the same volume. * seaweed-volume: forward inbound method and surface lookup failure in proxy reads Mirror the Go handler: proxy_request used to always send GET, turning a client HEAD into a replica download, and a failed master lookup returned 404 like missing data. Forward the inbound method and answer 500 for lookup failures in proxy mode. * volume: keep master lookup error detail out of the proxy error response * volume: bound the proxied-request wait in the method-preservation test * seaweed-volume: answer 404 for missing volumes in proxy-mode lookups Mirror Go's LookupVolumeId: a per-volume error in the master's lookup reply (e.g. a missing volume) means "no locations" and answers 404 downstream, not a lookup failure that proxy mode answers 500. * volume: guard nil lookup result in proxy path A master response that omits the requested vid leaves LookupVolumeId returning a nil *LookupResult with a nil error; dereferencing it panics the read handler. * seaweed-volume: answered lookup errors are missing data, not failures A master reply carrying a per-volume error with HTTP 404 (volume not found) now yields empty locations like Go's LookupResult, so proxy mode answers 404. Other failure statuses such as 503 during master warmup remain lookup failures that proxy mode answers 500. * seaweed-volume: drop hop-by-hop headers from replica proxy reads Go's inbound request drops Host, Content-Length and Transfer-Encoding into dedicated fields, so its forwarded header set never carries them. hyper keeps them; forwarding a client's Content-Length with no body stalls the replica's read, and forwarding its Host misnames the target. * volume: bound the proxy master lookup to the request context * volume: drop hop-by-hop and Connection-nominated headers from proxy reads * seaweed-volume: drop Connection-nominated headers from proxy reads * volume: skip the proxy lookup error when the inbound request is gone * volume: drop proxy-specific headers from replica proxy reads * seaweed-volume: drop proxy-specific headers from replica proxy reads Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: drop hop-by-hop headers from replica proxy responses * seaweed-volume: drop hop-by-hop headers from replica proxy responses * seaweed-volume: a failed lookup status without an error body is still a failure * volume: do not re-proxy an already-proxied read for a missing volume A proxied request that reaches a server which still lacks the volume was looked up and forwarded again, so a stale location answer could bounce the read between two servers until the client gave up. The Rust volume server's proxy_missing_volume already returns 404 for proxied=true; mirror it here. * seaweed-volume: replace stale proxied params in the forwarded query Go builds the proxied URL with query.Set(reqIsProxied), which replaces every proxied param the client sent. Appending after a client-supplied proxied=false left the replica's first-value lookup reading the stale marker, defeating the loop guard. --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
dfd9ecab05 |
mount: resume metadata-log replay where a dropped stream stopped (#12079)
* pb: checkpoint metadata-log replay progress per settled entry A chunk-mode subscriber's resume cursor only advanced after the whole LogFileRefs batch drained, so a stream dropped mid-replay resubscribed from the original stale position and re-read the same log files while live traffic kept appending - under a flapping filer the replay window never shrank, and each reconnect re-listed and re-fetched the same backlog. ReadLogFileRefs now reports every entry the merge has settled - delivered or filter-skipped - and FollowMetadata checkpoints StartTsNs on it, so a reconnect resumes where the previous stream stopped and the backlog drains across restarts. The peer aggregator wires the same progress into its resubscribe/offset position. * pb: emit a replay checkpoint only after the merge passes its timestamp Checkpointing an entry's timestamp the moment it settles could strand a same-timestamp entry still queued on another filer's stream: resuming is exclusive, so a reconnect would skip the unsettled half of the tie. The merge now emits a timestamp only once a later entry proves every tied entry was delivered (and once at clean EOF). * pb+filer: never regress the replay cursor below the settled checkpoint The drain's final assignment took the last settled entry's own timestamp, which can lag the log position progressFn just checkpointed - a skewed or legacy entry rewinds option.StartTsNs (and the peer's lastTsNs), so the next drain or a resubscribe re-reads the settled prefix. Guard both assignments so the cursor only moves forward. * pb: cover the resubscribe path of replay checkpoints A drain that aborts mid-replay leaves StartTsNs at the last checkpoint the merge passed, and the next subscribe resumes there instead of re-reading the whole ref set. The test drives the real makeSubscribeMetadataFunc path: refs, a mid-batch apply failure, then a resubscribe whose SinceNs must carry the checkpoint. * pb: emit the tail replay checkpoint only for a settled timestamp Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
0d721fccee |
fix(volume): keep preallocated growth above -volume.minFreeSpace (#12067)
* fix(volume): keep preallocated growth above -volume.minFreeSpace When free space fell below -volume.minFreeSpace, every volume went read-only and the master kept growing new volumes. With preallocation on, each new volume reserved its full size, driving the disk further below the floor (15 Oct incident: 66 empty volumes reserved about 1980 GiB). Two causes, both in weed/storage/store.go: - MaybeAdjustVolumeMax sized a disk's volume slots from raw free space, so space inside the floor was counted as available. It now counts only free space above the floor. - addVolume trusted the once-a-minute low-disk flag. A preallocated volume is now refused when the disk is low now, or when its reservation would not fit above the floor. Volumes without preallocation, including merges that copy existing data, are not affected. addVolume runs under a store-level lock, so two concurrent creates cannot both pass the room check against the same free space. * volume: mirror headroom guard in the rust store find_free_location only trusted the once-a-probe is_disk_space_low flag, so a burst of preallocated creates could still run the disk under -volume.minFreeSpace between probes, and maybe_adjust_volume_max sized slots from raw free space. Adds MinFreeSpace::is_low/available_space to match Go and applies both checks the same way. store.write() already serializes add_volume, so no extra lock is needed. * volume: read rust headroom tests from a fake disk The floor arithmetic used a live free-space reading, so unrelated disk activity could flip a test between pass and fail. Store.disk_stats_of is now the same seam Go uses (diskStatusOf): the growth-loop and floor tests run against a controlled disk, mirroring the Go suite one for one. * volume: hold back create metadata from the headroom check An exact-fit preallocation left free space exactly at the floor, so the .vif and .idx writes that follow a create could dip under it and get the location flagged read-only at the next probe. Reserve room for .vif on the data filesystem, and for .idx growth only when dir_idx shares it. * volume: mirror the metadata allowance in the rust store Same rule as the Go side: charge .vif against the checked filesystem and .idx only when dir_idx shares it, and grow the exact-fit test's fake disk by the per-create cost. * volume: skip the reservation check where create cannot preallocate * volume: mirror the platform preallocation gate in the rust store * volume.copy: count the preallocation reservation in the destination space check The copy path picked a location with room for the copied .dat, then reserved the master's full volume size limit on it. A volume smaller than the limit could land on a disk whose free space above -volume.minFreeSpace fit the copy but not the reservation, driving the disk under the floor the same way the growth loop did. Fetch the preallocation setting before choosing the location and require room for the reservation itself, only where CreateVolumeFile actually reserves blocks. * seaweed-volume: store min-free-space-adjusted available space like Go Go's DiskLocation.AvailableSpace holds free space above the -volume.minFreeSpace floor; the Rust probe cached raw free instead, so the VolumeCopy location check could accept a disk already inside the floor. Store the same adjusted figure and cover it with a test. * seaweed-volume: reserve the copied .dat like Go's CreateVolumeFile VolumeCopy called set_len for the reservation, which only grows a sparse file — no blocks are held, so a disk with room for the copied bytes but not the full reservation only fails mid-copy. preallocate_file fallocates KEEP_SIZE on Linux like Go, reserving the blocks up front; off Linux it is a no-op, matching PREALLOCATE_SUPPORTED. * seaweed-volume: charge a copy's destination pick for the .dat reservation Go's VolumeCopy now sizes its free-location check against the master's volume size limit when the platform preallocates, not just the copied bytes. The Rust pick only measured the copied bytes, so a destination could accept a copy whose reservation then dips under the floor. The config fetch moves ahead of the pick (same order as Go) and feeds both the space check and the reservation itself. * volume.copy: charge the picked location's free space under volumeCreateLock The copy path picked a destination off the once-a-minute AvailableSpace probe and released the decision long before the .dat reservation landed. Two copies could pass the same cached headroom and together drive the disk under -volume.minFreeSpace. ReserveFreeLocation serializes the pick under volumeCreateLock, verifies the disk still has live room for the reservation, and charges it against the location's cached free space until the next probe refreshes it. * seaweed-volume: mirror the copy pick's live-check and headroom charge Go's VolumeCopy now picks through Store.ReserveFreeLocation: live disk stats under the store write lock, then the reservation charged against the location's cached free space. The Rust plan_copy_destination gains the same store.reserve_free_location so two copies cannot spend the same headroom before the first .dat reservation lands. * volume.copy: saturate the free-space charge against a live-space drop The two statfs reads bracketing a pick can race a large external write; a plain subtraction would underflow and leave the location looking roomy until the next probe. * volume.copy: release the headroom reservation when a copy does not land ReserveFreeLocation charges the picked location's cached free space, but a failed copy never returned it, so the headroom stayed booked until the next disk-space probe and blocked retries. ReleaseFreeLocation refunds the charge; the copy's cleanup defer now registers with the reservation so every failure path (existing-replica delete, .note write, stream send) releases it, including the post-transfer mount/validate failures covered by the outer defer. * seaweed-volume: mirror the copy headroom release on failed copies release_free_location refunds reserve_free_location's charge. The copy returns it when the plan's existing-replica delete fails, when the .note write fails, and in cleanup_failed_copy, so a failed copy stops holding the headroom until the next disk-space probe. * volume.copy: keep pending copy headroom out of probe readings ReleaseFreeLocation refunded the reservation into AvailableSpace even when a CheckDiskSpace probe had already rebased the cache on live stats — the probe's reading still shows the in-flight copy's bytes as free, so the refund double-counted headroom and two copies could spend the same space, driving the disk under -volume.minFreeSpace. Track the outstanding charge in DiskLocation.ReservedSpace instead: probes subtract it so they cannot re-spend it, a failed copy removes only its own charge, and a landed copy commits it — the charge and the new usage cancel out. hasRoomForNewVolume respects pending headroom too, so a preallocating create cannot overrun an in-flight copy. * seaweed-volume: mirror pending copy headroom across disk probes reserved_space tracks the in-flight copy's charge separately from the cached free space: check_disk_space subtracts it, release removes only the failed copy's charge, a landed copy commits it, and has_room_for_new_volume respects it — matching the Go side. * volume.copy: settle a copy's headroom as its bytes land on disk The pending charge held the copy's full reservation until commit, but written and fallocated bytes already show up in statfs, so the live-space check counted them twice. A slow copy could then block other copies and volume growth despite free space above -volume.minFreeSpace. Settle the charge instead: commit the bytes CreateVolumeFile actually reserved (a failed fallocate leaves nothing committed), then settle each file's write progress as it lands. The failure paths now release only the unsettled remainder; allocatedSize is exported for the .dat's st_blocks. * seaweed-volume: mirror copy headroom settling as bytes land reserved_space keeps only the promised bytes statfs cannot see yet: preallocate_dat settles whatever blocks the filesystem actually reserved (0 when fallocate cannot), each copied file's progress settles the rest as it lands, and a failed copy releases only the unsettled remainder — matching the Go side. * volume.copy: release a failed copy's headroom charge only once Both defers fired on the same in-closure error: the inner released the unsettled charge and the outer released it again, driving ReservedSpace negative (an atomic.Uint64 underflow) and inflating AvailableSpace — the location would then fail every later headroom check, or worse, pass them with fabricated space. Zero the charge once it is released; the Rust mirror already releases via swap(0) for the same reason. * volume.copy: write into the preallocated .dat instead of staging over it The .dat stream staged into .copying and renamed over the file CreateVolumeFile had just fallocated, so the copy's transient footprint was preallocate+dat — more than the reservation the pick just checked — and the reservation itself was discarded at rename. Writing in place reuses the fallocated extent, keeps the peak at the reserved maximum, and leaves the copied volume preallocated like a created one. * seaweed-volume: mirror the in-place write into the preallocated .dat The .dat copy opened the just-fallocated file with truncate, freeing its blocks at open and spending the headroom again as the stream re-grew it. Append mode preserves them, matching the Go side. --------- Co-authored-by: chrislusf <chrislusf@gmail.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
|
b5af82409f |
fix: volume.fsck -volumeId purges live chunks referenced from outside the volume's bucket (#12076)
* fix: volume.fsck -volumeId purges live chunks referenced from outside the volume's bucket -volumeId narrowed the filer read to <bucketsPath>/<collection> when the selected volumes shared one bucketed collection (#9347). A filer started with -collection, or an fs.configure rule, stores files from any path in that collection, so the narrowed read misses their references and reports live chunks as orphans; with -reallyDeleteFromVolume they are deleted. #11794 already made -collection read from the root for this reason. Do the same for -volumeId: the flag keeps selecting which volumes are checked and purged, and the filer is always read from "/". Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> * test: volume.fsck keeps chunks referenced outside the volume's bucket --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Nguyễn Lộc Phúcanduser.email
|
937633a77d |
iceberg: fix CREATE TABLE rejecting a partition on the first column (#12072)
* iceberg: fix CREATE TABLE rejecting a partition on the first column Apache Spark number the columns of a new table from 0 and leave assigning fresh IDs to the catalog, so a table partitioned or sorted by its first column arrives with source-id 0. The create-table request decoded its partition spec and write order into iceberg-go's bound PartitionSpec and SortOrder, which require positive source IDs, so every such CREATE TABLE, and CTAS through stage-create, was refused with "Malformed request: Invalid request body". Decode both as iceberg.UnboundPartitionSpec and table.UnboundSortOrder, as iceberg-go documents for catalog create-table requests; newTableMetadata already resolves the placeholders through the request schema by name and assigns fresh IDs. A spec or order that does not resolve against the schema is now answered with 400 instead of 500, and the decode error message carries the underlying reason. Tests check that Spark's request bodies resolve to the same metadata as 1-based placeholders, that the handler accepts them, that unresolvable source IDs get 400, and add a Spark integration test that partitions by the first column. * iceberg: gofmt the create-table source-id tests --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
|
a6be3c89c2 |
filer: support nanosecond timestamps for conditional delete (#11339) (#12071)
* filer: support nanosecond timestamps for conditional delete (#11339) * shell: fs.verify prune carries the entry mtime in nanoseconds IfNotModifiedAfter now resolves to sub-second precision on the filer, so pass the looked-up mtime_ns along with mtime; the re-lookup check catches same-second rewrites that share the second, and the test mock mirrors the filer's seconds/nanoseconds cutoff. * shell: volume.fsck guards empty-directory deletes with mtime nanoseconds The looked-up mtime_ns joins the seconds in IfNotModifiedAfter so a directory rewritten within the same second is not dropped; the stub filer mirrors the filer's scale heuristic and the race fixtures use a realistic mtime so the nanosecond branch engages. * filer: cover the seconds guard and equality boundary Test second-resolution cutoffs on both sides of the entry's second and the exact-equality nanosecond guard that still deletes. * filer: compare the nanosecond guard as a time, not UnixNano UnixNano overflows for mtimes outside 1678-2262, letting a far-future entry slip past the guard; time.After compares exactly at every epoch. The seconds branch keeps Unix() so a same-second cutoff still deletes. * filer_pb: add explicit if_not_modified_after_ns to DeleteEntryRequest * filer: prefer explicit nanoseconds guard, keep if_not_modified_after as seconds * shell: send seconds and nanoseconds delete cutoffs, skip unexpressible mtimes * filer: merge conditional-delete guards into one nanosecond cutoff DeleteEntryMetaAndData carried both the seconds and nanoseconds guards even though the precedence between them is fixed, so every caller passed a dead parameter. The gRPC handler now folds if_not_modified_after into nanoseconds at the boundary, where (sec+1)*1e9-1 keeps the whole-second cutoff exact, and the filer takes a single ifNotModifiedAfterNs. * filer: hold the conditional-delete cutoff as a time, not int64 nanoseconds A seconds-only guard at or past year 2262 saturated to MaxInt64 nanoseconds (~2262), so a delete whose cutoff matched a far-future mtime was skipped while the request still reported success, and any entry between 2262 and the real cutoff survived pruning. The merged cutoff is now a time.Time, where a whole-second guard maps to the last nanosecond of its second for any int64 second. --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
b85281f0da |
s3: group ListMultipartUploads by delimiter into CommonPrefixes (#12069)
* s3: group ListMultipartUploads by delimiter into CommonPrefixes The delimiter was echoed but ignored, so every upload was listed. Keys that continue past the delimiter after the prefix are now listed once as a common prefix, counted toward max-uploads like an upload. * s3: bound tracked multipart common prefixes to the page Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: omit upload id marker when a page ends on a common prefix Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
b7b571d469 |
webdav: copy a file onto a collection URL without losing the collection (#12068)
* webdav: copy a file onto a collection URL without losing the collection An overwriting COPY of a file to dir/ removed dir, then OpenFile refused the name ending in / and the request failed with 403, leaving neither. Drop the trailing slash from the destination when the source is a file. * webdav: cover file-over-collection COPY through the real handler Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * webdav: keep a root destination out of the slash trim A file COPY or MOVE onto / would rewrite the Destination to an empty path, turning the existing refusal into x/net's bad-gateway error; leave the root untouched so RemoveAll refuses it as before. --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
96f310dec8 | docs: regenerate star history chart | ||
|
Chris Lu
|
ffa87c25fd |
ci: fix aarch64/amd64 musl rust builds on ubuntu-24.04 (#12070)
The 24.04 bump broke build-rust-binaries: aws-lc-sys C objects compile against noble's glibc 2.39 headers, which redirect sscanf/strtol to the C23 __isoc23_* variants musl does not provide, failing the link. Same fix as the enterprise build on seaweedfs/artifactory: - arm64: keep aarch64-linux-gnu-gcc, link a tiny forwarding shim that resolves __isoc23_sscanf/__isoc23_strtol to musl's vsscanf/strtol. - amd64: point CC at musl-gcc so C deps compile against musl headers. |
||
|
github-actions[bot]
|
153ab668c9 | 4.51 4.51 | ||
|
Chris Lu
|
6dd6ad512c |
iam/oidc: drop single-use web-identity tokens and the jti replay cache (#12065)
* iam/oidc: drop single-use web-identity tokens and the jti replay cache Reverts the single-use semantics added in #11936 (and the #12017 refactor): AssumeRoleWithWebIdentity consumed each web-identity token's jti into a bounded replay cache, so a second exchange of the same token was rejected until exp + skew. AWS STS accepts a web-identity token repeatedly until it expires, and Kubernetes workload identity depends on that: the kubelet keeps a projected ServiceAccount token until ~80% of its lifetime, so pod restarts, co-located containers sharing the token file, repeated CLI invocations, and pre-rotation SDK refreshes all re-present the same jti and were refused on 4.50. The replay machinery also carried a real maintenance cost — a bounded concurrent cache, fingerprint-keyed entries for jti-less tokens, adopt/merge handoffs so provider rebuilds could not reopen consumed tokens, and provider-reuse bookkeeping in the refresh path that existed only to preserve that state. Removing it restores AWS semantics and deletes the whole apparatus. Refs #12063. Alternative to the per-provider opt-out in #12064. * iam: reuse unchanged OIDC providers on refresh refreshOIDCProvidersFrom rebuilt every managed provider on any store change, so an unrelated write discarded each provider's JWKS cache and an unreachable issuer could then fail exchanges the cached keys would have allowed. Keep a per-record provider cache keyed on the record content fingerprint and only rebuild when the record changes. * iam/oidc: require exp in ValidateToken Dropping the single-use check also removed the implicit bound on expiry-less tokens: a signed web-identity token without exp could mint STS sessions indefinitely. Require the exp claim, matching AWS AssumeRoleWithWebIdentity which rejects tokens without one. * iam/oidc: bound idle connections on the provider transport A provider replaced on refresh drops its http.Transport without a close; give it an IdleConnTimeout so pooled connections and their goroutines cannot linger. * s3api: cover web-identity token reuse over HTTP Sign one token and exchange that exact string twice, directly covering the Kubernetes projected-token reuse this change restores. * test(iam): refresh reuses an unchanged record's provider Covers the provider-reuse cache: an unrelated record's refresh keeps a provider's JWKS state, a changed record rebuilds it. |
||
|
|
515380c349 |
s3: answer GET and HEAD of a delete marker as S3 does (#12051)
* s3: answer GET and HEAD of a delete marker as S3 does A delete marker as the current version now returns 404 NoSuchKey with x-amz-delete-marker: true and the marker's x-amz-version-id. Asking for the marker by versionId returns 405 MethodNotAllowed with the same headers and Last-Modified, instead of 404 NoSuchKey. * s3: check the tag-conditioned read grant before answering a delete marker The marker branch returned before the entry-based policy recheck the rest of the read path performs, so an s3:ExistingObjectTag-conditioned Deny did not apply to the delete-marker response and its version headers. * s3: send Allow: DELETE with the delete-marker 405 RFC 9110 15.5.6 requires a 405 answer to list the methods the resource supports; S3 sends allow: DELETE for a delete marker named by versionId. * s3: keep delete-marker headers on conditional-read misses If-Match or If-Unmodified-Since against a current delete marker short- circuited to a bare NoSuchKey. Carry the resolved marker through the conditional result so the 404 keeps x-amz-delete-marker and x-amz-version-id, matching the unconditional read. * s3: exercise delete-marker responses through GET and HEAD handlers * s3: read marker version id from the entry before the caller's guess A delete marker written outside the normal PUT path can miss the version-id attribute while the v_<id> file name still names it; the new x-amz-version-id header would then report "null". Reuse versionIdFromEntry so both the handler and conditional-miss paths emit the real id. * s3: answer 405 for versionId=null on a suspended-versioning marker The null marker lives in .versions while getSpecificObjectVersion only checks the plain path, so GET/HEAD ?versionId=null answered NoSuchVersion. On that miss, look up the null marker and answer it like any other marker named by versionId. * s3: propagate null-marker lookup failures instead of answering 404 A transient filer error reading .versions/v_null was swallowed as "no marker", so GET/HEAD answered NoSuchVersion for a live read failure. NotFound and absent entries still mean no marker; anything else is InternalError now. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
9620e7ebfe |
filer: auto-enable per-path commit records when a peer has its own store (#12062)
* filer: enable per-path commit records only when a peer has its own store Since seaweedfs#11698 every metadata change writes a per-path commit record into the filer store (an MpTs key, plus an MpDl key on deletes), and the records stay after the path is deleted. They order the events a filer replays from peer filers that keep their own store, since replay only runs when peerSignature != f.Signature. Filers that all share one store (one Redis, MySQL, TiKV, ...) never replay each other's events, so the records are never read, yet each change still costs one or two extra store writes and the key count grows with every path ever changed — unbounded growth for in-memory stores. This turns the records off by default and latches them on the first time a peer reports a different store signature — no option needed. The latch is persisted in the store (MpRep) so it survives restarts and converges filers sharing a store even before they discover the peer themselves. While off, a filer keeps an MpOff marker so enabling later bumps the records' generation (MpGen, folded into each record key) past the unrecorded window: a record older than a change made while off cannot let an older replayed peer event win. Ordering for unrecorded changes falls back to the entry's mtime, the same as entries written before the records existed. Supersedes the manual option approach in seaweedfs#12060. * filer: keep delete tombstones while records off, harden enable path Review on the first commit: - A delete committed while the records were off left no entry and no usable tombstone: after enabling, an older replayed create could resurrect the path. Deletes and rename sources are now always recorded under the generation-0 MpDl key — they are never fenced by a generation bump, and they cannot go stale because every delete is recorded. Creates/updates still write nothing while off. - bumpPathCommitGen could regress MpGen under concurrent bumpers on a shared store. It now loops: each pass writes only a generation higher than any it has seen, so concurrent bumps converge upward. - doSubscribeToOneFiler proceeded into replay when enablePathCommitRecords failed, running the whole session without ordering. It now aborts the subscribe so the reconnect retries with the records on. * filer: bind pending commit-record repairs to their generation A failed MpTs write kept in metaTsPending could land after a shared-store generation bump, writing a pre-bump timestamp as a current-generation record and overriding newer unrecorded state. Each pending change record now carries the generation it failed under: repairs write it only while that generation is current, and ordering reads ignore it otherwise. Pending delete records need no binding — MpDl keys are never fenced. * filer: never move the local records' generation backward The converge loop in bumpPathCommitGen seeded from zero and both it and syncPathCommitGen stored unconditionally: a stale writer could leave the stored generation below what this filer already adopted, and the next sync or bump then walked the local value back — orphaning records it wrote under the higher generation. The bump now seeds from the local generation and every store only moves it forward. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: name a records generation after the off marker it consumes Two filers consuming the same off marker each counted one past the stored generation, so the second could store N+2 after the first stored N+1 and write records under a key the other could not read until the next check. The bump now takes the marker's own timestamp — filers racing on one marker land on one value — a marker older than the stored generation is cleaned as a leftover, and a stored value that already passed the bump is adopted instead of counted past. A newly enabling filer whose marker was already consumed names its own off window with the current time, so its unrecorded period is still fenced. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: tell a fresh off marker from one the generation already consumed A marker whose timestamp reads at or below the stored generation was dropped as leftover, but a new off period can also read that old — the off filer's clock lagging, or a host clock moving back — and dropping it kept records from before that period live, so stale replay could win. The stored generation now names the marker it was bumped for: a marker the store still shows is consumed unless it is that one, however old it reads, and only the consumed marker itself is cleared without a bump. The generation trails in the value, so a whole-slice read still yields it on stores written before the pair was stored. Enabling fences the filer's own off window with the current time instead of a marker a sharer may already have consumed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: cover racing off-marker consumers settling on one generation * filer: adopt the stored generation, not the consumed marker it names A filer syncing with no off marker present read the value's first field — the consumed marker's timestamp — so a restart after a lagging-clock off period under-counted the generation and kept records the bump dropped. * filer: re-check the off marker before consuming it A sync whose marker read ran behind a later off period's consume cycle still bumped for its stale read: the stored generation already covered a different marker, so the bump overwrote the consumed name, counted the generation past records a newer consumption meant to keep, and cleared whatever marker was live by then. Each pass now re-reads the marker and only bumps or clears while it is still the one the pass saw. * filer: keep a failed generation read from resetting the records' generation The final generation read ignored errors and then stored whatever gen held — possibly zero — so a transient store failure could hide records the current generation wrote. Unexpected read errors now return before the local generation changes. Also cover a sync whose marker read runs behind another consumer's delete: it must adopt the stored generation, not bump for the stale read. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: fail a bump that cannot verify the stored generation A generation read that failed mid-bump was treated as absent: the write could land below a generation a racing bumper had stored, or re-store a marker that bumper had already consumed. Read failures now abort the pass like every other store error on the path. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: cover generation reads that fail instead of returning a value Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: carry the commit timestamp on the entry itself The per-path MpTs record only exists because a commit timestamp had nowhere else to live. Stamping commit_ts_ns on the entry before its store write - and reusing that stamp as the emitted event's TsNs - makes the entry its own commit record: replay and bootstrap traversal order on it with zero extra writes for creates and updates. Deletes still need a durable key since a removed entry cannot carry a timestamp, so MpDl tombstones stay (one write per delete, growth bounded by delete history). Entries predating the field fall back to mtime, matching what peers and old records already assume. This replaces the MpTs/MpOff/MpGen/MpRep machinery entirely: shared- store filers write nothing extra, and dedicated-store replication keeps last-writer-wins ordering plus tombstone coverage for deletes and ancestor drops. * filer: keep a directory update replay from tombstoning its subtree A same-path update wrote a delete record before the delete-and-reinsert, so a directory metadata change read as a subtree delete: a later peer event for a child older than the update was then skipped even though nobody deleted the children, leaving files missing on this filer. Only file updates take the tombstone now — the re-inserted directory entry's own commit stamp orders the path itself. * filer: order unstamped entries by their recorded commit before mtime Entries written before stamping existed have no CommitTsNs, so the ordering check fell back to the client-settable mtime and dropped the MpTs record those versions wrote: an entry committed at event time 300 with mtime 100 lost to a delayed peer event at 200 after upgrade. Read the legacy record for unstamped entries in localCommitTsNs, CommitTsNs, and the traverse stamp (which also stopped falling back to mtime, regressing entry_ts_ns to 0). * filer: release queued rename event stamps when the rename fails Each moved entry carried an in-flight stamp its queued event would adopt at emit time. When the move succeeded but a later child move or the commit failed, the handler rolled back without emitting the queued events or releasing their stamps, pinning the delivered and flushed watermarks below the aborted rename forever. Release every queued stamp on both failure exits in both rename handlers. * filer: apply a directory update replay with an in-place upsert Deleting the stored entry before inserting its replacement left a window where the directory had no entry and no ordering record — a tombstone cannot stand in for it without reading as a subtree delete that suppresses the children's own records. A stale peer event accepted in that gap could replay an older update or delete and sweep the directory's children. Upsert the replacement instead: the entry keeps its recorded commit order across the whole update, and a directory has no chunks or hard links for a delete to release. * filer: update an existing directory entry on replay instead of upserting InsertEntry is not an upsert on SQL stores configured without upsert support — PostgreSQL with enableUpsert=false issues a plain INSERT and its duplicate-key error does not match the MySQL-only fallback, so a replay updating a present directory failed on every retry. Pick UpdateEntry when the entry exists and InsertEntry when it is absent; UpdateEntry touches no row that is not there. * filer: detect a shared-store delete racing a replayed directory update UpdateEntry reports success even when the row was deleted between the find and the update — the UPDATE changes zero rows and the replayed change is dropped while the subscription offset advances. Re-read the entry after the update and retry the write, re-checking the recorded commit order each pass so a newer peer change wins instead of being resurrected over. * filer: declare commit_ts_ns in field-number order Field 6 sits between extended and hard_link_id; the declaration now reads in the same order. Wire format unchanged. Also syncs the Java client's checked-in copy of filer.proto, which the pb Makefile copies on gen. * filer: option to skip delete tombstones entirely filer.options.deleteTombstones=false stops writing MpDl records on deletes - a single filer or shared-store cluster never replays peer events, so the tombstones are dead weight. The tradeoff is explicit: if a dedicated-store peer does replicate, a stale event can resurrect a deleted path. * filer: drop delete tombstones entirely Deletes no longer write MpDl records, and the pending-repair machinery they needed is gone with them: per-mutation metadata overhead is now zero for every store topology. Ordering information lives only on the entry itself via commit_ts_ns. The accepted tradeoff: a dedicated-store peer replaying an event older than a delete can resurrect the path, and a kept directory no longer fences stale child creates. Recorded tombstones and MpTs records left by older builds are still read, so upgrades keep the protections they already have. * filer: stop reading legacy per-path records too Drops the MpTs/MpDl read path along with the writes: meta_path_ts.go keeps only the entry-field readers (CommitTsNs, EntryCommitTsNs, localCommitTsNs). Entries written before stamping existed fall back to mtime, and tombstones or commit records older builds left behind no longer fence replayed events. * filer: trim commit timestamp comments --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
53834117a6 |
mini: lift the post-mint allocation hold on first inventory report (#12061)
* mini: lift the post-mint allocation hold on first inventory report DeferAllocation armed a fixed warmup window whenever the leader minted a fresh topology id, so the first write on a brand-new weed mini blocked for the whole window even though no prior cluster state could exist. The window still has to cover a rebuilt master's veteran servers, which report on their first post-reconnect heartbeat. Mini has exactly one possible reporter — the embedded volume server, whose first heartbeat carries the full inventory — so hold allocation only until that report is applied. The bound raise and topology-id freeze checks run while the heartbeat is processed, so a wiped master state still raises the bound or trips the freeze before the hold lifts. Standalone weed master and weed server keep the fixed window: volume servers outside the process can still be on their way. * master: apply queued topology-id reports before opening the allocation gate SetTopologyInitPending(false) cleared the pending flag before replaying topology ids stashed while the local id was still being minted. A NextVolumeId landing in the drain window saw no gate and no freeze yet. The post-mint allocation hold used to cover the gap by delaying the winner past the freeze; once the hold can release on first inventory, the gate must stay closed until the queued checks have run. * mini: gate the first-report release on single-master mode releaseAllocOnFirstReport assumes the embedded volume server is the only possible reporter of prior cluster state. A mini started with -master.peers joins other masters, whose own volume servers can still be on their way when the first report lands, so keep the regular quiet-pulse hold there. * master: check reported topology id on empty full reports A HasNoVolumes heartbeat carries the same LastSeenTopologyId a populated list does, but the mismatch freeze only ran for non-empty reports. An EC-only veteran could therefore release the post-mint allocation hold through NoteInventoryApplied while its stale topology id never reached CheckReportedTopologyId. * master: drain the topology-id stash inside the init-gate lock CheckReportedTopologyId only held reportedTopologyIdsLock while stashing, so a report in flight during SetTopologyInitPending(false) could land its freeze after topologyInitPending cleared, letting a NextVolumeId slip through the gap. The whole check now runs under the lock, and the gate is released inside the same critical section as the drain. * mini, master: pin and enforce the single volume-server reporter isSingleMasterMode only checks master peers; SendHeartbeat still accepted any volume server, so a late foreign report could carry prior cluster state after the first-report release dropped the hold. Mini now pins the embedded volume server's address and the master refuses registration from any other reporter while ReleaseAllocOnFirstReport is set. * master: admit the volume-server reporter before applying heartbeat state * test: cover SendHeartbeat admission ordering A refused reporter must not freeze growth or raise the id bound, and the pinned reporter's stale topology id still must. * mini: re-derive the single-reporter release on every run The package-level master options survive in-process reruns, so a multi-master mini following a single-master one must not inherit the first-report release or a stale reporter pin. * master: admit addressless first heartbeats under the reporter pin An unregistered heartbeat with no Ip skipped the single-reporter admission check but still reached the seen-bound raise and the topology-id freeze before the delta-heartbeat skip. Admission now runs for every unregistered heartbeat; an empty address can never equal the pin. |
||
|
Chris LuandDevin
|
88b91cccba |
command: no extra localhost listener when -ip.bind is specified (#12055)
* util: add listenOnLocalhost parameter to NewIpAndLocalListeners Nil keeps the current behavior of also binding localhost when the host is a specific address; all callers pass nil for now. * command: add -ip.bind.localhost flag to opt out of the extra localhost listener Binding a command to a specific -ip.bind address also opened a second listener on localhost for every port, with no way to turn it off. Each command that calls NewIpAndLocalListeners (master, master.follower, filer, s3, iam, volume, sftp, mq.broker, mq.agent) now accepts -ip.bind.localhost=false to bind only the requested address. Embedded services get prefixed flags (s3.ip.bind.localhost, sftp.ip.bind.localhost, iam.ip.bind.localhost), and the command-level flag also disables them so a single switch turns off every extra listener the command opens. * mini: only require all-interface port availability while the loopback listener is on With -ip.bind.localhost=false each service binds just -ip.bind, so a port held on another address no longer collides; the all-interface precheck would reject or relocate usable ports. WebDAV, the Admin UI, and the Admin worker gRPC (which binds -admin.worker.ip, not -ip.bind) never open a loopback companion listener, so their checks now use the address they actually bind. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mq.broker: bind the gRPC listener to -ip.bind instead of all interfaces The listener used an empty host so -ip was ignored for binding and the localhost companion was never created, leaving -ip.bind.localhost with nothing to control. Give the broker the same -ip/-ip.bind pair the other servers use; embedded mode binds to -ip.bind like its peers. * server: add -sftp.ip.bind.localhost and -iam.ip.bind.localhost switches weed filer already exposes the prefixed opt-outs for its embedded sftp and iam; give weed server the same pair so each extra listener can be disabled individually while -ip.bind.localhost=false still turns them all off. * mq.broker: keep the default all-interface bind unless localhost is off An unset -ip.bind used to mean the standalone broker listened on every interface; defaulting it to -ip silently cut off multi-homed clients. Only fall back to -ip when -ip.bind.localhost=false, where a specific bind is needed for the opt-out to take effect. * mini: validate the Admin HTTP port on the address it will bind The port check always used -ip.bind while startMiniAdminWithWorker falls back to loopback when Admin has no credential, so a loopback-held port passed validation and killed the admin at startup. * mq.broker: keep the all-interface default bind when -ip.bind is unset -ip only becomes the fallback once -ip.bind.localhost=false needs a specific address for the listener. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mini: validate the Admin HTTP port on its effective bind IP When the Admin listener falls back to loopback (no credential or mTLS configured), the pre-flight port check now probes that same address instead of -ip.bind. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * util: keep NewIpAndLocalListeners source-compatible for 3-arg callers * command: drop the -ip.bind.localhost flag; an explicit -ip.bind drops the extra listener Instead of an opt-out flag, the loopback companion listener now simply follows whether the bind address was chosen: it is created only while -ip.bind is unset and the bind address falls back to -ip. Passing -ip.bind=<addr> binds each port to exactly that address. Each service options struct carries a listenOnLocalhost *bool that embedders (weed server, weed mini, embedded s3/sftp/iam in weed filer) set from the parent's -ip.bind state, captured before the bind flag is repointed or filled in by the -ip fallback; standalone commands derive it from their own flag before the fallback runs. iam and mq.agent gain a real -ip.bind flag so they have a way to pin a single address. mq.broker keeps its new -ip.bind flag but drops the -ip.bind.localhost fallback hack: empty still listens on all interfaces. * server: default embedded mq.agent brokers to the broker's bind address With an explicit -ip.bind the embedded broker no longer listens on localhost, so an unset -mq.agent.brokers defaulting to localhost:17777 could not reach it. Default to <serverBindIp>:<mq.broker.port> (the address the embedded broker binds), falling back to localhost when the bind address is a wildcard. An explicit -mq.agent.brokers still wins. Also fix the NewIpAndLocalListeners skip message to say localhost. * server: bind embedded mq.agent to serverBindIp The embedded agent's ipBind was never set, so startQueueAgent fell back to the advertised -ip address and an explicit -ip.bind did not pin the agent listener. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
fefac028ec |
feat: send EC shards in parallel per destination disk (#12050)
* feat: send EC shards in parallel per destination disk One stream per destination disk. max_shard_transfers unset means no global cap, 0 sends shards sequentially, and N is a global cap that still keeps one stream per disk. The value is persisted through TaskPolicy. The maintenance form is a checkbox plus Max transfers because unset and 0 differ. The key is intentionally omitted from GetConfigSpec. * ec shard distribution: treat negative max_shard_transfers as sequential The config layer (normalizedMaxShardTransfers) clamps negative values to 0, which selects sequential distribution. shardTransferCap diverged: a negative cap that reached ShardDistributionOptions directly would have selected uncapped parallel mode, the most aggressive setting. Align the options layer with the same normalization so invalid input degrades to the conservative mode. * ec: serialize shard transfers to a server when a target leaves disk choice to the receiver disk_id 0 is auto-placement on the receiving volume server, so an auto-placed stream can land on a disk another queued stream is pinned to. Grouping by (destNode, disk_id) alone let both run at once; collapse all of that server's transfers under the destNode key instead. * worker: clamp max_shard_transfers to the int32 wire range ReadIntConfig returned the 0 fallback for values above MaxInt32, turning a large positive cap into sequential mode. Read int64 and clamp into range instead. * worker: drop unused shard-transfer form helper The maintenance form renders and collects max_shard_transfers in plugin.templ JavaScript; the Go copy had no callers. * worker: bound max_shard_transfers conversion under an in-range guard --------- Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Jay HemnaniandChris Lu
|
0c9872f842 |
filer, volume: ignore Range when If-Range no longer matches (#12052)
* filer, volume: ignore Range when If-Range no longer matches A GET with Range and a stale If-Range validator got 206 with the requested bytes of the current file. RFC 9110 13.1.5: send the whole content instead, so a resumed download restarts rather than mixing two versions. * volume(rs): drop Range when If-Range no longer matches Mirror the Go fix for the Rust volume server: evaluate If-Range against the response ETag (strong comparison) or Last-Modified once the needle's validators are known, then drop the range on a stale validator so the whole current representation is sent. A dropped range on the RangeFromSource strategy falls back to a full stream of the data-file region; buffered and chunk-manifest paths already take Option<&str>. * volume(rs): fail If-Range when the header value is not readable * volume(rs): accept all HTTP-date formats in If-Range like Go * server: cover alternate HTTP-date formats in If-Range test --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
da1d7444b8 |
master: shorten the first-write hold after a fresh topology id (#12059)
* master: release the post-mint allocation hold after a quiet pulse DeferAllocation armed a fixed warmup window whenever the leader minted a fresh topology id, so the first write on a brand-new cluster blocked for the whole window even though no prior state could exist. The window still has to cover a rebuilt master's veteran servers, which report on their first post-reconnect heartbeat. Track the last heartbeat that could carry prior cluster state — a server joining, held volumes or shards, a nonzero id bound, or a foreign topology id — and lift the hold one pulse after it goes quiet, capped by the original window. * master: make the post-mint allocation hold configurable master.volume_growth.allocation_deferral overrides the warmup window a leader holds volume id allocation for after minting a fresh topology id. Unset keeps the existing window; "0s" disables the hold for setups that know the cluster is fresh (test harnesses, single-node labs); any other duration replaces the cap while the quiet-pulse early release still applies. * master: re-check the allocation deadline while it is held NextVolumeId slept for the whole computed delay, so a heartbeat landing mid-sleep that pushed the quiet deadline out was ignored — the pending allocation could still run ahead of a straggler's report. Wait in short slices, re-evaluating the deadline and the growth freeze each round. * master: count only ids above the issued bound as prior-state evidence Heartbeat echoes of volumes and shards this master allocated during the hold refreshed the quiet window, so each growth allocation re-extended the deferral until the hard cap. Evidence now requires an id beyond the bound this leader has issued, a higher reported bound, or a foreign topology id; a joining server (dn == nil) still counts once. * master: reject an unparseable allocation_deferral instead of disabling the hold Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: do not count a skipped heartbeat as a joining server Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: assert the allocation hold extends past the original quiet deadline Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: release the post-mint hold after two quiet pulses One pulse proved short for a veteran that reconnects on a staggered restart: with only five seconds of quiet, allocation could resume and reuse ids before a slower server landed its first heartbeat. Two pulses still halves the fresh-cluster first-write stall versus the full warmup cap, and known-fresh setups keep the "0s" override. * master: pick once more when assign times out as growth concludes The pick loop's final attempt can predate the growth it triggered concluding: growth registers its volumes in the layout before clearing the pending flag, so a timeout landing on that boundary used to shed a hard 406 over volumes that had just arrived. Pick once more after the loop in both the http and grpc assign paths before deciding how to shed. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
d15396fc4d |
s3tables: scope the anonymous identity's grant on the table catalogs (#12056)
* s3tables: scope the resolved anonymous identity instead of defaulting to allow Catalog authorization fell back to the open default whenever the request resolved to the configured anonymous identity, ignoring its grant. The fallback now applies only when no principal was resolved (zero-config), so anonymous requests are checked against the identity's configured actions like any other principal. The S3 Tables API and the Iceberg and Lance catalogs share this handler, so all three are covered. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: authenticate the catalog clients in the s3tables suites Anonymous calls into the Lance and Iceberg catalogs are now scoped by the anonymous identity's grant, so the suites that used to rely on the anonymous pass authenticate like their real deployments do: x-api-key for the Lance probes and REST drivers, client-credentials OAuth for pyiceberg. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * worker(lance): authenticate namespace calls with the storage credential Anonymous catalog calls are now scoped by the anonymous grant, so a worker that holds an S3 credential sends it as x-api-key on the namespace requests instead of relying on the anonymous pass. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: authenticate the pushdown catalog connection in catalog_lancedb Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: follow catalog redirects only within the same host reqwest forwards custom headers on a follow, so a redirect to another host would carry the x-api-key credential to a server it was never meant for. Keep same-host hops for proxies that normalize the path or upgrade the scheme. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: warn when the catalog api key travels in cleartext The x-api-key pair is a real storage credential: over plain http to a non-loopback host the network can read it. Warn at client build so the operator sees it; https and loopback stay quiet, and the key still sends since an http catalog is a valid deployment. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: require the same origin on catalog redirects A same-host hop can still downgrade https to http and put the api key on the wire; matching scheme, host, and port keeps the credential on the listener it was issued for. * lance: bound catalog redirect hops so loops fail fast The custom redirect policy drops reqwest's default 10-hop limit, so a catalog that redirects back to itself would hold each call open until the request timeout. Keep the bound alongside the origin check. * lance: gofmt the redirect helpers --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
39bdbc18db |
admin: keep read-only sessions out of the filer /etc tree (#12057)
* admin: keep read-only sessions out of the filer /etc tree The file browser read routes are open to any authenticated session, so a read-only login could download or view arbitrary filer paths. The filer stores the credential and cluster configuration under /etc (identity JSON with plaintext S3 secret keys, SSE-S3 keys, remote storage config), which let a view-only session recover live secrets that the user API redacts. Gate the read paths on the session role: read-only sessions get 403 for /api/files/download, /view, /properties, /metadata on /etc paths, the browser page redirects away, and recursive metadata exports skip the /etc subtree for read-only sessions. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * admin: only reject real .. path components in file browser paths path.Clean already resolves interior parent references, so any '..' surviving in the cleaned path can only be a leading escape from a relative input or part of a valid name. Matching on the substring rejected legitimate entries like /buckets/reports/report..csv. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * admin: cover the read-only /etc export exclusion with a test Pull the excluded-prefix selection into exportExcludedPrefixes so the role-dependent /etc skip is exercised directly — the cleanReadPath tests alone would still pass if the exclusion were dropped. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
403d51631b |
s3api: stop echoing stored tags and internal attributes on GET/HEAD (#12053)
* s3api: stop echoing stored tags and internal attributes on GET/HEAD setResponseHeaders copied every entry.Extended key into the response, only filtering xattr- and x-seaweedfs-. Stored object tags (X-Amz-Tagging-*) and the gateway's own attributes (Seaweed-X-Amz-Acl, -Owner, -Version-Id, -Object-Lock-Mode, ...) therefore leaked to any caller holding s3:GetObject, and to anonymous readers on public-read objects, without s3:GetObjectTagging or READ_ACP. Tag count, user metadata, storage class and the real S3 headers (x-amz-version-id, x-amz-object-lock-*) are unaffected. * s3api: match stored tag and internal attribute keys case-insensitively |
||
|
Chris Lu
|
52d91008ac |
ci: bump hosted runners to ubuntu-24.04
GitHub is retiring ubuntu-22.04; move all 94 runs-on pins across 40 workflows to ubuntu-24.04. rust_binaries_release: the arm64 cross step was jammy-era — noble keeps apt sources in deb822 ubuntu.sources (the sources.list edit was a silent no-op that left amd64 repos open to arm64 and 404'd) and ports suites must say noble. Pin deb822 stanzas to amd64 and add noble ports entries. |
||
|
github-actions[bot]
|
ec8f979350 | 4.50 4.50 | ||
|
Chris LuandDevin
|
1a0485a43d |
test: bound trino container memory + dump logs when it exits before ready (#12044)
* test: dump trino container state and logs when it exits before ready
When the Trino container dies during startup, waitForTrino only
reported the docker exec error ("container is not running"), which
gives no hint whether the JVM was OOM-killed, the image broke, or the
config is bad. Capture docker inspect exit state and the container
logs in the failure message so runner-side flakes are diagnosable
from the test output alone.
* test: cap trino container memory at 4g in catalog tests
Trino's jvm.config sizes the heap at 80% of host RAM. On a loaded
self-hosted runner the JVM attempts to claim most of system memory
at boot and the container is killed ~15s after docker run — the
flake seen in CI today. Bounding the container to 4g gives the JVM
a fixed ~3.2g ceiling (observed usage is ~750mb), immune to host
memory pressure.
* test: keep docker errors in trino container diagnostics
Report docker inspect/logs failures instead of dropping them, and
include the last docker exec output in the container-dead fatal so a
missing container still explains what happened.
* test: detect trino container exit in readiness fallback
When the polling loop times out while the container still appears
alive, the trino --version fallback may itself fail because the
container has since exited. Re-check its output for the
exited-container signature and route through the diagnostic dump.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: dump trino container diagnostics on readiness timeout too
The alive-but-never-ready timeout path reported only the last docker
exec error, losing the container state and logs that the dead-container
path already captures.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Chris LuandDevin
|
8c1a0a0cc2 |
filer: fix prefixed listing when the resume point sorts below the prefix (#12049)
* test: prefixed directory listing with start below the prefix A listing constrained to prefix "item-b" and resumed from "item" must return item-b.versions. The seek-based stores return an empty page because the iterator lands before the prefix range and breaks on the first non-matching name. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: clamp prefixed-listing seek to the prefix range Seek-based stores (leveldb, leveldb2, leveldb3, rocksdb, tikv, etcd) started the scan at startFileName and stopped at the first key outside the prefix. A startFileName below the prefix landed the iterator on a non-matching sibling first, so the listing returned an empty page even though matching entries followed. etcd had the mirror problem: a range beginning below the prefix end leaked non-matching entries. Begin the seek at max(startFileName, prefix), which is what foundationdb and the sql stores already do. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer(hbase): honor startFileName lower bound in prefixed listings The scan starts at the prefix but only skipped the exact startFileName, so entries sorting before it were returned again on resume. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: keep paging a prefixed listing while matches can still follow The fallback for stores without native prefix support stopped paging as soon as the last scanned name reached the prefix, dropping prefix matches that continue on the next page. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: return the last delivered name from the prefixed-listing fallback The fallback for stores without native prefix support pages raw entries and filters client-side. Keep the scan cursor and the returned cursor separate: resuming must continue after the last delivered match, not the last scanned name, or matches still inside the current raw page are skipped on the next page. Also keep paging while the scan cursor is inside or below the prefix range instead of stopping at the first name that reaches the prefix. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer(hbase): start prefixed-listing scans at the resume key Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
0c32eab48e | docs: regenerate star history chart | ||
|
Chris Lu
|
85470416cb |
volume: overlap the local needle write with the replica fan-out (#10074)
* volume: overlap the local needle write with the replica fan-out ReplicatedWrite wrote the needle to local disk and then fanned it out to the replicas serially, so a replicated write cost local + replicas. The needle is already fully buffered, so the two can run at once. Snapshot the fields the fan-out reads first (the local write stamps the volume TTL and updates needle flags, so the replica goroutine must not touch the shared needle), then run the local write and the replica fan-out concurrently and ack only after both. Where local disk and replica network are separate resources the write latency drops to max(local, replicas) instead of their sum; co-located on one disk the two writes contend, so this helps distributed placements, not single-host tests. * volume: return the local isUnchanged from the write goroutine * volume: trim overlap-write comments * volume-rs: overlap the replica fan-out with the local needle write * volume: test the overlapped local write and replica fan-out * volume: decode replica pairs only when replicas exist A single-copy or incoming replicate write never uses the decoded pair map, so a malformed Pairs blob only burned an error counter and a log line. Gate the decode on replicaCount. * volume: release the stalled replica on test failure paths An early t.Fatal left the httptest handler blocked on releaseReplica, so the deferred Close hung instead of reporting the failure. Release idempotently ahead of the server close. * test: mismatched If-None-Match serves the object over If-Modified-Since The precedence case asserted the old If-Modified-Since-first ordering and expected 304. The volume servers now evaluate If-None-Match first, so a non-matching tag must return 200 with the body even when If-Modified-Since would allow 304 (RFC 9110 13.1.3). |
||
|
Chris LuandDevin
|
e442eda88e |
s3: authorize POST policy uploads through the shared object check (#12047)
* s3: authorize POST policy uploads through the shared object check doesPolicySignatureMatch gated a verified POST Object signature on the identity's legacy Write action, which cannot represent attached IAM policies, STS session policies, or key-scoped grants. Users authorized only by an attached policy (e.g. s3:PutObject on bucket/*) were denied with AccessDenied even though the equivalent PUT succeeds. The handler already runs AuthorizeObjectWrite against the real object key after the form's x-amz-* fields are promoted to headers, so the coarse check only shadowed the full evaluation. Signature verification now just returns the identity; authorization stays in one place. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: POST Object authorizes attached-policy identities, still denies others Cover the signature check and AuthorizeObjectWrite for an identity whose only write grant comes from an attached IAM policy, plus the deny leg for an identity whose attached policy omits s3:PutObject. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
216c1032e6 |
test: mismatched If-None-Match serves the object over If-Modified-Since (#12045)
The precedence case asserted the old If-Modified-Since-first ordering and expected 304. The volume servers now evaluate If-None-Match first, so a non-matching tag must return 200 with the body even when If-Modified-Since would allow 304 (RFC 9110 13.1.3). |
||
|
|
03f773f087 |
redis2: pipeline entry reads when listing a directory (#12042)
ListDirectoryEntries fetched every member with its own GET, so a page of 1024 entries cost 1024 round trips. Fetch the values in pipelined batches of 128 as the loop reaches them and walk the replies in order. Each reply keeps its own error, so missing and expired members are cleaned up and skipped exactly as before, and an unreadable member still ends the listing at that name. Batching keeps a caller that lists with a huge limit and stops after a few entries from reading the whole directory. Co-authored-by: Patrick Schmidt <1310072+Woellchen@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Junker der Provinz
|
f9dac64374 |
helm: remove filer.redirectOnRead, which weed filer does not accept (#12040)
weed filer lost -redirectOnRead in |
||
|
|
7b43dec665 |
shell: run volume.configure.replication in parallel per volume server (#12041)
* shell: run volume.configure.replication in parallel per volume server Volume servers were configured one after another, and each server's result overwrote the previous error, so a failure on one server was lost when a later server succeeded. Run one job per volume server, bounded by -maxParallelization like the other shell commands, and return the errors from all servers. VolumeConfigure unmounts and remounts the volume, so replicas of the same volume on different servers take turns instead of all going offline at once. Co-authored-by: NyaMisty <5344431+NyaMisty@users.noreply.github.com> * shell: test that volume.configure.replication overlaps servers up to the limit * shell: assert every server ran in the parallel configure-replication test Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: NyaMisty <5344431+NyaMisty@users.noreply.github.com> Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
95d8a709e8 |
ci: run s3-go-tests on remote_storage changes (#12043)
The S3 Remote Cache tests exercise the remote-storage S3 client, but the path filter did not include weed/remote_storage, so a tagging change there can regress the suite without running it on the PR. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
e432b75822 |
remote_storage/s3: send only real object tags to the remote (#12039)
* remote_storage/s3: send only real object tags to the remote WriteFile and toTagging copied every entry.Extended attribute into the remote object's tag set, including Seaweed-internal keys such as Seaweed-X-Amz-Acl whose values are not legal S3 tag values. Remotes that validate x-amz-tagging now reject these uploads with InvalidTag. Only X-Amz-Tagging-* attributes are S3 tags: filter to that prefix, strip it, and encode through s3TaggingHeader so keys and values are properly escaped. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * remote_storage/s3: skip remote tagging call when the tag set is unchanged UpdateFileMetadata fired PutObjectTagging on any Extended diff, but most attribute changes are not tag changes. Compare the effective tag sets and return early when they match. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * remote_storage/s3: assert WriteFile errors in tag-header tests Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
81c90cb446 |
filer.backup: retry a failed checkpoint read instead of replaying from the epoch (#12036)
* filer.backup: retry a failed checkpoint read instead of replaying from the epoch Fixes #12035 * test: reset sink.local.directory in backup offset test cleanup The new test sets sink.local.directory on the shared viper but only reset the enabled flag, leaking a key that points at a deleted t.TempDir() to later tests in the package. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover the retried attempt re-reading the checkpoint The fake filer now fails only its first KvGet, and the test runs a second doFilerBackup attempt asserting the checkpoint is re-read and the attempt proceeds past the offset stage. LookupDirectoryEntry is stubbed so the SSE-S3 KEK probe returns a miss instead of retrying 10x2s. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
4c2c0ff21c |
s3: fix wildcard matcher skipping '*' targets in policy patterns (#12034)
* s3: check pattern '*' before literal equality in matchWildcardString When the target string carried a '*' at the offset of the pattern's '*', the literal-equality case consumed it as a one-character match and no backtrack point was recorded, so the comparison returned false. A Deny Resource carrying a policy variable, or an Allow carrying NotResource, could then be sidestepped by an object key containing '*' at the aligned offset. * test: cover literal asterisks in wildcard match targets --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
1d2d9dfed2 |
test(fuse): raise mini/filer and mount readiness waits to 90s (#12037)
Same startup-timing flake class as the Makefile waits: volume/filer readiness at 30s is marginal on loaded runners — observed failing on a github-hosted runner too (TestPOSIXCompliance + TestExternalPOSIXSuites). The process-death early return still reports real startup failures quickly. |