mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-20 21:14:25 +02:00
2095c905ba6a047aa4e2da4836280001a7eaecf6
15582
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandDevin
|
2095c905ba |
server: give heartbeat-options test a topology (#11853)
heartbeatResponse now reads MaxVolumeId and TopologyId off ms.Topo, but the test built a MasterServer without one and panicked, failing the package for every PR. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
6b6a268903 |
volume balance: stop replica-fallback moves between equally full servers (#11851)
* volume balance: unify the per-move convergence guard Evaluate the post-move utilization comparison through serverUtilization so the guard also applies when servers lack MaxVolumeCount info (raw-count mode), where it previously did not run at all. * volume balance: restrict fallback destinations to improving servers The score-based fallback inside createBalanceTask could redirect a move to any valid server, including ones as full as the source — between equally full servers the move only relocates the imbalance and the next detection cycle reverses it (#11839). Exclude from eligibleTargets any server whose post-move utilization would exceed the source's, so a fallback destination satisfies the same convergence guard as the loop-chosen target. * volume balance: skip a volume when the chosen target violates placement When the detection loop's target resolves but cannot host the replica (it already holds one, or is out of the filtered scope), createBalanceTask now returns nil so the loop advances to the next volume — matching shell volume.balance, which continues to the next candidate instead of redirecting the move to a different server. The score-based fallback is kept for the case where the target cannot be resolved at all; its allowed set is already limited to servers the move would improve. * volume balance: regression test for replica-constrained targets (#11839) Reproduces the reported loop: the emptiest server holds the second rp=001 replica of the lowest-ID volumes on the fuller servers, so it is never a legal target for them. Detection must skip those volumes and send the movable ones to the under-utilized server, and re-running detection after applying the plan must never move the same volume twice. |
||
|
Chris LuandDevin
|
79daffb2d2 |
test: retry retryable 503s in cold-start assign test (#11852)
The master intentionally sheds an assign that triggered volume growth once its wait window elapses (503 + Retry-After); on a slow CI box cold-start growth can outlast it, so the single-shot assertion flaked. Honor the documented retry contract like a real client, within a 60s deadline. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
207d06863a | docs: regenerate star history chart | ||
|
Chris LuandDevin
|
7e348951c2 |
s3: strip internal metadata headers on auth entry (#11850)
* s3: strip internal object-lock and version headers on auth entry putToFiler persists Seaweed-X-Amz-Object-Lock-Mode, -Retention-Until-Date, -Legal-Hold and -Version-Id request headers into authoritative entry metadata without a permission check, so an s3:PutObject-only writer could pin COMPLIANCE retention and legal hold on its own uploads, bypassing the dedicated-permission gate on the standard x-amz-object-lock-* headers. The server only sets these headers from validated inputs, so drop them in authenticateRequestInternal alongside the identity headers already scrubbed there. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: strip internal SSE headers on auth entry X-SeaweedFS-SSE-S3-Key-Data, -SSE-S3-Base-IV and -SSE-KMS-Base-IV are read from the request to drive multipart encryption, so a client could inject its own key material and IVs. The server sets them itself between handler stages of a request; drop inbound values in authenticateRequestInternal. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: scrub internal headers before the auth-enabled early return Auth and AuthPostPolicy skip authenticateRequestInternal when IAM is disabled, so client-supplied internal headers still reached putToFiler. Hoist the scrub into scrubInternalHeaders and run it unconditionally in both wrappers as well as the shared auth entry point. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: test internal header scrub through the auth-disabled wrapper Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
8b09c96cec |
s3: keep a signed content-length in the canonical request (#11849)
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
0fb6c0e77d |
filer.sync: unstick sync on chunks the source no longer has (#11845)
* filer.sync: keep the needle-404 clock from resetting on live volumes The give-up clock for a source chunk that cannot be produced is kept per volume so a vacuumed volume's files all inherit one wait. But a needle 404 and a volume lookup failure shared that clock, and sourceServed cleared it whenever any needle on the same volume read fine, so on a busy cluster the grace never elapsed for a single deleted needle on a healthy volume and filer.sync retried it forever (#11841). Give needle not-found answers their own per-volume clock that serving siblings does not reset. * filer.sync: an answered 404 needs no liveness probe before skipping After the missing-chunk gate gives up, onReplicateChunkError only skips the entry when a re-read of the most recently served chunk succeeds. That probe can never have been armed on a fresh sync, and its single stored file id can itself be vacuumed since — either way the entry stays pinned and the offset never advances. A needle 404 was answered by a live volume server, which is itself proof the source is serving requests, so skip on it directly. Keep the probe for the lookup-failure flavor, where no server answered and the silence can still mean a source outage. * filer.sync: skip a replayed version the source rewrote in place Supersession was decided by entry mtime alone, so a source entry whose chunks were swapped without an mtime bump — remote cache eviction and re-cache, an in-place repair — never read as superseded, and its vacuumed chunks were retried indefinitely. When a chunk is missing, compare the replayed chunk's file id against the live entry's resolved chunk set: absent means the replayed version is obsolete and its event is skipped on the first failure instead of after the give-up window. The same live-entry comparison now also decides the entry-level skip in onCorruptChunk/onReplicateChunkError. * filer.sync: give each missing needle its own grace clock A 404 is one needle's answer, but the clock was kept per volume: once the first needle's wait expired, every other needle on that volume inherited an already-elapsed clock and could be written off on its first sighting. Track the wait per file id and clear it only when that needle itself serves again, so a live volume's healthy needles neither shorten nor restart it. * filer.sync: keep the probe for filer-proxied 404s ReadPart proxied through the source filer collapses "no volume locations" into the same 404 a missing needle returns, so that answer cannot stand in for the liveness probe the way a direct volume read's can. Only treat 404 as an answered needle when the sink reads volume servers directly. * filer.sync: only a not-older live entry can supersede a replay Chunk-list comparison must respect the same ordering the mtime check does: a live entry older than the replayed version means the source was rolled back, so its differing chunks say nothing about whether the replayed data is still needed. Gate the content check on live mtime >= replayed mtime in both sourceMovedOn and sourceDroppedChunk. * filer.sync: compare chunk ids without resolving manifests resolvedChunkFileIds downloaded manifest files to expand chunk lists, so a manifest sitting on a dead volume made the supersession check itself fail and left the offset pinned behind a replica the source cannot read. File ids are immutable needle identities — the unresolved sets already compare chunk content exactly, and the check no longer performs I/O. * filer.sync: judge nested chunks by their top-level chunk A manifest child is not in the entry's chunk list even when the manifest still references it, so checking the live entry for the child id reported it dropped and returned the raw 404 — unwrapped, so the offset still pinned. Thread the entry-level chunk id down the manifest recursion and consult it for the dropped-chunk check instead. * filer.sync: let the manifest-child test exit through the gate Run the worker with a shortened grace period and assert the returned error is marked errSourceChunkMissing, instead of racing the worker still running against a restored RetryWaitTime. |
||
|
Jay HemnaniandChris Lu
|
60b06b6881 |
s3: sort ListMultipartUploads by key and honor key-marker (#11833)
* s3: sort ListMultipartUploads by key and honor key-marker * s3: skip upload folders without an object key in ListMultipartUploads * s3: order same-key multipart uploads by initiation time The upload id suffix is now a time-ordered UUIDv7, so sorting by upload id lists a key's uploads oldest first, like S3. Pagination is unchanged: ordering and the upload-id-marker comparison share the same key. selectMultipartUploads now builds the whole result, including NextKeyMarker and NextUploadIdMarker, and the pagination test pages with the response markers. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
dbcf6850b6 |
ci: fail the awscli multipart retry loop on the last attempt (#11848)
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
04eff7d613 |
ci: retry the awscli multipart download on transient failures (#11846)
The HEAD on the completed multipart object occasionally gets a 403 from a signature verification mismatch before the object read settles; retry the download instead of failing the run. Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
4feb30deff |
Federated volume-id bound via heartbeat stamps (#11763) (#11819)
* pb: heartbeat fields for the federated volume-id bound (#11763) Heartbeat carries max_volume_id_seen and last_seen_topology_id up, and HeartbeatResponse echoes max_volume_id and topology_id down, so the allocation bound is replicated to every volume server. * master: echo the volume-id bound and fold reported bounds in (#11763) Every heartbeat response carries the leader current max volume id and topology id; every heartbeat reported max_volume_id_seen is folded into the topology bound via UpAdjustMaxVolumeId. * volume: persist a per-dir vid stamp of the highest id ever seen (#11763) The store keeps max_volume_id_seen + last_seen_topology_id in a .vid_stamp file per volume dir, fed by heartbeat echoes, AllocateVolume, and local volume/EC creation, seeded at boot from stamps and local holdings, and reported in every heartbeat. A rebuilt master recovers the bound from the first surviving veteran report. * volume-rust: mirror the vid stamp and heartbeat echo handling (#11763) Same mechanism as the Go volume server: .vid_stamp per dir, bound folded in from add_location scans and allocate/mount calls, echoed fields applied on heartbeat responses, reported on every heartbeat. * volume: serialize vid stamp writes and fsync the directory (#11763) * volume-rust: serialize vid stamp writes and fsync the directory (#11763) * master: echo the allocation bound to connected servers on change The bound was only sent at connect time or on digest mismatch, so a healthy stream never learned later increases. If the high-id server then went offline and the master lost state, survivors restored a stale bound and the id could be reissued. Resend the heartbeat response whenever the bound or topology id moved. * volume: retry failed vid stamp saves on later updates A transient write failure left the in-memory bound raised but the stamp stale, and repeat updates at the same bound never retried. Track a dirty flag so any later update re-attempts the save. * volume: cover stamp bootstrap for pre-stamp dirs in test * volume-rust: retry failed vid stamp saves, mirroring Go * volume-rust: stamp every dir with the combined bound on add_location A dir whose own stamp sat below the combined bound kept its stale value after fold; losing the higher-stamped disk then restarted the server below ids it had already learned. Rewrite all stamps after each location folds in. * volume: stamp mounted volume and ec shard ids (#11763) * volume-rust: stamp mounted volume ids, mirroring Go (#11763) * volume: retry dirty vid stamp save on unchanged topology id (#11763) * volume-rust: retry dirty vid stamp save on unchanged topology id (#11763) * master: record echoed bound from the sent heartbeat response (#11763) |
||
|
Chris Lu
|
117ea58a83 |
s3tables: accept .puffin deletion vectors in table data/ (#11844)
* s3tables: accept .puffin data files for Iceberg v3 deletion vectors Iceberg v3 writes deletion vectors as Puffin files under a table's data/ directory, so merge-on-read DELETE and UPDATE on v3 tables were refused with AccessDenied by the table-bucket upload validator. Fixes #11842 * s3tables: name all accepted data formats in rejection error The message stopped at .avro even though .lance and now .puffin are accepted. * s3tables: cover deletion-vector filenames in data layout test Uses the names real engines emit: Spark/Iceberg Java {seq}-{task}-{uuid}-{counter}-deletes.puffin, UUID-suffixed names, and the same under a partition directory. |
||
|
dependabot[bot]
|
ce93660abb |
build(deps): bump github.com/linxGnu/grocksdb from 1.10.8 to 1.11.1 (#11836)
Bumps [github.com/linxGnu/grocksdb](https://github.com/linxGnu/grocksdb) from 1.10.8 to 1.11.1. - [Release notes](https://github.com/linxGnu/grocksdb/releases) - [Commits](https://github.com/linxGnu/grocksdb/compare/v1.10.8...v1.11.1) --- updated-dependencies: - dependency-name: github.com/linxGnu/grocksdb dependency-version: 1.11.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
73391a2e2f |
build(deps): bump github.com/tidwall/gjson from 1.19.0 to 1.20.0 (#11834)
Bumps [github.com/tidwall/gjson](https://github.com/tidwall/gjson) from 1.19.0 to 1.20.0. - [Commits](https://github.com/tidwall/gjson/compare/v1.19.0...v1.20.0) --- updated-dependencies: - dependency-name: github.com/tidwall/gjson dependency-version: 1.20.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
3a30ee177a |
build(deps): bump github.com/klauspost/compress from 1.20.1 to 1.20.3 (#11837)
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.20.1 to 1.20.3. - [Release notes](https://github.com/klauspost/compress/releases) - [Commits](https://github.com/klauspost/compress/compare/v1.20.1...v1.20.3) --- updated-dependencies: - dependency-name: github.com/klauspost/compress dependency-version: 1.20.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
109dd24f52 |
build(deps): bump github/codeql-action from 4.38.3 to 4.39.0 (#11840)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.38.3 to 4.39.0. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v4.38.3...v4.39.0) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.39.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
a2f8f4943c |
build(deps): bump gocloud.dev/pubsub/rabbitpubsub from 0.46.0 to 0.47.0 (#11835)
--- updated-dependencies: - dependency-name: gocloud.dev/pubsub/rabbitpubsub dependency-version: 0.47.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
4909843748 |
build(deps): bump modernc.org/sqlite from 1.57.0 to 1.60.1 (#11838)
Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.57.0 to 1.60.1. - [Changelog](https://gitlab.com/cznic/sqlite/blob/master/CHANGELOG.md) - [Commits](https://gitlab.com/cznic/sqlite/compare/v1.57.0...v1.60.1) --- updated-dependencies: - dependency-name: modernc.org/sqlite dependency-version: 1.60.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
0a96972a99 |
s3: verify X-Forwarded-Prefix only from trusted proxies (#11843)
* s3api: check the direct peer against the trusted-proxy allowlist * s3: verify X-Forwarded-Prefix only from trusted proxies * s3: note that trusted proxies must sanitize forwarded headers |
||
|
Jay Hemnani
|
4e5fed9e9e | s3: list the size of a directory marker that has a body (#11830) | ||
|
Chris LuandDevin
|
e7923c14d1 |
s3tables: honor resource policy Deny on remaining operations (#11825)
* s3tables: honor a table policy Deny for maintenance operations authorizeMaintenanceTable evaluated only the bucket policy, so a Deny in the table policy never vetoed PutTableMaintenanceConfiguration, GetTableMaintenanceConfiguration or GetTableMaintenanceJobStatus even though they act on the table ARN. The stored table policy is now checked for a matching Deny first via deniedByPolicy, a veto helper that keeps the admin exemption, and resource and bucket tags are loaded into the policy context so tag-conditioned Denies evaluate. readBucketPolicy is renamed to readPolicy and reused for the table path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: honor the stored policy's Deny on policy management operations PutTablePolicy, GetTablePolicy and DeleteTablePolicy only evaluated the bucket policy, so a Deny in the table's own policy was ignored and a bucket-level Allow could strip or replace the policy guarding a table. The existing table policy is now loaded and a matching Deny vetoes the call. PutTableBucketPolicy likewise consults the existing bucket policy for a Deny before replacing it. Resource and bucket tags populate the policy context so tag-conditioned Denies evaluate, and policy writes assert the policy authorized against is still in force, returning a conflict when it changed mid-request. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: apply a table policy Deny to tag operations TagResource, ListTagsForResource and UntagResource evaluated only the bucket policy; a Deny in the table's own policy is now honored when the target ARN is a table. ListTagsForResource and UntagResource also returned early when the resource carried no tags attribute, skipping the permission check entirely; the check now always runs. Policy evaluation uses the canonical resource ARN rebuilt from stored metadata, so a request ARN with a forged account or region cannot dodge the Deny. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: regression tests for resource policy denies Cover the case where a bucket policy grants an identity s3tables:* while the table policy denies it: the policy, tag and maintenance operations must stay denied so the protection cannot be stripped. Also cover a bucket policy that denies PutTableBucketPolicy under the open fallback, and a tag-conditioned Deny, which vetoes only while the stored tag matches. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
51a4c092c8 |
rust volume: release the store write lock during needle I/O (#11823)
* volume: give Volume interior locks mirroring Go's Volume's mutable state moves behind interior locks matching Go's per-volume lock domains, so the store can hand out shared handles instead of requiring exclusive &mut access: - io: RwLock<VolumeIo> owns the .dat/remote handles, needle map, append cursor, compaction bookkeeping and write watermarks — Go's dataFileAccessLock-adjacent mutable set; helpers take the guard as &mut VolumeIo so nothing re-locks. - super_block and volume_info each get a RwLock (superBlockAccessLock, volumeInfoRWLock); dir_idx gets its own so path building never contends with I/O. - no_write packs Go's noWriteLock pair into AtomicU8; last_modified_ts_seconds and compaction_byte_per_second become atomics. The data-file write lease (dataFileAccessLock) now covers every mutating entry that previously relied on borrow exclusivity: writes, deletes, blob writes, set_writable's flag/index swap, remote-file updates, local-backend swaps, index relocation, replica-placement rewrite, put_needle_index, close and destroy. set_read_only_persist leases only its can_delete idx-attach branch because mark_io_unavailable reaches it under a held lease. commit_compact takes try_write_lock — callers hold the store lock, so blocking would stall every volume behind one slow reader; WouldBlock surfaces as COMMIT_WAITS_FOR_READS for the caller to retry. Lock order is fixed: lease -> io -> volume_info/super_block/dir_idx. * storage: mount volumes as shared Arc<Volume> handles DiskLocation's map owns Arc<Volume> so the store can hand a caller a handle that stays valid across store-guard releases, instead of borrowing &mut Volume under the store write lock. - volumes: HashMap<VolumeId, Arc<Volume>>; find_volume returns a cloned Arc. - find_volume_mut and volumes_mut are gone — every Volume method is &self now, so shared handles cover what exclusive borrows did. - writable_volume_mut -> writable_volume(&self) -> Result<Arc<Volume>> keeps the same admission checks (exists, disk-space-low). - write_volume_needle/write_volume_needles/delete_volume_needle are &self; lifecycle methods that mutate the map or mounted state stay &mut self. * server: run needle I/O with the store guard released The store write lock previously wrapped needle writes and deletes end-to-end: process_batch held it across every grouped fsync, and the HTTP/gRPC fallbacks, tail receiver and fetch-and-write paths did the same. One volume's slow I/O then stalled admission and metadata on every other volume. - write_queue resolves each group's handle via writable_volume under a brief store read, then runs write_needles_grouped after the guard drops. - HTTP write/DELETE fallbacks, the gRPC batch-delete EC recheck + delete, tail receiver and fetch_and_write local writes, blob writes and the quarantine/sync/set_writable/remote-file paths all run under store.read() now that every volume method they reach is &self. - Lifecycle calls that mutate the volume map or mounted state (mount/unmount/delete, EC shard ops, compaction commits, load_new_volumes, heartbeat volume actions) keep store.write(). * test: cross-volume admission proceeds during another volume's write Regression coverage for the store-lock fix: - write_queue's test parks a volume-1 write inside its .dat append via a new needle_write_hook seam (mirroring needle_read_hook), then proves the store still hands out a lifecycle writer and a volume-2 handle — while volume 1's data-file lease still excludes same-volume writers. - tests/store_lock_repro.rs exercises the public API: concurrent read-guarded writes on two volumes, and a stale Arc<Volume> handle failing closed after unmount instead of writing into torn-down files. * test: drop store write guards that shared handles made read-only Test setup sites held store.write() purely to reach find_volume_mut; with Arc<Volume> and &self methods the guard is read-only (clippy readonly_write_lock). The volume lease still serializes the writes inside. * volume: drop a windows-only read_exact_at that shadows the io helper cargo's windows build failed E0255: the free fn copied with the interior-lock conversion collided with the unconditional crate::storage::io::read_exact_at import, which already carries the platform branches internally. * server: release the store read guard before needle I/O The read guard still ran through write_needle_blob_and_index, the tail receiver write, the fetch-and-write local write, and the HTTP write and DELETE fallbacks, so a slow append blocked mount/unmount writers (and, on writer-preferring locks, readers on other volumes). Resolve the Arc<Volume> under a short guard and write through the handle after it drops — the same scoping process_batch already used. * volume: serialize the read-only mark and its .vif save under the lease set_read_only_persist only leased the idx-attach branch, so concurrent mark-readonly and mark-writable calls could interleave flag updates and .vif writes — including two save_vif calls truncating and writing the file at once. The lease now spans the whole flag change and save, with a _held variant for mark_io_unavailable, which reaches this from write paths already holding the lease (Go: markVolumeReadonly runs under dataFileAccessLock). * volume,server: span the tier-down transition under one write lease update_remote_files cleared the remote-only write restriction and released its lease before open_local_dat_backend attached the local file, so a write could slip into the gap — pass check_writable, then fail appending to a missing dat_file. The handler now holds the lease across the reference trim, .vif save, and backend swap (and the rollback re-insert), via _held variants; the store read guard is scoped to the handle lookup. The swap still lands after the .vif trim — the ordering that keeps a crash from pointing the volume at a deleted remote object. * server: run a batch's volume groups on separate scoped threads A single batch can hold writes for several volumes; running the groups sequentially made every later volume wait behind a stalled append on an earlier one — the same head-of-line blocking the store guard release fixed, one level up. Each group now gets a scoped thread, preserving order within a group while independent volumes progress in parallel. * storage: close a volume handle a mount slot replaces With shared handles, inserting a new Volume over an existing map entry left the old handle alive in callers — two Volume instances could then write the same files with separate in-memory indexes. set_volume and create_volume now close() the replaced handle so its writers fail closed, matching what unload_volume already does. * test: key the write hook off a reserved needle id The hook registry is process-global and keyed by needle id, and other parallel tests write NeedleId(1) — an unrelated write could enter the barriers and leave the test's real writer parked forever. * server: wait out readers for vacuum commit without the store guard commit_compact try-locked the write lease because callers held the store write lock, returning COMMIT_WAITS_FOR_READS while a download was in flight — and the master reads that as a failed vacuum, discarding completed compaction work. The commit RPC now resolves the handle under a brief store read and blocks on the volume's lease, matching Go's CommitCompact under dataFileAccessLock: a slow reader stalls only this volume's vacuum. The try-lock path stays for callers that still hold the store lock. * server: scope the remaining store read guards to volume lookup Same class as the write-path fix: the batch-delete cookie check and tombstone, the query read, volume_needle_status, and the local chunk read all kept the read guard through the needle operation, so slow I/O there could still stall store writers. Each now resolves the Arc<Volume> under a short guard; the EC rechecks keep their guards. * seaweed-volume: fail closed on detached volume handles A handle resolved under a brief store read can outlive its mount: an unmount+remount between lookup and lease leaves the Arc pointing at a closed Volume while a new instance owns the files. Mark the handle in close() and reject it under the write lease in commit_compact_held, delete_needle, check_writable, set_read_only_persist and update_remote_files_held; the tier-down handler also checks the flag before deleting a remote object its replacement still reads. * seaweed-volume: drop the store guard before mark-readonly waits on the lease mark_volume_readonly held a store read across set_read_only_persist, which waits out the volume's leased readers; a slow download would stall every lifecycle operation behind the guard. Resolve the handle under a short read, then mark off the guard; a detached handle now maps to not_found. * seaweed-volume: retry batch deletes that raced an EC mount The store guard now drops before delete_needle, so ec.encode can mount shards between the pre-delete recheck and the tombstone; the encode then removes the .dat and the needle resurrects from pre-tombstone shards. Re-check has_ec_volume after each successful delete and return the retriable 503 so the filer requeues onto the EC path. * seaweed-volume: one write-queue worker per volume A stalled batch kept later batches for other volumes queued behind it: the single worker awaited process_batch before receiving again. Go's volume_write.go keeps one batching worker per volume, so give each volume its own FIFO worker spawned lazily, bounded by a global concurrent grouped-I/O semaphore and a global queued-request bound; workers exit only when their channel closes, and submit refuses unknown volume ids before spawning. * seaweed-volume: move the queued-write permit inside the request submit held the global queued-slot permit across the send and the response wait, so a canceled submit freed the slot while the worker still held the needle; repeated cancels could overflow the bound. The request now owns the permit until the worker drops it. * seaweed-volume: retire write workers whose volume left the store worker_for kept a sender per volume forever, so servers that mount and retire volumes accumulated idle workers. A worker now polls while idle and retires once its volume leaves the store: close the channel so racing senders fail fast, drain accepted writes, then release the map slot — a replacement worker can never start while this one still holds queued writes, so same-volume FIFO survives retirement. |
||
|
Chris Lu
|
0b3ed4502f |
Design + master mirror of max volume id outside raft (#11763) (#11818)
* design: durable volume-id allocation boundary (#11763) * master: mirror max volume id to a file outside raft (#11763) The allocation bound lived only in raft state, so a master that lost it rebuilt the bound from whatever volume servers were connected and could reissue ids held by offline servers. NextVolumeId now persists the bound to <mdir>/max_volume_id before the raft apply, and both raft implementations persist on apply and recovery so follower mirrors stay current. The file also carries the topology id, which survives raft state cleanup the same way the snapshot recovery did. * master: serialize and fully sync the max volume id mirror (#11763) * master: rewrite the mirror when the topology id changes at the same bound A same-bound raft apply carrying a new topology id was skipped by the lastPersistedMaxVolumeId check, leaving the mirrored topology id empty until the next allocation. Track the mirrored topology id too and rewrite when either value differs. * drop the design doc from the PR stack |
||
|
Chris Lu
|
ab559bca76 |
storage: hash index-metric bloom filter with xxhash
The reverse index walk dedupes seen NeedleIds with a BoomFilters bloom filter. Its default kernel is FNV-1, whose final step only XORs the last input byte into the hash's low bits, so keys sharing the high 7 bytes of a big-endian NeedleId land on correlated probe positions and false positives arrive in bursts of hundreds instead of the ~0.001 target rate. Each false positive hides a live-latest key and inflates the rebuilt deletion counters, which is what made TestFastLoadingNeedleMapMetrics flake past its +256 bound in CI (observed DeletedCount drift of 267). Swap the kernel for xxhash, which fully diffuses the key bytes; the observed false-positive count in 300 local runs of the test drops to max 2 (avg 0.23). |
||
|
|
d04d838ebd |
ec: compare live idx NeedleId sets before encode (#11812)
* ec: compare live idx NeedleId sets before encode Refuse encode when replicas' live .idx NeedleId sets differ. Walk only live rows (skip deleted and Size<=0) and compare NeedleId, ignoring Size so MIME/flags drift does not abort a matching id set. Mark the volume read-only for encode and reject writes after that. * fix(ec): compare all replicas then pick capacity source Log matching live idx sets with "replicas are consistent" for the shell CI check. After readonly, compare every replica's live NeedleId set, then require the encode source to have FreeVolumeCount>=2. Empty live-idx vs non-empty refuses encode instead of skipping superblock stubs. Drop the FileCount rebind after the idx gate. * volume: mark readonly under the write lock so checked appends finish first * ec: fix readonly deadlock and restore worker-host source preference MarkVolumeReadonly held dataFileAccessLock while calling reopenIdxForWrite, which re-acquires the same mutex and deadlocks TestMarkVolumeReadonlyCanDelete. Split out reopenIdxForWriteLocked for callers already holding the lock, matching the existing Locked pattern. The worker task lost its worker-host source preference when the idx gate replaced the replica union. Late-bind the source to a replica on the worker host using only the checked location set, with no FileCount rebinding. Drop the now-unused helpers, trim comments added by the earlier commits, and rename tests whose names still claimed convergence while asserting refusal. --------- Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com> Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
10ce19cb17 |
test: poll for TopologyId convergence in multi-master failover tests
A restarted goraft master wipes its raft state and only learns the TopologyId after the leader re-admits it and replays the entry. That trails HTTP readiness by several seconds, so the one-shot check in assertTopologyIdConsistent raced the propagation (CI run 39199033687: TopologyId applied on the follower ~2s after the test gave up). |
||
|
Mohd Quamar TyagiandMohd Quamar Tyagi
|
e20640acb4 |
fix(admin): improve Total Volumes count contrast (#11817)
fix(admin): improve Total Volumes count contrast on dashboard Co-authored-by: Mohd Quamar Tyagi <Tyagiquamar@users.noreply.github.com> |
||
|
Chris Lu
|
8ac5133abe |
admin: default to port 12535, keep 23646 as a best-effort compat listener (#11816)
* admin: default to port 12535, keep 23646 as a best-effort compat listener The default admin HTTP port 23646 derives a worker gRPC port of 33646 (port + GrpcPortOffset), inside the Linux default ephemeral range (32768+). One of our own startup dials can claim 33646 before the admin binds it, killing startup. This is the same failure #10209 fixed for the test port allocator, still reachable via the mini default (#10838) and via standalone `weed admin` defaults. Move the default admin port to 12535 (gRPC 22535) everywhere, keeping the http+10000 convention so worker -admin derivation, the Rust worker and host:http.grpc addressing keep working: - weed admin -port, weed mini -admin.port - weed worker -admin, the plugin worker fallback, Rust worker and weed-lance-worker defaults - help text, README, helm values, docker compose, tests So old workers, bookmarks and scripts keep working, admin also serves the same handler and the worker gRPC server on the previous default (-port.legacy / -admin.port.legacy, default 23646, 0 disables). The legacy gRPC port binds best-effort: it lives in the ephemeral range, so a failed bind only warns instead of reintroducing the startup failure. weed mini reserves it early like the primary port and keeps it out of its own port relocation. grpc.Server tracks every listener handed to Serve, so the compat listener stops with the primary. Also warn when an explicitly configured admin port puts the derived gRPC port at or above the ephemeral floor, the other fix suggested in the issue. Supersedes the pinned-grpc approach in #11814: pinning -admin.port.grpc off the +10000 convention breaks external workers that derive the admin gRPC port from the HTTP address. ClickHouse/ClickHouse#115512 pins -admin.port on their side; this makes that pin unnecessary. Fixes #10838 * admin: let services win the legacy ports, close compat listener on failure Review findings on the compat listeners: - Do not put the legacy ports in mini's reservedPorts: an explicit -s3.port=23646 or -master.port.grpc=33646 must be honored, not rejected or stolen by the early reservation. Mini now checks the finalized port set and skips only the conflicting compat listener, so HTTP and gRPC compat degrade independently and the service keeps its port. - Also skip the legacy HTTP listener when a mini service owns the port: a wildcard-bound service can still leave loopback bindable, which would let the compat listener shadow it for loopback clients. - Close legacyServer's raw listener in the deferred cleanup: Serve may not have registered it yet when startAdminServer returns early. - Helm: add admin.legacyPort (default 23646) and expose the legacy http/grpc ports through the admin Service, container ports and NetworkPolicy so older workers resolved via the Service keep working. * helm: only pass -port.legacy when it differs from the binary default ct install deploys the released image with the new chart, and 4.48 does not know -port.legacy, so the admin pod crash-loops. Omitting the flag at the default value is behavior-identical on new binaries (their default is also 23646) and keeps older releases installable; legacyPort: 0 or a custom port still renders the flag. |
||
|
github-actions[bot]
|
d3ee817b2e | docs: regenerate star history chart | ||
|
|
863efb3d35 |
webdav: delete a folder with its contents (#11811)
* webdav: delete a folder with its contents * webdav: guard the recursive delete against stale and confined paths - propagate stat lookup errors instead of returning an error-bearing FileInfo: OpenFile and DELETE would otherwise take the recursive removeAll on a path whose type was never confirmed, wiping a directory on a transient filer error - refuse wrappedFs.Rename from or to the served root, matching the RemoveAll guard and webdav.Dir - normalize trailing slashes in removeAll so the delete names the entry itself, and refuse the root for every caller * webdav: keep file-only deletes shallow and refuse ancestor destinations - removeAll used by OpenFile and failed-upload cleanup now deletes non-recursively, so a folder swapped in after the stat check is not emptied by a pending file delete - MOVE and COPY whose destination contains the source are refused before the handler removes the destination, which would otherwise take the source down with it * webdav: also refuse a destination inside the source a MOVE or COPY into its own subtree recurses into the destination it just created; refuse it with the same guard instead of burning the recursion budget for a 500 * webdav: allow a shallow copy into the source subtree a COPY with Depth: 0 creates only the destination collection and never reads children, so it is safe inside the source; only deep copies and moves are refused there * webdav: refuse a copy/move whose source is unreadable * webdav: cover the missing-source refusal in the guard test * webdav: 500 on a source lookup failure, 404 only when absent --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
8cb42cd69f |
build(deps): bump github.com/seaweedfs/raft from 1.2.1 to 1.2.2 (#11815)
Brings in the raft snapshot-recovery fixes: a recovered follower returns to Follower so it can campaign when the leader goes away (masters on ephemeral -mdir no longer wedge leaderless), and currentTerm stays monotonic and persisted through recovery. |
||
|
Chris Lu
|
6a594da739 |
filer: free orphaned hard-link chunks and remote objects in delete sweeps (#11809)
* filer: free shared chunks when a folder delete drops the last hard link doBatchDeleteFolderMetaAndData skipped chunk collection for every hard-linked child, relying on maybeDeleteHardLinks to retire the shared inode record. When the delete removes the last link the record is gone and nobody else will ever free those chunks. After the link callback runs, check whether the record survived; if it did not, queue the entry's chunks for deletion. * filer: free orphaned hard-link chunks in the replayed delete sweep The subtree sweep on a replayed directory delete skipped every hard-linked child, so a link that was the inode's last owner leaked its shared chunks. Check the inode record after the store delete; when it is gone, queue the child's chunks like any other swept entry. * filer: mirror remote-object cleanup in the replayed delete sweep Children a lagging peer alone still holds under a replayed directory delete were dropped with plain store deletes: their remote objects leaked and lazy remote fetches could resurrect them. The sweep now records event-stamped tombstones and deletes swept non-directory children through maybeDeleteFromRemoteIfUnchanged, which carries the same stat guard the write-back daemon's processDeleteEvent applies plus a modification-time check — a delayed replay cannot erase an object another writer replaced, even a same-size one. A failed remote delete fails the replay so the event is retried instead of orphaning the object. The directory's own tombstone comes from the event pipeline at the event timestamp and covers the subtree. Metadata-only deletes and cross-cluster events keep the origin's semantics. * filerstore: return real errors from KvGet on MongoDB, ArangoDB, Elastic These stores mapped every lookup failure to ErrKvNotFound, so a database error looked identical to a missing key: hard-link records could appear gone while links still owned their chunks, and path timestamp reads could treat committed state as absent. Return the underlying error and keep ErrKvNotFound for genuinely missing keys. * filer: bind remote-object deletes to the observed object version A guarded stat-then-delete still lets a replacement landing between the two requests go down with the deleted object. Add an optional conditional-delete operation to remote storage clients — S3 If-Match on the live ETag, GCS generation match, Azure IfMatch — and use it everywhere a delete is guarded by a stored stamp: the write-back daemon's delete, rename-source, and pending-rename paths, and the replayed-delete sweep. Backends without it keep the guarded stat-then-delete. * filer: restamp the remote entry after a metadata-only remote update Changing Content-Encoding rewrites the object in place on S3 and advances its LastModified, while the recorded stamp keeps the old mtime. A version-checked delete would then mistake the object for a replacement and keep it forever. Re-stat after UpdateFileMetadata so the saved stamp describes the live object. * filer: mirror the remote-delete ownership checks in the replayed sweep The sweep deleted swept children's remote objects on the listing's snapshot alone. Two gaps: a remote-only entry that took the path since — one a commit record may not cover, such as a lazy fetch materialized from the remote — reads the object as its only copy, and an entry removed while its rename copy was pending leaves the object at the source key orphaned. The per-child delete now runs under the child's entry lock with the commit-time check repeated, keeps a remote-only entry that holds the path, and releases the pending rename's source under the same ownership and stamp conditions the write-back daemon's releasePendingSource applies. * remote.sync: restamp only the object the metadata update touched The live stat after UpdateFileMetadata was saved unconditionally, so a replacement written between the update and the stat stamped the entry with an object it never described — and a later delete would then pass the freshness guard on the replacement's data. The restamp now requires the live object to still match the stored stamp on size and ETag; a different object keeps the old stamp, which the delete guard reads as replaced and keeps. * filer: skip unbudgeted reader-cache test on 32-bit TestReaderCacheUnboundedWithoutBudget proves the reader cache has no implicit memory budget by keeping 96 concurrent 4 MiB downloads in flight (>256 MiB). On linux/386 the weed/filer test binary already runs near the 4 GiB address-space limit, so the burst reliably crashes it with "out of memory" mid-suite. * filer: bail out of a swept child lock instead of waiting on it Replay already holds the event's path locks when the sweep reaches a descendant. A local rename can hold that descendant while waiting on one of the replayed paths, so blocking on the child lock deadlocks both. The sweep now tries the lock and reports ErrLockContended — transient, so the replication retry replays the event once the local write settles. * remote: report the stamp a metadata update produced A metadata write can rewrite the object it touches — S3 changes Content-Encoding through an in-place copy, which can mint a new ETag — so restamping from a fresh stat can reject the very object the update produced, or describe a replacement that landed in between. UpdateFileMetadata now returns the RemoteEntry of the object it wrote (S3 from the copy result, GCS from the update attributes, Azure from a post-update stat); nil means the object was untouched and callers keep the stored stamp. * remote: restamp Azure from the metadata write's own etag |
||
|
|
eea3f232da |
iam: scope CreateServiceAccount's ParentUser by policy resource (#11736)
* iam: scope CreateServiceAccount's ParentUser by policy resource AuthorizeIamAction evaluated the iam:CreateServiceAccount grant against a hardcoded arn:aws:iam:::* resource, so the policy's Resource never had a say; #11218 then bound the target to the caller outright. Evaluate self-target actions against the target identity's ARN instead: a wildcard resource keeps the deliberate any-target grant, a scoped resource confines issuance to the named parent identity, anything else is denied. buildResourceARN gains the ARN pass-through buildS3ResourceArn already has, so iam: ARNs reach the policy engine intact on the attached-policies route. Closes #11735 * ci: retrigger after unrelated weed/storage metrics flake * iam: fold IAM-ARN matching into awsIAMMatch, drop debug test - remove zz_debug_test.go (local debugging leftover) - fold awsIAMMatchResource into awsIAMMatch: the arn:aws:iam:: value prefix selects case-sensitive matching, so no separate name is needed - trim comments --------- Co-authored-by: batonogov <batonogov@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
4a57432488 |
s3: list children of the directory a ListObjects marker names (#11805)
* s3: resume into the directory a listing marker names * s3: keep delimited marker resumes exclusive * s3: descend a marker-named directory without repeating its key * s3: emit marker-named dir objects but keep internal folders hidden |
||
|
Jay HemnaniandChris Lu
|
006d4150f0 |
webdav: create a folder under its own name (#11810)
* webdav: create a folder under its own name * webdav: reuse NormalizePath for the MKCOL entry split --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
ab39e3a51b |
Rust volume: stamp ExpireAtSec into the live volume_info on .vif persist (#11813)
The worker fix reads the source's TTL destroy time through ReadVolumeFileStatus, which returns the in-memory volume_info. Go's SaveVolumeInfo stamps volumeInfo.ExpireAtSec before writing; Rust's save_vif wrote it only to the serialized file, so a Rust source kept reporting the stale creation-time value and the EC .vif would carry a wrong expiry. Stamp the live volume_info first, mirroring Go and save_volume_info. |
||
|
Chris Lu
|
8c55425d55 |
s3tables: tighten legacy OIDC subject re-anchoring (#11808)
* sts: recognize issuer-scoped parent user ids
ComputeParentUser output has a fixed shape (base64url SHA-256); let
callers distinguish it from bare subject strings.
* s3tables: keep legacy owner re-anchoring off local and scoped account ids
Re-anchoring a stored bare subject under the single trusted issuer must
only apply to strings that could actually be a legacy OIDC subject. A
stored owner naming the reserved or a configured local account, or an
already issuer-scoped id, can otherwise be claimed by any caller whose
token asserts that string as sub.
* s3tables: re-anchor legacy policy principals in map and array forms
A stored bare subject can appear as {"AWS": sub} or inside a principal
list, not only as a top-level string; re-anchoring only the string form
left legacy Deny statements unable to match the caller.
* s3tables: keep built-in legacy exclusions independent of the account checker
A missing local-account predicate must not disable re-anchoring entirely:
the reserved accounts, ARN-shaped values, and scoped-id shapes are never
OIDC subjects regardless, while legacy Deny statements for plain subjects
still need to resolve so a denied caller cannot fall through to an
identity grant.
* s3tables: exercise legacy Deny against an owner grant
The previous case denied the caller regardless of whether the statement
principal matched; make the caller the owner so a missed match shows.
Cover the array Deny form alongside the AWS map.
|
||
|
Chris Lu
|
88d9926b35 |
s3api: recheck suspended-version condition under the lock when the filer ignores condition_alt_key (#11806)
* s3api: recheck condition under lock when routed filer ignores condition_alt_key A filer that predates condition_alt_key silently drops the field and evaluates a suspended-versioning condition against the empty null slot alone, so a matching If-Match still gets a false 412. Recheck under the object lock instead of trusting that response: the lock path resolves the same entry the read path serves. Genuine precondition failures still return 412 from the recheck. * s3api: cover suspended If-Match against a filer ignoring condition_alt_key * s3api: resend against the resolved entry instead of the lock-path recheck Rechecking under the gateway object lock does not serialize against routed writers, which commit under the filer's path lock; a concurrent routed PUT could land between the recheck and the create and be overwritten with a stale precondition. Resend the same transaction with condition_key set to the resolved version file so the condition and the write stay under the same filer lock. When the resolved entry is the null slot, a delete marker, or unresolvable, the resend targets the object path and fails closed. |
||
|
Chris Lu
|
e596be10f0 |
s3api: evaluate object-tag policy conditions on delete, ACL and lock operations (#11807)
* s3api: resolve the target version in tag-condition policy lookups * s3api: apply tag-conditioned policy to object deletes * s3api: apply tag-conditioned policy to object ACL reads and writes * s3api: apply tag-conditioned policy to retention and legal hold * s3api: cover tag-conditioned delete, ACL and object-lock rechecks * s3api: let tag-conditioned grants redecide denied batch deletes * s3api: resolve the tagged version in delete policy tests |
||
|
Chris Lu
|
2ff7888546 |
filer: honor a storage rule's rack/data-center constraint instead of falling back (#11804)
filer: stop widening a storage rule's location on assign failure |
||
|
Nabil RagabandClaude Opus 5.5
|
6ab6926c94 |
s3: read chunks from HTTPS volume servers with the global HTTP client (#11800)
readChunkRange sent the volume lookup's http:// URL through a bare http.Client, so with [https.client]/[https.volume] configured every read of an SSE-KMS object larger than one chunk (and every ranged read through this path) failed with "client sent an HTTP request to an HTTPS server" and the GET returned 500. Normalise the URL's scheme and use util_http's global client, which carries the [https.client] TLS configuration, as the other volume reads already do. Drop the now-unused volumeServerHTTPClient. Fixes #11799 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris Lu
|
e6a0534e3b |
s3api: make ambiguous routed-PUT recovery atomic and stop transaction replay (#11787)
* filer_pb: map PRECONDITION_FAILED to a sentinel error A conditional CreateEntry that the owner's path lock refuses comes back as FilerError_PRECONDITION_FAILED; mapping it to ErrPreconditionFailed lets the S3 gateway tell a rejected precondition apart from a transport failure and report 412 instead of retrying the write. * s3api: make the ambiguous-PUT recovery create conditional under the lock createAfterAmbiguousRoute verified state and then issued an unconditional create, so a write landing between the lookup and the create was still clobbered by the stale re-commit (issue #11631). The recovery create now carries a WriteCondition — IF_NOT_EXISTS for a proven-absent entry, IF_ENTRY_EQUAL for a directory it may need to write onto — which the owner filer evaluates under its per-path lock, making the check and the write atomic. A refused precondition loops back to re-read the state instead of overwriting whatever won the race. * s3api: do not fail over a routed transaction past a possibly-sent request withFilerClientFailover retried the callback on the next filer on any transport error, so an ObjectTransaction whose response was lost could be re-applied over a write that committed in between (issue #11631). A gRPC error cannot distinguish "never sent" from "committed", but channel readiness can: a non-replayable callback now runs only on a connection that just reached READY, and failover continues only over channels that never came up — the one failure mode that provably sent nothing. * s3api: run ambiguous-route recovery for empty PUTs too The recovery was gated on a nonempty chunk list, so a zero-byte PUT whose routed transaction lost its response still fell back to an unconditional create that could overwrite a newer write and delete its chunks. Chunk-count mismatch on an empty upload list is enough to distinguish a superseding entry from this PUT's — there is nothing to save by skipping the check. * s3api: keep conditional recovery creates off the replay path The conditional create in the ambiguous-PUT fallback rode the replayable WithFilerClient failover, so a committed create whose response was lost could be re-sent to another filer — where a delete in between would let IF_NOT_EXISTS pass again and resurrect an entry pointing at reclaimed chunks. Conditional creates now run through the non-replayable failover; an ambiguous result falls through to confirmCreateLanded, which resolves the committed entry rather than resending. * s3api: do not mark a filer unhealthy for an answered request A reply that reached the callback — including an application-level failure like a refused write precondition — proves the filer is alive. Checking received before the health bookkeeping keeps definitive application errors from tripping the failure circuit or flagging a live routed owner as unreachable. * s3api: do not claim an empty stored entry as an ambiguous PUT's own Two chunkless entries compare equal under sameFileChunks, so an older empty object satisfies "the route landed" even though it predates this write — recovery would return success while leaving the request's Content-Type and metadata unapplied. A landed PUT stores the entry exactly as built, so require mime/size/md5/content and the extended attributes to match before claiming it; anything else stays uncertain and the client retries. * s3api: prove ambiguous empty PUTs by write id and pin recovery to the owner - Conditional recovery creates are sent to the routed owner alone instead of failing over: the condition only serializes with routed writes under the owner's per-path lock, and another filer's forwarding is only as correct as its ring view. An unreachable owner now yields an uncertain result rather than a create under a different serialization boundary. - A proven-absent path no longer satisfies an empty, chunkless PUT: the routed commit may have landed and been deleted, and there are no chunks to verify, so IF_NOT_EXISTS could resurrect a removed object. Recovery returns uncertain for the client to retry. - Chunkless identity moves from metadata equality to a per-write id (x-seaweedfs-write-id) stamped into Extended: an identical older object can match mime/md5/content but not the id. Entries without this write's id are not claimed, and confirmCreateLanded can now positively confirm a committed empty object instead of always reporting failure. |
||
|
Chris Lu
|
e6633146d8 |
filer: shrink zip spool threshold in tests to fix 32-bit OOM (#11801)
* filer: shrink zip spool threshold in tests to fix 32-bit OOM TestSpoolZipReaderTempFile and TestSpoolZipReaderLimit each allocate ~33-70 MiB to cross the 32 MiB in-memory spool threshold. On linux/386 the weed/filer test binary already runs near the 4 GiB address-space limit, so these allocations reliably crash it with "out of memory" mid-suite. Make zipSpoolMemoryLimit a var and drop it to 1 KiB in those tests, which exercises the same memory-vs-temp-file spool paths with KiB-sized buffers. * s3 lifecycle: update stale runShard and WalkerFunc call sites in tests Recent master changes added a walks param to runShard (#11797) and made WalkerFunc take []int, but replay_view_test.go and run_bounded_test.go still use the old signatures, breaking go vet and test builds on merge. |
||
|
Chris Lu
|
fe8221c453 |
fs.meta.load: wait for in-flight entry creation before returning (#11776)
* fs.meta.load: wait for in-flight entry creation before returning On io.EOF and error paths the command returned while CreateEntry goroutines were still running, so trailing entries could be silently dropped once the shell exited. Wait for them on every exit path and guard the shared error with a mutex. * fs.meta.load: surface worker errors on read/unmarshal failure paths The ReadFull and proto.Unmarshal error paths waited for in-flight CreateEntry workers but returned only the local error, hiding a failed create from the caller. Check loadErr after wg.Wait() like the other exit paths so the worker error is reported. Extract the load loop into loadEntries so the concurrency and error propagation can be exercised by unit tests, and add regression tests covering delayed creates at EOF and on corrupt input plus worker error reporting. |
||
|
github-actions[bot]
|
e32d8ca2a6 | docs: regenerate star history chart | ||
|
Chris Lu
|
602ddb31f2 |
s3: count stored versions as bucket contents on delete (#11775)
* s3: count stored versions as bucket contents A <key>.versions directory holding version entries is user data; skipping it let bucketHasUserObjects report an empty bucket while the recursive delete dropped every stored version. * test: rename stale reserved-folders case in bucket delete test .versions directories are now descended into, so the case only holds because the folder is empty. |
||
|
Chris Lu
|
44bb697495 |
s3api: hydrate runtime policies on propagated IAM updates (#11784)
* s3api: hydrate runtime policies on pushed identity upserts PutIdentity propagation (used on every IAM write when the credential store has a master client) carries the persisted identity proto. That proto records inline policies only as flattened legacy Actions; their runtime __inline_policy__ names live in the policy store and were dropped by the merge. For the interval until the next full reload, an identity whose policy denied e.g. DeleteObject routed to legacy CanDo, where "Write:bucket" authorizes deletes. Hydrate the pushed identity the same way a full load does, so the merged state always carries its runtime policy names and documents. * s3api: hydrate runtime policies on pushed group upserts PutGroup propagation has the same gap as identity pushes: the wire proto does not carry the group's runtime __inline_group_policy__ names, so members' attached-policy check lost them until the next reload. * s3api: attach only runtime inline policy names on pushes Listing the pushed identity's or group's own inline policies is enough to keep it on the attached-policy route; a whole-store hydration per pushed proto is not. Pushes now never write iam.policies, so a stale read can no longer overwrite a newer committed policy document. * s3api: carry inline policy documents on pushed upserts Attaching only runtime policy names left a window where a pushed identity or group referenced a document the local cache had never loaded, denying allowed requests until the next reload. Fetch the pushed entity own inline documents and publish them with the update; reads stay scoped to the pushed entity rather than the whole store, and inline runtime names are only ever populated from this store so a read cannot install a document older than the committed one. Resync the IAM manager only when a push actually installs documents. * s3api: order pushed inline policy reads against their installs A pushed UpsertIdentity or PutGroup read inline policy documents from the credential store and installed them without holding the ordering guard that full reloads already take, so a delayed push could overwrite a newer committed document with its stale read and restore a revoked grant until the next update or reload. Pushes now hold reloadMu across the store read and the install, the same snapshot-through-commit serialization the credential-manager load uses. MergeS3ApiConfiguration also applies config.Policies to the live iam.policies at commit instead of swapping in its earlier snapshot, so a policy installed while a merge prepared (a pushed inline document or a PutPolicy update) is no longer discarded. |
||
|
|
74ed12d6b3 |
s3 lifecycle: replay only the rules that fit the retention window (#11789)
* s3 lifecycle: replay only the rules that fit the retention window runShard handed drainShardEvents the full snapshot, so rules the retention window assigns to the walk were also matched during meta-log replay. Their matches are not due yet, and drainShardEvents freezes the cursor at the first event with a not-yet-due match. With a 3-day and a 30-day expiration rule on one bucket and RetentionWindow of 8 days, every shard's cursor stopped at the first 30-day object's PUT and stayed there until that object came due. Each pass then re-read the log from that point and re-dispatched every 3-day expiration already handled, which surfaces as NOOP_RESOLVED dispatches, a growing cursor_lag_max, and passes that still report status=ok. Snapshot.ReplayView drops the replay-kind actions promoted to the walk and keeps every other action as compiled, and the drain uses it. Promoted rules are served by the walk as before. RulesForShard, PromotedHash and ReplayView share one isPromoted predicate. With RetentionWindow unset no active rule is promoted, so behaviour without a configured window is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: scale the replay-view test with DaysToDuration The test took its retention window from DaysToDuration but dated its PUTs in hours, so under the s3tests build tag (one lifecycle day = 10s) every event fell before the seeded cursor and replay never ran. Express the event ages in DaysToDuration too, so the test exercises replay at both scales. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: rehabilitate scan_only mode in the replay drain view ReplayView shared the base CompiledAction pointers, so a replay-kind rule locked into ModeScanOnly by a prior compile stayed invisible to router.Route's event_driven gate even when the runtime retention window kept it in the replay partition - and it is absent from the walk view too, so the rule would be served by neither. Clone non-promoted replay-kind actions the way RulesForShard does (Mode forced to event_driven), reactivating the scan_only case whose bootstrap rendezvous already completed, while preserving the base active bit for genuinely pre-bootstrap actions. Disabled actions are excluded for parity with the other views, and DESIGN.md now documents the view. * s3 lifecycle: keep drain activation observable, require walker with retention ReplayView now shares pointers for event-driven actions so a mid-pass MarkActive flip remains visible to the drain; only a scan_only-locked action is cloned to event_driven, and reactivated only when its bootstrap rendezvous completed (tracked via a new bootstrapComplete field on CompiledAction). validate() rejects RetentionWindow without Walker: promoted rules would lose their only dispatch path while the drain cursor still advanced past their events. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
443ae1cd83 |
s3 lifecycle: share daily bucket walk and prune listings (#11797)
* s3 lifecycle: share the daily bucket walk across shards The daily pass invoked the walker once per shard, and every shard's walk listed the full bucket tree while keeping only its own 1/16 of the entries — 16 full listings per pass by default (#11791). Run now groups the shards by the walk branch they'll take (recovery vs steady, using the same cursor data and hash checks runShard applies) and starts one shared walk per group, so a pass lists each bucket at most twice. WalkerFunc and WalkBuckets take a shard set instead of a single shard. runShard keeps its own decision and waits on its group's walk; a shard whose freshly-loaded cursor disagrees with the plan falls back to a per-shard walk, so mid-pass cursor rewrites can't under-cover a shard. * s3 lifecycle: skip the .versions/ pre-pass on non-versioned buckets walkBucketTree listed every directory twice — once looking for .versions/ dirs, once for regular entries. On a bucket that never had versioning enabled the first pass can only find coincidentally-named user folders, so the extra listing is pure cost (#11791). FilerListFunc now takes the walk view and consults Snapshot.BucketVersioned; non-versioned buckets skip pass 1 entirely. A *-suffixed .versions user folder on such a bucket is recursed into as a regular directory so real objects inside still surface; a nil view keeps the two-pass behavior. * s3 lifecycle: prune walk directories to active rule prefixes The daily walk listed every directory under a bucket even when no rule prefix could reach it. Collect the active rule prefixes of each bucket from the walk view (new Snapshot.BucketPrefixes) and skip recursing into directories whose keys can never match; a bucket with no active actions is skipped entirely, saving the listing. .uploads stays in scope regardless of prefixes because AbortMPU rules match DestKey, and .versions expansion is pruned only conservatively (the logical key is a string-prefix of the dir key). * s3 lifecycle: join planned walks before Run returns planWalks starts a shared walk goroutine per group, but a shard whose authoritative cursor load then fails exits runShard without waiting on it. Run returned while the walk still listed or dispatched deletes against connections the caller closes on return. Join every planned walk after the shard waitgroup. * s3 lifecycle: treat object-locked buckets as versioned in the walk Object Lock requires versioning and every writer sets both extended attributes, but IsBucketVersioned only read ExtVersioningKey. A locked bucket missing that attribute would have its .versions dirs walked as ordinary folders, hiding version siblings from lifecycle. Classify ExtObjectLockEnabledKey as versioned too. |
||
|
Chris Lu
|
bc29ffbfbd |
helm: apply placement values to the chart's hook Jobs (#11795)
* helm: honor placement values in the bucket creation hook The post-install/post-upgrade bucket Job ignored nodeSelector, tolerations and affinity, so it could land on nodes the operator kept clear, or fail to schedule at all on a fully tainted cluster. The hook now accepts nodeSelector, tolerations and affinity under each createBucketsHook (filer.s3, s3, allInOne.s3), read from the same values section that supplies createBuckets. Unset nodeSelector and tolerations fall back to the workload the hook already borrows its security context from; affinity is explicit-only since the anti-affinity rules of a long-lived server do not suit a one-off Job. * helm: honor placement values in the volume resize hook The pre-install/pre-upgrade resize Job inherited nothing from the volume server, so on a cluster whose storage nodes are all tainted the Job could not be scheduled and the upgrade timed out. The hook now falls back to the volume server's nodeSelector and tolerations, and accepts explicit nodeSelector, tolerations and affinity under volume.resizeHook. Affinity stays explicit-only: the default podAntiAffinity against component=volume would keep the Job off every node already running a volume pod, which is most of them during an upgrade. Placement strings are rendered through tpl with $volumeName bound, same as the StatefulSet. |