mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-17 20:10:51 +02:00
ac03d3fd78b3b41b402fcadbed468aa3f63e8985
741
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
adaf3534fa |
rust: clippy-clean both crates and adopt the std APIs the 1.91 MSRV allows (#11312)
* rust: apply clippy --fix to both crates The mechanical part of a clippy sweep: `cargo clippy --all-targets --fix` on seaweed-volume and the seaweed-worker workspace, hand-reviewed. Both manifests declare their MSRV (1.91.1 and 1.94.1), so every suggestion clippy applied is within it: the collapsible_if sites become let chains (1.88, edition 2024), `% n == 0` becomes is_multiple_of (1.87), chunks_exact with a constant becomes as_chunks (1.88), repeat().take() becomes repeat_n (1.82), and io::Error::new(Other, ..) becomes io::Error::other (1.74). The rest is redundant clones, borrows, casts, closures and field names. Nothing here changes behaviour. The three let_and_return sites in needle_map.rs and store_ec.rs deserve a note: the `let result = ..; result` shape was a deliberate edition-2021 workaround to drop a redb guard before the table it borrows. Edition 2024 drops tail-expression temporaries before locals, which is why clippy now flags it, and the two comments that described the workaround say so instead. Manual edits on top of the tool output: the blocks clippy rewrote are re-indented the way rustfmt lays them out (only those blocks — the crate is not rustfmt-clean and a whole-crate fmt would bury this diff), the blank lines let_and_return left behind are removed, and the CRC legacy_value test compares against a literal worked out from the original shift formula rather than restating rotate_right. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust: clear the clippy warnings --fix cannot apply, and say why the rest stay Hand fixes for the lints clippy only reports. Behaviour is unchanged throughout; each rewrite is the one clippy names. - needless_range_loop (7): index loops over shard vectors become iterator loops. Where the old code indexed `v[..n]` the new loop iterates `v[..n]` so an undersized vector still panics the same way. - field_reassign_with_default (6): struct literals with `..Default`. - redundant_pattern_matching (3): `if let Err(_) = guard.check()` becomes `.is_err()`, which also releases the read guard at the end of the condition instead of at the end of the block. - manual_strip (2), manual_checked_ops, format_in_format_args, redundant_locals, wrong_self_convention (to_vif takes self by value, so it is into_vif; CompactEntry is Copy, so to_needle_value takes self). - type_complexity (2): `OrphanShardLoad` and `RawNeedleEntry` name two tuples that were spelled out inline. - new_without_default: CompactNeedleMap gets a Default that calls new(). - suspicious_open_options: a test helper spells out `.truncate(false)`, which is what `.create(true).write(true)` already did. What stays, and the attribute that says so: - too_many_arguments (10): `#[expect]` on each function. Folding 8–15 parameters into a struct is a design change, not a lint fix. - await_holding_lock / readonly_write_lock: one test holds the store write guard across a sleep on purpose, as a barrier that parks the copy task at the mount block. `#[expect(.., reason = ..)]` records it. - module_inception: needle/needle.rs mirrors the Go package layout. Two lints become crate-wide policy in `[lints.clippy]`, with the reason next to each: result_large_err, because every RPC path returns tonic::Status (176 bytes) and boxing it would change every handler signature; and needless_update, because `..Default::default()` on a protobuf message literal is what lets a proto gain a field without touching every constructor (all 11 sites are pb messages). The worker workspace gets the same table and its members opt in with `lints.workspace = true`; its generated plugin.rs also allows large_enum_variant on prost's oneof enums. Both crates are now clean under `cargo clippy --all-targets -- -D warnings`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust volume: use the std APIs the 1.91 MSRV already pays for The crate declares rust-version 1.91.1, so a few things the code still worked around are plain std now. All of them come from the 1.85–1.91 release notes; nothing here needs a newer toolchain than the manifest already requires. - std::sync::LazyLock (1.80) replaces the lazy_static! block in metrics.rs, and the lazy_static dependency goes. Every use site reads the same through Deref, so no caller changes. - Duration::from_mins / from_hours (1.91) replace `from_secs(v * 60)` and `from_secs(v * 3600)` in the option parser and the shard-location refresh TTLs. One difference for the parser: an absurd count that overflows u64 seconds now panics in release builds too, where the multiplication used to wrap. - Result::flatten (1.89) replaces `.and_then(|r| r)` on the replication join handle. - OsStr::display (1.87) replaces `to_string_lossy()` where the name was only being formatted; the output is byte-identical. - `#[allow]` becomes `#[expect]` (1.81) on the suppressions that are meant to be permanent, so a suppression that stops being needed becomes a warning rather than lingering. Doing that found four that already had: dead_code on ChunkManifest, base_name and last_io_error, and too_many_arguments on read_from_data_shards, which is down to seven parameters. Those attributes are deleted. The three allows that depend on cfg (a unix-only mutation, a linux-only field set, a profiling-only parameter) stay as allow, because expect would be unfulfilled on the other platforms. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * ci: add a commented-out clippy step to both Rust workflows Both crates are warning-free under `cargo clippy --all-targets -D warnings` now. Whether that becomes a gate is a policy call, so the step is present but commented out; uncommenting it is the whole change. The comment points at the `[lints.clippy]` table where crate-wide exceptions are recorded, so the gate does not become a reason to sprinkle allows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust volume: guard parse_duration against overflow panics Duration::from_mins/from_hours panic when the count overflows u64 seconds. Use checked_mul so an oversized CLI value falls back to the parser default instead of crashing volume startup. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
dependabot[bot]
|
fd33c07843 |
build(deps): bump github/codeql-action from 4.37.9 to 4.38.0 (#11311)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.9 to 4.38.0. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v4.37.9...v4.38.0) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.38.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
f1f6886d0e |
fix: rebase on latest master before pushing star history chart (#11240)
* fix: rebase on latest master before pushing star history chart The daily star history workflow git push was rejected with a non-fast-forward error because new commits landed on master between the checkout and the push. Fetch full history (fetch-depth: 0) and rebase the generated commit on top of the latest remote branch before pushing so the workflow no longer fails when master has moved. * fix: retry rebase-and-push to handle concurrent master updates Address review feedback: a one-shot rebase still races if master advances between the rebase and the push. Match the bounded retry loop used by java_release.yml — push first, and on rejection rebase and retry up to five times before failing. * fix: serialize runs and ensure every rebase is followed by a push Address review feedback: - Devin (line 54-55): the old loop rebased after the 5th failed push but never pushed the rebased commit. Restructure so each rebase (attempts 2-5) is followed by a push attempt, with a clear 5-attempt cap. - Greptile: overlapping runs could conflict on the SVG during rebase. Add a concurrency group (cancel-in-progress: true, matching the repo convention) so only one chart regeneration runs at a time. * fix: scope concurrency by ref and guard rebase against transient failures Address review feedback: - Devin (line 16): the global star-history concurrency group let a manual run on another branch cancel an in-flight daily master update. Scope the group by github.ref so only same-branch runs cancel each other. - Greptile (line 58): git pull --rebase runs under the fail-fast shell, so a transient fetch error or conflict aborted the whole step before remaining attempts ran. Guard the rebase so a failure aborts the in-progress rebase and continues to the next attempt instead. |
||
|
Chris Lu
|
0ce5ca42ea |
helm: supply admin auth in CI renders that enable admin (#11239)
PR #11236 added a render-time guard that fails the chart when admin.ip is non-loopback (default 0.0.0.0) and admin auth is not configured, since weed admin 4.46 refuses to bind a non-loopback address without authentication. Several pre-existing helm_ci.yml test cases enable admin.enabled=true as part of "everything on" renders without a password, so helm template now exits non-zero and the Verify template rendering step fails. Add admin.secret.adminPassword to the four render calls that turn on admin without auth (IAM gRPC opt-in, NetworkPolicy EVERYTHING, egress without kubeApiServer.cidrs, and the license ALL_ON dict), using the same key ci/admin-values.yaml already uses. |
||
|
Chris LuandCopilot
|
213f4c5d5c |
release: wait for Go proxy propagation (#11219)
Allow normal post-tag proxy propagation before dispatching downstream releases, while preserving the check that prevents them from pinning the previous commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
dependabot[bot]
|
34936c610d |
build(deps): bump helm/kind-action from 1.14.0 to 1.15.0 (#11212)
Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.14.0 to 1.15.0. - [Release notes](https://github.com/helm/kind-action/releases) - [Commits](https://github.com/helm/kind-action/compare/v1.14.0...v1.15.0) --- updated-dependencies: - dependency-name: helm/kind-action dependency-version: 1.15.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
5e3af51d91 |
build(deps): bump docker/setup-qemu-action from 4.2.0 to 4.3.0 (#11213)
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 4.2.0 to 4.3.0. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](https://github.com/docker/setup-qemu-action/compare/v4.2.0...v4.3.0) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-version: 4.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Eliah Rusin
|
ed5b342f0c |
rust volume: optional redb insert_before bulk load (#11205)
* rust volume: quick-repair redb on durable checkpoints set_quick_repair(true) on the durable checkpoint transaction so an OOM-killed volume server opens without a full-file repair scan. * rust volume: optional redb insert_before bulk load Behind redb-experimental-cursor (default off). Production binary stays on sorted insert(). CI unit tests run both feature settings. * rust volume: exercise insert_before across leaf splits Replace the 5-key cfg clone with a 4000-key reverse-order rebuild so CursorMut::insert_before hits page splits. CI runs the feature only on storage::needle_map unit tests. |
||
|
Chris Lu
|
d8a40ef750 |
ci: pin actions/setup-python to v7 in star_history workflow (#11191)
The star_history workflow referenced actions/setup-python@v8, which does not exist, causing the workflow to fail at the "Set up job" step. Pin to v7, matching the version used across the other workflows. |
||
|
Chris Lu
|
97154802c5 |
docs(star-history): make the chart taller (#11173)
Change the matplotlib figure size from (10, 4) to (10, 6) so the star history chart renders vertically longer in the README. The regenerated note/star_history.svg reflects the new 5:3 aspect ratio (720x432pt) instead of the previous flat 2.5:1 (720x288pt). |
||
|
Chris Lu
|
ff0d5a9adf |
ci(mount-windows): clean up processes and don't let Logs step fail job (#11174)
The "Mount and exercise" step left the weed.exe mini server and the final WinFsp mount running when it exited. The next step's pwsh.exe then failed with STATUS_DLL_INIT_FAILED (0xC0000142), failing a job whose actual test step had passed. The same code passed on both the PR branch and the next master run, so this was a transient launch failure — but it was caused by an unclean environment and made fatal by a diagnostic step. Tear down all weed.exe processes at the end of the test step so subsequent steps launch into a clean environment, and mark the Logs step continue-on-error so a diagnostic step can never fail the job on its own. |
||
|
Chris Lu
|
567578f08d |
docs(readme): replace star-history.com with self-generated chart (#11171)
* docs(readme): replace star-history.com with self-generated chart The star-history.com SVG is a third-party dependency that can rate limit or go down. Replace it with a GitHub Action that fetches stargazers via the REST API and renders an SVG with matplotlib, committing note/star_history.svg weekly. The README references the committed file directly, so the chart has no runtime dependency on any external service. * ci(star-history): run daily instead of weekly |
||
|
Chris Lu
|
5a515adab2 |
s3: HeadObject with partNumber returns the part's size and 206 (#11166)
* s3: HEAD with partNumber reports the part's size and range HeadObject set its headers from the total object size and then only validated the partNumber, so a client probing part 1 with HEAD got the whole object's Content-Length and a 200 while the same GET returned the part's size, a Content-Range and a 206. Resolve the part's byte range before the headers are written, through the range logic GetObject already used, and answer a partNumber HEAD as the ranged HEAD that AWS documents. * s3: answer an unsatisfiable partNumber with 416 InvalidPartNumber GET and HEAD rejected a partNumber past the number of parts with 400 InvalidPart, the code for a missing part in CompleteMultipartUpload. AWS answers a read of a part that does not exist with 416 InvalidPartNumber, which lets a client probing for the part count tell the two apart. The ceph suite pins RGW's 400 InvalidPart here, so the s3tests jobs patch that expectation the way they already patch prefix ordering. * s3: keep the whole-object checksum off a partNumber response The stored checksum covers the whole object, so it is already withheld from a ranged read. A partNumber HEAD now describes one part while the request carries no Range header, so exclude it there too rather than handing a client a checksum that does not match the bytes described. * s3: resolve a partNumber against the parts the object records Completion accepts ascending, not consecutive, part numbers, so the part count is not the highest part number. Comparing the two rejected an uploaded part 3 of a two-part object, and let a request for the absent part 2 fall through to the positional chunk lookup and serve part 3's bytes. Ask the recorded boundaries for the part instead, and keep the count comparison for objects written before boundaries were stored. * s3: apply a client Range within the part on HEAD too GET narrowed the part by a Range sent alongside partNumber; HEAD reported the whole part, so the two disagreed again for a request that carries both. Move the narrowing into the shared range lookup so either verb describes the same bytes. |
||
|
Chris LuandDevin
|
57a5285020 |
ci: turn off cosign's signing config alongside the bundle format (#11144)
Every image-signing job has failed since signing was added (#11129): must provide --new-bundle-format or --bundle where applicable with --signing-config or --use-signing-config Cosign 3 turned on two defaults, not one. The action only disabled --new-bundle-format to keep the .sig tag layout, but --use-signing-config is still on, and cosign refuses that pairing because the signing-config path has nowhere to write its verification material without a bundle. Disabling it too falls back to the default Fulcio and Rekor URLs, the same services the .sig layout always used. The verify step needs no change: cosign verify looks for a referrer bundle first and falls back to the .sig tag when there is none. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
1205630c43 |
ci: track the actions in the signing jobs by tag again (#11137)
The repository tracks actions by tag with dependabot; the signing jobs follow the same convention. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa |
||
|
Chris Lu
|
a8f763e717 |
Sign the published Docker images with cosign (#11129)
* ci: composite action that signs and verifies an image with cosign Keyless, by digest, with a verification pass against the calling workflow's own identity right after signing. Signatures use the .sig tag layout rather than the OCI-referrer bundle cosign 3 writes by default, since that is what the verifiers people run today read. Dependabot is pointed at the action so the cosign-installer pin keeps moving. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker release: sign every variant on both registries The merge job signs each variant's multi-arch index on GHCR and Docker Hub once the tag exists, recursively so the platform images are covered too. latest re-tags the same manifest and inherits the signature. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker dev: sign the dev image Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker latest: sign a latest rebuilt by hand Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker release: sign the foundationdb image Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker: sign the per-version foundationdb and rocksdb builds They push to the same repository as the releases, so an admission policy that verifies chrislusf/seaweedfs would otherwise reject them. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker: document image signature verification Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * ci: pin the actions the signing jobs newly run by commit These run with registry credentials and the OIDC token that signs under the repository's identity, so a retargeted tag upstream must not be able to reach them. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker latest: pass the dispatch tag through env, not the script Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker: complete Kyverno policy, digest note, identity scope Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker latest: keep the dispatch tag out of the manifest script too The step predates signing, but the job now holds the OIDC identity. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * ci: pin every action in the jobs that sign The jobs that hold the OIDC identity run these with registry credentials, so a retargeted tag upstream must not reach them. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker release: copy and sign the digest the run created, pin the rest crane copy and the signature both resolved the tag, which another publisher could move between the two steps. The index digest is read once, right after it is created, and the Docker Hub copy and both signatures use it. The manual latest rebuild gets the same treatment. The actions in these jobs are pinned to commits, crane to v0.22.0 by checksum, and the sparse checkout no longer keeps the token. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker latest: the signing job checks out the workflow's own commit The job only assembles and signs manifests, so nothing there needs the source_ref checkout; the local signing action now comes from the same revision as the workflow file that calls it. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker release: take the index digest from the create result imagetools create writes the descriptor it pushed with --metadata-file (buildx 0.32+, the runners ship 0.36), so the digest no longer comes from re-resolving the tag even within the same step. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa |
||
|
Chris Lu
|
59916d8978 |
helm: values-driven labels on every ingress (#11127)
* helm: values-driven labels on every ingress Each ingress already takes annotations from values, but its labels were a fixed block, so tools that select ingresses by label (ExternalDNS label filters, for one) had nothing to key on. Every ingress block now has a labels map rendered after the standard app.kubernetes.io labels, including the Traefik IngressRouteTCP that shares the filer gRPC values. Claude-Session: https://claude.ai/code/session_01L6eJGXtYkwe1W9QjGeUgr1 * helm ci: render check for ingress labels Claude-Session: https://claude.ai/code/session_01L6eJGXtYkwe1W9QjGeUgr1 |
||
|
Chris Lu
|
721499a05a |
release: judge downstream releases by their run, excluding runner-queue time (#11057)
Claude-Session: https://claude.ai/code/session_01Y228KU8MLsGmfpcbGxgjwh |
||
|
dependabot[bot]
|
8c3695fd4a |
build(deps): bump actions/setup-java from 5 to 6 (#11047)
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 5 to 6. - [Release notes](https://github.com/actions/setup-java/releases) - [Commits](https://github.com/actions/setup-java/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/setup-java dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
3a3d513cb8 |
build(deps): bump github/codeql-action from 4.37.8 to 4.37.9 (#11045)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.8 to 4.37.9. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v4.37.8...v4.37.9) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.9 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
dfa75ce231 |
build(deps): bump actions/upload-artifact from 4 to 7 (#10922)
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 7. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v4...v7) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
61d588e455 |
build(deps): bump github/codeql-action from 4.37.6 to 4.37.8 (#10923)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.6 to 4.37.8. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v4.37.6...v4.37.8) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.8 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
863fec6c3f |
S3: let a key that is a prefix of other keys be an object (#10912)
* filer: keep the sentinel when CreateEntry reports an update failure CreateEntry flattened the error UpdateEntry wraps, so errors.Is stopped matching and ErrExistingIsDirectory and ErrExistingIsFile never reached the S3 mapper, which answered a retryable 500 instead. * s3: let a key that is a prefix of other keys be an object S3 keys are flat, so "a/b" and "a/b/c" are independent objects that coexist in either write order. The filer stores a key as a path, so one of them has to live on the directory the other is nested under. Writing the nested key first refused the prefix key outright. Writing it second promoted the file to a directory, which kept its data but lost the key: an empty object left nothing to recognise it by and disappeared, and one with data listed under a trailing slash it never had. Mark the directory that carries such a key, and write the object onto it when the path is already a directory. The mark makes an empty prefix object visible to listings and readable by GET and HEAD, keeps the empty folder cleaner off it, and lists it under the key it was written with. Deleting the key strips the mark back off along with the data. * filer: keep a TTL off a directory that stands for an object An expired entry is deleted a row at a time, so expiring a directory removes it and leaves everything under it unreachable. Promoting a file to a directory carried its TTL across, and a promoted file is exactly the one that has keys nested under it. Drop the TTL on promotion, and leave one an older build wrote alone. The lifecycle worker still expires the object, through the delete that leaves the directory behind. * s3: delete the null version of a key other keys are nested under The routed delete cannot remove an entry that other keys live under, and answered a retryable 500 rather than falling back to the lock path the unversioned delete already falls back to. That path then looked the entry up under the bucket with the whole key as its name, so the demote wrote it back one directory too high and failed as not found. Fall back on any non-precondition error, and split the key before deleting it. Trailing-slash directory markers with children reach the same delete. * filer: keep the sentinel when MkFile and Mkdir report a create failure Same flattening one layer out: every mkFile caller lost the sentinel, so a CopyObject onto a key that other keys are nested under answered a retryable 500 where a PutObject of the same key answers 409. * s3: copy and rename a key that other keys are nested under Such a key is stored on the directory those keys live in, and copy and rename both refused it: the source lookup maps every directory entry to NoSuchKey, so a key a plain GET serves could not be copied or moved, and the destination side refused it as a directory conflict. The source is read through a view of the entry as the object it names. The destination is written the way a PutObject of that key writes it. A rename at either end copies the object's own data across and strips it off the source key rather than going through AtomicRenameEntry, which moves a directory by moving everything under it - the nested keys are not part of what is being renamed. |
||
|
Chris Lu
|
7ebf2ebac3 |
Build the Rust worker against the protoc that ships with the build (#10881)
* worker: compile plugin.proto with the protoc that ships with the build seaweed-volume already does this: protoc-bin-vendored carries the binary, so the build needs no package manager and every build sees the same version. An explicit PROTOC still wins, which is what lets the lance crates - whose own build scripts read the same variable - share it. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: point the worker builds at the vendored protoc The jobs installed protobuf-compiler for lance's build scripts. They read PROTOC, so pointing it at the binary protoc-bin-vendored already puts in the registry serves them without a system package - one less apt call on the way to a release, and the same protoc a developer's build uses. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docs: say what the worker build needs from protoc The lance crates' build scripts are the ones that need it, not ours, and they take the same vendored binary. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris Lu
|
cc8364a03e |
Ship the Rust maintenance worker with the release (#10879)
* worker: name the binary weed-worker It is the Rust side of `weed worker`, the way weed-volume is the Rust side of `weed volume`, and lance is the first family of jobs it carries rather than the only one it ever will. The crate keeps its own name: when a second family arrives the bin target moves to a crate of its own, under this name. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docker: ship the Rust maintenance worker in the image Lance table buckets need a worker that can read the format, and until now the only way to get one was a Rust toolchain and a cargo build. It now sits at /usr/bin/weed-worker beside the Rust volume server, reached as `docker run chrislusf/seaweedfs worker-rust --admin host:23646` — the verb mirrors volume-rust, so plain `worker` still runs the Go one. Taken pre-built or not at all: the lance jobs pull in arrow and datafusion, far too large a tree to compile inside the image build, so an architecture CI did not build for gets the empty placeholder the entrypoint refuses to exec, the way the Rust volume server already does. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: build the Rust worker for the container images The same native cross-compile the volume server uses, so the release, latest and dev images all carry it on amd64 and arm64. The artifact holds both binaries now, so it is named for that rather than for the volume server. Only the release directory each job builds is cached: with a debug profile beside it the worker's target/ reaches 24GB, against a 10GB cache budget. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: publish Rust worker binaries with the release Linux amd64 and arm64 only: the worker runs beside the cluster it maintains, and its dependency tree makes every extra target an expensive build. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: build and test the Rust workers on change Nothing built seaweed-worker in CI, so the release and the container images would have been the first place a break showed up. Tests run in release too, rather than compiling lance, arrow and datafusion again in another profile. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docs: say how to get a released worker Neither the image nor the release tarballs were mentioned; a toolchain and a cargo build read as the only way in. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: install protoc for the Rust worker builds lance's crates compile their own protos, and unlike seaweed-volume they do not vendor a protoc to do it with, so every job that builds the worker failed at lance-encoding's build script. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: do not persist credentials in the worker release checkout The upload step is handed a token explicitly; a cargo build script should not find another one sitting in the checkout's git config. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docker: keep the worker's argument boundaries Unquoted $@ splits on whitespace and expands globs, so an argument carrying either arrived as something else. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris Lu
|
c3f4d799b5 |
helm: install chart CI against an image tag that exists (#10877)
* helm: install chart CI against an image tag that exists The release bumps appVersion on master well before the container build publishes that tag, and the chart CI runs on the bump commit, so every release turns it red with ImagePullBackOff. Resolve the tag first and fall back to latest while the new one is still building. * helm: run the chart CI when the workflow itself changes * helm: bound the registry lookup in the chart CI An unbounded curl can hold the job, and the log did not say why the tag was rejected. Cap it and print the status. |
||
|
Chris Lu
|
34bb444f33 |
test: drive the Lance namespace with Spark (#10864)
* test: drive the Lance namespace with Spark
The counterpart of catalog_spark, which does this for the Iceberg REST
catalog. Spark is the engine most likely to be pointed at a lakehouse,
and it reaches the Lance catalog through the connector's DSV2 catalog -
org.lance.spark.LanceNamespaceSparkCatalog with impl=rest - over the same
routes every other client uses.
SHOW NAMESPACES -> ['`sparklance-lcephd80`.ml']
SHOW TABLES -> ['sparklance-lcephd80$ml$embeddings']
count -> 3
filtered -> [(2, 'two'), (3, 'three')]
count after a second commit -> 4
The second insert is there on purpose: a store that cannot order commits
fails on the second one, not the first.
Two things the run settled that were guesses beforehand. CREATE TABLE
works, because the connector declares through the namespace and writes the
data itself rather than pushing Arrow at the server. And SHOW TABLES
returns the namespace's own identifiers - bucket, namespace and name
joined by the delimiter - not bare Spark table names.
Credentials go under the catalog's storage.* prefix, which is handed to
lance as object_store options; a gateway without STS vends none, the same
trap the LanceDB suite documents.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame Spark for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: keep the ivy cache under the user's cache directory
It is mounted into a container running as root, so a shared temp path lets
another local user pre-create it and choose what Spark loads.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: assert the vector column's type, not only its name
A column that came back as array<double> or array<string> would still be
called vector and still pass.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: read the dataset off its location for real
The catalog being optional is the property that lets duckdb and pandas read
these tables; it was asserted in a comment and printed, never exercised.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the Spark Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: say the hosts in the README are placeholders
The suite passes dynamically allocated host.docker.internal ports.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
|
||
|
Chris Lu
|
4af6798639 |
helm: render the mysql secret and env only for the mysql filer store (#10872)
The db credential secret and the filer's WEED_MYSQL_* env were gated on filer.enabled alone, so a filer on mongodb, redis, postgres or leveldb2 got a generated mysql secret it never reads - kept forever by resource-policy: keep - plus a mysql-db-host pointing nowhere. Gate all of it on WEED_MYSQL_ENABLED, which is how the store is selected, plain keys and secret-backed ones alike. An enable flag the chart cannot read - a valueFrom, or one in secretExtraEnvironmentVars - counts as selected, so nothing is dropped from a filer that is actually on mysql. |
||
|
Chris Lu
|
96304b6870 |
S3: source config credentials from the environment, and let the chart point at an existing secret (#10868)
* s3: resolve ${VAR} in static config credentials from the environment
A deployment that keeps its S3 keys in a secret store had no way to hand
them to the gateway: -config takes a file, so the keys had to be written
into that file. Let a key in the static config name an environment
variable instead, and drop any credential whose reference stays unset so
the placeholder never becomes a usable key.
* helm: source the generated s3 identities from an existing secret
The only way to reuse credentials that already live in a Secret was to
hand-author the whole seaweedfs_s3_config JSON, since the literal keys in
values.yaml end up in git and a lookup-based keyRef renders empty under
helm template and Argo CD. Let s3.credentials.admin/read name a Secret and
its keys instead: the generated config references them as ${VAR} and the
gateway resolves them from the environment, so nothing is read from the
cluster at render time.
* s3: treat an empty environment value as an unresolved credential reference
A secret store can hand over a key that exists but is blank. Resolving it
would leave an access key whose signing secret is empty, so count it as
unresolved and drop the credential.
* helm: render the s3 secret when only the all-in-one auth flag is set
The all-in-one deployment mounts the s3 secret whenever any of the three
enableAuth flags is set, but the secret itself only rendered for the s3 and
filer flags, so allInOne.s3.enableAuth on its own left the pod waiting on a
secret nothing creates.
* helm ci: check the credential wiring on every workload that mounts it
The render check only looked at the standalone s3 deployment and only at
one of the four variables, so a helper that bound a variable to the wrong
secret key would still pass.
* helm: create the all-in-one s3 secret for every flag that mounts it
The all-in-one pod mounts the secret on any of the three enableAuth flags,
so keying its creation off allInOne.s3.enableAuth alone still left
filer.s3.enableAuth without filer.s3.enabled pointing at a secret nothing
creates. Mirror the deployment's own condition instead, and check each
flag renders both the mount and the secret.
* s3: reject a malformed credential reference instead of keying on it
A typo such as ${MY-VAR} matches no substitution, so it survived expansion
and the placeholder itself became the access key the gateway accepted.
Require every ${ in a static credential to open a well-formed reference.
|
||
|
Chris Lu
|
480795d40d |
release: cut the whole release from the version bump workflow (#10870)
* release: cut the whole release from the version bump workflow The bump workflow stopped after pushing the version commit, and the rest was manual: create the release, then run "Prepare release" in the csi-driver and the operator. It now pushes the tag itself, which is what starts the binary, container and helm workflows, creates the release with generated notes, and dispatches the other two repositories, waiting for both. Pushing the tag and reaching the other repositories both need RELEASE_PAT; GITHUB_TOKEN raises no events that start workflows. * release: tighten the release workflow after review Check out master explicitly: a dispatch can select any branch, and the tag, the commit and the release would then come off that branch while the downstream job dispatches master. Scope contents:write to the job that pushes; the downstream job talks to the other repositories with RELEASE_PAT and needs nothing here. Wait for the module proxy to serve the release commit as the tip before dispatching, instead of priming it and hoping. The dispatched workflows pin seaweedfs with `go get -u ...@latest`, so a stale tip means they release against a pre-release commit, silently. Identify the dispatched run by diffing the run list against the snapshot taken before dispatching, rather than assuming the newest run is ours. * release: wait on the downstream release, not on the run that makes it A dispatched run cannot be told apart from a concurrent one: the API does not report the inputs a run was dispatched with, so watching "the run that appeared after mine" can watch someone else's and report their result as ours. Wait for a release to appear in the downstream repository instead. That is the thing being waited for, and it holds however many runs are in flight. |
||
|
Chris Lu
|
5e7ab43ddd |
test: read Lance tables from DuckDB (#10866)
* test: read Lance tables from DuckDB
The LanceDB and Spark suites go through the catalog. DuckDB does not: its
lance extension reaches the data over S3 with no namespace involved, which
exercises the other half of the design - a table bucket's layout is a
valid Lance dataset directory, so a table stays readable when the catalog
is not in the path.
scan_rows=128
scan_columns=id,title,vector
filtered_rows=5
nearest=1,0,2
It also pins the one place the layout costs us. DuckDB's replacement scan
recognises a dataset by a .lance path suffix, and tables created through
this catalog deliberately have none: the catalog entry is the dataset
directory, a table name may not contain a dot, and a suffix would leak
into ARNs and policies. So __lance_scan is the way in, and the bare
SELECT ... FROM 's3://...' form does not see these tables.
The test asserts both halves - a suffixed path is read, a suffix-less one
is not - so if the extension ever recognises a bare directory, it fails
and says to update the documentation rather than leaving it wrong.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: require the catalog error from the suffix-less read
Any failure satisfied the old check - a missing extension, bad credentials,
an unreachable endpoint - so the assertion could pass without the
replacement scan ever classifying the path.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame DuckDB for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: order the aggregates the assertions read
string_agg over an unordered relation may return the names, and the vector
search's ids, in any order, so the expectations could fail on a run where
nothing changed.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the DuckDB Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
|
||
|
Chris Lu
|
0dfaa103d0 |
test: take a table through its whole life, for Iceberg and Lance (#10862)
* lance worker: share the integration tests' scaffolding The recorder that keeps what a handler sent, the config builder and the storage-option fallback all lived inside compaction.rs, so a second test binary would have had to copy them. They move to tests/common. The fallback now reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_ENDPOINT_URL from the environment, defaulting to what it used before. A harness can then point these tests at a gateway that checks what it is given rather than one that accepts anything. * lance worker: maintain one named table, for a harness to drive Compacts and cleans up whatever WEED_LANCE_TABLE names, through the handlers' own detect-then-execute path: a proposal the worker would not have made is not one worth running. The existing tests seed the tables they check. This one deliberately does not, so a harness that has already written a table and knows what is in it can have the real handlers maintain it and then read it back. * test: take a table through its whole life, for Iceberg and Lance Created in the catalog, filled by a real client, maintained by the worker, read again, dropped. The step nothing was checking is the read after maintenance: compaction once rewrote every dictionary-encoded column onto a single value and shipped, because the maintenance tests were thorough about sequence numbers, manifest entries and metadata versions and none of them opened the parquet file the worker had just written. So the assertion is a tally - row count, the cardinality of each dictionary-encoded column, and an md5 over whole rows - taken before maintenance and again after, required to be equal. The cardinalities name the failure that happened; the digest catches a rewrite that keeps every column's cardinality and hands the values to the wrong rows. A compaction that merged nothing fails rather than passes, or the read afterwards is checking a file the worker never wrote. The Iceberg half runs two clients. DuckDB is the one the corruption was reported against and the only one here that writes the deprecated PLAIN_DICTIONARY encoding, which parquet-go normalizes away on write, so a Go writer cannot produce it. PyIceberg writes the modern spelling. Pinning parquet-go back to v0.30.1 fails the DuckDB half and passes the PyIceberg one, which is why both are here. Lance maintenance lives in the Rust worker, so it runs there where cargo is installed and through the two lance calls those handlers wrap where it is not. WEED_LANCE_MAINTENANCE picks one instead of letting the test guess. * ci: run the table lifecycle tests CI maintains the Lance table through the lance library rather than the worker: a cold build of the lance crate costs more than the glue it would be checking, and the worker's own tests cover its handlers. The suite drives the Iceberg maintenance worker, so a change to it now triggers this workflow too. * test: let the lifecycle harness fail instead of skipping Setup failures all exited zero, so a cluster that would not come up, or a port allocation that lost, reported a green run for code nothing had executed. That is the failure mode this whole directory exists to close, and it was in the harness itself. Only a checkout without a weed binary skips now, and it runs the tests so each one says so rather than the package quietly passing. Everything else fails. The filer existence probe gets a deadline while I am here: it ran without one, so an unresponsive filer would hang the suite past every timeout the clients have. * test: make the lifecycle checks check what they claim to Three of them could pass without having looked. The DuckDB skip matched "syntax error", "not implemented" and "Failed to load" anywhere in the output, in any phase. A parse error in the SQL this test generates, or a refusal from our own catalog, would have taken the only coverage of the PLAIN_DICTIONARY encoding out of CI and left it green. It now matches the extension failing to install, and only in the phase that installs it. Everything past LOAD is ours and fails. The digests covered id, category and value. Compaction rewrites the whole row, so a defect confined to ts, or to a Lance vector, changed nothing either side of maintenance. Every persisted column goes in now, ts as microseconds so no timezone sits between the two runs. The Lance drop check caught every exception as proof the dataset was gone. pylance turns credential and transport failures into the same ValueError, so it only accepts the message that means not found. * docs: say up front which maintenance path the Lance half takes The opening summary said the worker maintains both tables. It maintains the Iceberg one always and the Lance one only where cargo is installed, which is not what CI does. |
||
|
Chris Lu
|
83753ccdad |
test: drive the Lance namespace with LanceDB (#10850)
* test: drive the Lance namespace with LanceDB
The Iceberg catalog is checked against Spark, Trino, ClickHouse, Doris,
Dremio and RisingWave. The Lance one had only its own reference client,
which is the same thing as checking it against ourselves.
LanceDB connects with connect_namespace("rest", ...), which speaks the
routes this catalog implements, so the suite exercises the protocol rather
than our idea of it: list the catalog, open a table through it, read the
schema, run a vector search and a filtered scan, create a table, and read
the same dataset straight off its URI with no catalog at all.
table_names -> ['lancedb-p0guidmm$ml$embeddings']
open_table -> 64 rows
search -> [1, 0, 2]
create_table -> 4 rows, listed by the catalog
direct read without the catalog -> 64 rows
Seeding is pylance, because the namespace records where a table lives and
does not carry its data. That split is the design rather than a limit of
the test.
One interop note the test encodes: a gateway without STS vends
storage_options carrying an endpoint and a region but no credentials, and
LanceDB uses what the namespace vends on some paths. The container gets
credentials in its environment as well, which is what a deployment without
STS would do.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: pin the LanceDB client, and index before searching
Three from review.
The client's dependencies were unpinned, so an unrelated upstream release
could change what an old commit reproduces. Pinned to the versions this
suite was verified against; the client is as much the thing under test as
the server.
The search was called ANN and was not: without an index LanceDB scans.
The test now builds an IVF_PQ index over 1024 rows first, which is worth
more than the wording fix - an index writes into a directory of the table
that the S3 door has to admit, and that guard has refused a Lance
directory before. It builds, covers all 1024 rows, and searches.
The assertion moved with it. Demanding the exact nearest neighbour was
right for a brute-force scan and wrong for a quantized index, which
answered 0 as readily as 1; both are correct, so the check is now the
neighbourhood.
And the pushdown check accepted any failure. It now requires the refusal
to be the catalog's Unsupported and requires that nothing was left behind,
or, when the client falls back, that the table is complete.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
|
||
|
Chris Lu
|
cd3db76eed |
ci: stop installing FUSE headers nothing links against (#10840)
Four workflows ran apt to install libfuse3-dev before every FUSE job. Nothing needs it: go-fuse implements the protocol in pure Go, no cgo in the tree references fuse, and the package does not even provide the fusermount3 the mount actually execs - fuse3 does, and it is already on the runner image, which is why the setuid-repair step finds it. So the step downloaded a dev package to build against headers no compiler ever opened, and it is the step that has been hanging whenever the Ubuntu mirror goes slow. Configuring /etc/fuse.conf is all that is left. |
||
|
Chris Lu
|
3b18a635df |
ci: call the apt helper from the workflow's working directory (#10832)
The e2e workflow sets defaults.run.working-directory: docker, so the call I added resolved to docker/docker/apt-install and every FUSE Mount run has failed with 'sudo: docker/apt-install: command not found' since it merged. |
||
|
Chris Lu
|
baead6901c |
ci: build protoc into the crate instead of installing it per job (#10830)
Every workflow that builds the Rust volume server first installed protoc from a package manager - twelve steps across apt, brew and choco. That is 37s per job on a good day, and this week archive.ubuntu.com stalled long enough for four jobs to burn their whole timeout without reaching a build. protoc-bin-vendored ships the compiler as a build-dependency, so it now arrives through the cargo registry the workflows already cache and there is nothing left to install. cargo build works on a machine with no protoc at all, which is worth as much locally as it is in CI. It also pins the version. The apt protoc on ubuntu-22.04 is 3.12, old enough to reject proto3 optional, which is why build.rs passes --experimental_allow_proto3_optional; the vendored one is 31.1. The flag stays, since it costs nothing and keeps a build against an older PROTOC working, and an explicit PROTOC still overrides the vendored binary for packagers who supply their own. |
||
|
Chris Lu
|
1564244b1a |
ci: install the runner's own packages through the mirror fallback too (#10831)
The e2e job overwrote the runner's sources.list with two azure-only lines and installed fuse from it, so the same mirror outage that took out the image builds failed the step outright - this time on the runner rather than inside the container, where the image-side fallback cannot reach. Install through the same helper, and widen its rewrite to match any archive host so it works whether the pristine list came from the base image (archive.ubuntu.com) or from a CI runner (azure.archive.ubuntu.com). Keeping the runner's original list also restores the security and backports pockets, which the hand-written two-line replacement dropped. Verified against the outage itself: with the pristine list pointed at Azure, the build logged the skip after Azure timed out for real and installed from archive.ubuntu.com. |
||
|
Chris Lu
|
1ddec72707 |
Recover from a dead volume server on the mount read path (#10798)
* mount: re-resolve volume locations after a failed chunk read NewChunkGroup passed nil as the ReaderCache's CacheInvalidator, so retryFetchAfterCacheInvalidation was dead code on the FUSE read path. A mount that cached a volume's locations while one server was down kept retrying that server after it died, then returned EIO, even though the master and filer both resolved the live replica. The S3 gateway already passes its filerClient; do the same for the mount. * test: FUSE integration tests for volume server failover One mount appends while a second tails, and a volume server is killed, started or restarted mid-stream against a 001-replicated cluster of three volume servers. Automates the scenario matrix reported for Docker Swarm mounts, including the large-file variant and a no-chaos control. * test: report the filer's own view when append content mismatches A mismatch between what the writer wrote and what the reader sees can come from either side's cache. Read the file back through the filer's HTTP handler as well, and let the mount verbosity be raised from the environment, so a failing run says which layer lost the data. * test: wait for the reader mount to converge before comparing A mount caches metadata for about a second, so reading the file the instant the writer's last close returned can legitimately come back short. Poll the reader until it matches or the timeout expires; content that is wrong rather than merely late never converges and still fails, now with the writer's mount and the filer's own view alongside it. * test: detect a failover cluster child that exited at startup Signal(0) succeeds for a zombie and nothing reaped these children until shutdown, so a process that died on startup looked alive until the readiness timeout expired. Reap each child as it is started and consult the result. * test: read a file the killed volume server actually holds Placement decides which two of three servers back each volume, so killing volume N and reading readfile-N could pass without the victim ever holding a replica of it. Resolve each file's volumes through the filer and the master, and pick one the victim backs, preferring a file the reader has not cached. * ci: stop persisting checkout credentials in the failover workflow The job does not use the token after cloning. Also tag the README's command block as bash and match the timeout the workflow actually uses. * test: discard the ignored errors errcheck flags in the failover harness * test: resolve manifests when mapping a file to its volumes A manifest chunk's own fid names the volume holding the manifest, not the volumes holding the data, so a large enough file would point the failover victim at the wrong server. * test: pin the stale-location recovery path with a primed reader Reading a file for the first time after a server dies proves nothing: the lookup is fresh and returns the survivor. Kill one holder and wait for the master to drop it, read a file on that volume so the reader caches the lone survivor, restart the first server, then kill the survivor. The reader's only cached location is now dead while the data is live elsewhere, which is the case the invalidator exists for: EIO without it, recovery with it. |
||
|
Chris Lu
|
1bcd55eba2 | go 1.26 (#10797) | ||
|
Chris Lu
|
1c926e8fac |
test: systematic EC interruption verification — exhaustive model check + deterministic kill matrix (#10764)
* ec: bounded-exhaustive model check of the volume lifecycle The randomized chaos harness samples the state space; this enumerates it. The lifecycle is a state machine whose steps mirror the pipelines in this package, and the checker explores every schedule within the bound: a crash at every step boundary, an error return running the rollback (itself crashable at every step), a volume-server restart applying the startup reconciliation rules in every quiescent state, and the prescribed restart-based recovery from every crashed state. Checked in every reachable state: durability (a readable copy always exists), at most one generation mounted, and — a property the sweep discipline turns out to guarantee — at most one generation's files on disk. From every quiescent state the recovery must converge to a clean volume. Runs in well under a second. * test: deterministic EC interruption matrix Enumerate every phase of every interruptible EC operation and kill a real weed shell exactly when the phase announces itself on the command output, instead of at a random moment: four encode phases, four decode phases, and the balance's move phase (set up with -rebalance=false so a move is guaranteed). Each scenario prepares its precondition, kills at the marker, runs the prescribed recovery, and verifies every stored byte still reads back identical. The interruption recoveries move out of the randomized ops into shared chaosRun helpers both drivers use. * test: make the randomized EC chaos walk opt-in The systematic layers — the interruption matrix and the lifecycle model check — carry the CI coverage deterministically; the randomized walk stays for exploratory runs, behind EC_CHAOS_SEED. * ci: bound the EC integration suite by the job budget, not go test's default The suite with the interruption matrix runs close to the default 10m binary timeout on slower runners. * test: require every interruption-matrix marker to appear A marker that never prints means a pipeline refactor renamed or dropped the progress line; silently degenerating into a no-interruption run would let CI pass without exercising the boundary the scenario names. Also recheck the marker channel after the wait: a shell that prints and exits at once makes both channels ready, and select picking the exit case must not report a printed marker as missed. |
||
|
Chris Lu
|
0de7ff5eb8 |
ci: run the gated redis store tests (#10746)
* redis2: route the orphan cleanup existence checks to the master * scaffold: the redis_cluster2 read routing key is useReadOnly * ci: run the gated redis store tests * redis2: poll for the redis expiry instead of a fixed sleep * redis2: assert the value key exists before testing its expiry |
||
|
Chris Lu
|
790e8d3fd6 |
clickhouse catalog test: cover latest ClickHouse and catalog-side CREATE TABLE (#10707)
* clickhouse catalog test: cover latest ClickHouse and catalog-side CREATE TABLE * verify catalog registration structurally and fix README image wording |
||
|
Chris Lu
|
214d3599d3 |
windows mount: cache file data, resolved paths and attributes (#10703)
* benchmark tool for mounted filesystems * ci: on-demand mount benchmark, native WinFsp vs rclone plus a Linux reference * windows mount: let the Windows cache manager cache file data WinFsp only turns the cache manager on for a file when FileInfoTimeout is infinite; at any finite value every application read and write is a synchronous trip into the mount process at whatever size the application issued. Metadata events already reach FspFileSystemNotify, which purges a changed file's cached pages and attributes, so an infinite timeout stays coherent. The dir listing, volume info and EA timeouts are pinned to one second so they do not silently inherit the infinity. * windows mount: cache resolved paths and attributes in the adapter WinFsp addresses every operation by path and has no FORGET, so the adapter walked the whole path through Lookup on each one, and in a directory the filer has not listed yet every walk was a filer round trip; nothing played the part of the kernel's dentry and attribute caches. The path cache owns one lookup reference per entry the way the kernel holds one until FORGET, serves attribute reads for files without an open handle, and is purged by the mount's own mutations and by metadata events, with the timeout as backstop. * windows mount: keep a closed file's attributes cached Open steals the path's cache entry for its handle and Release returned the reference with a purge, so the stat that follows every copied file walked to the filer again. Reading the handle's final attributes before it goes away and moving the reference back into the cache serves that stat locally, the way the kernel's attribute cache does after a close. Only if the path still names that inode, though: WinFsp reports the path the handle opened with, and after a delete-on-close or a rename caching it would resurrect an entry that is gone. * windows mount: persist entries at create, and let the flush stay at close WinFsp posts the cleanup and close that carry the flush after CloseHandle has returned, so deferring the filer entry to the flush let everything that reads through the filer race an unflushed close: a listing missed just-written files, and a directory rename moved a directory on the filer before its newest child existed there, leaving the straggler flush to recreate the child under the dead path. Flush-at-cleanup is not the answer either: it makes every handle's cleanup flush, and those flushes race the unlinks of delete-on-close, re-inserting the entry the unlink just removed. Persisting the entry at create takes the ordering question away. * mount: flush written pages before a truncate shrinks past them The shrink trims chunks, but written pages that have not become chunks yet are invisible to it, so the next flush wrote them back and the file grew again, resurrecting the truncated bytes. Windows hits this on every write-then-shrink because its flush runs after CloseHandle, but the gap is platform-neutral. * mount: order a file's unlink against its in-flight flush Unlink set the handle's deleted flag bare, so a flush already past its own check of that flag wrote the entry back right after the delete removed it, and a delete-on-close file outlived its last handle. The flag is now set under the handle's flush lock and re-checked under it, so a flush either completes before the delete or sees the flag and skips. An eagerly created handle also starts clean: the dirty mark existed to make the deferred filer create happen at flush, and eager creates have nothing to flush. |
||
|
dependabot[bot]
|
6dd83c823d |
build(deps): bump github/codeql-action from 4.37.4 to 4.37.6 (#10699)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.4 to 4.37.6. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v4.37.4...v4.37.6) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.6 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
3a61debaa5 |
filer: rebuild peer metadata subscriptions after a master reconnect (#10648)
* filer: keep the existing peer subscription on a repeated add A cluster node add for a peer that is already followed restarted the subscription, dropping the metadata events between the two runs. * master: tell a connecting client the current cluster membership Cluster node updates are only broadcast to the clients connected at that moment. A filer that lost its master stream while a peer came back never learned about the peer, and stopped replicating its metadata for good. * test: a filer joining the master learns about the filers already there * test: a filer resubscribes to a peer that registered while it was disconnected Runs the reported sequence against real processes: filer2 leaves, filer1 is paused and its master stream is broken, filer2 registers again, and filer1 has to replicate from it after reconnecting. |
||
|
Chris Lu
|
213eb4c23a |
s3tables: add ClickHouse iceberg catalog integration test (#10637)
* s3tables: add ClickHouse iceberg catalog integration test * ci: run the ClickHouse iceberg catalog test * s3tables: bound setup HTTP calls in the ClickHouse test * s3tables: pin the ClickHouse writer image dependencies |
||
|
Chris Lu
|
fb92d46e2d |
helm: enterprise license Secret, and a persistent-claim option for master data (#10601)
* helm: mount an enterprise license Secret into every component
Running the enterprise image under this chart meant hand-rolling
extraVolumes and extraVolumeMounts on every component. Missing one is
easy and quiet: a component without the license silently drops to
community mode, and on the admin that surfaces only as Data Recovery and
Point-in-Time Recovery refusing to enable, with the master looking fine.
Add global.seaweedfs.license.existingSecret. The Secret is mounted
read-only into master, volume, filer, s3, sftp, admin, worker and
all-in-one, and SEAWEED_LICENSE points every process at the file rather
than relying on the binary's search paths, which depend on the working
directory.
The mount is a directory, never a subPath: kubelet refreshes Secret
contents in place, but a subPath is resolved once at container start and
never updates, which would break license renewal. There is deliberately
no checksum annotation on the pod template either — that would roll every
pod on renewal, the opposite of what is wanted. Verified on kind: the
renewed file reached a running master ~70s after the Secret was patched,
same pod UID, restartCount 0.
Also documents that only the master re-reads the license on a timer
today; the other components pick a renewal up on their next restart.
* helm: keep master data on a claim by default
The master's -mdir holds its Raft log and snapshots, and with them the
cluster's topology UUID — the identity an enterprise license is issued
against. It defaulted to a hostPath under /ssd, which does not follow a
rescheduled pod: the master came back with an empty data directory, a
freshly generated cluster UUID, and a license that no longer matched.
With the chart's default of a single master replica there is no peer to
recover the identity from either.
Default master.data.type to persistentVolumeClaim, sized 1Gi (Raft state
is small). hostPath stays available for anyone who wants it.
This is breaking for existing releases: volumeClaimTemplates is immutable,
so helm upgrade on a release installed with the old default fails with
"updates to statefulset spec for fields other than ... are forbidden".
Verified on kind, along with both ways out — pinning
master.data.type=hostPath upgrades cleanly, and the documented migration
(stop the master, pre-seed a claim named after the StatefulSet, upgrade)
preserves the cluster UUID. Seeding has to happen while the master is
stopped; copying into a live pod loses the state, since the running
master rewrites its Raft files before the restart.
* helm: mount the license on masters only
The master is what reads the license file: it validates it, enforces the
capacity limit and binds it to the cluster UUID. Mounting the Secret on
volume, filer, s3, sftp, admin and worker put it in six more containers
that never look at it, so drop it there and keep master plus all-in-one,
which runs `weed server -master`.
Two fixes from review while here:
- project only the configured key out of the Secret, so an unrelated
key in the same Secret is not exposed to the container. Verified the
key-scoped projection still updates in place: patching the Secret
reached the running master in ~50s, same pod UID, restartCount 0.
- drop SEAWEED_LICENSE from merged extraEnvironmentVars while a
license Secret is configured. It used to be possible to render the
key twice in one container, with the user's value winning over the
path the chart actually mounts.
CI now pins the scope (master only, all-in-one separately), the
key-scoped projection, readOnly, and that SEAWEED_LICENSE renders once.
* helm: fix the documented master-data migration
The seed pod in the migration never mounted the claim it was supposed to
seed, so following the steps verbatim copied the Raft state onto the
pod's ephemeral filesystem and threw it away with the pod — landing the
reader in exactly the empty-claim, new-cluster-UUID state the section
exists to avoid. Give the pod the volume.
The names were assembled as <release>-seaweedfs-*, which is wrong
whenever the release name already contains the chart name or an override
is set; read the StatefulSet name from the cluster instead and derive the
claim from it. Also scope the procedure to the chart's single-master
default, and create the Secret in the release namespace.
Trims the enterprise prose this section had accumulated: this is the OSS
chart, and the master-data default is a durability fix that stands on its
own.
* helm: quote the projected license key, reserve it on the secret env path
A secretKey that YAML reads as a non-string (123, yes, no) rendered
unquoted into the volume's items, so the projection would not match the
Secret's key. Quote both key and path.
all-in-one renders secretExtraEnvironmentVars itself, outside the merge
helper that already drops SEAWEED_LICENSE, so an entry there could still
render the variable twice. Skip it there too while a license Secret is
configured. The master template has no such block, so this is the only
remaining path.
* helm: correct the license helper comments after scoping to masters
* helm: keep hostPath as the master data default
Defaulting master.data.type to a claim broke every existing release:
volumeClaimTemplates is immutable on a StatefulSet, so helm upgrade
failed with "updates to statefulset spec for fields other than ... are
forbidden" before it changed anything.
Keep hostPath as the default and document the claim as the option to
choose — for a new install, or for an existing one via the migration
already in the README. The chart supported both types all along; only
the default moves back.
Every immutable field of every rendered StatefulSet is now identical to
upstream under default values, so an in-place upgrade cannot trip the
API. Verified on kind: install with the unmodified upstream chart,
upgrade to this branch (ok), upgrade again turning the license Secret on
(ok, volume added in place). A fresh install with
master.data.type=persistentVolumeClaim binds its claim as before.
The whole PR is additive now: nothing renders differently until a value
is set.
* helm: scope the migration's StatefulSet lookup to the release
* helm: trim the comments added by this change
|
||
|
Chris Lu
|
a5e8254ffd |
s3: give a versioned metadata-only copy its own chunks (#10594)
* s3: give a versioned metadata-only copy its own chunks A self-copy that only rewrites metadata clones the source entry, chunk fids and all, and writes the clone back. With no versioning that is exactly right: the clone replaces the entry it came from, so one entry owns the needles the whole time. Under versioning the clone lands in a new .versions/ file and the source stays live, and nothing refcounts a plain shared chunk list -- deleting either version (a NoncurrentVersionExpiration rule, say) frees needles the other still points at, and the next vacuum makes that permanent. rclone hits this on every upload, since it stamps mtime with exactly this copy. Take the metadata-only path only where the write replaces the entry it read: the bare key of a bucket without versioning. Versioned, suspended, and versionId-pinned copies fall through to the regular copy path, which gives the destination its own chunks. * s3: reencrypt a versioned SSE-KMS key rotation instead of reusing the chunks A same-object copy that changes the KMS key id hands the source chunks straight back, on the assumption that the copy overwrites the entry they came from. A versioned bucket writes a new version beside the source instead, so the two end up sharing needles that nothing refcounts, and deleting either one frees the other's data. Reuse the chunks only when the destination really is the source entry; otherwise fall through to the reencrypt path, which also gives the new version the key it asked for rather than leaving it on the old one. * s3: make one predicate decide whether a copy replaces its source The metadata-only branch and the key-rotation strategy both answer the same question -- does this copy write back to the entry it read -- so let them share one predicate instead of pairing a same-destination check with it separately at each site. * test(s3): fail the copy regression tests when the vacuum does not run The helper swallowed a failed or non-200 request to the master, so a vacuum that never ran turned both chunk-ownership assertions into no-ops: the tombstoned needles were still readable and the surviving version looked fine either way. Require the endpoint, the request, and a 200. * ci(s3): run every versioning test in the regression gate The gate named the tests it wanted, so a new regression test sat there uncovered until someone remembered this file -- it fooled me into thinking two tests added in this PR never ran anywhere, when the comprehensive job had them all along. Invert it: run everything, and name a test only to keep it out. The delete job beside this one already works that way, and the suite costs about two minutes. Only the pagination stress tests are excluded; they build 1500+ versions, skip themselves without ENABLE_STRESS_TESTS, and have their own make target. Go's regexp has no negation, so the pattern is still assembled from a listing, the way the volume-server integration workflow does it. Note the trailing $$: make eats a lone trailing $ and takes the anchor with it. |
||
|
Chris Lu
|
69aa6d7adc |
test(s3): give the copying suite room for a collection per bucket (#10590)
Every test bucket is its own collection and each grows 7 volumes, so the suite asks for 140 while the job caps the volume server at 100. Slots come back only when the volume server's next full heartbeat tells the master the deleted collections are gone, and the suite finishes inside one 5s pulse: the run survives on whichever buckets happened to be dropped before that single tick. The last run cleared by two slots, this one wedged the final PutObject with 'No writable volumes and no free volumes left'. |
||
|
Chris Lu
|
3c549b33ab |
ci: deal volume server tests across shards instead of bucketing by letter (#10576)
Both workflows split the suite with ^Test[A-H] / ^Test[I-S] / ^Test[T-Z]. Test names cluster, so shard 2 drew 50 of the 114 grpc tests and 30 of the 64 http ones, and spent 13m51s against 7m48s and 9m09s for its peers. Listing the tests and dealing them out one at a time splits them 38/38/38 and 21/22/21, and keeps splitting evenly as tests are added. The pattern is computed once into the environment rather than repeated in the summary step, where the two copies had to be kept in agreement by hand. |