mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-13 17:48:29 +02:00
d69f45ee83d33e82ef024bd9d846ee9cbc847cd8
746
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
f11cecc697 |
helm: keep the S3 config checksum stable across the first upgrade (fixes #11719) (#11720)
* helm: keep the S3 config checksum stable across the first upgrade (fixes #11719) checksum/s3config hashed the rendered s3-secret.yaml. That template keeps its generated S3 keys with lookup, so rendering it again for the checksum drew new keys whenever lookup found nothing: on install the filer, S3 and all-in-one pods each carried a checksum of a different set of keys than the Secret got, and the first upgrade with unchanged values corrected it and restarted them. Hash the values s3-secret.yaml is built from instead: the enable and auth switches, existingConfigSecret, credentials, the legacy-secret settings, the audit log configs and the Secret name. Only those, so scaling the S3 gateway or changing its resources does not restart the filer. A change to any of them still restarts the pods. Keys that the chart generated stay out of the hash; they only change if the Secret is deleted by hand. The new render test fails on the old template (two checksums for one Secret, different on every render) and passes now; it also checks that the checksum follows the credentials and filer.s3.auditLogConfig and ignores s3.replicas and s3.resources. On kind, an upgrade with unchanged values no longer replaces the filer pod. * helm: check the S3 checksum per workload, all-in-one included Each workload's checksum/s3config is now read on its own (--show-only), so a workload that loses the annotation fails the test instead of being hidden by sort -u, and the all-in-one deployment gets the same checks. Verified against two broken copies: one without the filer annotation, one with the old all-in-one checksum; the test fails on both. The helper comment now says what happens when the Secret is deleted to rotate generated keys. * ci(helm): fail the S3 checksum test on a missing or unrendered checksum A failed helm template or a render without the annotation produced an empty checksum, and the inequality assertions then passed against it. s3_checksum now returns non-zero unless it found a checksum, and the changed-value renders are assigned before comparing so failures cannot hide inside test. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Ege Zambelli <egezamb@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
c96eb7b8ee |
kms/azure: fix encrypt/decrypt round trip for Azure Key Vault (#11692)
* kms/azure: fix encrypt/decrypt round trip for Azure Key Vault The provider stored the wrapped data key as string(encryptResult.Result) in the JSON envelope, sent the encryption context as AAD to an RSA-OAEP key, and passed a full key URL to the azkeys client in the name position. Each of those breaks a round trip on its own. - split the key URL into the (name, version) pair azkeys expects before Encrypt, Decrypt and GetKey; a plain name still resolves to the latest version, and decrypt keeps the stored version so objects stay readable after a key rotation - store the wrapped key base64-encoded and decode it strictly, so the JSON envelope no longer replaces the raw bytes with U+FFFD - stop sending AAD: Key Vault rejects it on RSA-OAEP with BadParameter, so the encryption context is logged and dropped instead * kms/azure: reject a key URL that names another vault splitKeyID dropped the host of a full Key Vault URL, so a key ID from a different vault silently resolved to this vault's same-named key. Return an error instead of encrypting under a key the caller did not ask for. * kms/azure: bind encryption context via an envelope digest RSA-OAEP rejects AAD, so removing it left the encryption context unauthenticated: a wrapped key could be decrypted under a different object's context. Record a sha256 digest of the marshaled context in the envelope's provider_specific field on encrypt and verify it before calling Decrypt, restoring the binding without AAD. * kms/azure: treat an explicit :443 port as the same vault * kms/azure: accept an absent context digest only for empty contexts * kms/azure: normalize both hosts when comparing key URLs to the vault splitKeyID stripped :443 and a trailing dot from the configured vault but only :443 from the key URL, so a key URL naming the same vault with a trailing DNS dot was rejected before Azure was ever called. Compare both sides through vaultHost so they are normalized identically. * kms/azure: reject non-key vault URLs in splitKeyID, document digest limits A Key Vault URL that does not name a key under /keys/, has an empty key name, or carries extra path segments now fails fast instead of being passed to the client as a key name, where it would surface as a confusing vault-side error. Also note that the envelope context digest is a client-side mismatch check, not vault-authenticated AAD, and only allocate providerSpecific when a context is present. * ci: compile and test azurekms-gated code weed/kms/azure is excluded from every default build, so nothing in CI compiled it; that is how the provider shipped unregistered. Build the tree and run the kms tests with -tags azurekms on every Go change. --------- Co-authored-by: Yi-111-a <> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
838c554e33 |
s3api: reject SSE headers PutObject cannot honor, as CopyObject does (#11642)
* s3api: reject SSE headers PutObject cannot honor, as CopyObject does PutObject and CreateMultipartUpload did not check the server-side encryption headers they were given. An x-amz-server-side-encryption value that names no method, such as "aes:kms", matched none of the SSE paths, so the object was stored unencrypted and the request answered 200. SSE-C together with x-amz-server-side-encryption was also accepted, and one of the two silently won. CopyObject already rejects both through validateEncryptionCompatibility. Run the same check, with the same error codes, before PutObject and CreateMultipartUpload store anything. * s3api: answer rejected SSE headers with InvalidArgument, as S3 does S3 rejects an unknown x-amz-server-side-encryption value and SSE-C combined with another method with InvalidArgument. PutObject and CreateMultipartUpload now return that code, with S3's messages, through two new error codes; CopyObject keeps its own. Also run ceph/s3-tests' test_put_obj_enc_conflict_c_s3, _c_kms and _bad_enc_kms in CI, which check both handlers' responses end to end. * s3api: close the remaining ways a PUT could skip requested encryption - A repeated x-amz-server-side-encryption header is rejected: the encryption paths apply only the first value, so extra values could hide the method the client asked for. - KMS options (key id, encryption context, bucket key) are rejected unless the method is aws:kms, matching the S3 InvalidArgument error. - CreateMultipartUpload validates before auto-create, so a refused upload cannot leave a bucket behind. - Directory markers encrypt their inline content through the shared SSE path and record the same entry metadata as regular objects, instead of storing requested-encrypted bytes in plaintext. * s3api: read back what the SSE marker write stores - Repeated SSE-C and KMS option headers are rejected alongside a repeated x-amz-server-side-encryption, closing the same first-value bypass for customer-key and KMS fields. - Algorithm validity is checked before KMS options so an unsupported value keeps the "not supported" error. - serveDirectoryContent decrypts marker content with the stored SSE metadata and returns the SSE headers, so an encrypted marker reads back what was written; its Content-Length now reflects the bytes actually served. * s3api: HEAD of a marker skips decryption and keeps the stored size HEAD returns no body, so it now runs only the SSE-C key check instead of decrypting — matching HeadObjectHandler and avoiding a KMS round-trip — and chunk-backed directory entries report Attributes.FileSize again rather than the length of their (empty) inline content. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: keep GET Content-Length to the bytes serveDirectoryContent writes The FileSize override described chunk-backed markers on HEAD, but GET sends only the inline content, so it promised bytes it never wrote. * s3api: stream a chunk-backed directory on GET like any object A directory promoted over an uploaded object keeps the object chunks with empty inline content, so serving only entry.Content made GET deliver nothing while HEAD reported FileSize. GET now routes those entries through the regular volume-server stream, keeping the two methods consistent. * s3api: answer a busy volume read with RequestBytesExceed --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Younsung Lee
|
68af030e38 |
helm: add Gateway API HTTPRoute for master, volume, filer, s3 and admin (#11625)
* helm: add Gateway API HTTPRoute for master, volume, filer, s3 and admin Each component that offers an Ingress can now be exposed through a Gateway API HTTPRoute instead, via <component>.httpRoute. It is disabled by default, so existing renders are unchanged. parentRefs, hostnames, labels and annotations pass through as given. Each rule may set matches, filters, timeouts and backendRefs, and a rule without backendRefs routes to the component's own service and port, following all-in-one mode the same way the Ingress templates do. An empty rules list yields a single rule sending all traffic there. Signed-off-by: younsl <cysl@kakao.com> * helm: fix HTTPRoute backend selection for S3 on filer and all-in-one - s3: with S3 on the filer the route targets filer.s3.port, the port the s3 Service exposes. The all-in-one Service is chosen only when allInOne.s3 is enabled, so a standalone S3 next to all-in-one routes to the s3 Service. - master: render in all-in-one mode and route to the all-in-one Service, like the volume and filer routes. - Name routes with seaweedfs.componentName, keeping them within 63 characters like the Services they point to. - CI: cover the filer S3 port and the all-in-one master route. Signed-off-by: younsl <cysl@kakao.com> * helm: document the Gateway API HTTPRoute values in the chart README Signed-off-by: younsl <cysl@kakao.com> * helm: link the Gateway API docs from the chart README Signed-off-by: younsl <cysl@kakao.com> * helm: scope the filer S3 port note in the chart README Signed-off-by: younsl <cysl@kakao.com> --------- Signed-off-by: younsl <cysl@kakao.com> |
||
|
Upayan
|
b14dd1cee4 |
helm: drop fromToml dependency in security-configmap.yaml (fixes #11611) (#11614)
* helm: drop fromToml dependency in security-configmap.yaml (fixes #11611) fromToml was added to Helm in v3.17.0 (helm/helm#12026, merged 2024-09-12, one day after v3.16.0 was cut). This chart declares no minimum Helm version (no Chart.yaml kubeVersion, nothing in the README), and the call in security-configmap.yaml:21 is an unconditional *parse*-time failure on Helm < v3.17.0 - Go's text/template parses a file's entire body before evaluating any {{if}}, so this breaks the chart (any topology, any values) even when securityConfigEnabled is false and the ConfigMap would render nothing. Replaces the fromToml-based dig lookup with a small regex-based helper (seaweedfs.existingTomlKey) that reads the same four "key = ..." JWT signing-key values out of a previously-rendered security.toml, preserving the existing fallback-to-random behavior exactly. Verified: - helm lint (v3.16.3 and v4.3.0): clean - helm template with chart defaults: byte-identical output to the unpatched chart rendered via Helm v4 (which has fromToml) - the disabled/default path is untouched - helm template with security enabled, no prior ConfigMap: identical structure to the unpatched chart (helm v4), modulo the expected random key - Real helm install + helm upgrade round trip (live lookup, since "helm template" never evaluates lookup, even under the original fromToml code): the JWT signing key is identical across both releases - confirms key persistence across upgrades is preserved, not just "renders without erroring" - helm template with chart defaults, Helm v3.16.3: previously failed with a parse error naming fromToml as undefined; now renders successfully Fixes #11611. * helm: harden existingTomlKey against commented key lines and CRLF Addresses two review findings from greptile-apps on PR #11614: - The key-line regex matched the first "key = ..." anywhere in the section block, including a commented-out "# key = ..." line, which would shadow a real active key on a hand-edited or otherwise non-chart-generated ConfigMap. Anchored to line start with the Go regexp multiline flag ((?m)^key...), which a line starting with "#" cannot match. - The section-header match required an exact "]\n", so a ConfigMap with CRLF line endings would fail to match the block at all and regenerate the key instead of reusing it. Changed to "]\r?\n". Also adds a CI test ("Verify JWT signing key persistence across upgrades") exercising all of this end to end with a real helm install -> edit the live ConfigMap -> helm upgrade cycle, matching the existing "Verify SFTP host key secret lifecycle" test's shape: both edge cases are reproduced against a real ConfigMap and asserted on the post-upgrade rendered security.toml. Verified locally (same commands as the new CI step) against a real cluster before pushing. * helm: preserve JWT keys across supported TOML layouts * ci: use setup-python interpreter for JWT upgrade checks * helm: preserve keys under quoted TOML section headers * helm: ignore unrelated quoted TOML section headers |
||
|
Peter Dodd
|
8ebef03509 |
helm: expose s3 update strategy, lifecycle, and termination grace period (#11602)
The standalone s3 Deployment hardcoded terminationGracePeriodSeconds and offered no way to set a container lifecycle or the Deployment strategy, so operators could not add a preStop delay to drain endpoints before SIGTERM or hold maxUnavailable at 0 during rollouts. Add s3.updateStrategy, s3.lifecycle and s3.terminationGracePeriodSeconds. Defaults render the same manifest as before. |
||
|
zhao-ycandChris Lu
|
16e66b1bad |
fix(s3): initialize destination ACLs for CopyObject (#11599)
* fix(s3): initialize destination ACLs for CopyObject Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com> * fix(s3): re-check routed self-copy eligibility on the locked read routeInPlace was decided on the pre-lock entry, but the PATCH body re-reads the entry. A concurrent write changing file mode or MIME in between left the routed PATCH installing new ACL keys while Attributes kept the stale mode. Evaluate eligibility against the re-read entry and retry the self-copy under the distributed lock when it no longer qualifies. * fix(s3): guard metadata self-copies against concurrent writes Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com> * ci: raise s3api unit-test timeout to 9m The suite crossed the 5m binary timeout on the hosted runner (local run is ~4.3m and still growing). The job-level limit is already 10m. * ci: allow setup time before the S3 API test suite Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com> --------- Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
hajimohammadi.net
|
0d93dec145 | helm: mount TLS certificates in bucket hook (#11589) | ||
|
|
43abe21ffa |
Helm: Add Opt-in Read-Only Root Filesystem Support (#11562)
* security context changes Signed-off-by: Subhadeep Maity <smaity@slb.com> * root file changes Signed-off-by: Subhadeep Maity <smaity@slb.com> * Skip tmp mount when extras provide one; use allInOne context for the bucket hook * Mount tmp for secondary containers; keep user /tmp on the main container --------- Signed-off-by: Subhadeep Maity <smaity@slb.com> Signed-off-by: Subhadeep Maity <322813880+deepnemesis@users.noreply.github.com> Co-authored-by: Subhadeep Maity <smaity@slb.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
7c7834c98e |
Helm: Add Configurable Security Contexts for Chart-Managed Workloads (#11558)
* security context changes Signed-off-by: Subhadeep Maity <smaity@slb.com> * extending examples Signed-off-by: Subhadeep Maity <322813880+deepnemesis@users.noreply.github.com> * updating examples Signed-off-by: Subhadeep Maity <322813880+deepnemesis@users.noreply.github.com> * examples Signed-off-by: Subhadeep Maity <322813880+deepnemesis@users.noreply.github.com> * Update k8s/charts/seaweedfs/values.yaml Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update k8s/charts/seaweedfs/values.yaml Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * review comments Signed-off-by: Subhadeep Maity <322813880+deepnemesis@users.noreply.github.com> --------- Signed-off-by: Subhadeep Maity <smaity@slb.com> Signed-off-by: Subhadeep Maity <322813880+deepnemesis@users.noreply.github.com> Co-authored-by: Subhadeep Maity <smaity@slb.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
Javier Garcia
|
8fdcf69eb0 |
s3api: report the stored checksum in GetObjectAttributes (#11529)
GetObjectAttributes accepted the Checksum attribute but never filled it in, as its comment said SeaweedFS did not store S3 checksums. PutObject and CompleteMultipartUpload store them now, and HeadObject returns them. Fill in Checksum from the same entry fields, with the ChecksumType and ChecksumCRC64NVME members the response did not have. Also run ceph/s3-tests' test_get_checksum_object_attributes in CI. |
||
|
Javier Garcia
|
988fc4f7ba |
s3api: do not store aws-chunked in an object's Content-Encoding (#11528)
* s3api: do not store aws-chunked in an object's Content-Encoding aws-chunked in Content-Encoding names the SigV4 streaming framing of the request body, which the gateway decodes on upload. PutObject and CreateMultipartUpload stored the header as sent, so an object uploaded with "gzip, aws-chunked" was served with that Content-Encoding, and one uploaded with "aws-chunked" alone was served as aws-chunked. S3 drops aws-chunked and keeps the other encodings. Also run ceph/s3-tests' test_object_content_encoding_aws_chunked in CI. * s3api: read every Content-Encoding field, and drop aws-chunked on copy A client can send aws-chunked and the object's own encoding as separate Content-Encoding fields. Only the first was read, so "aws-chunked" followed by "gzip" left the object without its gzip. Combine all the fields before dropping aws-chunked. CopyObject with the REPLACE directive stored the requested Content-Encoding as sent: drop aws-chunked there too. |
||
|
Ishwar Kanse
|
5fcf12c080 |
docker: publish release and latest images for linux/ppc64le and linux/s390x (#11517)
Add ppc64le and s390x to the normal and large_disk release image matrices, next to arm/v7 and 386, and to the manual container_latest fallback so that rebuilding `latest` does not drop them from the manifest. The Go binary is cross-compiled without cgo on the build host, and Dockerfile.go_build already falls back to empty Rust volume server and worker placeholders on platforms without pre-built Rust binaries. Only the small final-stage RUN steps run emulated, so these two platforms use QEMU on ubuntu-latest like arm/v7. Refs #5558 |
||
|
dependabot[bot]
|
c54f1fe69c |
build(deps): bump github/codeql-action from 4.38.1 to 4.38.2 (#11509)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.38.1 to 4.38.2. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v4.38.1...v4.38.2) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.38.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Ping QiuandClaude Opus 5
|
635f69a821 |
test(s3tables): add an OLake Iceberg catalog integration test (#11441)
OLake (github.com/datazip-inc/olake) is a CDC and ingestion engine that writes Apache Iceberg. It covers two paths none of the existing catalog suites reach. It is a strict Java Iceberg client. OLake does not write Iceberg from Go — its Go process spawns a Java sidecar over gRPC and writes through the official Apache Iceberg library, because the Go library has no equality deletes and CDC needs them. That makes it the client class weed/s3api/iceberg's metadata compliance backfill exists to serve: the one that fails with "Cannot parse missing long current-snapshot-id" when spec-required keys are omitted. And it produces equality deletes. Its upsert path commits operation=overwrite with an equality-delete file and a delete manifest. ClickHouse, Doris, Trino, Spark and DuckDB all only append, so nothing else in this directory exercises a delete manifest at all. Six subtests: the destination check reaches SUCCEEDED and actually loads the REST catalog; discover enumerates the source; a full sync commits a snapshot; PyIceberg reads back what the Java writer committed; an update plus a re-sync records an overwrite carrying equality deletes with a delete manifest in the current snapshot; and the catalog does not rewrite manifests the official Java writer produced, which gives the manifest-repair path a negative test to go with the ClickHouse positive one. What it deliberately does not assert is a delete-applied read. PyIceberg refuses to scan a table carrying equality deletes (apache/iceberg#6568) while reading its metadata fine, and an engine that can apply them costs a multi-gigabyte image. Recording the commit correctly is the catalog's contract; applying deletes on read is the engine's. The README says so, says the read half was verified by hand once with StarRocks, and warns against later "upgrading" this to a PyIceberg rows read — which would either fail or, if PyIceberg ever starts skipping deletes instead of raising, pass by not looking. The workflow job asserts the suite actually ran rather than trusting a green exit, for the same reason: at least one top-level PASS and zero SKIP. No product change — the destination config is the generic catalog_type=rest with the standard OAuth2 client-credentials flow, and s3_path_style is not even set, since OLake turns it on itself whenever s3_endpoint is non-empty. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
dependabot[bot]
|
559ec33498 |
build(deps): bump github/codeql-action from 4.38.0 to 4.38.1 (#11423)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.38.0 to 4.38.1. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v4.38.0...v4.38.1) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.38.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
d6397fdc50 |
build(deps): bump dorny/paths-filter from 3 to 4 (#11422)
Bumps [dorny/paths-filter](https://github.com/dorny/paths-filter) from 3 to 4. - [Release notes](https://github.com/dorny/paths-filter/releases) - [Changelog](https://github.com/dorny/paths-filter/blob/master/CHANGELOG.md) - [Commits](https://github.com/dorny/paths-filter/compare/v3...v4) --- updated-dependencies: - dependency-name: dorny/paths-filter dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
264030c08c |
build(deps): bump docker/setup-qemu-action from 4.3.0 to 4.4.0 (#11421)
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 4.3.0 to 4.4.0. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](https://github.com/docker/setup-qemu-action/compare/v4.3.0...v4.4.0) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-version: 4.4.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
7643f4f541 |
rust: a seaweed-common crate for the address and TLS helpers both crates carry (#11358)
* rust: a seaweed-common crate for the address and TLS helpers both crates carry seaweed-volume and seaweed-worker are separate cargo trees with separate lockfiles and no root manifest, so anything both of them need has had to be written twice. Two of those copies are a correctness risk rather than a typing cost, and this crate is where they stop being copies. address.rs is the HTTP<->gRPC port rule: `host:port` means gRPC on port+10000, `host:port.grpcPort` names it outright. The two copies had already drifted — the worker's bracketed IPv6 literals, the volume server's did not — so the rule lives here once, returning a typed AddressError whose Display text is the volume server's original wording, with join_host_port public beside it. A test asserts two of those messages in full rather than by substring, because the wording is the contract its callers hand to a Status or an io::Error; the other three end in a std ParseIntError message, which is std's to reword. The enum is #[non_exhaustive] so a future variant is not a breaking change for either consumer. The tests are both crates' cases together, plus the IPv6, already-bracketed and normalisation cases neither copy covered on its own. tls.rs is install_default_crypto_provider. Both binaries link aws-lc-rs and ring transitively, so rustls cannot auto-select and tonic's client TLS panics on first use; each binary has to pin one and it has to be the same one, which is exactly the kind of choice that should not exist twice. It is safe to share because `cargo tree -i rustls` resolves a single rustls in each tree (0.23.37 in seaweed-volume, 0.23.43 in seaweed-worker) and cargo unifies all semver-compatible `rustls = "0.23"` requirements into one crate per binary, so this crate writes the same process-wide static its consumer reads. rustls is already in both graphs — directly in the volume server, through tonic's tls-aws-lc in seaweed-worker-core — so the dependency adds no crate to either. rust-version is 1.91.1, the lower of the two consumers' floors, so depending on this crate cannot raise either tree's MSRV; verified with `cargo +1.91.1 check --all-targets`. The lockfile is committed even though this is a library: CI builds it directly, so a committed lock is what makes those runs reproducible and their caches stable. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * rust: take the address and TLS helpers from seaweed-common Both public signatures are kept, so no caller outside the two wrapper files changes. parse_grpc_address stays `Result<String, String>` and maps the typed error through Display; server_to_grpc_address stays `Option<String>` and drops it with .ok(). Their doc comments and the volume server's 13 call sites are otherwise untouched. Three behaviours change, each in the direction of the copy that was already right: - The volume server now brackets IPv6 literals. `::1:19333` used to come back as `::1:29333`, which build_grpc_endpoint rejects with "invalid gRPC endpoint http://::1:19333: invalid authority" — an IPv6 master or EC peer could not be dialled at all. Two tests in grpc_client.rs pin it, one on the string and one on the endpoint the string builds. - The volume server now emits the *parsed* gRPC port of the dotted form instead of the original text it had just validated, so `host:8080.018080` and `host:8080.+18080` come back as `host:18080` rather than as authorities the URI parser rejects. Same port either way; only malformed spellings change. - The worker's dotted form now validates the HTTP port it discards. `server_to_grpc_address("host:abc.18080")` used to answer Some("host:18080"); it now answers None, which is what the volume server's copy has always done. install_default_crypto_provider becomes a re-export in both trees, so `crate::security::tls::install_default_crypto_provider` and `weed_lance_worker::tls::install_default_crypto_provider` still resolve. The lance crate's `rustls = "0.23"` was its only direct use of rustls and goes away with the body; seaweed-common states the same requirement, so neither the resolved version nor the enabled features move in either lockfile. The PEM test fixtures stay where they are. The two tests that use them are not duplicates: the volume server's exercises build_grpc_endpoint, and the lance one exists precisely because aws-lc-rs and ring are both linked in that crate's graph. Only the literals are shared, and exporting test fixtures from a library to dedupe two constants costs more than it saves. A path dependency outside both trees means every build context that copies one crate directory has to copy the other. The repo has one: the Rust source-build stage of docker/Dockerfile.go_build, which now copies seaweed-common beside seaweed-volume. Every workflow whose `paths:` filter keys on a crate directory gains `seaweed-common/**` — the two Rust test workflows, rust_binaries_dev, container_dev and performance. The tag- and dispatch-triggered ones (rust_binaries_release, container_release_unified, container_latest) have no `paths:` filter and need nothing. The two Rust test workflows also run `cargo test` in seaweed-common, from their unit-test job, because a path dependency is not a workspace member and neither tree's own `cargo test` reaches it. Each step builds into its job's cached target directory, and both cache keys now hash seaweed-common/Cargo.lock as well so a change there invalidates the cache it would otherwise silently reuse. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docker: keep go_build working for BRANCH revisions without seaweed-common The rust_builder stage copies seaweed-common unconditionally now that seaweed-volume path-depends on it, but BRANCH can name any revision — including ones that predate the crate. Create the directory in the builder stage so the COPY always has a source; an empty dir beside an old seaweed-volume is harmless. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
44ba070d83 |
helm: document admin.allowInsecureBind and drop stale comment
- README and values.yaml now describe the allowInsecureBind escape hatch alongside the non-loopback bind guard - remove a PR reference from the CI test comment |
||
|
Yongun SeongandClaude Sonnet 5
|
062238bb5c |
helm: let admin.allowInsecureBind satisfy the admin auth render guard
PR #11185 made `weed admin` refuse to bind a non-loopback address without -adminPassword or mTLS. PR #11228 added -allowInsecureNoAuth as an explicit opt-out for operators who restrict admin access some other way (e.g. a NetworkPolicy plus an authenticating reverse proxy). The chart's render-time guard added by #11236 (admin-statefulset.yaml, seaweedfs.admin.authEnabled) predates -allowInsecureNoAuth and only recognizes password-based auth, so there was no values.yaml path to express that choice: the chart would fail(...) even though the binary itself would start fine with a warning. Add admin.allowInsecureBind (default false) to the seaweedfs.admin.authEnabled helper's checks; when true it renders -allowInsecureNoAuth on the admin command and satisfies the render guard alongside the existing password-based checks. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Chris Lu
|
01bb3b3053 |
s3api: add Snowflake s3compat API integration tests (#11394)
* s3api: add Snowflake s3compat API integration tests Run the upstream snowflakedb/snowflake-s3compat-api-test-suite against a local SeaweedFS server in CI. test/s3/snowflake/run.sh starts weed server with S3 (-s3.autoCreateBucket=false so missing-bucket PUTs return NoSuchBucket), prepares the fixtures the suite needs (versioned bucket, deny-all-policy bucket, >1000-object prefix), clones the suite, patches it to path-style addressing, and runs mvn -Dtest=S3CompatApiTest. The suite also exposed that GetBucketLocation returned 404 NoSuchBucket for a malformed bucket name; validate the name first and return 400 InvalidBucketName like AWS. * test: harden snowflake s3compat runner per review - Pin the upstream suite to a tested commit (SUITE_REV) instead of the moving default branch - Bind the test server to loopback only - Require the AccessDenied error code when verifying the denied bucket - Fix README so go install runs in a subshell - checkout with persist-credentials: false - Make the concurrency group unique per PR, and widen path filters to the storage/operation/wdclient/cluster/pb packages the S3 stack uses * test: advertise loopback ip for snowflake test server -ip.bind 127.0.0.1 alone left the volume server advertising the host's primary address, so chunk uploads were refused. Also set -ip 127.0.0.1 and disable the Iceberg/Lance listeners so the harness is loopback-only and does not collide with other local services. |
||
|
Chris Lu
|
2d2619f0b4 |
ci: make telemetry deploy work on Oracle Linux 7 (#11377)
* telemetry/server: tidy module dependencies * ci: make telemetry deploy work on Oracle Linux 7 * ci: install telemetry unit and logrotate on every deploy * ci: abort telemetry deploy on install failures |
||
|
Eliah Rusin
|
4fc9ada2ec | ci: run seaweed-volume unit tests on Windows (#11349) | ||
|
|
166af06a2b |
rust: cargo fmt both crates, with a commented-out fmt --check CI step (#11329)
* rust: migrate seaweed-volume and seaweed-worker to tonic 0.14 / prost 0.14
tonic 0.14 boxes the contents of tonic::Status, which is what made every
RPC path trip clippy's result_large_err; the allow for that lint goes in
the next commit. The prost codec moved out of tonic into tonic-prost and
tonic-prost-build, so both build scripts now call
tonic_prost_build::configure() and both crates depend on tonic-prost for
the generated code. The `tls` feature was split into a per-backend
feature; `tls-aws-lc` is the same backend both crates already install
through rustls::crypto::aws_lc_rs.
tonic 0.14 depends on axum 0.8 and tower 0.5, which would have left a
second axum and a second tower in each tree next to the 0.7 / 0.4 the
crates named themselves. Bumping them keeps one copy of each: axum 0.8
only changes the path-parameter syntax for the routes here (`/:vid` ->
`/{vid}`, `/*path` -> `/{*path}`), tower 0.5 needs the `util` feature
named explicitly for ServiceExt::oneshot (it used to arrive through
tonic's feature unification), and tower-http 0.6 is the matching
release.
Lock files move only through cargo's own resolution for the new
versions; no other dependency was refreshed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust: drop the result_large_err allow now that tonic::Status is boxed
tonic 0.14 stores Status behind a Box, so Result<_, Status> is no longer
a large-Err type and clippy has nothing to say about it. Both crates
pass `cargo clippy --all-targets -- -D warnings` without the allow
(seaweed-volume in both feature sets), so the policy entry and its
comment go.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: drop the unused headers argument of try_expand_chunk_manifest
The parameter was already named `_headers`; nothing in the body reads it.
With it gone the function is under clippy's argument threshold and the
expect goes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: pass EC peer reads an EcInterval instead of ten arguments
fetch_one_interval, read_remote_ec_shard_interval,
do_read_remote_ec_shard_interval and recover_one_remote_ec_shard_interval
all took the same (vid, needle_id, shard_id, shard_offset, size,
expected_encode_ts_ns) tuple, and the two that reconstruct also took the
location map with the data/parity counts. Those are now EcInterval (Copy)
and EcShardMap (a borrow of the map plus the counts). The fan-out inside
recovery builds its per-shard request with `EcInterval { shard_id: sid,
..iv }`, which is the one place the old argument list was easy to get
wrong. Bodies destructure at the top, so the code below the signatures
is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: give the EC encoder an EcEncodeLayout and an EncodeRun
encode_dat_file took the Reed-Solomon shape and three block sizes as five
loose integers; they are now one Copy struct, EcEncodeLayout, which is
what Go calls ECContext. The per-row and per-batch helpers took the same
six sinks and the offsets; they become methods on EncodeRun, which owns
the borrows for one run, so each call names only the offset and block
size that vary. The byte-level work is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: describe a .dat rebuild with DatRebuild instead of nine arguments
write_dat_file_from_shards, its _with_dirs twin and the private
write_dat_file were three layers over one nine-argument signature. One
public function now takes a DatRebuild, whose shard_dirs is None when
every shard sits beside the .dat and Some(dirs) for the cross-disk
reconciled layout. The field docs carry what the function doc used to
say about the encode-time size and the block layout.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: split copy_file_from_source's fifteen arguments into two structs
CopyFileSpec is the per-file request (what to ask the source for, where
it lands, whether its bytes count as progress); CopyProgress is the
sender, throttler and report state that all three files of one
VolumeCopy share, held by &mut across the calls. The three production
call sites now read as the .dat/.idx/.vif literals they are, instead of
positional trues and falses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: create volumes from a VolumeSpec
Volume::new, DiskLocation::create_volume and Store::add_volume each
took the same five-value tail of Go's NewVolume argument list:
collection, replica placement, TTL, preallocation and needle version.
That tail is now VolumeSpec, a Copy struct whose Default is what almost
every test wanted anyway (empty collection, no replication, no TTL, no
preallocation, current version), so most of the 104 call sites shrink
to `&VolumeSpec::default()` or name the one field they set. The id,
directories, index kind and disk type stay positional because they
differ at every site.
Two imports that only test modules use moved into those modules, and
DiskLocation no longer imports ReplicaPlacement.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-worker: run cargo fmt
Layout only; no token in the workspace changes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: run cargo fmt
Layout only; no token in the crate changes. Every earlier Rust PR here
formatted only the blocks it touched so as not to drown its diff in
this one, and this commit is that debt paid in a single place. rustfmt
needed two passes to settle one block in handlers.rs; the committed
form is the fixed point, so `cargo fmt --check` is clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* ci: add a commented-out cargo fmt --check step to both Rust workflows
Same shape as the commented clippy step from #11312: the check is
written out so that making formatting a gate is a one-line uncomment,
and whether to do that stays a maintainer call.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
|
||
|
|
adaf3534fa |
rust: clippy-clean both crates and adopt the std APIs the 1.91 MSRV allows (#11312)
* rust: apply clippy --fix to both crates The mechanical part of a clippy sweep: `cargo clippy --all-targets --fix` on seaweed-volume and the seaweed-worker workspace, hand-reviewed. Both manifests declare their MSRV (1.91.1 and 1.94.1), so every suggestion clippy applied is within it: the collapsible_if sites become let chains (1.88, edition 2024), `% n == 0` becomes is_multiple_of (1.87), chunks_exact with a constant becomes as_chunks (1.88), repeat().take() becomes repeat_n (1.82), and io::Error::new(Other, ..) becomes io::Error::other (1.74). The rest is redundant clones, borrows, casts, closures and field names. Nothing here changes behaviour. The three let_and_return sites in needle_map.rs and store_ec.rs deserve a note: the `let result = ..; result` shape was a deliberate edition-2021 workaround to drop a redb guard before the table it borrows. Edition 2024 drops tail-expression temporaries before locals, which is why clippy now flags it, and the two comments that described the workaround say so instead. Manual edits on top of the tool output: the blocks clippy rewrote are re-indented the way rustfmt lays them out (only those blocks — the crate is not rustfmt-clean and a whole-crate fmt would bury this diff), the blank lines let_and_return left behind are removed, and the CRC legacy_value test compares against a literal worked out from the original shift formula rather than restating rotate_right. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust: clear the clippy warnings --fix cannot apply, and say why the rest stay Hand fixes for the lints clippy only reports. Behaviour is unchanged throughout; each rewrite is the one clippy names. - needless_range_loop (7): index loops over shard vectors become iterator loops. Where the old code indexed `v[..n]` the new loop iterates `v[..n]` so an undersized vector still panics the same way. - field_reassign_with_default (6): struct literals with `..Default`. - redundant_pattern_matching (3): `if let Err(_) = guard.check()` becomes `.is_err()`, which also releases the read guard at the end of the condition instead of at the end of the block. - manual_strip (2), manual_checked_ops, format_in_format_args, redundant_locals, wrong_self_convention (to_vif takes self by value, so it is into_vif; CompactEntry is Copy, so to_needle_value takes self). - type_complexity (2): `OrphanShardLoad` and `RawNeedleEntry` name two tuples that were spelled out inline. - new_without_default: CompactNeedleMap gets a Default that calls new(). - suspicious_open_options: a test helper spells out `.truncate(false)`, which is what `.create(true).write(true)` already did. What stays, and the attribute that says so: - too_many_arguments (10): `#[expect]` on each function. Folding 8–15 parameters into a struct is a design change, not a lint fix. - await_holding_lock / readonly_write_lock: one test holds the store write guard across a sleep on purpose, as a barrier that parks the copy task at the mount block. `#[expect(.., reason = ..)]` records it. - module_inception: needle/needle.rs mirrors the Go package layout. Two lints become crate-wide policy in `[lints.clippy]`, with the reason next to each: result_large_err, because every RPC path returns tonic::Status (176 bytes) and boxing it would change every handler signature; and needless_update, because `..Default::default()` on a protobuf message literal is what lets a proto gain a field without touching every constructor (all 11 sites are pb messages). The worker workspace gets the same table and its members opt in with `lints.workspace = true`; its generated plugin.rs also allows large_enum_variant on prost's oneof enums. Both crates are now clean under `cargo clippy --all-targets -- -D warnings`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust volume: use the std APIs the 1.91 MSRV already pays for The crate declares rust-version 1.91.1, so a few things the code still worked around are plain std now. All of them come from the 1.85–1.91 release notes; nothing here needs a newer toolchain than the manifest already requires. - std::sync::LazyLock (1.80) replaces the lazy_static! block in metrics.rs, and the lazy_static dependency goes. Every use site reads the same through Deref, so no caller changes. - Duration::from_mins / from_hours (1.91) replace `from_secs(v * 60)` and `from_secs(v * 3600)` in the option parser and the shard-location refresh TTLs. One difference for the parser: an absurd count that overflows u64 seconds now panics in release builds too, where the multiplication used to wrap. - Result::flatten (1.89) replaces `.and_then(|r| r)` on the replication join handle. - OsStr::display (1.87) replaces `to_string_lossy()` where the name was only being formatted; the output is byte-identical. - `#[allow]` becomes `#[expect]` (1.81) on the suppressions that are meant to be permanent, so a suppression that stops being needed becomes a warning rather than lingering. Doing that found four that already had: dead_code on ChunkManifest, base_name and last_io_error, and too_many_arguments on read_from_data_shards, which is down to seven parameters. Those attributes are deleted. The three allows that depend on cfg (a unix-only mutation, a linux-only field set, a profiling-only parameter) stay as allow, because expect would be unfulfilled on the other platforms. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * ci: add a commented-out clippy step to both Rust workflows Both crates are warning-free under `cargo clippy --all-targets -D warnings` now. Whether that becomes a gate is a policy call, so the step is present but commented out; uncommenting it is the whole change. The comment points at the `[lints.clippy]` table where crate-wide exceptions are recorded, so the gate does not become a reason to sprinkle allows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust volume: guard parse_duration against overflow panics Duration::from_mins/from_hours panic when the count overflows u64 seconds. Use checked_mul so an oversized CLI value falls back to the parser default instead of crashing volume startup. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
dependabot[bot]
|
fd33c07843 |
build(deps): bump github/codeql-action from 4.37.9 to 4.38.0 (#11311)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.9 to 4.38.0. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v4.37.9...v4.38.0) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.38.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
f1f6886d0e |
fix: rebase on latest master before pushing star history chart (#11240)
* fix: rebase on latest master before pushing star history chart The daily star history workflow git push was rejected with a non-fast-forward error because new commits landed on master between the checkout and the push. Fetch full history (fetch-depth: 0) and rebase the generated commit on top of the latest remote branch before pushing so the workflow no longer fails when master has moved. * fix: retry rebase-and-push to handle concurrent master updates Address review feedback: a one-shot rebase still races if master advances between the rebase and the push. Match the bounded retry loop used by java_release.yml — push first, and on rejection rebase and retry up to five times before failing. * fix: serialize runs and ensure every rebase is followed by a push Address review feedback: - Devin (line 54-55): the old loop rebased after the 5th failed push but never pushed the rebased commit. Restructure so each rebase (attempts 2-5) is followed by a push attempt, with a clear 5-attempt cap. - Greptile: overlapping runs could conflict on the SVG during rebase. Add a concurrency group (cancel-in-progress: true, matching the repo convention) so only one chart regeneration runs at a time. * fix: scope concurrency by ref and guard rebase against transient failures Address review feedback: - Devin (line 16): the global star-history concurrency group let a manual run on another branch cancel an in-flight daily master update. Scope the group by github.ref so only same-branch runs cancel each other. - Greptile (line 58): git pull --rebase runs under the fail-fast shell, so a transient fetch error or conflict aborted the whole step before remaining attempts ran. Guard the rebase so a failure aborts the in-progress rebase and continues to the next attempt instead. |
||
|
Chris Lu
|
0ce5ca42ea |
helm: supply admin auth in CI renders that enable admin (#11239)
PR #11236 added a render-time guard that fails the chart when admin.ip is non-loopback (default 0.0.0.0) and admin auth is not configured, since weed admin 4.46 refuses to bind a non-loopback address without authentication. Several pre-existing helm_ci.yml test cases enable admin.enabled=true as part of "everything on" renders without a password, so helm template now exits non-zero and the Verify template rendering step fails. Add admin.secret.adminPassword to the four render calls that turn on admin without auth (IAM gRPC opt-in, NetworkPolicy EVERYTHING, egress without kubeApiServer.cidrs, and the license ALL_ON dict), using the same key ci/admin-values.yaml already uses. |
||
|
Chris LuandCopilot
|
213f4c5d5c |
release: wait for Go proxy propagation (#11219)
Allow normal post-tag proxy propagation before dispatching downstream releases, while preserving the check that prevents them from pinning the previous commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
dependabot[bot]
|
34936c610d |
build(deps): bump helm/kind-action from 1.14.0 to 1.15.0 (#11212)
Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.14.0 to 1.15.0. - [Release notes](https://github.com/helm/kind-action/releases) - [Commits](https://github.com/helm/kind-action/compare/v1.14.0...v1.15.0) --- updated-dependencies: - dependency-name: helm/kind-action dependency-version: 1.15.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
5e3af51d91 |
build(deps): bump docker/setup-qemu-action from 4.2.0 to 4.3.0 (#11213)
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 4.2.0 to 4.3.0. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](https://github.com/docker/setup-qemu-action/compare/v4.2.0...v4.3.0) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-version: 4.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Eliah Rusin
|
ed5b342f0c |
rust volume: optional redb insert_before bulk load (#11205)
* rust volume: quick-repair redb on durable checkpoints set_quick_repair(true) on the durable checkpoint transaction so an OOM-killed volume server opens without a full-file repair scan. * rust volume: optional redb insert_before bulk load Behind redb-experimental-cursor (default off). Production binary stays on sorted insert(). CI unit tests run both feature settings. * rust volume: exercise insert_before across leaf splits Replace the 5-key cfg clone with a 4000-key reverse-order rebuild so CursorMut::insert_before hits page splits. CI runs the feature only on storage::needle_map unit tests. |
||
|
Chris Lu
|
d8a40ef750 |
ci: pin actions/setup-python to v7 in star_history workflow (#11191)
The star_history workflow referenced actions/setup-python@v8, which does not exist, causing the workflow to fail at the "Set up job" step. Pin to v7, matching the version used across the other workflows. |
||
|
Chris Lu
|
ff0d5a9adf |
ci(mount-windows): clean up processes and don't let Logs step fail job (#11174)
The "Mount and exercise" step left the weed.exe mini server and the final WinFsp mount running when it exited. The next step's pwsh.exe then failed with STATUS_DLL_INIT_FAILED (0xC0000142), failing a job whose actual test step had passed. The same code passed on both the PR branch and the next master run, so this was a transient launch failure — but it was caused by an unclean environment and made fatal by a diagnostic step. Tear down all weed.exe processes at the end of the test step so subsequent steps launch into a clean environment, and mark the Logs step continue-on-error so a diagnostic step can never fail the job on its own. |
||
|
Chris Lu
|
567578f08d |
docs(readme): replace star-history.com with self-generated chart (#11171)
* docs(readme): replace star-history.com with self-generated chart The star-history.com SVG is a third-party dependency that can rate limit or go down. Replace it with a GitHub Action that fetches stargazers via the REST API and renders an SVG with matplotlib, committing note/star_history.svg weekly. The README references the committed file directly, so the chart has no runtime dependency on any external service. * ci(star-history): run daily instead of weekly |
||
|
Chris Lu
|
5a515adab2 |
s3: HeadObject with partNumber returns the part's size and 206 (#11166)
* s3: HEAD with partNumber reports the part's size and range HeadObject set its headers from the total object size and then only validated the partNumber, so a client probing part 1 with HEAD got the whole object's Content-Length and a 200 while the same GET returned the part's size, a Content-Range and a 206. Resolve the part's byte range before the headers are written, through the range logic GetObject already used, and answer a partNumber HEAD as the ranged HEAD that AWS documents. * s3: answer an unsatisfiable partNumber with 416 InvalidPartNumber GET and HEAD rejected a partNumber past the number of parts with 400 InvalidPart, the code for a missing part in CompleteMultipartUpload. AWS answers a read of a part that does not exist with 416 InvalidPartNumber, which lets a client probing for the part count tell the two apart. The ceph suite pins RGW's 400 InvalidPart here, so the s3tests jobs patch that expectation the way they already patch prefix ordering. * s3: keep the whole-object checksum off a partNumber response The stored checksum covers the whole object, so it is already withheld from a ranged read. A partNumber HEAD now describes one part while the request carries no Range header, so exclude it there too rather than handing a client a checksum that does not match the bytes described. * s3: resolve a partNumber against the parts the object records Completion accepts ascending, not consecutive, part numbers, so the part count is not the highest part number. Comparing the two rejected an uploaded part 3 of a two-part object, and let a request for the absent part 2 fall through to the positional chunk lookup and serve part 3's bytes. Ask the recorded boundaries for the part instead, and keep the count comparison for objects written before boundaries were stored. * s3: apply a client Range within the part on HEAD too GET narrowed the part by a Range sent alongside partNumber; HEAD reported the whole part, so the two disagreed again for a request that carries both. Move the narrowing into the shared range lookup so either verb describes the same bytes. |
||
|
Chris Lu
|
1205630c43 |
ci: track the actions in the signing jobs by tag again (#11137)
The repository tracks actions by tag with dependabot; the signing jobs follow the same convention. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa |
||
|
Chris Lu
|
a8f763e717 |
Sign the published Docker images with cosign (#11129)
* ci: composite action that signs and verifies an image with cosign Keyless, by digest, with a verification pass against the calling workflow's own identity right after signing. Signatures use the .sig tag layout rather than the OCI-referrer bundle cosign 3 writes by default, since that is what the verifiers people run today read. Dependabot is pointed at the action so the cosign-installer pin keeps moving. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker release: sign every variant on both registries The merge job signs each variant's multi-arch index on GHCR and Docker Hub once the tag exists, recursively so the platform images are covered too. latest re-tags the same manifest and inherits the signature. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker dev: sign the dev image Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker latest: sign a latest rebuilt by hand Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker release: sign the foundationdb image Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker: sign the per-version foundationdb and rocksdb builds They push to the same repository as the releases, so an admission policy that verifies chrislusf/seaweedfs would otherwise reject them. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker: document image signature verification Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * ci: pin the actions the signing jobs newly run by commit These run with registry credentials and the OIDC token that signs under the repository's identity, so a retargeted tag upstream must not be able to reach them. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker latest: pass the dispatch tag through env, not the script Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker: complete Kyverno policy, digest note, identity scope Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker latest: keep the dispatch tag out of the manifest script too The step predates signing, but the job now holds the OIDC identity. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * ci: pin every action in the jobs that sign The jobs that hold the OIDC identity run these with registry credentials, so a retargeted tag upstream must not reach them. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker release: copy and sign the digest the run created, pin the rest crane copy and the signature both resolved the tag, which another publisher could move between the two steps. The index digest is read once, right after it is created, and the Docker Hub copy and both signatures use it. The manual latest rebuild gets the same treatment. The actions in these jobs are pinned to commits, crane to v0.22.0 by checksum, and the sparse checkout no longer keeps the token. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker latest: the signing job checks out the workflow's own commit The job only assembles and signs manifests, so nothing there needs the source_ref checkout; the local signing action now comes from the same revision as the workflow file that calls it. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa * docker release: take the index digest from the create result imagetools create writes the descriptor it pushed with --metadata-file (buildx 0.32+, the runners ship 0.36), so the digest no longer comes from re-resolving the tag even within the same step. Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa |
||
|
Chris Lu
|
59916d8978 |
helm: values-driven labels on every ingress (#11127)
* helm: values-driven labels on every ingress Each ingress already takes annotations from values, but its labels were a fixed block, so tools that select ingresses by label (ExternalDNS label filters, for one) had nothing to key on. Every ingress block now has a labels map rendered after the standard app.kubernetes.io labels, including the Traefik IngressRouteTCP that shares the filer gRPC values. Claude-Session: https://claude.ai/code/session_01L6eJGXtYkwe1W9QjGeUgr1 * helm ci: render check for ingress labels Claude-Session: https://claude.ai/code/session_01L6eJGXtYkwe1W9QjGeUgr1 |
||
|
Chris Lu
|
721499a05a |
release: judge downstream releases by their run, excluding runner-queue time (#11057)
Claude-Session: https://claude.ai/code/session_01Y228KU8MLsGmfpcbGxgjwh |
||
|
dependabot[bot]
|
8c3695fd4a |
build(deps): bump actions/setup-java from 5 to 6 (#11047)
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 5 to 6. - [Release notes](https://github.com/actions/setup-java/releases) - [Commits](https://github.com/actions/setup-java/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/setup-java dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
3a3d513cb8 |
build(deps): bump github/codeql-action from 4.37.8 to 4.37.9 (#11045)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.8 to 4.37.9. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v4.37.8...v4.37.9) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.9 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
dfa75ce231 |
build(deps): bump actions/upload-artifact from 4 to 7 (#10922)
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 7. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v4...v7) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
61d588e455 |
build(deps): bump github/codeql-action from 4.37.6 to 4.37.8 (#10923)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.6 to 4.37.8. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v4.37.6...v4.37.8) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.8 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
863fec6c3f |
S3: let a key that is a prefix of other keys be an object (#10912)
* filer: keep the sentinel when CreateEntry reports an update failure CreateEntry flattened the error UpdateEntry wraps, so errors.Is stopped matching and ErrExistingIsDirectory and ErrExistingIsFile never reached the S3 mapper, which answered a retryable 500 instead. * s3: let a key that is a prefix of other keys be an object S3 keys are flat, so "a/b" and "a/b/c" are independent objects that coexist in either write order. The filer stores a key as a path, so one of them has to live on the directory the other is nested under. Writing the nested key first refused the prefix key outright. Writing it second promoted the file to a directory, which kept its data but lost the key: an empty object left nothing to recognise it by and disappeared, and one with data listed under a trailing slash it never had. Mark the directory that carries such a key, and write the object onto it when the path is already a directory. The mark makes an empty prefix object visible to listings and readable by GET and HEAD, keeps the empty folder cleaner off it, and lists it under the key it was written with. Deleting the key strips the mark back off along with the data. * filer: keep a TTL off a directory that stands for an object An expired entry is deleted a row at a time, so expiring a directory removes it and leaves everything under it unreachable. Promoting a file to a directory carried its TTL across, and a promoted file is exactly the one that has keys nested under it. Drop the TTL on promotion, and leave one an older build wrote alone. The lifecycle worker still expires the object, through the delete that leaves the directory behind. * s3: delete the null version of a key other keys are nested under The routed delete cannot remove an entry that other keys live under, and answered a retryable 500 rather than falling back to the lock path the unversioned delete already falls back to. That path then looked the entry up under the bucket with the whole key as its name, so the demote wrote it back one directory too high and failed as not found. Fall back on any non-precondition error, and split the key before deleting it. Trailing-slash directory markers with children reach the same delete. * filer: keep the sentinel when MkFile and Mkdir report a create failure Same flattening one layer out: every mkFile caller lost the sentinel, so a CopyObject onto a key that other keys are nested under answered a retryable 500 where a PutObject of the same key answers 409. * s3: copy and rename a key that other keys are nested under Such a key is stored on the directory those keys live in, and copy and rename both refused it: the source lookup maps every directory entry to NoSuchKey, so a key a plain GET serves could not be copied or moved, and the destination side refused it as a directory conflict. The source is read through a view of the entry as the object it names. The destination is written the way a PutObject of that key writes it. A rename at either end copies the object's own data across and strips it off the source key rather than going through AtomicRenameEntry, which moves a directory by moving everything under it - the nested keys are not part of what is being renamed. |
||
|
Chris Lu
|
7ebf2ebac3 |
Build the Rust worker against the protoc that ships with the build (#10881)
* worker: compile plugin.proto with the protoc that ships with the build seaweed-volume already does this: protoc-bin-vendored carries the binary, so the build needs no package manager and every build sees the same version. An explicit PROTOC still wins, which is what lets the lance crates - whose own build scripts read the same variable - share it. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: point the worker builds at the vendored protoc The jobs installed protobuf-compiler for lance's build scripts. They read PROTOC, so pointing it at the binary protoc-bin-vendored already puts in the registry serves them without a system package - one less apt call on the way to a release, and the same protoc a developer's build uses. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docs: say what the worker build needs from protoc The lance crates' build scripts are the ones that need it, not ours, and they take the same vendored binary. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris Lu
|
cc8364a03e |
Ship the Rust maintenance worker with the release (#10879)
* worker: name the binary weed-worker It is the Rust side of `weed worker`, the way weed-volume is the Rust side of `weed volume`, and lance is the first family of jobs it carries rather than the only one it ever will. The crate keeps its own name: when a second family arrives the bin target moves to a crate of its own, under this name. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docker: ship the Rust maintenance worker in the image Lance table buckets need a worker that can read the format, and until now the only way to get one was a Rust toolchain and a cargo build. It now sits at /usr/bin/weed-worker beside the Rust volume server, reached as `docker run chrislusf/seaweedfs worker-rust --admin host:23646` — the verb mirrors volume-rust, so plain `worker` still runs the Go one. Taken pre-built or not at all: the lance jobs pull in arrow and datafusion, far too large a tree to compile inside the image build, so an architecture CI did not build for gets the empty placeholder the entrypoint refuses to exec, the way the Rust volume server already does. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: build the Rust worker for the container images The same native cross-compile the volume server uses, so the release, latest and dev images all carry it on amd64 and arm64. The artifact holds both binaries now, so it is named for that rather than for the volume server. Only the release directory each job builds is cached: with a debug profile beside it the worker's target/ reaches 24GB, against a 10GB cache budget. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: publish Rust worker binaries with the release Linux amd64 and arm64 only: the worker runs beside the cluster it maintains, and its dependency tree makes every extra target an expensive build. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: build and test the Rust workers on change Nothing built seaweed-worker in CI, so the release and the container images would have been the first place a break showed up. Tests run in release too, rather than compiling lance, arrow and datafusion again in another profile. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docs: say how to get a released worker Neither the image nor the release tarballs were mentioned; a toolchain and a cargo build read as the only way in. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: install protoc for the Rust worker builds lance's crates compile their own protos, and unlike seaweed-volume they do not vendor a protoc to do it with, so every job that builds the worker failed at lance-encoding's build script. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: do not persist credentials in the worker release checkout The upload step is handed a token explicitly; a cargo build script should not find another one sitting in the checkout's git config. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docker: keep the worker's argument boundaries Unquoted $@ splits on whitespace and expands globs, so an argument carrying either arrived as something else. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris Lu
|
c3f4d799b5 |
helm: install chart CI against an image tag that exists (#10877)
* helm: install chart CI against an image tag that exists The release bumps appVersion on master well before the container build publishes that tag, and the chart CI runs on the bump commit, so every release turns it red with ImagePullBackOff. Resolve the tag first and fall back to latest while the new one is still building. * helm: run the chart CI when the workflow itself changes * helm: bound the registry lookup in the chart CI An unbounded curl can hold the job, and the log did not say why the tag was rejected. Cap it and print the status. |
||
|
Chris Lu
|
34bb444f33 |
test: drive the Lance namespace with Spark (#10864)
* test: drive the Lance namespace with Spark
The counterpart of catalog_spark, which does this for the Iceberg REST
catalog. Spark is the engine most likely to be pointed at a lakehouse,
and it reaches the Lance catalog through the connector's DSV2 catalog -
org.lance.spark.LanceNamespaceSparkCatalog with impl=rest - over the same
routes every other client uses.
SHOW NAMESPACES -> ['`sparklance-lcephd80`.ml']
SHOW TABLES -> ['sparklance-lcephd80$ml$embeddings']
count -> 3
filtered -> [(2, 'two'), (3, 'three')]
count after a second commit -> 4
The second insert is there on purpose: a store that cannot order commits
fails on the second one, not the first.
Two things the run settled that were guesses beforehand. CREATE TABLE
works, because the connector declares through the namespace and writes the
data itself rather than pushing Arrow at the server. And SHOW TABLES
returns the namespace's own identifiers - bucket, namespace and name
joined by the delimiter - not bare Spark table names.
Credentials go under the catalog's storage.* prefix, which is handed to
lance as object_store options; a gateway without STS vends none, the same
trap the LanceDB suite documents.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame Spark for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: keep the ivy cache under the user's cache directory
It is mounted into a container running as root, so a shared temp path lets
another local user pre-create it and choose what Spark loads.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: assert the vector column's type, not only its name
A column that came back as array<double> or array<string> would still be
called vector and still pass.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: read the dataset off its location for real
The catalog being optional is the property that lets duckdb and pandas read
these tables; it was asserted in a comment and printed, never exercised.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the Spark Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: say the hosts in the README are placeholders
The suite passes dynamically allocated host.docker.internal ports.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
|