* vector: boto3 s3vectors conformance harness
test/vectors spins a mini cluster, an s3 gateway with -port.vectors and
-vectors.grpc, and a weed-vector-server, then exercises the AWS client
end to end: bucket and index lifecycle, put/get/list/delete vectors,
and a filtered query asserting distances and metadata.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run the s3vectors conformance subset
An e2e job in rust-vector-tests: builds weed and weed-vector-server,
then runs test/vectors against a mini cluster plus the vectors gateway.
The workflow now also watches vector.proto, its stubs, and test/vectors.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: fix e2e protoc and codespell findings
The e2e job needed the same vendored-protoc step as the unit-test job —
lance's build scripts locate protoc themselves. Codespell flags the AWS
`$nin` operator literal and the `lits` variable: exempt `nin` in
.codespellrc like the existing word exemptions, rename the variable,
and fold the negation into the IN/NOT IN symbol directly.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: classify catalog errors by code, not message text
* s3: drop the implicit vector namespace on vector-bucket delete
* test: point the vector engine at the gateway lance port
* test: bind the vectors gateway listeners to loopback
* test: make the filter and distance assertions load-bearing
* test: rebuild the harness binaries on every run
* ci: fix the vectors end-to-end job
* ci: run the vector tests on gateway and command changes
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: retry the timing-sensitive s3 lifecycle expiration tests once
test_lifecycle_expiration and test_lifecyclev2_expiration assert exact
object counts after fixed sleeps. The lifecycle worker expires objects
via meta-log replay, and under the suite's write flood a shard's replay
can trail an object's PUT event past the test's window (observed on
master and PR runs, SQL-store job only). Split them out of the main
list and allow one retry; each attempt is independent so real
regressions still fail.
* ci: allow a second retry for s3 lifecycle expiration tests
* vector: project _distance and _rowid explicitly
Lance warns that a knn projection without _distance will stop
auto-including it; spell both scoring columns out.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: reference-comparison tests
The spike's exit gate: assert the engine's answers equal a bare
scan().nearest() on the same pinned version — over a fresh index, over
unindexed fragments, over a two-segment index after optimize_indices,
and after a delete. A version-pinned handle querying through
maintenance sees only its own version.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: weed-vector spike CLI
describe / coverage / query / get against a namespace-registered table
or a plain dataset URI, so the spike runs against a real SeaweedFS Lance
table bucket and against a local fixture alike. Flag and env names
mirror weed-worker.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: rust-vector-tests workflow
Build and test seaweed-vector in release on pushes and PRs touching it
or its path-dependency trees, mirroring rust-worker-tests.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: derive Serialize for the spike CLI's JSON output
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: --postfilter on the spike CLI
Track the Query field rename; the flag stays an explicit opt-out of
prefiltering for comparison runs.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: assert the pinned query's answers
The maintenance test passed even with zero pinned hits: an empty result
satisfies all() trivially. Assert the hit count and compare the pinned
answers against the reference on the pinned handle.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: vector tests also watch plugin.proto
weed-vector-core path-depends on weed-lance-worker, which compiles
weed/pb/plugin.proto through seaweed-worker-core; a proto-only change
would skip these checks and could break this workspace.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: describe prints layout and coverage
The describe doc comment promised the vector layout and index coverage;
print both, and propagate count_rows errors instead of printing 0.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The 24.04 bump broke build-rust-binaries: aws-lc-sys C objects compile against noble's glibc 2.39 headers, which redirect sscanf/strtol to the C23 __isoc23_* variants musl does not provide, failing the link.
Same fix as the enterprise build on seaweedfs/artifactory:
- arm64: keep aarch64-linux-gnu-gcc, link a tiny forwarding shim that
resolves __isoc23_sscanf/__isoc23_strtol to musl's vsscanf/strtol.
- amd64: point CC at musl-gcc so C deps compile against musl headers.
GitHub is retiring ubuntu-22.04; move all 94 runs-on pins across 40 workflows to ubuntu-24.04. rust_binaries_release: the arm64 cross step was jammy-era — noble keeps apt sources in deb822 ubuntu.sources (the sources.list edit was a silent no-op that left amd64 repos open to arm64 and 404'd) and ports suites must say noble. Pin deb822 stanzas to amd64 and add noble ports entries.
The S3 Remote Cache tests exercise the remote-storage S3 client, but the
path filter did not include weed/remote_storage, so a tagging change
there can regress the suite without running it on the PR.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(s3tables): tolerate stale FUSE mounts in disk diagnostics
* ci(s3tables): bound df diagnostics with timeout, explain once
A stale FUSE mount can also hang statfs, not just fail it, so wrap df in
timeout 15; one comment at the first site covers all sixteen.
* ci: tolerate stale FUSE mounts in df diagnostics elsewhere
The same self-hosted runners serve the other workflow files; a bare
df -h there can stall or fail a job the same way.
* ci: unmount disconnected FUSE test mounts before jobs start
* ci: skip FUSE cleanup when the variant checkout is skipped
* ci: reclaim the crashed run's test dir once its stale FUSE mount is detached
* ci: stop tracking python bytecode for the FUSE cleanup action
The helper's __pycache__ slipped into the tree; ignore it so local
test runs cannot re-add it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run the FUSE cleanup action's unit tests on action changes
test_cleanup.py was never wired into CI, so a regression in cleanup.py
would ship silently. A path-filtered workflow runs it on every change
under .github/actions/.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: match every FUSE test mount layout in the cleanup action
The candidates list only matched seaweedfs_fuse_test_*/mount, so stale
mounts left by fuse_failover and fuse_p2p runs (mountN leaves under
seaweedfs_fuse_* dirs) and the posix-compliance workflow's fixed
/tmp/seaweedfs/mount were never reclaimed and could still wedge df.
* ci: run the FUSE mount cleanup in the fuse_failover and fuse_p2p workflows
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: serialize quarantine records with their layout marks
A quarantine publishes a (node, vid) record and marks the replica read-only at the layout level; release and heartbeat flag updates must observe that pair as one state. Store-then-mark let a concurrent release delete the record before the mark landed, stranding the surviving replica read-only; mark-then-store let a heartbeat clear the mark before the record was visible, returning a conflicting replica to the writable list.
quarantineLock now covers every record<->mark pair: flagDuplicateVolumeId's quarantine, releaseResolvedQuarantine's undo, the isQuarantined-gated marks on registration, and the quarantine-folded read-only flag writes on heartbeat paths. Remote server pushes stay outside the lock; a push that lands after the record was released is undone, and a failed release push re-adds the record for a later retry.
The release test now waits for in-flight async flags to settle instead of asserting on a single read.
* topology: fold quarantine into RegisterVolume's flag rebuild
RegisterVolume rewrites every replica's location read-only flag from what
its node last reported, and reported flags deliberately exclude quarantine.
Any registration of a shared vid -- a reconnected replica, a new copy, the
self-heal and delta-heartbeat re-register paths -- therefore cleared the
quarantine mark on every replica in the location, while the gate afterwards
only re-marked the registering node. Two call sites had no gate at all.
RegisterVolume now takes an isQuarantined predicate and evaluates it under
accessLock for every replica it touches, so the mark is never dropped in
the first place: a release unmarks after the rebuild, and a quarantine
mark re-applies on the next rebuild, with no window between.
* topology: keep quarantine when a vacuum commit re-adds a replica
SetVolumeAvailable re-adds the replica into the location list; a re-added
slot starts with a cleared read-only flag, so a quarantined replica came
back flag-free and the next ensureCorrectWritables could return a
conflicting volume to the writable list. Pass the quarantine state in and
set the flag back under the same accessLock, with the caller holding
quarantineLock so release cannot interleave between the check and the mark.
* topology: merge quarantine records instead of overwriting them
enforceQuarantine wrote back the state copy it loaded before its remote
push, so a concurrent writer's enforced=true was lost: the server stayed
read-only while the record said unenforced, and a later release skipped
the writable push it owed. The write-back now reloads the record under
quarantineLock and merges -- enforced wins from either side, reported is
taken from the newer pre-enforcement observation only while no push has
landed yet.
All three compensating writable pushes had the same hole in reverse: a
failed un-quarantine left the server read-only with no record to retry
from, and release's restore blindly overwrote any record a fresh flag
pass had created in between. They now go through restoreQuarantineRecord,
which adds the record only when absent and otherwise sets enforced on
the current one, preserving its reported flag.
* topology: hold quarantineLock over registration's quarantine check-and-mark
RegisterVolume reads the quarantine record and marks the replica
read-only under accessLock, but the pair ran without quarantineLock.
A release could push writable, watch the record get re-read, and then
delete it: the replica stayed read-only with no record left to clear,
and unchanged writable heartbeats skip SetReplicaReadOnlyFlag so the
mark never repaired. Holding quarantineLock around the check-and-mark
in all three production callers serializes it against the release
clear-and-delete.
* topology: serialize remote quarantine pushes behind a record recheck
releaseResolvedQuarantine deletes the record under quarantineLock and
sends the writable request after unlock. A conflict flagged while that
request is in flight creates a fresh record and marks the replica
read-only again, and the older writable request can still land last,
leaving the server writable under an active quarantine. The reverse
flap strands a stale read-only mark the same way.
Route every QuarantineVolumeFn call through pushQuarantine, which
holds quarantinePushLock across the remote call and re-checks the
record under quarantineLock right before sending: a read-only push
requires the pair to exist, a writable push requires it to be gone.
A skipped push means the decision already moved on, so the final
server state always follows the newest record.
* topology: serialize quarantine pushes per replica, not globally
A single quarantinePushLock held across the remote mark call let one slow
volume server (up to the 10s client timeout) delay pushes for unrelated
replicas. Keying the lock by (node, vid) keeps pushes to one replica
ordered in record-decision order while pushes to others run in parallel.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: restore the layout read-only mark with a released record
A failed writable push re-armed the quarantine record but left the
replica marked writable in the layout, so PickForWrite could route to a
replica the server still holds read-only.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: wait out async duplicate checks in the release-retry test
TestFailedReleasePushRestoresQuarantineMark asserts layout marks around a
release pass, but the two RegisterVolumeLayout calls each start a
background flagDuplicateVolumeId that can still be mid-flight. A late
mark could land after the final release and fail the writable assertion
even though the code behaved correctly. Track the async passes in a
WaitGroup and drain them before the explicit flag call so the rest of
the test is deterministic.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: keep quarantine when a replica's layout key is unreadable
* topology: keep a failed writable push's restore ordered under the push lock
* topology: retry the quarantine release on every duplicate check
* ci: isolate Java and S3 test servers on persistent runners
* ci: drop the duplicated -ip flags in the lifecycle test server
The bind flags were already on the command; the isolation pass added a
second pair.
* topology: retry a failed quarantine writable push on the next report
* topology: retry pending quarantine releases off the leader pulse
A failed writable push re-arms the quarantine record with a retry that only
enforceQuarantine fires, and that runs only for volumes a heartbeat carries.
Once the server has reported the replica read-only it drops out of the delta
reports, and restoreQuarantineRecord keeps the stored flag matching the
server's so the digests agree and no full list is requested - the pending
release then never retries and the idle volume stays read-only forever. Run
the same release pass from the leader's pulse loop so a pending retry no
longer waits on a report.
* topology: exercise the pulse-driven quarantine release retry
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* release: emit digest-bound provenance and SBOM attestations
Per-platform builds now record SLSA provenance at mode=max and a
BuildKit SBOM, so the builder identity and resolved dependencies are
captured instead of the empty predicate the 4.48 build record carried.
After the multi-arch index is assembled and signed, the merge job
publishes GitHub artifact attestations bound to the merged digest and a
syft SPDX SBOM of the image, giving digest-pinning consumers a
verifiable record on the exact index they pull.
* release: keep unfixed findings in the trivy SARIF upload
ignore-unfixed dropped every finding without an available patch, so the
uploaded report understated the image's exposure. The scan stays
report-only; only the completeness of what is uploaded changes.
* release: generate and attest SBOMs per platform digest
An index-wide syft scan only describes its default platform, so the SBOM
attested to the merged multi-arch digest misrepresented every other
manifest. Scan and attest each platform digest instead, which also drops
the sbom-action upload defaults that raced release-asset permissions and
artifact names across the merge matrix.
* release: resolve per-platform index to its image manifest before syft/cosign
push-by-digest records point at OCI indexes (image + attestation
manifests), so scanning or attesting the index digest picks the
runner's platform and fails outright for the other arches under
set -e. Resolve the non-attestation manifest inside each index and
bind the SBOM attestation to the image digest a same-platform pull
of the merged index actually returns.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: invoke pip via python -m in s3tests clone steps
The self-hosted runner reaches the Clone s3-tests step with the
setup-python toolcache interpreter on PATH but no pip shim, so the
bare 'pip install' exits 127. python -m pip resolves through the
interpreter that is already on PATH, matching the helm_ci workflow.
* ci: invoke pip via python3 -m in s3tests clone steps
The adjacent fix_s3_tests_bucket_conflicts step already runs python3, so
python3 is the interpreter guaranteed on the self-hosted PATH; a missing
python alias would fail the same way the missing pip shim did.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The rocksdb release image fails to compile: grocksdb v1.11.1 calls
C.rocksdb_block_based_options_set_block_align,
C.rocksdb_block_based_options_set_index_block_search_type, and
C.rocksdb_block_based_options_set_separate_key_value_in_data_block, all
of which only exist in RocksDB v11.x headers. grocksdb v1.11.1's own
build script pins 11.1.2, so move every default from v10.10.1 to
v11.1.2 to match.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Self-hosted runners do not ship fuse3, so the setuid repair step
failed with 'no fusermount3 in PATH' before the FUSE jobs could run.
Install the package instead of failing outright.
The registry container is started with --restart=always but never
removed during teardown, so once a machine runs the job, every later
run on that machine fails with a container name conflict. Reuse the
running container when it exists and recreate it only when missing or
broken.
Some self-hosted runners (e.g. seaweedfs-server03) also lack the
Docker Compose plugin, so install it via docker/setup-compose-action,
which is a no-op where compose is already available.
The workflow relied on a Go toolchain preinstalled on the runner
image; self-hosted runners without setup-go fail at autobuild with
'go: executable file not found'.
Follow-up to #11985/#11986: setup-go/setup-java/setup-python steps
added or restructured since (e.g. by #11981's shared-artifact layout)
were missed. On self-hosted runners setup-go restores/saves a ~5GB
module cache archive per job — pure waste when the module cache is
already on local disk — and the restore blocks the job for minutes.
cargo registry/target and docker-image caches are already local on a
persistent self-hosted runner; the save/restore pass is pure overhead
(the setup-go cache-save stall on awscli-tests was the same pattern).
Gate every actions/cache step on runner.environment == 'github-hosted'.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: skip setup-go cache on self-hosted runners
The module cache is already local on a persistent self-hosted runner;
archiving ~/go/pkg/mod into cache.tzst at every job's end is pure cost
and can stall (a ~4-minute zstdmt hang was observed on awscli-tests).
runner.environment gates the cache to GitHub-hosted runners only.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: gate the remaining setup-go caches too
The first pass only covered go-version-file steps; go-version steps in
nine workflows (including awscli-tests' own file) kept uploading the
cache archive on self-hosted runners.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: cache docker build layers for container_dev
The Go-build image rebuilds its layers on every run; type=gha backend
caching reuses them across runs and platforms.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: pin container build to the pushed sha under layer caching
Dockerfile.go_build clones master inside the image; a cache-from layer
would otherwise keep serving the previous commit's clone and publish
stale code under the dev tag. Passing BRANCH=github.sha makes the clone
layer's key commit-specific while toolchain layers still hit.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: pass BRANCH via build-args so the per-commit cache key reaches buildx
docker/build-push-action has no `args` input — the BRANCH override was
silently dropped, leaving the build arg at its `master` default and the
clone/checkout layers serving a stale commit under the gha cache.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: build test binaries once, share them across shard jobs
Each of the 6 rust-volume-go-tests shards, the rust-integration-tests
job, and the 6 volume-server-integration-tests shards rebuilt the same
weed and weed-volume binaries — 13 duplicate builds per run. A
build-test-binaries / build-weed job now compiles them once under the
existing cargo cache and publishes them as artifacts; the consumers
download and chmod them instead of building.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: preserve binary paths through the artifact round-trip
upload-artifact stores single files flat, so weed/weed would come back
at the workspace root and the consumers' expected paths break. Pack the
binaries in a tar before upload and extract on download.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: extract the shared test binaries in the rust go-test shards
The download step leaves rust-test-binaries.tar at the workspace root;
without untarring it the chmod fails on missing files and every shard
fails before running a test.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: cancel superseded workflow runs on PR pushes
PR-triggered workflows without a concurrency group keep running (and
holding runner slots) after a force-push or quick follow-up commit
supersedes them. Add the same group/cancel-in-progress block the go
workflow already uses, keyed on head_ref, to the remaining PR workflows.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: key concurrency groups on PR number or ref, not head_ref
github.head_ref is empty on tag pushes and manual runs (every run then
shares one group and cancels the last) and collides across forks when
two PRs share a branch name. pull_request.number is unique per PR;
github.ref covers push/tag/dispatch. Applied to all workflows using the
pattern, including pre-existing ones.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: restrict cancel-in-progress to PR and branch-push runs
* ci: serialize container_latest manual runs by destination tag
Two workflow_dispatch runs sharing an image_tag could otherwise build
concurrently: their per-arch images publish to the same platform tags,
so one run's create-manifest job can combine platforms from different
source refs. Group dispatch runs by the destination tag (with
cancel-in-progress already disabled for non-PR/non-branch events) so
same-tag publishes queue instead of interleaving.
* ci: serialize manual publish runs by destination tag in remaining container workflows
* ci: group foundationdb release dispatch runs by release tag
A manual run now shares the tag-push concurrency group for the same
release, so a re-publish of a tag queues behind (or ahead of) the push
run instead of racing it on the same image tag.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: serialize foundationdb_version dispatch runs by destination tag
Two manual runs resolving to the same docker tag now share a concurrency
group (explicit image_tag, else the computed
foundationdb_<fdb>_seaweedfs_<ref> default), so they queue instead of
racing pushes to the same tag.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: drop unsupported toLower() from foundationdb_version concurrency group
* ci: key rocksdb_version dispatch group on its published tag
* ci: queue publish runs via turnstyle instead of cancel-free shared groups
* Revert "ci: queue publish runs via turnstyle instead of cancel-free shared groups"
This reverts commit 039148f8d5.
* ci: queue same-tag publish runs instead of replacing the pending one
Dispatch/tag-only workflows can never cancel in progress, so opt their
concurrency groups into queue: max — up to 100 same-tag publishes wait
in line rather than each new dispatch cancelling the pending one.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The separate master/volume/filer processes in framework.go bound fixed
ports, discarded child output, and never noticed an early process exit,
so a slow or dead volume server burned the whole 30s timeout with no
logs. Switch to the same weed-mini approach the fuse_test package
framework already uses: dynamic filer ports, child output captured to
log files, and readiness waits that bail out the moment the process
dies. The external-mount path (TEST_MOUNT_POINT) is unchanged.
The posix-compliance workflow mounts with -allowOthers, but fusermount
refuses allow_other unless user_allow_other is set in /etc/fuse.conf,
mirroring the fuse-integration workflow FUSE setup.
* s3lifecycle: schedule ExpirationDays from version-file writes
On a versioned bucket a PUT lands in the meta-log as the version file
plus the .versions/ directory update — there is no bare-key event, so
the version-folder skip dropped every version-file event for all kinds
but ExpiredDeleteMarker. ExpirationDays therefore only fired via the
recovery walker's cold-start / rule-change pass: once a shard had a
saved cursor, no daily run ever scheduled another delete marker.
A version-file write is the latest for its logical key at write time,
so route it through the same event-driven evaluation the regular path
runs. The match carries the file's identity with an empty VersionID —
the dispatcher resolves the current latest at due time, and a fresher
PUT meanwhile resolves to STALE_IDENTITY there.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* lifecycle test: versioned Expiration fires on the second daily run
Establishes the shard cursor with a first pass, then PUTs and backdates
a new versioned object and runs again. The delete marker must appear
without a rule change forcing a recovery walk.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run TestLifecycleVersionedBucketExpiresOnSecondRun in the lifecycle matrix
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: add POSIX compliance test suite
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: add posix-compliance workflow
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: skip non-executable weed path candidates
* fuse integration: size readv buffer to the written segment
* fuse integration: reference posix_Makefile explicitly in recursive make calls
GNU make does not propagate -f through MAKEFLAGS, so bare $(MAKE) re-read the default Makefile in this directory.
* fuse integration: fail posix test targets when go test fails
Every suite pipes go test through tee under /bin/sh, so the recipe status was tee's and test failures stayed green. Run recipes under bash -o pipefail and emit the color escapes with printf since bash echo does not expand them.
* fuse integration: run only benchmarks in benchmark-posix
go test -bench=. without -run also executes the whole suite, mixing compliance output into the benchmark log.
* fuse integration: emit single-number test counts in reports
grep -c prints 0 and exits nonzero when nothing matches, so '|| echo 0' produced two lines that broke the expr arithmetic and JSON fields. awk prints exactly one count either way.
* fuse integration: skip the sendfile test on macOS
macOS sendfile(2) only accepts a socket destination; the test copies into a regular file and always gets ENOTSOCK.
* fuse integration: honor TestConfig and own only self-started resources
NewFuseTestFramework dropped its config argument and Cleanup read a fresh
DefaultTestConfig, so SkipCleanup never did anything. Cleanup also skipped
teardown whenever TEST_SKIP_CLUSTER_SETUP was set, even when TEST_MOUNT_POINT
was empty and Setup had started the cluster itself. The framework now keeps
the config it was built with, runs external-mount tests inside a per-run work
dir under the mount, and only skips teardown when an external mount was
actually used.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: pass the debug flag before the subcommand
-v is a weed global flag; appended after master/volume/filer/mount it lands
on the subcommand flagset, which rejects it and the process dies at flag
parsing whenever EnableDebug is on.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: require a real mount before running tests
Setup creates the mount point before weed mount starts, so statting and
listing it succeeded on the plain directory and tests ran against local
disk whenever the mount was slow or died. waitForMount now waits until the
mount point's device id differs from its parent's.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: resolve the weed binary to an absolute path
The started processes run with cmd.Dir at the framework's temp dir, so a
relative candidate like ../weed no longer resolves. Prefer PATH, then probe
the usual build outputs and return absolute paths only.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: build the external test work dir with MkdirTemp
The suite staged cloned tools and generated scripts under a predictable
/tmp path another user could pre-create and swap mid-run.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: use the pjdfstest tree setup-pjdfstest installs
The test gated on a pjdfstest binary in PATH but then proved a suite it
cloned itself, so a tree installed by make setup-pjdfstest was never found.
Look for the built suite under external_tools first, then clone into the
work dir as before.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: require a real mount before posix tests
The wait loop fell back to listing the mount point, which passes on the
directory created before weed mount runs — a dead or slow mount meant the
suite exercised local disk. Only mountpoint counts now, and a dead mount
process fails fast instead of timing out.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: collect failure logs before cleanup deletes them
The always-on cleanup step removed /tmp/seaweedfs before the failure-only
debug step could copy the service logs, so failed runs uploaded nothing.
Run cleanup after log collection.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: point MOUNT_POINT at TEST_MOUNT_POINT
Targets that work against MOUNT_POINT (pjdfstest, nfstest, fio configs)
ignored TEST_MOUNT_POINT and ran against the local-disk default while the
go tests used the external mount.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: skip the permission-denied check as root
Root bypasses mode-bit checks, so writing the read-only file succeeded and
the assertion failed in root-run environments like containers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: align the posix docs with the suite
The doc listed a docker-test-posix target and make variables that do not
exist, and ran go test against single files that cannot compile alone. The
roadmap also still claimed the platform-specific tests this suite now
implements were all skipped.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: honor SkipCleanup in the mini-based framework too
Same gap as the fuse-package framework: Cleanup read a fresh
DefaultTestConfig, so a test's SkipCleanup setting was ignored.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: normalize config in Setup and clean the work dir too
Setup(nil) panicked on config.EnableDebug and external-mount Cleanup left
the temp dir behind; use the stored config for the start* calls and remove
tempDir alongside the external work dir.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: point external posix suites at the workflow FUSE mount
* fuse integration: create the fio target directory before running
* fuse integration: use unix.O_DIRECT instead of a hard-coded x86 value
0x4000 is O_DIRECT only on x86 and amd64; on arm64 it is O_DIRECTORY.
The test module already depends on golang.org/x/sys.
* fuse integration: run autoreconf and configure before building pjdfstest
A fresh pjdfstest clone is autotools-based and has no Makefile, so the
Go-side setup fallback always failed and the suite silently skipped.
* ci: require external posix tools only when setup installed them
setup-external-tools always exits 0 because each sub-target tolerates
failure, so POSIX_REQUIRE_EXTERNAL_TOOLS was set even with no tools
installed and the suite then hard-failed on the missing suites.
* ci: fail the posix workflow when the mount point is not a FUSE mount
The file write/read check passes on the plain directory created before
weed mount, so a dead mount let the whole suite grade the local disk.
* fuse integration: skip when the host has no FUSE device node
Setup failed the test outright on machines without /dev/fuse instead of
skipping; cover every suite entry point by skipping inside Setup.
* fuse integration: drop the chmod 666 /dev/fuse troubleshooting tip
World-writable device access is a security risk; the fuse group
membership on the line below is the safe fix.
* fuse integration: require go 1.21+ in check-go, not 1.20+
* fuse integration: refuse external suites on an unmounted path
* fuse integration: verify TEST_MOUNT_POINT is a real mount
* fuse integration: clean TEST_MOUNT_POINT before the parent-device check
filepath.Dir of a trailing-slash path returns the mount itself, so a mounted TEST_MOUNT_POINT ending in / was refused.
* fuse integration: detect WinFsp mounts on Windows in mountedAt
A directory mount is a reparse point and a drive mount is a volume root; a plain directory is now refused instead of accepted.
* fuse integration: strip a trailing slash off MOUNT_POINT in require-mounted
The mount-table grep fallback on hosts without mountpoint(8) would not match a slash-suffixed entry.
* fuse integration: require an external mount to be a FUSE filesystem
A device-ID boundary alone does not prove FUSE: a tmpfs mount would pass mountedAt and report its results as FUSE compliance. Check the filesystem type (statfs magic on linux, fstypename on the BSDs); platforms without a check keep the mount and write verification.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The HEAD on the completed multipart object occasionally gets a 403 from a
signature verification mismatch before the object read settles; retry the
download instead of failing the run.
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
* helm: keep the S3 config checksum stable across the first upgrade (fixes#11719)
checksum/s3config hashed the rendered s3-secret.yaml. That template
keeps its generated S3 keys with lookup, so rendering it again for the
checksum drew new keys whenever lookup found nothing: on install the
filer, S3 and all-in-one pods each carried a checksum of a different
set of keys than the Secret got, and the first upgrade with unchanged
values corrected it and restarted them.
Hash the values s3-secret.yaml is built from instead: the enable and
auth switches, existingConfigSecret, credentials, the legacy-secret
settings, the audit log configs and the Secret name. Only those, so
scaling the S3 gateway or changing its resources does not restart the
filer. A change to any of them still restarts the pods. Keys that the
chart generated stay out of the hash; they only change if the Secret
is deleted by hand.
The new render test fails on the old template (two checksums for one
Secret, different on every render) and passes now; it also checks that
the checksum follows the credentials and filer.s3.auditLogConfig and
ignores s3.replicas and s3.resources. On kind, an upgrade with unchanged
values no longer replaces the filer pod.
* helm: check the S3 checksum per workload, all-in-one included
Each workload's checksum/s3config is now read on its own (--show-only),
so a workload that loses the annotation fails the test instead of being
hidden by sort -u, and the all-in-one deployment gets the same checks.
Verified against two broken copies: one without the filer annotation,
one with the old all-in-one checksum; the test fails on both.
The helper comment now says what happens when the Secret is deleted to
rotate generated keys.
* ci(helm): fail the S3 checksum test on a missing or unrendered checksum
A failed helm template or a render without the annotation produced an
empty checksum, and the inequality assertions then passed against it.
s3_checksum now returns non-zero unless it found a checksum, and the
changed-value renders are assigned before comparing so failures cannot
hide inside test.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Ege Zambelli <egezamb@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* kms/azure: fix encrypt/decrypt round trip for Azure Key Vault
The provider stored the wrapped data key as string(encryptResult.Result)
in the JSON envelope, sent the encryption context as AAD to an RSA-OAEP
key, and passed a full key URL to the azkeys client in the name
position. Each of those breaks a round trip on its own.
- split the key URL into the (name, version) pair azkeys expects before
Encrypt, Decrypt and GetKey; a plain name still resolves to the latest
version, and decrypt keeps the stored version so objects stay readable
after a key rotation
- store the wrapped key base64-encoded and decode it strictly, so the JSON
envelope no longer replaces the raw bytes with U+FFFD
- stop sending AAD: Key Vault rejects it on RSA-OAEP with BadParameter,
so the encryption context is logged and dropped instead
* kms/azure: reject a key URL that names another vault
splitKeyID dropped the host of a full Key Vault URL, so a key ID from a
different vault silently resolved to this vault's same-named key. Return
an error instead of encrypting under a key the caller did not ask for.
* kms/azure: bind encryption context via an envelope digest
RSA-OAEP rejects AAD, so removing it left the encryption context
unauthenticated: a wrapped key could be decrypted under a different
object's context. Record a sha256 digest of the marshaled context in
the envelope's provider_specific field on encrypt and verify it before
calling Decrypt, restoring the binding without AAD.
* kms/azure: treat an explicit :443 port as the same vault
* kms/azure: accept an absent context digest only for empty contexts
* kms/azure: normalize both hosts when comparing key URLs to the vault
splitKeyID stripped :443 and a trailing dot from the configured vault but
only :443 from the key URL, so a key URL naming the same vault with a
trailing DNS dot was rejected before Azure was ever called. Compare both
sides through vaultHost so they are normalized identically.
* kms/azure: reject non-key vault URLs in splitKeyID, document digest limits
A Key Vault URL that does not name a key under /keys/, has an empty key
name, or carries extra path segments now fails fast instead of being
passed to the client as a key name, where it would surface as a
confusing vault-side error.
Also note that the envelope context digest is a client-side mismatch
check, not vault-authenticated AAD, and only allocate providerSpecific
when a context is present.
* ci: compile and test azurekms-gated code
weed/kms/azure is excluded from every default build, so nothing in CI
compiled it; that is how the provider shipped unregistered. Build the
tree and run the kms tests with -tags azurekms on every Go change.
---------
Co-authored-by: Yi-111-a <>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3api: reject SSE headers PutObject cannot honor, as CopyObject does
PutObject and CreateMultipartUpload did not check the server-side
encryption headers they were given. An x-amz-server-side-encryption
value that names no method, such as "aes:kms", matched none of the SSE
paths, so the object was stored unencrypted and the request answered
200. SSE-C together with x-amz-server-side-encryption was also accepted,
and one of the two silently won.
CopyObject already rejects both through validateEncryptionCompatibility.
Run the same check, with the same error codes, before PutObject and
CreateMultipartUpload store anything.
* s3api: answer rejected SSE headers with InvalidArgument, as S3 does
S3 rejects an unknown x-amz-server-side-encryption value and SSE-C
combined with another method with InvalidArgument. PutObject and
CreateMultipartUpload now return that code, with S3's messages, through
two new error codes; CopyObject keeps its own.
Also run ceph/s3-tests' test_put_obj_enc_conflict_c_s3, _c_kms and
_bad_enc_kms in CI, which check both handlers' responses end to end.
* s3api: close the remaining ways a PUT could skip requested encryption
- A repeated x-amz-server-side-encryption header is rejected: the
encryption paths apply only the first value, so extra values could
hide the method the client asked for.
- KMS options (key id, encryption context, bucket key) are rejected
unless the method is aws:kms, matching the S3 InvalidArgument error.
- CreateMultipartUpload validates before auto-create, so a refused
upload cannot leave a bucket behind.
- Directory markers encrypt their inline content through the shared
SSE path and record the same entry metadata as regular objects,
instead of storing requested-encrypted bytes in plaintext.
* s3api: read back what the SSE marker write stores
- Repeated SSE-C and KMS option headers are rejected alongside a
repeated x-amz-server-side-encryption, closing the same first-value
bypass for customer-key and KMS fields.
- Algorithm validity is checked before KMS options so an unsupported
value keeps the "not supported" error.
- serveDirectoryContent decrypts marker content with the stored SSE
metadata and returns the SSE headers, so an encrypted marker reads
back what was written; its Content-Length now reflects the bytes
actually served.
* s3api: HEAD of a marker skips decryption and keeps the stored size
HEAD returns no body, so it now runs only the SSE-C key check instead of
decrypting — matching HeadObjectHandler and avoiding a KMS round-trip —
and chunk-backed directory entries report Attributes.FileSize again
rather than the length of their (empty) inline content.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: keep GET Content-Length to the bytes serveDirectoryContent writes
The FileSize override described chunk-backed markers on HEAD, but GET
sends only the inline content, so it promised bytes it never wrote.
* s3api: stream a chunk-backed directory on GET like any object
A directory promoted over an uploaded object keeps the object chunks
with empty inline content, so serving only entry.Content made GET
deliver nothing while HEAD reported FileSize. GET now routes those
entries through the regular volume-server stream, keeping the two
methods consistent.
* s3api: answer a busy volume read with RequestBytesExceed
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* helm: add Gateway API HTTPRoute for master, volume, filer, s3 and admin
Each component that offers an Ingress can now be exposed through a
Gateway API HTTPRoute instead, via <component>.httpRoute. It is disabled
by default, so existing renders are unchanged.
parentRefs, hostnames, labels and annotations pass through as given.
Each rule may set matches, filters, timeouts and backendRefs, and a rule
without backendRefs routes to the component's own service and port,
following all-in-one mode the same way the Ingress templates do. An
empty rules list yields a single rule sending all traffic there.
Signed-off-by: younsl <cysl@kakao.com>
* helm: fix HTTPRoute backend selection for S3 on filer and all-in-one
- s3: with S3 on the filer the route targets filer.s3.port, the port the
s3 Service exposes. The all-in-one Service is chosen only when
allInOne.s3 is enabled, so a standalone S3 next to all-in-one routes
to the s3 Service.
- master: render in all-in-one mode and route to the all-in-one Service,
like the volume and filer routes.
- Name routes with seaweedfs.componentName, keeping them within 63
characters like the Services they point to.
- CI: cover the filer S3 port and the all-in-one master route.
Signed-off-by: younsl <cysl@kakao.com>
* helm: document the Gateway API HTTPRoute values in the chart README
Signed-off-by: younsl <cysl@kakao.com>
* helm: link the Gateway API docs from the chart README
Signed-off-by: younsl <cysl@kakao.com>
* helm: scope the filer S3 port note in the chart README
Signed-off-by: younsl <cysl@kakao.com>
---------
Signed-off-by: younsl <cysl@kakao.com>
* deps: pin github.com/seaweedfs/goexif back to v1.0.3
The fork's newest tagged release is v1.0.3; the v2.0.0+incompatible
requirement resolved to older, untagged code and breaks isolated
builds that fetch it from the proxy.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: stop dependabot bumping seaweedfs/goexif past its latest tag
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/kafka: settle goexif at v1.0.3 too
The kafka test module recorded v2.0.0+incompatible in its own requires,
so MVS kept selecting it over the pinned v1.0.3 in the root module.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* helm: drop fromToml dependency in security-configmap.yaml (fixes#11611)
fromToml was added to Helm in v3.17.0 (helm/helm#12026, merged
2024-09-12, one day after v3.16.0 was cut). This chart declares no
minimum Helm version (no Chart.yaml kubeVersion, nothing in the
README), and the call in security-configmap.yaml:21 is an
unconditional *parse*-time failure on Helm < v3.17.0 - Go's
text/template parses a file's entire body before evaluating any
{{if}}, so this breaks the chart (any topology, any values) even when
securityConfigEnabled is false and the ConfigMap would render nothing.
Replaces the fromToml-based dig lookup with a small regex-based helper
(seaweedfs.existingTomlKey) that reads the same four "key = ..." JWT
signing-key values out of a previously-rendered security.toml,
preserving the existing fallback-to-random behavior exactly.
Verified:
- helm lint (v3.16.3 and v4.3.0): clean
- helm template with chart defaults: byte-identical output to the
unpatched chart rendered via Helm v4 (which has fromToml) - the
disabled/default path is untouched
- helm template with security enabled, no prior ConfigMap: identical
structure to the unpatched chart (helm v4), modulo the expected
random key
- Real helm install + helm upgrade round trip (live lookup, since
"helm template" never evaluates lookup, even under the original
fromToml code): the JWT signing key is identical across both
releases - confirms key persistence across upgrades is preserved,
not just "renders without erroring"
- helm template with chart defaults, Helm v3.16.3: previously failed
with a parse error naming fromToml as undefined; now renders
successfully
Fixes#11611.
* helm: harden existingTomlKey against commented key lines and CRLF
Addresses two review findings from greptile-apps on PR #11614:
- The key-line regex matched the first "key = ..." anywhere in the
section block, including a commented-out "# key = ..." line, which
would shadow a real active key on a hand-edited or otherwise
non-chart-generated ConfigMap. Anchored to line start with the
Go regexp multiline flag ((?m)^key...), which a line starting with
"#" cannot match.
- The section-header match required an exact "]\n", so a ConfigMap
with CRLF line endings would fail to match the block at all and
regenerate the key instead of reusing it. Changed to "]\r?\n".
Also adds a CI test ("Verify JWT signing key persistence across
upgrades") exercising all of this end to end with a real
helm install -> edit the live ConfigMap -> helm upgrade cycle,
matching the existing "Verify SFTP host key secret lifecycle" test's
shape: both edge cases are reproduced against a real ConfigMap and
asserted on the post-upgrade rendered security.toml.
Verified locally (same commands as the new CI step) against a real
cluster before pushing.
* helm: preserve JWT keys across supported TOML layouts
* ci: use setup-python interpreter for JWT upgrade checks
* helm: preserve keys under quoted TOML section headers
* helm: ignore unrelated quoted TOML section headers
The standalone s3 Deployment hardcoded terminationGracePeriodSeconds and
offered no way to set a container lifecycle or the Deployment strategy, so
operators could not add a preStop delay to drain endpoints before SIGTERM
or hold maxUnavailable at 0 during rollouts.
Add s3.updateStrategy, s3.lifecycle and s3.terminationGracePeriodSeconds.
Defaults render the same manifest as before.
* fix(s3): initialize destination ACLs for CopyObject
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
* fix(s3): re-check routed self-copy eligibility on the locked read
routeInPlace was decided on the pre-lock entry, but the PATCH body
re-reads the entry. A concurrent write changing file mode or MIME in
between left the routed PATCH installing new ACL keys while Attributes
kept the stale mode. Evaluate eligibility against the re-read entry and
retry the self-copy under the distributed lock when it no longer
qualifies.
* fix(s3): guard metadata self-copies against concurrent writes
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
* ci: raise s3api unit-test timeout to 9m
The suite crossed the 5m binary timeout on the hosted runner (local run
is ~4.3m and still growing). The job-level limit is already 10m.
* ci: allow setup time before the S3 API test suite
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
---------
Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
GetObjectAttributes accepted the Checksum attribute but never filled it
in, as its comment said SeaweedFS did not store S3 checksums. PutObject
and CompleteMultipartUpload store them now, and HeadObject returns them.
Fill in Checksum from the same entry fields, with the ChecksumType and
ChecksumCRC64NVME members the response did not have.
Also run ceph/s3-tests' test_get_checksum_object_attributes in CI.
* s3api: do not store aws-chunked in an object's Content-Encoding
aws-chunked in Content-Encoding names the SigV4 streaming framing of
the request body, which the gateway decodes on upload. PutObject and
CreateMultipartUpload stored the header as sent, so an object uploaded
with "gzip, aws-chunked" was served with that Content-Encoding, and one
uploaded with "aws-chunked" alone was served as aws-chunked. S3 drops
aws-chunked and keeps the other encodings.
Also run ceph/s3-tests' test_object_content_encoding_aws_chunked in CI.
* s3api: read every Content-Encoding field, and drop aws-chunked on copy
A client can send aws-chunked and the object's own encoding as separate
Content-Encoding fields. Only the first was read, so "aws-chunked"
followed by "gzip" left the object without its gzip. Combine all the
fields before dropping aws-chunked. CopyObject with the REPLACE
directive stored the requested Content-Encoding as sent: drop
aws-chunked there too.
Add ppc64le and s390x to the normal and large_disk release image
matrices, next to arm/v7 and 386, and to the manual container_latest
fallback so that rebuilding `latest` does not drop them from the
manifest.
The Go binary is cross-compiled without cgo on the build host, and
Dockerfile.go_build already falls back to empty Rust volume server and
worker placeholders on platforms without pre-built Rust binaries. Only
the small final-stage RUN steps run emulated, so these two platforms use
QEMU on ubuntu-latest like arm/v7.
Refs #5558
OLake (github.com/datazip-inc/olake) is a CDC and ingestion engine that writes
Apache Iceberg. It covers two paths none of the existing catalog suites reach.
It is a strict Java Iceberg client. OLake does not write Iceberg from Go — its
Go process spawns a Java sidecar over gRPC and writes through the official
Apache Iceberg library, because the Go library has no equality deletes and CDC
needs them. That makes it the client class weed/s3api/iceberg's metadata
compliance backfill exists to serve: the one that fails with "Cannot parse
missing long current-snapshot-id" when spec-required keys are omitted.
And it produces equality deletes. Its upsert path commits operation=overwrite
with an equality-delete file and a delete manifest. ClickHouse, Doris, Trino,
Spark and DuckDB all only append, so nothing else in this directory exercises
a delete manifest at all.
Six subtests: the destination check reaches SUCCEEDED and actually loads the
REST catalog; discover enumerates the source; a full sync commits a snapshot;
PyIceberg reads back what the Java writer committed; an update plus a re-sync
records an overwrite carrying equality deletes with a delete manifest in the
current snapshot; and the catalog does not rewrite manifests the official Java
writer produced, which gives the manifest-repair path a negative test to go
with the ClickHouse positive one.
What it deliberately does not assert is a delete-applied read. PyIceberg
refuses to scan a table carrying equality deletes (apache/iceberg#6568) while
reading its metadata fine, and an engine that can apply them costs a
multi-gigabyte image. Recording the commit correctly is the catalog's
contract; applying deletes on read is the engine's. The README says so, says
the read half was verified by hand once with StarRocks, and warns against
later "upgrading" this to a PyIceberg rows read — which would either fail or,
if PyIceberg ever starts skipping deletes instead of raising, pass by not
looking.
The workflow job asserts the suite actually ran rather than trusting a green
exit, for the same reason: at least one top-level PASS and zero SKIP.
No product change — the destination config is the generic catalog_type=rest
with the standard OAuth2 client-credentials flow, and s3_path_style is not even
set, since OLake turns it on itself whenever s3_endpoint is non-empty.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
