test: add s3 integration tests for SQLite Cloud Backed Storage (#12243)

* test: add s3 integration tests for SQLite Cloud Backed Storage (#12235)

SQLite CBS stores a database as named objects in a bucket and relies on
conditional PUT — put-if-absent for new blocks and ETag compare-and-swap
on the manifest — so concurrent clients cannot corrupt the database.

The suite runs a weed mini cluster (conditions are evaluated under the
object's write lock on the owner filer via the routed-write path) and
covers:

- the CBS storage primitives: create/put/fetch/list/delete/destroy
- If-None-Match: * put-if-absent, success and 412 rejection
- If-Match ETag compare-and-swap, including stale-ETag rejection and
  the re-read/retry loop CBS clients run on 412, on plain and
  versioned buckets
- racing put-if-absent and manifest CAS writers: exactly one commit
  wins, every update survives
- conditional CompleteMultipartUpload, sequential and raced
- an end-to-end bcv-style upload, download-and-verify, and checkpoint

A matching job is added to the S3 mutation regression workflow.

* test/s3: drop issue references from the sqlite cbs suite comments

* test/s3: abort every losing upload after the conditional multipart race

* test/s3: assert the winning writer's body after the conditional write races

* test/s3: spell out the get-returned etag in the cbs suite

* test/s3: assert the completed multipart ETag the cbs suite feeds back

The conditional-completion checks pass completeOut.ETag into If-Match;
if the response carried no ETag the check would silently run
unconditional.

* test/s3: drop the unused generation parameter from the cbs block body
This commit is contained in:
Chris Lu authored and GitHub committed 2026-10-30 03:17:16 -07:00
1 parent 12dac68d66
commit e80430db8e
3 files changed
+1205

No files matched your search

@@ -6,6 +6,7 @@ on:
- 'weed/s3api/**'
- 'test/s3/delete/**'
- 'test/s3/distributed_lock/**'
- 'test/s3/sqlite_cbs/**'
- 'test/s3/versioning/**'
- 'test/volume_server/framework/**'
- 'docker/compose/s3.json'
@@ -149,3 +150,41 @@ jobs:
name: s3-distributed-lock-regression-logs
path: test/s3/distributed_lock/tmp/seaweedfs_s3_distributed_lock_*
retention-days: 3
s3-sqlite-cbs-regressions:
name: S3 SQLite CBS Regression Tests
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS
run: |
go build -o weed/weed -buildvcs=false ./weed
- name: Run sqlite cbs regressions
timeout-minutes: 15
env:
TMPDIR: ${{ github.workspace }}/test/s3/sqlite_cbs/tmp
S3_SQLITE_CBS_KEEP_LOGS: "1"
WEED_BINARY: ${{ github.workspace }}/weed/weed
run: |
set -x
mkdir -p "$TMPDIR"
go test -v -count=1 -timeout=12m ./test/s3/sqlite_cbs
- name: Upload sqlite cbs logs on failure
if: failure()
uses: actions/upload-artifact@v7
with:
name: s3-sqlite-cbs-regression-logs
path: test/s3/sqlite_cbs/tmp/seaweedfs_sqlite_cbs_*
retention-days: 3