mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-27 07:20:44 +01:00
test: boto3 s3vectors conformance harness (P1) (#12105)
* vector: boto3 s3vectors conformance harness test/vectors spins a mini cluster, an s3 gateway with -port.vectors and -vectors.grpc, and a weed-vector-server, then exercises the AWS client end to end: bucket and index lifecycle, put/get/list/delete vectors, and a filtered query asserting distances and metadata. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: run the s3vectors conformance subset An e2e job in rust-vector-tests: builds weed and weed-vector-server, then runs test/vectors against a mini cluster plus the vectors gateway. The workflow now also watches vector.proto, its stubs, and test/vectors. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: fix e2e protoc and codespell findings The e2e job needed the same vendored-protoc step as the unit-test job — lance's build scripts locate protoc themselves. Codespell flags the AWS `$nin` operator literal and the `lits` variable: exempt `nin` in .codespellrc like the existing word exemptions, rename the variable, and fold the negation into the IN/NOT IN symbol directly. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: classify catalog errors by code, not message text * s3: drop the implicit vector namespace on vector-bucket delete * test: point the vector engine at the gateway lance port * test: bind the vectors gateway listeners to loopback * test: make the filter and distance assertions load-bearing * test: rebuild the harness binaries on every run * ci: fix the vectors end-to-end job * ci: run the vector tests on gateway and command changes --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
c55b7c5a29
commit
a9d6bfc51f
5 files changed
+242
-8
No files matched your search
@@ -8,6 +8,11 @@ on:
|
||||
- 'seaweed-worker/**'
|
||||
- 'seaweed-common/**'
|
||||
- 'weed/pb/plugin.proto'
|
||||
- 'weed/pb/vector.proto'
|
||||
- 'weed/pb/vector_pb/**'
|
||||
- 'weed/s3api/**'
|
||||
- 'weed/command/**'
|
||||
- 'test/vectors/**'
|
||||
- '.github/workflows/rust-vector-tests.yml'
|
||||
push:
|
||||
branches: [ master, main ]
|
||||
@@ -16,6 +21,11 @@ on:
|
||||
- 'seaweed-worker/**'
|
||||
- 'seaweed-common/**'
|
||||
- 'weed/pb/plugin.proto'
|
||||
- 'weed/pb/vector.proto'
|
||||
- 'weed/pb/vector_pb/**'
|
||||
- 'weed/s3api/**'
|
||||
- 'weed/command/**'
|
||||
- 'test/vectors/**'
|
||||
- '.github/workflows/rust-vector-tests.yml'
|
||||
|
||||
concurrency:
|
||||
@@ -74,3 +84,41 @@ jobs:
|
||||
|
||||
- name: Run unit tests
|
||||
run: cd seaweed-vector && cargo test --release --workspace
|
||||
|
||||
|
||||
vectors-e2e:
|
||||
name: S3 Vectors End-to-End
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
- uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version-file: go.mod
|
||||
- name: Build weed
|
||||
run: go build -o test/vectors/weed ./weed
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
# Same as the unit-test job: lance's build scripts need a protoc; use
|
||||
# the vendored one so no system package is required.
|
||||
- name: Use the vendored protoc
|
||||
run: |
|
||||
cd seaweed-vector
|
||||
cargo fetch
|
||||
version=$(awk '/^name = "protoc-bin-vendored-linux-x86_64"$/{found=1; next} found && /^version = /{gsub(/"/,"",$3); print $3; exit}' Cargo.lock)
|
||||
test -n "$version" || { echo "protoc-bin-vendored-linux-x86_64 is not in Cargo.lock" >&2; exit 1; }
|
||||
protoc=$(find ~/.cargo/registry/src -path "*protoc-bin-vendored-linux-x86_64-$version/bin/protoc" | head -1)
|
||||
test -x "$protoc" || { echo "no vendored protoc $version in the registry" >&2; exit 1; }
|
||||
echo "PROTOC=$protoc" >> "$GITHUB_ENV"
|
||||
- name: Build weed-vector-server
|
||||
run: cargo build --release --manifest-path seaweed-vector/Cargo.toml -p weed-vector-server
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- name: Install test deps
|
||||
run: pip install boto3 pytest
|
||||
- name: Run the s3vectors conformance subset
|
||||
working-directory: test/vectors
|
||||
run: |
|
||||
cp ../../seaweed-vector/target/release/weed-vector-server weed-vector-server
|
||||
./run_test.sh
|
||||
Reference in new issue
Block a user