mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-29 08:20:42 +01:00
Chris Lu
190ab32cfb
* vector: weed-vector registers with the weed master weed-vector-server --masters holds a KeepConnected stream to the weed master, the same mechanism filer and s3 use via wdclient — the server shows up in ListClusterNodes(client_type=vector). Resends registration and rediscovers on failure. * vector: s3 discovers weed-vector through the master With -vectors.grpc unset the vectors gateway resolves one registered weed-vector via ListClusterNodes on the masters it already derives from filer config — discovery replaces the static dial for the single-worker OSS path. The conformance harness registers the engine to mini's master and drops -vectors.grpc. * vector: default-fill registration fields ..Default::default() keeps the source compiling against proto variants that carry extra fields, so the same file works in either tree. * vector: serve the Vectors API on the main s3 port The vectors router sits on the main -port listener behind a dispatch that checks the SigV4 credential scope for /s3vectors/ — no separate port needed. -port.vectors stays for explicit dedicated-port deployments; the conformance harness now hits vectors through :8334. * vector: drop -port.vectors entirely It was new — no compat to preserve. The main -port listener serves the API whenever a weed-vector backend is reachable. * vector: drop -vectors.grpc; discovery is the only locator weed-vector always registers to master, the gateway always resolves it through ListClusterNodes — a static address just forks membership. * vector: binary is weed-vector Matches the enterprise binary; package and lib names are unchanged. * vector: review fixes — cluster type, grpc port, lazy resolve, leader hint, advertise - cluster: register 'vector' as a node type so ListClusterNodes returns weed-vector members - s3: dial masters via ToGrpcAddress (HTTP port + 10000) - s3: backend resolves lazily through vectorDiscovery — workers joining after gateway startup work on first query, and address changes re-dial - master_client: convert master HTTP addr to grpc +10000, honor http(s) scheme, follow the leader hint in KeepConnectedResponse, and --advertise covers wildcard/unreachable listen addresses * vector: no --advertise flag; derive the outbound address A wildcard listen resolves to the source IP the master sees via a route lookup, so registration always advertises something reachable without another flag. * vector: review fixes — plaintext dispatch, conn leaks, discovery backoff The two plaintext http servers now serve the vectors dispatch too — signed s3vectors calls on the main port reach the right router. discoverVectorWorker closes each master conn and bounds lookups to 5s; a dead or absent worker no longer re-hits master on every RPC — the resolver backs off for 5s between rediscovery attempts. * vector: 1 GiB grpc cap, dead-worker invalidation, CLI rename Valid PutVectors batches exceed the tonic 4 MiB default — match the gateway's 1 GiB cap. A replaced worker no longer sticks: any RPC error resets the cached address and the next call rediscovers through the master. The core CLI bin renames to weed-vector-cli so the two crates do not collide on the same output path. * vector: retry-only-transport reset, periodic re-resolve, cli rename - reset() takes the failed client and runs only on transport errors — an InvalidArgument on a healthy conn no longer interrupts unrelated requests, and a concurrent re-resolve wins the race. - cli() re-resolves once the interval passes even with a live client, so a worker that moves addresses is picked up. - verify.rs spike docs use weed-vector-cli after the bin rename. * vector: only Unavailable counts as a transport failure A request deadline or an application error can fire on a healthy connection — resetting the shared conn on those churned reconnects.
82 lines
2.8 KiB
Python
82 lines
2.8 KiB
Python
"""Boto3 conformance subset for the S3 Vectors API gateway.
|
|
|
|
Run via `make test` — a mini cluster, an s3 gateway serving the vectors API on its main port, and a
|
|
weed-vector are up by then.
|
|
"""
|
|
|
|
import boto3
|
|
import pytest
|
|
|
|
ENDPOINT = "http://127.0.0.1:8334"
|
|
BUCKET = "vectors-bucket"
|
|
INDEX = "idx"
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def vectors():
|
|
return boto3.client(
|
|
"s3vectors",
|
|
endpoint_url=ENDPOINT,
|
|
aws_access_key_id="test",
|
|
aws_secret_access_key="test",
|
|
region_name="us-east-1",
|
|
)
|
|
|
|
|
|
def test_vector_lifecycle(vectors):
|
|
vectors.create_vector_bucket(vectorBucketName=BUCKET)
|
|
buckets = [b["vectorBucketName"] for b in vectors.list_vector_buckets()["vectorBuckets"]]
|
|
assert BUCKET in buckets
|
|
|
|
vectors.create_index(
|
|
vectorBucketName=BUCKET,
|
|
indexName=INDEX,
|
|
dataType="float32",
|
|
dimension=4,
|
|
distanceMetric="euclidean",
|
|
)
|
|
idx = vectors.get_index(vectorBucketName=BUCKET, indexName=INDEX)["index"]
|
|
assert idx["dimension"] == 4
|
|
|
|
vectors.put_vectors(
|
|
vectorBucketName=BUCKET,
|
|
indexName=INDEX,
|
|
vectors=[
|
|
{"key": "a", "data": {"float32": [1, 0, 0, 0]}, "metadata": {"color": "red"}},
|
|
{"key": "b", "data": {"float32": [0.95, 0.05, 0, 0]}, "metadata": {"color": "blue"}},
|
|
{"key": "c", "data": {"float32": [0.9, 0.1, 0, 0]}, "metadata": {"color": "red"}},
|
|
],
|
|
)
|
|
|
|
got = vectors.get_vectors(
|
|
vectorBucketName=BUCKET, indexName=INDEX, keys=["a"], returnData=True, returnMetadata=True
|
|
)["vectors"]
|
|
assert got[0]["key"] == "a" and got[0]["metadata"]["color"] == "red"
|
|
|
|
listed = vectors.list_vectors(vectorBucketName=BUCKET, indexName=INDEX)["vectors"]
|
|
assert sorted(v["key"] for v in listed) == ["a", "b", "c"]
|
|
|
|
hits = vectors.query_vectors(
|
|
vectorBucketName=BUCKET,
|
|
indexName=INDEX,
|
|
queryVector={"float32": [1, 0, 0, 0]},
|
|
topK=2,
|
|
returnDistance=True,
|
|
returnMetadata=True,
|
|
filter={"color": {"$eq": "red"}},
|
|
)["vectors"]
|
|
# b is the second-nearest vector but blue, so an ignored filter would
|
|
# return ["a", "b"] here.
|
|
assert [h["key"] for h in hits] == ["a", "c"]
|
|
distances = {h["key"]: h["distance"] for h in hits}
|
|
assert distances["a"] == pytest.approx(0.0, abs=1e-6)
|
|
assert distances["c"] == pytest.approx(0.02, abs=1e-4)
|
|
assert all(h["metadata"]["color"] == "red" for h in hits)
|
|
|
|
vectors.delete_vectors(vectorBucketName=BUCKET, indexName=INDEX, keys=["b"])
|
|
listed = vectors.list_vectors(vectorBucketName=BUCKET, indexName=INDEX)["vectors"]
|
|
assert sorted(v["key"] for v in listed) == ["a", "c"]
|
|
|
|
vectors.delete_index(vectorBucketName=BUCKET, indexName=INDEX)
|
|
vectors.delete_vector_bucket(vectorBucketName=BUCKET)
|