vector: integrate engine health, admin management, and local deployment (#12517)

* vector: expose request metrics and registration-aware health endpoints

* vector: report immutable read cache and storage fetch metrics

* docs: describe vector engine health, metrics, and maintenance

* vector: count canceled requests when their handler is dropped

* vector: account for uncached large immutable storage reads

* vector: reliably release the stalled credential reader in tests

* vector: observe large reads as their streamed body is consumed

* vector: bound heartbeats on established master registration streams

* test: exercise hardened vector maintenance on Go and Rust volumes

* vector: register engines in their configured filer group

* vector: make retried index creation preserve compatible datasets

* vector: allow registration to advertise a separate address

* test: release stalled credential readers without blocking FIFO cleanup

* test: allow hardened vector coverage in a named filer group

* vector: discover engines in the same filer group as their gateway

* vector: fall back when the advertised address is empty

* admin: expose vector readiness, maintenance, and management tools (#12520)

* s3vectors: share vector discovery with trusted management clients

* admin: show vector engine readiness and recent maintenance activity

* admin: suggest S3 Vectors IAM actions from the supported operations

* admin: manage vector indexes, policies, tags, encryption, and query data

* admin: validate SSE-S3 support before creating encrypted vector buckets

* admin: discover vector engines in their registration group

* admin: probe vector engines concurrently with independent time budgets

* admin: verify encryption capability on the remote S3 gateway

* admin: defer discovery until after the HTTP server can start

* admin: distinguish unknown vector health services from failed engines

* admin: count workers that can execute vector index maintenance

* admin: allow read-only users to query vectors and inspect tags

* vector: scope gateway and admin discovery to the configured filer group

* admin: refresh policy state after mutations and serialize policy controls

* vector: drain active RPCs before closing replaced discovery connections

* vector: respect request cancellation while waiting for discovery

* vector: prefer another registered engine after a transport failure

* admin: check all masters before reporting no registered vector engines

* iam: honor attached vector policies without broadening legacy grants

* vector: allow short-lived discovery clients to close their connections

* vector: refresh cached master addresses for each discovery lookup

* iam: enforce vector bucket tag conditions in attached policies

* iam: authorize vector bucket creation with the requested tags

* test: exercise vector IAM grants and tag denies on Go and Rust volumes

* admin: reject authenticated vector mutations without an admin role

* test: run vector IAM assertions before the shared engine is stopped

* vector: add native mini startup, shell tools, and a runnable Compose example (#12522)

* s3vectors: share vector discovery with trusted management clients

* admin: show vector engine readiness and recent maintenance activity

* admin: suggest S3 Vectors IAM actions from the supported operations

* admin: manage vector indexes, policies, tags, encryption, and query data

* admin: validate SSE-S3 support before creating encrypted vector buckets

* admin: discover vector engines in their registration group

* admin: probe vector engines concurrently with independent time budgets

* admin: verify encryption capability on the remote S3 gateway

* admin: defer discovery until after the HTTP server can start

* admin: distinguish unknown vector health services from failed engines

* admin: count workers that can execute vector index maintenance

* admin: allow read-only users to query vectors and inspect tags

* vector: scope gateway and admin discovery to the configured filer group

* admin: refresh policy state after mutations and serialize policy controls

* vector: drain active RPCs before closing replaced discovery connections

* vector: respect request cancellation while waiting for discovery

* vector: prefer another registered engine after a transport failure

* admin: check all masters before reporting no registered vector engines

* iam: honor attached vector policies without broadening legacy grants

* vector: allow short-lived discovery clients to close their connections

* vector: refresh cached master addresses for each discovery lookup

* iam: enforce vector bucket tag conditions in attached policies

* iam: authorize vector bucket creation with the requested tags

* test: exercise vector IAM grants and tag denies on Go and Rust volumes

* admin: reject authenticated vector mutations without an admin role

* test: run vector IAM assertions before the shared engine is stopped

* mini: optionally start and supervise the vector engine

* mini: bootstrap vector buckets with existing Lance catalog helpers

* shell: manage vector buckets, indexes, and vector data

* docs: provide a runnable vector and maintenance worker Compose stack

* shell: scope vector discovery to the session and close its connections

* mini: honor the bind address for vector and metrics listeners

* mini: reserve vector ports before assigning gRPC and compatibility listeners

* mini: wait for vector storage setup and master registration

* mini: create requested vector buckets without requiring the Lance HTTP listener

* shell: refresh master addresses during vector discovery

* mini: resolve sibling vector executables with platform extensions

* vector: preserve uncertain creates and fix mini listener allocation (#12525)

* vector: retain catalog records when index creation has an unknown outcome

* mini: keep chosen listener ports reserved throughout allocation

* mini: resolve hostname binds before starting vector listeners

* vector: fix integration test imports and recover interrupted index creation (#12533)

* shell: register vector operations from the command package

* s3vectors: resume interrupted index initialization safely

* mini: exclude disabled WebDAV from listener allocation

* s3vectors: retain uncertain creates across refused retries

* s3vectors: claim pending indexes before retrying initialization

* s3vectors: compare resolved distance metrics during recovery

* s3vectors: preserve metadata keys when retrying initialization

* s3vectors: verify ownership before initializing new catalog records

docker/s3vectors/README.md

@@ -0,0 +1,39 @@
+# Vector buckets locally
+
+Run on Linux containers (amd64 or arm64):
+
+```sh
+docker compose up -d
+export AWS_ACCESS_KEY_ID=vector-demo
+export AWS_SECRET_ACCESS_KEY=vector-demo-secret
+export AWS_DEFAULT_REGION=us-east-1
+aws --endpoint-url http://localhost:8333 s3vectors create-index \
+  --vector-bucket-name embeddings --index-name images \
+  --data-type float32 --dimension 3 --distance-metric cosine
+aws --endpoint-url http://localhost:8333 s3vectors put-vectors \
+  --vector-bucket-name embeddings --index-name images \
+  --vectors '[{"key":"first","data":{"float32":[1,0,0]}}]'
+aws --endpoint-url http://localhost:8333 s3vectors query-vectors \
+  --vector-bucket-name embeddings --index-name images \
+  --query-vector '{"float32":[1,0,0]}' --top-k 1
+```
+
+The engine registers with the master. Wait for that registration before creating
+an index. The Lance worker builds and refreshes ANN indexes as data grows;
+queries can scan data before an ANN index is built. The demo's admin UI is at
+http://localhost:12535 (user `admin`, password `vector-demo`). Demo credentials
+and plaintext cluster connections are for this local example. Production
+settings are in the Helm/operator TLS and authentication documentation.
+
+`docker compose down` preserves the named data volume. Add `-v` to discard it.
+Override `SEAWEEDFS_IMAGE` to test a newer image. With metrics-enabled engines,
+append `--metrics-port=9329 --metrics-ip=0.0.0.0` to the vector command and scrape
+`vector:9329/metrics` inside the Compose network.
+
+A native development setup can use `weed mini -vector -vectorBucket=embeddings`
+with `weed-vector` beside `weed` or on PATH. `-vector.binary` selects an explicit
+binary, and `-vector.metricsPort=9329` enables HTTP readiness and metrics on a
+compatible engine. The child inherits the cluster's configuration directory and
+stops with mini. Run `weed-worker` separately for ANN maintenance. `weed shell`
+provides `s3vectors -operation <Operation> -file request.json` for the same API
+operations using trusted cluster credentials.

docker/s3vectors/compose.yaml

@@ -0,0 +1,34 @@
+services:
+  seaweedfs:
+    image: ${SEAWEEDFS_IMAGE:-chrislusf/seaweedfs:4.53}
+    command: mini -dir=/data -ip=seaweedfs -ip.bind=0.0.0.0 -admin.password=vector-demo -admin.worker.ip=0.0.0.0 -tableBucket=embeddings:LANCE
+    environment: &environment
+      WEED_JWT_FILER_SIGNING_KEY: vector-demo-cluster-key
+      AWS_ACCESS_KEY_ID: vector-demo
+      AWS_SECRET_ACCESS_KEY: vector-demo-secret
+    ports:
+      - "127.0.0.1:8333:8333"
+      - "127.0.0.1:12535:12535"
+    volumes:
+      - data:/data
+    healthcheck:
+      test: [CMD-SHELL, "wget -q -O /dev/null http://127.0.0.1:9333/cluster/status"]
+      interval: 5s
+      timeout: 3s
+      retries: 30
+  vector:
+    image: ${SEAWEEDFS_IMAGE:-chrislusf/seaweedfs:4.53}
+    command: vector --listen=vector:8090 --masters=seaweedfs:9333
+    environment: *environment
+    depends_on:
+      seaweedfs:
+        condition: service_healthy
+  worker:
+    image: ${SEAWEEDFS_IMAGE:-chrislusf/seaweedfs:4.53}
+    command: worker-rust --admin=seaweedfs:12535 --masters=seaweedfs:9333 --namespace=http://seaweedfs:9101
+    environment: *environment
+    depends_on:
+      seaweedfs:
+        condition: service_healthy
+volumes:
+  data: