S3 Vectors: the gateway resolves the table it sends to weed-vector (#12468)

* lance catalog: in-process DescribeTable for trusted callers DescribeTable and DescribeTableInternal resolve a table's location and storage options without the HTTP layer; the internal form runs the lookup through a trusted manager so the caller's catalog grants do not gate a resolution its own permission already authorized. Same code as the Enterprise repository. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector.proto: Table carries its location and storage options The gateway resolves the table it addresses, so the engine can open it without a describe round trip of its own. Both fields are optional; an empty location means the engine looks the table up as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: test the in-process describe Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: resolve the table before calling the engine The gateway describes the table in process, through the trusted catalog manager, and sends its location and storage options with every engine call. Storage options are vended for the caller's principal, as the Enterprise router does for queries. A catalog failure answers with the code the engine would have given. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: open a table at the location the gateway sent A request whose table carries a location is opened there, with the storage options the gateway vended over the engine's fallback; the catalog is described only for a table that comes without one, as from an older gateway. dataset::open_at and dataset::drop_at take a description the caller already holds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: run the engine without a catalog VECTOR_ENGINE_NO_CATALOG=1 gives the engine an unreachable catalog URL; every CI leg sets it, so the suite proves the gateway resolves each table it sends. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: an internal describe may write a vector index With credential vending on, the options vended to the engine followed the caller's s3tables grants, which never allow writing the vector namespace, so every write the gateway sent would have run with read-only credentials and reads would have lost their version lease. The engine is that namespace's writer and the caller's vector permission authorized the operation, so an internal describe of a vector table vends write-capable options; other namespaces keep the UpdateTable check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: resolve tables for the engine only while the catalog runs With -port.lance=0 the gateway has no catalog to resolve through; the engine then keeps looking tables up itself, as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: a deregistered table describes as NoSuchTable So the gateway answers NotFoundException for it, not InternalFailure. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: trust a sent location only for a colocated vector index A caller that reaches the engine could otherwise name any storage the engine's credentials cover. A location is taken from the request only when the id names a vector index and the location is its colocated path, which the engine writes already; anything else is described through the catalog as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: scope the read cache by storage endpoint The cache was keyed by bucket and path alone, so the same path reached through another endpoint would have shared entries. Each open wraps the store with the cache scoped to the endpoint its options name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep a configured endpoint over one a request names A caller that reaches the engine could otherwise send its fallback credentials to an endpoint of the caller's choosing. When the engine's --storage-option names an aws_endpoint, a table whose request names another one is described through the catalog instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: use request options only where credentials cannot leave A request's storage options are taken only when they hold the keys a gateway's describe produces, so a proxy or an endpoint alias cannot route the engine's credentials elsewhere, and an endpoint only when it is the configured one or the engine holds no credentials to send. Anything else sends the table to the catalog. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: give the engine its endpoint with its credentials Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a drop checks the request's options like an open Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: an unpinned endpoint needs the request to sign for itself Without configured credentials the S3 store still signs with the AWS default chain, so an endpoint from a request is taken only when the request skips signing or brings its own credentials. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a trailing slash does not unpin the endpoint, and say so A configured aws_endpoint written with a trailing slash matched nothing the gateway sends, so every table fell back to the catalog in silence. The comparison ignores the slash, and a fallback for a resolved table is logged at debug. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: resolved by the gateway, or refused The engine no longer has a catalog: --namespace and --filer-signing-key are gone, a request whose table names no location is InvalidArgument, and one whose location is not the index's own or whose storage options the engine's credentials must not be used with is PermissionDenied. A drop removes the colocated path it was given. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: the catalog resolves vector tables whether or not its port is served newLanceServer builds the Lance catalog for the S3 Vectors API; the listener on -port.lance is optional. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors tests: start the engine without a catalog Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: the aux listener test builds the catalog before serving it Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: an unpinned endpoint takes only credentials the request signs with alone Lance fills an option a request leaves out from the engine's environment, under any of object_store's names for it (TOKEN, SESSION_TOKEN, ...), and asks the AWS default chain when the key pair is incomplete. A request that named its own endpoint with only an access key id, or a key pair while the engine's environment held a session token, would have sent the engine's credentials there. Such a request now needs a full key pair, and its own session token whenever the environment would supply one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: accept --namespace and --filer-signing-key, ignored Both flags shipped in 4.52 and are documented, so dropping them made an existing command line exit at startup with an unexpected-argument error. They are kept hidden and ignored, with a warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors design: the engine is started without a catalog Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: the trust check takes the environment it reads The options test read the process environment through signs_alone, so a host exporting AWS_SESSION_TOKEN or TOKEN failed it. The caller now passes the variable names in; describe passes the engine's own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: log both endpoints when a request's options are refused The gateway turns the PermissionDenied into a bare AccessDenied and logs nothing, so a pinned aws_endpoint written in another form than the one the gateway sends (host case, a default port, an IP for a name) failed every call with no trace of which value differed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: the Lance catalog sets the credential role whether or not Iceberg runs -iceberg.credentialRole reached the S3 server only while the Iceberg listener started, so with -port.iceberg=0 the Lance catalog installed its vendor but vending stayed off, and the vector engine got no credentials for the tables the gateway resolves. Both catalogs now set the role through one helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: set the credential role once, before either catalog serves Both catalogs wrote the role as they were built; the second write landed while the first was already serving requests that read it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-07 12:37:41 +01:00 · 2026-11-06 19:39:09 -08:00
1 parent 7beac0e8ea
commit 2fe480a6e2
@@ -32,7 +32,7 @@ bytes = "1"
 clap = { version = "4", features = ["derive", "env"] }
 futures = "0.3"
 moka = { version = "0.12", features = ["sync"] }
-object_store = "0.13"
+object_store = { version = "0.13", features = ["aws"] }
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"
 toml = "0.8"
@@ -4,7 +4,6 @@ use anyhow::{Context, Result};
 use clap::Parser;
 use tokio_stream::wrappers::TcpListenerStream;
 use tonic::transport::Server;
-use weed_lance_worker::catalog::{NamespaceClient, SigningKey};
 use weed_lance_worker::dataset::FallbackOptions;
 use weed_vector_server::pb::vector::vector_service_server::VectorServiceServer;
 use weed_vector_server::read_cache::ReadCache;
@@ -15,18 +14,12 @@ use weed_vector_server::tls::{self, ServerTls};
 /// The VectorService the S3 gateway fans vector calls out to.
 #[derive(Parser)]
 struct Args {
-    /// Lance Namespace base URL: the S3 gateway's Lance port (-s3.port.lance).
-    #[arg(
-        long,
-        env = "WEED_VECTOR_NAMESPACE",
-        default_value = "http://localhost:9101"
-    )]
-    namespace: String,
    /// gRPC listen address.
    #[arg(long, env = "WEED_VECTOR_LISTEN", default_value = "127.0.0.1:8090")]
    listen: String,
-    /// object_store option fallback for deployments that vend no credentials:
-    /// key=value, repeatable.
+    /// object_store options for deployments whose gateway vends no
+    /// credentials: key=value, repeatable. Credentials given here are used
+    /// only with the aws_endpoint given here.
    #[arg(long = "storage-option", alias = "storage.option", value_parser = parse_option)]
    storage_options: Vec<(String, String)>,
    /// Comma-separated master addresses to register this server with; the
@@ -40,10 +33,6 @@ struct Args {
    /// tenth of the index's partitions, at least 8; more under a filter.
    #[arg(long, env = "WEED_VECTOR_NPROBES")]
    nprobes: Option<std::num::NonZeroUsize>,
-    /// The `jwt.filer_signing.key` of the cluster. The Lance namespace vends
-    /// credentials that write index data only to a caller holding it.
-    #[arg(long, env = "WEED_FILER_SIGNING_KEY")]
-    filer_signing_key: Option<String>,
    /// The cluster's security.toml, for gRPC TLS: this server's identity is
    /// [grpc.vector], the masters are dialed with [grpc.client]. By default
    /// the file weed processes find in -config_dir, ., ~/.seaweedfs and
@@ -54,6 +43,13 @@ struct Args {
    /// first.
    #[arg(long, alias = "config_dir", env = "WEED_VECTOR_CONFIG_DIR")]
    config_dir: Option<PathBuf>,
+    /// Ignored: the gateway resolves every table it sends. Still accepted so
+    /// an existing command line starts.
+    #[arg(long, hide = true)]
+    namespace: Option<String>,
+    /// Ignored, like --namespace.
+    #[arg(long, hide = true)]
+    filer_signing_key: Option<String>,
 }

 fn parse_option(s: &str) -> Result<(String, String), String> {
@@ -123,6 +119,11 @@ fn advertised_addr(listen: &str, masters: &[String]) -> String {
 async fn main() -> Result<()> {
    tracing_subscriber::fmt::init();
    let args = Args::parse_from(normalize_args(std::env::args().collect()));
+    if args.namespace.is_some() || args.filer_signing_key.is_some() {
+        tracing::warn!(
+            "--namespace and --filer-signing-key are ignored: the gateway resolves every table it sends"
+        );
+    }
    weed_lance_worker::tls::install_default_crypto_provider();
    let security = Security::load(args.security_toml.as_deref(), args.config_dir.as_deref())?;
    if let Some(path) = security.path() {
@@ -154,17 +155,12 @@ async fn main() -> Result<()> {
        ));
    }
    let fallback: FallbackOptions = args.storage_options.into_iter().collect();
-    let signing_key = SigningKey::new(args.filer_signing_key);
-    let client = NamespaceClient::new(args.namespace.clone())
-        .with_fallback(&fallback)
-        .with_signing_key(signing_key.as_ref());
-    let mut service = VectorServer::new(client, fallback)
-        .with_nprobes(args.nprobes.map(std::num::NonZeroUsize::get));
+    let mut service =
+        VectorServer::new(fallback).with_nprobes(args.nprobes.map(std::num::NonZeroUsize::get));
    if args.read_cache_mb > 0 {
        service = service.with_read_cache(ReadCache::new(args.read_cache_mb << 20));
    }
    tracing::info!(
-        namespace = %args.namespace,
        listen = %args.listen,
        "weed-vector; serving gRPC {}, dialing masters {}",
        match &server_tls {
@@ -208,4 +204,21 @@ mod tests {
        let args = Args::try_parse_from(["weed-vector", "--nprobes", "12"]).unwrap();
        assert_eq!(args.nprobes.map(std::num::NonZeroUsize::get), Some(12));
    }
+
+    #[test]
+    fn an_earlier_command_line_still_parses() {
+        let args = Args::try_parse_from(normalize_args(
+            [
+                "weed-vector",
+                "--namespace",
+                "http://localhost:9101",
+                "-filer-signing-key",
+                "k",
+            ]
+            .map(String::from)
+            .to_vec(),
+        ))
+        .unwrap();
+        assert!(args.namespace.is_some() && args.filer_signing_key.is_some());
+    }
 }
@@ -37,26 +37,40 @@ struct Entry {

 /// Wraps each object store Lance opens; clones share one cache.
 #[derive(Clone)]
-pub struct ReadCache(moka::sync::Cache<Key, Entry>);
+pub struct ReadCache {
+    cache: moka::sync::Cache<Key, Entry>,
+    /// Keeps entries apart from those of another [`Self::scoped`] view, for
+    /// stores that share a prefix but not their bytes.
+    scope: Arc<str>,
+}

 impl ReadCache {
    pub fn new(capacity_bytes: u64) -> Self {
-        Self(
-            moka::sync::Cache::builder()
+        Self {
+            cache: moka::sync::Cache::builder()
                .max_capacity(capacity_bytes)
                .weigher(|key: &Key, entry: &Entry| {
                    u32::try_from(key.path.as_ref().len() + entry.bytes.len()).unwrap_or(u32::MAX)
                })
                .build(),
-        )
+            scope: "".into(),
+        }
+    }
+
+    /// The same cache, keyed apart under `scope`.
+    pub fn scoped(&self, scope: &str) -> Self {
+        Self {
+            cache: self.cache.clone(),
+            scope: scope.into(),
+        }
    }
 }

 impl fmt::Debug for ReadCache {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.debug_struct("ReadCache")
-            .field("entries", &self.0.entry_count())
-            .field("bytes", &self.0.weighted_size())
+            .field("entries", &self.cache.entry_count())
+            .field("bytes", &self.cache.weighted_size())
            .finish()
    }
 }
@@ -65,8 +79,8 @@ impl WrappingObjectStore for ReadCache {
    fn wrap(&self, store_prefix: &str, original: Arc<dyn ObjectStore>) -> Arc<dyn ObjectStore> {
        Arc::new(CachedStore {
            inner: original,
-            store: store_prefix.into(),
-            cache: self.0.clone(),
+            store: format!("{}|{store_prefix}", self.scope).into(),
+            cache: self.cache.clone(),
        })
    }
 }
@@ -277,7 +291,6 @@ mod tests {
    use object_store::ObjectStoreExt;
    use object_store::memory::InMemory;
    use tonic::Request;
-    use weed_lance_worker::catalog::NamespaceClient;
    use weed_lance_worker::dataset::FallbackOptions;

    fn cached(inner: Arc<InMemory>) -> Arc<dyn ObjectStore> {
@@ -373,8 +386,8 @@ mod tests {
                .unwrap();
            read(&store, &path, 0..(8 << 10)).await;
        }
-        cache.0.run_pending_tasks();
-        assert!(cache.0.weighted_size() <= 64 << 10, "{cache:?}");
+        cache.cache.run_pending_tasks();
+        assert!(cache.cache.weighted_size() <= 64 << 10, "{cache:?}");
    }

    #[tokio::test]
@@ -392,40 +405,15 @@ mod tests {
        let backing = whole.as_ptr_range();
        read(&store, &path, 10..20).await;
        let key = Key {
-            store: "memory".into(),
+            store: "|memory".into(),
            path,
            range: Some(10..20),
        };
-        let cached = cache.0.get(&key).unwrap().bytes;
+        let cached = cache.cache.get(&key).unwrap().bytes;
        assert_eq!(cached.len(), 10);
        assert!(!backing.contains(&cached.as_ptr()));
    }

-    /// A catalog that answers every request with `location`.
-    async fn catalog_at(location: String) -> String {
-        use tokio::io::{AsyncReadExt, AsyncWriteExt};
-        let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
-        let url = format!("http://{}", listener.local_addr().unwrap());
-        let body = serde_json::json!({ "location": location }).to_string();
-        tokio::spawn(async move {
-            loop {
-                let (mut conn, _) = listener.accept().await.unwrap();
-                let body = body.clone();
-                tokio::spawn(async move {
-                    let mut request = [0u8; 8192];
-                    let _ = conn.read(&mut request).await;
-                    let response = format!(
-                        "HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{body}",
-                        body.len()
-                    );
-                    let _ = conn.write_all(response.as_bytes()).await;
-                    let _ = conn.shutdown().await;
-                });
-            }
-        });
-        url
-    }
-
    fn vector(key: &str, data: Vec<f32>) -> pb::Vector {
        pb::Vector {
            key: key.to_string(),
@@ -492,15 +480,15 @@ mod tests {
    /// dropped and recreated at the same location answers with its new rows.
    #[tokio::test]
    async fn cached_reads_follow_writes_and_a_recreated_index() {
-        let location = format!("shared-memory://cache-{}/vector/x", uuid_like(),);
-        let server = VectorServer::new(
-            NamespaceClient::new(catalog_at(location.clone()).await),
-            FallbackOptions::new(),
-        )
-        .with_read_cache(ReadCache::new(16 << 20));
+        let bucket = format!("cache-{}", uuid_like());
+        let location = format!("shared-memory://{bucket}/vector/x");
+        let server = VectorServer::new(FallbackOptions::new())
+            .with_schemes(&["shared-memory"])
+            .with_read_cache(ReadCache::new(16 << 20));
        let table = pb::Table {
-            id: vec!["bkt".into(), "vector".into(), "x".into()],
-            version: 0,
+            id: vec![bucket, "vector".into(), "x".into()],
+            location: location.clone(),
+            ..Default::default()
        };

        create(&server, &table, 4).await;
@@ -12,12 +12,16 @@ use lance::io::ObjectStoreParams;
 use lance::io::StorageOptionsAccessor;
 use lance::io::WrappingObjectStore;
 use lance::table::format::Fragment;
+use object_store::aws::AmazonS3ConfigKey;
 use prost_types::{ListValue, Struct, Value, value::Kind};
 use std::collections::{BTreeMap, HashMap};
+use std::ffi::OsString;
+use std::str::FromStr;
 use std::sync::Arc;
 use tonic::{Request, Response, Status};
-use weed_lance_worker::catalog::NamespaceClient;
+use weed_lance_worker::catalog::TableDescription;
 use weed_lance_worker::dataset::{self, FallbackOptions};
+use weed_lance_worker::jobs::vector_index::is_vector_table;
 use weed_vector_core::index;
 use weed_vector_core::query::{self, Query};
 use weed_vector_core::table::VectorTable;
@@ -35,17 +39,14 @@ fn invalid(err: impl Into<anyhow::Error>) -> Status {

 fn open_status(err: anyhow::Error) -> Status {
    let missing = err.chain().any(|cause| {
-        cause
-            .downcast_ref::<reqwest::Error>()
-            .is_some_and(|e| e.status() == Some(reqwest::StatusCode::NOT_FOUND))
-            || cause.downcast_ref::<lance::Error>().is_some_and(|e| {
-                matches!(
-                    e,
-                    lance::Error::DatasetNotFound { .. }
-                        | lance::Error::NotFound { .. }
-                        | lance::Error::VersionNotFound { .. }
-                )
-            })
+        cause.downcast_ref::<lance::Error>().is_some_and(|e| {
+            matches!(
+                e,
+                lance::Error::DatasetNotFound { .. }
+                    | lance::Error::NotFound { .. }
+                    | lance::Error::VersionNotFound { .. }
+            )
+        })
    });
    if missing {
        Status::not_found(format!("{:#}", err))
@@ -58,6 +59,69 @@ const KEY_COLUMN: &str = "key";
 const METADATA_COLUMN: &str = "metadata";
 const VECTOR_COLUMN: &str = "vector";
 const ROW_ID_COLUMN: &str = "_rowid";
+const ENDPOINT_OPTION: &str = "aws_endpoint";
+/// The options a gateway's describe produces. Any other key a request sends,
+/// such as a proxy or an endpoint alias, could route the engine's credentials
+/// elsewhere, so it is not taken from the request.
+const SENT_OPTIONS: [&str; 8] = [
+    ENDPOINT_OPTION,
+    "allow_http",
+    "aws_region",
+    "aws_skip_signature",
+    "aws_access_key_id",
+    "aws_secret_access_key",
+    "aws_session_token",
+    "expires_at_millis",
+];
+const CREDENTIAL_OPTIONS: [&str; 3] = [
+    "aws_access_key_id",
+    "aws_secret_access_key",
+    "aws_session_token",
+];
+
+/// Whether a request's options may be used with the engine's fallback: known
+/// keys only, and an endpoint that is the configured one, or any when no
+/// credential of the engine's can reach it: none configured, and the request
+/// either skips signing or signs with its own credentials alone. `env` names
+/// the engine's environment variables.
+fn options_trusted(
+    sent: &HashMap<String, String>,
+    fallback: &FallbackOptions,
+    env: impl IntoIterator<Item = OsString>,
+) -> bool {
+    if sent.keys().any(|key| !SENT_OPTIONS.contains(&key.as_str())) {
+        return false;
+    }
+    match (sent.get(ENDPOINT_OPTION), fallback.get(ENDPOINT_OPTION)) {
+        (None, _) => true,
+        (Some(sent), Some(pinned)) => sent.trim_end_matches('/') == pinned.trim_end_matches('/'),
+        (Some(_), None) => {
+            let configured = CREDENTIAL_OPTIONS
+                .iter()
+                .any(|key| fallback.contains_key(*key));
+            let unsigned = sent.get("aws_skip_signature").is_some_and(|v| v == "true");
+            !configured && (unsigned || signs_alone(sent, env))
+        }
+    }
+}
+
+/// Whether lance signs with the request's credentials alone. Lance fills an
+/// option the request leaves out from an environment variable under any of
+/// object_store's names for it, and asks the AWS default chain when the key
+/// pair is incomplete.
+fn signs_alone(sent: &HashMap<String, String>, env: impl IntoIterator<Item = OsString>) -> bool {
+    let has = |key: &str| sent.contains_key(key);
+    let env_token = || {
+        env.into_iter().any(|name| {
+            name.to_str()
+                .and_then(|name| AmazonS3ConfigKey::from_str(&name.to_ascii_lowercase()).ok())
+                == Some(AmazonS3ConfigKey::Token)
+        })
+    };
+    has("aws_access_key_id")
+        && has("aws_secret_access_key")
+        && (has("aws_session_token") || !env_token())
+}

 fn schema_for(
    dimension: usize,
@@ -370,23 +434,50 @@ fn rows_as_vectors(
    out
 }

+/// The description a gateway sent with the table. Only a vector index at its
+/// colocated path is taken, since the engine is that path's writer already;
+/// any other location would point the engine at storage the catalog does not
+/// know.
+fn resolved(table: &pb::Table, schemes: &[&str]) -> Option<TableDescription> {
+    let colocated = dataset::table_location(&table.id)?;
+    let path = colocated.strip_prefix("s3://")?;
+    let location = table.location.trim_end_matches('/');
+    let trusted = is_vector_table(&table.id)
+        && schemes
+            .iter()
+            .any(|scheme| location == format!("{scheme}://{path}"));
+    trusted.then(|| TableDescription {
+        location: location.to_string(),
+        storage_options: table.storage_options.clone(),
+        managed_versioning: false,
+    })
+}
+
 pub struct VectorServer {
-    client: NamespaceClient,
    fallback: FallbackOptions,
-    read_cache: Option<Arc<dyn WrappingObjectStore>>,
+    read_cache: Option<ReadCache>,
    nprobes: Option<usize>,
+    /// URI schemes a table's location may have.
+    schemes: &'static [&'static str],
 }

 impl VectorServer {
-    pub fn new(client: NamespaceClient, fallback: FallbackOptions) -> Self {
+    pub fn new(fallback: FallbackOptions) -> Self {
        Self {
-            client,
            fallback,
            read_cache: None,
            nprobes: None,
+            schemes: &["s3"],
        }
    }

+    /// Tables on an in-process store, for tests.
+    #[cfg(test)]
+    pub(crate) fn with_schemes(mut self, schemes: &'static [&'static str]) -> Self {
+        self.schemes = schemes;
+        self
+    }
+
    /// Partitions every query probes, instead of the index's own default.
    pub fn with_nprobes(mut self, nprobes: Option<usize>) -> Self {
        self.nprobes = nprobes;
@@ -395,21 +486,64 @@ impl VectorServer {

    /// Serve repeated reads of write-once table files from `cache`.
    pub fn with_read_cache(mut self, cache: ReadCache) -> Self {
-        self.read_cache = Some(Arc::new(cache));
+        self.read_cache = Some(cache);
        self
    }

-    async fn open(&self, table: &pb::Table) -> Result<Dataset> {
-        let opened = dataset::open_with(
-            &self.client,
-            &table.id,
-            &self.fallback,
-            self.read_cache.clone(),
-        )
-        .await?;
-        let mut dataset = opened.dataset;
+    /// The table as the gateway resolved it. A request that names no
+    /// location, another location than the index's own, or storage options
+    /// the engine's credentials must not be used with is refused.
+    fn describe(&self, table: &pb::Table) -> Result<TableDescription, Status> {
+        if table.location.is_empty() {
+            return Err(Status::invalid_argument("the table names no location"));
+        }
+        let description = resolved(table, self.schemes).ok_or_else(|| {
+            Status::permission_denied(format!(
+                "{} is not the location of index {}",
+                table.location,
+                table.id.join("/")
+            ))
+        })?;
+        let env = std::env::vars_os().map(|(name, _)| name);
+        if !options_trusted(&description.storage_options, &self.fallback, env) {
+            // The gateway answers AccessDenied without logging, so this is
+            // where an operator sees a pinned endpoint that does not match.
+            let endpoint = |options: &HashMap<String, String>| {
+                options.get(ENDPOINT_OPTION).cloned().unwrap_or_default()
+            };
+            tracing::warn!(
+                index = %table.id.join("/"),
+                sent = %endpoint(&description.storage_options),
+                pinned = %endpoint(&self.fallback),
+                "refused the request's storage options: an aws_endpoint other than the pinned one, a key other than the gateway's, or credentials the engine would have to complete"
+            );
+            return Err(Status::permission_denied(
+                "the storage options would send the engine's credentials elsewhere",
+            ));
+        }
+        Ok(description)
+    }
+
+    async fn open(&self, table: &pb::Table) -> Result<Dataset, Status> {
+        let description = self.describe(table)?;
+        // The options name the store the bytes come from; the cache is scoped
+        // by it so the same bucket and path on another endpoint never share.
+        let cache = self.read_cache.as_ref().map(|cache| {
+            let options = dataset::table_options(&description, &self.fallback);
+            let endpoint = options
+                .get(ENDPOINT_OPTION)
+                .map(String::as_str)
+                .unwrap_or("");
+            Arc::new(cache.scoped(endpoint)) as Arc<dyn WrappingObjectStore>
+        });
+        let mut dataset = dataset::open_at(&description, &self.fallback, cache)
+            .await
+            .map_err(open_status)?;
        if table.version != 0 {
-            dataset = dataset.checkout_version(table.version).await?;
+            dataset = dataset
+                .checkout_version(table.version)
+                .await
+                .map_err(|err| open_status(err.into()))?;
        }
        Ok(dataset)
    }
@@ -433,11 +567,7 @@ impl VectorService for VectorServer {
            return Err(Status::invalid_argument("dimension must be 1..=4096"));
        }
        // The gateway has already created the table through the catalog.
-        let description = self
-            .client
-            .describe_table(&table.id)
-            .await
-            .map_err(internal)?;
+        let description = self.describe(&table)?;
        let options = dataset::table_options(&description, &self.fallback);
        let schema = schema_for(dimension, metric, &req.non_filterable_metadata_keys);
        let empty = RecordBatch::new_empty(schema.clone());
@@ -470,7 +600,8 @@ impl VectorService for VectorServer {
        let table = req
            .table
            .ok_or_else(|| Status::invalid_argument("missing table"))?;
-        dataset::drop(&self.client, &table.id, &self.fallback)
+        let description = self.describe(&table)?;
+        dataset::drop_at(&table.id, &description, &self.fallback)
            .await
            .map_err(|err| {
                if err.is::<dataset::DropRefused>() {
@@ -490,7 +621,7 @@ impl VectorService for VectorServer {
        let table = req
            .table
            .ok_or_else(|| Status::invalid_argument("missing table"))?;
-        let dataset = self.open(&table).await.map_err(open_status)?;
+        let dataset = self.open(&table).await?;
        if req.vectors.is_empty() {
            return Ok(Response::new(pb::PutVectorsResponse {
                version: dataset.version().version,
@@ -532,7 +663,7 @@ impl VectorService for VectorServer {
        let table = req
            .table
            .ok_or_else(|| Status::invalid_argument("missing table"))?;
-        let dataset = self.open(&table).await.map_err(open_status)?;
+        let dataset = self.open(&table).await?;
        // Lease the resolved version so maintenance cannot reclaim its files
        // while this query runs against it.
        let _lease = weed_vector_core::maintenance::try_pin(&dataset)
@@ -556,7 +687,7 @@ impl VectorService for VectorServer {
        let table = req
            .table
            .ok_or_else(|| Status::invalid_argument("missing table"))?;
-        let mut dataset = self.open(&table).await.map_err(open_status)?;
+        let mut dataset = self.open(&table).await?;
        if !req.keys.is_empty() {
            dataset
                .delete(&key_list(&req.keys))
@@ -582,7 +713,7 @@ impl VectorService for VectorServer {
                "segment_index must be below segment_count, which is at most {MAX_SEGMENTS}"
            )));
        }
-        let dataset = self.open(&table).await.map_err(open_status)?;
+        let dataset = self.open(&table).await?;
        let offset: u64 = if req.next_token.is_empty() {
            0
        } else {
@@ -649,7 +780,7 @@ impl VectorService for VectorServer {
        let table = req
            .table
            .ok_or_else(|| Status::invalid_argument("missing table"))?;
-        let dataset = self.open(&table).await.map_err(open_status)?;
+        let dataset = self.open(&table).await?;
        // Lease the resolved version so maintenance cannot reclaim its files
        // while this query runs against it.
        let _lease = weed_vector_core::maintenance::try_pin(&dataset)
@@ -941,45 +1072,210 @@ mod tests {
        assert_eq!(next, None);
    }

-    /// A catalog that answers one describe with `location`.
-    async fn catalog_describing(location: &str) -> String {
-        use tokio::io::{AsyncReadExt, AsyncWriteExt};
-        let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
-        let url = format!("http://{}", listener.local_addr().unwrap());
-        let body = serde_json::json!({ "location": location }).to_string();
-        tokio::spawn(async move {
-            let (mut conn, _) = listener.accept().await.unwrap();
-            let mut request = [0u8; 8192];
-            let _ = conn.read(&mut request).await;
-            let response = format!(
-                "HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{body}",
-                body.len()
-            );
-            conn.write_all(response.as_bytes()).await.unwrap();
-            let _ = conn.shutdown().await;
-            let _ = tokio::io::copy(&mut conn, &mut tokio::io::sink()).await;
-        });
-        url
-    }
-
+    /// A location that is not the index's own is refused before anything is
+    /// opened or removed.
    #[tokio::test]
    async fn drop_index_refuses_a_location_outside_the_table() {
-        for location in ["s3://bkt/vector", "s3://bkt/vector/prod"] {
-            let server = VectorServer::new(
-                NamespaceClient::new(catalog_describing(location).await),
-                FallbackOptions::new(),
-            );
+        let server = VectorServer::new(FallbackOptions::new());
+        for location in [
+            "s3://bkt/vector",
+            "s3://bkt/vector/prod",
+            "s3://other/vector/x",
+        ] {
            let request = pb::DropIndexRequest {
                table: Some(pb::Table {
                    id: vec!["bkt".into(), "vector".into(), "x".into()],
-                    version: 0,
+                    location: location.into(),
+                    ..Default::default()
                }),
            };
            let err = server
                .drop_index(Request::new(request))
                .await
                .expect_err(location);
-            assert_eq!(err.code(), tonic::Code::FailedPrecondition, "{location}");
+            assert_eq!(err.code(), tonic::Code::PermissionDenied, "{location}");
        }
    }
+
+    /// Only a vector index at its colocated path is taken from the request.
+    #[test]
+    fn resolved_takes_only_a_colocated_vector_index() {
+        let table = |id: &[&str], location: &str| pb::Table {
+            id: id.iter().map(|s| s.to_string()).collect(),
+            location: location.into(),
+            storage_options: HashMap::from([("aws_endpoint".to_string(), "http://gw".to_string())]),
+            ..Default::default()
+        };
+        let s3 = &["s3"];
+        let index = ["bkt", "vector", "x"];
+        for location in ["s3://bkt/vector/x", "s3://bkt/vector/x/"] {
+            let description = resolved(&table(&index, location), s3).expect(location);
+            assert_eq!(description.location, "s3://bkt/vector/x");
+            assert_eq!(
+                description
+                    .storage_options
+                    .get("aws_endpoint")
+                    .map(String::as_str),
+                Some("http://gw")
+            );
+        }
+        assert!(resolved(&table(&index, ""), s3).is_none());
+        assert!(resolved(&table(&index, "s3://other/vector/x"), s3).is_none());
+        assert!(resolved(&table(&index, "s3://bkt/vector/y"), s3).is_none());
+        assert!(resolved(&table(&index, "file://bkt/vector/x"), s3).is_none());
+        assert!(resolved(&table(&index, "memory://bkt/vector/x"), &["s3", "memory"]).is_some());
+        assert!(resolved(&table(&["bkt", "ns", "x"], "s3://bkt/ns/x"), s3).is_none());
+        assert!(resolved(&table(&["bkt", "x"], "s3://bkt/x"), s3).is_none());
+    }
+
+    /// A request names the table's location or is refused.
+    #[tokio::test]
+    async fn a_table_without_its_location_is_refused() {
+        let server = VectorServer::new(FallbackOptions::new());
+        for (location, code) in [
+            ("", tonic::Code::InvalidArgument),
+            ("s3://other/vector/x", tonic::Code::PermissionDenied),
+        ] {
+            let err = server
+                .get_vectors(Request::new(pb::GetVectorsRequest {
+                    table: Some(pb::Table {
+                        id: vec!["bkt".into(), "vector".into(), "x".into()],
+                        location: location.into(),
+                        ..Default::default()
+                    }),
+                    keys: vec!["k001".into()],
+                    ..Default::default()
+                }))
+                .await
+                .expect_err(location);
+            assert_eq!(err.code(), code, "{location}");
+        }
+    }
+
+    /// A request's options are used with the engine's credentials only when
+    /// they cannot send those credentials elsewhere.
+    #[test]
+    fn options_are_trusted_only_where_credentials_cannot_leave() {
+        let options = |pairs: &[(&str, &str)]| -> HashMap<String, String> {
+            pairs
+                .iter()
+                .map(|(k, v)| (k.to_string(), v.to_string()))
+                .collect()
+        };
+        let creds = options(&[("aws_access_key_id", "k"), ("aws_secret_access_key", "s")]);
+        let pinned = options(&[
+            ("aws_access_key_id", "k"),
+            ("aws_secret_access_key", "s"),
+            (ENDPOINT_OPTION, "http://gw"),
+        ]);
+        let gateway = options(&[(ENDPOINT_OPTION, "http://gw"), ("allow_http", "true")]);
+        let elsewhere = options(&[(ENDPOINT_OPTION, "http://elsewhere")]);
+        let unsigned = options(&[
+            (ENDPOINT_OPTION, "http://gw"),
+            ("aws_skip_signature", "true"),
+        ]);
+        let vended = options(&[
+            (ENDPOINT_OPTION, "http://gw"),
+            ("aws_access_key_id", "v"),
+            ("aws_secret_access_key", "v"),
+        ]);
+        let none = FallbackOptions::new();
+        let trusted = |sent: &HashMap<String, String>, fallback: &FallbackOptions| {
+            options_trusted(sent, fallback, Vec::<OsString>::new())
+        };
+
+        assert!(trusted(&gateway, &pinned));
+        assert!(trusted(
+            &options(&[(ENDPOINT_OPTION, "http://gw/")]),
+            &pinned
+        ));
+        assert!(!trusted(&elsewhere, &pinned));
+        assert!(!trusted(&gateway, &creds), "no configured endpoint");
+        assert!(!trusted(&gateway, &none), "the default chain could sign");
+        assert!(trusted(&unsigned, &none));
+        assert!(trusted(&vended, &none));
+        assert!(
+            !options_trusted(&vended, &none, [OsString::from("AWS_SESSION_TOKEN")]),
+            "the environment would add its session token"
+        );
+        assert!(
+            !trusted(
+                &options(&[(ENDPOINT_OPTION, "http://gw"), ("aws_access_key_id", "v")]),
+                &none
+            ),
+            "a key without its secret is signed by the default chain"
+        );
+        assert!(
+            !trusted(&vended, &creds),
+            "configured credentials stay home"
+        );
+        assert!(
+            trusted(&options(&[("aws_region", "r")]), &creds),
+            "no endpoint sent"
+        );
+        for key in ["proxy_url", "aws_endpoint_url_s3", "aws_endpoint_url"] {
+            let routed = options(&[(key, "http://elsewhere"), ("aws_skip_signature", "true")]);
+            assert!(!trusted(&routed, &none), "{key}");
+        }
+    }
+
+    /// A request signs alone only with a full key pair, and with its own
+    /// session token whenever the environment would supply one.
+    #[test]
+    fn a_request_signs_alone_without_the_environment() {
+        let sent = |pairs: &[(&str, &str)]| -> HashMap<String, String> {
+            pairs
+                .iter()
+                .map(|(k, v)| (k.to_string(), v.to_string()))
+                .collect()
+        };
+        let env = |names: &[&str]| -> Vec<OsString> { names.iter().map(OsString::from).collect() };
+        let pair = sent(&[("aws_access_key_id", "v"), ("aws_secret_access_key", "v")]);
+        let with_token = sent(&[
+            ("aws_access_key_id", "v"),
+            ("aws_secret_access_key", "v"),
+            ("aws_session_token", "t"),
+        ]);
+
+        assert!(signs_alone(&pair, env(&["AWS_REGION", "HOME"])));
+        assert!(!signs_alone(&sent(&[("aws_access_key_id", "v")]), env(&[])));
+        assert!(!signs_alone(
+            &sent(&[("aws_secret_access_key", "v")]),
+            env(&[])
+        ));
+        for name in [
+            "AWS_SESSION_TOKEN",
+            "aws_session_token",
+            "TOKEN",
+            "SESSION_TOKEN",
+        ] {
+            assert!(!signs_alone(&pair, env(&[name])), "{name}");
+            assert!(signs_alone(&with_token, env(&[name])), "{name}");
+        }
+    }
+
+    /// Options the engine must not use are refused, for a drop as for an open.
+    #[tokio::test]
+    async fn untrusted_options_are_refused() {
+        let sent = |endpoint: &str| pb::Table {
+            id: vec!["bkt".into(), "vector".into(), "x".into()],
+            location: "s3://bkt/vector/x".into(),
+            storage_options: HashMap::from([(ENDPOINT_OPTION.to_string(), endpoint.to_string())]),
+            ..Default::default()
+        };
+        let pinned =
+            FallbackOptions::from([(ENDPOINT_OPTION.to_string(), "http://gw".to_string())]);
+        let server = VectorServer::new(pinned);
+        let description = server.describe(&sent("http://gw")).unwrap();
+        assert_eq!(description.location, "s3://bkt/vector/x");
+        let err = server.describe(&sent("http://elsewhere")).unwrap_err();
+        assert_eq!(err.code(), tonic::Code::PermissionDenied);
+        let err = server
+            .drop_index(Request::new(pb::DropIndexRequest {
+                table: Some(sent("http://elsewhere")),
+            }))
+            .await
+            .unwrap_err();
+        assert_eq!(err.code(), tonic::Code::PermissionDenied);
+    }
 }