10

S3 Vectors API

Chris Lu edited this page 2026-11-05 12:54:16 -08:00

Table of Contents

• S3 Vectors API
• Operations
• Vector buckets
• Indexes
• Tags
• Distance metrics
• Writing vectors
• Reading vectors
• Querying
• Metadata filters
• Errors

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

S3 Vectors API

What the SeaweedFS S3 Vectors API supports, and where it differs from AWS. For setup see S3 Vector Bucket; for permissions see S3 Vector Bucket Security.

Operations

Requests go to the S3 port, signed with SigV4 for the service s3vectors. Use an AWS CLI or SDK s3vectors client with its endpoint set to the SeaweedFS S3 address; every example on these pages was run with the AWS CLI and boto3.

Operation	Supported
CreateVectorBucket, GetVectorBucket, ListVectorBuckets, DeleteVectorBucket	yes
PutVectorBucketPolicy, GetVectorBucketPolicy, DeleteVectorBucketPolicy	yes
CreateIndex, GetIndex, ListIndexes, DeleteIndex	yes
PutVectors, GetVectors, ListVectors, DeleteVectors, QueryVectors	yes
TagResource, UntagResource, ListTagsForResource	yes, see Tags

Every bucket and index can be addressed by name or by ARN. ARNs always use region us-east-1 and account 000000000000, e.g. arn:aws:s3vectors:us-east-1:000000000000:bucket/media/index/movies; an ARN with another partition, region or account, or a name that disagrees with the ARN, is rejected with ValidationException. The region a request is signed for is not checked.

Vector buckets

• Names: 3 to 63 characters, lowercase letters, digits and hyphens, starting and ending with a letter or digit.
• Deleting a bucket that still has indexes fails with ConflictException.
• Encryption: with encryptionConfiguration {"sseType": "AES256"}, every object of the bucket's indexes is written with SSE-S3: HeadObject on them shows ServerSideEncryption: AES256, and queries work as in an unencrypted bucket. GetVectorBucket returns the setting. It needs an SSE-S3 key on the S3 gateway (s3.sse.kek or s3.sse.key in security.toml; weed mini creates one); without a key, CreateVectorBucket is refused with AES256 encryption needs an SSE-S3 key on the S3 gateway: set s3.sse.kek or s3.sse.key in security.toml. aws:kms is rejected with unsupported sseType "aws:kms".
• A vector bucket is a table bucket with format LANCE, and its objects are visible through the S3 API under vector/<index>/.

Indexes

s3v.create_index(
    vectorBucketName="media", indexName="movies",
    dataType="float32", dimension=384, distanceMetric="cosine",
    metadataConfiguration={"nonFilterableMetadataKeys": ["description"]},
)

• Names: lowercase letters, digits, - and _, starting with a letter or digit. The AWS CLI and SDKs require 3 to 63 characters before sending.
• dataType: float32 only (only float32 vectors are supported).
• dimension: 1 to 4096 (dimension must be 1..=4096).
• nonFilterableMetadataKeys: those keys are stored and returned, but a filter that names one is rejected with metadata key "description" is not filterable.
• Deleting an index removes its vectors. An index created again under the same name starts empty and may have a different dimension or metric.

Tags

s3v.tag_resource(resourceArn=bucket_arn, tags={"team": "search", "env": "dev"})
s3v.untag_resource(resourceArn=bucket_arn, tagKeys=["env"])
s3v.list_tags_for_resource(resourceArn=bucket_arn)["tags"]  # {"team": "search"}

• resourceArn is a vector bucket or index ARN. An index has its own tags and does not inherit the bucket's.
• CreateVectorBucket and CreateIndex accept tags too.
• A resource can carry at most 50 tags, counting the ones it already has. A TagResource that would exceed that is rejected with ValidationException.
• Keys are 1 to 128 characters and values up to 256, made of letters, digits, spaces and _ . : / = + - @.
• TagResource overwrites the value of a key that is already set. UntagResource ignores keys that are not set.
• A missing bucket or index gets NotFoundException.
• A vector bucket policy can allow or deny requests by these tags. See S3 Vector Bucket Security#tag-conditions.

Distance metrics

The metric is fixed when the index is created and used by every query; it is returned by GetIndex and in each QueryVectors response as distanceMetric. The metric name is not case-sensitive.

distanceMetric	distance in results	Example: query [1, 0]
euclidean (default when omitted)	Euclidean distance	[3, 4] from [0, 0] → 5.0
cosine	1 − cosine similarity	[10, 0] → 0.0
dotproduct	1 − dot product	[2, 0] → -1.0, [1, 1] → 0.0, [-1, 0] → 2.0

Results are ordered by ascending distance. dotproduct is a SeaweedFS addition; AWS offers euclidean and cosine. Any other name is rejected with unknown distanceMetric.

Writing vectors

s3v.put_vectors(vectorBucketName="media", indexName="movies", vectors=[
    {"key": "alien", "data": {"float32": [0.2, 0.8, 0.1]},
     "metadata": {"genre": "scifi", "year": 1979, "cast": {"lead": "Weaver"}, "tags": ["space", "horror"]}},
])

• PutVectors is an upsert by key: writing an existing key replaces its vector and metadata.
• A batch is all or nothing: 1 to 500 vectors, and if any vector has the wrong dimension, or a key appears twice in the batch, the whole request fails with ValidationException and nothing is written.
• Metadata is any JSON object: strings, numbers, booleans, arrays and nested objects come back unchanged. Keys may contain characters such as '.
• DeleteVectors takes 1 to 500 keys.
• The request body is limited to 32 MiB. A larger request is refused with ValidationException: request body exceeds 32 MiB, with or without Content-Length. In our test 500 vectors of 4096 dimensions made bodies of 9.8, 19.6 and 25.4 MiB depending on the digits per value, all accepted; with 13 digits per value the body was 33.2 MiB and was refused.

Reading vectors

• GetVectors takes 1 to 100 keys; keys that do not exist are left out of the result. Data and metadata are returned only when returnData / returnMetadata are set.
• ListVectors pages through every vector: 500 per page by default, maxResults up to 1000, continued with nextToken. An invalid nextToken is rejected.
• Parallel listing: pass segmentCount (up to 16) and a segmentIndex from 0 to segmentCount - 1 to list one disjoint part of the index; the parts together hold every vector exactly once. segmentIndex without segmentCount is rejected.

Querying

s3v.query_vectors(
    vectorBucketName="media", indexName="movies",
    queryVector={"float32": [0.15, 0.85, 0.15]}, topK=10,
    filter={"$and": [{"genre": "scifi"}, {"year": {"$gte": 1978}}]},
    returnDistance=True, returnMetadata=True,
)

• topK: 1 to 10000. topK larger than the index returns every vector, as long as no vector index is built (see the last point).
• The query vector must have the index's dimension.
• distance is in the result only with returnDistance, and metadata only with returnMetadata.
• Search is exact until a vector index is built: without one, every query compares against every vector in the index. Once the Lance Maintenance Worker has built an approximate index (from 50,000 vectors by default), a query searches part of it (see S3 Vector Bucket#how-it-works) and returns only what it finds there, so a topK near the size of the index, or a filter that few vectors match, can return fewer results than exist. In a test with 4,000 vectors in 63 partitions, topK 4000 returned 2,173 vectors, and a filter that 400 vectors matched returned 210 of them for topK 500.

Metadata filters

Filter	Matches
{"genre": "scifi"}	genre equals "scifi", or is an array that holds "scifi"
{"genre": {"$eq": "scifi"}}, {"$ne": ...}	equal / not equal; for an array, holds / does not hold the value
{"year": {"$gt": 1977}}, $gte, $lt, $lte	numeric comparison; the operand must be a number ($gt expects a number)
{"genre": {"$in": ["drama", "western"]}}, $nin	equal to (for an array, holds) one of / none of the values
{"tags": {"$exists": true}}	the key is present (or absent, with false)
{"$and": [ ... ]}, {"$or": [ ... ]}	all / any of the sub-filters
{"genre": "scifi", "year": 2014}	several keys in one object: all must match

Behaviour to know:

• Arrays match by their elements. With "tags": ["space", "classic"], {"tags": "space"}, {"tags": {"$eq": "classic"}} and {"tags": {"$in": ["prison", "space"]}} all match. $ne and $nin are their opposites, so {"tags": {"$ne": "space"}} does not match. $gt, $gte, $lt and $lte do not look inside arrays: {"scores": {"$gt": 2}} does not match "scores": [1, 2.5].
• Types are compared strictly, inside arrays too. 1, "1", true and "true" are four different values: {"scores": 1} matches [1, 2.5] but not ["1", "2.5"], and {"flags": true} matches [true] but not ["true"].
• Nested objects are not searched. A dotted path such as {"info.lang": "fr"} matches nothing, and {"info": {"lang": "fr"}} is rejected with unsupported operator lang.
• Array matching needs a gateway and a weed-vector of the same release. With an older weed-vector, a number or boolean can also match a string element with the same text.
• An unknown operator is rejected (unsupported operator $regex), and so is a filter on a non-filterable key.

Errors

Error	When
ValidationException (400)	invalid parameters, limits exceeded (including a body over 32 MiB), invalid policy
AccessDeniedException (403)	not allowed; see S3 Vector Bucket Security
AccessDenied (403)	the credential is unknown or the signature is wrong
NotFoundException (404)	bucket, index or policy does not exist
ConflictException (409)	bucket or index already exists; bucket still has indexes
ServiceUnavailableException (503)	no weed-vector engine is registered
InternalFailure (500)	an engine error, with the message internal error, see the S3 gateway log; the gateway logs the cause (see S3 Vector Bucket#troubleshooting)

Introduction

• Quick Start with weed mini
• Simplest S3 Bucket and User Setup
• Components
• Blob Store Architecture
• Getting Started
• Production Setup
• A typical step‐by‐step example
• Benchmarks
• FAQ
• Applications

API

• Master Server API
• Volume Server API
• Filer Server API
• Client Libraries
• SeaweedFS Java Client

Configuration

• Replication
• Store file with a Time To Live
• Failover Master Server
• Erasure coding for warm storage
• EC Bitrot Detection
• Server Startup via Systemd
• Environment Variables

Filer

• Filer Setup
• Directories and Files
• File Operations Quick Reference
• Data Structure for Large Files
• Filer Data Encryption
• Filer Commands and Operations
• Filer JWT Use
• TUS Resumable Uploads

Filer Stores

• Filer Cassandra Setup
• Filer Redis Setup
• Super Large Directories
• Path-Specific Filer Store
• Choosing a Filer Store
• Customize Filer Store

Management

• Admin UI
• Admin UI OIDC
• Worker
• Plugin Worker Scheduling

Advanced Filer Configurations

• Migrate to Filer Store
• Migrate Between Filer Stores
• Add New Filer Store
• Filer Store Replication
• Filer Active Active cross cluster continuous synchronization
• Filer as a Key-Large-Value Store
• Path Specific Configuration
• Filer Change Data Capture
• Filer Operation Serialization

FUSE Mount

• Mount on Windows
• FIO benchmark
• fstab and systemd mount
• POSIX Compliance
• Distributed POSIX Locks
• P2P reading in weed mount
• Mount over the Internet

WebDAV

SFTP Server

Cloud Drive

• Cloud Drive Benefits
• Cloud Drive Architecture
• Configure Remote Storage
• Azure Blob Storage Authentication
• Mount Remote Storage
• Cache Remote Storage
• Cloud Drive Quick Setup
• Gateway to Remote Object Storage

AWS S3 API

• Amazon S3 API
• Supported APIs vs Minio
• S3 Lifecycle
  • S3 Lifecycle Recipes
  • S3 Lifecycle Operator Guide
  • S3 Lifecycle Monitoring
  • S3 Lifecycle Troubleshooting
  • S3 Lifecycle Architecture
• S3 Lifecycle vs Volume TTL
• S3 Conditional Operations
• S3 CORS
• S3 Object Lock and Retention
• S3 Object Versioning
• S3 RenameObject
• S3 API Benchmark
• S3 API FAQ
• S3 Bucket Quota
• S3 Rate Limiting
• S3 API Audit log
• S3 Nginx Proxy
• Docker Compose for S3
• Image Gateway

S3 Table Bucket

• S3 Table Bucket
• S3 Table Bucket Commands
• S3 Tables Security
• SeaweedFS Iceberg Catalog
• Iceberg REST Catalog API
• Iceberg Table Maintenance
• SeaweedFS Lance Catalog
• Lance Maintenance Worker

S3 Vector Bucket

• S3 Vector Bucket
• S3 Vectors API
• S3 Vector Bucket Security
• S3 Vector Bucket Performance

Iceberg Integrations

• Spark Iceberg Integration
• Trino Iceberg Integration
• Dremio Iceberg Integration
• DuckDB Iceberg Integration
• Doris Iceberg Integration
• RisingWave Iceberg Integration
• Lakekeeper Iceberg Integration
• OLake Go Iceberg Integration

Lance Integrations

• LanceDB Integration
• Spark Lance Integration
• DuckDB Lance Integration

S3 Authentication & IAM

• S3 Configuration - Start Here
• S3 Credentials (-s3.config)
• OIDC Integration (-s3.iam.config)
• Kubernetes ServiceAccount Authentication (IRSA-style)
• S3 Policy Variables
• S3 Policy Conditions
• S3 Bucket Policies
• Amazon IAM API
• AWS IAM CLI
• weed shell - Shell IAM Commands

Server-Side Encryption

• Server-Side Encryption
• Server-Side Encryption SSE-KMS
• Server-Side Encryption SSE-C

S3 Client Tools

• AWS CLI with SeaweedFS
• s3cmd with SeaweedFS
• rclone with SeaweedFS
• restic with SeaweedFS
• nodejs with Seaweed S3

Machine Learning

• TensorFlow with SeaweedFS
• Distributing AI Model Files for Multi-GPU Loading

HDFS

• Hadoop Compatible File System
• run Spark on SeaweedFS
• run HBase on SeaweedFS
• Run Trino on SeaweedFS
• Hadoop Benchmark
• HDFS via S3 connector

Replication and Backup

• Async Replication to another Filer [Deprecated]
• Async Backup
• Async Filer Metadata Backup
• Async Replication to Cloud [Deprecated]
• Kubernetes Backups and Recovery with K8up

Metadata Change Events

• Filer Metadata Events
• Filer Notification Webhook

Messaging

• Structured Data Lake with SMQ and SQL
• Seaweed Message Queue
• SQL Queries on Message Queue
• SQL Quick Reference
• PostgreSQL-compatible Server weed db
• Pub-Sub to SMQ to SQL
• Kafka to Kafka Gateway to SMQ to SQL

Use Cases

• Use Cases
• Actual Users

Operations

• System Metrics
• weed shell
• Data Backup
• Deployment to Kubernetes and Minikube
• Helm Chart Recipes
• Deployment with seaweed-up
  • seaweed-up Command Reference
  • Cluster Plan Workflow
  • Cluster Plan Inventory Reference
  • Cluster Plan Day 2 Operations

Rust Volume Server

• Rust Volume Server

Advanced

• Large File Handling
• Optimization
• Optimization for Many Small Buckets
• Volume Management
• Tiered Storage
• Cloud Tier
• Cloud Monitoring
• Load Command Line Options from a file
• SRV Service Discovery
• Volume Files Structure

Security

• Security Overview
• Security Configuration
• Cryptography and FIPS Compliance
• Run Blob Storage on Public Internet

Misc Use Case Examples

• UrBackup with SeaweedFS
• Docker Image Registry with SeaweedFS
• SeaweedFS in Docker Swarm
• Words from SeaweedFS Users
• Independent Benchmarks
• Hardware