Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-11-09 21:38:19 +01:00
Code Issues Packages Projects Releases Wiki Activity

docs: showcase vector buckets and cluster features (#12513)

Browse Source 
* docs: showcase vector buckets and cluster features

* docs: include built-in Iceberg catalog in table bucket overview

* docs: keep feature diagram at cluster level
This commit is contained in:
Chris Lu authored and GitHub committed 2026-11-09 08:52:12 -08:00
1 parent 643784df0d
commit 620a1c3f39
2 files changed
+106 -3
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+23 -3
View File
@@ -17,7 +17,9 @@ SeaweedFS is a simple and highly scalable distributed file system. There are two
1. to store billions of files!
2. to serve the files fast!
One `weed` binary serves an S3 object store, a POSIX file system, and a lakehouse with S3 Tables, all over the same data. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently. Both read and write operations have O(1) complexity and can run at the full speed supported by the underlying hardware.
One SeaweedFS cluster serves an S3 object store, a POSIX file system, a lakehouse with S3 Tables and a built-in Iceberg REST Catalog, and [vector search with S3 Vector Buckets](#vector-search-with-s3-vector-buckets), all on shared storage. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently. Blob reads and writes have O(1) complexity and can run at the full speed supported by the underlying hardware.
![SeaweedFS feature overview: S3 objects, POSIX files, table buckets with an Iceberg REST Catalog and Lance namespace, and S3 Vector Buckets share one cluster of masters, filers, and volume servers, with cloud caching, tiering, and cross-cluster replication.](note/seaweedfs-features.svg)
- [Download Binaries for different platforms](https://github.com/seaweedfs/seaweedfs/releases/latest)
- [Wiki Documentation](https://github.com/seaweedfs/seaweedfs/wiki)
@@ -40,6 +42,7 @@ Table of Contents
* [Scalable](#scalable)
* [The most complete S3 API](#the-most-complete-s3-api)
* [A data warehouse with S3 Tables](#a-data-warehouse-with-s3-tables)
* [Vector search with S3 Vector Buckets](#vector-search-with-s3-vector-buckets)
* [A fast cache for cloud storage](#a-fast-cache-for-cloud-storage)
* [Active-active replication and more](#active-active-replication-and-more)
* [Architecture](#architecture)
@@ -204,12 +207,13 @@ On one laptop, [`weed benchmark`][Benchmarks] writes 1KB files at 15,700 per sec
## The most complete S3 API ##
The S3 gateway implements the object, bucket, S3 Tables, IAM, and STS APIs on one endpoint, so the AWS SDKs and CLI, rclone, restic, Spark, and Trino work unchanged.
The S3 gateway implements the object, bucket, S3 Tables, S3 Vectors, IAM, and STS APIs on one endpoint, so the AWS SDKs and CLI, rclone, restic, Spark, and Trino work unchanged. S3 Vectors uses the `weed-vector` service described [below](#vector-search-with-s3-vector-buckets).
| API | Operations |
| --- | --- |
| S3 bucket and object | 73 |
| S3 Tables | 36 |
| S3 Vectors | 19 |
| IAM | 39 |
| STS | 5 |
@@ -223,7 +227,7 @@ The full operation list is in [Amazon S3 API][AmazonS3API], and [Supported APIs
## A data warehouse with S3 Tables ##
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. The S3 Tables API and Iceberg REST Catalog share the same namespaces, table metadata, and permissions. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials. [OLake Go][OLake] is a high-performance, open-source EL (Extract–Load) platform that bridges operational databases and open lakehouse storage - writing Apache Iceberg tables or Parquet files on Amazon S3.
* [Automated table maintenance][IcebergMaintenance]: compaction, snapshot expiration, orphan file removal, and manifest rewriting, configured per bucket or table through the S3 Tables maintenance APIs, and the same for [Lance][LanceMaintenance].
@@ -232,6 +236,22 @@ SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold A
`S3_TABLE_BUCKET=warehouse ./weed mini -dir=./data` brings the whole stack up on a laptop.
## Vector search with S3 Vector Buckets ##
Keep embeddings alongside your objects and tables in the same SeaweedFS cluster. S3 Vector Buckets expose the AWS S3 Vectors API for semantic search, retrieval-augmented generation (RAG), and recommendations, using Lance tables on SeaweedFS storage.
* Create vector buckets and indexes; insert, retrieve, list, and delete vectors with the AWS SDKs or CLI pointed at your S3 endpoint.
* Find the nearest vectors with cosine or Euclidean distance, filter by metadata, and return distances and metadata with the results.
* Control access with `s3vectors:*` IAM actions and vector bucket policies; tag buckets and indexes for organization and access control.
Run the separate `weed-vector` binary alongside the cluster. For the local cluster started with `weed mini` above:
```bash
weed-vector --masters=localhost:9333
```
It registers with the master and the S3 gateway discovers it automatically. An AWS S3 Vectors client uses the same S3 endpoint at http://localhost:8333. The vector engine stores its data through the cluster's filers and volume servers, so embeddings share the storage layer's replication and erasure coding. See the [AWS SDK examples](test/s3vectors/s3vectors_test.py) for bucket, index, and query operations.
## A fast cache for cloud storage ##
[Cloud Drive][CloudDrive] mounts a bucket from S3, Google Cloud Storage, Azure, Backblaze B2, Wasabi, Storj, or any S3-compatible store into SeaweedFS and serves it at local speed:
note/seaweedfs-features.svg
+83
View File
@@ -0,0 +1,83 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1080" height="650" viewBox="0 0 1080 650" role="img" aria-labelledby="title description">
<title id="title">SeaweedFS: one cluster for objects, files, tables, and vectors</title>
<desc id="description">S3 object storage, a POSIX file system, table buckets with a built-in Iceberg REST Catalog and Lance namespace, and S3 Vector Buckets share one SeaweedFS storage cluster. Masters coordinate the cluster, filers manage namespaces and metadata, and volume servers store replicated or erasure-coded blobs. Cloud caching, cloud tiering, and cross-cluster replication extend the cluster.</desc>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M 0 0 L 10 5 L 0 10 z" fill="#75928e"/>
</marker>
</defs>
<rect width="1080" height="650" rx="20" fill="#f5f9f8"/>
<g font-family="Arial, Helvetica, sans-serif">
<text x="40" y="43" fill="#0f766e" font-size="15" font-weight="700" letter-spacing="2">SEAWEEDFS</text>
<text x="40" y="86" fill="#163832" font-size="32" font-weight="700">One cluster. Objects, files, tables, vectors.</text>
<!-- Four ways to use the same storage cluster. -->
<g>
<rect x="40" y="116" width="238" height="184" rx="14" fill="#fff" stroke="#cbded8"/>
<g transform="translate(62 136)" fill="none" stroke="#187b67" stroke-width="2.2" stroke-linejoin="round">
<path d="m0 10 18-9 18 9-18 9z M0 10v21l18 9 18-9V10 M18 19v21"/>
</g>
<text x="62" y="207" fill="#163832" font-size="22" font-weight="700">Object storage</text>
<text x="62" y="237" fill="#48635e" font-size="18">S3 API</text>
<text x="62" y="261" fill="#48635e" font-size="17">Apps, backups, media</text>
</g>
<g>
<rect x="294" y="116" width="238" height="184" rx="14" fill="#fff" stroke="#cbded8"/>
<g transform="translate(316 140)" fill="none" stroke="#2676a3" stroke-width="2.2" stroke-linejoin="round">
<path d="M0 7V2h15l6 6h17v27H0V7z M0 12h38"/>
</g>
<text x="316" y="207" fill="#163832" font-size="22" font-weight="700">File system</text>
<text x="316" y="237" fill="#48635e" font-size="18">POSIX / FUSE</text>
<text x="316" y="261" fill="#48635e" font-size="17">WebDAV, SFTP, HDFS</text>
</g>
<g>
<rect x="548" y="116" width="238" height="184" rx="14" fill="#fff" stroke="#cbded8"/>
<g transform="translate(570 138)" fill="none" stroke="#7a62ab" stroke-width="2.2" stroke-linejoin="round">
<rect x="0" y="0" width="38" height="36" rx="3"/>
<path d="M0 12h38 M0 24h38 M13 12v24 M26 12v24"/>
</g>
<text x="570" y="207" fill="#163832" font-size="22" font-weight="700">Table buckets</text>
<text x="570" y="237" fill="#48635e" font-size="18">S3 Tables</text>
<text x="570" y="261" fill="#48635e" font-size="17">Iceberg REST Catalog</text>
<text x="570" y="285" fill="#48635e" font-size="17">Lance namespace</text>
</g>
<g>
<rect x="802" y="116" width="238" height="184" rx="14" fill="#e4f5ee" stroke="#187b67" stroke-width="1.5"/>
<g transform="translate(824 138)" fill="none" stroke="#187b67" stroke-width="2.2">
<path d="m5 29 13-22 18 22H5z M18 7v30 M5 29l13 8 18-8"/>
<g fill="#e4f5ee">
<circle cx="5" cy="29" r="4"/>
<circle cx="18" cy="7" r="4"/>
<circle cx="36" cy="29" r="4"/>
<circle cx="18" cy="37" r="4"/>
</g>
</g>
<text x="824" y="207" fill="#163832" font-size="22" font-weight="700">Vector buckets</text>
<text x="824" y="237" fill="#266354" font-size="18">S3 Vectors</text>
<text x="824" y="261" fill="#266354" font-size="17">Search, RAG, embeddings</text>
</g>
<!-- A shared connection expresses capabilities, not request routing. -->
<path d="M159 300v28h762v-28 M413 300v28 M667 300v28 M540 328v30" fill="none" stroke="#75928e" stroke-width="2" marker-end="url(#arrow)"/>
<rect x="40" y="362" width="1000" height="174" rx="16" fill="#163e35"/>
<text x="540" y="400" text-anchor="middle" fill="#fff" font-size="25" font-weight="700">One shared storage cluster</text>
<g fill="#fff" font-size="20" font-weight="700" text-anchor="middle">
<text x="213" y="445">Masters</text>
<text x="540" y="445">Filers</text>
<text x="867" y="445">Volume servers</text>
</g>
<path d="M376 425v51 M704 425v51" stroke="#497266"/>
<g fill="#cfe4dc" font-size="17" text-anchor="middle">
<text x="213" y="473">Coordination &amp; placement</text>
<text x="540" y="473">Namespaces &amp; metadata</text>
<text x="867" y="473">Replicated or erasure-coded blobs</text>
</g>
<text x="540" y="511" text-anchor="middle" fill="#cfe4dc" font-size="17">Add volume servers to grow capacity</text>
<path d="M284 536v23 M796 536v23" stroke="#75928e" stroke-width="2" stroke-dasharray="5 5"/>
<rect x="40" y="560" width="488" height="58" rx="12" fill="#fff" stroke="#cbded8"/>
<rect x="552" y="560" width="488" height="58" rx="12" fill="#fff" stroke="#cbded8"/>
<text x="284" y="596" text-anchor="middle" fill="#305c50" font-size="19">Cloud caching &amp; tiering</text>
<text x="796" y="596" text-anchor="middle" fill="#305c50" font-size="19">Cross-cluster replication</text>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 5.3 KiB