add Blob Store Architecture and Helm Chart Recipes

Blob Store Architecture takes the master and volume server walkthrough
that used to live in the README: assign, write, read, replication and
data center placement, then how the master tracks volumes instead of
files, and the disk notes.

Helm Chart Recipes gives copy-paste values files for a three-node S3
cluster, a development cluster, a lakehouse with the Iceberg catalog
exposed, filer metadata on PostgreSQL, and node-local disks. Every
recipe renders against the current chart.

Claude-Session: https://claude.ai/code/session_014apMEkkquAtAYp89paTkAT
Chris Lu committed 2026-09-02 23:12:55 -07:00
1 parent bb04d8e2cf
commit 5eb8ff8c1a
3 files changed
+414

No files matched your search

+205
@@ -0,0 +1,205 @@
The blob store is the bottom layer of SeaweedFS: master servers and volume servers. Everything else, the [[Filer|Directories and Files]], the [[S3 API|Amazon S3 API]], [[FUSE Mount]], and the [[S3 Table Bucket]] catalogs, is built on top of it. You can also use it directly as a key-to-blob store, which is what this page walks through first, before explaining how the master assigns and locates volumes.
## Using the blob store directly
By default, the master node runs on port 9333, and the volume nodes run on port 8080.
Let's start one master node, and two volume nodes on port 8080 and 8081. Ideally, they should be started from different machines. We'll use localhost as an example.
SeaweedFS uses HTTP REST operations to read, write, and delete. The responses are in JSON or JSONP format.
### Start Master Server
```
> ./weed master
```
### Start Volume Servers
```
> weed volume -dir="/tmp/data1" -max=5 -master="localhost:9333" -port=8080 &
> weed volume -dir="/tmp/data2" -max=10 -master="localhost:9333" -port=8081 &
```
### Write A Blob
A blob, also referred as a needle, a chunk, or mistakenly as a file, is just a byte array. It can have attributes, such as name, mime type, create or update time, etc. But basically it is just a byte array of a relatively small size, such as 2 MB ~ 64 MB. The size is not fixed.
To upload a blob: first, send a HTTP POST, PUT, or GET request to `/dir/assign` to get an `fid` and a volume server URL:
```
> curl http://localhost:9333/dir/assign
{"count":1,"fid":"3,01637037d6","url":"127.0.0.1:8080","publicUrl":"localhost:8080"}
```
Second, to store the blob content, send a HTTP multi-part POST request to `url + '/' + fid` from the response:
```
> curl -F file=@/home/chris/myphoto.jpg http://127.0.0.1:8080/3,01637037d6
{"name":"myphoto.jpg","size":43234,"eTag":"1cc0118e"}
```
To update, send another POST request with updated blob content.
For deletion, send an HTTP DELETE request to the same `url + '/' + fid` URL:
```
> curl -X DELETE http://127.0.0.1:8080/3,01637037d6
```
### Save Blob Id
Now, you can save the `fid`, 3,01637037d6 in this case, to a database field.
The number 3 at the start represents a volume id. After the comma, it's one file key, 01, and a file cookie, 637037d6.
The volume id is an unsigned 32-bit integer. The file key is an unsigned 64-bit integer. The file cookie is an unsigned 32-bit integer, used to prevent URL guessing.
The file key and file cookie are both coded in hex. You can store the <volume id, file key, file cookie> tuple in your own format, or simply store the `fid` as a string.
If stored as a string, in theory, you would need 8+1+16+8=33 bytes. A char(33) would be enough, if not more than enough, since most uses will not need 2^32 volumes.
If space is really a concern, you can store the file id in the binary format. You would need one 4-byte integer for volume id, 8-byte long number for file key, and a 4-byte integer for the file cookie. So 16 bytes are more than enough.
### Read a Blob
Here is an example of how to render the URL.
First look up the volume server's URLs by the file's volumeId:
```
> curl http://localhost:9333/dir/lookup?volumeId=3
{"volumeId":"3","locations":[{"publicUrl":"localhost:8080","url":"localhost:8080"}]}
```
Since (usually) there are not too many volume servers, and volumes don't move often, you can cache the results most of the time. Depending on the replication type, one volume can have multiple replica locations. Just randomly pick one location to read.
Now you can take the public URL, render the URL or directly read from the volume server via URL:
```
http://localhost:8080/3,01637037d6.jpg
```
Notice we add a file extension ".jpg" here. It's optional and just one way for the client to specify the file content type.
If you want a nicer URL, you can use one of these alternative URL formats:
```
http://localhost:8080/3/01637037d6/my_preferred_name.jpg
http://localhost:8080/3/01637037d6.jpg
http://localhost:8080/3,01637037d6.jpg
http://localhost:8080/3/01637037d6
http://localhost:8080/3,01637037d6
```
If you want to get a scaled version of an image, you can add some params:
```
http://localhost:8080/3/01637037d6.jpg?height=200&width=200
http://localhost:8080/3/01637037d6.jpg?height=200&width=200&mode=fit
http://localhost:8080/3/01637037d6.jpg?height=200&width=200&mode=fill
```
### Rack-Aware and Data Center-Aware Replication
SeaweedFS applies the replication strategy at a volume level. So, when you are getting a blob id, you can specify the replication strategy. For example:
```
curl http://localhost:9333/dir/assign?replication=001
```
The replication parameter options are:
```
000: no replication
001: replicate once on the same rack
010: replicate once on a different rack, but same data center
100: replicate once on a different data center
200: replicate twice on two different data center
110: replicate once on a different rack, and once on a different data center
```
More details about replication can be found in [[Replication]].
You can also set the default replication strategy when starting the master server.
### Allocate Blob Key on Specific Data Center
Volume servers can be started with a specific data center name:
```
weed volume -dir=/tmp/1 -port=8080 -dataCenter=dc1
weed volume -dir=/tmp/2 -port=8081 -dataCenter=dc2
```
When requesting a blob key, an optional "dataCenter" parameter can limit the assigned volume to the specific data center. For example, this specifies that the assigned volume should be limited to 'dc1':
```
http://localhost:9333/dir/assign?dataCenter=dc1
```
### Other Features
* [[No Single Point of Failure|Failover Master Server]]
* [Insert with your own keys](Optimization#insert-with-your-own-keys)
* [Chunking large files](Optimization#upload-large-files)
* [Collection as a Simple Name Space](Optimization#collection-as-a-simple-name-space)
The full HTTP surface is in [[Master Server API]] and [[Volume Server API]].
## How the master and volume servers work
Usually distributed file systems split each file into chunks. A central server keeps a mapping of filenames to chunks, and also which chunks each chunk server has.
The main drawback is that the central server can't handle many small files efficiently, and since all read requests need to go through the central master, so it might not scale well for many concurrent users.
Instead of managing chunks, SeaweedFS manages data volumes in the master server. Each data volume is 32GB in size, and can hold a lot of blobs. And each storage node can have many data volumes. So the master node only needs to store the metadata about the volumes, which is a fairly small amount of data and is generally stable.
The actual blob metadata, which are the blob volume, offset, and size, is stored in each volume on volume servers. Since each volume server only manages metadata of blobs on its own disk, with only 16 bytes for each blob, all access can read the metadata just from memory and only needs one disk operation to actually read file data.
For comparison, consider that an xfs inode structure in Linux is 536 bytes.
### Master Server and Volume Server
The architecture is fairly simple. The actual data is stored in volumes on storage nodes. One volume server can have multiple volumes, and can both support read and write access with basic authentication.
All volumes are managed by a master server. The master server contains the volume id to volume server mapping. This is fairly static information, and can be easily cached.
On each write request, the master server also generates a file key, which is a growing 64-bit unsigned integer. Since write requests are not generally as frequent as read requests, one master server should be able to handle the concurrency well.
### Write and Read files
When a client sends a write request, the master server returns (volume id, file key, file cookie, volume node URL) for the blob. The client then contacts the volume node and POSTs the blob content.
When a client needs to read a blob based on (volume id, file key, file cookie), it asks the master server by the volume id for the (volume node URL, volume node public URL), or retrieves this from a cache. Then the client can GET the content, or just render the URL on web pages and let browsers fetch the content.
### Saving memory
All blob metadata stored on a volume server is readable from memory without disk access. Each file takes just a 16-byte map entry of <64bit key, 32bit offset, 32bit size>. Of course, each map entry has its own space cost for the map. But usually the disk space runs out before the memory does.
### Tiered Storage to the cloud
The local volume servers are much faster, while cloud storages have elastic capacity and are actually more cost-efficient if not accessed often (usually free to upload, but relatively costly to access). With the append-only structure and O(1) access time, SeaweedFS can take advantage of both local and cloud storage by offloading the warm data to the cloud.
Usually hot data are fresh and warm data are old. SeaweedFS puts the newly created volumes on local servers, and optionally upload the older volumes on the cloud. If the older data are accessed less often, this literally gives you unlimited capacity with limited local servers, and still fast for new data.
With the O(1) access time, the network latency cost is kept at minimum.
If the hot/warm data is split as 20/80, with 20 servers, you can achieve storage capacity of 100 servers. That's a cost saving of 80%! Or you can repurpose the 80 servers to store new data also, and get 5X storage throughput.
See [[Cloud Tier]] for the setup.
### The Filer on top
Built on top of the blob store, the SeaweedFS Filer adds directory structure to create a file system. The directory structure is an interface that is implemented in many key-value stores or databases, see [[Filer Stores]].
The content of a file is mapped to one or many blobs, distributed to multiple volumes on multiple volume servers.
## Disks
### Hard Drive Performance
When testing read performance on SeaweedFS, it basically becomes a performance test of your hard drive's random read speed. Hard drives usually get 100MB/s~200MB/s.
### Solid State Disk
To modify or delete small files, SSD must delete a whole block at a time, and move content in existing blocks to a new block. SSD is fast when brand new, but will get fragmented over time and you have to garbage collect, compacting blocks. SeaweedFS is friendly to SSD since it is append-only. Deletion and compaction are done on volume level in the background, not slowing reading and not causing fragmentation.
+207
@@ -0,0 +1,207 @@
Copy-paste `values.yaml` files for the shapes people ask about most. Every recipe renders against the current chart. Install any of them with:
```bash
helm repo add seaweedfs https://seaweedfs.github.io/seaweedfs/helm
helm install seaweedfs seaweedfs/seaweedfs -n seaweedfs --create-namespace -f values.yaml
```
The chart README in [k8s/charts/seaweedfs](https://github.com/seaweedfs/seaweedfs/tree/master/k8s/charts/seaweedfs) lists every key. For the operator instead of the chart, see [[Deployment to Kubernetes and Minikube]].
## Three-node S3 cluster
Three masters for Raft failover, three volume servers, two filers, two S3 gateways, every write stored twice, and a bucket ready to use.
```yaml
global:
seaweedfs:
enableReplication: true
replicationPlacement: "001" # one extra copy on another server; "002" for two
master:
replicas: 3
data:
type: persistentVolumeClaim
storageClass: local-path
size: 1Gi
volume:
replicas: 3 # at least 1 + the sum of the replication digits
dataDirs:
- name: data
type: persistentVolumeClaim
storageClass: local-path
size: 500Gi
maxVolumes: 0 # size the volume count from the disk
filer:
replicas: 2
enablePVC: true
storageClass: local-path
storage: 20Gi
s3:
enabled: true
replicas: 2
enableAuth: true
credentials:
admin:
accessKey: admin
secretKey: change-me
createBuckets:
- name: app-storage
```
How the pieces fit:
* `replicationPlacement` is three digits, `XYZ`: X extra copies in other data centers, Y in other racks of the same data center, Z on other servers of the same rack. Total copies are 1 + X + Y + Z, and you need at least that many volume servers in the matching places. Without rack or data center labels every volume server is in the same rack, so `001` needs two volume servers and `002` needs three. `enableReplication` sets both `master.defaultReplication` and `filer.defaultReplicaPlacement` to this value. See [[Replication]].
* `volume.replicas` is the number of volume server pods, one per node with the chart's default anti-affinity. It is independent of the replication digits except for the minimum above.
* `maxVolumes: 0` lets each volume server compute how many volumes fit on its disk. The chart's master defaults to 1000MB volumes (`master.volumeSizeLimitMB`); raise it to `30000` for large disks with few buckets, keep it small when you expect many buckets, since each bucket grows its own volumes.
* Master data on a claim keeps the cluster identity when a master pod is rescheduled. The chart defaults to `hostPath`, which does not follow the pod.
* Two filers on the default LevelDB store [[sync with each other|Filer Store Replication]]. For three or more filers, or a lot of metadata, put the store in a database (next recipes).
* `s3.enableAuth` with `s3.credentials` writes the identities into the S3 config secret; the `read` identity gets random keys unless you set it too. `createBuckets` runs once as a post-install hook.
The endpoint inside the cluster is `seaweedfs-s3.seaweedfs.svc:8333`. From a laptop:
```bash
kubectl -n seaweedfs port-forward svc/seaweedfs-s3 8333:8333
AWS_ACCESS_KEY_ID=admin AWS_SECRET_ACCESS_KEY=change-me \
aws --endpoint-url http://localhost:8333 s3 ls s3://app-storage/
```
Expose it with `s3.ingress` or a `LoadBalancer` under `s3.service.type`. Grow the cluster by raising `volume.replicas`; new volume servers take new writes and nothing moves until you run `volume.balance` in [[weed shell]].
## Single node for development
One of everything, on the cluster's default storage class, no authentication.
```yaml
master:
data:
type: persistentVolumeClaim
size: 1Gi
volume:
dataDirs:
- name: data
type: persistentVolumeClaim
size: 20Gi
maxVolumes: 0
filer:
enablePVC: true
storage: 5Gi
s3:
enabled: true
createBuckets:
- name: test
```
For a laptop without Kubernetes, `weed mini` or the Docker image does the same in one process, see [[Quick Start with weed mini]].
## Lakehouse: S3 Tables with the Iceberg REST catalog
The three-node recipe plus the catalog port on the S3 gateway. Table buckets are created through the S3 Tables API rather than `createBuckets`.
```yaml
global:
seaweedfs:
enableReplication: true
replicationPlacement: "001"
master:
replicas: 3
data:
type: persistentVolumeClaim
storageClass: local-path
size: 1Gi
volume:
replicas: 3
dataDirs:
- name: data
type: persistentVolumeClaim
storageClass: local-path
size: 500Gi
maxVolumes: 0
filer:
replicas: 2
enablePVC: true
storageClass: local-path
storage: 20Gi
s3:
enabled: true
replicas: 2
enableAuth: true
icebergPort: 8181
credentials:
admin:
accessKey: admin
secretKey: change-me
```
Create a table bucket, then point engines at the S3 endpoint and the catalog:
```bash
export S3_ENDPOINT=http://seaweedfs-s3.seaweedfs.svc:8333
aws s3tables create-table-bucket --endpoint-url $S3_ENDPOINT --name warehouse
```
The Iceberg REST catalog is `http://seaweedfs-s3.seaweedfs.svc:8181`, or set `s3.icebergIngress` to reach it from outside. Engine setup is in [[Spark Iceberg Integration]], [[Trino Iceberg Integration]], [[DuckDB Iceberg Integration]], and the other pages under [[S3 Table Bucket]]. `s3.lancePort` (default 9101) serves the [[SeaweedFS Lance Catalog]] the same way.
## Filer metadata on PostgreSQL
For three or more filers, or when metadata outgrows a local LevelDB, keep it in a database the filers share. Any store from [[Filer Stores]] works the same way through `WEED_<STORE>_*` environment variables; this is `postgres2`.
```yaml
filer:
replicas: 3
extraEnvironmentVars:
WEED_LEVELDB2_ENABLED: "false"
WEED_POSTGRES2_ENABLED: "true"
WEED_POSTGRES2_HOSTNAME: "postgres.database.svc"
WEED_POSTGRES2_PORT: "5432"
WEED_POSTGRES2_DATABASE: "seaweedfs"
WEED_POSTGRES2_SSLMODE: "disable"
secretExtraEnvironmentVars:
WEED_POSTGRES2_USERNAME:
secretKeyRef:
name: postgres-credentials
key: username
WEED_POSTGRES2_PASSWORD:
secretKeyRef:
name: postgres-credentials
key: password
s3:
enabled: true
replicas: 3
```
The filer creates its table on first start. `WEED_LEVELDB2_ENABLED` must be turned off, since the chart enables it by default and two enabled stores is an error. The filers no longer need a claim, so `enablePVC` can stay off.
## Bare metal: node-local disks
Skip the storage class and write straight to each node's disks. One volume server per node, every listed directory becomes a `-dir` on that pod.
```yaml
volume:
replicas: 3
dataDirs:
- name: nvme0
type: hostPath
hostPathPrefix: /mnt/nvme0
maxVolumes: 0
- name: hdd0
type: hostPath
hostPathPrefix: /mnt/hdd0
maxVolumes: 0
```
The chart mounts `<hostPathPrefix>/object_store/` for each entry. Pin the pods with `volume.nodeSelector` if only some nodes have the disks. To steer buckets between disk types, tag the directories in the same order with `volume.extraArgs: ["-disk", "ssd,hdd"]`, see [[Tiered Storage]]. Master and filer data default to `hostPath` too; give them claims as in the recipes above, or accept that they stay on the node they started on.
## Cross-cluster replication
Two clusters from any recipe above can replicate to each other continuously with `weed filer.sync`, active-active or active-passive, from any pod or host that reaches both filers. The command, the operator sidecar form, and the topology rules are in [[Filer Active Active cross cluster continuous synchronization]].
+2
@@ -2,6 +2,7 @@
* [[Quick Start with weed mini]]
* [[Simplest S3 Bucket and User Setup]]
* [[Components]]
* [[Blob Store Architecture]]
* [[Getting Started]]
* [[Production Setup]]
* [[A typical step‐by‐step example]]
@@ -194,6 +195,7 @@
* [[weed shell]]
* [[Data Backup]]
* [[Deployment to Kubernetes and Minikube]]
* [[Helm Chart Recipes]]
* [[Deployment with seaweed-up]]
* [[seaweed-up Command Reference]]
* [[Cluster Plan Workflow]]