From 8d883e29677fff67b5c72470fcedaf5f75ff07bb Mon Sep 17 00:00:00 2001 From: "Dave St.Germain" Date: Thu, 8 Feb 2024 12:18:20 -0500 Subject: [PATCH 001/149] Added asyncio python client --- Client-Libraries.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Client-Libraries.md b/Client-Libraries.md index 828d44a..f63f6e3 100644 --- a/Client-Libraries.md +++ b/Client-Libraries.md @@ -25,6 +25,7 @@ Name | Author | Language [Python-weed](https://github.com/darkdarkfruit/python-weed) | Darkdarkfruit | Python [Django-weed](https://github.com/ProstoKSI/django-weed) | ProstoKSI | Python [Pyweed](https://github.com/utek/pyweed) | Utek | Python +[Aioseaweedfs](https://code.pobblelabs.org/fossil/aioseaweed) | davestgermain | Python [Scala SeaweedFS client](https://github.com/chiradip/WeedFsScalaClient) | Chiradip | Scala [Seaweedrb](https://github.com/jguest/seaweedrb) | John Guest | Ruby [SeaweedFs.NET](https://github.com/piechpatrick/SeaweedFs.Client) | piechpatrick | C# From 4dcff74014ac51abfbe7ed78795241aa7d665e59 Mon Sep 17 00:00:00 2001 From: jerebear12 <72420925+jerebear12@users.noreply.github.com> Date: Mon, 12 Feb 2024 12:43:56 -0600 Subject: [PATCH 002/149] Added cors.allowed_origins.values option to config example --- Security-Configuration.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/Security-Configuration.md b/Security-Configuration.md index 8b28d95..7e68c01 100644 --- a/Security-Configuration.md +++ b/Security-Configuration.md @@ -11,6 +11,11 @@ $ weed scaffold -config=security # /etc/seaweedfs/security.toml # this file is read by master, volume server, and filer +# comma separated origins allowed to make requests to the filer and s3 gateway. +# enter in this format: https://domain.com, or http://localhost:port +[cors.allowed_origins] +values = "*" + # this jwt signing key is read by master and volume server, and it is used for write operations: # - the Master server generates the JWT, which can be used to write a certain file on a volume server # - the Volume server validates the JWT on writing From a567d5a2a733c6cff659c38098dcee8b1e9098b3 Mon Sep 17 00:00:00 2001 From: Marcell Mars Date: Wed, 14 Feb 2024 16:20:29 +0100 Subject: [PATCH 003/149] adds https:// to Markdown link to ensure it navigates correctly and doesn't concatenate to the current page's URL. --- Run-Blob-Storage-on-Public-Internet.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Run-Blob-Storage-on-Public-Internet.md b/Run-Blob-Storage-on-Public-Internet.md index f9f3174..6b86c92 100644 --- a/Run-Blob-Storage-on-Public-Internet.md +++ b/Run-Blob-Storage-on-Public-Internet.md @@ -144,7 +144,7 @@ services: I haven't tested this, but this is how I would go about making it HA: - Run master server on 3 or 5 physical servers - Run caddy as a sidecar on every server that runs a master -- Install caddy from own Dockerfile, adding [redis plugin](github.com/gamalan/caddy-tlsredis) for distributed SSL cert storage and distributed locks for SSL cert issuing. +- Install caddy from own Dockerfile, adding [redis plugin](https://github.com/gamalan/caddy-tlsredis) for distributed SSL cert storage and distributed locks for SSL cert issuing. - Run redis in high-availability mode, for example by [following this docker swarm guide](https://medium.com/@emmano3h/redis-high-availability-with-docker-swarm-2142a4d80b49). Caddy redis plugin probably doesn't allow multiple IP addresses, so might have to add `haproxy` sidecar to every caddy sidecar as well to load balance to redis cluster. - Update `command` of volume servers docker-compose file to add all master server IPs From 26ac6ae3e39b5e21d3c232e82d17c8fd55763625 Mon Sep 17 00:00:00 2001 From: Robert Ames Date: Wed, 14 Feb 2024 11:13:51 -0600 Subject: [PATCH 004/149] prefer `/etc/motd` vs. `/etc/passwd` --- s3cmd-with-SeaweedFS.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/s3cmd-with-SeaweedFS.md b/s3cmd-with-SeaweedFS.md index a7b634e..012c0d5 100644 --- a/s3cmd-with-SeaweedFS.md +++ b/s3cmd-with-SeaweedFS.md @@ -43,31 +43,31 @@ Bucket 's3://newbucket/' created $ s3cmd ls s3:// 2019-01-01 01:30 s3://newbucket -$ s3cmd put /etc/passwd s3://newbucket +$ s3cmd put /etc/motd s3://newbucket WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. -upload: '/etc/passwd' -> 's3://newbucket/passwd' [1 of 1] +upload: '/etc/motd' -> 's3://newbucket/motd' [1 of 1] 6804 of 6804 100% in 0s 87.93 kB/s done -$ s3cmd get s3://newbucket/passwd -download: 's3://newbucket/passwd' -> './passwd' [1 of 1] +$ s3cmd get s3://newbucket/motd +download: 's3://newbucket/motd' -> './motd' [1 of 1] 6804 of 6804 100% in 0s 595.33 kB/s done # change the file -$ vi passwd +$ vi motd -$ s3cmd sync passwd s3://newbucket/ +$ s3cmd sync motd s3://newbucket/ WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. -upload: 'passwd' -> 's3://newbucket/passwd' [1 of 1] +upload: 'motd' -> 's3://newbucket/motd' [1 of 1] 22 of 22 100% in 0s 6.45 kB/s done Done. Uploaded 22 bytes in 1.0 seconds, 22.00 B/s. $ s3cmd ls s3://newbucket/ -2019-01-01 01:32 22 s3://newbucket/passwd +2019-01-01 01:32 22 s3://newbucket/motd -$ s3cmd del s3://newbucket/passwd -delete: 's3://newbucket/passwd' +$ s3cmd del s3://newbucket/motd +delete: 's3://newbucket/motd' $ s3cmd rb s3://newbucket Bucket 's3://newbucket/' removed -``` \ No newline at end of file +``` From 74ffd64137f1a25db0681f53301fe19a7a5960f1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Fr=C4=85ckiewicz?= <1836094+SystemZ@users.noreply.github.com> Date: Tue, 20 Feb 2024 20:18:18 +0100 Subject: [PATCH 005/149] explain better what happens when saving and loading metadata, explain one example case when to do it --- Filer-Stores.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/Filer-Stores.md b/Filer-Stores.md index 38e130a..2c67698 100644 --- a/Filer-Stores.md +++ b/Filer-Stores.md @@ -30,12 +30,14 @@ The Filer Store persists all file metadata and directory information. | TiKV | O(logN)| unlimited | Distributed, Fast |Atomic|Yes| Yes| Scalable. High Availability. Need to manually build. | #### Switching between different Stores -It is easy to switch between different filer stores. +It is easy to switch between different filer stores. +This is useful if you started with embedded DB and want to move to distributed one or in reverse. For example: ```sh -# first save current filer meta data +# make sure you don't create or modify files in filer before this to maintain consistency of metadata +# save current filer meta data to local disk $ weed shell > fs.cd / @@ -45,6 +47,7 @@ total 65 directories, 292 files meta data for http://localhost:8888/ is saved to localhost-8888-20190417-005421.meta > exit +# you can turn off old filer if you have metadata saved to file # now switch to a new filer, and load the previously saved metadata $ weed shell > fs.meta.load localhost-8888-20190417-005421.meta From f4d5d942928ad0d929356059a40ab5dac0690cf1 Mon Sep 17 00:00:00 2001 From: Robert Ames Date: Sat, 24 Feb 2024 08:36:03 -0600 Subject: [PATCH 006/149] Updated restic with SeaweedFS (markdown) --- restic-with-SeaweedFS.md | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/restic-with-SeaweedFS.md b/restic-with-SeaweedFS.md index 6d4ff3c..347b898 100644 --- a/restic-with-SeaweedFS.md +++ b/restic-with-SeaweedFS.md @@ -3,14 +3,31 @@ See https://restic.github.io On mac: `brew install restic` -### Configuration -Set these environment variables. The key values do not matter. +### Weed Configuration + +Set the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables in both your `weed` environment as well as your `restic` environment. + +The key values do not matter _but they must be provided_, as `restic` expects (and effectively requires) non-anonymous access when communicating with an S3 bucket. + +This may involve configuring your `weed` server similar to the following: + ```bash -export AWS_ACCESS_KEY_ID=any-key-id -export AWS_SECRET_ACCESS_KEY=any-access-key +export AWS_ACCESS_KEY_ID="any-key-id" +export AWS_SECRET_ACCESS_KEY="any-access-key" + +export WEED_S3_CONFIGURE_CMD="s3.configure \ + -access_key \"$AWS_ACCESS_KEY_ID\" \ + -secret_key \"$AWS_SECRET_ACCESS_KEY\" \ + -user iam \ + -actions Admin \ + -apply" +echo "$WEED_S3_CONFIGURE_CMD" | weed shell ``` -### Execute commands +### Execute Restic Commands + +See: https://github.com/seaweedfs/seaweedfs/wiki/s3cmd-with-SeaweedFS + First, create the bucket: ```bash s3cmd mb s3://resticbucket @@ -18,6 +35,9 @@ s3cmd mb s3://resticbucket Then, initialize the restic bucket and backup to it: ```console +$ export AWS_ACCESS_KEY_ID="any-key-id" +$ export AWS_SECRET_ACCESS_KEY="any-access-key" + $ restic -r s3:http://localhost:8333/resticbucket init $ restic -r s3:http://localhost:8333/resticbucket backup /Users/chris/dev/gopath/bin/ From 60fa206d43d7f905568e01597dd0ddf6c5f8e8c9 Mon Sep 17 00:00:00 2001 From: jerebear12 <72420925+jerebear12@users.noreply.github.com> Date: Tue, 27 Feb 2024 10:42:31 -0600 Subject: [PATCH 007/149] Updated security.toml to include new filer.expose_directory_metadata.enabled configuration option. --- Security-Configuration.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/Security-Configuration.md b/Security-Configuration.md index 7e68c01..560f561 100644 --- a/Security-Configuration.md +++ b/Security-Configuration.md @@ -31,6 +31,13 @@ expires_after_seconds = 10 # seconds [access] ui = false +# by default the filer UI is enabled. This can be a security risk if the filer is exposed to the public +# and the JWT for reads is not set. If you don't want the public to have access to the objects in your +# storage, and you haven't set the JWT for reads it is wise to disable access to directory metadata. +# This disables access to the Filer UI, and will no longer return directory metadata in GET requests. +[filer.expose_directory_metadata] +enabled = true + # this jwt signing key is read by master and volume server, and it is used for read operations: # - the Master server generates the JWT, which can be used to read a certain file on a volume server # - the Volume server validates the JWT on reading From f07b35d448ba04bee2319b4ddf989b9d4d3bcf57 Mon Sep 17 00:00:00 2001 From: Rootspring Date: Wed, 28 Feb 2024 17:04:32 +0530 Subject: [PATCH 008/149] Updated Cloud Monitoring (markdown) --- Cloud-Monitoring.md | 72 ++------------------------------------------- 1 file changed, 2 insertions(+), 70 deletions(-) diff --git a/Cloud-Monitoring.md b/Cloud-Monitoring.md index 2a3dcf3..ec6d94b 100644 --- a/Cloud-Monitoring.md +++ b/Cloud-Monitoring.md @@ -1,73 +1,5 @@ -Seaweed Cloud Monitoring is a service provided by seaweedfs.com. It is an attempt to help SeaweedFS to grow organically, +Seaweed Cloud Monitoring was a service provided by seaweedfs.com to attempt to help SeaweedFS to grow organically, * Iterate faster without upgrading the whole cluster. * Help SeaweedFS admin to manage the clusters via internet. -# Architecture - -``` - Browser => Seaweed Cloud <==> Seaweed Agent <==> SeaweedFS cluster -``` -The Seaweed Agent is a simple docker image `chrislusf/seaweed_agent`. Just need to run the docker image, setting the filer address and a password. To upgrade, just pull the latest image and restart. - -The docker image should be able to run for recent SeaweedFS clusters since 2.58 or so. There are no specific requirement for the SeaweedFS cluster. - -It subscribes to metadata changes from a SeaweedFS cluster, and build up statistics. When it starts, it may take some time to process existing metadata logs, if there are a lot of files or a lot of updates. - -When accessing the monitoring URL, the Seaweed Cloud would proxy the requests to the agent, get the statistics, and render to the web page. There are no data persisted on the Seaweed Cloud. (It is running on a free-tier web server.) - -## Features -Currently it has very basic features and all in beta. - -Existing features: -* Near real time directory statistics on disk usage, file count and directory count. -* Set quota for any directory. -* Automatically delete empty folders in buckets (under `/buckets`), to be more compatible with AWS S3 object store. - -![](CloudMonitoring.png) - -# Start Seaweed Agent - -To try it out, run these with docker. It should print out a URL which you can visit. - -``` -docker run --pull always -ti chrislusf/seaweed_agent swagent -filer 192.168.2.11:8888 -password=your_password - -# or set the password via environment variables - -docker run --pull always -e SEAWEED_PASSWORD=your_password -ti chrislusf/seaweed_agent swagent -filer 192.168.2.11:8888 - -``` - -The Seaweed Agent is still constantly evolving. So better always use the latest version. - -# Get Cloud Monitoring URL - -The container instance should have similar output, which you can find the Cloud Monitoring URL: -``` -docker run --pull always -e SEAWEED_PASSWORD=abc -ti chrislusf/seaweed_agent swagent -filer 192.168.2.10:8888 -2021/09/14 05:35:41 connecting to grpc.seaweedfs.com:4772 -2021/09/14 05:35:41 read /etc/seaweedfs/filer.conf: ReadEntry: filer: no entry is found in filer store -2021/09/14 05:35:41 tracking 192.168.2.10:8888: 1970-01-01 00:00:00 +0000 UTC -2021/09/14 05:35:42 connected to grpc.seaweedfs.com:4772 - ---- -Free Monitoring Data URL: -https://cloud.seaweedfs.com/ui/593HZ8CV9VF5976A8S0UABCDEFGHIJKL ---- -Filer version: 30GB 2.67 - -``` - -## Find the URL through `weed shell` -Now when starting `weed shell`, it should also print out the Cloud Monitoring URL: -``` -$ weed shell -master: localhost:9333 filer: localhost:8888 -I0913 22:50:51 96301 masterclient.go:96] adminShell masterClient Connecting to master localhost:9333 - ---- -Free Monitoring Data URL: -https://cloud.seaweedfs.com/ui/593HZ8CV9VF5976A8S0UABCDEFGHIJKL ---- -> -``` \ No newline at end of file +This service has now been deprecated and is no longer usable \ No newline at end of file From bd0e0c26de3aae1df59f8049ba4117abb73ce182 Mon Sep 17 00:00:00 2001 From: Rootspring Date: Wed, 28 Feb 2024 17:05:21 +0530 Subject: [PATCH 009/149] Updated Cloud Monitoring (markdown) --- Cloud-Monitoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Cloud-Monitoring.md b/Cloud-Monitoring.md index ec6d94b..88e9761 100644 --- a/Cloud-Monitoring.md +++ b/Cloud-Monitoring.md @@ -2,4 +2,4 @@ Seaweed Cloud Monitoring was a service provided by seaweedfs.com to attempt to h * Iterate faster without upgrading the whole cluster. * Help SeaweedFS admin to manage the clusters via internet. -This service has now been deprecated and is no longer usable \ No newline at end of file +**This service has now been deprecated and is no longer usable** \ No newline at end of file From 2f98741fc30d3cb0ec396ed45ee3987c8ec0e86f Mon Sep 17 00:00:00 2001 From: Rootspring Date: Wed, 28 Feb 2024 17:07:31 +0530 Subject: [PATCH 010/149] publicIp doesnt seem to exist anymore? --- Production-Setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Production-Setup.md b/Production-Setup.md index 079304a..880eff8 100644 --- a/Production-Setup.md +++ b/Production-Setup.md @@ -30,7 +30,7 @@ to ensure SeaweedFS uses the correct IP, for cluster communication. Append `-ip=xx.xx.xx.xx` to specify the appropriate address. If you wish to use a different IP address for user-facing services, then -set `-publicIp=yy.yy.yy.yy` as well. +set `-ip.bind=yy.yy.yy.yy` as well. ### For single node setup From f6755e814030c010510ab313ae9a51b91befa69c Mon Sep 17 00:00:00 2001 From: Seyed Mahdi Sadegh Shobeiri <36403983+SmsS4@users.noreply.github.com> Date: Fri, 1 Mar 2024 13:32:33 +0330 Subject: [PATCH 011/149] Created Volume Files Structure (markdown) --- Volume-Files-Structure.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 Volume-Files-Structure.md diff --git a/Volume-Files-Structure.md b/Volume-Files-Structure.md new file mode 100644 index 0000000..cb9ff6e --- /dev/null +++ b/Volume-Files-Structure.md @@ -0,0 +1 @@ +![Untitled-2023-09-24-1403](https://github.com/seaweedfs/seaweedfs/assets/36403983/41512a67-305c-4019-97ef-17a8ea836aca) From eee4ce1669f9f5c18b853e3231bf3edc7c8e5e2a Mon Sep 17 00:00:00 2001 From: Seyed Mahdi Sadegh Shobeiri <36403983+SmsS4@users.noreply.github.com> Date: Fri, 1 Mar 2024 13:33:47 +0330 Subject: [PATCH 012/149] Updated _Sidebar (markdown) --- _Sidebar.md | 1 + 1 file changed, 1 insertion(+) diff --git a/_Sidebar.md b/_Sidebar.md index 1d8cc3e..b4b4deb 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -117,6 +117,7 @@ * [[Cloud Monitoring]] * [[Load Command Line Options from a file]] * [[SRV Service Discovery]] +* [[Volume Files Structure]] ### Security * [[Security Overview]] From b17041e1e47ccaeae754ec6622c598ab17f57759 Mon Sep 17 00:00:00 2001 From: Nathan Date: Mon, 4 Mar 2024 21:52:01 -0800 Subject: [PATCH 013/149] Added typescript library for seaweed --- Client-Libraries.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Client-Libraries.md b/Client-Libraries.md index f63f6e3..2b834c5 100644 --- a/Client-Libraries.md +++ b/Client-Libraries.md @@ -14,6 +14,7 @@ Name | Author | Language [@trubavuong/seaweedfs](https://www.npmjs.com/package/@trubavuong/seaweedfs) | Vuong Tru | Javascript [@trubavuong/fastify-seaweedfs](https://www.npmjs.com/package/@trubavuong/fastify-seaweedfs) | Vuong Tru | Javascript [seaweedfs client plug-in for egg.js](https://github.com/aliwalker/egg-seaweed-client) | aliwalker | Javascript +[seaweedTS - Typescript library for seaweedfs](https://www.npmjs.com/package/seaweedts) | Nathan Nesbitt | Typescript [Weedo](https://github.com/ginuerzh/weedo) | Ginuerzh | Go [goseaweed](https://github.com/tnextday/goseaweed) | tnextday | Go [goseaweedfs](https://github.com/linxGnu/goseaweedfs) | linxGnu | Go From 799e48c46f345602111c1afde96f2b0a0db3cc6a Mon Sep 17 00:00:00 2001 From: Ming Yong Date: Mon, 11 Mar 2024 15:29:15 +0800 Subject: [PATCH 014/149] Updated Actual Users (markdown) --- Actual-Users.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Actual-Users.md b/Actual-Users.md index 80ee7dc..0092a9d 100644 --- a/Actual-Users.md +++ b/Actual-Users.md @@ -13,6 +13,8 @@ Add your name here if your company is using SeaweedFS. * [techbay](http://www.qualebs.com/techbay/) +* [Imagineapp](http://www.imagineapp.co/) + * ![SANMS](https://i.imgur.com/JNyx3CC_d.jpg?maxwidth=640&shape=thumb&fidelity=high) * 京东登月平台 (http://geek.csdn.net/news/detail/228285) \ No newline at end of file From b350242fccc7e0f4fda8b1b9b30c03ce6c7d8f15 Mon Sep 17 00:00:00 2001 From: Seyed Mahdi Sadegh Shobeiri <36403983+SmsS4@users.noreply.github.com> Date: Sat, 16 Mar 2024 19:28:31 +0330 Subject: [PATCH 015/149] Updated Volume Files Structure (markdown) --- Volume-Files-Structure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Volume-Files-Structure.md b/Volume-Files-Structure.md index cb9ff6e..7882925 100644 --- a/Volume-Files-Structure.md +++ b/Volume-Files-Structure.md @@ -1 +1 @@ -![Untitled-2023-09-24-1403](https://github.com/seaweedfs/seaweedfs/assets/36403983/41512a67-305c-4019-97ef-17a8ea836aca) +![struc](https://github.com/seaweedfs/seaweedfs/assets/36403983/3f29f4ed-a085-4e5e-81fe-1581d1a8b3c7) From 7d86df6f2cfc3b299055fc22933f01ff1c6952ee Mon Sep 17 00:00:00 2001 From: Nikita Korolev <66738864+doc-sheet@users.noreply.github.com> Date: Fri, 22 Mar 2024 17:27:26 +0300 Subject: [PATCH 016/149] Created Error reporting to sentry (markdown) --- Error-reporting-to-sentry.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 Error-reporting-to-sentry.md diff --git a/Error-reporting-to-sentry.md b/Error-reporting-to-sentry.md new file mode 100644 index 0000000..fa86a06 --- /dev/null +++ b/Error-reporting-to-sentry.md @@ -0,0 +1,13 @@ +Since [https://github.com/seaweedfs/seaweedfs/commit/953f5713490343f3cb4d36eb17bae9e8716068b4](probably 3.65) release will be possible to send errors to sentry. + +To enable reporting some environment variables must be set +* SENTRY_DSN (from project settings) +* SENTRY_ENVIRONMENT (production, prod, test or whatever is used in organization) +* SENTRY_RELEASE (weed@3.64) + +No other settings involved at this moment. + +SENTRY_DSN is the only required option and could be obtained from project settings. + +SENTRY_ENVIRONMENT and SENTRY_RELEASE may be empty or not set at all. +Assign it to what match your organization's flow. \ No newline at end of file From 00b68d3574f770edda5c29c9ad672fc946f35780 Mon Sep 17 00:00:00 2001 From: Nikita Korolev <66738864+doc-sheet@users.noreply.github.com> Date: Fri, 22 Mar 2024 17:33:56 +0300 Subject: [PATCH 017/149] Updated Error reporting to sentry (markdown) --- Error-reporting-to-sentry.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Error-reporting-to-sentry.md b/Error-reporting-to-sentry.md index fa86a06..b9e571b 100644 --- a/Error-reporting-to-sentry.md +++ b/Error-reporting-to-sentry.md @@ -1,4 +1,6 @@ -Since [https://github.com/seaweedfs/seaweedfs/commit/953f5713490343f3cb4d36eb17bae9e8716068b4](probably 3.65) release will be possible to send errors to sentry. +# Overview + +Since [probably 3.65](https://github.com/seaweedfs/seaweedfs/commit/953f5713490343f3cb4d36eb17bae9e8716068b4) release will be possible to send errors to sentry. To enable reporting some environment variables must be set * SENTRY_DSN (from project settings) From 3ef0082c26a4d952b9dae5830ba7532901f34b50 Mon Sep 17 00:00:00 2001 From: Scott Edlund Date: Tue, 9 Apr 2024 02:55:14 +0800 Subject: [PATCH 018/149] Simplification and grammar --- Components.md | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/Components.md b/Components.md index 7bc6506..1023e54 100644 --- a/Components.md +++ b/Components.md @@ -1,12 +1,10 @@ # SeaweedFS Components -SeaweedFS comprises 3 main conceptual components. The master service and the volume service together provide a distributed object store, with user-configurable replication and redundancy. The optional filer and S3 service are additional layers on top of the object store. Each of these services may run as one or separate instances, on various actual servers. +SeaweedFS comprises 3 main components. The master service and the volume service together provide a distributed object store, with user-configurable replication and redundancy. The optional filer and S3 service are additional layers on top of the object store. Each of these services may run as one or separate processes, on one or more operating systems. ## Master service -The essential master service works smart, not hard. - -It represents a cluster of 1 (or 3 or more servers) that own a consistent view of the entire SeaweedFS cluster and communicate it to all participating nodes, through a Leader elected via Raft protocol. +It represents a cluster of 1 (or 3 or n/2+1) that own a consistent view of the entire SeaweedFS cluster and communicate it to all participating nodes, through a leader elected via [Raft](https://raft.github.io/) protocol. The number of servers in the master service must always be odd, to ensure that a majority consensus can be formed. You're best off keeping this number down, a small number of stable servers is better than a large pool of flakey boxes. 1 or 3 is typical. @@ -18,17 +16,13 @@ If the leader is unavailable, the raft consensus protocol ensures that a new lea ## Volume service -The volume service is where the hard work is done. - It packs many objects (files and chunks of files) efficiently into larger individual volumes, which can be arbitrarily large blocks on disk. Redundancy and replication of data is managed at the volume level, not on a per-object level. Each volume server sends periodic heartbeats with status and volume information back to the leader, via a master. ## Filer service -The optional filer service does heavy lifting so you don't have to. - -It organizes SeaweedFS volumes and objects, into user-visible paths (like URLs or file systems) over HTTP or UNIX FUSE mounts. +It organizes SeaweedFS volumes and objects into user-visible paths (like URLs or file systems) over HTTP or UNIX FUSE mounts. Filer provides a convenient and common abstraction that can be used to provide normal looking filesystems, or web APIs for down/uploads, to existing applications without modification. @@ -38,7 +32,7 @@ This optional service provides AWS style S3 buckets, similar to the filer servic ## Volume Concept -The volume in SeaweedFS means a single actual file consists of many small files. When master starts, it configures the volume file size, default to 30GB. At the beginning, there are 8 volumes created. +In SeaweedFS, a volume is a single file consisting of many small files. When a master server starts, it sets the volume file maximum size to 30GB (see: `-volumeSizeLimitMB`). At volume server initialization, it will create 8 of these volumes (see: `-max`). Each volume has its own TTL and replication. From c635d26df13a00b0b92a792715b05d96359ea1ff Mon Sep 17 00:00:00 2001 From: Max! Date: Sat, 27 Apr 2024 14:48:13 +0200 Subject: [PATCH 019/149] replace manifest deployment with a helm local path install --- Kubernetes-Backups-and-Recovery-with-K8up.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/Kubernetes-Backups-and-Recovery-with-K8up.md b/Kubernetes-Backups-and-Recovery-with-K8up.md index d7eb115..f071bc0 100644 --- a/Kubernetes-Backups-and-Recovery-with-K8up.md +++ b/Kubernetes-Backups-and-Recovery-with-K8up.md @@ -132,8 +132,7 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 4. Deploy via Helm ```bash - helm template . -f test-values.yaml > manifests.yaml - kubectl apply -f manifests.yaml + helm install seaweedfs . -f test-values.yaml --wait ``` 5. Expose the S3 endpoint @@ -496,8 +495,7 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 - Deploy via Helm ```bash - helm template . -f restore-values.yaml > manifests.yaml - kubectl apply -f manifests.yaml + helm install seaweedfs . -f restore-values.yaml --wait ``` 7. Update your alias for your server: From 2dcf09eaf96fc4724ba0d80b3898d0df13798010 Mon Sep 17 00:00:00 2001 From: chrislu Date: Mon, 29 Apr 2024 06:26:23 -0700 Subject: [PATCH 020/149] AllowEmptyFolder checks during S3 object deletion --- Amazon-S3-API.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/Amazon-S3-API.md b/Amazon-S3-API.md index 473be3a..3520bff 100644 --- a/Amazon-S3-API.md +++ b/Amazon-S3-API.md @@ -83,11 +83,7 @@ Not included: SeaweedFS has directories while AWS S3 only has objects with "fake" directories. In AWS S3, if the last file is deleted in a directory, the directory will disappear also. -To be consistent with AWS S3, SeaweedFS tries to skip empty folders while listing. You can use `weed s3 -allowEmptyFolder` to toggle this behavior. - -This is not so ideal. Another approach is to list current directory when deleting a file, which will slow down quite a bit especially when deleting multiple files. SeaweedFS did not take this approach. - -The last approach, which is most efficient, is to maintain counters for each folder, and drop the folder as soon as it becomes empty. This is implemented in [[Cloud Monitoring]]. +To be consistent with AWS S3, SeaweedFS tries to check whether the folder is empty after each deletion. You can use `weed s3 -allowEmptyFolder` to toggle this behavior. # S3 Authentication From 1b7568f02a02c71d5184f0d9ac8215d4769391f1 Mon Sep 17 00:00:00 2001 From: shenxingwuying Date: Tue, 7 May 2024 14:40:08 +0800 Subject: [PATCH 021/149] Fix an stale option in the file mount start command and make filer's default config more clearly --- Production-Setup.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/Production-Setup.md b/Production-Setup.md index 880eff8..5160c6c 100644 --- a/Production-Setup.md +++ b/Production-Setup.md @@ -128,7 +128,7 @@ If currently only one filer is needed, just use one filer with default filer sto You can always migrate to other scalable filer store by export and import the filer meta data. See [[Filer Stores]] -Run `weed scaffold -config=filer` to generate an example `filer.toml` file. +Run `weed scaffold -config=filer` to generate an example `filer.toml` file. This file choose `leveldb2` as the filer store by default which stores file meta in local on disk. `leveldb2` only support one filer. The filer store to choose depends on your requirements, your existing data stores, etc. @@ -165,9 +165,10 @@ The endpoint is `http://:8333`. Run -`weed mount -filer= -chunkCacheCountLimit=xxx -chunkSizeLimitMB=4` +`weed mount -filer= -cacheCapacityMB=xxx -chunkSizeLimitMB=4 -dir=mount_point_dir` -* `-chunkCacheCountLimit` means how many entries cached in memory, default to 1000. With default `-chunkSizeLimitMB` set to 4, it may take up to 4x1000 MB memory. If all files are bigger than 4MB. +* `-cacheCapacityMB` means file chunk read cache capacity in MB with tiered cache(memory + disk), default 0 which means chunk cache for read is disabled. +* `-chunkSizeLimitMB` local write buffer size, also chunk large file, default 2 MB. * `-replication` is the replication level for each file. It overwrites replication settings on both filer and master. * `-volumeServerAccess=[direct|publicUrl|filerProxy]` is used if master, volume server, and filer are inside a cluster, but `weed mount` is outside of the cluster. With this option set to `filerProxy`, only filer needs to be exposed to outside. All read write access to volume servers will be proxied by filer. From 2ba6ca0bcdb53419567ab98a91e1a0accc279e6b Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Thu, 16 May 2024 16:04:30 -0700 Subject: [PATCH 022/149] Updated _Sidebar (markdown) --- _Sidebar.md | 1 + 1 file changed, 1 insertion(+) diff --git a/_Sidebar.md b/_Sidebar.md index b4b4deb..73e88a1 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -48,6 +48,7 @@ ### [[FUSE Mount]] * [[FIO benchmark]] +* [[fstab]] ### [[WebDAV]] From 703b8e85a9e581757a481102b7de04baed894b6d Mon Sep 17 00:00:00 2001 From: Benjamin Arntzen Date: Fri, 17 May 2024 00:09:52 +0100 Subject: [PATCH 023/149] Created fstab (markdown) --- fstab.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 fstab.md diff --git a/fstab.md b/fstab.md new file mode 100644 index 0000000..c51f5f7 --- /dev/null +++ b/fstab.md @@ -0,0 +1,15 @@ +To mount SeaweedFS using `/etc/fstab` (such as on boot): + +* Follow the directions in https://github.com/seaweedfs/seaweedfs/wiki/FUSE-Mount for setting up a SeaweedFS mount subtype for FUSE (hint: `cp weed /sbin/weed`) +* Install SeaweedFS as usual, making sure any specific settings (such as `security.toml`) are in the correct location to be read +* Add the fstab entry as described below + +If you have a single filer server, this is the syntax you will use: + +`fuse /path/to/mountpoint fuse.weed filer=localhost:8888,filer.path=/,defaults,_netdev 0 0` + +If you have multiple filer servers, this is the syntax you will use: + +`fuse /path/to/mountpoint fuse.weed filer='192.168.0.1:8888,192.168.0.2:8888',filer.path=/,defaults,_netdev 0 0` + +Place the appropriate line into `/etc/fstab` and attempt to mount your filesystem. If you encounter any issues, run `weed mount` directly in verbose/debug mode to diagnose the issue, and switch back to fstab-style mounting once you have resolved the issues. \ No newline at end of file From 881d93a5375948f8d43edf6ad4cf7744e21efd2f Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Thu, 16 May 2024 16:37:25 -0700 Subject: [PATCH 024/149] Updated _Sidebar (markdown) --- _Sidebar.md | 1 + 1 file changed, 1 insertion(+) diff --git a/_Sidebar.md b/_Sidebar.md index 73e88a1..cf3b39e 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -32,6 +32,7 @@ ### [[Filer Stores]] * [[Filer Cassandra Setup]] * [[Filer Redis Setup]] +* [[Filer YugabyteDB Setup]] * [[Super Large Directories]] * [[Path-Specific Filer Store]] * [[Choosing a Filer Store]] From ad269e03e412c12bc1cca6d82b54010e8458f8fd Mon Sep 17 00:00:00 2001 From: Jack Date: Fri, 17 May 2024 13:56:07 +0100 Subject: [PATCH 025/149] Tensorflow master branch link now 404's. It's unclear where/if this document has been moved to, but I found an old reference. --- TensorFlow-with-SeaweedFS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TensorFlow-with-SeaweedFS.md b/TensorFlow-with-SeaweedFS.md index 3425ee5..5f0fcac 100644 --- a/TensorFlow-with-SeaweedFS.md +++ b/TensorFlow-with-SeaweedFS.md @@ -18,7 +18,7 @@ model.fit(train_dataset, ...) ``` # TensorFlow on SeaweedFS S3 -[TensorFlow already supports S3](https://github.com/tensorflow/examples/blob/master/community/en/docs/deploy/s3.md) +[TensorFlow already supports S3](https://github.com/tensorflow/examples/blame/tflmm/v0.2.4/community/en/docs/deploy/s3.md) (Old Link) Here is an adaption of it with unnecessary content removed. From 60410f97c5fe6b029cb2d5ff277a099e40896eb5 Mon Sep 17 00:00:00 2001 From: Vlad Date: Wed, 22 May 2024 11:16:36 +1100 Subject: [PATCH 026/149] Updated Filer Stores (markdown) --- Filer-Stores.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Filer-Stores.md b/Filer-Stores.md index 2c67698..46f3a1c 100644 --- a/Filer-Stores.md +++ b/Filer-Stores.md @@ -55,6 +55,12 @@ $ weed shell total 65 directories, 292 files localhost-8888-20190417-005421.meta is loaded to http://localhost:8888/ +# optionally, you can use concurrency to speed up the meta load process +# -concurrency=N specifies the number of parallel meta load operations to filer +# caution: this may not work correctly with some databases like Redis + +# $ weed shell +# > fs.meta.load -concurrency=1 localhost-8888-20190417-005421.meta ``` ### Extending Storage Options From 29821e9582660187904297b525ec5075a70be2bd Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Thu, 23 May 2024 09:01:23 -0700 Subject: [PATCH 027/149] Updated Cloud Drive Benefits (markdown) --- Cloud-Drive-Benefits.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/Cloud-Drive-Benefits.md b/Cloud-Drive-Benefits.md index 26fd5ec..745a2ea 100644 --- a/Cloud-Drive-Benefits.md +++ b/Cloud-Drive-Benefits.md @@ -1,17 +1,17 @@ # Introduction -> NOTE: SeaweedFS provides **two mechanisms** to use cloud storage: -> 1) **SeaweedFS Cloud Drive** (**<== You are here**) -> - in this case, you can **mount** an S3 bucket to the Seaweedfs file system (in the filer), and access the remote files -> through SeaweedFS. Effectively, SeaweedFS caches the files from the cloud storage. -> - In this mode, the file structure in cloud store is exactly matching the SeaweedFS structure - so every -> file in SeaweedFS will also become a file in the cloud storage provider. -> - This is useful in case you want to use the files inside the cloud provider's infrastructure. -> - However, this does **not support file encryption** in any way (obviously), as the files are put to Cloud Storage as is. -> 2) [Tiered Storage with Cloud Tier](https://github.com/seaweedfs/seaweedfs/wiki/Cloud-Tier) -> - In this mode, seaweedFS **moves full volume files to the cloud storage provider**, so files which are 1 GB (in our case) big. -> - This mode supports [Filer Data Encryption](https://github.com/seaweedfs/seaweedfs/wiki/Filer-Data-Encryption) transparently. -> - The chunk files uploaded to the cloud provider are not usable outside of SeaweedFS. +SeaweedFS provides **two mechanisms** to use cloud storage: + 1) **SeaweedFS Cloud Drive** (**<== You are here**) + - in this case, you can **mount** an S3 bucket to the Seaweedfs file system (in the filer), and access the remote files + through SeaweedFS. Effectively, SeaweedFS caches the files from the cloud storage. + - In this mode, the file structure in cloud store is exactly matching the SeaweedFS structure - so every + file in SeaweedFS will also become a file in the cloud storage provider. + - This is useful in case you want to use the files inside the cloud provider's infrastructure. + - However, this does **not support file encryption** in any way (obviously), as the files are put to Cloud Storage as is. + 2) [Tiered Storage with Cloud Tier](https://github.com/seaweedfs/seaweedfs/wiki/Cloud-Tier) + - In this mode, seaweedFS **moves full volume files to the cloud storage provider**, so files which are 1 GB (in our case) big. + - This mode supports [Filer Data Encryption](https://github.com/seaweedfs/seaweedfs/wiki/Filer-Data-Encryption) transparently. + - The chunk files uploaded to the cloud provider are not usable outside of SeaweedFS. ## Cloud is not for everyone From 996e6815b4bf4b57fc4be8ef517730489c4d4acc Mon Sep 17 00:00:00 2001 From: swerter Date: Fri, 31 May 2024 10:23:21 +0200 Subject: [PATCH 028/149] Fix a small typo --- Cloud-Drive-Quick-Setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Cloud-Drive-Quick-Setup.md b/Cloud-Drive-Quick-Setup.md index 0af1363..24a89b5 100644 --- a/Cloud-Drive-Quick-Setup.md +++ b/Cloud-Drive-Quick-Setup.md @@ -101,7 +101,7 @@ These cache or uncache jobs can vary wildly. Here are some examples: # uncache file size older than 3600 seconds > remote.uncache -dir=/buckets/bucket1 -maxAge=3600 # uncache file size more than 10240 bytes -> remote.cache -dir=/buckets/bucket1 -minSize=10240 +> remote.uncache -dir=/buckets/bucket1 -minSize=10240 ``` From 9c10fa0b41b38ac17b3d0abc54507b65e69d8515 Mon Sep 17 00:00:00 2001 From: Paul Donald Date: Wed, 12 Jun 2024 13:03:59 +0200 Subject: [PATCH 029/149] Updated Production Setup (markdown) --- Production-Setup.md | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/Production-Setup.md b/Production-Setup.md index 5160c6c..4abf9a9 100644 --- a/Production-Setup.md +++ b/Production-Setup.md @@ -2,16 +2,16 @@ A robust production setup requires more configuration -- and care -- than the [[Getting Started]] guide. There are multiple layers of components. Please follow the steps to set them up from bottom up, one by one. -* Setup object storage - * Setup Masters +* Set up object storage + * Set up Masters * Add volume servers -* Setup file storage +* Set up file storage * Choose filer store * Setup Filer -And then, choose the component you want to setup -* Setup S3 -* Setup FUSE mount +And then, choose the component you want to set up +* Set up S3 +* Set up FUSE mount * Cluster Maintenance ## Prerequisites @@ -25,8 +25,8 @@ Make sure the ports are open. By default | Filer | 8888 | 18888 | | S3 | 8333 | | -If you have multi-homed servers (many IP addresses and interfaces), you need -to ensure SeaweedFS uses the correct IP, for cluster communication. Append +If you have multi-homed servers (many IP addresses and interfaces), +ensure SeaweedFS uses the correct IP for cluster communication. Append `-ip=xx.xx.xx.xx` to specify the appropriate address. If you wish to use a different IP address for user-facing services, then @@ -46,17 +46,17 @@ empty spaces, you need to reduce the volume size. weed server -filer -s3 -ip=xx.xx.xx.xx -volume.max=0 -master.volumeSizeLimitMB=1024 ``` -# Setup object storage +# Set up object storage -## Setup Masters +## Set up Masters ### One master is fine -If there are 2 machines, it is not possible to achieve consensus. Just do not bother to setup multiple masters. +If there are 2 machines, it is not possible to achieve consensus. Just do not bother to set up multiple masters. Even for large clusters, it is totally fine to have one single master. The load on master is very light. It is unlikely to go down. You can always just restart it since it only has soft states collected from volume servers. -### Setup masters +### Set up masters OK. Your CTO just wants multiple masters. To do so, see [[Failover Master Server]] for details. @@ -115,12 +115,12 @@ Additional notes: Now the object store setup is completed. You can visit `http://:9333/` to check it around. -* Make sure the Free volume count is not zero. -* Try to assign some file ids to trigger a volume allocation. +* Ensure the Free volume count is not zero. +* Try to assign some file IDs to trigger a volume allocation. If you only use SeaweedFS object store, that is all. -# Setup file storage +# Set up file storage ## Choose filer store @@ -161,7 +161,7 @@ weed filer -s3 -s3.config= -port=8333 The endpoint is `http://:8333`. -## Setup FUSE mount +## Set up FUSE mount Run @@ -174,7 +174,7 @@ Run ## Cluster Maintenance -In a cluster, volume servers can go down. But automatic rebalancing will be problematic. It can cause unexpected busy network activities. For example, the heartbeat of a volume server may come and go, causing unnecessary busy system. Current recommended strategy is to keep existing data readonly, and automatically add new writable volumes. +In a cluster, volume servers can go down. But automatic rebalancing will be problematic. It can cause unexpected busy network activities. For example, the heartbeat of a volume server may come and go, causing an unnecessarily busy system. Currently recommended strategy is to keep the existing data readonly, and automatically add new writable volumes. There are `volume.balance` and `volume.fix.replication` commands in `weed shell`. You can configure them to run during off hours. From 009e4886f61cacde9374a776111285ac783c5998 Mon Sep 17 00:00:00 2001 From: JesseBot Date: Sat, 22 Jun 2024 08:43:39 +0200 Subject: [PATCH 030/149] fixed grammar, syntax highlighting, and spacing --- Data-Backup.md | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/Data-Backup.md b/Data-Backup.md index 9d38ed9..ebe6264 100644 --- a/Data-Backup.md +++ b/Data-Backup.md @@ -1,28 +1,30 @@ ## A Simple Backup Strategy -The most important thing for storage is to keep data safe and not losing them. It gives you a comfort of thought that your data is safely backup. +The most important thing for storage is to keep data safe and not lose it. It gives you a comfort of thought that your data is safely backed up. -However, we do not want to always copy the whole data files over. We want to do it incrementally. +However, we do not always want to copy the all the data files over. We want to do it incrementally. -"weed backup" command is your friend. +`weed backup` command is your friend. -Run "weed backup" command on any machine that have enough disk spaces. Assuming we want to backup volume 5. +Run `weed backup` command on any machine that has enough disk space. Assuming we want to backup volume 5: - weed backup -server=master:port -dir=. -volumeId=5 +```bash +weed backup -server=master:port -dir=. -volumeId=5 +``` -If local volume 5 does not exist, it will be created. All remote needle entries are fetched and compared to local needle entries. The delta is calculated and local missing files are fetched from the volume server. +If local volume 5 does not exist, it will be created. All remote needle entries are fetched and compared to local needle entries. The delta is calculated, and local missing files are fetched from the volume server. -If you specify `-volumeId=87`, but volume 87 does not exist, it's ok. No files will be created locally. This gives the opportunity that you can create a backup script simply looping from 1 to 100. All existing volumes will be backed up. The non-existing volumes can also be backed up when they are created remotely. +If you specify `-volumeId=87`, but volume 87 does not exist, it's ok. No files will be created locally. This gives the opportunity for you to create a backup script simply looping from 1 to 100. All existing volumes will be backed up. The non-existing volumes can also be backed up when they are created remotely. -The backup scripts is just one command, not a continuous running service though. High Availability servers will be added later. +The backup scripts are just one command, not a continuous running service though. High Availability servers will be added later. ## How to create a mirror of a cluster ### To Start -- Pause operations on cluster you want to backup. This is to avoid a mismatch in the two data sets(the volumes and the Filer metadata) you will be moving separately and combining in the backup cluster. +- Pause operations on cluster you want to backup. This is to avoid a mismatch in the two data sets (the volumes and the Filer metadata) you will be moving separately and combining in the backup cluster. - Install SeaweedFS on a new machine/cluster of machines. Use the same version of SeaweedFS if you can to avoid any issues that could otherwise come up. -- *Do not start your volume servers yet!!* This is to avoid SeaweedFS creating it's own volumes(we will be using the volumes backed up from the current operational cluster). +- *Do not start your volume servers yet!!* This is to avoid SeaweedFS creating it's own volumes (we will be using the volumes backed up from the current operational cluster). ### Prepare the New Cluster and Backup Your Data @@ -32,18 +34,23 @@ The backup scripts is just one command, not a continuous running service though. ### Backup the Metadata -run `fs.meta.save` on the cluster you are pulling from and save the output. This can look like: +Run `fs.meta.save` on the cluster you are pulling from and save the output. This can look like: + ```sh # You will need permission to create a file in the destination directory # I recommend changing the file name because the default naming convention is not very readable BUT it does show the date the file was created which can be good information to store and know. fs.meta.save -o=[yourlocaldir]/[yourfilename].meta ``` + then download the file on the Filer machine you are using in the backup cluster. This can look like: + ```sh # This tool requires that the remote machine be accessable via SSH and that you have the password scp [hostmachineusername]@[hostmachineip]:/remote_directory/file /local/directory ``` + run `fs.meta.load` in the backup cluster on the Filer: + ```sh fs.meta.load [filepath/filename.meta] ``` From 02e211a5267aa8854a6fd541eaaadc6c04c2080c Mon Sep 17 00:00:00 2001 From: zako91 <56262719+zako91@users.noreply.github.com> Date: Sat, 29 Jun 2024 15:09:08 +0200 Subject: [PATCH 031/149] weed mount with non-root user --- FUSE-Mount.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/FUSE-Mount.md b/FUSE-Mount.md index f0da1b6..ad51a67 100644 --- a/FUSE-Mount.md +++ b/FUSE-Mount.md @@ -310,4 +310,10 @@ The issue is with samba.conf. If you see NT_STATUS_ACCESS_DENIED error, try to a Size: total number of volumes * volume size limit Used: (Logical Total Size of files - Logical Deleted File Size) * replication = physical disk size taken Available = Size - Used +``` +### Can't mount as non-root user ### +``` +Workaround is to use Linux capabilities on weed executable. +setcap cap_net_raw,cap_net_admin,cap_dac_override+eip /usr/local/bin/weed +After that you need to use "weed mount" with option allowOthers=false ``` \ No newline at end of file From 277c843b5cb06fa7139da65334088781bf9ec012 Mon Sep 17 00:00:00 2001 From: zako91 <56262719+zako91@users.noreply.github.com> Date: Mon, 1 Jul 2024 02:04:51 +0200 Subject: [PATCH 032/149] Updated FUSE Mount (markdown) --- FUSE-Mount.md | 1 + 1 file changed, 1 insertion(+) diff --git a/FUSE-Mount.md b/FUSE-Mount.md index ad51a67..4219fb7 100644 --- a/FUSE-Mount.md +++ b/FUSE-Mount.md @@ -312,6 +312,7 @@ Used: (Logical Total Size of files - Logical Deleted File Size) * replication = Available = Size - Used ``` ### Can't mount as non-root user ### +From https://github.com/seaweedfs/seaweedfs/issues/877 ``` Workaround is to use Linux capabilities on weed executable. setcap cap_net_raw,cap_net_admin,cap_dac_override+eip /usr/local/bin/weed From 9ac690d3ef6cbc550e51c527a126721063dbd6e5 Mon Sep 17 00:00:00 2001 From: Scott Edlund Date: Fri, 19 Jul 2024 13:52:01 +0200 Subject: [PATCH 033/149] use single pkill command instead of kill and pgrep --- Volume-Management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Volume-Management.md b/Volume-Management.md index cc60503..86ff0de 100644 --- a/Volume-Management.md +++ b/Volume-Management.md @@ -65,4 +65,4 @@ The balancing plan will try to evenly spread the number of writable and readonly Run `weed shell` and `volume.mount -node : -volumeId ` to mount a volume file. -You can also execute locally on volume server `kill -s HUP $(pgrep -f "weed volume")` to mount for all new volume files. +You can also execute locally on volume server `pkill -HUP -f "weed volume"` to mount all new volume files. From 9bad248ce699d838eee1d913ea5e4978bf2548b0 Mon Sep 17 00:00:00 2001 From: Scott Edlund Date: Fri, 19 Jul 2024 17:26:47 +0200 Subject: [PATCH 034/149] grammar --- Volume-Management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Volume-Management.md b/Volume-Management.md index 86ff0de..df25792 100644 --- a/Volume-Management.md +++ b/Volume-Management.md @@ -65,4 +65,4 @@ The balancing plan will try to evenly spread the number of writable and readonly Run `weed shell` and `volume.mount -node : -volumeId ` to mount a volume file. -You can also execute locally on volume server `pkill -HUP -f "weed volume"` to mount all new volume files. +To mount all new volume files you can send a hang-up signal to the volume server causing a reload with a command such as `pkill -HUP -f "weed volume"`. From 6fb8d269e777bdff0dcc06fac229858d1b7d484a Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Thu, 8 Aug 2024 08:49:00 -0700 Subject: [PATCH 035/149] Updated _Sidebar (markdown) --- _Sidebar.md | 1 - 1 file changed, 1 deletion(-) diff --git a/_Sidebar.md b/_Sidebar.md index cf3b39e..73e88a1 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -32,7 +32,6 @@ ### [[Filer Stores]] * [[Filer Cassandra Setup]] * [[Filer Redis Setup]] -* [[Filer YugabyteDB Setup]] * [[Super Large Directories]] * [[Path-Specific Filer Store]] * [[Choosing a Filer Store]] From 74550298127a40ba794b1dac43a2cb6b86bf98a7 Mon Sep 17 00:00:00 2001 From: Shaun Alexander <37967220+sierra-alpha@users.noreply.github.com> Date: Sat, 10 Aug 2024 09:14:40 +1200 Subject: [PATCH 036/149] I was following along these docs and realised that the `filer.remote.sync`command was not the intended command I believe. It should be `filer.remote.gateway`, I got a warning that `-createBucketAt` was not an option for `filer.remote.sync` leading me to make this change. --- Gateway-to-Remote-Object-Storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Gateway-to-Remote-Object-Storage.md b/Gateway-to-Remote-Object-Storage.md index d0eb1c5..35ea64b 100644 --- a/Gateway-to-Remote-Object-Storage.md +++ b/Gateway-to-Remote-Object-Storage.md @@ -43,7 +43,7 @@ It is OK to pause it, and resume. It is also OK to change the `-createBucketAt=xxx` value to a different one, since it only affects new bucket creation. ``` -$ weed filer.remote.sync -createBucketAt=cloud1 +$ weed filer.remote.gateway -createBucketAt=cloud1 synchronize /buckets, default new bucket creation in cloud1 ... ``` From ff293acfd2372b0e09bdada3adf20bb4a3181350 Mon Sep 17 00:00:00 2001 From: Dan Goodman Date: Sun, 11 Aug 2024 14:33:12 -0400 Subject: [PATCH 037/149] Updated Amazon S3 API (markdown) --- Amazon-S3-API.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Amazon-S3-API.md b/Amazon-S3-API.md index 3520bff..56920e5 100644 --- a/Amazon-S3-API.md +++ b/Amazon-S3-API.md @@ -4,7 +4,7 @@ To be compatible with Amazon S3 API, a separate "weed s3" command is provided. T `weed s3` will start a stateless gateway server to bridge the Amazon S3 API to SeaweedFS Filer. For convenience, `weed server -s3` will start a master, a volume server, a filer, and the S3 gateway. And `weed filer -s3` can start a filer and the S3 gateway together also. -Each bucket is stored in one collection, and mapped to folder /buckets/ by default. +Each bucket is stored in one collection, and mapped to folder `/buckets/` by default. A bucket can be deleted efficiently by deleting the whole collection. From 7a240412bba6d458f882af72f8e3db7dd84816b8 Mon Sep 17 00:00:00 2001 From: chrislu Date: Mon, 12 Aug 2024 17:40:32 -0700 Subject: [PATCH 038/149] update maven --- Hadoop-Benchmark.md | 2 +- Hadoop-Compatible-File-System.md | 32 +++++++++++++------------- Run-Blob-Storage-on-Public-Internet.md | 2 +- Run-Presto-on-SeaweedFS.md | 10 ++++---- SeaweedFS-Java-Client.md | 6 ++--- run-HBase-on-SeaweedFS.md | 2 +- run-Spark-on-SeaweedFS.md | 14 +++++------ 7 files changed, 34 insertions(+), 34 deletions(-) diff --git a/Hadoop-Benchmark.md b/Hadoop-Benchmark.md index 7b0d5cd..7f2e352 100644 --- a/Hadoop-Benchmark.md +++ b/Hadoop-Benchmark.md @@ -26,7 +26,7 @@ Then get the seaweedfs hadoop client jar. ``` cd share/hadoop/common/lib/ -wget https://oss.sonatype.org/service/local/repositories/releases/content/com/github/chrislusf/seaweedfs-hadoop2-client/3.59/seaweedfs-hadoop2-client-3.59.jar +wget https://repo1.maven.org/maven2/com/seaweedfs/seaweedfs-hadoop2-client/3.71/seaweedfs-hadoop2-client-3.71.jar ``` # TestDFSIO Benchmark diff --git a/Hadoop-Compatible-File-System.md b/Hadoop-Compatible-File-System.md index 4fcf397..e96d56a 100644 --- a/Hadoop-Compatible-File-System.md +++ b/Hadoop-Compatible-File-System.md @@ -10,44 +10,44 @@ $ mvn install # build for hadoop2 $cd $GOPATH/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2 $ mvn package -$ ls -al target/seaweedfs-hadoop2-client-3.59.jar +$ ls -al target/seaweedfs-hadoop2-client-3.71.jar # build for hadoop3 $cd $GOPATH/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3 $ mvn package -$ ls -al target/seaweedfs-hadoop3-client-3.59.jar +$ ls -al target/seaweedfs-hadoop3-client-3.71.jar ``` Maven ``` - com.github.chrislusf + com.seaweedfs seaweedfs-hadoop3-client - 3.59 + 3.71 or - com.github.chrislusf + com.seaweedfs seaweedfs-hadoop2-client - 3.59 + 3.71 ``` Or you can download the latest version from MavenCentral -* https://mvnrepository.com/artifact/com.github.chrislusf/seaweedfs-hadoop2-client - * [seaweedfs-hadoop2-client-3.59.jar](https://oss.sonatype.org/service/local/repositories/releases/content/com/github/chrislusf/seaweedfs-hadoop2-client/3.59/seaweedfs-hadoop2-client-3.59.jar) -* https://mvnrepository.com/artifact/com.github.chrislusf/seaweedfs-hadoop3-client - * [seaweedfs-hadoop3-client-3.59.jar](https://oss.sonatype.org/service/local/repositories/releases/content/com/github/chrislusf/seaweedfs-hadoop3-client/3.59/seaweedfs-hadoop3-client-3.59.jar) +* https://central.sonatype.com/artifact/com.seaweedfs/seaweedfs-hadoop2-client/versions + * [seaweedfs-hadoop2-client-3.71.jar](https://repo1.maven.org/maven2/com/seaweedfs/seaweedfs-hadoop2-client/3.71/seaweedfs-hadoop2-client-3.71.jar) +* https://central.sonatype.com/artifact/com.seaweedfs/seaweedfs-hadoop3-client/versions + * [seaweedfs-hadoop3-client-3.71.jar](https://repo1.maven.org/maven2/com/seaweedfs/seaweedfs-hadoop3-client/3.71/seaweedfs-hadoop3-client-3.71.jar) # Test SeaweedFS on Hadoop Suppose you are getting a new Hadoop installation. Here are the minimum steps to get SeaweedFS to run. -You would need to start a weed filer first, build the seaweedfs-hadoop2-client-3.59.jar -or seaweedfs-hadoop3-client-3.59.jar, and do the following: +You would need to start a weed filer first, build the seaweedfs-hadoop2-client-3.71.jar +or seaweedfs-hadoop3-client-3.71.jar, and do the following: ``` # optionally adjust hadoop memory allocation @@ -60,12 +60,12 @@ $ echo "" > etc/hadoop/mapred-site.xml # on hadoop2 $ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \ -Dfs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \ - -libjars ./seaweedfs-hadoop2-client-3.59.jar \ + -libjars ./seaweedfs-hadoop2-client-3.71.jar \ -ls / # or on hadoop3 $ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \ -Dfs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \ - -libjars ./seaweedfs-hadoop3-client-3.59.jar \ + -libjars ./seaweedfs-hadoop3-client-3.71.jar \ -ls / ``` @@ -112,9 +112,9 @@ $ bin/hadoop classpath # Copy SeaweedFS HDFS client jar to one of the folders $ cd ${HADOOP_HOME} # for hadoop2 -$ cp ./seaweedfs-hadoop2-client-3.59.jar share/hadoop/common/lib/ +$ cp ./seaweedfs-hadoop2-client-3.71.jar share/hadoop/common/lib/ # or for hadoop3 -$ cp ./seaweedfs-hadoop3-client-3.59.jar share/hadoop/common/lib/ +$ cp ./seaweedfs-hadoop3-client-3.71.jar share/hadoop/common/lib/ ``` Now you can do this: diff --git a/Run-Blob-Storage-on-Public-Internet.md b/Run-Blob-Storage-on-Public-Internet.md index 6b86c92..3e3f77a 100644 --- a/Run-Blob-Storage-on-Public-Internet.md +++ b/Run-Blob-Storage-on-Public-Internet.md @@ -14,7 +14,7 @@ We will use 2 servers. Server 1 will host master, 2x volumes (2 disks, one volum # todo: use 2 step build process, copy over weed binary to fresh container (do not need curl and tar at runtime) FROM alpine RUN apk update && apk add wget tar -RUN wget https://github.com/seaweedfs/seaweedfs/releases/download/3.59/linux_amd64_large_disk.tar.gz +RUN wget https://github.com/seaweedfs/seaweedfs/releases/download/3.71/linux_amd64_large_disk.tar.gz RUN tar -xf linux_amd64_large_disk.tar.gz RUN chmod +x weed RUN mv weed /usr/bin/ diff --git a/Run-Presto-on-SeaweedFS.md b/Run-Presto-on-SeaweedFS.md index 7d200e2..f214f31 100644 --- a/Run-Presto-on-SeaweedFS.md +++ b/Run-Presto-on-SeaweedFS.md @@ -5,10 +5,10 @@ The installation steps are divided into 2 steps: * https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+Administration ### Configure Hive Metastore to support SeaweedFS -1. Copy the seaweedfs-hadoop2-client-3.59.jar to hive lib directory,for example: +1. Copy the seaweedfs-hadoop2-client-3.71.jar to hive lib directory,for example: ``` -cp seaweedfs-hadoop2-client-3.59.jar /opt/hadoop/share/hadoop/common/lib/ -cp seaweedfs-hadoop2-client-3.59.jar /opt/hive-metastore/lib/ +cp seaweedfs-hadoop2-client-3.71.jar /opt/hadoop/share/hadoop/common/lib/ +cp seaweedfs-hadoop2-client-3.71.jar /opt/hive-metastore/lib/ ``` 2. Modify core-site.xml modify core-site.xml to support SeaweedFS, 30888 is the filer port @@ -50,9 +50,9 @@ metastore.thrift.port is the access port exposed by the Hive Metadata service it Follow instructions for installation of Presto: * https://prestosql.io/docs/current/installation/deployment.html ### Configure Presto to support SeaweedFS -1. Copy the seaweedfs-hadoop2-client-3.59.jar to Presto directory,for example: +1. Copy the seaweedfs-hadoop2-client-3.71.jar to Presto directory,for example: ``` -cp seaweedfs-hadoop2-client-3.59.jar /opt/presto-server-347/plugin/hive-hadoop2/ +cp seaweedfs-hadoop2-client-3.71.jar /opt/presto-server-347/plugin/hive-hadoop2/ ``` 2. Modify core-site.xml diff --git a/SeaweedFS-Java-Client.md b/SeaweedFS-Java-Client.md index 6f31c1c..0f7414f 100644 --- a/SeaweedFS-Java-Client.md +++ b/SeaweedFS-Java-Client.md @@ -13,15 +13,15 @@ $ mvn install Gradle ```gradle -implementation 'com.github.chrislusf:seaweedfs-client:3.59' +implementation 'com.github.chrislusf:seaweedfs-client:3.71' ``` Maven ```xml - com.github.chrislusf + com.seaweedfs seaweedfs-client - 3.59 + 3.71 ``` Or you can download the latest version from MavenCentral diff --git a/run-HBase-on-SeaweedFS.md b/run-HBase-on-SeaweedFS.md index a97d5fa..c738a72 100644 --- a/run-HBase-on-SeaweedFS.md +++ b/run-HBase-on-SeaweedFS.md @@ -1,7 +1,7 @@ # Installation for HBase Two steps to run HBase on SeaweedFS -1. Copy the seaweedfs-hadoop2-client-3.59.jar to `${HBASE_HOME}/lib` +1. Copy the seaweedfs-hadoop2-client-3.71.jar to `${HBASE_HOME}/lib` 1. And add the following 2 properties in `${HBASE_HOME}/conf/hbase-site.xml` ``` diff --git a/run-Spark-on-SeaweedFS.md b/run-Spark-on-SeaweedFS.md index 1bc6936..b05ab60 100644 --- a/run-Spark-on-SeaweedFS.md +++ b/run-Spark-on-SeaweedFS.md @@ -11,12 +11,12 @@ To make these files visible to Spark, set HADOOP_CONF_DIR in $SPARK_HOME/conf/sp ## installation not inheriting from Hadoop cluster configuration -Copy the seaweedfs-hadoop2-client-3.59.jar to all executor machines. +Copy the seaweedfs-hadoop2-client-3.71.jar to all executor machines. Add the following to spark/conf/spark-defaults.conf on every node running Spark ``` -spark.driver.extraClassPath=/path/to/seaweedfs-hadoop2-client-3.59.jar -spark.executor.extraClassPath=/path/to/seaweedfs-hadoop2-client-3.59.jar +spark.driver.extraClassPath=/path/to/seaweedfs-hadoop2-client-3.71.jar +spark.executor.extraClassPath=/path/to/seaweedfs-hadoop2-client-3.71.jar ``` And modify the configuration at runtime: @@ -37,8 +37,8 @@ And modify the configuration at runtime: 1. change the spark-defaults.conf ``` -spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.59.jar -spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.59.jar +spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.71.jar +spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.71.jar spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem ``` @@ -81,8 +81,8 @@ spark.history.fs.cleaner.enabled=true spark.history.fs.logDirectory=seaweedfs://localhost:8888/spark2-history/ spark.eventLog.dir=seaweedfs://localhost:8888/spark2-history/ -spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.59.jar -spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.59.jar +spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.71.jar +spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.71.jar spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888 ``` From 481cfa0958b4f2020015d848e82de4a3e955b534 Mon Sep 17 00:00:00 2001 From: Dan Goodman Date: Sun, 18 Aug 2024 10:45:58 -0400 Subject: [PATCH 039/149] fix typo --- Super-Large-Directories.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Super-Large-Directories.md b/Super-Large-Directories.md index 2f7b35d..a02bcc7 100644 --- a/Super-Large-Directories.md +++ b/Super-Large-Directories.md @@ -53,7 +53,7 @@ So for large directories configured in Redis, SeaweedFS skips this operation, so The consequences are: * The directory listing for this folder is not supported. -* The filer meta dada import and export for this folder is not supported. You can still do it for specific child folders though. +* The filer meta data import and export for this folder is not supported. You can still do it for specific child folders though. * Once this is configured, it can not be changed back easily. You will need to write code to iterate all sub entries for that. # How to configure it? From 7f746510359265dc96a3f6d754f07c351e9a645d Mon Sep 17 00:00:00 2001 From: Erik <166780591+Erikdre@users.noreply.github.com> Date: Mon, 19 Aug 2024 05:01:50 +0200 Subject: [PATCH 040/149] Updated Home (markdown) --- Home.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Home.md b/Home.md index 9a35d99..611b700 100644 --- a/Home.md +++ b/Home.md @@ -1,4 +1,4 @@ -# Welcome to the SeaweedFS wiki! +* # Welcome to the SeaweedFS wiki! SeaweedFS is a versatile and efficient storage system designed to meet the needs of modern sysadmins managing a mix of blob, object, file, and data warehouse storage requirements. Its architecture guarantees fast access times, with constant-time (O(1)) disk seeks, regardless of the size of the dataset. This makes it an excellent choice for environments where speed and efficiency are critical. From e164a4976567f43298fdb917f2ad732f28d6bee4 Mon Sep 17 00:00:00 2001 From: Scott Edlund Date: Fri, 30 Aug 2024 15:51:42 +0200 Subject: [PATCH 041/149] grammar --- Volume-Management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Volume-Management.md b/Volume-Management.md index df25792..02804e9 100644 --- a/Volume-Management.md +++ b/Volume-Management.md @@ -1,4 +1,4 @@ -When running large clusters, it is common to add more volume servers, or some volume servers are down, or some volume servers are replaced. These topology changes can cause missing volume replicas, or unbalanced number of volumes on volume servers. +When managing large clusters, it's common to add more volume servers, have some servers go down, or replace others. These changes can lead to missing volume replicas or an uneven distribution of volumes across the servers. ## Optimize volumes See [[Optimization]] page on how to optimize for concurrent writes and concurrent reads. From 1bdee1b22ec01a7527c18ddd3867ce96f0407ea6 Mon Sep 17 00:00:00 2001 From: Vraj Shah <89588024+Vrajs16@users.noreply.github.com> Date: Thu, 19 Sep 2024 23:15:16 -0400 Subject: [PATCH 042/149] Created _Footer (markdown) Destroyed _Footer (markdown) Updated Home (markdown) Revert e164a4976567f43298fdb917f2ad732f28d6bee4...e17ab3c8394b007e5d6d7f189d06c88fc673c669 From 1f851d3df33fd692e89c5edbb97e28c8cd824fb6 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Mon, 30 Sep 2024 12:01:33 -0700 Subject: [PATCH 043/149] Updated Home (markdown) --- Home.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Home.md b/Home.md index 611b700..9a35d99 100644 --- a/Home.md +++ b/Home.md @@ -1,4 +1,4 @@ -* # Welcome to the SeaweedFS wiki! +# Welcome to the SeaweedFS wiki! SeaweedFS is a versatile and efficient storage system designed to meet the needs of modern sysadmins managing a mix of blob, object, file, and data warehouse storage requirements. Its architecture guarantees fast access times, with constant-time (O(1)) disk seeks, regardless of the size of the dataset. This makes it an excellent choice for environments where speed and efficiency are critical. From 2c53293ac1edab357fc496c33e45b2b18d94ef50 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Wed, 2 Oct 2024 14:54:23 -0700 Subject: [PATCH 044/149] Updated _Sidebar (markdown) --- _Sidebar.md | 1 + 1 file changed, 1 insertion(+) diff --git a/_Sidebar.md b/_Sidebar.md index 73e88a1..0d85853 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -73,6 +73,7 @@ * [[S3 Bucket Quota]] * [[S3 API Audit log]] * [[S3 Nginx Proxy]] +* [[Docker Compose for S3]] ### AWS IAM * [[Amazon IAM API]] From fb40ab7c508e6412d13f1f44f8f6df3c444462b7 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Wed, 2 Oct 2024 15:58:17 -0700 Subject: [PATCH 045/149] Created Docker Compose for S3 (markdown) --- Docker-Compose-for-S3.md | 42 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 Docker-Compose-for-S3.md diff --git a/Docker-Compose-for-S3.md b/Docker-Compose-for-S3.md new file mode 100644 index 0000000..387f4d8 --- /dev/null +++ b/Docker-Compose-for-S3.md @@ -0,0 +1,42 @@ +To use SeaweedFS S3 API, here is the simplest form: + +``` +version: '3.8' + +services: + seaweedfs-s3: + image: chrislusf/seaweedfs + container_name: seaweedfs-s3 + ports: + - "8333:8333" + entrypoint: /bin/sh -c + command: | + "echo '{ + \"identities\": [ + { + \"name\": \"anonymous\", + \"actions\": [ + \"Read\" + ] + }, + { + \"name\": \"some_admin_user\", + \"credentials\": [ + { + \"accessKey\": \"some_access_key1\", + \"secretKey\": \"some_secret_key1\" + } + ], + \"actions\": [ + \"Admin\", + \"Read\", + \"List\", + \"Tagging\", + \"Write\" + ] + } + ] + }' > /etc/seaweedfs/config.json && \ + weed server -s3 -s3.config /etc/seaweedfs/config.json" + restart: unless-stopped +``` \ No newline at end of file From fe6d93875fb05dbc2a5c37da89cb2bddd3b3c7ca Mon Sep 17 00:00:00 2001 From: Er2 Date: Thu, 3 Oct 2024 18:20:15 +0300 Subject: [PATCH 046/149] Updated s3cmd with SeaweedFS (markdown) --- s3cmd-with-SeaweedFS.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/s3cmd-with-SeaweedFS.md b/s3cmd-with-SeaweedFS.md index 012c0d5..c81eea9 100644 --- a/s3cmd-with-SeaweedFS.md +++ b/s3cmd-with-SeaweedFS.md @@ -35,6 +35,25 @@ use_https = No signature_v2 = False ``` +### Reverse proxy sub-path configuration + +This is undefined behavior as AWS S3 servers always have sub-domains instead of sub-paths. \ +Use this only if you can't create (sub-)domain and use other port! + +For s3cmd, `X-Forwarded-Host` is required to be `your.server/s3` but not for other utils +so it's better to use `signature_v2 = True` in s3cfg + +Example for Caddy web server +``` +redir /s3 /s3/ +handle_path /s3/* { + reverse_proxy localhost:8333 { + header_up X-Forwarded-Host your.server/s3 + header_up X-Forwarded-Prefix /s3 + } +} +``` + ### Execute commands ``` $ s3cmd mb s3://newbucket From 5d55565e21059eb837d9dcb7186fffd508503ebf Mon Sep 17 00:00:00 2001 From: Er2 Date: Thu, 3 Oct 2024 18:22:36 +0300 Subject: [PATCH 047/149] Updated AWS CLI with SeaweedFS (markdown) --- AWS-CLI-with-SeaweedFS.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/AWS-CLI-with-SeaweedFS.md b/AWS-CLI-with-SeaweedFS.md index 0c0ac37..548c299 100644 --- a/AWS-CLI-with-SeaweedFS.md +++ b/AWS-CLI-with-SeaweedFS.md @@ -19,6 +19,23 @@ $ aws configure set default.s3.signature_version s3v4 ``` +### Reverse proxy sub-path configuration + +This is undefined behavior as AWS S3 servers always have sub-domains instead of sub-paths. \ +Use this only if you can't create (sub-)domain and use other port! + +AWS CLI appends sub-path before actual path so need to add `X-Forwarded-Prefix` header (set to `/s3` for example) + +Example for Caddy web server +``` +redir /s3 /s3/ +handle_path /s3/* { + reverse_proxy localhost:8333 { + header_up X-Forwarded-Prefix /s3 + } +} +``` + ### Execute commands ``` # list buckets From 32f301ff169eff178988292b5fef0ad3117a83f5 Mon Sep 17 00:00:00 2001 From: Er2 Date: Thu, 3 Oct 2024 18:23:22 +0300 Subject: [PATCH 048/149] Updated rclone with SeaweedFS (markdown) --- rclone-with-SeaweedFS.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/rclone-with-SeaweedFS.md b/rclone-with-SeaweedFS.md index 97a1b8f..b541a19 100644 --- a/rclone-with-SeaweedFS.md +++ b/rclone-with-SeaweedFS.md @@ -19,6 +19,23 @@ chunk_size = 50Mi force_path_style = true ``` +### Reverse proxy sub-path configuration + +This is undefined behavior as AWS S3 servers always have sub-domains instead of sub-paths. \ +Use this only if you can't create (sub-)domain and use other port! + +rclone (like AWS CLI) appends sub-path before actual path so need to add `X-Forwarded-Prefix` header (set to `/s3` for example) + +Example for Caddy web server +``` +redir /s3 /s3/ +handle_path /s3/* { + reverse_proxy localhost:8333 { + header_up X-Forwarded-Prefix /s3 + } +} +``` + ### Execute commands copy files From 54899815a8d9d99836887ff638968503a42374b8 Mon Sep 17 00:00:00 2001 From: Max! Date: Mon, 14 Oct 2024 12:58:31 +0200 Subject: [PATCH 049/149] fix typo in helm command --- Kubernetes-Backups-and-Recovery-with-K8up.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Kubernetes-Backups-and-Recovery-with-K8up.md b/Kubernetes-Backups-and-Recovery-with-K8up.md index f071bc0..f9ccc42 100644 --- a/Kubernetes-Backups-and-Recovery-with-K8up.md +++ b/Kubernetes-Backups-and-Recovery-with-K8up.md @@ -60,7 +60,7 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 5. Install k8up ```bash - repo add k8up-io https://k8up-io.github.io/k8up + helm repo add k8up-io https://k8up-io.github.io/k8up helm repo update kubectl apply -f https://github.com/k8up-io/k8up/releases/download/k8up-4.4.3/k8up-crd.yaml From ab43ab0b2f06441fd8f6f9d0269d878122c019a3 Mon Sep 17 00:00:00 2001 From: Max! Date: Wed, 16 Oct 2024 19:39:02 +0200 Subject: [PATCH 050/149] update docs to work with v3.77 --- Kubernetes-Backups-and-Recovery-with-K8up.md | 210 +++++++++++++------ 1 file changed, 146 insertions(+), 64 deletions(-) diff --git a/Kubernetes-Backups-and-Recovery-with-K8up.md b/Kubernetes-Backups-and-Recovery-with-K8up.md index f9ccc42..332ee29 100644 --- a/Kubernetes-Backups-and-Recovery-with-K8up.md +++ b/Kubernetes-Backups-and-Recovery-with-K8up.md @@ -62,7 +62,7 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 ```bash helm repo add k8up-io https://k8up-io.github.io/k8up helm repo update - + kubectl apply -f https://github.com/k8up-io/k8up/releases/download/k8up-4.4.3/k8up-crd.yaml helm install k8up k8up-io/k8up ``` @@ -84,11 +84,12 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 export PATH=$PATH:$HOME/minio-binaries/ ``` -2. Clone repo and cd to the helm dir +2. Download SeaweedFS, unzip, then cd to the helm dir ```bash - git clone https://github.com/seaweedfs/seaweedfs - cd seaweedfs/k8s/charts/seaweedfs + wget https://github.com/seaweedfs/seaweedfs/archive/refs/tags/3.77.zip + unzip 3.77.zip + cd seaweedfs-3.77/k8s/charts/seaweedfs ``` 3. Create a minimal values file for the Seaweedfs deployment which adds annotations for K8up. @@ -96,84 +97,129 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 ```bash /bin/cat << EOF > test-values.yaml master: + enabled: true data: type: "persistentVolumeClaim" - size: "5Gi" + size: "10G" storageClass: "local-path" annotations: "k8up.io/backup": "true" + livenessProbe: + periodSeconds: 5 + readinessProbe: + periodSeconds: 5 + volume: + enabled: true + readMode: proxy + dataDirs: + - name: data + type: "persistentVolumeClaim" + size: "10G" + storageClass: "local-path" + annotations: + "k8up.io/backup": "true" + maxVolumes: 0 + idx: {} + livenessProbe: + periodSeconds: 5 + readinessProbe: + periodSeconds: 5 + + filer: + enabled: true + encryptVolumeData: true + enablePVC: true + storage: 10Gi + defaultReplicaPlacement: "000" data: type: "persistentVolumeClaim" - size: "5Gi" + size: "10G" storageClass: "local-path" annotations: "k8up.io/backup": "true" - filer: - enablePVC: true - storage: 5Gi - data: - type: "persistentVolumeClaim" - size: "5Gi" - storageClass: "local-path" - annotations: s3: enabled: true + enableAuth: true port: 8333 httpsPort: 0 allowEmptyFolder: false - domainName: "" - enableAuth: false - skipAuthSecretCreation: false - auditLogConfig: {} + createBuckets: + - name: shared + anonymousRead: false + livenessProbe: + periodSeconds: 5 + readinessProbe: + periodSeconds: 5 + + s3: + enabled: false + cosi: + enabled: false EOF ``` -4. Deploy via Helm +4. Deploy via Helm (takes longer on slow drives) ```bash helm install seaweedfs . -f test-values.yaml --wait ``` -5. Expose the S3 endpoint +5. Expose the filer service via a LoadBalancer (servicelb in k3s). This will let us view the admin UI as well as reach the S3 endpoint during the demo. ```bash /bin/cat << EOF > service.yaml apiVersion: v1 kind: Service metadata: - name: swfs-s3-nodeport labels: - "app.kubernetes.io/name": "seaweedfs" + app.kubernetes.io/component: filer + app.kubernetes.io/instance: seaweedfs + app.kubernetes.io/name: seaweedfs + name: seaweedfs-filer-lb spec: - type: NodePort ports: - - port: 8333 - nodePort: 30000 + - name: swfs-filer + port: 8888 protocol: TCP + targetPort: 8888 + - name: swfs-filer-grpc + port: 18888 + protocol: TCP + targetPort: 18888 + - name: swfs-s3 + port: 8333 + protocol: TCP + targetPort: 8333 + - name: metrics + port: 9327 + protocol: TCP + targetPort: 9327 selector: - "app.kubernetes.io/name": "seaweedfs" + app.kubernetes.io/component: filer + app.kubernetes.io/name: seaweedfs + type: LoadBalancer EOF ``` - -6. Export your NodeIP as an env var + +6. Export your LoadBalancer IP address as an env var ```bash export NODE_IP="" ``` 7. Create an alias for your server using your S3 CLI tool: - + - You can find the `admin_access_key_id` and `admin_secret_access_key` values in the secret `seaweedfs-s3-secret` ```bash - mc alias set seaweedfs http://$NODE_IP:30000 $admin_access_key_id $admin_secret_access_key + mc alias set seaweedfs http://$NODE_IP:8333 $admin_access_key_id $admin_secret_access_key ``` 8. Create a bucket that will hold our demo data ```bash - mc mb seaweedfs s3://backups + mc mb seaweedfs/backups ``` 8. Add some data to the bucket @@ -181,19 +227,27 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 ```bash mc cp ./some-file seaweedfs/backups/ ``` - + +9. Verify its there + + ```bash + mc ls seaweedfs/backups + ``` + +10. Open the Web UI at http://$NODE_IP:8888 in a browser to view or add more data. +

Configure scheduled backups of SeaweedFS to B2

1. Create a secret containing your external S3 credentials - You will need to get these from your provider (Backblaze, Wasabi etc..): - + ```bash export ACCESS_KEY_ID=$(echo -n "" | base64) - + export ACCESS_SECRET_KEY=$(echo -n "" |base64) ``` - + ```bash /bin/cat << EOF > backblaze-secret.yaml apiVersion: v1 @@ -208,15 +262,15 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 kubectl apply -f backblaze-secret.yaml ``` - + 2. Create a secret containing a random password for restic - - Generate a password and base64 encode it. - + - Generate a password. + ```bash - export RESTIC_PASS=$(openssl rand -base64 32) + export RESTIC_PASS="" ``` - + - Create a secret manifest ```bash @@ -226,11 +280,11 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 metadata: name: restic-repo type: Opaque - data: + stringData: "password": "$RESTIC_PASS" EOF ``` - + - Create the secret ```bash @@ -244,9 +298,9 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 ```bash export BACKUP_S3_URL="" export BACKUP_S3_BUCKET="" - ``` + ``` - - Create a manifest for the backup + - Create a manifest for the backup ```bash /bin/cat << EOF > backup.yaml @@ -281,7 +335,7 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 EOF ``` - - Create the backup + - Create the backup and let it run ```bash kubectl apply -f backup.yaml @@ -289,12 +343,14 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034

Restore SeaweedFS from B2 backups

-1. Uninstall SeaweedFS and delete your scheduled backup +1. Uninstall SeaweedFS, delete the PVCs, secrets, and scheduled backup ```bash - k delete -f manifests.yaml - kubectl delete -f backup.yaml + helm uninstall seaweedfs + kubectl delete pvc data-default-seaweedfs-master-0 + kubectl delete pvc data-filer-seaweedfs-filer-0 + kubectl delete pvc data-seaweedfs-volume-0 ``` 2. Create PVCs to hold our restored data @@ -344,7 +400,7 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 storage: 10Gi EOF ``` - + - Create the PVCs ```bash @@ -356,7 +412,7 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 ```bash # the password used in your restic-repo secret export RESTIC_PASSWORD="" - + # Your S3 credentials export AWS_ACCESS_KEY_ID="" export AWS_SECRET_ACCESS_KEY="" @@ -378,9 +434,9 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 3 snapshots ``` -4. Use the K8up CLI or a declarative setup to restore data to the PVC. You will need to do this for each PVC that needs to be restored - - - Example manifest for a S3-to-PVC restore job which uses the restic snapshots shown above. +4. Use the K8up CLI or a declarative setup to restore data to the PVC. You will need to do this for each PVC that needs to be restored + + - Example manifest for a S3-to-PVC restore job which uses the restic snapshots shown above. ```bash /bin/cat << EOF > s3-to-pvc.yaml @@ -460,7 +516,7 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 ```bash kubectl apply -f s3-to-pvc.yaml ``` - + 5. Re-deploy Seaweedfs from the existing PVCs - Create a manifest that targets the PVCs we created @@ -468,44 +524,70 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 ```bash /bin/cat << EOF > restore-values.yaml master: + enabled: true data: type: "existingClaim" claimName: "swfs-master-data" + livenessProbe: + periodSeconds: 5 + readinessProbe: + periodSeconds: 5 + volume: - data: - type: "existingClaim" - claimName: "swfs-volume-data" + enabled: true + readMode: proxy + dataDirs: + - name: data + type: "existingClaim" + claimName: "swfs-volume-data" + maxVolumes: 0 + idx: {} + livenessProbe: + periodSeconds: 5 + readinessProbe: + periodSeconds: 5 + filer: + enabled: true + encryptVolumeData: true enablePVC: true + storage: 10Gi + defaultReplicaPlacement: "000" data: type: "existingClaim" claimName: "swfs-filer-data" s3: enabled: true + enableAuth: false port: 8333 httpsPort: 0 allowEmptyFolder: false - domainName: "" - enableAuth: false - skipAuthSecretCreation: false - auditLogConfig: {} + livenessProbe: + periodSeconds: 5 + readinessProbe: + periodSeconds: 5 + + s3: + enabled: false + cosi: + enabled: false EOF ``` - Deploy via Helm - + ```bash helm install seaweedfs . -f restore-values.yaml --wait ``` 7. Update your alias for your server: - + - get the `admin_access_key_id` and `admin_secret_access_key` from the secret `seaweedfs-s3-secret` ```bash mc alias set seaweedfs http://$NODE_IP:30000 $admin_access_key_id $admin_secret_access_key ``` - + - View for your data: ```bash From fc9a769ed86f6f0c86d8216cbf7e6bef7a6eae03 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Tue, 22 Oct 2024 08:48:12 -0700 Subject: [PATCH 051/149] Updated Path Specific Configuration (markdown) --- Path-Specific-Configuration.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Path-Specific-Configuration.md b/Path-Specific-Configuration.md index 3081f63..33e23a6 100644 --- a/Path-Specific-Configuration.md +++ b/Path-Specific-Configuration.md @@ -99,6 +99,8 @@ Usage of fs.configure: assign writes with this ttl -volumeGrowthCount int the number of physical volumes to add if no writable volumes + -worm + worm mode, If true, a file can only be changed once, after which it becomes readonly and undeletable, see https://en.wikipedia.org/wiki/Write_once_read_many ``` From 0586615ff6e8a7fabd126ef8c7ba140635f7459c Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Tue, 22 Oct 2024 09:06:49 -0700 Subject: [PATCH 052/149] Updated Path Specific Configuration (markdown) --- Path-Specific-Configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Path-Specific-Configuration.md b/Path-Specific-Configuration.md index 33e23a6..0ba50a2 100644 --- a/Path-Specific-Configuration.md +++ b/Path-Specific-Configuration.md @@ -100,7 +100,7 @@ Usage of fs.configure: -volumeGrowthCount int the number of physical volumes to add if no writable volumes -worm - worm mode, If true, a file can only be changed once, after which it becomes readonly and undeletable, see https://en.wikipedia.org/wiki/Write_once_read_many + write-once-read-many, written files are readonly ``` From c4e1941d0cc18e06a7509d50c22ea4f19dcd7bb5 Mon Sep 17 00:00:00 2001 From: Clouder Date: Sun, 27 Oct 2024 09:16:50 +0800 Subject: [PATCH 053/149] fix(docs): minor mistake --- Production-Setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Production-Setup.md b/Production-Setup.md index 4abf9a9..36948f1 100644 --- a/Production-Setup.md +++ b/Production-Setup.md @@ -71,7 +71,7 @@ weed master -mdir=/data/seaweedfs/master -peers=ip1:9333,ip2:9333,ip3:9333 -ip=i Additional notes: * Depending on the available disk space on each volume server, the master may need to reduce maximum volume size, e.g., add `-volumeSizeLimitMB=1024`. This will ensure -each volume has several volumes. On the note, you can't change `volumeSizeLimitMB` later. +each volume server has several volumes. On the note, you can't change `volumeSizeLimitMB` later. * Since it is for production, you may also want to add `-metrics.address=`. See [[System Metrics]]. From 1788981bf8e48f2338b22e5a2a51480c87342c5f Mon Sep 17 00:00:00 2001 From: Yichuan Sun <582582zxcv@163.com> Date: Tue, 5 Nov 2024 22:30:50 +0800 Subject: [PATCH 054/149] Fix a wrong parameter --- s3cmd-with-SeaweedFS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/s3cmd-with-SeaweedFS.md b/s3cmd-with-SeaweedFS.md index c81eea9..a7edfb2 100644 --- a/s3cmd-with-SeaweedFS.md +++ b/s3cmd-with-SeaweedFS.md @@ -30,7 +30,7 @@ Make sure the `.s3cfg` file has these values # Setup endpoint host_base = localhost:8333 host_bucket = localhost:8333 -use_https = No +use_https = False # Enable S3 v4 signature APIs signature_v2 = False ``` From 618baff75785be2b02d3f9b93ec5c9d5f5f9d1f7 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Thu, 14 Nov 2024 08:29:07 -0800 Subject: [PATCH 055/149] Updated Production Setup (markdown) --- Production-Setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Production-Setup.md b/Production-Setup.md index 36948f1..a3266f5 100644 --- a/Production-Setup.md +++ b/Production-Setup.md @@ -156,7 +156,7 @@ Follow [[Amazon S3 API]] to generate a json config file, to assign accessKey and Start s3 together with filer. This avoids the setup for s3 to support multiple filers. ``` -weed filer -s3 -s3.config= -port=8333 +weed filer -s3 -s3.config= -s3.port=8333 ``` The endpoint is `http://:8333`. From 7e1fb49c35b89a4b5c1848a7c74d47bbcbe52e86 Mon Sep 17 00:00:00 2001 From: zemul Date: Mon, 18 Nov 2024 13:23:15 +0800 Subject: [PATCH 056/149] Updated Optimization (markdown) --- Optimization.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Optimization.md b/Optimization.md index 8ed14ea..2bffc68 100644 --- a/Optimization.md +++ b/Optimization.md @@ -50,6 +50,7 @@ copy_1 = 7 # create 1 x 7 = 7 actual volumes copy_2 = 6 # create 2 x 6 = 12 actual volumes copy_3 = 3 # create 3 x 3 = 9 actual volumes copy_other = 1 # create n x 1 = n actual volumes +threshold = 0.9 # create threshold ``` ## Increase concurrent reads From 55838d8ab08a41f045ed2c59e9e19f0c14aed70f Mon Sep 17 00:00:00 2001 From: chrislu Date: Mon, 18 Nov 2024 22:47:15 -0800 Subject: [PATCH 057/149] java 3.80 --- Hadoop-Benchmark.md | 2 +- Hadoop-Compatible-File-System.md | 24 ++++++++++++------------ Run-Blob-Storage-on-Public-Internet.md | 2 +- Run-Presto-on-SeaweedFS.md | 10 +++++----- SeaweedFS-Java-Client.md | 4 ++-- run-HBase-on-SeaweedFS.md | 2 +- run-Spark-on-SeaweedFS.md | 14 +++++++------- 7 files changed, 29 insertions(+), 29 deletions(-) diff --git a/Hadoop-Benchmark.md b/Hadoop-Benchmark.md index 7f2e352..f6ac2fb 100644 --- a/Hadoop-Benchmark.md +++ b/Hadoop-Benchmark.md @@ -26,7 +26,7 @@ Then get the seaweedfs hadoop client jar. ``` cd share/hadoop/common/lib/ -wget https://repo1.maven.org/maven2/com/seaweedfs/seaweedfs-hadoop2-client/3.71/seaweedfs-hadoop2-client-3.71.jar +wget https://repo1.maven.org/maven2/com/seaweedfs/seaweedfs-hadoop2-client/3.80/seaweedfs-hadoop2-client-3.80.jar ``` # TestDFSIO Benchmark diff --git a/Hadoop-Compatible-File-System.md b/Hadoop-Compatible-File-System.md index e96d56a..0af4eb3 100644 --- a/Hadoop-Compatible-File-System.md +++ b/Hadoop-Compatible-File-System.md @@ -10,12 +10,12 @@ $ mvn install # build for hadoop2 $cd $GOPATH/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2 $ mvn package -$ ls -al target/seaweedfs-hadoop2-client-3.71.jar +$ ls -al target/seaweedfs-hadoop2-client-3.80.jar # build for hadoop3 $cd $GOPATH/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3 $ mvn package -$ ls -al target/seaweedfs-hadoop3-client-3.71.jar +$ ls -al target/seaweedfs-hadoop3-client-3.80.jar ``` Maven @@ -23,7 +23,7 @@ Maven com.seaweedfs seaweedfs-hadoop3-client - 3.71 + 3.80 or @@ -31,23 +31,23 @@ or com.seaweedfs seaweedfs-hadoop2-client - 3.71 + 3.80 ``` Or you can download the latest version from MavenCentral * https://central.sonatype.com/artifact/com.seaweedfs/seaweedfs-hadoop2-client/versions - * [seaweedfs-hadoop2-client-3.71.jar](https://repo1.maven.org/maven2/com/seaweedfs/seaweedfs-hadoop2-client/3.71/seaweedfs-hadoop2-client-3.71.jar) + * [seaweedfs-hadoop2-client-3.80.jar](https://repo1.maven.org/maven2/com/seaweedfs/seaweedfs-hadoop2-client/3.80/seaweedfs-hadoop2-client-3.80.jar) * https://central.sonatype.com/artifact/com.seaweedfs/seaweedfs-hadoop3-client/versions - * [seaweedfs-hadoop3-client-3.71.jar](https://repo1.maven.org/maven2/com/seaweedfs/seaweedfs-hadoop3-client/3.71/seaweedfs-hadoop3-client-3.71.jar) + * [seaweedfs-hadoop3-client-3.80.jar](https://repo1.maven.org/maven2/com/seaweedfs/seaweedfs-hadoop3-client/3.80/seaweedfs-hadoop3-client-3.80.jar) # Test SeaweedFS on Hadoop Suppose you are getting a new Hadoop installation. Here are the minimum steps to get SeaweedFS to run. -You would need to start a weed filer first, build the seaweedfs-hadoop2-client-3.71.jar -or seaweedfs-hadoop3-client-3.71.jar, and do the following: +You would need to start a weed filer first, build the seaweedfs-hadoop2-client-3.80.jar +or seaweedfs-hadoop3-client-3.80.jar, and do the following: ``` # optionally adjust hadoop memory allocation @@ -60,12 +60,12 @@ $ echo "" > etc/hadoop/mapred-site.xml # on hadoop2 $ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \ -Dfs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \ - -libjars ./seaweedfs-hadoop2-client-3.71.jar \ + -libjars ./seaweedfs-hadoop2-client-3.80.jar \ -ls / # or on hadoop3 $ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \ -Dfs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \ - -libjars ./seaweedfs-hadoop3-client-3.71.jar \ + -libjars ./seaweedfs-hadoop3-client-3.80.jar \ -ls / ``` @@ -112,9 +112,9 @@ $ bin/hadoop classpath # Copy SeaweedFS HDFS client jar to one of the folders $ cd ${HADOOP_HOME} # for hadoop2 -$ cp ./seaweedfs-hadoop2-client-3.71.jar share/hadoop/common/lib/ +$ cp ./seaweedfs-hadoop2-client-3.80.jar share/hadoop/common/lib/ # or for hadoop3 -$ cp ./seaweedfs-hadoop3-client-3.71.jar share/hadoop/common/lib/ +$ cp ./seaweedfs-hadoop3-client-3.80.jar share/hadoop/common/lib/ ``` Now you can do this: diff --git a/Run-Blob-Storage-on-Public-Internet.md b/Run-Blob-Storage-on-Public-Internet.md index 3e3f77a..05b8926 100644 --- a/Run-Blob-Storage-on-Public-Internet.md +++ b/Run-Blob-Storage-on-Public-Internet.md @@ -14,7 +14,7 @@ We will use 2 servers. Server 1 will host master, 2x volumes (2 disks, one volum # todo: use 2 step build process, copy over weed binary to fresh container (do not need curl and tar at runtime) FROM alpine RUN apk update && apk add wget tar -RUN wget https://github.com/seaweedfs/seaweedfs/releases/download/3.71/linux_amd64_large_disk.tar.gz +RUN wget https://github.com/seaweedfs/seaweedfs/releases/download/3.80/linux_amd64_large_disk.tar.gz RUN tar -xf linux_amd64_large_disk.tar.gz RUN chmod +x weed RUN mv weed /usr/bin/ diff --git a/Run-Presto-on-SeaweedFS.md b/Run-Presto-on-SeaweedFS.md index f214f31..f699f4a 100644 --- a/Run-Presto-on-SeaweedFS.md +++ b/Run-Presto-on-SeaweedFS.md @@ -5,10 +5,10 @@ The installation steps are divided into 2 steps: * https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+Administration ### Configure Hive Metastore to support SeaweedFS -1. Copy the seaweedfs-hadoop2-client-3.71.jar to hive lib directory,for example: +1. Copy the seaweedfs-hadoop2-client-3.80.jar to hive lib directory,for example: ``` -cp seaweedfs-hadoop2-client-3.71.jar /opt/hadoop/share/hadoop/common/lib/ -cp seaweedfs-hadoop2-client-3.71.jar /opt/hive-metastore/lib/ +cp seaweedfs-hadoop2-client-3.80.jar /opt/hadoop/share/hadoop/common/lib/ +cp seaweedfs-hadoop2-client-3.80.jar /opt/hive-metastore/lib/ ``` 2. Modify core-site.xml modify core-site.xml to support SeaweedFS, 30888 is the filer port @@ -50,9 +50,9 @@ metastore.thrift.port is the access port exposed by the Hive Metadata service it Follow instructions for installation of Presto: * https://prestosql.io/docs/current/installation/deployment.html ### Configure Presto to support SeaweedFS -1. Copy the seaweedfs-hadoop2-client-3.71.jar to Presto directory,for example: +1. Copy the seaweedfs-hadoop2-client-3.80.jar to Presto directory,for example: ``` -cp seaweedfs-hadoop2-client-3.71.jar /opt/presto-server-347/plugin/hive-hadoop2/ +cp seaweedfs-hadoop2-client-3.80.jar /opt/presto-server-347/plugin/hive-hadoop2/ ``` 2. Modify core-site.xml diff --git a/SeaweedFS-Java-Client.md b/SeaweedFS-Java-Client.md index 0f7414f..b32a13c 100644 --- a/SeaweedFS-Java-Client.md +++ b/SeaweedFS-Java-Client.md @@ -13,7 +13,7 @@ $ mvn install Gradle ```gradle -implementation 'com.github.chrislusf:seaweedfs-client:3.71' +implementation 'com.github.chrislusf:seaweedfs-client:3.80' ``` Maven @@ -21,7 +21,7 @@ Maven com.seaweedfs seaweedfs-client - 3.71 + 3.80 ``` Or you can download the latest version from MavenCentral diff --git a/run-HBase-on-SeaweedFS.md b/run-HBase-on-SeaweedFS.md index c738a72..a73a3d4 100644 --- a/run-HBase-on-SeaweedFS.md +++ b/run-HBase-on-SeaweedFS.md @@ -1,7 +1,7 @@ # Installation for HBase Two steps to run HBase on SeaweedFS -1. Copy the seaweedfs-hadoop2-client-3.71.jar to `${HBASE_HOME}/lib` +1. Copy the seaweedfs-hadoop2-client-3.80.jar to `${HBASE_HOME}/lib` 1. And add the following 2 properties in `${HBASE_HOME}/conf/hbase-site.xml` ``` diff --git a/run-Spark-on-SeaweedFS.md b/run-Spark-on-SeaweedFS.md index b05ab60..555b07a 100644 --- a/run-Spark-on-SeaweedFS.md +++ b/run-Spark-on-SeaweedFS.md @@ -11,12 +11,12 @@ To make these files visible to Spark, set HADOOP_CONF_DIR in $SPARK_HOME/conf/sp ## installation not inheriting from Hadoop cluster configuration -Copy the seaweedfs-hadoop2-client-3.71.jar to all executor machines. +Copy the seaweedfs-hadoop2-client-3.80.jar to all executor machines. Add the following to spark/conf/spark-defaults.conf on every node running Spark ``` -spark.driver.extraClassPath=/path/to/seaweedfs-hadoop2-client-3.71.jar -spark.executor.extraClassPath=/path/to/seaweedfs-hadoop2-client-3.71.jar +spark.driver.extraClassPath=/path/to/seaweedfs-hadoop2-client-3.80.jar +spark.executor.extraClassPath=/path/to/seaweedfs-hadoop2-client-3.80.jar ``` And modify the configuration at runtime: @@ -37,8 +37,8 @@ And modify the configuration at runtime: 1. change the spark-defaults.conf ``` -spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.71.jar -spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.71.jar +spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.80.jar +spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.80.jar spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem ``` @@ -81,8 +81,8 @@ spark.history.fs.cleaner.enabled=true spark.history.fs.logDirectory=seaweedfs://localhost:8888/spark2-history/ spark.eventLog.dir=seaweedfs://localhost:8888/spark2-history/ -spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.71.jar -spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.71.jar +spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.80.jar +spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.80.jar spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888 ``` From bb488988866b7f9e79cd7da719ab0ba1f5110d57 Mon Sep 17 00:00:00 2001 From: Max Denushev Date: Thu, 21 Nov 2024 20:22:45 +0300 Subject: [PATCH 058/149] Add fstab config example --- FUSE-Mount.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/FUSE-Mount.md b/FUSE-Mount.md index 4219fb7..2c168a7 100644 --- a/FUSE-Mount.md +++ b/FUSE-Mount.md @@ -50,6 +50,12 @@ mount -t weed fuse /mnt -o "filer='192.168.0.1:8888,192.168.0.2:8888',filer.path Now you can operate the SeaweedFS files, browsing or modifying directories and files, in local file system. To unmount, just shut it down the "weed mount". +#### Mount with fstab + +``` +weed#fuse /mnt fuse _netdev,filer='192.168.0.1:8888',filer.path=/ 0 0 +``` + #### Mount outside of a SeaweedFS cluster In addition to connecting to filer server, `weed mount` also directly connects to volume servers directly for better performance. From e61a7b6314c1bd39756a960cf25b2fe8f9ad8b76 Mon Sep 17 00:00:00 2001 From: dsyscms <132198855+dsyscms@users.noreply.github.com> Date: Mon, 25 Nov 2024 17:11:43 +0800 Subject: [PATCH 059/149] fix filer s3 argument --- S3-API-FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/S3-API-FAQ.md b/S3-API-FAQ.md index e61cc53..5a9d203 100644 --- a/S3-API-FAQ.md +++ b/S3-API-FAQ.md @@ -43,7 +43,7 @@ Having separate LevelDB instance, or separate SQL tables, will help to isolate t Due to the semantics of the S3 API, empty directories (aka prefixes) aren't shown. However, an entry is still stored in the filer metadata store. When workload access patterns create many unique directories and then remove all the objects inside those directories, the filer metadata store can grow unbounded with orphaned directories. These directories are visible in the filer metadata store itself, but not using the S3 API. -If the filer argument `-allowEmptyFolder=false` is set, the orphaned directories are cleaned up during list requests for non bucket-level directories. Normally this works well, but if the workload never performs a list operation, the orphaned directories may never be cleaned up. To force cleanup, simply list an existing, non bucket-level directory. +If the filer argument `-s3.allowEmptyFolder=false` is set, the orphaned directories are cleaned up during list requests for non bucket-level directories. Normally this works well, but if the workload never performs a list operation, the orphaned directories may never be cleaned up. To force cleanup, simply list an existing, non bucket-level directory. Example using rclone: From bdc18edb4e6fc245e2608ad1405c23d68c29cc5e Mon Sep 17 00:00:00 2001 From: Michael Spradling Date: Tue, 3 Dec 2024 22:29:20 -0500 Subject: [PATCH 060/149] Update replication section of wiki for new format. --- Replication.md | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/Replication.md b/Replication.md index c6ec1bf..d888d30 100644 --- a/Replication.md +++ b/Replication.md @@ -83,17 +83,7 @@ No change to Submitting, Reading, and Deleting files. *Note: This subject to change.* -Value | Meaning ----|--- -000 | no replication, just one copy -001 | replicate once on the same rack -010 | replicate once on a different rack in the same data center -100 | replicate once on a different data center -200 | replicate twice on two other different data center -110 | replicate once on a different rack, and once on a different data center -... | ... - -So if the replication type is xyz +The replication type is defined by a 3 digit string as follows: Column | Meaning ---|--- @@ -101,9 +91,18 @@ Column | Meaning **y** | number of replica in other racks in the same data center **z** | number of replica in other servers in the same rack -x,y,z each can be 0, 1, or 2. So there are 9 possible replication types, and can be easily extended. -Each replication type will physically create x+y+z+1 copies of volume data files. +Each digit in the replication string can have a value of 0, 1, or 2, representing the number of additional copies of the data to be maintained in the seaweed cluster beyond the original. Data remains readable as long as at least one replica is accessible. However, writes are permitted only if the number of replicas is achievable based on the configuration values. With these parameters, there are 27 possible replication configurations. Each configuration physically creates x+y+z+1 copies of the data volume. +Here are some possible replication configuration examples and what they mean: +Value | Meaning +---|--- +000 | no replication, just one copy +001 | replicate once on the same rack +010 | replicate once on a different rack in the same data center +100 | replicate once on a different data center +200 | replicate twice on two other different data center +110 | replicate once on a different rack within the same data center and once on a different data center +... | ... ## Allocate File Key on specific data center From a6334674e7ab860295fb56a78cee7b1880669253 Mon Sep 17 00:00:00 2001 From: Michael Spradling Date: Tue, 3 Dec 2024 22:52:10 -0500 Subject: [PATCH 061/149] Update replication configuration for new replication type 256) --- Replication.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/Replication.md b/Replication.md index d888d30..b6e7883 100644 --- a/Replication.md +++ b/Replication.md @@ -91,7 +91,8 @@ Column | Meaning **y** | number of replica in other racks in the same data center **z** | number of replica in other servers in the same rack -Each digit in the replication string can have a value of 0, 1, or 2, representing the number of additional copies of the data to be maintained in the seaweed cluster beyond the original. Data remains readable as long as at least one replica is accessible. However, writes are permitted only if the number of replicas is achievable based on the configuration values. With these parameters, there are 27 possible replication configurations. Each configuration physically creates x+y+z+1 copies of the data volume. +The replication string represents the additional number of copies of data that will be maintained by the seaweed cluster beyond the original data volume. Data remains readable as long as at least on replica is accessible. However, writes are permitted only if the number of replicas is achievable based on the replication configuration string. The max value that can be specified is '255'. The total number of copies of this volume will be the sum of the digits + 1. The original data is not counted. + Here are some possible replication configuration examples and what they mean: Value | Meaning @@ -99,10 +100,12 @@ Value | Meaning 000 | no replication, just one copy 001 | replicate once on the same rack 010 | replicate once on a different rack in the same data center +052 | replicate to five different racks within the same data center and two different volumes within the same rack 100 | replicate once on a different data center 200 | replicate twice on two other different data center 110 | replicate once on a different rack within the same data center and once on a different data center ... | ... +255 | replicate twice on two different data centers, five different racks and 5 different volumes servers with respect to where the original volume exists. ## Allocate File Key on specific data center From af21bceffdc4cf837f3ca3a7784f224cb6309fc9 Mon Sep 17 00:00:00 2001 From: Michael Spradling Date: Wed, 4 Dec 2024 22:40:46 -0500 Subject: [PATCH 062/149] typo --- Filer-Store-Replication.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Filer-Store-Replication.md b/Filer-Store-Replication.md index dd57198..55dfde9 100644 --- a/Filer-Store-Replication.md +++ b/Filer-Store-Replication.md @@ -27,7 +27,7 @@ Knowing all the peers, one filer will keep its own metadata updated: This is tightly related to FUSE Mount, which streams filer meta data changes from one filer. When using multiple filers but without peer file metadata updates, a FUSE mount can only see the changes applied to the connected filer. -So aggregating metadata updates form its peers is required when the filers are using either shared or dedicated filer stores. +So aggregating metadata updates from its peers is required when the filers are using either shared or dedicated filer stores. ``` FUSE mount <----> filer1 -- filer2 From 1d84deb0a80c8f3643938e7f92a440999fc80997 Mon Sep 17 00:00:00 2001 From: jovial <41611371+JovialYip@users.noreply.github.com> Date: Thu, 5 Dec 2024 15:21:14 +0800 Subject: [PATCH 063/149] Updated Master Server API (markdown) --- Master-Server-API.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Master-Server-API.md b/Master-Server-API.md index 4b96549..a28e05b 100644 --- a/Master-Server-API.md +++ b/Master-Server-API.md @@ -235,7 +235,7 @@ curl "http://localhost:9333/dir/status?pretty=y" ### Check Volume Status ``` -curl localhost:9333/vol/status?pretty=y +curl "localhost:9333/vol/status?pretty=y" { "Version": "30GB 2.24 ", "Volumes": { From dedc5d964814616ecd7b714960efc93187d38c38 Mon Sep 17 00:00:00 2001 From: Lisandro Pin Date: Thu, 19 Dec 2024 18:20:54 +0100 Subject: [PATCH 064/149] Document new topology-aware EC rebalancing logic. --- Erasure-Coding-for-warm-storage.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/Erasure-Coding-for-warm-storage.md b/Erasure-Coding-for-warm-storage.md index 41e378a..71071a8 100644 --- a/Erasure-Coding-for-warm-storage.md +++ b/Erasure-Coding-for-warm-storage.md @@ -47,7 +47,7 @@ The scripts have 3 steps related to erasure coding. ### Erasure Encode Sealed Data `ec.encode` command will find volumes that are almost full and has been stale for a period of time. -The default command is `ec.encode -fullPercent=95 -quietFor=1h`. It will find volumes at least 95% of the maximum volume size, which is usually 30GB, and have no updates for 1 hour. +The default command is `ec.encode -fullPercent=95 -quietFor=1h -rebalance`. It will find volumes at least 95% of the maximum volume size, which is usually 30GB, and have no updates for 1 hour. Once the encoding is completed, EC shards are re-balanced; see [EC data balancing](#ec-data-balancing) below. Note that if you have any collections, i.e. because you're using s3 where every bucket is a collection you explicitly need to specify the collection for the ec.encode command for them to be erasure coded `ec.encode -collection="collection" -fullPercent=95 -quietFor=1h`. @@ -67,6 +67,14 @@ With servers added or removed, some data shards may not be laid out optimally. F The default command is `ec.balance -force`. It will try to spread the data shards evenly to minimize the data shard loss risk. +EC shard re-balancing happens in three steps: + + 1. Duplicate EC shards for the same volume + server are deleted. + 2. EC shards are balanced across volumes for all racks. + 3. EC shards are balanced across volumes for individual racks. + +Destination volume/racks for EC shards are selected based on capacity, favoring those with less preexisting shards in order to ensure an uniform distribution. Additionally, EC shards selection obey default [replica placement settings](Replication#the-meaning-of-replication-type) for the master server. + ## How the read works? When all data shards are online, the read for one file key are assigned to one volume server (A) that has at least one data shard for the volume. Server A will read its copy of index file, and locate the volume server (B), and read from server B for the file key. From cd2ce02f00b41ddd011cb0d4da95c1c3fdf2c3d8 Mon Sep 17 00:00:00 2001 From: MGSousa <31368750+MGSousa@users.noreply.github.com> Date: Wed, 25 Dec 2024 11:10:34 +0000 Subject: [PATCH 065/149] Updated FAQ (markdown) --- FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/FAQ.md b/FAQ.md index b4e673d..5c3401d 100644 --- a/FAQ.md +++ b/FAQ.md @@ -168,7 +168,7 @@ If you **must** have custom gRPC ports, you can specify a custom gRPC port when For example: ``` weed master -port=9333 -port.grpc=9444 - weed volume -port=8080 -port.grpc=8444 -msever=localhost:9333.9444 + weed volume -port=8080 -port.grpc=8444 -mserver=localhost:9333.9444 weed filer -port=8888 -port.grpc=9999 -master=localhost:9333.9444 weed shell -filer=localhst:8888.9999 -master=localhost:9333.9444 weed mount -dir=mm -filer=localhst:8888.9999 From 3d475633393446e9c014091a5be66ed2aa1e7195 Mon Sep 17 00:00:00 2001 From: Tom Crasset <25140344+tcrasset@users.noreply.github.com> Date: Thu, 9 Jan 2025 17:05:13 +0100 Subject: [PATCH 066/149] Updated Getting Started (markdown) --- Getting-Started.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Getting-Started.md b/Getting-Started.md index 9fe0785..a407826 100644 --- a/Getting-Started.md +++ b/Getting-Started.md @@ -85,7 +85,7 @@ docker run -p 8080:8080 -p 18080:18080 --name volume --link master chrislusf/sea But with Compose it's easiest. To startup just run: ``` -docker-compose -f docker/seaweedfs-compose.yml -p seaweedfs up +docker compose -f docker/seaweedfs-compose.yml -p seaweedfs up ``` ## Using SeaweedFS in docker From dba464d458f5ca8925cd9a1074bc4795ea237c1d Mon Sep 17 00:00:00 2001 From: Jack Date: Tue, 14 Jan 2025 10:37:25 +0000 Subject: [PATCH 067/149] Since version [3.60](https://github.com/seaweedfs/seaweedfs/commit/7a50762b9ed05f1410a24f52571b49a25cd561eb) without any release notes. --- Cloud-Tier.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Cloud-Tier.md b/Cloud-Tier.md index b45b1a6..552e1f8 100644 --- a/Cloud-Tier.md +++ b/Cloud-Tier.md @@ -82,6 +82,9 @@ Multiple S3 buckets are supported. Usually you just need to configure one backen ### Rclone +> NOTE: Since version 3.60 the rclone backend is removed +> in standard builds to reduce size for common use cases. Use the `_full` weed binary. + Here's an example of a configuration for an Rclone storage backend: ``` From b762401d61dd4087e1bbc09aefb445a17617eecd Mon Sep 17 00:00:00 2001 From: Tom Crasset <25140344+tcrasset@users.noreply.github.com> Date: Wed, 15 Jan 2025 14:08:00 +0100 Subject: [PATCH 068/149] Specifying `/*` is the only way to get it to parse the bucket as a valid bucket --- AWS-IAM-CLI.md | 1 - 1 file changed, 1 deletion(-) diff --git a/AWS-IAM-CLI.md b/AWS-IAM-CLI.md index 8b4aa7e..191fd73 100644 --- a/AWS-IAM-CLI.md +++ b/AWS-IAM-CLI.md @@ -42,7 +42,6 @@ echo ' "s3:List*" ], "Resource": [ - "arn:aws:s3:::EXAMPLE-BUCKET", "arn:aws:s3:::EXAMPLE-BUCKET/*" ] } From ebf83b154456e52814bd41b02a715d3958233d49 Mon Sep 17 00:00:00 2001 From: Tom Crasset <25140344+tcrasset@users.noreply.github.com> Date: Thu, 23 Jan 2025 15:59:46 +0100 Subject: [PATCH 069/149] Updated Filer JWT Use (markdown) --- Filer-JWT-Use.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Filer-JWT-Use.md b/Filer-JWT-Use.md index 71ca200..945ec6d 100644 --- a/Filer-JWT-Use.md +++ b/Filer-JWT-Use.md @@ -1,3 +1,7 @@ +## Introduction + +See [the security overview](https://github.com/seaweedfs/seaweedfs/wiki/Security-Overview#securing-filer-http-with-jwt) for a more complete description. + ## How to get a valid JWT The Filer won't generate JWTs, you will have to configure another service to create a JWT with the same signing key set in the `security.toml` file described on the Security Configuration wiki page. When generating the JWT, ensure that the timeout in seconds matches the timeout set in the `security.toml` file. @@ -9,3 +13,5 @@ The Filer supports receiving a JWT in three different forms: 1. Through the `Authorization: Bearer ` header 2. Via the request's query parameters: http://localhost:8888/buckets/all?jwt=token 3. In an HTTP-only cookie named AT(Access Token) + + From 6de8b61333111ae22057b681340086814406feea Mon Sep 17 00:00:00 2001 From: Pasha Finkelshteyn Date: Wed, 5 Feb 2025 15:31:50 +0100 Subject: [PATCH 070/149] fixes Gradle artifact coordinates in docs --- SeaweedFS-Java-Client.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SeaweedFS-Java-Client.md b/SeaweedFS-Java-Client.md index b32a13c..517d8d4 100644 --- a/SeaweedFS-Java-Client.md +++ b/SeaweedFS-Java-Client.md @@ -13,7 +13,7 @@ $ mvn install Gradle ```gradle -implementation 'com.github.chrislusf:seaweedfs-client:3.80' +implementation 'com.seaweedfs:seaweedfs-client:3.80' ``` Maven From ca4badd2173b4376706b44a98b9a0ae08955f0d7 Mon Sep 17 00:00:00 2001 From: chalet Date: Thu, 6 Feb 2025 17:10:29 +0800 Subject: [PATCH 071/149] Updated Cloud Tier (markdown) --- Cloud-Tier.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Cloud-Tier.md b/Cloud-Tier.md index 552e1f8..91475ea 100644 --- a/Cloud-Tier.md +++ b/Cloud-Tier.md @@ -76,6 +76,7 @@ Multiple S3 buckets are supported. Usually you just need to configure one backen aws_secret_access_key = "" # if empty, loads from the shared credentials file (~/.aws/credentials). region = "us-west-2" bucket = "one_bucket_two" # an existing bucket + force_path_style = "false" # The default value is true. If you are using Alibaba Cloud OSS (path style is no longer supported), please set it to false. storage_class = "STANDARD_IA" ``` From 1e78645034cdb072138e171785ab1403ede050ed Mon Sep 17 00:00:00 2001 From: Tom Crasset <25140344+tcrasset@users.noreply.github.com> Date: Tue, 11 Feb 2025 15:53:30 +0100 Subject: [PATCH 072/149] Updated S3 Nginx Proxy (markdown) --- S3-Nginx-Proxy.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/S3-Nginx-Proxy.md b/S3-Nginx-Proxy.md index a27eb63..917f375 100644 --- a/S3-Nginx-Proxy.md +++ b/S3-Nginx-Proxy.md @@ -2,6 +2,11 @@ It's a common concept to put a proxy in front of S3 that handles requests. Nginx For virtual-hosted style URL buckets, you'll need to add a [wildcard DNS record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) for your S3 subdomain. + +Make sure the config sets the `X-Forwarded-Host` and optionally the `X-Forwarded-Port` if you are using a non-standard port. Otherwise, the AWS Signature v4 will not be correctly computed by the server, as it uses the Host header to compute the signature on the client side. + +Additionally, make sure that `proxy-request-buffering` is `off` (default is `on`), as the proxy will buffer the request, and send the request to the backend as a whole instead of chunked, and again the signature computed by the client side will be different as it would have taken into account the `Transfer-Encoding: chunked` header that is dropped by the proxy when it buffers. + ### Example Nginx config ``` From b70027481c0b8832b208188d59c2cd8443d1adfd Mon Sep 17 00:00:00 2001 From: Tom Crasset <25140344+tcrasset@users.noreply.github.com> Date: Tue, 11 Feb 2025 15:56:06 +0100 Subject: [PATCH 073/149] Updated S3 Nginx Proxy (markdown) --- S3-Nginx-Proxy.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/S3-Nginx-Proxy.md b/S3-Nginx-Proxy.md index 917f375..382649a 100644 --- a/S3-Nginx-Proxy.md +++ b/S3-Nginx-Proxy.md @@ -5,7 +5,7 @@ For virtual-hosted style URL buckets, you'll need to add a [wildcard DNS record] Make sure the config sets the `X-Forwarded-Host` and optionally the `X-Forwarded-Port` if you are using a non-standard port. Otherwise, the AWS Signature v4 will not be correctly computed by the server, as it uses the Host header to compute the signature on the client side. -Additionally, make sure that `proxy-request-buffering` is `off` (default is `on`), as the proxy will buffer the request, and send the request to the backend as a whole instead of chunked, and again the signature computed by the client side will be different as it would have taken into account the `Transfer-Encoding: chunked` header that is dropped by the proxy when it buffers. +Additionally, make sure that `proxy_request_buffering` is `off` (default is `on`), as the proxy will buffer the request, and send the request to the backend as a whole instead of chunked, and again the signature computed by the client side will be different as it would have taken into account the `Transfer-Encoding: chunked` header that is dropped by the proxy when it buffers. ### Example Nginx config @@ -31,11 +31,15 @@ server { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Port $server_port; + proxy_set_header X-Forwarded-Proto $scheme; proxy_connect_timeout 300; proxy_http_version 1.1; proxy_set_header Connection ""; + proxy_request_buffering off; chunked_transfer_encoding off; # If bucket subdomain is not empty, From fd5fb80620b351526ca5857046a97581759683e8 Mon Sep 17 00:00:00 2001 From: Tom Crasset <25140344+tcrasset@users.noreply.github.com> Date: Mon, 17 Feb 2025 16:15:53 +0100 Subject: [PATCH 074/149] add BucketLifecycleConfiguration as supported bucket operations --- Amazon-S3-API.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/Amazon-S3-API.md b/Amazon-S3-API.md index 56920e5..07cd224 100644 --- a/Amazon-S3-API.md +++ b/Amazon-S3-API.md @@ -32,7 +32,7 @@ See https://github.com/seaweedfs/seaweedfs/wiki/Path-Specific-Configuration # Supported APIs -Currently, the following APIs are supported. +Currently, the following APIs are supported. Some addiotional endpoints might be supported but are not in this list. ``` // Object operations @@ -56,6 +56,9 @@ Currently, the following APIs are supported. * DeleteBucket * HeadBucket * ListBuckets +* PutBucketLifecycleConfiguration (partially, only for TTL) +* GetBucketLifecycleConfiguration (partially, only for TTL) +* DeleteBucketLifecycleConfiguration (partially, only for TTL) // Multipart upload operations * NewMultipartUpload From e2d1e2656e3b47b3a9fa4289a3095ab95388da38 Mon Sep 17 00:00:00 2001 From: Tom Crasset <25140344+tcrasset@users.noreply.github.com> Date: Mon, 17 Feb 2025 16:18:30 +0100 Subject: [PATCH 075/149] hint that list of supported endpoints might not be complete and direct user to where they can check --- Amazon-S3-API.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/Amazon-S3-API.md b/Amazon-S3-API.md index 07cd224..d08f558 100644 --- a/Amazon-S3-API.md +++ b/Amazon-S3-API.md @@ -32,7 +32,11 @@ See https://github.com/seaweedfs/seaweedfs/wiki/Path-Specific-Configuration # Supported APIs -Currently, the following APIs are supported. Some addiotional endpoints might be supported but are not in this list. +Currently, the following APIs are supported. + +Some additional endpoints might be (partially), supported but are not in this list. +To be sure, you can look at the function defined in the files `weed/s3api/s3api_*_handlers_*.go`. + ``` // Object operations From 4beff24f491534701cbf753355b973076abcb24f Mon Sep 17 00:00:00 2001 From: Tom Crasset <25140344+tcrasset@users.noreply.github.com> Date: Mon, 17 Feb 2025 16:26:57 +0100 Subject: [PATCH 076/149] explain how to set per-directory TTL using the s3 api --- S3-API-FAQ.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/S3-API-FAQ.md b/S3-API-FAQ.md index 5a9d203..c1ca16e 100644 --- a/S3-API-FAQ.md +++ b/S3-API-FAQ.md @@ -52,3 +52,40 @@ rclone lsf seaweedfs:my-bucket/dir ``` If the directory `dir` exists in `my-bucket`, the orphaned metadata will be cleaned up. Note that due to slight API usage differences, `rclone ls` does not trigger cleanup, but `rclone lsf` will. + + +## Setting TTL + +It is possible to set a TTL for a specific directory using the S3 API. They are set using [`PutBucketLifecycleConfiguration`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html). + +Example of a JSON configuration is below. It is an equivalent of calling the following with `weed shell`. +``` +fs.configure -locationPrefix /buckets/f341868e-baff-4e20-896a-08bc148e32f9/my-directory-whose-files-will-expire-in-20-days -ttl 20d -apply +``` +: + +``` +{ + "Rules": [ + { + "Status": "Enabled", + "Filter": { + "Prefix": "my-directory-whose-files-will-expire-in-20-days" + }, + "Expiration": { + "Days": 20d + } + } + ] +} +``` +Save that in a `.json` file and call it, for example, via the `aws` cli: + +``` +BUCKET_NAME=f341868e-baff-4e20-896a-08bc148e32f9 +aws --endpoint-url http://127.0.0.1:8333 s3api put-bucket-lifecycle-configuration --bucket $BUCKET_NAME --lifecycle-configuration "file://lifecycle_policy.json" +``` + +Note that you don't need to add the part `/buckets/$BUCKET_NAME` in the configurations "Filter.Prefix" (contrary to using `fs.configure`, this is taken care of for you in the S3 API. + + From 076186b9ab30afb17d5c59965ef884ac267276c2 Mon Sep 17 00:00:00 2001 From: Tom Crasset <25140344+tcrasset@users.noreply.github.com> Date: Mon, 17 Feb 2025 16:27:15 +0100 Subject: [PATCH 077/149] link to s3 api ttl --- Store-file-with-a-Time-To-Live.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/Store-file-with-a-Time-To-Live.md b/Store-file-with-a-Time-To-Live.md index 9d3021c..1d345ec 100644 --- a/Store-file-with-a-Time-To-Live.md +++ b/Store-file-with-a-Time-To-Live.md @@ -72,3 +72,7 @@ For deploying to production, the TTL volume maximum size should be taken into co It's recommended not to mix the TTL volumes and non TTL volumes in the same cluster. This is because the volume maximum size, default to 30GB, is configured on the volume master at the cluster level. We could implement the configuration for max volume size for each TTL. However, it could get fairly verbose. Maybe later if it is strongly desired. + + +## Via S3 API +This can also be set via the S3 api using LifecycleConfiguration. See the section about that [here](https://github.com/seaweedfs/seaweedfs/wiki/S3-API-FAQ#setting-ttl). \ No newline at end of file From 6adf49776de765a1d5148932224af6e4732b6272 Mon Sep 17 00:00:00 2001 From: Tom Crasset <25140344+tcrasset@users.noreply.github.com> Date: Tue, 18 Feb 2025 16:23:43 +0100 Subject: [PATCH 078/149] show how to run the commands using the admin --- AWS-IAM-CLI.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/AWS-IAM-CLI.md b/AWS-IAM-CLI.md index 191fd73..690588a 100644 --- a/AWS-IAM-CLI.md +++ b/AWS-IAM-CLI.md @@ -16,6 +16,13 @@ s3.configure -apply -user admin -access_key some_access_key1 -secret_key some_se ## Create S3 credentials +Make sure you are using the admin: + +```bash +export AWS_ACCESS_KEY_ID=some_access_key1 +export AWS_SECRET_ACCESS_KEY=some_secret_key1 +``` + Create user and access key ``` aws --endpoint http://127.0.0.1:8111 iam create-access-key --user-name Bob From e6f831eedf0bcaef62b57461bae79e468748a826 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Mon, 10 Mar 2025 21:01:08 -0700 Subject: [PATCH 079/149] Updated _Sidebar (markdown) --- _Sidebar.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/_Sidebar.md b/_Sidebar.md index 0d85853..e031890 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -97,9 +97,12 @@ * [[Async Replication to Cloud]] [Deprecated] * [[Kubernetes Backups and Recovery with K8up]] -### Messaging +### Metadata Change Events * [[Filer Metadata Events]] +### Messaging +* [[Seaweed Message Queue]] + ### Use Cases * [[Use Cases]] * [[Actual Users]] From d4453b56733f0ab82bdea072bacfd59d46689219 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Mon, 10 Mar 2025 21:25:35 -0700 Subject: [PATCH 080/149] Created Seaweed Message Queue (markdown) --- Seaweed-Message-Queue.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 Seaweed-Message-Queue.md diff --git a/Seaweed-Message-Queue.md b/Seaweed-Message-Queue.md new file mode 100644 index 0000000..a602293 --- /dev/null +++ b/Seaweed-Message-Queue.md @@ -0,0 +1,23 @@ +# Introduction + +# Architecture + +The structured messages are published to message brokers, persisted to SeaweedFS, and consumed by message subscribers. To simplify client libraries, a messaging agent with gRPC APIs is added for easier message publishing and subscribing. +``` + Publishers => gRPC Publish APIs => Agent => Brokers => Agent => gRPC Subscribe APIs => Subscribers + ^ + | + v + SeaweedFS +``` + +The Agent can be run either on the server side, or as a sidecar on each client. + +# Features +* Structured Messages: SeaweedFS is to store unstructured data files. Seaweed Message Queue is to store structured messages. +* The messages stored on SeaweedFS can be converted into Parquet files, saving disk spaces with more efficient columnar compression. +* The Parquet files can be consumed in streams via Seaweed messaging brokers. +* The Parquet files can be read in batches directly via SeaweedFS. +* Message published successfully are acknowledged, or receive an error if not successful. +* Message consume offsets are tracked and persisted. +* Consumer APIs can process messages in multi-threads while still ensuring serial processing the same message key. From cf0fff4c778dddc1e9febd274ee5b4e845554120 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Mon, 10 Mar 2025 23:24:03 -0700 Subject: [PATCH 081/149] Updated Seaweed Message Queue (markdown) --- Seaweed-Message-Queue.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Seaweed-Message-Queue.md b/Seaweed-Message-Queue.md index a602293..e2c1347 100644 --- a/Seaweed-Message-Queue.md +++ b/Seaweed-Message-Queue.md @@ -14,10 +14,10 @@ The structured messages are published to message brokers, persisted to SeaweedFS The Agent can be run either on the server side, or as a sidecar on each client. # Features -* Structured Messages: SeaweedFS is to store unstructured data files. Seaweed Message Queue is to store structured messages. -* The messages stored on SeaweedFS can be converted into Parquet files, saving disk spaces with more efficient columnar compression. -* The Parquet files can be consumed in streams via Seaweed messaging brokers. -* The Parquet files can be read in batches directly via SeaweedFS. -* Message published successfully are acknowledged, or receive an error if not successful. +* Structured Messages: SeaweedFS is used to store unstructured data files, while Seaweed Message Queue is used to store structured messages. +* Messages stored in SeaweedFS can be converted into Parquet files, saving disk space with more efficient columnar compression. +* The messages in Parquet files can be streamed via Seaweed messaging brokers. +* The Parquet files can be read in batches directly from SeaweedFS. +* Messages published successfully are acknowledged, while unsuccessful ones receive an error. * Message consume offsets are tracked and persisted. -* Consumer APIs can process messages in multi-threads while still ensuring serial processing the same message key. +* Consumer APIs can process messages in multiple threads while still ensuring serial processing of messages with the same key. From 327409dc52a9fefe211a4d93428c90e605c3f0ec Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Mon, 10 Mar 2025 23:26:00 -0700 Subject: [PATCH 082/149] Updated Seaweed Message Queue (markdown) --- Seaweed-Message-Queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Seaweed-Message-Queue.md b/Seaweed-Message-Queue.md index e2c1347..b207f09 100644 --- a/Seaweed-Message-Queue.md +++ b/Seaweed-Message-Queue.md @@ -2,7 +2,7 @@ # Architecture -The structured messages are published to message brokers, persisted to SeaweedFS, and consumed by message subscribers. To simplify client libraries, a messaging agent with gRPC APIs is added for easier message publishing and subscribing. +The structured messages are published to message brokers, persisted in SeaweedFS, and consumed by message subscribers. To simplify client libraries, a messaging agent with gRPC APIs is introduced, enabling easier client libraries for message publishing and subscribing. ``` Publishers => gRPC Publish APIs => Agent => Brokers => Agent => gRPC Subscribe APIs => Subscribers ^ From fb18c6ae711ab18cba88f37e229f87758228a35e Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Tue, 11 Mar 2025 21:29:54 -0700 Subject: [PATCH 083/149] Updated S3 Nginx Proxy (markdown) --- S3-Nginx-Proxy.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/S3-Nginx-Proxy.md b/S3-Nginx-Proxy.md index 382649a..6ec7af9 100644 --- a/S3-Nginx-Proxy.md +++ b/S3-Nginx-Proxy.md @@ -31,15 +31,15 @@ server { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; - proxy_set_header X-Forwarded-Host $host; - proxy_set_header X-Forwarded-Port $server_port; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Port $server_port; proxy_set_header X-Forwarded-Proto $scheme; proxy_connect_timeout 300; proxy_http_version 1.1; proxy_set_header Connection ""; - proxy_request_buffering off; + proxy_request_buffering off; chunked_transfer_encoding off; # If bucket subdomain is not empty, From 68cc2acfd4787e27190c24372a170e496adb382e Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Thu, 13 Mar 2025 09:39:55 -0700 Subject: [PATCH 084/149] Updated Seaweed Message Queue (markdown) --- Seaweed-Message-Queue.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/Seaweed-Message-Queue.md b/Seaweed-Message-Queue.md index b207f09..62d794d 100644 --- a/Seaweed-Message-Queue.md +++ b/Seaweed-Message-Queue.md @@ -18,6 +18,11 @@ The Agent can be run either on the server side, or as a sidecar on each client. * Messages stored in SeaweedFS can be converted into Parquet files, saving disk space with more efficient columnar compression. * The messages in Parquet files can be streamed via Seaweed messaging brokers. * The Parquet files can be read in batches directly from SeaweedFS. -* Messages published successfully are acknowledged, while unsuccessful ones receive an error. + +## Features on Publishing +* Messages published successfully are acknowledged asynchronously. + +## Features on Subscribing * Message consume offsets are tracked and persisted. * Consumer APIs can process messages in multiple threads while still ensuring serial processing of messages with the same key. + From ca1f4321eb4dd0edf9af01b4f4f23e888eaa2400 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Thu, 13 Mar 2025 09:45:07 -0700 Subject: [PATCH 085/149] Updated Seaweed Message Queue (markdown) --- Seaweed-Message-Queue.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Seaweed-Message-Queue.md b/Seaweed-Message-Queue.md index 62d794d..5a26424 100644 --- a/Seaweed-Message-Queue.md +++ b/Seaweed-Message-Queue.md @@ -1,5 +1,11 @@ # Introduction +Seaweed Message Queue has +* Structured message : all messages need to have a schema +* Streamed publishing with async acknowledgements +* Messages are stored in Parquet files, for both streaming and batch reading. +* Unlimited storage. + # Architecture The structured messages are published to message brokers, persisted in SeaweedFS, and consumed by message subscribers. To simplify client libraries, a messaging agent with gRPC APIs is introduced, enabling easier client libraries for message publishing and subscribing. From 64df87597bcb59c55d4fe7d8efc3a5df5f01e3eb Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Thu, 13 Mar 2025 09:46:12 -0700 Subject: [PATCH 086/149] Updated Seaweed Message Queue (markdown) --- Seaweed-Message-Queue.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Seaweed-Message-Queue.md b/Seaweed-Message-Queue.md index 5a26424..883d542 100644 --- a/Seaweed-Message-Queue.md +++ b/Seaweed-Message-Queue.md @@ -5,6 +5,7 @@ Seaweed Message Queue has * Streamed publishing with async acknowledgements * Messages are stored in Parquet files, for both streaming and batch reading. * Unlimited storage. +* Scalable stateless message brokers. # Architecture From aeed4676c7b5d8d52275e9167e3449a550835aa3 Mon Sep 17 00:00:00 2001 From: GabrielxD Date: Tue, 18 Mar 2025 13:24:42 +0800 Subject: [PATCH 087/149] fix: make bucket subdomain optional in server_name regex pattern --- S3-Nginx-Proxy.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/S3-Nginx-Proxy.md b/S3-Nginx-Proxy.md index 6ec7af9..24c44e6 100644 --- a/S3-Nginx-Proxy.md +++ b/S3-Nginx-Proxy.md @@ -22,7 +22,7 @@ server { # The regex will support path style as well as virtual-hosted style bucket URLs # path style: http://s3.yourdomain.com/mybucket # virtual-hosted style: http://mybucket.s3.yourdomain.com - server_name ~^(?:(?[^.]+)\.)s3\.yourdomain\.com; + server_name ~^(?:(?[^.]+)\.)?s3\.yourdomain\.com; ignore_invalid_headers off; client_max_body_size 0; @@ -31,15 +31,15 @@ server { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; - proxy_set_header X-Forwarded-Host $host; - proxy_set_header X-Forwarded-Port $server_port; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Port $server_port; proxy_set_header X-Forwarded-Proto $scheme; proxy_connect_timeout 300; proxy_http_version 1.1; proxy_set_header Connection ""; - proxy_request_buffering off; + proxy_request_buffering off; chunked_transfer_encoding off; # If bucket subdomain is not empty, From 17ee9c2dea46d1e62960e0eec3b09dbe78dc99c0 Mon Sep 17 00:00:00 2001 From: chrislu Date: Tue, 18 Mar 2025 08:13:55 -0700 Subject: [PATCH 088/149] updates related to cassandra2 --- Filer-Cassandra-Setup.md | 5 +++-- Super-Large-Directories.md | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Filer-Cassandra-Setup.md b/Filer-Cassandra-Setup.md index b98c9b6..884b961 100644 --- a/Filer-Cassandra-Setup.md +++ b/Filer-Cassandra-Setup.md @@ -18,10 +18,11 @@ create keyspace seaweedfs WITH replication = { use seaweedfs; CREATE TABLE filemeta ( + dirhash bigint, directory varchar, name varchar, meta blob, - PRIMARY KEY (directory, name) + PRIMARY KEY ((dirhash, directory), name) ) WITH CLUSTERING ORDER BY (name ASC); ``` @@ -33,7 +34,7 @@ Try run ```weed filer -h``` to see an example filer.toml file. The file should b Here is the shortest example for Cassandra ```bash -[cassandra] +[cassandra2] enabled = true keyspace="seaweedfs" hosts=[ diff --git a/Super-Large-Directories.md b/Super-Large-Directories.md index a02bcc7..f905ab1 100644 --- a/Super-Large-Directories.md +++ b/Super-Large-Directories.md @@ -61,7 +61,7 @@ The consequences are: In `filer.toml` for Cassandra/Redis, there is an option `superLargeDirectories`. For example, if you will have a lot of user data under `/home/users` ``` -[cassandra] +[cassandra2] ... superLargeDirectories = [ "/home/users", From 416d50be763c158201b9552155d6580cf9285137 Mon Sep 17 00:00:00 2001 From: chrislu Date: Tue, 18 Mar 2025 08:16:36 -0700 Subject: [PATCH 089/149] Update Data-Backup.md --- Data-Backup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Data-Backup.md b/Data-Backup.md index ebe6264..a9bef62 100644 --- a/Data-Backup.md +++ b/Data-Backup.md @@ -22,7 +22,7 @@ The backup scripts are just one command, not a continuous running service though ### To Start -- Pause operations on cluster you want to backup. This is to avoid a mismatch in the two data sets (the volumes and the Filer metadata) you will be moving separately and combining in the backup cluster. +- Pause operations on cluster you want to back up. This is to avoid a mismatch in the two data sets (the volumes and the Filer metadata) you will be moving separately and combining in the backup cluster. - Install SeaweedFS on a new machine/cluster of machines. Use the same version of SeaweedFS if you can to avoid any issues that could otherwise come up. - *Do not start your volume servers yet!!* This is to avoid SeaweedFS creating it's own volumes (we will be using the volumes backed up from the current operational cluster). From 19f3665a619c4892e5c0024c31ab6b007e1d3db4 Mon Sep 17 00:00:00 2001 From: name_snrl <72071763+name-snrl@users.noreply.github.com> Date: Thu, 20 Mar 2025 14:37:43 +0900 Subject: [PATCH 090/149] Updated fstab (markdown) --- fstab.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fstab.md b/fstab.md index c51f5f7..9e42e5b 100644 --- a/fstab.md +++ b/fstab.md @@ -1,5 +1,5 @@ To mount SeaweedFS using `/etc/fstab` (such as on boot): - + * Follow the directions in https://github.com/seaweedfs/seaweedfs/wiki/FUSE-Mount for setting up a SeaweedFS mount subtype for FUSE (hint: `cp weed /sbin/weed`) * Install SeaweedFS as usual, making sure any specific settings (such as `security.toml`) are in the correct location to be read * Add the fstab entry as described below From b4ebaf09fec05843517d4da07d0e58bc0d7cf211 Mon Sep 17 00:00:00 2001 From: name_snrl <72071763+name-snrl@users.noreply.github.com> Date: Thu, 20 Mar 2025 15:37:44 +0900 Subject: [PATCH 091/149] few words about integration with systemd --- fstab.md => fstab-(poor-systemd-support).md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) rename fstab.md => fstab-(poor-systemd-support).md (58%) diff --git a/fstab.md b/fstab-(poor-systemd-support).md similarity index 58% rename from fstab.md rename to fstab-(poor-systemd-support).md index 9e42e5b..c00c360 100644 --- a/fstab.md +++ b/fstab-(poor-systemd-support).md @@ -3,6 +3,7 @@ To mount SeaweedFS using `/etc/fstab` (such as on boot): * Follow the directions in https://github.com/seaweedfs/seaweedfs/wiki/FUSE-Mount for setting up a SeaweedFS mount subtype for FUSE (hint: `cp weed /sbin/weed`) * Install SeaweedFS as usual, making sure any specific settings (such as `security.toml`) are in the correct location to be read * Add the fstab entry as described below +* But make sure that `systemd-fstab-generator(8)` is not used on your system If you have a single filer server, this is the syntax you will use: @@ -12,4 +13,18 @@ If you have multiple filer servers, this is the syntax you will use: `fuse /path/to/mountpoint fuse.weed filer='192.168.0.1:8888,192.168.0.2:8888',filer.path=/,defaults,_netdev 0 0` -Place the appropriate line into `/etc/fstab` and attempt to mount your filesystem. If you encounter any issues, run `weed mount` directly in verbose/debug mode to diagnose the issue, and switch back to fstab-style mounting once you have resolved the issues. \ No newline at end of file +Place the appropriate line into `/etc/fstab` and attempt to mount your filesystem. If you encounter any issues, run `weed mount` directly in verbose/debug mode to diagnose the issue, and switch back to fstab-style mounting once you have resolved the issues. + +## Systemd + +* No matter what systemd options (`nofail`, `x-systemd.device-timeout`, `x-systemd.mount-timeout`, etc.) you add to /etc/fstab, you won’t be able to make `systemd.mount(5)` handle the mount properly. You will always get an error when starting mount unit, even though the filesystem ends up being mounted. +* `x-systemd.automount` doesn’t work at all. +* The `_netdev` results in errors like: + + ```log + kernel: fuse: Unknown parameter '_netdev' + ``` + + p.s. Not sure if it’s related. (systemd 255.6) + +In general, if your system is deeply integrated with systemd, it is better to create `systemd.service(5)` to mount. \ No newline at end of file From 55ed2b885c8ff989e3e519d5da650be65f3aa42c Mon Sep 17 00:00:00 2001 From: name_snrl <72071763+name-snrl@users.noreply.github.com> Date: Thu, 20 Mar 2025 15:40:06 +0900 Subject: [PATCH 092/149] Updated _Sidebar (markdown) --- _Sidebar.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_Sidebar.md b/_Sidebar.md index e031890..a53d8bd 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -48,7 +48,7 @@ ### [[FUSE Mount]] * [[FIO benchmark]] -* [[fstab]] +* [[fstab (poor systemd support)]] ### [[WebDAV]] From 3f52fd498c1eab11a06c07f5bec278e65064ae93 Mon Sep 17 00:00:00 2001 From: chrislu Date: Thu, 20 Mar 2025 08:36:13 -0700 Subject: [PATCH 093/149] Revert "Updated _Sidebar (markdown)" This reverts commit 55ed2b885c8ff989e3e519d5da650be65f3aa42c. --- _Sidebar.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_Sidebar.md b/_Sidebar.md index a53d8bd..e031890 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -48,7 +48,7 @@ ### [[FUSE Mount]] * [[FIO benchmark]] -* [[fstab (poor systemd support)]] +* [[fstab]] ### [[WebDAV]] From 948a70df5c0d9d2d27561b96de53bde07a29d2db Mon Sep 17 00:00:00 2001 From: chrislu Date: Thu, 20 Mar 2025 08:36:51 -0700 Subject: [PATCH 094/149] revert file renaming --- fstab-(poor-systemd-support).md => fstab.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename fstab-(poor-systemd-support).md => fstab.md (100%) diff --git a/fstab-(poor-systemd-support).md b/fstab.md similarity index 100% rename from fstab-(poor-systemd-support).md rename to fstab.md From ae0727b575ed9fd680f41a54c6b20e34242e683d Mon Sep 17 00:00:00 2001 From: chrislu Date: Thu, 20 Mar 2025 08:53:43 -0700 Subject: [PATCH 095/149] Update Seaweed-Message-Queue.md --- Seaweed-Message-Queue.md | 151 +++++++++++++++++++++++++++++++++++---- 1 file changed, 137 insertions(+), 14 deletions(-) diff --git a/Seaweed-Message-Queue.md b/Seaweed-Message-Queue.md index 883d542..dff397a 100644 --- a/Seaweed-Message-Queue.md +++ b/Seaweed-Message-Queue.md @@ -1,15 +1,21 @@ # Introduction -Seaweed Message Queue has +Seaweed Message Queue (SMQ) is a distributed messaging system built on top of SeaweedFS. It provides: + * Structured message : all messages need to have a schema * Streamed publishing with async acknowledgements -* Messages are stored in Parquet files, for both streaming and batch reading. -* Unlimited storage. -* Scalable stateless message brokers. +* Messages are stored in Parquet files, for both streaming and batch reading +* Disaggregated storage +* Scalable stateless message brokers # Architecture -The structured messages are published to message brokers, persisted in SeaweedFS, and consumed by message subscribers. To simplify client libraries, a messaging agent with gRPC APIs is introduced, enabling easier client libraries for message publishing and subscribing. +The system consists of three main components: + +1. **Message Queue Agent**: A gRPC server that provides a simplified interface for clients +2. **Message Queue Brokers**: Stateless brokers that handle message routing and storage +3. **SeaweedFS**: The underlying storage system that persists messages in Parquet format + ``` Publishers => gRPC Publish APIs => Agent => Brokers => Agent => gRPC Subscribe APIs => Subscribers ^ @@ -21,15 +27,132 @@ The structured messages are published to message brokers, persisted in SeaweedFS The Agent can be run either on the server side, or as a sidecar on each client. # Features -* Structured Messages: SeaweedFS is used to store unstructured data files, while Seaweed Message Queue is used to store structured messages. -* Messages stored in SeaweedFS can be converted into Parquet files, saving disk space with more efficient columnar compression. -* The messages in Parquet files can be streamed via Seaweed messaging brokers. -* The Parquet files can be read in batches directly from SeaweedFS. -## Features on Publishing -* Messages published successfully are acknowledged asynchronously. +## Core Features +* Structured Messages: SeaweedFS is used to store unstructured data files, while Seaweed Message Queue is used to store structured messages +* Messages stored in SeaweedFS can be converted into Parquet files, saving disk space with more efficient columnar compression +* The messages in Parquet files can be streamed via Seaweed messaging brokers +* The Parquet files can be read in batches directly from SeaweedFS -## Features on Subscribing -* Message consume offsets are tracked and persisted. -* Consumer APIs can process messages in multiple threads while still ensuring serial processing of messages with the same key. +## Publishing Features +* Messages published successfully are acknowledged asynchronously +* Partition-based message routing +* Schema validation for message structure + +## Subscribing Features +* Message consume offsets are tracked and persisted on the server side +* Consumer APIs can process messages in parallel while still ensuring serial processing of messages with the same key +* Configurable sliding window for concurrent message processing +* Ability to start consuming from specific timestamps or offsets + +# Usage Examples + +## Starting the Services + +1. Start a Message Queue Broker: +```bash +weed mq.broker -port=17777 -master=localhost:9333 +``` + +2. Start a Message Queue Agent: +```bash +weed mq.agent -port=16777 -broker=localhost:17777 +``` + +## Defining Message Schema + +Messages in SMQ must have a defined schema. Here's an example of defining a message type: + +```go +type MyRecord struct { + Key []byte + Field1 []byte + Field2 string + Field3 int32 + Field4 int64 + Field5 float32 + Field6 float64 + Field7 bool +} + +func MyRecordType() *schema_pb.RecordType { + return schema.RecordTypeBegin(). + WithField("key", schema.TypeBytes). + WithField("field1", schema.TypeBytes). + WithField("field2", schema.TypeString). + WithField("field3", schema.TypeInt32). + WithField("field4", schema.TypeInt64). + WithField("field5", schema.TypeFloat). + WithField("field6", schema.TypeDouble). + WithField("field7", schema.TypeBoolean). + RecordTypeEnd() +} +``` + +## Publishing Messages + +```go +// Create a publish session +session, err := agent_client.NewPublishSession( + "localhost:16777", // agent address + schema.NewSchema("my_namespace", "my_topic", MyRecordType()), + 6, // partition count + "publisher1", // client name +) + +// Publish a message +myRecord := &MyRecord{ + Key: []byte("key1"), + Field1: []byte("value1"), + Field2: "string value", + Field3: 123, + Field4: 456, + Field5: 1.23, + Field6: 4.56, + Field7: true, +} + +err := session.PublishMessageRecord(myRecord.Key, myRecord.ToRecordValue()) +``` + +## Subscribing to Messages + +```go +// Create a subscribe session +session, err := agent_client.NewSubscribeSession( + "localhost:16777", // agent address + &agent_client.SubscribeOption{ + ConsumerGroup: "my-group", + ConsumerGroupInstanceId: "consumer1", + Topic: topic.NewTopic("my_namespace", "topmy_topicic"), + OffsetType: schema_pb.OffsetType_RESUME_OR_EARLIEST, + MaxSubscribedPartitions: 3, // maximum number of partitions this consumer instance can subscribe + SlidingWindowSize: 16, // concurrently process up-to 16 messages with different message key + }, +) + +// Subscribe to messages +session.SubscribeMessageRecord( + func(key []byte, recordValue *schema_pb.RecordValue) { + record := FromRecordValue(recordValue) + fmt.Printf("Received: %+v\n", record) + }, + func() { + fmt.Println("Subscription completed") + }, +) +``` + +# Configuration + +## Broker Configuration +* `-port`: gRPC server port (default: 17777) +* `-master`: comma-separated master servers +* `-filerGroup`: share metadata with other filers in the same group +* `-dataCenter`: prefer volumes in this data center +* `-rack`: prefer volumes in this rack + +## Agent Configuration +* `-port`: gRPC server port (default: 16777) +* `-broker`: comma-separated message queue brokers From 97880395a8429d904407caf50ddcdabae6319a3f Mon Sep 17 00:00:00 2001 From: MaratKarimov Date: Mon, 31 Mar 2025 17:46:51 +0300 Subject: [PATCH 096/149] Updated Filer Stores (markdown) --- Filer-Stores.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Filer-Stores.md b/Filer-Stores.md index 46f3a1c..534f54f 100644 --- a/Filer-Stores.md +++ b/Filer-Stores.md @@ -28,6 +28,7 @@ The Filer Store persists all file metadata and directory information. | ElasticSearch| O(logN)| unlimited | Distributed, Fast || Yes| | Scalable, Searchable. Need to manually build. | | HBase | O(logN)| unlimited | Distributed, Fast | | Native| | Scalable | | TiKV | O(logN)| unlimited | Distributed, Fast |Atomic|Yes| Yes| Scalable. High Availability. Need to manually build. | +| Tarantool | O(logN)| unlimited | Local or Distributed, Fast ||Yes|| Scalable. High Availability. Need to manually build. | #### Switching between different Stores It is easy to switch between different filer stores. From c645ad8758abfdf7c0b5453a9c0a8637f4789d54 Mon Sep 17 00:00:00 2001 From: Ignatiev Mikhail Date: Fri, 4 Apr 2025 11:07:49 +0300 Subject: [PATCH 097/149] Updated nodejs with Seaweed S3 (markdown) --- nodejs-with-Seaweed-S3.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/nodejs-with-Seaweed-S3.md b/nodejs-with-Seaweed-S3.md index f6ce0ca..6cfe5ed 100644 --- a/nodejs-with-Seaweed-S3.md +++ b/nodejs-with-Seaweed-S3.md @@ -20,6 +20,11 @@ const s3client = new S3Client({ region: `us-east-1`, // force path style for compatibility reasons forcePathStyle: true, + // dual stack endpoint is not supported by seaweed + useDualstackEndpoint: false, + // checksum validation should be disabled, overwise `x-amz-checksum` will be injected directly into files + responseChecksumValidation: `WHEN_REQUIRED`, + requestChecksumCalculation: 'WHEN_REQUIRED', // credentials is mandatory and s3 authorization should be enabled with `s3.configure` credentials: { accessKeyId: `same as -accesskey`, From e7ecc7825726a2240b11823790e06050d730fbcf Mon Sep 17 00:00:00 2001 From: Cliff Dyer Date: Fri, 11 Apr 2025 14:09:23 -0400 Subject: [PATCH 098/149] Random connection to .ru looks sketchy --- Getting-Started.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/Getting-Started.md b/Getting-Started.md index a407826..cac06c1 100644 --- a/Getting-Started.md +++ b/Getting-Started.md @@ -141,16 +141,14 @@ You can use docker volumes to persist data: ```bash # start our weed server daemonized docker run --name weed -d -p 9333:9333 -p 8080:8080 -p 18080:8080 \ - -v seaweedvolume:/data chrislusf/seaweedfs server -dir="/data" \ - -publicIp="$(curl -s cydev.ru/ip)" + -v seaweedvolume:/data chrislusf/seaweedfs server -dir="/data" ``` Alternatively, you can mount a directory on the host machine into the container: ```bash # start our weed server daemonized docker run --name weed -d -p 9333:9333 -p 8080:8080 -p 18080:8080 \ - -v /opt/weedfs/data:/data chrislusf/seaweedfs server -dir="/data" \ - -publicIp="$(curl -s cydev.ru/ip)" + -v /opt/weedfs/data:/data chrislusf/seaweedfs server -dir="/data" ``` Note that according to [Docker's documentation](https://docs.docker.com/storage/volumes/), volumes are the preferred mechanism for persisting data. From f61fef3b25b3801ba545af82db0466d2bf6712b0 Mon Sep 17 00:00:00 2001 From: orthoxerox Date: Sun, 11 May 2025 13:24:05 +0300 Subject: [PATCH 099/149] Documenting PR #6775 --- Hadoop-Compatible-File-System.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/Hadoop-Compatible-File-System.md b/Hadoop-Compatible-File-System.md index 0af4eb3..76e47ea 100644 --- a/Hadoop-Compatible-File-System.md +++ b/Hadoop-Compatible-File-System.md @@ -78,6 +78,7 @@ Both reads and writes are working fine. 1. `fs.AbstractFileSystem.seaweedfs.impl`: Add the SeaweedFS implementation of Hadoop AbstractFileSystem to delegates to the existing SeaweedFS FileSystem and is only necessary for use with Hadoop 3.x. 1. `fs.seaweed.buffer.size`: Optionally change the default buffer size 4194304 to a larger number. It will be the default chunk size. 1. `fs.seaweed.volume.server.access`: `[direct|publicUrl|filerProxy]` Optionally access volume servers via their publicUrl settings, or use filer as a proxy. This is useful when volume servers are inside a cluster and not directly accessible. + 1. `fs.seaweed.filer.cn`: if you have enabled gRPC security on your filer and the certificate doesn't match the hostname of the filer, set this to the CN of your filer's TLS certificate (see [[[[Security-Configuration]], `grpc.filer.cert`). Seaweed doesn't use host name verification anywhere, but Java libraries for gRPC do, so to sidestep this you need to pass them the expected CN instead. ``` @@ -102,6 +103,10 @@ Both reads and writes are working fine. direct + + fs.seaweed.filer.cn + filer01 + ``` * Deploy the SeaweedFS HDFS client jar From 899595847bea929153bdc463a69ed8babb23905e Mon Sep 17 00:00:00 2001 From: orthoxerox Date: Sun, 11 May 2025 13:24:52 +0300 Subject: [PATCH 100/149] editing typo --- Hadoop-Compatible-File-System.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Hadoop-Compatible-File-System.md b/Hadoop-Compatible-File-System.md index 76e47ea..7e914f7 100644 --- a/Hadoop-Compatible-File-System.md +++ b/Hadoop-Compatible-File-System.md @@ -78,7 +78,7 @@ Both reads and writes are working fine. 1. `fs.AbstractFileSystem.seaweedfs.impl`: Add the SeaweedFS implementation of Hadoop AbstractFileSystem to delegates to the existing SeaweedFS FileSystem and is only necessary for use with Hadoop 3.x. 1. `fs.seaweed.buffer.size`: Optionally change the default buffer size 4194304 to a larger number. It will be the default chunk size. 1. `fs.seaweed.volume.server.access`: `[direct|publicUrl|filerProxy]` Optionally access volume servers via their publicUrl settings, or use filer as a proxy. This is useful when volume servers are inside a cluster and not directly accessible. - 1. `fs.seaweed.filer.cn`: if you have enabled gRPC security on your filer and the certificate doesn't match the hostname of the filer, set this to the CN of your filer's TLS certificate (see [[[[Security-Configuration]], `grpc.filer.cert`). Seaweed doesn't use host name verification anywhere, but Java libraries for gRPC do, so to sidestep this you need to pass them the expected CN instead. + 1. `fs.seaweed.filer.cn`: if you have enabled gRPC security on your filer and the certificate doesn't match the hostname of the filer, set this to the CN of your filer's TLS certificate (see [[Security-Configuration]], `grpc.filer.cert`). Seaweed doesn't use host name verification anywhere, but Java libraries for gRPC do, so to sidestep this you need to pass them the expected CN instead. ``` From b890bb77f761de40edd3406fffd9ad45d976c59d Mon Sep 17 00:00:00 2001 From: orthoxerox Date: Sun, 11 May 2025 13:35:29 +0300 Subject: [PATCH 101/149] Updated the scaffolded toml, added a warning for enterprise users. --- Security-Configuration.md | 45 +++++++++++++++++++++++++++++++++------ 1 file changed, 39 insertions(+), 6 deletions(-) diff --git a/Security-Configuration.md b/Security-Configuration.md index 560f561..5b78e96 100644 --- a/Security-Configuration.md +++ b/Security-Configuration.md @@ -11,7 +11,7 @@ $ weed scaffold -config=security # /etc/seaweedfs/security.toml # this file is read by master, volume server, and filer -# comma separated origins allowed to make requests to the filer and s3 gateway. +# comma separated origins allowed to make requests to the filer and s3 gateway. # enter in this format: https://domain.com, or http://localhost:port [cors.allowed_origins] values = "*" @@ -86,26 +86,57 @@ cert = "" key = "" allowed_commonNames = "" # comma-separated SSL certificate common names +[grpc.s3] +cert = "" +key = "" +allowed_commonNames = "" # comma-separated SSL certificate common names + [grpc.msg_broker] cert = "" key = "" allowed_commonNames = "" # comma-separated SSL certificate common names +[grpc.msg_agent] +cert = "" +key = "" +allowed_commonNames = "" # comma-separated SSL certificate common names + # use this for any place needs a grpc client # i.e., "weed backup|benchmark|filer.copy|filer.replicate|mount|s3|upload" [grpc.client] cert = "" key = "" -# volume server https options -# Note: work in progress! -# this does not work with other clients, e.g., "weed filer|mount" etc, yet. +# https client for master|volume|filer|etc connection +# It is necessary that the parameters [https.volume]|[https.master]|[https.filer] are set [https.client] -enabled = true +enabled = false +cert = "" +key = "" +ca = "" + +# volume server https options [https.volume] cert = "" key = "" +ca = "" +# master server https options +[https.master] +cert = "" +key = "" +ca = "" + +# filer server https options +[https.filer] +cert = "" +key = "" +ca = "" +# disable_tls_verify_client_cert = true|false (default: false) + +# white list. It's checking request ip address. +[guard] +white_list = "" ``` The following command is what I used to generate the private key and certificate files, using https://github.com/square/certstrap. To compile this tool, you can run `go get github.com/square/certstrap` - or alternatively `brew install certstrap` if you are on Mac OS and use [Homebrew](https://brew.sh). @@ -174,4 +205,6 @@ Java gRPC uses Netty's SslContext. From https://netty.io/wiki/sslcontextbuilder- If you are using existing certificates: make sure they all have the **Extended Key Usage** 'TLS Web Server Authentication' AND 'TLS Web Client Authentication' set - as grpc uses them for both use-cases! -Else you will see those errors: `error reading server preface: remote error: tls: bad certificate` \ No newline at end of file +Else you will see those errors: `error reading server preface: remote error: tls: bad certificate` + +Choose your CA carefully when using existing certificates in an enterprise environment. Since Seaweed only checks the certificate against the CA and optionally validates the CN of the certificate against a whitelist, do not use the root CA of the company or any other CA you don't control, as this means someone else can generate a client certificate your cluster will accept as legitimate. Instead, generate an intermediate CA for each Seaweed cluster and use it to generate server and client certificates. This intermediate CA is the one you should use in `grpc.ca` and `https.*.ca` properties. \ No newline at end of file From 8d3f343ba25db9c0da573ab04f3c040769f3348f Mon Sep 17 00:00:00 2001 From: Corbin McNeely-Smith <58151731+restingbull@users.noreply.github.com> Date: Wed, 4 Jun 2025 12:22:28 -0500 Subject: [PATCH 102/149] Add offset to the query parameters (see https://github.com/seaweedfs/seaweedfs/pull/2694) --- Filer-Server-API.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Filer-Server-API.md b/Filer-Server-API.md index 8cbf220..fced80b 100644 --- a/Filer-Server-API.md +++ b/Filer-Server-API.md @@ -36,7 +36,8 @@ You can append to any HTTP API with &pretty=y to see a formatted json output. | ttl | time to live, examples, 3m: 3 minutes, 4h: 4 hours, 5d: 5 days, 6w: 6 weeks, 7M: 7 months, 8y: 8 years | empty | | maxMB | max chunk size | empty | | mode | file mode | 0660 | -| op | file operation, currently only support "append" | empty | +| offset | incompatible with `op=append`. Defines the number of bytes from the file beginning to insert the uploaded chunk | empty | +| op | file operation, currently only support "append", incompatible with `offset=`. | empty | | skipCheckParentDir | Ensuring parent directory exists cost one metadata API call. Skipping this can reduce network latency. | false | | header: `Content-Type` | used for auto compression | empty | | header: `Content-Disposition` | used as response content-disposition | empty | From 13c7c4339cfef7100437ddc7ecb4f2e29cdca1b0 Mon Sep 17 00:00:00 2001 From: Anders Aarvik Date: Fri, 6 Jun 2025 11:29:45 +0200 Subject: [PATCH 103/149] The hash on uploadId query string in the GET request create consistency for multipart uploads. This will cause errors otherwise when using multiple filers for a single object with upload parts. --- S3-Nginx-Proxy.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/S3-Nginx-Proxy.md b/S3-Nginx-Proxy.md index 24c44e6..088af2a 100644 --- a/S3-Nginx-Proxy.md +++ b/S3-Nginx-Proxy.md @@ -10,7 +10,11 @@ Additionally, make sure that `proxy_request_buffering` is `off` (default is `on` ### Example Nginx config ``` -upstream seaweedfs { server localhost:8333 fail_timeout=0; keepalive 20;} +upstream seaweedfs { + hash $arg_uploadId consistent; + server localhost:8333 fail_timeout=0; + keepalive 20; +} ## Also you can use unix domain socket instead for better performance: # upstream seaweedfs { server unix:/tmp/seaweedfs-s3-8333.sock; keepalive 20;} From 70fb395d3e43e15cb76cd67d31172945509b29c6 Mon Sep 17 00:00:00 2001 From: Anders Aarvik Date: Mon, 9 Jun 2025 08:13:36 +0200 Subject: [PATCH 104/149] Updated S3 Nginx Proxy (markdown) --- S3-Nginx-Proxy.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/S3-Nginx-Proxy.md b/S3-Nginx-Proxy.md index 088af2a..e63fc45 100644 --- a/S3-Nginx-Proxy.md +++ b/S3-Nginx-Proxy.md @@ -11,6 +11,8 @@ Additionally, make sure that `proxy_request_buffering` is `off` (default is `on` ``` upstream seaweedfs { + # Hash on uploadId query string in the GET request create consistency for multipart uploads, + # only necessary when using local embedded filer store (leveldb) hash $arg_uploadId consistent; server localhost:8333 fail_timeout=0; keepalive 20; From 3f2062c1dc0274c295abfdb12175dbb250a92c44 Mon Sep 17 00:00:00 2001 From: Anders Aarvik Date: Mon, 9 Jun 2025 08:17:45 +0200 Subject: [PATCH 105/149] Added a comment in connection to load balancing and embedded filer store --- Filer-Store-Replication.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Filer-Store-Replication.md b/Filer-Store-Replication.md index 55dfde9..a1c6bba 100644 --- a/Filer-Store-Replication.md +++ b/Filer-Store-Replication.md @@ -14,7 +14,7 @@ So when a filer was added or removed from the cluster, there is no configuration What's more, adding a fresh new filer will automatically synchronize the metadata with other filers. Resuming a paused filer will also resume the metadata synchronization from when it was stopped. Everything will/should just work! -Note however that using multiple replicated embedded filers is only eventually consistent. So using this setup behind a load-balancer might not work as expected. +Note however that using multiple replicated embedded filers is only eventually consistent. So using this setup behind a load-balancer might not work as expected, although this can be solved with load balancer configuration, such as hashing connections based on the `uploadId` GET query, or temporary sticky sessions to ensure the objects requested are available. # Metadata synchronization From 7d53223c79859486f5336c2007cb5a3370f08a63 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sun, 6 Jul 2025 22:31:37 -0700 Subject: [PATCH 106/149] Updated _Sidebar (markdown) --- _Sidebar.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/_Sidebar.md b/_Sidebar.md index e031890..d9762d2 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -37,6 +37,10 @@ * [[Choosing a Filer Store]] * [[Customize Filer Store]] +### Management +* [[Admin UI]] +* [[Worker]] + ### Advanced Filer Configurations * [[Migrate to Filer Store]] * [[Add New Filer Store|Customize Filer Store]] From 6ad192fa9cda6d16174773100133a57e4f6173f8 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sun, 6 Jul 2025 22:33:17 -0700 Subject: [PATCH 107/149] Created Admin UI (markdown) --- Admin-UI.md | 202 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 202 insertions(+) create mode 100644 Admin-UI.md diff --git a/Admin-UI.md b/Admin-UI.md new file mode 100644 index 0000000..3d468e2 --- /dev/null +++ b/Admin-UI.md @@ -0,0 +1,202 @@ +# Weed Admin + +The `weed admin` command starts a modern web-based administration interface for SeaweedFS cluster management. + +## Overview + +The admin interface provides a comprehensive web UI for managing SeaweedFS clusters, including: +- **Cluster topology visualization and monitoring** +- **Volume management and operations** +- **File browser and management** +- **System metrics and performance monitoring** +- **Configuration management** +- **Maintenance operations** + +The admin interface automatically discovers filers from the master servers and runs a gRPC server for worker connections on HTTP port + 10000. + +## Usage + +```bash +weed admin [options] +``` + +## Options + +| Option | Default | Description | +|--------|---------|-------------| +| `-port` | 23646 | Admin server port | +| `-masters` | localhost:9333 | Comma-separated master servers | +| `-dataDir` | "" | Directory to store admin configuration and data files | +| `-adminUser` | admin | Admin interface username | +| `-adminPassword` | "" | Admin interface password (if empty, auth is disabled) | + +## Examples + +### Basic Usage + +```bash +# Start admin interface on default port (23646) +weed admin -masters=localhost:9333 + +# Start with custom port and multiple masters +weed admin -port=8080 -masters="master1:9333,master2:9333" + +# Start with specific data directory +weed admin -port=23646 -masters="localhost:9333" -dataDir="/var/lib/seaweedfs-admin" + +# Start with home directory expansion +weed admin -port=23646 -masters="localhost:9333" -dataDir="~/seaweedfs-admin" +``` + +### With Authentication + +```bash +# Enable authentication +weed admin -adminUser=admin -adminPassword=secret123 -masters="localhost:9333" +``` + +### Production Deployment + +```bash +# Production setup with data persistence and authentication +weed admin \ + -port=23646 \ + -masters="master1:9333,master2:9333,master3:9333" \ + -dataDir="/var/lib/seaweedfs-admin" \ + -adminUser=admin \ + -adminPassword=strongpassword123 +``` + +## Data Directory + +The data directory (`-dataDir`) is used to persist admin configuration and maintenance data: + +- **If specified**: Configuration and data are persisted to disk +- **If not specified**: All configuration is kept in memory only +- **Path expansion**: Supports tilde (`~`) expansion for home directory +- **Auto-creation**: Directory is automatically created if it doesn't exist +- **Format**: Configuration files are stored in JSON format for easy editing + +## Security and Authentication + +### Authentication + +- **Disabled by default**: If `-adminPassword` is not set, no authentication is required +- **Session-based**: When enabled, uses secure session management with auto-generated session keys +- **User credentials**: Login with `-adminUser` and `-adminPassword` + +### TLS/HTTPS Configuration + +The admin server reads TLS configuration from `security.toml`: + +```toml +[https.admin] +cert = "/etc/ssl/admin.crt" +key = "/etc/ssl/admin.key" +ca = "/etc/ssl/ca.crt" # optional, for mutual TLS +``` + +- **HTTPS**: Automatically enabled if `https.admin.key` is configured +- **Mutual TLS**: Enabled if `https.admin.ca` is configured +- **Certificate loading**: Certificates are loaded from the security configuration + +### Security Best Practices + +1. **Set strong passwords**: Use strong `-adminPassword` for production +2. **Configure TLS**: Use HTTPS for production deployments +3. **Firewall rules**: Restrict admin interface access to authorized networks +4. **Regular updates**: Keep SeaweedFS updated for security patches + +## Worker Communication + +The admin server also runs a gRPC server for worker connections: + +- **Port**: HTTP port + 10000 (e.g., if admin runs on 23646, gRPC runs on 33646) +- **Purpose**: Handles maintenance worker connections and task distribution +- **TLS**: Uses `[grpc.admin]` configuration from `security.toml` +- **Fallback**: Workers fall back to insecure connections if TLS is unavailable + +## Configuration File + +The admin server reads configuration from `security.toml` in the following order: +1. Current directory (`.`) +2. `$HOME/.seaweedfs/` +3. `/usr/local/etc/seaweedfs/` +4. `/etc/seaweedfs/` + +### Generate Example Configuration + +```bash +# Generate example security.toml +weed scaffold -config=security +``` + +## Features + +### Automatic Service Discovery + +- **Master discovery**: Connects to specified master servers +- **Filer discovery**: Automatically discovers filers from masters +- **Health monitoring**: Monitors cluster health and status + +### Web Interface + +- **Modern UI**: Bootstrap-based responsive design +- **Real-time updates**: Live cluster status and metrics +- **File management**: Browse and manage files through web interface +- **Maintenance operations**: Execute maintenance tasks from the UI + +### API Endpoints + +The admin interface provides RESTful API endpoints for: +- Cluster status and topology +- Volume management +- File operations +- System metrics +- Configuration management + +## Troubleshooting + +### Common Issues + +1. **No filers discovered**: + - Check master server connectivity + - Verify master addresses are correct + - Ensure masters are running and accessible + +2. **Authentication not working**: + - Verify `-adminPassword` is set correctly + - Check session cookie settings + - Clear browser cache and cookies + +3. **TLS/HTTPS issues**: + - Verify certificate paths in `security.toml` + - Check certificate validity and permissions + - Ensure certificates are in PEM format + +4. **Worker connections failing**: + - Check if gRPC port (HTTP port + 10000) is accessible + - Verify TLS configuration for worker connections + - Check firewall rules for gRPC port + +### Debug Information + +Enable debug logging for detailed troubleshooting: + +```bash +# Run with verbose logging +weed admin -v=4 -masters="localhost:9333" +``` + +## Related Commands + +- [`weed worker`](Weed-Worker.md): Start maintenance workers that connect to admin +- [`weed master`](https://github.com/seaweedfs/seaweedfs/wiki/Master-Server): Start master servers +- [`weed filer`](https://github.com/seaweedfs/seaweedfs/wiki/Filer-Server): Start filer servers +- [`weed scaffold`](https://github.com/seaweedfs/seaweedfs/wiki/Scaffold): Generate configuration files + +## See Also + +- [SeaweedFS Architecture](https://github.com/seaweedfs/seaweedfs/wiki/SeaweedFS-Architecture) +- [Security Configuration](https://github.com/seaweedfs/seaweedfs/wiki/Security-Configuration) +- [Maintenance Operations](https://github.com/seaweedfs/seaweedfs/wiki/Maintenance) \ No newline at end of file From ff1bea50c4a89a9d0ce7f3469fa0d0a213bab0fb Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sun, 6 Jul 2025 22:37:31 -0700 Subject: [PATCH 108/149] Created Worker (markdown) --- Worker.md | 283 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 283 insertions(+) create mode 100644 Worker.md diff --git a/Worker.md b/Worker.md new file mode 100644 index 0000000..8d95857 --- /dev/null +++ b/Worker.md @@ -0,0 +1,283 @@ +# This is still work in progress! + +# Weed Worker + +The `weed worker` command starts a maintenance worker that connects to an admin server to process cluster maintenance tasks. + +## Overview + +Workers are distributed maintenance agents that connect to the admin server to process various maintenance tasks such as: +- **Vacuum**: Reclaim disk space by removing deleted files +- **Erasure Coding**: Convert volumes to erasure-coded format for storage efficiency +- **Remote Upload**: Upload volumes to remote/cloud storage +- **Replication**: Fix replication issues and maintain data consistency +- **Balance**: Redistribute volumes across volume servers for load balancing + +Workers automatically register with the admin server and receive tasks based on their capabilities and current load. + +## Usage + +```bash +weed worker [options] +``` + +## Options + +| Option | Default | Description | +|--------|---------|-------------| +| `-admin` | localhost:23646 | Admin server address | +| `-capabilities` | vacuum,erasure_coding,balance | Comma-separated list of task types this worker can handle | +| `-maxConcurrent` | 2 | Maximum number of concurrent tasks | +| `-heartbeat` | 30s | Heartbeat interval to admin server | +| `-taskInterval` | 5s | Task request interval | + +## Examples + +### Basic Usage + +```bash +# Start worker connecting to local admin server +weed worker -admin=localhost:23646 + +# Connect to remote admin server +weed worker -admin=admin.example.com:23646 + +# Start worker with custom admin server and port +weed worker -admin=192.168.1.100:8080 +``` + +### Capability Configuration + +```bash +# Worker that only handles vacuum tasks +weed worker -admin=localhost:23646 -capabilities=vacuum + +# Worker that handles vacuum and replication tasks +weed worker -admin=localhost:23646 -capabilities=vacuum,replication + +# Worker with all capabilities (default) +weed worker -admin=localhost:23646 -capabilities=vacuum,ec,remote,replication,balance + +# Worker using capability aliases +weed worker -admin=localhost:23646 -capabilities=vacuum,ec,remote,replication +``` + +### Performance Tuning + +```bash +# High-performance worker with more concurrent tasks +weed worker -admin=localhost:23646 -maxConcurrent=8 + +# More frequent task requests for busy clusters +weed worker -admin=localhost:23646 -taskInterval=2s + +# Custom heartbeat interval +weed worker -admin=localhost:23646 -heartbeat=10s +``` + +## Task Capabilities + +Workers can be configured to handle specific types of maintenance tasks: + +### Available Task Types + +| Capability | Description | +|------------|-------------| +| `vacuum` | Reclaim disk space by removing deleted files | +| `erasure_coding` | Convert volumes to erasure-coded format | +| `balance` | Redistribute volumes for load balancing | + +## Worker Architecture + +### Worker Lifecycle + +1. **Registration**: Worker connects to admin server via gRPC +2. **Capabilities**: Worker reports its capabilities to admin +3. **Task Request**: Worker periodically requests tasks from admin +4. **Task Execution**: Worker processes assigned tasks +5. **Heartbeat**: Worker sends periodic heartbeats to admin +6. **Graceful Shutdown**: Worker completes current tasks before stopping + +### Connection Details + +- **Protocol**: gRPC connection to admin server +- **Port**: Admin HTTP port + 10000 (e.g., admin on 23646 → gRPC on 33646) +- **Security**: Supports TLS using `[grpc.worker]` configuration +- **Fallback**: Falls back to insecure connection if TLS unavailable + +## Configuration + +### Security Configuration + +Workers read TLS configuration from `security.toml`: + +```toml +[grpc.worker] +cert = "/etc/ssl/worker.crt" +key = "/etc/ssl/worker.key" +ca = "/etc/ssl/ca.crt" +``` + +### Worker Identification + +- **Worker ID**: Automatically generated unique identifier +- **Address**: Worker's network address (auto-detected) +- **Capabilities**: Reported task capabilities +- **Status**: Current worker status (active, idle, busy) + +## Task Processing + +### Concurrent Task Handling + +- **Max Concurrent**: Configurable via `-maxConcurrent` (default: 2) +- **Task Queue**: Workers maintain internal task queues +- **Load Balancing**: Admin distributes tasks based on worker load +- **Task Completion**: Workers report task completion status + +### Task Request Cycle + +1. Worker requests tasks from admin server +2. Admin assigns tasks based on worker capabilities and load +3. Worker processes tasks concurrently +4. Worker reports task completion/failure +5. Cycle repeats based on `-taskInterval` + +## Monitoring and Status + +### Worker Status + +Workers report the following status information: +- **Worker ID**: Unique identifier +- **Current Load**: Number of active tasks +- **Capabilities**: Supported task types +- **Last Heartbeat**: Timestamp of last heartbeat +- **Tasks Completed**: Total completed tasks +- **Tasks Failed**: Total failed tasks +- **Uptime**: Worker uptime duration + +### Health Monitoring + +- **Heartbeat**: Periodic heartbeat to admin server +- **Task Timeout**: Tasks have configurable timeouts +- **Error Reporting**: Failed tasks are reported to admin +- **Automatic Retry**: Failed tasks may be retried + +## Best Practices + +### Deployment + +1. **Multiple Workers**: Deploy multiple workers for redundancy +2. **Capability Specialization**: Consider specialized workers for specific tasks +3. **Resource Allocation**: Ensure adequate CPU and memory for concurrent tasks +4. **Network Connectivity**: Ensure reliable connection to admin server + +### Performance + +1. **Concurrent Tasks**: Tune `-maxConcurrent` based on available resources +2. **Task Interval**: Adjust `-taskInterval` based on cluster activity +3. **Heartbeat Frequency**: Balance between responsiveness and overhead +4. **Resource Monitoring**: Monitor worker resource usage + +### Security + +1. **TLS Configuration**: Use TLS for production deployments +2. **Network Security**: Secure communication between workers and admin +3. **Access Control**: Limit worker deployment to trusted systems +4. **Certificate Management**: Manage and rotate TLS certificates + +## Troubleshooting + +### Common Issues + +1. **Cannot connect to admin server**: + - Verify admin server address and port + - Check network connectivity + - Ensure admin server is running + - Verify gRPC port (admin HTTP port + 10000) + +2. **No tasks received**: + - Check worker capabilities match available tasks + - Verify worker registration with admin + - Check admin server logs for task assignment + - Ensure worker is not overloaded + +3. **TLS connection failures**: + - Verify `security.toml` configuration + - Check certificate paths and permissions + - Ensure certificates are valid + - Check certificate compatibility + +4. **Task execution failures**: + - Check worker logs for error details + - Verify worker has necessary permissions + - Check disk space and resources + - Ensure target volumes are accessible + +### Debug Information + +Enable debug logging: + +```bash +# Run with verbose logging +weed worker -admin=localhost:23646 -v=4 +``` + +### Worker Logs + +Workers log important events: +- Connection status to admin server +- Task assignments and completion +- Error conditions and failures +- Heartbeat and health information + +## Task-Specific Information + +### Vacuum Tasks + +- **Purpose**: Reclaim disk space from deleted files +- **Requirements**: Access to volume servers +- **Duration**: Varies based on volume size and deleted data +- **Impact**: Temporary increase in I/O during vacuum process + +### Erasure Coding Tasks + +- **Purpose**: Convert volumes to erasure-coded format +- **Requirements**: Multiple volume servers for redundancy +- **Duration**: Long-running, depends on volume size +- **Impact**: Reduces storage requirements but increases complexity + +### Remote Upload Tasks + +- **Purpose**: Upload volumes to remote/cloud storage +- **Requirements**: Cloud storage credentials and connectivity +- **Duration**: Depends on volume size and upload bandwidth +- **Impact**: Enables tiered storage and backup strategies + +### Replication Tasks + +- **Purpose**: Fix replication consistency issues +- **Requirements**: Access to master and volume servers +- **Duration**: Quick, depends on replication factor +- **Impact**: Ensures data consistency and availability + +### Balance Tasks + +- **Purpose**: Redistribute volumes across volume servers +- **Requirements**: Multiple volume servers +- **Duration**: Depends on data movement requirements +- **Impact**: Improves cluster load distribution + +## Related Commands + +- [`weed admin`](Weed-Admin.md): Start admin server that manages workers +- [`weed master`](https://github.com/seaweedfs/seaweedfs/wiki/Master-Server): Start master servers +- [`weed volume`](https://github.com/seaweedfs/seaweedfs/wiki/Volume-Server): Start volume servers +- [`weed scaffold`](https://github.com/seaweedfs/seaweedfs/wiki/Scaffold): Generate configuration files + +## See Also + +- [SeaweedFS Architecture](https://github.com/seaweedfs/seaweedfs/wiki/SeaweedFS-Architecture) +- [Maintenance Operations](https://github.com/seaweedfs/seaweedfs/wiki/Maintenance) +- [Security Configuration](https://github.com/seaweedfs/seaweedfs/wiki/Security-Configuration) +- [Erasure Coding](https://github.com/seaweedfs/seaweedfs/wiki/Erasure-Coding) +- [Remote Storage](https://github.com/seaweedfs/seaweedfs/wiki/Remote-Storage) \ No newline at end of file From 81f462cee230f4015212b0c60c3e46ad53564d17 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sun, 6 Jul 2025 22:38:13 -0700 Subject: [PATCH 109/149] Updated Admin UI (markdown) --- Admin-UI.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Admin-UI.md b/Admin-UI.md index 3d468e2..214ea2e 100644 --- a/Admin-UI.md +++ b/Admin-UI.md @@ -1,3 +1,5 @@ +# This is still work in progress. Some features work, some not. + # Weed Admin The `weed admin` command starts a modern web-based administration interface for SeaweedFS cluster management. From 3779dd210d8440867dfd280490a9e73d41cc984a Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sun, 6 Jul 2025 22:38:38 -0700 Subject: [PATCH 110/149] Updated Admin UI (markdown) --- Admin-UI.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Admin-UI.md b/Admin-UI.md index 214ea2e..f113925 100644 --- a/Admin-UI.md +++ b/Admin-UI.md @@ -1,4 +1,4 @@ -# This is still work in progress. Some features work, some not. +# This is still work in progress. Some features work, some not. Everything is subject to change. # Weed Admin From ab03887e37350dcb430476eafbd914864836b8b0 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Sun, 13 Jul 2025 20:15:01 -0700 Subject: [PATCH 111/149] add object locking and retention --- Amazon-S3-API.md | 160 ++++++++++++ S3-API-FAQ.md | 62 +++++ S3-Object-Lock-and-Retention.md | 424 ++++++++++++++++++++++++++++++++ _Sidebar.md | 1 + 4 files changed, 647 insertions(+) create mode 100644 S3-Object-Lock-and-Retention.md diff --git a/Amazon-S3-API.md b/Amazon-S3-API.md index d08f558..7443ba7 100644 --- a/Amazon-S3-API.md +++ b/Amazon-S3-API.md @@ -72,6 +72,25 @@ To be sure, you can look at the function defined in the files `weed/s3api/s3api_ * PutObjectPart * CopyObjectPart * ListObjectParts + +// Object Versioning +* PutBucketVersioning +* GetBucketVersioning +* ListObjectVersions +* GetObject (with version ID) +* PutObject (with versioning) +* DeleteObject (with version ID) +* CopyObject (with version ID) +* RestoreObject (partial) + +// Object Lock and Retention +* GetObjectLockConfiguration +* PutObjectLockConfiguration +* GetObjectRetention +* PutObjectRetention +* GetObjectLegalHold +* PutObjectLegalHold +* BypassGovernanceRetention (via x-amz-bypass-governance-retention header) ``` Not included: @@ -85,6 +104,8 @@ Not included: | DeleteObject deletes a folder | Yes | No | | same path for both a file and a folder | No | Yes | | allows more than "/" as a delimiter | No | Yes | +| Object Versioning | Yes | Yes | +| MFA Delete for versioning | No | Yes | ## Empty folders @@ -245,6 +266,145 @@ Read:bucket/a/b/* Presigned URL is supported. See [[AWS-CLI-with-SeaweedFS#presigned-url]] for example. +# S3 Object Versioning + +SeaweedFS supports S3 object versioning, which allows you to keep multiple variants of an object in the same bucket. This provides data protection against accidental deletion or modification. + +## Enable Versioning + +To enable versioning on a bucket, use the `PutBucketVersioning` API: + +```bash +aws s3api put-bucket-versioning \ + --bucket my-bucket \ + --versioning-configuration Status=Enabled +``` + +## Check Versioning Status + +To check the versioning status of a bucket: + +```bash +aws s3api get-bucket-versioning --bucket my-bucket +``` + +Response: +```json +{ + "Status": "Enabled" +} +``` + +## Suspend Versioning + +To suspend versioning (not disable completely): + +```bash +aws s3api put-bucket-versioning \ + --bucket my-bucket \ + --versioning-configuration Status=Suspended +``` + +## List Object Versions + +To list all versions of objects in a bucket: + +```bash +aws s3api list-object-versions --bucket my-bucket +``` + +Response includes both object versions and delete markers: +```json +{ + "Versions": [ + { + "Key": "example.txt", + "VersionId": "v_1234567890abcdef", + "IsLatest": true, + "LastModified": "2023-12-01T10:00:00Z", + "ETag": "\"abcdef1234567890\"", + "Size": 1024 + } + ], + "DeleteMarkers": [ + { + "Key": "deleted-file.txt", + "VersionId": "v_fedcba0987654321", + "IsLatest": true, + "LastModified": "2023-12-01T11:00:00Z" + } + ] +} +``` + +## Access Specific Versions + +### Get a specific version of an object: +```bash +aws s3api get-object \ + --bucket my-bucket \ + --key example.txt \ + --version-id v_1234567890abcdef \ + output.txt +``` + +### Copy a specific version: +```bash +aws s3api copy-object \ + --copy-source my-bucket/example.txt?versionId=v_1234567890abcdef \ + --bucket my-bucket \ + --key example-copy.txt +``` + +### Delete a specific version: +```bash +aws s3api delete-object \ + --bucket my-bucket \ + --key example.txt \ + --version-id v_1234567890abcdef +``` + +## Versioning Behavior + +### When Versioning is Enabled: +- **PUT Object**: Creates a new version with a unique version ID +- **GET Object**: Returns the latest version (unless version ID is specified) +- **DELETE Object**: Creates a delete marker (soft delete) +- **DELETE Object with version ID**: Permanently deletes that specific version + +### When Versioning is Suspended: +- **PUT Object**: Overwrites the object with version ID "null" +- **GET Object**: Returns the current version +- **DELETE Object**: Permanently deletes the object + +## Storage Layout + +SeaweedFS stores versioned objects using the following structure: +``` +/buckets/my-bucket/ +├── example.txt # Current version (if versioning suspended) +└── example.txt.versions/ + ├── v_1234567890abcdef # Version 1 + ├── v_fedcba0987654321 # Version 2 + └── v_abcdef1234567890 # Version 3 (latest) +``` + +## Limitations + +- **Version ID Format**: SeaweedFS generates version IDs in the format `v_<32-char-hex>` +- **Restore Operations**: Only partial support for `RestoreObject` API +- **Lifecycle Policies**: Version-specific lifecycle rules are not fully implemented +- **MFA Delete**: Not currently supported + +## Best Practices + +1. **Enable versioning before storing important data** to ensure all versions are captured +2. **Monitor storage usage** as versioning can increase storage consumption +3. **Implement lifecycle policies** to automatically clean up old versions +4. **Use version-specific operations** when you need to access historical data +5. **Consider performance impact** when listing many versions of objects + + ## Multiple S3 Nodes If you need to setup multiple S3 nodes, you can just start multiple s3 instances pointing to a filer. diff --git a/S3-API-FAQ.md b/S3-API-FAQ.md index c1ca16e..728c6d6 100644 --- a/S3-API-FAQ.md +++ b/S3-API-FAQ.md @@ -88,4 +88,66 @@ aws --endpoint-url http://127.0.0.1:8333 s3api put-bucket-lifecycle-configuratio Note that you don't need to add the part `/buckets/$BUCKET_NAME` in the configurations "Filter.Prefix" (contrary to using `fs.configure`, this is taken care of for you in the S3 API. +## Does SeaweedFS support S3 object versioning? + +Yes, SeaweedFS supports S3 object versioning. You can enable versioning on a bucket using the `PutBucketVersioning` API. When enabled, SeaweedFS will store multiple versions of an object in the same bucket, providing data protection against accidental deletion or modification. + +Key features supported: +- Enable/suspend versioning on buckets +- List all versions of objects +- Get, copy, and delete specific versions +- Delete markers for soft deletion + +For detailed documentation and examples, see [[Amazon S3 API#s3-object-versioning]]. + +## How does versioning affect storage usage? + +When versioning is enabled, each uploaded object creates a new version instead of overwriting the existing one. This means: +- Storage usage will increase as you accumulate versions +- All versions are preserved until explicitly deleted +- Delete operations create delete markers (soft delete) rather than immediately removing data + +To manage storage growth, you should: +- Monitor storage usage regularly +- Implement lifecycle policies to automatically clean up old versions +- Use version-specific deletions for permanent removal when needed + +## Does SeaweedFS support S3 Object Lock? + +Yes! SeaweedFS provides comprehensive support for S3 Object Lock features, including: + +### Object Lock Features +- **Governance Mode**: Objects can be deleted/modified by users with `s3:BypassGovernanceRetention` permission +- **Compliance Mode**: Objects cannot be deleted/modified by any user until retention expires +- **Legal Hold**: Objects cannot be deleted/modified until legal hold is explicitly removed + +### Supported APIs +- `GetObjectLockConfiguration` / `PutObjectLockConfiguration` (bucket-level) +- `GetObjectRetention` / `PutObjectRetention` (object-level) +- `GetObjectLegalHold` / `PutObjectLegalHold` (object-level) +- Governance bypass via `x-amz-bypass-governance-retention` header + +### Requirements +- Object Lock must be enabled when creating the bucket (cannot be added later) +- Versioning is automatically enabled and required for Object Lock +- Compatible with standard AWS S3 tools and SDKs + +For complete documentation, examples, and best practices, see [[S3 Object Lock and Retention]]. + +## What's the difference between Governance and Compliance modes? + +**Governance Mode**: +- Designed for internal governance and compliance requirements +- Can be bypassed by users with proper permissions (`s3:BypassGovernanceRetention`) +- Admin users can always bypass governance retention +- Suitable for testing and development environments + +**Compliance Mode**: +- Designed for regulatory compliance (SEC, FINRA, etc.) +- Cannot be bypassed by any user, including root/admin +- Provides the highest level of data protection +- Suitable for production environments with strict compliance requirements + +Both modes prevent accidental deletion and provide audit trails for compliance purposes. + diff --git a/S3-Object-Lock-and-Retention.md b/S3-Object-Lock-and-Retention.md new file mode 100644 index 0000000..6af1ebb --- /dev/null +++ b/S3-Object-Lock-and-Retention.md @@ -0,0 +1,424 @@ +# S3 Object Lock and Retention + +SeaweedFS provides comprehensive support for Amazon S3 Object Lock functionality, including object versioning, retention policies, legal holds, and WORM (Write Once Read Many) compliance features. + +## Overview + +Object Lock is a feature that allows you to store objects using a WORM (Write Once Read Many) model. It helps prevent objects from being deleted or overwritten for a fixed amount of time or indefinitely. Object Lock works in conjunction with versioning and is enabled at the bucket level. + +## Prerequisites + +- **Versioning Required**: Object Lock can only be enabled on buckets that have versioning enabled +- **Immutable Setting**: Object Lock can only be enabled when creating a bucket, not on existing buckets +- **S3 API**: All Object Lock operations are available through the S3 API + +## Key Features + +### 1. Retention Modes + +#### Governance Mode +- Objects are protected from deletion or modification by most users +- Users with `s3:BypassGovernanceRetention` permission can override protection +- Admin users can always bypass governance retention +- Suitable for internal compliance and governance requirements + +#### Compliance Mode +- Objects cannot be deleted or modified by any user, including the root user +- Retention periods cannot be shortened +- Provides the highest level of protection +- Suitable for regulatory compliance requirements + +### 2. Legal Hold +- Independent protection mechanism that can be applied to any object +- When enabled, objects cannot be deleted or modified until the legal hold is removed +- Can be used alongside retention policies +- Useful for litigation, investigation, or audit requirements + + + +## API Support + +SeaweedFS implements the complete S3 Object Lock API: + +### Bucket-Level Operations +- `GET /?object-lock` - Get bucket Object Lock configuration +- `PUT /?object-lock` - Set bucket Object Lock configuration + +### Object-Level Operations +- `GET /{object}?retention` - Get object retention settings +- `PUT /{object}?retention` - Set object retention settings +- `GET /{object}?legal-hold` - Get object legal hold status +- `PUT /{object}?legal-hold` - Set object legal hold status + +### Versioning Operations +- `GET /?versioning` - Get bucket versioning status +- `PUT /?versioning` - Set bucket versioning status +- `GET /?versions` - List object versions + +## Setup and Configuration + +### 1. Enable Object Lock on Bucket Creation + +Object Lock must be enabled when creating a bucket: + +```bash +# Create bucket with Object Lock enabled +aws s3api create-bucket \ + --bucket my-secure-bucket \ + --object-lock-enabled-for-bucket + +# Enable versioning (automatically done with Object Lock) +aws s3api put-bucket-versioning \ + --bucket my-secure-bucket \ + --versioning-configuration Status=Enabled +``` + +### 2. Configure Governance Bypass Permissions + +Grant users permission to bypass governance retention: + +```json +{ + "identities": [ + { + "name": "compliance-admin", + "credentials": [ + { + "accessKey": "admin123", + "secretKey": "secret123" + } + ], + "actions": [ + "Read:my-secure-bucket/*", + "Write:my-secure-bucket/*", + "BypassGovernanceRetention:my-secure-bucket/*" + ] + } + ] +} +``` + +## Usage Examples + +### 1. Object Retention + +#### Set Governance Retention +```bash +# Set 30-day governance retention +aws s3api put-object-retention \ + --bucket my-secure-bucket \ + --key important-document.pdf \ + --retention '{ + "Mode": "GOVERNANCE", + "RetainUntilDate": "2024-12-31T23:59:59Z" + }' +``` + +#### Set Compliance Retention +```bash +# Set 7-year compliance retention +aws s3api put-object-retention \ + --bucket my-secure-bucket \ + --key regulatory-record.json \ + --retention '{ + "Mode": "COMPLIANCE", + "RetainUntilDate": "2031-01-01T00:00:00Z" + }' +``` + +#### Get Object Retention +```bash +aws s3api get-object-retention \ + --bucket my-secure-bucket \ + --key important-document.pdf +``` + +### 2. Legal Hold + +#### Apply Legal Hold +```bash +aws s3api put-object-legal-hold \ + --bucket my-secure-bucket \ + --key investigation-file.doc \ + --legal-hold Status=ON +``` + +#### Remove Legal Hold +```bash +aws s3api put-object-legal-hold \ + --bucket my-secure-bucket \ + --key investigation-file.doc \ + --legal-hold Status=OFF +``` + +#### Check Legal Hold Status +```bash +aws s3api get-object-legal-hold \ + --bucket my-secure-bucket \ + --key investigation-file.doc +``` + +### 3. Governance Bypass + +#### Delete Object with Governance Bypass +```bash +# User with bypass permission can delete governance-protected objects +aws s3api delete-object \ + --bucket my-secure-bucket \ + --key document.pdf \ + --bypass-governance-retention +``` + +#### Bulk Delete with Governance Bypass +```bash +# Delete multiple objects with governance bypass +aws s3api delete-objects \ + --bucket my-secure-bucket \ + --delete file://delete-objects.json \ + --bypass-governance-retention +``` + +### 4. Object Versioning + +#### List Object Versions +```bash +aws s3api list-object-versions \ + --bucket my-secure-bucket \ + --prefix documents/ +``` + +#### Get Specific Version +```bash +aws s3api get-object \ + --bucket my-secure-bucket \ + --key document.pdf \ + --version-id "3/L4kqtJlcpXroDTDmpUMLUo" +``` + +#### Delete Specific Version +```bash +aws s3api delete-object \ + --bucket my-secure-bucket \ + --key document.pdf \ + --version-id "3/L4kqtJlcpXroDTDmpUMLUo" +``` + +## Permission System + +### IAM Actions + +SeaweedFS supports the following S3 IAM actions for Object Lock in identity-based permissions: + +- `s3:GetObjectRetention` - Get object retention settings +- `s3:PutObjectRetention` - Set object retention settings +- `s3:GetObjectLegalHold` - Get object legal hold status +- `s3:PutObjectLegalHold` - Set object legal hold status +- `s3:BypassGovernanceRetention` - Bypass governance retention +- `s3:GetBucketObjectLockConfiguration` - Get bucket Object Lock configuration +- `s3:PutBucketObjectLockConfiguration` - Set bucket Object Lock configuration + +### Identity-Based Permissions + +Configure user permissions in `identities.json`: + +```json +{ + "identities": [ + { + "name": "read-only-user", + "credentials": [{"accessKey": "ro123", "secretKey": "secret123"}], + "actions": [ + "Read:my-secure-bucket/*", + "GetObjectRetention:my-secure-bucket/*", + "GetObjectLegalHold:my-secure-bucket/*" + ] + }, + { + "name": "compliance-admin", + "credentials": [{"accessKey": "admin123", "secretKey": "secret123"}], + "actions": [ + "Read:my-secure-bucket/*", + "Write:my-secure-bucket/*", + "Admin:my-secure-bucket/*", + "BypassGovernanceRetention:my-secure-bucket/*" + ] + } + ] +} +``` + +## Error Handling + +### Common Error Scenarios + +1. **Object Lock Not Enabled** + - Error: `InvalidRequest` + - Solution: Enable Object Lock when creating the bucket + +2. **Versioning Not Enabled** + - Error: `InvalidRequest` + - Solution: Enable versioning before configuring Object Lock + +3. **Governance Bypass Not Permitted** + - Error: `AccessDenied` + - Solution: Grant `s3:BypassGovernanceRetention` permission + +4. **Compliance Mode Cannot Be Bypassed** + - Error: `AccessDenied` + - Solution: Wait for retention period to expire + +5. **Object Under Legal Hold** + - Error: `AccessDenied` + - Solution: Remove legal hold before deleting/modifying + +### Error Response Examples + +```xml + + + AccessDenied + Object is under COMPLIANCE mode retention and cannot be deleted + /my-secure-bucket/protected-file.pdf + 12345 + +``` + +## WORM Compliance + +SeaweedFS Object Lock provides WORM (Write Once Read Many) compliance: + +### Features +- **Immutable Objects**: Once written, objects cannot be modified +- **Retention Enforcement**: Objects cannot be deleted before retention expiry +- **Legal Hold Support**: Additional protection for litigation/investigation +- **Audit Trail**: All access attempts are logged for compliance + +### Use Cases +- **Regulatory Compliance**: Financial records, healthcare data, legal documents +- **Data Archiving**: Long-term storage with guaranteed integrity +- **Backup Protection**: Prevent accidental or malicious deletion +- **Audit Requirements**: Maintain evidence for legal proceedings + +## Best Practices + +### 1. Retention Planning +- **Start with Governance**: Use governance mode for testing and development +- **Compliance for Production**: Use compliance mode for regulatory requirements +- **Gradual Implementation**: Start with short retention periods and increase gradually + +### 2. Permission Management +- **Principle of Least Privilege**: Grant minimum required permissions +- **Separate Roles**: Different permissions for different use cases +- **Regular Audits**: Review and update permissions regularly + +### 3. Monitoring and Alerting +- **Monitor Bypass Attempts**: Track governance bypass usage +- **Retention Expiry Alerts**: Set up alerts for expiring retention periods +- **Failed Access Attempts**: Monitor and investigate access denials + +### 4. Backup and Recovery +- **Multiple Copies**: Maintain backups of Object Lock configurations +- **Cross-Region Replication**: Replicate to different regions for disaster recovery +- **Version Management**: Keep track of object versions and their retention status + +## Performance Considerations + +### Storage Efficiency +- **Versioning Overhead**: Each version consumes storage space +- **Retention Metadata**: Minimal overhead for Object Lock metadata +- **Cleanup Strategies**: Plan for version cleanup after retention expires + +### Access Patterns +- **Read-Heavy Workloads**: Object Lock is optimized for read operations +- **Write-Once Pattern**: Best suited for write-once, read-many scenarios +- **Batch Operations**: Use batch operations for bulk retention management + +## Limitations + +### Current Limitations +- **Bucket-Level Only**: Object Lock must be enabled during bucket creation +- **No Modification**: Existing buckets cannot be converted to Object Lock +- **Retention Limits**: Maximum 100 years retention period +- **Version Limits**: Consider version proliferation in high-write scenarios + +### AWS Compatibility +SeaweedFS Object Lock is fully compatible with AWS S3 Object Lock: +- Same API endpoints and parameters +- Identical error responses +- Compatible with AWS CLI and SDKs +- Consistent behavior across governance and compliance modes + +## Troubleshooting + +### Common Issues + +1. **Cannot Enable Object Lock** + - Check if versioning is enabled + - Ensure bucket is being created (not existing) + - Verify permissions for bucket operations + +2. **Retention Settings Not Applied** + - Verify Object Lock is enabled on bucket + - Check retention date format (ISO 8601) + - Ensure retention mode is valid (GOVERNANCE/COMPLIANCE) + +3. **Governance Bypass Fails** + - Verify user has `s3:BypassGovernanceRetention` permission + - Check if bypass header is included in request + - Ensure object is in GOVERNANCE mode (not COMPLIANCE) + +### Debug Commands + +```bash +# Check bucket Object Lock configuration +aws s3api get-object-lock-configuration --bucket my-secure-bucket + +# Check bucket versioning status +aws s3api get-bucket-versioning --bucket my-secure-bucket + +# Check object retention +aws s3api get-object-retention --bucket my-secure-bucket --key file.pdf + +# Check legal hold status +aws s3api get-object-legal-hold --bucket my-secure-bucket --key file.pdf +``` + +## Migration Guide + +### From Non-Object Lock Buckets + +1. **Create New Bucket**: Create new bucket with Object Lock enabled +2. **Copy Data**: Use `aws s3 sync` to copy data to new bucket +3. **Apply Retention**: Set retention policies on copied objects +4. **Update Applications**: Update applications to use new bucket +5. **Cleanup**: Remove old bucket after validation + +### From Other S3 Providers + +Object Lock configurations can be migrated using standard S3 API calls: + +```bash +# Export current configuration +aws s3api get-object-lock-configuration --bucket source-bucket > config.json + +# Apply to SeaweedFS bucket +aws s3api put-object-lock-configuration \ + --bucket target-bucket \ + --object-lock-configuration file://config.json +``` + + + +## Related Documentation + +- [S3 API FAQ](S3-API-FAQ.md) +- [Amazon S3 API](Amazon-S3-API.md) +- [Security Configuration](Security-Configuration.md) +- [Production Setup](Production-Setup.md) + +## Support + +For issues or questions about Object Lock features: +- GitHub Issues: [SeaweedFS Issues](https://github.com/seaweedfs/seaweedfs/issues) +- Community Forum: [SeaweedFS Discussions](https://github.com/seaweedfs/seaweedfs/discussions) +- Documentation: [SeaweedFS Wiki](https://github.com/seaweedfs/seaweedfs/wiki) \ No newline at end of file diff --git a/_Sidebar.md b/_Sidebar.md index d9762d2..f5382b8 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -67,6 +67,7 @@ ### AWS S3 API * [[Amazon S3 API]] +* [[S3 Object Lock and Retention]] * [[AWS CLI with SeaweedFS]] * [[s3cmd with SeaweedFS]] * [[rclone with SeaweedFS]] From 342cb066d0c2f589f82ec6ef4116d3b5c5f1ead2 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Sun, 13 Jul 2025 20:30:23 -0700 Subject: [PATCH 112/149] Update Admin-UI.md --- Admin-UI.md | 295 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 286 insertions(+), 9 deletions(-) diff --git a/Admin-UI.md b/Admin-UI.md index f113925..c29ca28 100644 --- a/Admin-UI.md +++ b/Admin-UI.md @@ -12,7 +12,6 @@ The admin interface provides a comprehensive web UI for managing SeaweedFS clust - **File browser and management** - **System metrics and performance monitoring** - **Configuration management** -- **Maintenance operations** The admin interface automatically discovers filers from the master servers and runs a gRPC server for worker connections on HTTP port + 10000. @@ -71,7 +70,7 @@ weed admin \ ## Data Directory -The data directory (`-dataDir`) is used to persist admin configuration and maintenance data: +The data directory (`-dataDir`) is used to persist admin configuration data: - **If specified**: Configuration and data are persisted to disk - **If not specified**: All configuration is kept in memory only @@ -114,7 +113,7 @@ ca = "/etc/ssl/ca.crt" # optional, for mutual TLS The admin server also runs a gRPC server for worker connections: - **Port**: HTTP port + 10000 (e.g., if admin runs on 23646, gRPC runs on 33646) -- **Purpose**: Handles maintenance worker connections and task distribution +- **Purpose**: Handles worker connections and task distribution - **TLS**: Uses `[grpc.admin]` configuration from `security.toml` - **Fallback**: Workers fall back to insecure connections if TLS is unavailable @@ -143,10 +142,41 @@ weed scaffold -config=security ### Web Interface -- **Modern UI**: Bootstrap-based responsive design -- **Real-time updates**: Live cluster status and metrics -- **File management**: Browse and manage files through web interface -- **Maintenance operations**: Execute maintenance tasks from the UI +The admin interface provides a comprehensive web-based management console with the following sections: + +#### Dashboard +- **Cluster Overview**: Real-time cluster status and health metrics +- **System Statistics**: Total volumes, files, size, and volume size limits +- **Node Status**: Master, filer, volume server, and message broker status +- **Data Centers**: Geographic distribution of storage nodes + +#### Object Store Management +- **S3 Buckets**: View, create, delete, and manage S3-compatible buckets +- **Bucket Details**: Quota management and configuration +- **User Management**: Create and manage S3 API users with permissions +- **Access Keys**: Generate and manage access/secret key pairs +- **Policies**: Manage bucket policies and user permissions + +#### File Browser +- **Directory Navigation**: Browse filesystem hierarchy through web interface +- **File Operations**: Upload, download, delete, and manage files +- **File Properties**: View file metadata, permissions, and storage details +- **Bulk Operations**: Multi-select for batch operations + +#### Cluster Management +- **Master Servers**: View master node status, leadership, and connectivity +- **Filer Servers**: Monitor filer instances and metadata operations +- **Volume Servers**: Track storage nodes, capacity, and health status +- **Volume Management**: View volume distribution, replication, and status +- **Collections**: Monitor data collections and their volume allocation + +#### Message Queue Management +- **Brokers**: View message queue broker status and configuration +- **Topics**: Manage topics, partitions, and message retention +- **Subscribers**: Monitor subscriber connections and consumer lag +- **Topic Details**: View message statistics and partition distribution + + ### API Endpoints @@ -192,13 +222,260 @@ weed admin -v=4 -masters="localhost:9333" ## Related Commands -- [`weed worker`](Weed-Worker.md): Start maintenance workers that connect to admin + - [`weed master`](https://github.com/seaweedfs/seaweedfs/wiki/Master-Server): Start master servers - [`weed filer`](https://github.com/seaweedfs/seaweedfs/wiki/Filer-Server): Start filer servers - [`weed scaffold`](https://github.com/seaweedfs/seaweedfs/wiki/Scaffold): Generate configuration files +## Admin UI Pages Reference + +### Accessing the Admin Interface + +Once the admin server is running, access the web interface at: +``` +http://localhost:23646 +``` + +Or with custom port: +``` +http://localhost:PORT +``` + +### Dashboard (/) + +The main dashboard provides a comprehensive overview of your SeaweedFS cluster: + +**Key Metrics**: +- Total volumes, files, and storage size +- Volume size limit configuration +- Cluster health status + +**Cluster Topology**: +- Master server status and leader election +- Filer server instances and connections +- Volume server distribution and capacity +- Message broker status (if enabled) +- Data center geographic distribution + +**Real-time Updates**: The dashboard automatically refreshes to show current cluster status. + +### Object Store Management + +#### S3 Buckets (/object-store/buckets) + +Manage S3-compatible storage buckets: + +**Features**: +- View all buckets with creation dates and sizes +- Create new buckets with Object Lock support +- Delete buckets (with confirmation) +- Set bucket quotas and limits +- Export bucket list to CSV + +**Bucket Operations**: +- **Create Bucket**: Specify name and optional Object Lock enablement +- **Delete Bucket**: Remove empty buckets with confirmation dialog +- **Quota Management**: Set storage limits per bucket +- **Bucket Details**: View detailed statistics and configuration + +#### User Management (/object-store/users) + +Manage S3 API users and their permissions: + +**User Operations**: +- Create new users with email and permissions +- Edit existing user permissions and details +- Delete users with confirmation +- Generate and manage access keys +- View user activity and permissions + +**Permission Types**: +- **Admin**: Full access to all buckets and operations +- **Read**: Read access to specified buckets +- **Write**: Write access to specified buckets +- **List**: List bucket contents +- **Tagging**: Manage object tags +- **Object Lock Permissions**: + - **BypassGovernanceRetention**: Override governance retention + - **GetObjectRetention**: Read object retention settings + - **PutObjectRetention**: Modify object retention + - **GetObjectLegalHold**: Read legal hold status + - **PutObjectLegalHold**: Set legal hold + - **GetBucketObjectLockConfiguration**: Read bucket Object Lock config + - **PutBucketObjectLockConfiguration**: Modify bucket Object Lock config + +**Access Key Management**: +- Generate new access/secret key pairs +- View existing access keys (secret keys are masked) +- Delete unused access keys +- Copy keys to clipboard + +#### Policies (/object-store/policies) + +Manage bucket policies and access control: + +**Policy Operations**: +- Create JSON-based bucket policies +- Edit existing policies with syntax validation +- Delete policies +- Validate policy syntax before saving +- View policy effects and permissions + +### File Browser (/files) + +Web-based file system interface: + +**Navigation**: +- Browse directory hierarchy +- Navigate with breadcrumb navigation +- Search files and folders +- Sort by name, size, or modification date + +**File Operations**: +- **Upload**: Single or multiple file upload with progress tracking +- **Download**: Direct file download or streaming +- **Delete**: Remove files and folders with confirmation +- **Create Folders**: New directory creation +- **Rename**: File and folder renaming +- **Copy/Move**: File management operations + +**Advanced Features**: +- Drag-and-drop file upload +- Bulk selection for batch operations +- File property viewing (size, permissions, metadata) +- Preview for supported file types +- Export file listings to CSV + +### Cluster Management + +#### Master Servers (/cluster/masters) + +Monitor master server cluster: + +**Information Displayed**: +- Master server addresses and ports +- Leader election status +- Connection health and response times +- Configuration synchronization status +- Cluster membership changes + +#### Filer Servers (/cluster/filers) + +Track filer instances: + +**Monitoring**: +- Filer server addresses and health +- Metadata store backend status +- Connected clients and operations +- Performance metrics and response times + +#### Volume Servers (/cluster/volume-servers) + +Manage storage nodes: + +**Server Information**: +- Server addresses and capacity +- Free space and utilization +- Active volume counts +- Data center and rack assignment +- Health status and connectivity + +#### Volume Management (/cluster/volumes) + +Detailed volume tracking: + +**Volume Details**: +- Volume ID and size information +- Replication status and factor +- Read/write statistics +- Storage location and server mapping +- Collection assignment + +**Operations**: +- View volume distribution across servers +- Monitor replication health +- Track volume growth and utilization + +#### Collections (/cluster/collections) + +Monitor data collections: + +**Collection Information**: +- Collection names and volume counts +- Replication configuration +- Storage distribution +- Growth patterns and capacity planning + +### Message Queue Management + +#### Brokers (/mq/brokers) + +Monitor message queue brokers: + +**Broker Status**: +- Broker addresses and health +- Topic assignment and leadership +- Connection counts and throughput +- Configuration and settings + +#### Topics (/mq/topics) + +Manage message queue topics: + +**Topic Operations**: +- Create new topics with partition configuration +- View topic statistics and message counts +- Manage topic retention policies +- Monitor consumer lag and throughput + +#### Topic Details (/mq/topics/{namespace}/{topic}) + +Detailed topic information: + +**Statistics**: +- Message production and consumption rates +- Partition distribution and leadership +- Subscriber connections and lag +- Storage utilization per partition + + + +**Configuration Options**: +- Task scheduling parameters +- Worker connection settings +- Retry policies and timeouts +- Resource allocation limits + +## Navigation and UI Features + +### Responsive Design +- Mobile-friendly interface +- Collapsible sidebar navigation +- Responsive tables and charts +- Touch-friendly controls + +### Real-time Updates +- Live cluster status monitoring +- Automatic page refresh for dynamic content +- WebSocket connections for real-time data +- Progress indicators for long-running operations + +### Security Features +- Session-based authentication +- CSRF protection +- Secure cookie handling +- TLS/HTTPS support + +### Accessibility +- Keyboard navigation support +- Screen reader compatibility +- High contrast mode support +- Semantic HTML structure + ## See Also - [SeaweedFS Architecture](https://github.com/seaweedfs/seaweedfs/wiki/SeaweedFS-Architecture) - [Security Configuration](https://github.com/seaweedfs/seaweedfs/wiki/Security-Configuration) -- [Maintenance Operations](https://github.com/seaweedfs/seaweedfs/wiki/Maintenance) \ No newline at end of file + +- [S3 Object Lock and Retention](S3-Object-Lock-and-Retention.md) +- [S3 API FAQ](S3-API-FAQ.md) \ No newline at end of file From e4a7db192f499c1412818ec1f977fecf6885eca6 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Mon, 14 Jul 2025 23:02:01 -0700 Subject: [PATCH 113/149] update s3 api supports --- Amazon-S3-API.md | 136 +------------- S3-CORS.md | 393 ++++++++++++++++++++++++++++++++++++++++ S3-Object-Versioning.md | 137 ++++++++++++++ 3 files changed, 537 insertions(+), 129 deletions(-) create mode 100644 S3-CORS.md create mode 100644 S3-Object-Versioning.md diff --git a/Amazon-S3-API.md b/Amazon-S3-API.md index 7443ba7..3ab5c71 100644 --- a/Amazon-S3-API.md +++ b/Amazon-S3-API.md @@ -63,6 +63,9 @@ To be sure, you can look at the function defined in the files `weed/s3api/s3api_ * PutBucketLifecycleConfiguration (partially, only for TTL) * GetBucketLifecycleConfiguration (partially, only for TTL) * DeleteBucketLifecycleConfiguration (partially, only for TTL) +* GetBucketCors +* PutBucketCors +* DeleteBucketCors // Multipart upload operations * NewMultipartUpload @@ -270,139 +273,14 @@ Presigned URL is supported. See [[AWS-CLI-with-SeaweedFS#presigned-url]] for exa SeaweedFS supports S3 object versioning, which allows you to keep multiple variants of an object in the same bucket. This provides data protection against accidental deletion or modification. -## Enable Versioning +For detailed information about object versioning, see the dedicated [[S3-Object-Versioning]] page. -To enable versioning on a bucket, use the `PutBucketVersioning` API: -```bash -aws s3api put-bucket-versioning \ - --bucket my-bucket \ - --versioning-configuration Status=Enabled -``` +# S3 Cross-Origin Resource Sharing (CORS) -## Check Versioning Status +SeaweedFS supports S3-compatible Cross-Origin Resource Sharing (CORS) configuration, allowing web applications to make cross-origin requests to your S3 buckets. CORS is essential for web applications that need to access resources from different domains. -To check the versioning status of a bucket: - -```bash -aws s3api get-bucket-versioning --bucket my-bucket -``` - -Response: -```json -{ - "Status": "Enabled" -} -``` - -## Suspend Versioning - -To suspend versioning (not disable completely): - -```bash -aws s3api put-bucket-versioning \ - --bucket my-bucket \ - --versioning-configuration Status=Suspended -``` - -## List Object Versions - -To list all versions of objects in a bucket: - -```bash -aws s3api list-object-versions --bucket my-bucket -``` - -Response includes both object versions and delete markers: -```json -{ - "Versions": [ - { - "Key": "example.txt", - "VersionId": "v_1234567890abcdef", - "IsLatest": true, - "LastModified": "2023-12-01T10:00:00Z", - "ETag": "\"abcdef1234567890\"", - "Size": 1024 - } - ], - "DeleteMarkers": [ - { - "Key": "deleted-file.txt", - "VersionId": "v_fedcba0987654321", - "IsLatest": true, - "LastModified": "2023-12-01T11:00:00Z" - } - ] -} -``` - -## Access Specific Versions - -### Get a specific version of an object: -```bash -aws s3api get-object \ - --bucket my-bucket \ - --key example.txt \ - --version-id v_1234567890abcdef \ - output.txt -``` - -### Copy a specific version: -```bash -aws s3api copy-object \ - --copy-source my-bucket/example.txt?versionId=v_1234567890abcdef \ - --bucket my-bucket \ - --key example-copy.txt -``` - -### Delete a specific version: -```bash -aws s3api delete-object \ - --bucket my-bucket \ - --key example.txt \ - --version-id v_1234567890abcdef -``` - -## Versioning Behavior - -### When Versioning is Enabled: -- **PUT Object**: Creates a new version with a unique version ID -- **GET Object**: Returns the latest version (unless version ID is specified) -- **DELETE Object**: Creates a delete marker (soft delete) -- **DELETE Object with version ID**: Permanently deletes that specific version - -### When Versioning is Suspended: -- **PUT Object**: Overwrites the object with version ID "null" -- **GET Object**: Returns the current version -- **DELETE Object**: Permanently deletes the object - -## Storage Layout - -SeaweedFS stores versioned objects using the following structure: -``` -/buckets/my-bucket/ -├── example.txt # Current version (if versioning suspended) -└── example.txt.versions/ - ├── v_1234567890abcdef # Version 1 - ├── v_fedcba0987654321 # Version 2 - └── v_abcdef1234567890 # Version 3 (latest) -``` - -## Limitations - -- **Version ID Format**: SeaweedFS generates version IDs in the format `v_<32-char-hex>` -- **Restore Operations**: Only partial support for `RestoreObject` API -- **Lifecycle Policies**: Version-specific lifecycle rules are not fully implemented -- **MFA Delete**: Not currently supported - -## Best Practices - -1. **Enable versioning before storing important data** to ensure all versions are captured -2. **Monitor storage usage** as versioning can increase storage consumption -3. **Implement lifecycle policies** to automatically clean up old versions -4. **Use version-specific operations** when you need to access historical data -5. **Consider performance impact** when listing many versions of objects +For detailed information about CORS configuration, see the dedicated [[S3-CORS]] page. ## Multiple S3 Nodes diff --git a/S3-CORS.md b/S3-CORS.md new file mode 100644 index 0000000..f50789f --- /dev/null +++ b/S3-CORS.md @@ -0,0 +1,393 @@ +# S3 Cross-Origin Resource Sharing (CORS) + +SeaweedFS supports S3-compatible Cross-Origin Resource Sharing (CORS) configuration, allowing web applications to make cross-origin requests to your S3 buckets. CORS is essential for web applications that need to access resources from different domains. + +## Overview + +CORS defines a way for web applications running at one domain to access resources at another domain. When a web application tries to access your S3 bucket from a different domain, the browser will first send a preflight OPTIONS request to check if the cross-origin request is allowed. + +SeaweedFS handles CORS through: +- **Bucket-level CORS configuration**: Each bucket can have its own CORS rules +- **Persistent storage**: CORS configurations are stored in bucket metadata +- **Automatic header handling**: CORS middleware automatically applies appropriate headers +- **Preflight request support**: Proper handling of OPTIONS requests + +## CORS Configuration + +### Basic CORS Rule Structure + +A CORS configuration consists of one or more CORS rules. Each rule defines: + +```xml + + + rule-id + http://example.com + GET + POST + Content-Type + ETag + 3600 + + +``` + +### CORS Rule Elements + +- **ID** (optional): A unique identifier for the rule +- **AllowedOrigin** (required): Specifies which origins are allowed to access the bucket +- **AllowedMethod** (required): HTTP methods that are allowed (GET, PUT, POST, DELETE, HEAD) +- **AllowedHeader** (optional): Headers that are allowed in the actual request +- **ExposeHeader** (optional): Headers that browsers can access from the response +- **MaxAgeSeconds** (optional): How long browsers can cache the preflight response + +## Managing CORS Configuration + +### Set CORS Configuration + +Use the `PutBucketCors` API to set CORS configuration for a bucket: + +```bash +aws s3api put-bucket-cors \ + --bucket my-bucket \ + --cors-configuration file://cors-config.json +``` + +Example `cors-config.json`: +```json +{ + "CORSRules": [ + { + "ID": "allow-all-origins", + "AllowedOrigins": ["*"], + "AllowedMethods": ["GET", "POST", "PUT", "DELETE", "HEAD"], + "AllowedHeaders": ["*"], + "ExposeHeaders": ["ETag", "x-amz-version-id"], + "MaxAgeSeconds": 3600 + } + ] +} +``` + +### Get CORS Configuration + +Retrieve the current CORS configuration for a bucket: + +```bash +aws s3api get-bucket-cors --bucket my-bucket +``` + +Response: +```json +{ + "CORSRules": [ + { + "ID": "allow-all-origins", + "AllowedOrigins": ["*"], + "AllowedMethods": ["GET", "POST", "PUT", "DELETE", "HEAD"], + "AllowedHeaders": ["*"], + "ExposeHeaders": ["ETag", "x-amz-version-id"], + "MaxAgeSeconds": 3600 + } + ] +} +``` + +### Delete CORS Configuration + +Remove CORS configuration from a bucket: + +```bash +aws s3api delete-bucket-cors --bucket my-bucket +``` + +## CORS Rule Examples + +### Example 1: Allow Specific Domain + +```json +{ + "CORSRules": [ + { + "ID": "allow-example-domain", + "AllowedOrigins": ["https://example.com"], + "AllowedMethods": ["GET", "POST"], + "AllowedHeaders": ["Content-Type", "Authorization"], + "ExposeHeaders": ["ETag"], + "MaxAgeSeconds": 1800 + } + ] +} +``` + +### Example 2: Allow Multiple Domains + +```json +{ + "CORSRules": [ + { + "ID": "allow-multiple-domains", + "AllowedOrigins": [ + "https://app.example.com", + "https://staging.example.com", + "https://localhost:3000" + ], + "AllowedMethods": ["GET", "PUT", "POST", "DELETE"], + "AllowedHeaders": ["*"], + "ExposeHeaders": ["ETag", "x-amz-version-id"], + "MaxAgeSeconds": 3600 + } + ] +} +``` + +### Example 3: Wildcard Domain Support + +```json +{ + "CORSRules": [ + { + "ID": "allow-subdomain-wildcard", + "AllowedOrigins": ["https://*.example.com"], + "AllowedMethods": ["GET", "POST"], + "AllowedHeaders": ["Content-Type"], + "MaxAgeSeconds": 1800 + } + ] +} +``` + +### Example 4: Multiple Rules + +```json +{ + "CORSRules": [ + { + "ID": "read-only-rule", + "AllowedOrigins": ["*"], + "AllowedMethods": ["GET", "HEAD"], + "AllowedHeaders": ["*"], + "MaxAgeSeconds": 3600 + }, + { + "ID": "write-rule", + "AllowedOrigins": ["https://admin.example.com"], + "AllowedMethods": ["PUT", "POST", "DELETE"], + "AllowedHeaders": ["Content-Type", "Authorization"], + "ExposeHeaders": ["ETag"], + "MaxAgeSeconds": 1800 + } + ] +} +``` + +## Web Application Integration + +### JavaScript Example + +```javascript +// Configure your S3 client +const AWS = require('aws-sdk'); +const s3 = new AWS.S3({ + accessKeyId: 'your-access-key', + secretAccessKey: 'your-secret-key', + endpoint: 'http://localhost:8333', + s3ForcePathStyle: true, + region: 'us-east-1' +}); + +// Upload file from web application +const uploadFile = async (file, bucket, key) => { + const params = { + Bucket: bucket, + Key: key, + Body: file, + ContentType: file.type + }; + + try { + const result = await s3.upload(params).promise(); + console.log('Upload successful:', result); + return result; + } catch (error) { + console.error('Upload failed:', error); + throw error; + } +}; + +// Download file from web application +const downloadFile = async (bucket, key) => { + const params = { + Bucket: bucket, + Key: key + }; + + try { + const result = await s3.getObject(params).promise(); + return result.Body; + } catch (error) { + console.error('Download failed:', error); + throw error; + } +}; +``` + +### HTML Upload Form Example + +```html + + + + S3 Upload Example + + + + + + + + +``` + +## CORS Rule Evaluation + +SeaweedFS evaluates CORS rules in the following order: + +1. **Rule Matching**: The first rule that matches the request origin is used +2. **Origin Validation**: Check if the request origin matches any `AllowedOrigin` +3. **Method Validation**: For preflight requests, validate the requested method +4. **Header Validation**: For preflight requests, validate all requested headers +5. **Response Building**: Build appropriate CORS headers based on the matched rule + +### Preflight Request Handling + +For preflight requests (OPTIONS method), SeaweedFS: + +1. Checks if the origin matches an allowed origin +2. Validates the requested method against allowed methods +3. Validates all requested headers against allowed headers +4. Returns appropriate CORS headers if all validations pass +5. Returns 403 Forbidden if any validation fails + +### Actual Request Handling + +For actual requests (GET, POST, PUT, DELETE, etc.), SeaweedFS: + +1. Checks if the origin matches an allowed origin +2. Validates the request method against allowed methods +3. Applies appropriate CORS headers to the response +4. Continues with normal request processing + +## Performance and Caching + +### CORS Configuration Caching + +- CORS configurations are cached in memory for 5 minutes +- Cache is automatically invalidated when configuration changes +- Multiple S3 nodes share the same cached configuration + +### Browser Caching + +- Use `MaxAgeSeconds` to control how long browsers cache preflight responses +- Longer cache times reduce preflight requests but delay configuration changes +- Recommended values: 1800-3600 seconds (30-60 minutes) + +## Security Considerations + +### Origin Validation + +- Never use `*` for `AllowedOrigin` in production unless absolutely necessary +- Specify exact domains or use specific wildcard patterns +- Validate all origins against your application's requirements + +### Method Restrictions + +- Only allow necessary HTTP methods +- Restrict write operations (PUT, POST, DELETE) to trusted origins +- Consider separate rules for read-only vs. write operations + +### Header Security + +- Avoid using `*` for `AllowedHeaders` in production +- Only allow headers that your application actually needs +- Be cautious with authorization headers + +## Troubleshooting + +### Common Issues + +1. **CORS policy error**: Check that your origin is listed in `AllowedOrigins` +2. **Method not allowed**: Ensure the HTTP method is in `AllowedMethods` +3. **Header blocked**: Add required headers to `AllowedHeaders` +4. **Preflight failure**: Verify all preflight requirements are met + +### Debugging Tips + +- Use browser developer tools to inspect CORS headers +- Check server logs for CORS-related errors +- Test with simple requests first, then add complexity +- Verify bucket-level CORS configuration is correct + +### Testing CORS Configuration + +```bash +# Test preflight request +curl -X OPTIONS \ + -H "Origin: https://example.com" \ + -H "Access-Control-Request-Method: GET" \ + -H "Access-Control-Request-Headers: Content-Type" \ + http://localhost:8333/my-bucket/test-object + +# Test actual request +curl -X GET \ + -H "Origin: https://example.com" \ + http://localhost:8333/my-bucket/test-object +``` + +## Limitations + +- Maximum 100 CORS rules per bucket +- Wildcard support is limited to `*` character +- Complex regex patterns in origins are not supported +- CORS configuration is per-bucket, not per-object + +## Best Practices + +1. **Specify exact origins** instead of using wildcards when possible +2. **Use appropriate MaxAgeSeconds** to balance performance and flexibility +3. **Implement proper error handling** in your web applications +4. **Test CORS configuration thoroughly** before deploying to production +5. **Monitor CORS usage** and adjust rules as needed +6. **Keep CORS rules simple** and well-documented +7. **Use separate rules** for different access patterns (read vs. write) \ No newline at end of file diff --git a/S3-Object-Versioning.md b/S3-Object-Versioning.md new file mode 100644 index 0000000..24780b4 --- /dev/null +++ b/S3-Object-Versioning.md @@ -0,0 +1,137 @@ +# S3 Object Versioning + +SeaweedFS supports S3 object versioning, which allows you to keep multiple variants of an object in the same bucket. This provides data protection against accidental deletion or modification. + +## Enable Versioning + +To enable versioning on a bucket, use the `PutBucketVersioning` API: + +```bash +aws s3api put-bucket-versioning \ + --bucket my-bucket \ + --versioning-configuration Status=Enabled +``` + +## Check Versioning Status + +To check the versioning status of a bucket: + +```bash +aws s3api get-bucket-versioning --bucket my-bucket +``` + +Response: +```json +{ + "Status": "Enabled" +} +``` + +## Suspend Versioning + +To suspend versioning (not disable completely): + +```bash +aws s3api put-bucket-versioning \ + --bucket my-bucket \ + --versioning-configuration Status=Suspended +``` + +## List Object Versions + +To list all versions of objects in a bucket: + +```bash +aws s3api list-object-versions --bucket my-bucket +``` + +Response includes both object versions and delete markers: +```json +{ + "Versions": [ + { + "Key": "example.txt", + "VersionId": "v_1234567890abcdef", + "IsLatest": true, + "LastModified": "2023-12-01T10:00:00Z", + "ETag": "\"abcdef1234567890\"", + "Size": 1024 + } + ], + "DeleteMarkers": [ + { + "Key": "deleted-file.txt", + "VersionId": "v_fedcba0987654321", + "IsLatest": true, + "LastModified": "2023-12-01T11:00:00Z" + } + ] +} +``` + +## Access Specific Versions + +### Get a specific version of an object: +```bash +aws s3api get-object \ + --bucket my-bucket \ + --key example.txt \ + --version-id v_1234567890abcdef \ + output.txt +``` + +### Copy a specific version: +```bash +aws s3api copy-object \ + --copy-source my-bucket/example.txt?versionId=v_1234567890abcdef \ + --bucket my-bucket \ + --key example-copy.txt +``` + +### Delete a specific version: +```bash +aws s3api delete-object \ + --bucket my-bucket \ + --key example.txt \ + --version-id v_1234567890abcdef +``` + +## Versioning Behavior + +### When Versioning is Enabled: +- **PUT Object**: Creates a new version with a unique version ID +- **GET Object**: Returns the latest version (unless version ID is specified) +- **DELETE Object**: Creates a delete marker (soft delete) +- **DELETE Object with version ID**: Permanently deletes that specific version + +### When Versioning is Suspended: +- **PUT Object**: Overwrites the object with version ID "null" +- **GET Object**: Returns the current version +- **DELETE Object**: Permanently deletes the object + +## Storage Layout + +SeaweedFS stores versioned objects using the following structure: +``` +/buckets/my-bucket/ +├── example.txt # Current version (if versioning suspended) +└── example.txt.versions/ + ├── v_1234567890abcdef # Version 1 + ├── v_fedcba0987654321 # Version 2 + └── v_abcdef1234567890 # Version 3 (latest) +``` + +## Limitations + +- **Version ID Format**: SeaweedFS generates version IDs in the format `v_<32-char-hex>` +- **Restore Operations**: Only partial support for `RestoreObject` API +- **Lifecycle Policies**: Version-specific lifecycle rules are not fully implemented +- **MFA Delete**: Not currently supported + +## Best Practices + +1. **Enable versioning before storing important data** to ensure all versions are captured +2. **Monitor storage usage** as versioning can increase storage consumption +3. **Implement lifecycle policies** to automatically clean up old versions +4. **Use version-specific operations** when you need to access historical data +5. **Consider performance impact** when listing many versions of objects \ No newline at end of file From 36ddae236fb4371093fb3a74a880613645e8af62 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Wed, 23 Jul 2025 02:05:01 -0700 Subject: [PATCH 114/149] Update Environment-Variables.md --- Environment-Variables.md | 57 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/Environment-Variables.md b/Environment-Variables.md index 09c1705..45b1518 100644 --- a/Environment-Variables.md +++ b/Environment-Variables.md @@ -13,6 +13,37 @@ weed master # Weed prefix For `v`, `logtostderr`, `stderrthreshold`, `vmoudle`, `options`, `logdir`, `alsologtostderr`, `log_backtrace_at` , and `config_dir` you have to use `WEED_` as prefix for environment variable like this `WEED_CONFIG_DIR=/tmp` +# S3 Admin Credentials +For S3 API server, you can use standard AWS environment variables to set default admin credentials: + +```shell +export AWS_ACCESS_KEY_ID=your_access_key +export AWS_SECRET_ACCESS_KEY=your_secret_key +weed s3 -filer=localhost:8888 +``` + +These environment variables will supplement existing configuration by adding admin credentials when: +- Both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are set +- No identity with the same access key already exists + +**Configuration Priority:** +1. **File or Filer configuration** is loaded first (if provided) +2. **Environment variables** are added second, supplementing the existing configuration +3. Environment variables are **skipped** if an access key conflict exists + +This approach ensures that: +- Existing S3 configuration files (with `admin_access` etc.) are preserved +- Environment variables provide additional admin access without conflicts +- AWS standard environment variables work seamlessly alongside existing setups + +The admin identity from environment variables will have: +- Name: `admin-` followed by the access key (or first 8 characters if longer than 8) +- Access Key: value of `AWS_ACCESS_KEY_ID` +- Secret Key: value of `AWS_SECRET_ACCESS_KEY` +- Permissions: Full admin access to all S3 operations + +This follows the standard AWS credential convention, making it compatible with existing AWS tooling and workflows. + # Docker This is useful for using docker and docker compose You have to override entrypoint to `weed` because defautl [entrypoint](https://github.com/seaweedfs/seaweedfs/blob/master/docker/entrypoint.sh) use default values for `volumeSizeLimitMB`, `volumePreallocate`, `mdir`, `dir`, and `max` and setting environment variables won't change these values. @@ -39,3 +70,29 @@ services: entrypoint: weed command: master ``` + +## Docker Compose with S3 Admin Credentials +```yaml +version: '3.9' +services: + filer: + image: chrislusf/seaweedfs:latest + ports: + - 8888:8888 + environment: + # ... other filer environment variables + entrypoint: weed + command: filer -master=master:9333 + + s3: + image: chrislusf/seaweedfs:latest + ports: + - 8333:8333 + environment: + AWS_ACCESS_KEY_ID: s3admin + AWS_SECRET_ACCESS_KEY: s3secret + entrypoint: weed + command: s3 -filer=filer:8888 + depends_on: + - filer +``` From 6503963c27c510a69fbf4a31bdb11b35894485bd Mon Sep 17 00:00:00 2001 From: chrislusf Date: Wed, 23 Jul 2025 11:45:04 -0700 Subject: [PATCH 115/149] add s3 credentials --- Environment-Variables.md | 53 ++------ S3-Credentials.md | 257 +++++++++++++++++++++++++++++++++++++++ _Sidebar.md | 1 + 3 files changed, 271 insertions(+), 40 deletions(-) create mode 100644 S3-Credentials.md diff --git a/Environment-Variables.md b/Environment-Variables.md index 45b1518..49e4672 100644 --- a/Environment-Variables.md +++ b/Environment-Variables.md @@ -14,50 +14,28 @@ weed master For `v`, `logtostderr`, `stderrthreshold`, `vmoudle`, `options`, `logdir`, `alsologtostderr`, `log_backtrace_at` , and `config_dir` you have to use `WEED_` as prefix for environment variable like this `WEED_CONFIG_DIR=/tmp` # S3 Admin Credentials -For S3 API server, you can use standard AWS environment variables to set default admin credentials: - -```shell -export AWS_ACCESS_KEY_ID=your_access_key -export AWS_SECRET_ACCESS_KEY=your_secret_key -weed s3 -filer=localhost:8888 -``` - -These environment variables will supplement existing configuration by adding admin credentials when: -- Both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are set -- No identity with the same access key already exists - -**Configuration Priority:** -1. **File or Filer configuration** is loaded first (if provided) -2. **Environment variables** are added second, supplementing the existing configuration -3. Environment variables are **skipped** if an access key conflict exists - -This approach ensures that: -- Existing S3 configuration files (with `admin_access` etc.) are preserved -- Environment variables provide additional admin access without conflicts -- AWS standard environment variables work seamlessly alongside existing setups - -The admin identity from environment variables will have: -- Name: `admin-` followed by the access key (or first 8 characters if longer than 8) -- Access Key: value of `AWS_ACCESS_KEY_ID` -- Secret Key: value of `AWS_SECRET_ACCESS_KEY` -- Permissions: Full admin access to all S3 operations - -This follows the standard AWS credential convention, making it compatible with existing AWS tooling and workflows. +For S3 API server authentication, see the dedicated **[S3 Credentials](S3-Credentials)** page which covers: +- Configuration file setup (highest priority) +- Filer configuration (medium priority) +- Environment variables as fallback (lowest priority) +- AWS standard environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) +- Complete authentication examples and troubleshooting # Docker -This is useful for using docker and docker compose -You have to override entrypoint to `weed` because defautl [entrypoint](https://github.com/seaweedfs/seaweedfs/blob/master/docker/entrypoint.sh) use default values for `volumeSizeLimitMB`, `volumePreallocate`, `mdir`, `dir`, and `max` and setting environment variables won't change these values. -## Docker +You can set environment variables easily in Docker: ```shell -docker run --entrypoint weed -it -e IP_BIND=0.0.0.0 -e MDIR=/tmp -e PORT=5000 -e VOLUMEPREALLOCATE=true chrislusf/seaweedfs:3.45 master +docker run --name master -d -p 9333:9333 -p 19333:19333 \ + -e MDIR="/data" -e PORT="9333" \ + chrislusf/seaweedfs:latest \ + master ``` -## Docker Compose +## Docker Compose with Environment Variables ```yaml version: '3.9' services: master: - image: chrislusf/seaweedfs:3.45 + image: chrislusf/seaweedfs:latest ports: - 9333:9333 - 19333:19333 @@ -69,12 +47,7 @@ services: # or `VOLUMEPREALLOCATE:` entrypoint: weed command: master -``` -## Docker Compose with S3 Admin Credentials -```yaml -version: '3.9' -services: filer: image: chrislusf/seaweedfs:latest ports: diff --git a/S3-Credentials.md b/S3-Credentials.md new file mode 100644 index 0000000..30c175e --- /dev/null +++ b/S3-Credentials.md @@ -0,0 +1,257 @@ +# S3 Credentials + +SeaweedFS S3 API supports multiple authentication methods with a clear priority system. This page explains how to configure S3 credentials for your SeaweedFS setup. + +## Authentication Methods + +### 1. Configuration File (Highest Priority) + +Create a JSON configuration file and use the `-config` option: + +```json +{ + "identities": [ + { + "name": "admin_user", + "credentials": [ + { + "accessKey": "admin_access_key", + "secretKey": "admin_secret_key" + } + ], + "actions": ["Admin", "Read", "Write"] + }, + { + "name": "read_only_user", + "credentials": [ + { + "accessKey": "readonly_access_key", + "secretKey": "readonly_secret_key" + } + ], + "actions": ["Read"] + } + ] +} +``` + +Start S3 server with config file: +```bash +weed s3 -config=/path/to/s3.json -filer=localhost:8888 +``` + +### 2. Filer Configuration (Medium Priority) + +Store configuration in the filer using the credential manager. This allows dynamic configuration updates without restarting the S3 server. + +### 3. Environment Variables (Fallback) + +Use AWS standard environment variables as a fallback when no other configuration is available: + +```bash +export AWS_ACCESS_KEY_ID=your_access_key +export AWS_SECRET_ACCESS_KEY=your_secret_key +weed s3 -filer=localhost:8888 +``` + +**Important**: Environment variables are only used when: +- No `-config` option is provided +- No configuration is available from the filer +- Both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are set + +## Priority System + +SeaweedFS uses the following priority order for S3 credentials: + +1. **Configuration File** (if `-config` option is provided) +2. **Filer Configuration** (if available and no config file) +3. **Environment Variables** (fallback only) + +Higher priority methods completely override lower priority methods - there is no merging or supplementing. + +## Configuration Examples + +### Production Setup +```bash +# Use dedicated configuration file +weed s3 -config=/etc/seaweedfs/s3.json -filer=filer1:8888,filer2:8888 +``` + +### Development Setup +```bash +# Use environment variables for quick setup +export AWS_ACCESS_KEY_ID=dev_access_key +export AWS_SECRET_ACCESS_KEY=dev_secret_key +weed s3 -filer=localhost:8888 +``` + +### Docker Compose +```yaml +version: '3.9' +services: + s3: + image: chrislusf/seaweedfs:latest + ports: + - 8333:8333 + environment: + AWS_ACCESS_KEY_ID: s3admin + AWS_SECRET_ACCESS_KEY: s3secret + entrypoint: weed + command: s3 -filer=filer:8888 + depends_on: + - filer +``` + +## Credential Features + +### Actions +Identities can have different permission levels: +- `Admin`: Full access to all S3 operations +- `Read`: Read-only access +- `Write`: Read and write access +- `Read_ACP`: Read access control permissions +- `Write_ACP`: Write access control permissions + +### Multiple Credentials +Each identity can have multiple access key/secret key pairs: + +```json +{ + "name": "multi_key_user", + "credentials": [ + { + "accessKey": "key1", + "secretKey": "secret1" + }, + { + "accessKey": "key2", + "secretKey": "secret2" + } + ], + "actions": ["Read", "Write"] +} +``` + +### Account Management +Identities can be associated with accounts for better organization and cross-account access control. + +## Anonymous Access + +By default, if no credentials are configured, SeaweedFS allows anonymous access to all S3 operations. To enable authentication: + +1. Configure at least one identity using any of the methods above +2. Authentication will be automatically enabled +3. All requests will require valid credentials + +## Configuration Reloading + +SeaweedFS supports different reloading mechanisms depending on which authentication method you use: + +| Configuration Method | Auto Reload | Manual Reload | Live Reload | +|---------------------|-------------|---------------|-------------| +| **Configuration File** (`-config` option) | ❌ No | ✅ SIGHUP | ❌ No | +| **Filer Configuration** (credential manager) | ✅ Yes | ✅ Yes | ✅ Yes | +| **Environment Variables** | ❌ No | ❌ No | ❌ No | + +### Static Configuration Files + +When using the `-config` option, you can reload the configuration by sending a SIGHUP signal: + +```bash +# Find the S3 server process ID +ps aux | grep "weed s3" + +# Send SIGHUP signal to reload configuration +kill -HUP + +# Or if using systemd +systemctl reload seaweedfs-s3 +``` + +The server will log the reload: +``` +I0723 12:34:56.789 s3api_server.go:98] Loaded 3 identities from config file /etc/seaweedfs/s3.json +``` + +### Filer-based Configuration + +Filer-based configurations automatically reload when changes are detected: + +```bash +# Changes are automatically applied +weed shell +> s3.configure -user=newuser -access_key=key123 -secret_key=secret123 -actions=Admin -apply +``` + +The server will automatically detect and apply changes: +``` +I0723 12:35:12.456 auth_credentials_subscribe.go:55] updated /etc/seaweedfs/iam/identity.json +``` + +### Environment Variables + +Environment variable changes require a complete restart of the S3 server: + +```bash +# Update environment variables +export AWS_ACCESS_KEY_ID=new_access_key +export AWS_SECRET_ACCESS_KEY=new_secret_key + +# Restart the S3 server +systemctl restart seaweedfs-s3 +``` + +### Verifying Configuration Reloads + +Monitor the logs to verify configuration updates: + +```bash +# Watch for reload messages +tail -f /var/log/seaweedfs/s3.log | grep -E "updated|Loaded.*identities" + +# Check current configuration via shell +weed shell +> s3.configure +``` + +## Troubleshooting + +### Common Issues + +**Environment variables not working:** +- Check that no `-config` option is provided +- Verify no configuration exists in the filer +- Ensure both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are set + +**Configuration file not loading:** +- Verify the file path is correct +- Check JSON syntax is valid +- Ensure the file is readable by the SeaweedFS process + +**Invalid credentials error:** +- Verify access key and secret key are correct +- Check that the identity has the required actions/permissions +- Ensure the credential store is properly configured + +### Debug Commands + +Check current configuration: +```bash +# View current identities (if using filer store) +weed shell +> s3.configure -list +``` + +Test credentials: +```bash +# Test with AWS CLI +aws --endpoint-url=http://localhost:8333 s3 ls +``` + +## Security Best Practices + +1. **Use Configuration Files in Production**: Environment variables are visible in process lists +2. **Rotate Credentials Regularly**: Update access keys and secret keys periodically +3. **Principle of Least Privilege**: Grant only the minimum required permissions +4. **Secure Storage**: Store configuration files with appropriate file permissions +5. **Monitor Access**: Enable audit logging to track S3 API usage \ No newline at end of file diff --git a/_Sidebar.md b/_Sidebar.md index f5382b8..e3f307d 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -66,6 +66,7 @@ * [[Gateway to Remote Object Storage]] ### AWS S3 API +* [[S3 Credentials]] * [[Amazon S3 API]] * [[S3 Object Lock and Retention]] * [[AWS CLI with SeaweedFS]] From 04d496dc3a3d07fe0ad9a76e2ffedc215cb6ac27 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Wed, 23 Jul 2025 11:52:24 -0700 Subject: [PATCH 116/149] correction about admin ui --- Kubernetes-Backups-and-Recovery-with-K8up.md | 2 +- S3-Credentials.md | 39 +++++++++++++++++++- 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/Kubernetes-Backups-and-Recovery-with-K8up.md b/Kubernetes-Backups-and-Recovery-with-K8up.md index 332ee29..01f780f 100644 --- a/Kubernetes-Backups-and-Recovery-with-K8up.md +++ b/Kubernetes-Backups-and-Recovery-with-K8up.md @@ -165,7 +165,7 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034 helm install seaweedfs . -f test-values.yaml --wait ``` -5. Expose the filer service via a LoadBalancer (servicelb in k3s). This will let us view the admin UI as well as reach the S3 endpoint during the demo. +5. Expose the filer service via a LoadBalancer (servicelb in k3s). This will let us view the filer's file browser UI as well as reach the S3 endpoint during the demo. ```bash /bin/cat << EOF > service.yaml diff --git a/S3-Credentials.md b/S3-Credentials.md index 30c175e..e07be8c 100644 --- a/S3-Credentials.md +++ b/S3-Credentials.md @@ -44,7 +44,27 @@ weed s3 -config=/path/to/s3.json -filer=localhost:8888 Store configuration in the filer using the credential manager. This allows dynamic configuration updates without restarting the S3 server. -### 3. Environment Variables (Fallback) +### 3. Admin UI (Web Interface) + +Use the SeaweedFS Admin UI to manage S3 credentials through a web interface: + +```bash +# Start the admin interface (separate from filer) +weed admin -masters=localhost:9333 + +# Access the admin UI (default port 23646) +http://localhost:23646 +``` + +Navigate to **Object Store → Users** (`/object-store/users`) to: +- **Create Users**: Add new S3 users with email and permissions +- **Edit Permissions**: Modify existing user access levels +- **Manage Access Keys**: Generate and delete access key pairs +- **View User Details**: Check user activity and current permissions + +The Admin UI stores credentials in the filer using the same filer configuration method, so changes are automatically synchronized across all S3 servers connected to the same filer. + +### 4. Environment Variables (Fallback) Use AWS standard environment variables as a fallback when no other configuration is available: @@ -65,10 +85,13 @@ SeaweedFS uses the following priority order for S3 credentials: 1. **Configuration File** (if `-config` option is provided) 2. **Filer Configuration** (if available and no config file) -3. **Environment Variables** (fallback only) +3. **Admin UI** (web interface that stores in filer configuration) +4. **Environment Variables** (fallback only) Higher priority methods completely override lower priority methods - there is no merging or supplementing. +**Important**: Admin UI and Filer Configuration both use the same underlying storage (filer), so they have the same effective priority. The Admin UI provides a user-friendly interface for managing what is stored as filer configuration. + ## Configuration Examples ### Production Setup @@ -151,6 +174,7 @@ SeaweedFS supports different reloading mechanisms depending on which authenticat |---------------------|-------------|---------------|-------------| | **Configuration File** (`-config` option) | ❌ No | ✅ SIGHUP | ❌ No | | **Filer Configuration** (credential manager) | ✅ Yes | ✅ Yes | ✅ Yes | +| **Admin UI** (web interface) | ✅ Yes | ✅ Yes | ✅ Yes | | **Environment Variables** | ❌ No | ❌ No | ❌ No | ### Static Configuration Files @@ -188,6 +212,17 @@ The server will automatically detect and apply changes: I0723 12:35:12.456 auth_credentials_subscribe.go:55] updated /etc/seaweedfs/iam/identity.json ``` +### Admin UI Configuration + +Admin UI changes are automatically applied in real-time since they use the same filer-based storage: + +1. **Access Admin UI**: Navigate to `http://localhost:23646` +2. **Go to Users**: Click **Object Store → Users** +3. **Make Changes**: Create, edit, or delete users through the web interface +4. **Automatic Sync**: Changes are immediately applied to all connected S3 servers + +The server will show the same automatic detection messages as filer-based configuration since they share the same underlying storage mechanism. + ### Environment Variables Environment variable changes require a complete restart of the S3 server: From 40aac1ab4693b6a5a57f85026eb0e08892e0c78f Mon Sep 17 00:00:00 2001 From: chrislusf Date: Wed, 23 Jul 2025 16:46:30 -0700 Subject: [PATCH 117/149] how to configure --- S3-Credentials.md | 153 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) diff --git a/S3-Credentials.md b/S3-Credentials.md index e07be8c..007b34c 100644 --- a/S3-Credentials.md +++ b/S3-Credentials.md @@ -158,6 +158,159 @@ Each identity can have multiple access key/secret key pairs: ### Account Management Identities can be associated with accounts for better organization and cross-account access control. +## Bucket-Specific Permissions + +SeaweedFS supports restricting user access to specific buckets using bucket-scoped actions. This allows you to create users who have full access to one bucket but no access to other buckets. + +### Single Bucket Full Access + +To create a user with full access to only one specific bucket, use bucket-scoped actions: + +```json +{ + "identities": [ + { + "name": "bucket1_user", + "credentials": [ + { + "accessKey": "bucket1_access_key", + "secretKey": "bucket1_secret_key" + } + ], + "actions": [ + "Read:mybucket", + "Write:mybucket", + "List:mybucket", + "Tagging:mybucket", + "Admin:mybucket" + ] + } + ] +} +``` + +This user can: +- ✅ Read, write, list, and tag objects in `mybucket` +- ✅ Create and delete objects in `mybucket` +- ✅ Manage bucket settings for `mybucket` +- ❌ Access any other buckets +- ❌ Create new buckets (requires global Admin action) + +### Bucket-Specific Actions + +Actions can be scoped to specific buckets using the format `Action:BucketName`: + +| Action Format | Description | Example | +|--------------|-------------|---------| +| `Read:bucket1` | Read access to bucket1 only | Get objects from bucket1 | +| `Write:bucket1` | Write access to bucket1 only | Put/delete objects in bucket1 | +| `List:bucket1` | List access to bucket1 only | List objects in bucket1 | +| `Admin:bucket1` | Admin access to bucket1 only | Bucket management for bucket1 | +| `Tagging:bucket1` | Tagging access to bucket1 only | Manage object tags in bucket1 | + +### Multiple Bucket Access + +Users can have access to multiple specific buckets: + +```json +{ + "name": "multi_bucket_user", + "credentials": [{"accessKey": "key", "secretKey": "secret"}], + "actions": [ + "Read:bucket1", + "Write:bucket1", + "List:bucket1", + "Read:bucket2", + "List:bucket2" + ] +} +``` + +This user has: +- Full read/write access to `bucket1` +- Read-only access to `bucket2` +- No access to any other buckets + +### Wildcard Support + +SeaweedFS supports wildcard patterns for flexible bucket access: + +```json +{ + "name": "prefix_user", + "credentials": [{"accessKey": "key", "secretKey": "secret"}], + "actions": [ + "Read:user-*", + "Write:user-*", + "List:user-*" + ] +} +``` + +This user can access all buckets starting with `user-` (like `user-data`, `user-logs`, etc.). + +### Object-Level Permissions + +You can restrict access to specific paths within a bucket: + +```json +{ + "name": "path_limited_user", + "credentials": [{"accessKey": "key", "secretKey": "secret"}], + "actions": [ + "Read:mybucket/uploads/*", + "Write:mybucket/uploads/*", + "List:mybucket" + ] +} +``` + +This user can: +- Only read/write objects under `mybucket/uploads/` path +- List the bucket to see the directory structure +- Cannot access objects in other paths within the bucket + +### Configuration Methods + +Bucket-specific permissions work with all configuration methods: + +#### Dynamic Configuration (weed shell) +```bash +# Create user with access to specific bucket +s3.configure -access_key=bucket1user -secret_key=bucket1pass -buckets=mybucket -user=bucket1_user -actions=Read,Write,List,Tagging,Admin -apply +``` + +#### Static Configuration File +Use the JSON examples shown above in your configuration file. + +#### Admin UI +1. Navigate to **Object Store → Users** +2. Create a new user +3. In the permissions section, specify bucket-scoped actions like `Read:mybucket` + +#### Environment Variables +Environment variables create global admin access and cannot be scoped to specific buckets. + +### Best Practices + +1. **Principle of Least Privilege**: Grant only the minimum permissions needed +2. **Use Specific Bucket Names**: Avoid wildcards unless necessary for flexibility +3. **Separate Users for Different Buckets**: Create dedicated users for each bucket or application +4. **Test Permissions**: Verify users can only access intended buckets +5. **Monitor Access**: Use audit logs to track bucket access patterns + +### Troubleshooting + +**User can access other buckets:** +- Verify no global actions (`Read`, `Write`, `Admin`) are granted +- Check for wildcard patterns that might be too broad +- Ensure bucket names in actions match exactly + +**User cannot access intended bucket:** +- Verify bucket name spelling in actions +- Check that all required actions are granted (e.g., `List` for listing objects) +- Test with AWS CLI: `aws --endpoint-url=http://localhost:8333 s3 ls s3://mybucket` + ## Anonymous Access By default, if no credentials are configured, SeaweedFS allows anonymous access to all S3 operations. To enable authentication: From d7af82f4cb1810e65340537167cee91ec41feabe Mon Sep 17 00:00:00 2001 From: chrislusf Date: Fri, 1 Aug 2025 13:08:56 -0700 Subject: [PATCH 118/149] support X-Forwarded-Prefix https://github.com/seaweedfs/seaweedfs/issues/7064 --- Amazon-S3-API.md | 19 +++++++++++ S3-API-FAQ.md | 66 +++++++++++++++++++++++++++++++++++ S3-Nginx-Proxy.md | 87 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 172 insertions(+) diff --git a/Amazon-S3-API.md b/Amazon-S3-API.md index 3ab5c71..9ab64bd 100644 --- a/Amazon-S3-API.md +++ b/Amazon-S3-API.md @@ -283,6 +283,25 @@ SeaweedFS supports S3-compatible Cross-Origin Resource Sharing (CORS) configurat For detailed information about CORS configuration, see the dedicated [[S3-CORS]] page. +# Reverse Proxy Support + +SeaweedFS S3 API supports deployment behind reverse proxies with full AWS Signature v4 authentication compatibility. This includes support for: + +- **X-Forwarded-Host**: Preserves original host header for signature verification +- **X-Forwarded-Prefix**: Handles URL path prefix stripping by reverse proxies +- **Standard forwarded headers**: X-Forwarded-For, X-Forwarded-Proto, etc. + +## Path Prefix Handling + +When using reverse proxies that strip URL prefixes (e.g., `/s3/`, `/api/s3/`), SeaweedFS automatically handles signature verification for both the original prefixed path and the stripped path. This ensures seamless operation with: + +- API gateways +- Multi-tenant deployments +- Subpath hosting scenarios + +For detailed configuration examples and setup instructions, see the dedicated [[S3-Nginx-Proxy]] page. + + ## Multiple S3 Nodes If you need to setup multiple S3 nodes, you can just start multiple s3 instances pointing to a filer. diff --git a/S3-API-FAQ.md b/S3-API-FAQ.md index 728c6d6..84e8098 100644 --- a/S3-API-FAQ.md +++ b/S3-API-FAQ.md @@ -150,4 +150,70 @@ For complete documentation, examples, and best practices, see [[S3 Object Lock a Both modes prevent accidental deletion and provide audit trails for compliance purposes. +## S3 authentication fails when using reverse proxy + +### Symptom + +When accessing SeaweedFS S3 API through a reverse proxy, you might encounter signature verification errors such as: +- `SignatureDoesNotMatch` errors +- Authentication failures for presigned URLs +- Inconsistent behavior between direct access and proxied access + +### Common Causes and Solutions + +**1. Missing X-Forwarded-Host header** + +The reverse proxy must set the `X-Forwarded-Host` header to preserve the original host information for signature calculation. + +```nginx +proxy_set_header X-Forwarded-Host $host; +``` + +**2. URL path prefix stripping without X-Forwarded-Prefix** + +If your reverse proxy strips URL prefixes (e.g., `/s3/bucket/object` → `/bucket/object`), you must set the `X-Forwarded-Prefix` header: + +```nginx +# For /s3/ subpath +location /s3/ { + proxy_set_header X-Forwarded-Prefix /s3; + rewrite ^/s3/(.*) /$1 break; + proxy_pass http://seaweedfs; +} +``` + +**3. Request buffering enabled** + +Nginx request buffering can interfere with chunked transfer encoding and signature verification: + +```nginx +proxy_request_buffering off; +proxy_buffering off; +``` + +**4. Missing or incorrect forwarded headers** + +Ensure all necessary headers are forwarded: + +```nginx +proxy_set_header Host $host; +proxy_set_header X-Forwarded-Host $host; +proxy_set_header X-Forwarded-Proto $scheme; +proxy_set_header X-Forwarded-For $remote_addr; +``` + +### Testing Your Configuration + +You can test your reverse proxy configuration using AWS CLI: + +```bash +# Test basic bucket listing +aws s3 ls --endpoint-url https://yourdomain.com/s3 + +# Test presigned URL generation and access +aws s3 presign s3://test-bucket/test-object --endpoint-url https://yourdomain.com/s3 +``` + +For detailed configuration examples, see the [[S3-Nginx-Proxy]] documentation. + diff --git a/S3-Nginx-Proxy.md b/S3-Nginx-Proxy.md index e63fc45..0499cd7 100644 --- a/S3-Nginx-Proxy.md +++ b/S3-Nginx-Proxy.md @@ -5,10 +5,41 @@ For virtual-hosted style URL buckets, you'll need to add a [wildcard DNS record] Make sure the config sets the `X-Forwarded-Host` and optionally the `X-Forwarded-Port` if you are using a non-standard port. Otherwise, the AWS Signature v4 will not be correctly computed by the server, as it uses the Host header to compute the signature on the client side. +## Reverse Proxy with URL Path Prefixes + +SeaweedFS S3 API supports the `X-Forwarded-Prefix` header for scenarios where a reverse proxy strips URL path prefixes before forwarding requests. This is common when hosting the S3 API under a subpath like `/s3/` or `/api/s3/`. + +### How X-Forwarded-Prefix Works + +When a reverse proxy strips a URL prefix: +1. **Client request**: `https://example.com/s3/my-bucket/my-object` +2. **Proxy strips prefix** and forwards: `https://backend:8333/my-bucket/my-object` +3. **Proxy adds header**: `X-Forwarded-Prefix: /s3` + +SeaweedFS will: +1. First attempt signature verification using the **original path** (`/s3/my-bucket/my-object`) +2. Fall back to verification using the **stripped path** (`/my-bucket/my-object`) if the first attempt fails + +This ensures both regular S3 requests and presigned URLs work correctly with reverse proxies that strip prefixes. + +### Example Use Cases + +- **API Gateway**: `/api/s3/bucket/object` → `/bucket/object` +- **Multi-tenant setup**: `/tenant1/s3/bucket/object` → `/bucket/object` +- **Subpath hosting**: `/storage/s3/bucket/object` → `/bucket/object` + +### Important Notes + +- The `X-Forwarded-Prefix` header should contain the stripped prefix (e.g., `/s3`) +- Both regular S3 authentication and presigned URLs are supported +- This feature works with all S3 operations that require signature verification + Additionally, make sure that `proxy_request_buffering` is `off` (default is `on`), as the proxy will buffer the request, and send the request to the backend as a whole instead of chunked, and again the signature computed by the client side will be different as it would have taken into account the `Transfer-Encoding: chunked` header that is dropped by the proxy when it buffers. ### Example Nginx config +#### Standard Configuration (without URL prefix stripping) + ``` upstream seaweedfs { # Hash on uploadId query string in the GET request create consistency for multipart uploads, @@ -58,6 +89,62 @@ server { proxy_pass http://seaweedfs; } + ssl on; + ssl_certificate /{path_to_ssl_cert}/cert.pem; + ssl_certificate_key /{path_to_ssl_cert}/key.pem; +} +``` + +#### Configuration with URL Prefix Stripping (X-Forwarded-Prefix) + +For scenarios where you need to host SeaweedFS S3 API under a subpath: + +``` +upstream seaweedfs { + hash $arg_uploadId consistent; + server localhost:8333 fail_timeout=0; + keepalive 20; +} + +server { + listen 443 ssl; + server_name yourdomain.com; + + ignore_invalid_headers off; + client_max_body_size 0; + proxy_buffering off; + + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $remote_addr; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Port $server_port; + proxy_set_header X-Forwarded-Proto $scheme; + + proxy_connect_timeout 300; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_request_buffering off; + chunked_transfer_encoding off; + + # S3 API under /s3/ subpath + location /s3/ { + # Set the X-Forwarded-Prefix header to the stripped prefix + proxy_set_header X-Forwarded-Prefix /s3; + + # Strip the /s3 prefix before forwarding to backend + rewrite ^/s3/(.*) /$1 break; + + proxy_pass http://seaweedfs; + } + + # Alternative: S3 API under /api/s3/ subpath + location /api/s3/ { + proxy_set_header X-Forwarded-Prefix /api/s3; + rewrite ^/api/s3/(.*) /$1 break; + proxy_pass http://seaweedfs; + } + ssl on; ssl_certificate /{path_to_ssl_cert}/cert.pem; ssl_certificate_key /{path_to_ssl_cert}/key.pem; From d72d8785032bfc964084934afa1e7a6f30143464 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Mon, 11 Aug 2025 12:02:53 -0700 Subject: [PATCH 119/149] update for proxy headers X-Forwarded-Port --- Amazon-S3-API.md | 1 + S3-API-FAQ.md | 3 +++ S3-Nginx-Proxy.md | 4 +++- 3 files changed, 7 insertions(+), 1 deletion(-) diff --git a/Amazon-S3-API.md b/Amazon-S3-API.md index 9ab64bd..8686336 100644 --- a/Amazon-S3-API.md +++ b/Amazon-S3-API.md @@ -288,6 +288,7 @@ For detailed information about CORS configuration, see the dedicated [[S3-CORS]] SeaweedFS S3 API supports deployment behind reverse proxies with full AWS Signature v4 authentication compatibility. This includes support for: - **X-Forwarded-Host**: Preserves original host header for signature verification +- **X-Forwarded-Port**: Automatically combines with X-Forwarded-Host for non-standard ports - **X-Forwarded-Prefix**: Handles URL path prefix stripping by reverse proxies - **Standard forwarded headers**: X-Forwarded-For, X-Forwarded-Proto, etc. diff --git a/S3-API-FAQ.md b/S3-API-FAQ.md index 84e8098..557b764 100644 --- a/S3-API-FAQ.md +++ b/S3-API-FAQ.md @@ -198,10 +198,13 @@ Ensure all necessary headers are forwarded: ```nginx proxy_set_header Host $host; proxy_set_header X-Forwarded-Host $host; +proxy_set_header X-Forwarded-Port $server_port; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-For $remote_addr; ``` +**Note**: SeaweedFS automatically combines `X-Forwarded-Host` and `X-Forwarded-Port` for signature verification, omitting standard ports (80 for HTTP, 443 for HTTPS). + ### Testing Your Configuration You can test your reverse proxy configuration using AWS CLI: diff --git a/S3-Nginx-Proxy.md b/S3-Nginx-Proxy.md index 0499cd7..24ac967 100644 --- a/S3-Nginx-Proxy.md +++ b/S3-Nginx-Proxy.md @@ -3,7 +3,7 @@ It's a common concept to put a proxy in front of S3 that handles requests. Nginx For virtual-hosted style URL buckets, you'll need to add a [wildcard DNS record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) for your S3 subdomain. -Make sure the config sets the `X-Forwarded-Host` and optionally the `X-Forwarded-Port` if you are using a non-standard port. Otherwise, the AWS Signature v4 will not be correctly computed by the server, as it uses the Host header to compute the signature on the client side. +Make sure the config sets the `X-Forwarded-Host` and optionally the `X-Forwarded-Port` if you are using a non-standard port. SeaweedFS will automatically combine these headers to reconstruct the correct host information for signature verification. ## Reverse Proxy with URL Path Prefixes @@ -31,6 +31,8 @@ This ensures both regular S3 requests and presigned URLs work correctly with rev ### Important Notes - The `X-Forwarded-Prefix` header should contain the stripped prefix (e.g., `/s3`) +- `X-Forwarded-Port` is automatically combined with `X-Forwarded-Host` for non-standard ports +- Standard ports (80 for HTTP, 443 for HTTPS) are omitted from the host header automatically - Both regular S3 authentication and presigned URLs are supported - This feature works with all S3 operations that require signature verification From b33efe06323e3b9b9770fe081a2abe233338deeb Mon Sep 17 00:00:00 2001 From: chrislusf Date: Mon, 11 Aug 2025 12:05:07 -0700 Subject: [PATCH 120/149] copy --- File-Operations-Quick-Reference.md | 172 +++++++++++++++++++++++++++++ Filer-Commands-and-Operations.md | 30 +++++ Filer-Server-API.md | 123 +++++++++++++++++++++ _Sidebar.md | 1 + 4 files changed, 326 insertions(+) create mode 100644 File-Operations-Quick-Reference.md diff --git a/File-Operations-Quick-Reference.md b/File-Operations-Quick-Reference.md new file mode 100644 index 0000000..62e2e8b --- /dev/null +++ b/File-Operations-Quick-Reference.md @@ -0,0 +1,172 @@ +# File Operations Quick Reference + +This page provides a quick reference for common file operations in SeaweedFS using the HTTP API. + +## Basic File Operations + +### Upload a File + +**Small files (direct upload):** +```bash +# Upload with PUT +curl -X PUT "http://localhost:8888/path/to/file.txt" -d "file content" + +# Upload with POST (multipart) +curl -X POST "http://localhost:8888/path/to/file.txt" -F "file=@localfile.txt" +``` + +### Download a File + +```bash +# Get file content +curl "http://localhost:8888/path/to/file.txt" + +# Download to local file +curl "http://localhost:8888/path/to/file.txt" -o localfile.txt +``` + +### Delete a File + +```bash +# Delete single file +curl -X DELETE "http://localhost:8888/path/to/file.txt" + +# Delete directory (recursive) +curl -X DELETE "http://localhost:8888/path/to/directory/?recursive=true" +``` + +## Advanced File Operations + +### Move/Rename Files + +```bash +# Move file to new location +curl -X POST "http://localhost:8888/new/path/file.txt?mv.from=/old/path/file.txt" + +# Rename file in same directory +curl -X POST "http://localhost:8888/path/to/newname.txt?mv.from=/path/to/oldname.txt" +``` + +### Copy Files ⭐ **New Feature** + +```bash +# Copy file (preserves original) +curl -X POST "http://localhost:8888/backup/file.txt?cp.from=/original/file.txt" + +# Copy with automatic name resolution +curl -X POST "http://localhost:8888/backup/?cp.from=/original/file.txt" +# Result: /backup/file.txt +``` + +### List Directory Contents + +```bash +# List directory +curl -H "Accept: application/json" "http://localhost:8888/path/to/directory/?pretty=y" + +# List with pagination +curl -H "Accept: application/json" "http://localhost:8888/path/?limit=10&lastFileName=somefile.txt" +``` + +### Create Directory + +```bash +# Create empty directory +curl -X POST "http://localhost:8888/path/to/new/directory/" +``` + +## File Metadata Operations + +### Get File Information + +```bash +# Get file metadata +curl -I "http://localhost:8888/path/to/file.txt" +``` + +### File Attributes and Tagging + +```bash +# Set custom attributes +curl -X PUT "http://localhost:8888/path/to/file.txt" \ + -H "Seaweed-Custom-Attribute: value" \ + -F "file=@localfile.txt" + +# Set TTL (time to live) +curl -X POST "http://localhost:8888/path/to/file.txt?ttl=3600" \ + -F "file=@localfile.txt" +``` + +## Response Codes + +| HTTP Code | Operation | Meaning | +| --------- | --------- | ------- | +| 200 OK | GET | File retrieved successfully | +| 201 Created | POST/PUT | File uploaded successfully | +| 204 No Content | DELETE, COPY, MOVE | Operation completed successfully | +| 400 Bad Request | Any | Invalid parameters or unsupported operation | +| 404 Not Found | GET, DELETE | File or directory not found | +| 409 Conflict | POST/PUT | File already exists (when using exclusive creation) | + +## Operation Comparison + +| Operation | Source File | Use Case | Performance | +| --------- | ----------- | -------- | ----------- | +| **Upload** | External → SeaweedFS | Add new files | Network dependent | +| **Download** | SeaweedFS → External | Retrieve files | Network dependent | +| **Move** (`mv.from`) | Deleted | Rename/relocate | Very fast (metadata only) | +| **Copy** (`cp.from`) | Preserved | Backup/duplicate | Depends on file size | +| **Delete** | Deleted | Remove files | Fast | + +## Best Practices + +### When to Use Copy vs Move + +**Use Copy (`cp.from`) when:** +- Creating backups before modifications +- Duplicating configuration templates +- Staging files for testing +- Need to preserve original file + +**Use Move (`mv.from`) when:** +- Renaming files +- Reorganizing file structure +- Moving files between directories +- Don't need original file + +### Performance Tips + +1. **Batch operations**: Group multiple operations when possible +2. **Small files**: Use direct PUT for files < 1MB +3. **Large files**: POST with multipart automatically chunks files +4. **Copy operations**: Server-side copy is much faster than download + upload +5. **Directory operations**: Plan directory structure to minimize reorganization + +### Error Handling + +```bash +# Check operation status +if curl -X POST "http://localhost:8888/dest?cp.from=/src" -w "%{http_code}" -o /dev/null -s | grep -q "204"; then + echo "Copy successful" +else + echo "Copy failed" +fi +``` + +## Related Documentation + +- [Filer Server API](Filer-Server-API.md) - Complete API reference +- [Filer Commands and Operations](Filer-Commands-and-Operations.md) - Command-line tools +- [Getting Started](Getting-Started.md) - Installation and setup +- [FAQ](FAQ.md) - Common questions and answers + +## Examples Repository + +For more complex examples and use cases, see: +- [SeaweedFS Examples](https://github.com/seaweedfs/seaweedfs/tree/master/examples) (if available) +- Community-contributed scripts and tools +- Language-specific client libraries + +--- + +💡 **Tip**: Use the `?pretty=y` parameter with JSON responses to get formatted output for easier reading during development and testing. diff --git a/Filer-Commands-and-Operations.md b/Filer-Commands-and-Operations.md index 68b2169..7085a08 100644 --- a/Filer-Commands-and-Operations.md +++ b/Filer-Commands-and-Operations.md @@ -23,6 +23,36 @@ The above `weed copy` command is very efficient. It will contact the master serv This put very little loads on filer and the master server. Data is only transmitted between the local machine and the volume server. +## Copy files within Filer + +SeaweedFS also supports copying files within the filer using the HTTP API. This is useful for creating backups, duplicates, or templates without downloading and re-uploading files. + +### HTTP API Copy + +```bash +# Copy a file to a new location +curl -X POST 'http://localhost:8888/path/to/destination?cp.from=/path/to/source' + +# Copy with automatic name resolution +curl -X POST 'http://localhost:8888/backup/?cp.from=/important/config.json' +# Creates: /backup/config.json +``` + +**Features:** +- **Efficient**: Server-side copy without client data transfer +- **Independent chunks**: Creates new chunk copies (not shared references) +- **Atomic operation**: Either succeeds completely or fails with no partial state +- **Preserves metadata**: File attributes, timestamps, and permissions maintained + +**Comparison of copy methods:** + +| Method | Use Case | Data Transfer | Performance | +| ------ | -------- | ------------- | ----------- | +| `weed filer.copy` | Local files → Filer | Client → Volume Server | Good for initial uploads | +| `cp.from` HTTP API | File → File within Filer | Volume Server → Volume Server | Excellent for server-side copies | + +For more details, see the [Filer Server API documentation](Filer-Server-API.md#copy-files). + ## Register a file on Filer As mentioned above, the (path, fileId, fileSize) can be registered on filer with this gRPC call. diff --git a/Filer-Server-API.md b/Filer-Server-API.md index fced80b..ca63655 100644 --- a/Filer-Server-API.md +++ b/Filer-Server-API.md @@ -237,6 +237,129 @@ Notice that the tag names follow http header key convention, with the first char | ---- | -- | -- | | mv.from | move from one file or directory to another location | Required field | +### Copy files + +SeaweedFS supports efficient file copying using the `cp.from` parameter. This operation creates a complete copy of a file while preserving the original file. + +#### Basic Usage +```bash +# Copy a file to the same directory with a new name +> curl -X POST 'http://localhost:8888/documents/report_backup.pdf?cp.from=/documents/report.pdf' + +# Copy a file to a different directory +> curl -X POST 'http://localhost:8888/backup/important.txt?cp.from=/projects/important.txt' + +# Copy with automatic name resolution (uses source filename) +> curl -X POST 'http://localhost:8888/backup/?cp.from=/projects/important.txt' +# Creates: /backup/important.txt +``` + +#### Parameters +| POST Parameter | Description | Default | +| ---- | -- | -- | +| cp.from | Source file path to copy from. Must be a valid file path. | Required field | + +#### How Copy Works + +**Small Files (< chunk size):** +- Content is stored directly in the filer metadata +- Copy operation duplicates the content bytes +- Very fast, only metadata operation required + +**Large Files (chunked):** +- File data is stored as chunks on volume servers +- Copy operation reads data from source chunks and writes to new chunks +- Creates independent chunk copies (not shared references) +- Preserves file integrity and allows independent deletion + +#### Examples + +**Copy a configuration file:** +```bash +curl -X POST 'http://localhost:8888/config/app.conf.backup?cp.from=/config/app.conf' +``` + +**Copy a large media file:** +```bash +curl -X POST 'http://localhost:8888/media/backup/video.mp4?cp.from=/media/original/video.mp4' +``` + +**Copy with path resolution:** +```bash +# If destination ends with /, uses source filename +curl -X POST 'http://localhost:8888/backup/?cp.from=/important/data.json' +# Result: /backup/data.json +``` + +#### Response Codes + +| HTTP Code | Description | +| ---- | -- | +| 204 No Content | Copy operation completed successfully | +| 400 Bad Request | Invalid source path, missing cp.from parameter, or directory copy attempt | +| 404 Not Found | Source file does not exist | +| 500 Internal Server Error | Volume server error or chunk copy failure | + +#### Performance Characteristics + +- **Small files**: Near-instant (metadata only) +- **Large files**: Proportional to file size (requires data transfer) +- **Network efficient**: Direct volume-to-volume transfer when possible +- **Atomic operation**: Either completes fully or fails with no partial state + +#### Limitations + +- **Directory copying**: Not supported (returns 400 error) +- **Cross-cluster copying**: Limited to same SeaweedFS cluster +- **Concurrent access**: Source file should not be modified during copy + +#### Error Examples + +**Attempting to copy a directory:** +```bash +curl -X POST 'http://localhost:8888/new_folder/?cp.from=/existing_folder/' +# Returns: 400 Bad Request - "directory copying not yet supported" +``` + +**Source file not found:** +```bash +curl -X POST 'http://localhost:8888/copy.txt?cp.from=/nonexistent.txt' +# Returns: 400 Bad Request - "failed to get src entry" +``` + +**Missing cp.from parameter:** +```bash +curl -X POST 'http://localhost:8888/copy.txt' +# Returns: Normal file upload behavior (not a copy operation) +``` + +#### Comparison with Move Operation + +| Operation | Source File | Use Case | Speed | +| ---- | ---- | ---- | ---- | +| `mv.from` | Deleted | Rename/relocate files | Very fast (metadata only) | +| `cp.from` | Preserved | Backup/duplicate files | Depends on file size | + +#### Best Practices + +1. **Backup workflows**: Use copy for creating backups before modifications +2. **Template files**: Copy configuration templates to create new instances +3. **Data migration**: Copy files before cross-cluster transfers +4. **Testing**: Copy production files to staging environments + +#### Shell Command Equivalent + +The SeaweedFS copy operation is similar to: +```bash +# SeaweedFS copy +curl -X POST 'http://localhost:8888/dst?cp.from=/src' + +# Unix equivalent +cp /src /dst +``` + +Note: Directory copying is not currently supported. Only individual files can be copied. + ### Create an empty folder Folders usually are created automatically when uploading a file. To create an empty file, you can use this: ``` diff --git a/_Sidebar.md b/_Sidebar.md index e3f307d..50a172e 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -24,6 +24,7 @@ ### Filer * [[Filer Setup]] * [[Directories and Files]] +* [[File Operations Quick Reference]] * [[Data Structure for Large Files]] * [[Filer Data Encryption]] * [[Filer Commands and Operations]] From f234e69c413424f614ddc0955072310b3f8b1ae6 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Sat, 23 Aug 2025 09:08:57 -0700 Subject: [PATCH 121/149] support SSE-C SSE-KMS Conditional read and write --- Amazon-S3-API.md | 82 +++++++++ S3-Conditional-Operations.md | 84 +++++++++ Server-Side-Encryption-SSE-C.md | 137 ++++++++++++++ Server-Side-Encryption-SSE-KMS.md | 293 ++++++++++++++++++++++++++++++ Server-Side-Encryption.md | 108 +++++++++++ _Sidebar.md | 6 + 6 files changed, 710 insertions(+) create mode 100644 S3-Conditional-Operations.md create mode 100644 Server-Side-Encryption-SSE-C.md create mode 100644 Server-Side-Encryption-SSE-KMS.md create mode 100644 Server-Side-Encryption.md diff --git a/Amazon-S3-API.md b/Amazon-S3-API.md index 8686336..c8ad299 100644 --- a/Amazon-S3-API.md +++ b/Amazon-S3-API.md @@ -55,6 +55,20 @@ To be sure, you can look at the function defined in the files `weed/s3api/s3api_ * PutObjectTagging * DeleteObjectTagging +// Server-Side Encryption (NEW) +* PutObject (with SSE-KMS, SSE-C) +* GetObject (with automatic decryption) +* HeadObject (with encryption metadata) +* CopyObject (with encryption/decryption) +* Multipart uploads with encryption + +// Conditional Operations (NEW) +* All object operations support conditional headers: + - If-Match + - If-None-Match + - If-Modified-Since + - If-Unmodified-Since + // Bucket operations * PutBucket * DeleteBucket @@ -109,6 +123,12 @@ Not included: | allows more than "/" as a delimiter | No | Yes | | Object Versioning | Yes | Yes | | MFA Delete for versioning | No | Yes | +| Server-Side Encryption (SSE-KMS) | Yes | Yes | +| Server-Side Encryption (SSE-C) | Yes | Yes | +| KMS Providers (Multi-cloud) | Yes | No | +| Conditional Headers (All operations) | Yes | Yes | +| Range requests with SSE-KMS | Yes | Yes | +| Range requests with SSE-C | No | No | ## Empty folders @@ -116,6 +136,68 @@ SeaweedFS has directories while AWS S3 only has objects with "fake" directories. To be consistent with AWS S3, SeaweedFS tries to check whether the folder is empty after each deletion. You can use `weed s3 -allowEmptyFolder` to toggle this behavior. +# Server-Side Encryption + +SeaweedFS supports AWS S3-compatible server-side encryption to protect your data at rest. Two encryption methods are available: + +- **[SSE-KMS](Server-Side-Encryption-SSE-KMS)**: External Key Management Service providers (AWS KMS, Google Cloud KMS, OpenBao/Vault) +- **[SSE-C](Server-Side-Encryption-SSE-C)**: Customer-provided encryption keys (maximum customer control) + +All encryption types support: +- Automatic encryption/decryption +- Bucket default encryption +- Multipart upload encryption +- Cross-encryption copy operations +- AWS S3 compatibility + +For detailed setup guides and examples, see: +- **[Server-Side Encryption Overview](Server-Side-Encryption)** +- **[SSE-KMS Guide](Server-Side-Encryption-SSE-KMS)** + +## Quick Examples + +```bash +# SSE-KMS (Key Management Service) +aws s3 cp file.txt s3://mybucket/kms-encrypted.txt --server-side-encryption aws:kms --ssekms-key-id alias/my-key + +# SSE-C (Customer-provided keys) +aws s3 cp file.txt s3://mybucket/customer-encrypted.txt --sse-c AES256 --sse-c-key fileb://my-key.bin +``` + +# S3 Conditional Operations + +SeaweedFS supports AWS S3-compatible conditional headers for safe concurrent operations and efficient caching: + +- **If-Match**: Execute only if ETag matches (optimistic locking) +- **If-None-Match**: Execute only if ETag doesn't match (prevent overwrites, caching) +- **If-Modified-Since**: Execute only if modified after date (conditional downloads) +- **If-Unmodified-Since**: Execute only if not modified after date (safe updates) + +Conditional operations enable: +- **Optimistic concurrency control**: Prevent lost updates +- **Efficient caching**: Reduce bandwidth with 304 Not Modified +- **Atomic operations**: Operations only proceed when safe +- **Data integrity**: Prevent accidental overwrites + +For detailed usage patterns and examples, see **[S3 Conditional Operations](S3-Conditional-Operations)**. + +## Quick Examples + +```bash +# Get current ETag +ETAG=$(aws s3api head-object --bucket mybucket --key file.txt --query ETag --output text) + +# Conditional update (optimistic locking) +curl -X PUT -H "If-Match: $ETAG" -d "updated content" "http://localhost:8333/mybucket/file.txt" + +# Conditional download (caching) +curl -H "If-None-Match: $ETAG" "http://localhost:8333/mybucket/file.txt" +# Returns 304 Not Modified if unchanged + +# Prevent overwrite (atomic create) +curl -X PUT -H "If-None-Match: *" -d "new content" "http://localhost:8333/mybucket/newfile.txt" +``` + # S3 Authentication By default, the access key and secret key to access `weed s3` is not authenticated. To enable credential based access, you can choose static or dynamic configuration: diff --git a/S3-Conditional-Operations.md b/S3-Conditional-Operations.md new file mode 100644 index 0000000..b31424f --- /dev/null +++ b/S3-Conditional-Operations.md @@ -0,0 +1,84 @@ +# S3 Conditional Operations + +SeaweedFS supports AWS S3-compatible conditional headers for safe concurrent access patterns, optimistic locking, and efficient conditional operations. + +## Supported Conditional Headers + +| Header | Applies To | Condition | Use Case | +|--------|------------|-----------|----------| +| **If-Match** | GET, PUT, COPY | ETag matches | Ensure object hasn't changed | +| **If-None-Match** | GET, PUT, COPY | ETag doesn't match | Prevent overwrites, caching | +| **If-Modified-Since** | GET, COPY | Modified after date | Conditional downloads | +| **If-Unmodified-Since** | GET, PUT, COPY | Not modified after date | Safe modifications | + +## HTTP Examples + +### Conditional GET (Caching) + +```bash +# First, get the object and note its ETag +curl -I "http://localhost:8333/mybucket/myfile.txt" +# Response: ETag: "d41d8cd98f00b204e9800998ecf8427e" + +# Conditional GET - only download if object changed +curl -H "If-None-Match: \"d41d8cd98f00b204e9800998ecf8427e\"" \ + "http://localhost:8333/mybucket/myfile.txt" +``` + +### Conditional PUT (Optimistic Locking) + +```bash +# Safe update - only modify if object hasn't changed +curl -X PUT -H "If-Match: \"d41d8cd98f00b204e9800998ecf8427e\"" \ + -H "Content-Type: text/plain" \ + --data "Updated content" \ + "http://localhost:8333/mybucket/myfile.txt" + +# Prevent overwrites - only create if object doesn't exist +curl -X PUT -H "If-None-Match: *" \ + -H "Content-Type: text/plain" \ + --data "New file content" \ + "http://localhost:8333/mybucket/newfile.txt" +``` + +### If-Modified-Since + +```bash +# Only download if modified after specific date +curl -H "If-Modified-Since: Wed, 15 Jan 2024 10:00:00 GMT" \ + "http://localhost:8333/mybucket/log-file.txt" +``` + +### If-Unmodified-Since + +```bash +# Update only if not modified since last read +curl -X PUT \ + -H "If-Unmodified-Since: Wed, 15 Jan 2024 10:00:00 GMT" \ + -H "Content-Type: text/plain" \ + --data "Updated content" \ + "http://localhost:8333/mybucket/document.txt" +``` + +### Copy Operations + +```bash +# Copy only if source object hasn't changed +curl -X PUT \ + -H "x-amz-copy-source: /source-bucket/source-file.txt" \ + -H "x-amz-copy-source-if-match: \"source-etag\"" \ + "http://localhost:8333/dest-bucket/dest-file.txt" +``` + +## HTTP Status Codes + +| Status Code | Condition | Meaning | +|-------------|-----------|---------| +| **200 OK** | Condition met | Operation succeeded | +| **304 Not Modified** | If-None-Match matched | Object unchanged (GET only) | +| **412 Precondition Failed** | Condition not met | Operation blocked by condition | + +## Related Documentation + +- [Amazon S3 API](Amazon-S3-API.md) +- [Server-Side Encryption](Server-Side-Encryption.md) \ No newline at end of file diff --git a/Server-Side-Encryption-SSE-C.md b/Server-Side-Encryption-SSE-C.md new file mode 100644 index 0000000..429ea8c --- /dev/null +++ b/Server-Side-Encryption-SSE-C.md @@ -0,0 +1,137 @@ +# Server-Side Encryption with Customer-provided Keys (SSE-C) + +SeaweedFS supports AWS S3-compatible Server-Side Encryption with Customer-provided Keys (SSE-C), allowing clients to encrypt objects using their own encryption keys. + +## Overview + +SSE-C enables client-side key management with server-side encryption: + +- **Client provides**: AES-256 encryption key and MD5 hash +- **SeaweedFS handles**: Encryption/decryption operations transparently +- **Security**: Keys are never stored on the server + +## Required Headers + +For all SSE-C operations, provide these headers: + +```http +X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256 +X-Amz-Server-Side-Encryption-Customer-Key: +X-Amz-Server-Side-Encryption-Customer-Key-MD5: +``` + +## HTTP Examples + +### Upload Encrypted Object + +```bash +# Generate a 256-bit key +KEY=$(openssl rand -base64 32) +KEY_MD5=$(echo -n "$KEY" | base64 -d | md5sum | cut -d' ' -f1) + +# Upload encrypted object +curl -X PUT "http://localhost:8333/bucket/encrypted-file.txt" \ + -H "X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256" \ + -H "X-Amz-Server-Side-Encryption-Customer-Key: $KEY" \ + -H "X-Amz-Server-Side-Encryption-Customer-Key-MD5: $KEY_MD5" \ + -H "Content-Type: text/plain" \ + --data "This content will be encrypted" +``` + +### Download Encrypted Object + +```bash +# Download and decrypt object (must use same key) +curl "http://localhost:8333/bucket/encrypted-file.txt" \ + -H "X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256" \ + -H "X-Amz-Server-Side-Encryption-Customer-Key: $KEY" \ + -H "X-Amz-Server-Side-Encryption-Customer-Key-MD5: $KEY_MD5" +``` + +### Get Object Metadata + +```bash +# Get metadata for encrypted object +curl -I "http://localhost:8333/bucket/encrypted-file.txt" \ + -H "X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256" \ + -H "X-Amz-Server-Side-Encryption-Customer-Key: $KEY" \ + -H "X-Amz-Server-Side-Encryption-Customer-Key-MD5: $KEY_MD5" +``` + +### Copy Operations + +```bash +# Copy encrypted object to new location (same key) +curl -X PUT "http://localhost:8333/bucket/copied-file.txt" \ + -H "x-amz-copy-source: /bucket/encrypted-file.txt" \ + -H "X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256" \ + -H "X-Amz-Server-Side-Encryption-Customer-Key: $KEY" \ + -H "X-Amz-Server-Side-Encryption-Customer-Key-MD5: $KEY_MD5" \ + -H "x-amz-copy-source-server-side-encryption-customer-algorithm: AES256" \ + -H "x-amz-copy-source-server-side-encryption-customer-key: $KEY" \ + -H "x-amz-copy-source-server-side-encryption-customer-key-md5: $KEY_MD5" + +# Copy encrypted object with different key +NEW_KEY=$(openssl rand -base64 32) +NEW_KEY_MD5=$(echo -n "$NEW_KEY" | base64 -d | md5sum | cut -d' ' -f1) + +curl -X PUT "http://localhost:8333/bucket/reencrypted-file.txt" \ + -H "x-amz-copy-source: /bucket/encrypted-file.txt" \ + -H "X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256" \ + -H "X-Amz-Server-Side-Encryption-Customer-Key: $NEW_KEY" \ + -H "X-Amz-Server-Side-Encryption-Customer-Key-MD5: $NEW_KEY_MD5" \ + -H "x-amz-copy-source-server-side-encryption-customer-algorithm: AES256" \ + -H "x-amz-copy-source-server-side-encryption-customer-key: $KEY" \ + -H "x-amz-copy-source-server-side-encryption-customer-key-md5: $KEY_MD5" +``` + +### AWS CLI Usage + +```bash +# Upload with SSE-C +aws s3 cp file.txt s3://mybucket/file.txt \ + --sse-c AES256 \ + --sse-c-key fileb://customer-key.bin + +# Download with SSE-C +aws s3 cp s3://mybucket/file.txt downloaded-file.txt \ + --sse-c AES256 \ + --sse-c-key fileb://customer-key.bin + +# Copy with SSE-C (same key) +aws s3 cp s3://mybucket/file.txt s3://mybucket/file-copy.txt \ + --sse-c AES256 \ + --sse-c-key fileb://customer-key.bin \ + --sse-c-copy-source AES256 \ + --sse-c-copy-source-key fileb://customer-key.bin +``` + +## Error Codes + +| Error | HTTP Status | Description | +|-------|-------------|-------------| +| `InvalidEncryptionAlgorithmError` | 400 | Algorithm must be "AES256" | +| `InvalidArgument` | 400 | Invalid key format or MD5 mismatch | +| `InvalidRequest` | 400 | Missing SSE-C headers | + +## Common Issues + +**Wrong algorithm:** +```bash +X-Amz-Server-Side-Encryption-Customer-Algorithm: AES128 # Error! +``` + +**Invalid key length:** +```bash +X-Amz-Server-Side-Encryption-Customer-Key: dGVzdA== # Error! (too short) +``` + +**Missing key for encrypted object:** +```bash +curl http://localhost:8333/bucket/encrypted-file.txt # Error! +``` + +## Related Documentation + +- [Server-Side Encryption](Server-Side-Encryption.md) +- [Amazon S3 API](Amazon-S3-API.md) diff --git a/Server-Side-Encryption-SSE-KMS.md b/Server-Side-Encryption-SSE-KMS.md new file mode 100644 index 0000000..9d49b45 --- /dev/null +++ b/Server-Side-Encryption-SSE-KMS.md @@ -0,0 +1,293 @@ +# SSE-KMS: Server-Side Encryption with Key Management Service + +SeaweedFS SSE-KMS supports multiple external Key Management Service providers for enterprise-grade encryption key management. This guide covers setup and configuration for AWS KMS, Google Cloud KMS, and OpenBao/Vault. + +## Supported KMS Providers + +| Provider | Status | Use Cases | +|----------|--------|-----------| +| **AWS KMS** | Full support | AWS-centric deployments | +| **Google Cloud KMS** | Full support | GCP-centric deployments | +| **OpenBao/Vault** | Full support | Hybrid/on-premises environments | +| **Azure Key Vault** | Planned | Future Azure deployments | + +## Quick Start Guide + +### 1. Configure SeaweedFS + +KMS providers are configured in the S3 config JSON file: + +```json +{ + "identities": [ + { + "name": "admin", + "credentials": [{"accessKey": "admin", "secretKey": "password"}], + "actions": ["Admin", "Read", "Write"] + } + ], + "kms": { + "default_provider": "openbao", + "providers": { + "openbao": { + "type": "openbao", + "address": "http://localhost:8200", + "token": "root-token", + "transit_path": "transit", + "cache_enabled": true, + "cache_ttl": "1h" + } + } + } +} +``` + +### 2. Start S3 with KMS Config + +```bash +# Start S3 API with KMS and IAM configuration +weed s3 -config=s3_kms_config.json -port=8333 +``` + +**Note:** The S3 config JSON file contains both KMS provider settings AND IAM-style access control (identities, credentials, permissions). + +### 3. Test the Integration + +```bash +# Upload object with SSE-KMS +aws s3 cp test-file.txt s3://mybucket/test-file.txt \ + --server-side-encryption aws:kms \ + --ssekms-key-id alias/my-key +``` + +--- + +## AWS KMS Integration + +### Step 1: Create KMS Key + +```bash +# Create customer-managed KMS key +aws kms create-key --description "SeaweedFS encryption key" + +# Create key alias +aws kms create-alias \ + --alias-name alias/seaweedfs-key \ + --target-key-id +``` + +### Step 2: Configure SeaweedFS + +```json +{ + "identities": [ + { + "name": "admin", + "credentials": [{"accessKey": "admin", "secretKey": "password"}], + "actions": ["Admin", "Read", "Write"] + } + ], + "kms": { + "default_provider": "aws-kms", + "providers": { + "aws-kms": { + "type": "aws-kms", + "region": "us-east-1", + "key_id": "alias/seaweedfs-key" + } + } + } +} +``` + +### Step 3: Test AWS KMS + +```bash +# Upload with AWS KMS encryption +aws s3 cp file.txt s3://mybucket/file.txt \ + --server-side-encryption aws:kms \ + --ssekms-key-id alias/seaweedfs-key +``` + +--- + +## Google Cloud KMS Integration + +### Step 1: Create KMS Resources + +```bash +# Create key ring +gcloud kms keyrings create seaweedfs-keyring --location us-east1 + +# Create encryption key +gcloud kms keys create seaweedfs-key \ + --keyring seaweedfs-keyring \ + --location us-east1 \ + --purpose encryption +``` + +### Step 2: Configure SeaweedFS + +```json +{ + "identities": [ + { + "name": "admin", + "credentials": [{"accessKey": "admin", "secretKey": "password"}], + "actions": ["Admin", "Read", "Write"] + } + ], + "kms": { + "default_provider": "gcp-kms", + "providers": { + "gcp-kms": { + "type": "gcp-kms", + "project_id": "my-project-id", + "location": "us-east1", + "key_ring": "seaweedfs-keyring", + "key_name": "seaweedfs-key", + "credentials_file": "/etc/seaweedfs/gcp-kms-key.json" + } + } + } +} +``` + +--- + +## OpenBao/Vault Integration + +### Step 1: Setup OpenBao/Vault + +```bash +# Start OpenBao in dev mode (for testing) +openbao server -dev -dev-root-token-id="root-token" + +# Enable transit secrets engine +openbao secrets enable transit + +# Create encryption key +openbao write -f transit/keys/seaweedfs-key +``` + +### Step 2: Configure SeaweedFS + +```json +{ + "identities": [ + { + "name": "admin", + "credentials": [{"accessKey": "admin", "secretKey": "password"}], + "actions": ["Admin", "Read", "Write"] + } + ], + "kms": { + "default_provider": "openbao", + "providers": { + "openbao": { + "type": "openbao", + "address": "http://localhost:8200", + "token": "root-token", + "transit_path": "transit", + "cache_enabled": true, + "cache_ttl": "1h" + } + } + } +} +``` + +### Step 3: Test OpenBao/Vault + +```bash +# Upload with Vault encryption +aws s3 cp file.txt s3://mybucket/file.txt \ + --server-side-encryption aws:kms \ + --ssekms-key-id seaweedfs-key +``` + +--- + +## Azure Key Vault Integration + +**Status**: Planned (not yet implemented) + +Azure Key Vault integration is planned for future releases. When available, it will support: +- Azure AD authentication +- HSM-backed keys +- Integration with Azure Monitor + +--- + +## Multi-Provider Configuration + +### Provider Selection Strategies + +```json +{ + "identities": [...], + "kms": { + "default_provider": "openbao", + "providers": { + "openbao": { + "type": "openbao", + "address": "https://vault.internal:8200", + "token": "root-token", + "transit_path": "transit" + }, + "aws-kms": { + "type": "aws-kms", + "region": "us-east-1", + "key_id": "alias/seaweedfs-aws" + }, + "gcp-kms": { + "type": "gcp-kms", + "project_id": "my-gcp-project", + "location": "global", + "key_ring": "seaweedfs-keyring", + "key_name": "seaweedfs-gcp-key", + "credentials_file": "/etc/seaweedfs/gcp-key.json" + } + }, + "buckets": { + "financial-data": {"provider": "openbao"}, + "ml-models": {"provider": "gcp-kms"}, + "general-storage": {"provider": "aws-kms"} + } + } +} +``` + +## Security Best Practices + +### 1. Key Management + +- Grant minimal required KMS permissions +- Use resource-based policies where possible +- Implement proper key rotation policies +- Document key usage and ownership + +### 2. Access Control + +```json +{ + "identities": [ + { + "name": "admin", + "credentials": [{"accessKey": "admin", "secretKey": "password"}], + "actions": ["Admin", "Read", "Write"] + }, + { + "name": "readonly", + "credentials": [{"accessKey": "readonly", "secretKey": "password"}], + "actions": ["Read"] + } + ] +} +``` + +## Related Documentation + +- **[Server-Side Encryption](Server-Side-Encryption.md)**: Overview of all SSE types +- **[SSE-C Guide](Server-Side-Encryption-SSE-C.md)**: Customer-provided keys +- **[Amazon S3 API](Amazon-S3-API.md)**: Main S3 API documentation diff --git a/Server-Side-Encryption.md b/Server-Side-Encryption.md new file mode 100644 index 0000000..6e6368e --- /dev/null +++ b/Server-Side-Encryption.md @@ -0,0 +1,108 @@ +# Server-Side Encryption (SSE) + +SeaweedFS supports AWS S3-compatible server-side encryption to protect data at rest. Two encryption methods are available, each offering different levels of control over encryption keys and management. + +## Overview + +| Encryption Type | Key Management | Use Case | +|------------------|----------------|----------| +| **SSE-KMS** | External KMS providers | Enterprise key management, audit trails | +| **SSE-C** | Customer-provided | Full customer control, regulatory compliance | + +## Encryption Types + +### SSE-KMS (Server-Side Encryption with Key Management Service) +- **Best for**: Enterprise environments with external KMS infrastructure +- **Key management**: External providers (AWS KMS, Google Cloud KMS, OpenBao/Vault) +- **Features**: Centralized key management, detailed audit logs, per-bucket key assignment +- **Configuration**: Requires KMS provider setup in S3 config +- **Documentation**: [SSE-KMS Guide](Server-Side-Encryption-SSE-KMS.md) + +### SSE-C (Server-Side Encryption with Customer-Provided Keys) +- **Best for**: Maximum control over encryption keys, regulatory compliance +- **Key management**: Customer provides encryption keys with each request +- **Features**: No key storage on server, customer has full control +- **Configuration**: Keys provided via HTTP headers +- **Documentation**: [SSE-C Guide](Server-Side-Encryption-SSE-C.md) + +## Quick Start + +### SSE-KMS (Enterprise) +```bash +# Configure KMS in s3 config file (see KMS Providers Integration guide) +# Then upload with KMS encryption +aws s3 cp file.txt s3://mybucket/file.txt \ + --server-side-encryption aws:kms \ + --ssekms-key-id test-key-123 +``` + +### SSE-C (Customer Keys) +```bash +# Generate customer key +openssl rand 32 > customer-key.bin + +# Upload with customer-provided key +aws s3 cp file.txt s3://mybucket/file.txt \ + --sse-c AES256 \ + --sse-c-key fileb://customer-key.bin +``` + +## Configuration + +### Basic Setup +Configure KMS providers and IAM settings in the S3 config file: + +```json +{ + "identities": [ + { + "name": "admin", + "credentials": [{"accessKey": "admin", "secretKey": "password"}], + "actions": ["Admin", "Read", "Write"] + } + ], + "kms": { + "default_provider": "openbao", + "providers": { + "openbao": { + "type": "openbao", + "address": "http://localhost:8200", + "token": "root-token", + "transit_path": "transit" + } + } + } +} +``` + +**Note:** The S3 config JSON file contains both KMS encryption settings AND IAM-style access control (user identities, credentials, permissions). + +### Start S3 API with Encryption Support +```bash +# Start with KMS config +weed s3 -config=s3_kms_config.json +``` + +## Features + +**Supported Operations:** +- All standard S3 operations (PUT, GET, HEAD, COPY, DELETE) +- Multipart uploads with consistent encryption +- Cross-encryption copy operations +- Object metadata preservation +- Range requests (SSE-KMS only; SSE-C does not support range requests) + +**AWS S3 Compatibility:** +- Identical API behavior and headers +- Compatible with all S3 clients and SDKs +- Same error codes and responses + +## Implementation Notes + +- **SSE-KMS**: Supports AWS KMS, Google Cloud KMS, OpenBao/Vault +- **SSE-C**: Full support with security best practices +- **Azure Key Vault**: Not yet implemented (planned) + +**Note**: SSE-S3 (server-managed keys) is currently in development and not yet production-ready. + +For detailed setup guides, examples, and configuration options, see the individual encryption method documentation linked above. \ No newline at end of file diff --git a/_Sidebar.md b/_Sidebar.md index 50a172e..5aa94e3 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -69,6 +69,7 @@ ### AWS S3 API * [[S3 Credentials]] * [[Amazon S3 API]] +* [[S3 Conditional Operations]] * [[S3 Object Lock and Retention]] * [[AWS CLI with SeaweedFS]] * [[s3cmd with SeaweedFS]] @@ -82,6 +83,11 @@ * [[S3 Nginx Proxy]] * [[Docker Compose for S3]] +### Server-Side Encryption +* [[Server-Side Encryption]] +* [[Server-Side Encryption SSE-KMS]] +* [[Server-Side Encryption SSE-C]] + ### AWS IAM * [[Amazon IAM API]] * [[AWS IAM CLI]] From 5e1342f10c99d167e9194c85fe412a16d41d32d5 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Mon, 8 Sep 2025 01:38:14 -0700 Subject: [PATCH 122/149] add SQL --- SQL-Queries-on-Message-Queue.md | 627 ++++++++++++++++++++++++++++++++ SQL-Quick-Reference.md | 195 ++++++++++ _Sidebar.md | 2 + 3 files changed, 824 insertions(+) create mode 100644 SQL-Queries-on-Message-Queue.md create mode 100644 SQL-Quick-Reference.md diff --git a/SQL-Queries-on-Message-Queue.md b/SQL-Queries-on-Message-Queue.md new file mode 100644 index 0000000..e9e26fd --- /dev/null +++ b/SQL-Queries-on-Message-Queue.md @@ -0,0 +1,627 @@ +# SQL Queries on Message Queue + +SeaweedFS provides a powerful SQL query engine that allows you to query Message Queue topics using standard SQL syntax. This feature enables analytics, reporting, and data exploration on your message data using familiar SQL tools and PostgreSQL-compatible clients. + +## Overview + +The SQL query engine bridges the gap between SeaweedFS's Message Queue and traditional SQL databases by providing: + +- **PostgreSQL Wire Protocol Compatibility** - Use any PostgreSQL client, tool, or application +- **Real-time + Historical Data** - Query both live messages and archived Parquet data +- **Standard SQL Operations** - SELECT, aggregations, filtering, and schema operations +- **Multiple Interface Options** - Database server mode and interactive CLI +- **Secure Authentication** - MD5, password, and trust authentication methods + +## Architecture + +``` +┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ +│ SQL Clients │ │ PostgreSQL │ │ SeaweedFS │ +│ (psql, apps) │────│ Wire Protocol │────│ SQL Engine │ +└─────────────────┘ └─────────────────┘ └─────────────────┘ + │ + ┌─────────────────┐ + │ Hybrid Scanner │ + │ • Live Messages │ + │ • Parquet Files │ + └─────────────────┘ +``` + +## Quick Start + +### 1. Start the Database Server + +```bash +# Start PostgreSQL-compatible database server +weed db -auth=md5 -users='{"admin":"secret","analyst":"readonly"}' + +# Or use a JSON file for credentials +echo '{"admin":"secret","analyst":"readonly"}' > users.json +weed db -auth=md5 -users="@users.json" -port=5432 +``` + +### 2. Connect with psql + +```bash +# Connect using psql +PGPASSWORD=secret psql -h localhost -p 5432 -U admin -d default + +# Or with connection string +psql "postgresql://admin:secret@localhost:5432/default" +``` + +### 3. Start Querying + +```sql +-- List available databases (MQ namespaces) +SHOW DATABASES; + +-- Switch to a namespace +USE my_namespace; + +-- List tables (MQ topics) +SHOW TABLES; + +-- Query message data +SELECT * FROM user_events WHERE _ts > '2025-01-01' LIMIT 10; + +-- Perform basic aggregations +SELECT COUNT(*) as total_events FROM user_events; +``` + +## Commands and Interfaces + +### Database Server Mode (`weed db`) + +Starts a PostgreSQL-compatible database server that accepts connections from any PostgreSQL client. + +```bash +# Basic usage +weed db + +# Production setup with MD5 authentication +weed db -auth=md5 -users="@/etc/seaweedfs/users.json" \ + -host=0.0.0.0 -port=5432 + +# With TLS encryption +weed db -auth=md5 -users="@users.json" \ + -tls-cert=/etc/ssl/server.crt \ + -tls-key=/etc/ssl/server.key + +# Custom configuration +weed db -auth=md5 -users='{"admin":"pass"}' \ + -port=5433 \ + -master=master1:9333 \ + -max-connections=200 \ + -idle-timeout=2h +``` + +**Options:** +- `-auth`: Authentication method (`trust`, `password`, `md5`) +- `-users`: User credentials (JSON format or file) +- `-host`: Database server host (default: `localhost`) +- `-port`: Database server port (default: `5432`) +- `-master`: SeaweedFS master server address +- `-database`: Default database name +- `-max-connections`: Maximum concurrent connections +- `-idle-timeout`: Connection idle timeout +- `-tls-cert`, `-tls-key`: TLS certificate and key files + +### Interactive CLI Mode (`weed sql`) + +Provides an interactive SQL shell for quick queries and exploration. + +```bash +# Start interactive SQL shell +weed sql -master=localhost:9333 + +# Connect to specific namespace +weed sql -master=localhost:9333 -namespace=analytics + +# Execute single query +weed sql -master=localhost:9333 -exec="SHOW TABLES" +``` + +## Authentication and Security + +### Authentication Methods + +1. **`trust`** (Default - Development Only) + - No authentication required + - Suitable for local development only + +2. **`md5`** (Recommended for Production) + - Secure MD5 hash + salt authentication + - Compatible with all PostgreSQL clients + - Safe even without TLS + +3. **`password`** (Development + TLS Only) + - Clear text password transmission + - Requires TLS encryption for production use + +### Credential Formats + +#### JSON Format (Inline) +```bash +weed db -auth=md5 -users='{"admin":"secret","analyst":"readonly"}' + +# Supports any special characters +weed db -auth=md5 -users='{"admin":"pass;with;semicolons","user":"pass:with:colons"}' +``` + +#### File Format (Recommended) +```bash +# Create users file +cat > /etc/seaweedfs/users.json << EOF +{ + "admin": "strong_password_123!", + "analyst": "readonly_user_456", + "developer": "dev_access_789" +} +EOF + +# Use file-based credentials +weed db -auth=md5 -users="@/etc/seaweedfs/users.json" +``` + +### TLS Encryption + +```bash +# Generate certificates (example) +openssl req -x509 -newkey rsa:2048 -keyout server.key -out server.crt -days 365 -nodes + +# Start with TLS +weed db -auth=md5 -users="@users.json" \ + -tls-cert=server.crt -tls-key=server.key +``` + +## SQL Operations + +### Schema Operations + +```sql +-- List all databases (MQ namespaces) +SHOW DATABASES; + +-- Switch database context +USE namespace_name; + +-- List tables in current database (MQ topics) +SHOW TABLES; + +-- Describe table schema +DESCRIBE table_name; +SHOW COLUMNS FROM table_name; + +-- Note: CREATE TABLE, DROP TABLE and ALTER TABLE are not supported +``` + +### Data Queries + +```sql +-- Basic SELECT +SELECT * FROM user_events LIMIT 10; + +-- Filtering with WHERE clauses +SELECT * FROM user_events +WHERE event_type = 'login' + AND _ts > '2025-01-01'; + +-- Basic aggregations (limited support) +SELECT COUNT(*) FROM user_events; +SELECT MIN(timestamp), MAX(timestamp) FROM user_events; +SELECT SUM(value_column), AVG(value_column) FROM user_events; + +-- System columns (available on all topics) +SELECT + _ts, -- Message timestamp (formatted, supports string parsing in WHERE) + _key, -- Message key + _source, -- Data source (parquet file or live) + * +FROM user_events; +``` + +**Note:** The `_ts` system column supports automatic parsing of timestamp strings in WHERE clauses. Supported formats include `'2025-01-01'`, `'2025-01-01T15:30:00Z'`, `'2025-01-01 15:30:00'`, etc. Other timestamp columns require exact value matching. + +### Supported WHERE Clause Operations + +```sql +-- Comparison operators +SELECT * FROM user_events WHERE user_id = 123; +SELECT * FROM user_events WHERE _ts > '2025-01-01'; +SELECT * FROM user_events WHERE value <= 100; + +-- Pattern matching +SELECT * FROM user_events WHERE event_type LIKE 'login%'; + +-- IN clause +SELECT * FROM user_events WHERE event_type IN ('login', 'logout', 'signup'); + +-- Combining conditions +SELECT * FROM user_events +WHERE event_type = 'purchase' + AND _ts > '2025-01-01' + AND value > 50; +``` + +### INTERVAL Expressions and Timestamp Arithmetic + +```sql +-- Basic INTERVAL expressions +SELECT INTERVAL '1 hour'; +SELECT INTERVAL '30 minutes'; +SELECT INTERVAL '24 hours'; +SELECT INTERVAL '7 days'; + +-- Timestamp arithmetic with INTERVAL +SELECT NOW() - INTERVAL '1 hour' as one_hour_ago; +SELECT CURRENT_TIMESTAMP - INTERVAL '24 hours' as yesterday; + +-- Using INTERVAL in WHERE clauses for time-based filtering +SELECT * FROM user_events +WHERE _ts >= NOW() - INTERVAL '1 hour'; + +SELECT * FROM user_events +WHERE _ts >= CURRENT_TIMESTAMP - INTERVAL '24 hours' + AND _ts < CURRENT_TIMESTAMP - INTERVAL '1 hour'; + +-- Supported INTERVAL units +SELECT NOW() - INTERVAL '1 second'; +SELECT NOW() - INTERVAL '5 minutes'; +SELECT NOW() - INTERVAL '2 hours'; +SELECT NOW() - INTERVAL '3 days'; +SELECT NOW() - INTERVAL '1 week'; +``` + +### Limitations + +**Not Supported:** +- `ORDER BY` clauses +- `GROUP BY` clauses +- `HAVING` clauses +- `JOIN` operations +- `CREATE TABLE` statements +- `DROP TABLE` statements +- `ALTER TABLE` statements +- Complex aggregations with grouping +- Window functions +- Subqueries + +## Client Examples + +### Python (psycopg2) + +```python +import psycopg2 +import pandas as pd + +# Connect to SeaweedFS SQL server +conn = psycopg2.connect( + host="localhost", + port=5432, + user="admin", + password="secret", + database="default" +) + +# Query data +query = """ +SELECT event_type, user_id, _ts +FROM user_events +WHERE _ts > '2025-01-01' +LIMIT 100 +""" + +# Use pandas for easy data analysis +df = pd.read_sql(query, conn) +print(df) + +# Close connection +conn.close() +``` + +### Java JDBC + +```java +import java.sql.*; + +public class SeaweedFSQuery { + public static void main(String[] args) throws SQLException { + String url = "jdbc:postgresql://localhost:5432/default"; + Connection conn = DriverManager.getConnection(url, "admin", "secret"); + + String query = "SELECT * FROM user_events LIMIT 10"; + Statement stmt = conn.createStatement(); + ResultSet rs = stmt.executeQuery(query); + + while (rs.next()) { + System.out.println("User ID: " + rs.getInt("user_id")); + System.out.println("Event: " + rs.getString("event_type")); + System.out.println("Timestamp: " + rs.getTimestamp("timestamp")); + } + + conn.close(); + } +} +``` + +### Go (lib/pq) + +```go +package main + +import ( + "database/sql" + "fmt" + "log" + _ "github.com/lib/pq" +) + +func main() { + connStr := "host=localhost port=5432 user=admin password=secret dbname=default sslmode=disable" + db, err := sql.Open("postgres", connStr) + if err != nil { + log.Fatal(err) + } + defer db.Close() + + query := "SELECT event_type, user_id, _ts FROM user_events WHERE _ts > '2025-01-01' LIMIT 100" + rows, err := db.Query(query) + if err != nil { + log.Fatal(err) + } + defer rows.Close() + + for rows.Next() { + var eventType string + var userID int + var timestamp string + if err := rows.Scan(&eventType, &userID, ×tamp); err != nil { + log.Fatal(err) + } + fmt.Printf("Event: %s, User: %d, Time: %s\n", eventType, userID, timestamp) + } +} +``` + +### Node.js (pg) + +```javascript +const { Client } = require('pg'); + +const client = new Client({ + host: 'localhost', + port: 5432, + user: 'admin', + password: 'secret', + database: 'default' +}); + +async function queryData() { + await client.connect(); + + const query = ` + SELECT event_type, user_id, timestamp + FROM user_events + WHERE timestamp > $1 + LIMIT 100 + `; + + const result = await client.query(query, ['2024-01-01']); + + console.log('Recent Events:'); + result.rows.forEach(row => { + console.log(`${row.event_type} by user ${row.user_id} at ${row.timestamp}`); + }); + + await client.end(); +} + +queryData().catch(console.error); +``` + +## Data Sources and Architecture + +### Hybrid Message Scanner + +The SQL engine uses a hybrid approach to query both real-time and historical data: + +1. **Live Messages** - Queries unflushed messages directly from MQ brokers +2. **Parquet Files** - Queries archived/flushed messages from Parquet storage +3. **Seamless Integration** - Results are merged to provide complete data view + +### System Columns + +Every topic automatically includes system columns: + +- `_ts` - Message timestamp (formatted timestamp) +- `_key` - Message partition key +- `_source` - Data source identifier (parquet file path or "live") + +### Schema Evolution + +The SQL engine supports backward-compatible schema evolution: +- New columns can be added to existing topics +- Old queries continue to work with new data +- Missing columns return NULL values + +## Performance Optimization + +### Fast Path Aggregations + +The engine optimizes common aggregation queries: + +```sql +-- These queries use optimized fast paths +SELECT COUNT(*) FROM user_events; +SELECT MIN(timestamp) FROM user_events; +SELECT MAX(timestamp) FROM user_events; + +-- Add WHERE clauses for more specific queries +SELECT COUNT(*) FROM user_events WHERE event_type = 'login'; +``` + +### Query Best Practices + +1. **Use time-based filtering** for large datasets: + ```sql + SELECT * FROM events WHERE _ts >= '2025-01-01' AND _ts < '2025-02-01'; + ``` + +2. **Limit result sets** for exploration: + ```sql + SELECT * FROM events WHERE _ts > '2025-01-01' LIMIT 1000; + ``` + +3. **Use appropriate indexes** on frequently queried columns (when supported) + +4. **Leverage system columns** for debugging: + ```sql + SELECT _source, _ts FROM events LIMIT 100; + ``` + +## BI Tool Integration + +### Apache Superset + +```python +# Database URI for Apache Superset +postgresql://admin:secret@localhost:5432/default +``` + +### Grafana + +```yaml +# Grafana datasource configuration +apiVersion: 1 +datasources: + - name: SeaweedFS + type: postgres + url: localhost:5432 + database: default + user: admin + password: secret + sslmode: disable +``` + +### Tableau + +Use the PostgreSQL connector with: +- Server: `localhost` +- Port: `5432` +- Database: `default` +- Username: `admin` +- Password: `secret` + +## Troubleshooting + +### Common Issues + +1. **"Database server not running"** + ```bash + # Check if weed db is running + ps aux | grep "weed db" + + # Check port availability + netstat -ln | grep 5432 + ``` + +2. **"Authentication failed"** + ```bash + # Verify user credentials + cat users.json + + # Test with trust authentication + weed db -auth=trust + ``` + +3. **"No data returned"** + ```sql + -- Check if topics exist + SHOW TABLES; + + -- Check data sources + SELECT _source FROM topic_name LIMIT 100; + + -- Check timestamp ranges + SELECT MIN(timestamp), MAX(timestamp) FROM topic_name; + ``` + +4. **"Connection timeout"** + ```bash + # Increase timeout settings + weed db -idle-timeout=24h -max-connections=50 + + # Check network connectivity + telnet localhost 5432 + ``` + +### Debug Mode + +Enable debug logging for troubleshooting: + +```bash +# Set debug mode +export GLOG_v=2 + +# Start with verbose logging +weed db -auth=md5 -users="@users.json" -v=2 +``` + +### Performance Issues + +1. **Slow queries on large datasets**: + - Add time-based WHERE clauses + - Use LIMIT for exploration + - Consider data partitioning strategies + +2. **High memory usage**: + - Reduce concurrent connections + - Limit result set sizes + - Monitor broker memory usage + +3. **Network timeouts**: + - Increase idle timeout settings + - Check network stability between components + - Use connection pooling in applications + +## Production Deployment + +### Recommended Setup + +```bash +# Production database server +weed db \ + -auth=md5 \ + -users="@/etc/seaweedfs/users.json" \ + -host=0.0.0.0 \ + -port=5432 \ + -master=master1:9333,master2:9333,master3:9333 \ + -max-connections=100 \ + -idle-timeout=1h \ + -tls-cert=/etc/ssl/seaweedfs-db.crt \ + -tls-key=/etc/ssl/seaweedfs-db.key +``` + +### High Availability + +- Run multiple `weed db` instances behind a load balancer +- Use master server failover configuration +- Monitor connection health and query performance +- Implement connection pooling in client applications + +### Security Checklist + +- [ ] Use MD5 or password authentication (never trust in production) +- [ ] Enable TLS encryption for data in transit +- [ ] Store credentials in secure files with proper permissions +- [ ] Limit database server network access with firewalls +- [ ] Monitor authentication attempts and query patterns +- [ ] Regular security updates and credential rotation + +## Related Documentation + +- [Seaweed Message Queue](Seaweed-Message-Queue.md) +- [Security Configuration](Security-Configuration.md) +- [Production Setup](Production-Setup.md) +- [System Metrics](System-Metrics.md) diff --git a/SQL-Quick-Reference.md b/SQL-Quick-Reference.md new file mode 100644 index 0000000..12b2f62 --- /dev/null +++ b/SQL-Quick-Reference.md @@ -0,0 +1,195 @@ +# SQL Quick Reference + +Quick reference guide for SeaweedFS SQL queries on Message Queue topics. + +## Commands + +### Start Database Server +```bash +# Basic (development) +weed db + +# Production with MD5 auth +weed db -auth=md5 -users='{"admin":"secret"}' -host=0.0.0.0 + +# With TLS encryption +weed db -auth=md5 -users="@users.json" -tls-cert=server.crt -tls-key=server.key +``` + +### Interactive CLI +```bash +# Start SQL shell +weed sql + +# Execute single query +weed sql -exec="SHOW TABLES" +``` + +## Authentication + +### Credential Formats +```bash +# JSON inline +-users='{"user1":"pass1","user2":"pass2"}' + +# JSON file +-users="@/path/to/users.json" +``` + +### Auth Methods +- `trust` - No auth (dev only) +- `md5` - Hash + salt (recommended) +- `password` - Clear text (TLS required) + +## Client Connections + +### psql +```bash +# Basic connection +psql -h localhost -p 5432 -U admin -d default + +# With password +PGPASSWORD=secret psql -h localhost -p 5432 -U admin -d default + +# Connection string +psql "postgresql://admin:secret@localhost:5432/default" +``` + +### Programming Languages +```python +# Python +import psycopg2 +conn = psycopg2.connect(host="localhost", port=5432, user="admin", password="secret", database="default") +``` + +```java +// Java +String url = "jdbc:postgresql://localhost:5432/default"; +Connection conn = DriverManager.getConnection(url, "admin", "secret"); +``` + +```go +// Go +db, err := sql.Open("postgres", "host=localhost port=5432 user=admin password=secret dbname=default sslmode=disable") +``` + +## SQL Operations + +### Schema Commands +```sql +SHOW DATABASES; -- List MQ namespaces +USE namespace_name; -- Switch database +SHOW TABLES; -- List MQ topics +DESCRIBE table_name; -- Show table schema +-- Note: CREATE TABLE, DROP TABLE and ALTER TABLE not supported +``` + +### Query Commands +```sql +-- Basic queries +SELECT * FROM events LIMIT 10; +SELECT * FROM events WHERE _ts > '2025-01-01'; +SELECT * FROM events WHERE status IN ('active', 'completed'); +SELECT COUNT(*) FROM events; + +-- System columns (available on all tables) +SELECT _ts, _key, _source, * FROM events; + +-- Aggregations (optimized) +SELECT COUNT(*) FROM events; +SELECT MIN(timestamp), MAX(timestamp) FROM events; +``` + +### Time-based Queries +```sql +-- Time filtering with system timestamp column (_ts) +-- Automatic string-to-timestamp conversion for _ts system column +SELECT * FROM events +WHERE _ts >= '2025-01-01' + AND _ts < '2025-02-01' +LIMIT 1000; + +-- INTERVAL expressions and timestamp arithmetic +SELECT NOW() - INTERVAL '1 hour'; +SELECT * FROM events WHERE _ts >= NOW() - INTERVAL '24 hours'; +SELECT * FROM events WHERE _ts >= CURRENT_TIMESTAMP - INTERVAL '1 day'; + +-- BETWEEN clauses with INTERVAL arithmetic +SELECT * FROM events WHERE _ts BETWEEN NOW() - INTERVAL '1 week' AND NOW(); +SELECT * FROM events WHERE _ts BETWEEN '2025-01-01' AND '2025-12-31'; + +-- Current time functions +SELECT NOW(), CURRENT_TIMESTAMP, CURRENT_DATE FROM events LIMIT 1; +``` + +## System Columns + +Every topic includes these system columns: +- `_ts` - Message timestamp (formatted timestamp, supports automatic string-to-timestamp conversion in WHERE clauses) +- `_key` - Message partition key +- `_source` - Data source ("live" or parquet file path) + +**Note:** The `_ts` column supports automatic parsing of timestamp strings in WHERE clauses. Formats supported: `'2025-01-01'`, `'2025-01-01T15:30:00Z'`, `'2025-01-01 15:30:00'`, etc. + +## Troubleshooting + +### Check Status +```sql +-- Verify tables exist +SHOW TABLES; + +-- Check data sources +SELECT _source FROM table_name LIMIT 100; + +-- Verify time range +SELECT MIN(timestamp), MAX(timestamp) FROM table_name; +``` + +### Common Issues +1. **No data**: Check `SHOW TABLES` and topic names +2. **Auth failed**: Verify credentials in users file +3. **Timeouts**: Increase `-idle-timeout` setting +4. **Slow queries**: Add WHERE clauses and LIMIT + +### Debug Mode +```bash +# Enable verbose logging +export GLOG_v=2 +weed db -v=2 ... +``` + +## Performance Tips + +1. **Filter by time** for large datasets: + ```sql + WHERE _ts >= '2025-01-01' AND _ts < '2025-02-01' + ``` + +2. **Use LIMIT** for exploration: + ```sql + SELECT * FROM events WHERE _ts > '2025-01-01' LIMIT 1000 + ``` + +3. **Fast aggregations** (basic functions only): + ```sql + SELECT COUNT(*) FROM events; -- Optimized + SELECT MIN(timestamp), MAX(timestamp) FROM events; -- Optimized + ``` + +4. **Check data sources**: + ```sql + SELECT _source, _ts FROM events LIMIT 100; + ``` + +## Limitations + +**Not Supported:** +- `ORDER BY`, `GROUP BY`, `HAVING` clauses +- `JOIN` operations between tables +- `CREATE TABLE`, `DROP TABLE`, `ALTER TABLE` statements +- Window functions and subqueries +- Complex aggregations with grouping + +## Links + +For detailed documentation, see [[SQL Queries on Message Queue]]. diff --git a/_Sidebar.md b/_Sidebar.md index 5aa94e3..609f078 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -115,6 +115,8 @@ ### Messaging * [[Seaweed Message Queue]] +* [[SQL Queries on Message Queue]] +* [[SQL Quick Reference]] ### Use Cases * [[Use Cases]] From 51574f24369363d004131672ae77daa62c4ea97a Mon Sep 17 00:00:00 2001 From: chrislusf Date: Mon, 8 Sep 2025 02:05:25 -0700 Subject: [PATCH 123/149] support "is null" "is not null" --- SQL-Quick-Reference.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/SQL-Quick-Reference.md b/SQL-Quick-Reference.md index 12b2f62..2b0bcef 100644 --- a/SQL-Quick-Reference.md +++ b/SQL-Quick-Reference.md @@ -92,6 +92,11 @@ SELECT * FROM events WHERE _ts > '2025-01-01'; SELECT * FROM events WHERE status IN ('active', 'completed'); SELECT COUNT(*) FROM events; +-- NULL checking operations +SELECT * FROM events WHERE status IS NULL; +SELECT * FROM events WHERE status IS NOT NULL; +SELECT * FROM events WHERE user_id IS NOT NULL AND status = 'active'; + -- System columns (available on all tables) SELECT _ts, _key, _source, * FROM events; @@ -131,6 +136,33 @@ Every topic includes these system columns: **Note:** The `_ts` column supports automatic parsing of timestamp strings in WHERE clauses. Formats supported: `'2025-01-01'`, `'2025-01-01T15:30:00Z'`, `'2025-01-01 15:30:00'`, etc. +## NULL Value Handling + +### NULL Checking Operations +```sql +-- Check for NULL values +SELECT * FROM events WHERE description IS NULL; + +-- Check for non-NULL values +SELECT * FROM events WHERE user_id IS NOT NULL; + +-- Combine with other conditions +SELECT * FROM events +WHERE user_id IS NOT NULL + AND status = 'active' + AND _ts >= '2025-01-01'; + +-- Filter out records with missing data +SELECT * FROM events WHERE user_id IS NOT NULL AND description IS NOT NULL; +``` + +### NULL Value Semantics +- **Empty strings** are treated as valid values (not NULL) +- **Missing fields** in records are considered NULL +- **Boolean, numeric, and timestamp values** are never NULL once present +- **Bytes values** are treated as non-NULL even if empty +- NULL values are excluded from aggregate functions like `COUNT(column_name)` + ## Troubleshooting ### Check Status @@ -150,6 +182,7 @@ SELECT MIN(timestamp), MAX(timestamp) FROM table_name; 2. **Auth failed**: Verify credentials in users file 3. **Timeouts**: Increase `-idle-timeout` setting 4. **Slow queries**: Add WHERE clauses and LIMIT +5. **NULL filtering**: Use `IS NULL` / `IS NOT NULL` instead of `= NULL` / `!= NULL` ### Debug Mode ```bash @@ -181,6 +214,11 @@ weed db -v=2 ... SELECT _source, _ts FROM events LIMIT 100; ``` +5. **Filter NULL values for cleaner results**: + ```sql + SELECT * FROM events WHERE user_id IS NOT NULL LIMIT 100; + ``` + ## Limitations **Not Supported:** From d231a0f9fdd73e7b4f70682c55b347045d90ed74 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Thu, 11 Sep 2025 10:06:23 -0700 Subject: [PATCH 124/149] add keycloak integration --- Keycloak-Integration.md | 187 ++++++++++++++++++++++++++++++++++++++++ _Sidebar.md | 1 + 2 files changed, 188 insertions(+) create mode 100644 Keycloak-Integration.md diff --git a/Keycloak-Integration.md b/Keycloak-Integration.md new file mode 100644 index 0000000..645375e --- /dev/null +++ b/Keycloak-Integration.md @@ -0,0 +1,187 @@ +# Keycloak Integration (OIDC) with SeaweedFS S3 Gateway + +This guide shows how to integrate Keycloak (OpenID Connect) with SeaweedFS S3 Gateway using the advanced IAM and STS configuration. It supports both: + +- Direct OIDC authentication to S3 with Bearer tokens +- OIDC to STS role assumption using trust policies and role mapping + +## Prerequisites + +- A running Keycloak server and a realm (e.g. `seaweedfs`) +- A Keycloak client (e.g. `seaweedfs-s3`) created in that realm +- SeaweedFS with advanced IAM enabled via `-iam.config` + +## Step 1: Configure Keycloak + +1) Create a client +- Client ID: `seaweedfs-s3` +- Access Type: public or confidential (confidential requires a client secret) +- Standard Flow: enabled (for browser login flows) + +2) Add role/group claims to tokens +- Add a mapper of type "Group Membership" or "User Realm Role" that puts roles/groups into a top-level claim: + - Claim name: `groups` (recommended) or `roles` + - Add to ID token: true + - Add to Access token: true + +3) Confirm OIDC discovery +- Open your realm discovery document and note the issuer and certs endpoints (Keycloak Quarkus defaults): + - Issuer: `https://KEYCLOAK/realms/` + - JWKS (certs): `https://KEYCLOAK/realms//protocol/openid-connect/certs` + - UserInfo: `https://KEYCLOAK/realms//protocol/openid-connect/userinfo` + +Note: For older Keycloak distributions, issuer may include `/auth` in the path. + +## Step 2: Prepare SeaweedFS IAM config + +Create an IAM configuration JSON file (e.g. `/etc/seaweed/iam_keycloak.json`) and reference it with `weed s3 -iam.config=...`. + +Minimal example with Keycloak OIDC provider, role mapping, roles, and policies: + +```json +{ + "sts": { + "tokenDuration": "1h", + "maxSessionLength": "12h", + "issuer": "seaweedfs-sts", + "signingKey": "c2Vhd2VlZGZzLXNpZ25pbmcta2V5LTMyLWNoYXJzLWxvbmc=" + }, + "providers": [ + { + "name": "keycloak", + "type": "oidc", + "enabled": true, + "config": { + "issuer": "https://KEYCLOAK/realms/seaweedfs", + "clientId": "seaweedfs-s3", + "clientSecret": "", + "jwksUri": "https://KEYCLOAK/realms/seaweedfs/protocol/openid-connect/certs", + "userInfoUri": "https://KEYCLOAK/realms/seaweedfs/protocol/openid-connect/userinfo", + "scopes": ["openid", "profile", "email", "roles", "groups"], + "roleMapping": { + "rules": [ + { "claim": "groups", "value": "admins", "role": "arn:seaweed:iam::role/S3AdminRole" }, + { "claim": "groups", "value": "developers", "role": "arn:seaweed:iam::role/S3WriteRole" } + ], + "defaultRole": "arn:seaweed:iam::role/S3ReadOnlyRole" + } + } + } + ], + "policies": [ + { + "name": "S3ReadOnlyPolicy", + "document": { + "Version": "2012-10-17", + "Statement": [ + { "Effect": "Allow", "Action": ["s3:List*", "s3:Get*"], "Resource": ["*"] } + ] + } + }, + { + "name": "S3WritePolicy", + "document": { + "Version": "2012-10-17", + "Statement": [ + { "Effect": "Allow", "Action": ["s3:List*", "s3:Get*", "s3:Put*", "s3:DeleteObject"], "Resource": ["*"] } + ] + } + }, + { + "name": "S3AdminPolicy", + "document": { + "Version": "2012-10-17", + "Statement": [ + { "Effect": "Allow", "Action": ["s3:*"] , "Resource": ["*"] } + ] + } + } + ], + "roles": [ + { + "roleName": "S3ReadOnlyRole", + "roleArn": "arn:seaweed:iam::role/S3ReadOnlyRole", + "attachedPolicies": ["S3ReadOnlyPolicy"], + "trustPolicy": { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { "Federated": "*" }, + "Action": ["sts:AssumeRoleWithWebIdentity"], + "Condition": { + "StringEquals": { "seaweed:Issuer": "https://KEYCLOAK/realms/seaweedfs" } + } + } + ] + } + }, + { + "roleName": "S3WriteRole", + "roleArn": "arn:seaweed:iam::role/S3WriteRole", + "attachedPolicies": ["S3WritePolicy"], + "trustPolicy": { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { "Federated": "*" }, + "Action": ["sts:AssumeRoleWithWebIdentity"], + "Condition": { + "StringEquals": { "seaweed:Issuer": "https://KEYCLOAK/realms/seaweedfs" } + } + } + ] + } + }, + { + "roleName": "S3AdminRole", + "roleArn": "arn:seaweed:iam::role/S3AdminRole", + "attachedPolicies": ["S3AdminPolicy"], + "trustPolicy": { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { "Federated": "*" }, + "Action": ["sts:AssumeRoleWithWebIdentity"], + "Condition": { + "StringEquals": { "seaweed:Issuer": "https://KEYCLOAK/realms/seaweedfs" } + } + } + ] + } + } + ] +} +``` + +Notes: +- Set `signingKey` to a strong random secret (base64-encoded 32+ bytes). All S3 gateway instances must share the same STS `issuer` and `signingKey`. +- Explicit `jwksUri` and `userInfoUri` are recommended for Keycloak. +- Ensure your Keycloak mappers populate a top-level `groups` (or `roles`) claim. + +## Step 3: Start the S3 Gateway + +```bash +weed s3 -filer=filer:8888 -port=8333 -iam.config=/etc/seaweed/iam_keycloak.json +``` + +For multi-instance deployments, use the same IAM config on each instance. + +## Using It + +- Direct OIDC to S3 (Bearer): obtain a Keycloak access or ID token for client `seaweedfs-s3` and call S3 with: + +```bash +curl -H "Authorization: Bearer $KEYCLOAK_TOKEN" http://s3-gateway:8333/ +``` + +- Role selection: SeaweedFS maps OIDC claims via `roleMapping`. With the example above, users in `admins` get `S3AdminRole`, `developers` get `S3WriteRole`, others default to `S3ReadOnlyRole`. + +## Troubleshooting + +- Invalid token: verify token `iss` equals the configured provider `issuer` and `aud` or `azp` equals the client ID. +- JWKS errors: ensure `jwksUri` is reachable from the S3 gateway. For Keycloak, use the `.../protocol/openid-connect/certs` endpoint. +- No roles applied: confirm Keycloak mapper emits `groups` (or `roles`) at top-level in the token, and adjust `roleMapping` rules accordingly. +- Trust policy denied: ensure `seaweed:Issuer` in the trust policy matches your Keycloak realm issuer exactly. diff --git a/_Sidebar.md b/_Sidebar.md index 609f078..9d0656e 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -91,6 +91,7 @@ ### AWS IAM * [[Amazon IAM API]] * [[AWS IAM CLI]] +* [[Keycloak Integration]] ### Machine Learning * [[TensorFlow with SeaweedFS]] From 15ca722eaaa88362a2f125a15e38f323a77d07df Mon Sep 17 00:00:00 2001 From: chrislusf Date: Thu, 11 Sep 2025 11:34:44 -0700 Subject: [PATCH 125/149] update for SSE --- Server-Side-Encryption-SSE-KMS.md | 34 +++++++++++++++++++++++-------- Server-Side-Encryption.md | 28 +++++++++++++++++-------- 2 files changed, 46 insertions(+), 16 deletions(-) diff --git a/Server-Side-Encryption-SSE-KMS.md b/Server-Side-Encryption-SSE-KMS.md index 9d49b45..3576867 100644 --- a/Server-Side-Encryption-SSE-KMS.md +++ b/Server-Side-Encryption-SSE-KMS.md @@ -1,6 +1,7 @@ # SSE-KMS: Server-Side Encryption with Key Management Service -SeaweedFS SSE-KMS supports multiple external Key Management Service providers for enterprise-grade encryption key management. This guide covers setup and configuration for AWS KMS, Google Cloud KMS, and OpenBao/Vault. +SeaweedFS SSE-KMS supports multiple external Key Management Service providers for enterprise-grade encryption key management. This guide covers setup and configuration for AWS KMS, Google Cloud KMS, OpenBao/Vault. +and Azure Key Vault (experimental, build tag `azurekms`). ## Supported KMS Providers @@ -9,7 +10,7 @@ SeaweedFS SSE-KMS supports multiple external Key Management Service providers fo | **AWS KMS** | Full support | AWS-centric deployments | | **Google Cloud KMS** | Full support | GCP-centric deployments | | **OpenBao/Vault** | Full support | Hybrid/on-premises environments | -| **Azure Key Vault** | Planned | Future Azure deployments | +| **Azure Key Vault** | Experimental (build tag `azurekms`) | Azure-centric deployments | ## Quick Start Guide @@ -208,14 +209,31 @@ aws s3 cp file.txt s3://mybucket/file.txt \ --- -## Azure Key Vault Integration +## Azure Key Vault Integration (Experimental) -**Status**: Planned (not yet implemented) +Azure Key Vault support exists behind the build tag `azurekms` and is considered experimental. To enable it, build SeaweedFS with the tag and configure the provider: -Azure Key Vault integration is planned for future releases. When available, it will support: -- Azure AD authentication -- HSM-backed keys -- Integration with Azure Monitor +```bash +# Build with Azure KMS support (example) +go build -tags azurekms ./weed +``` + +```json +{ + "kms": { + "providers": { + "azure": { + "type": "azure", + "vault_url": "https://.vault.azure.net/", + "tenant_id": "", + "client_id": "", + "client_secret": "", + "use_default_creds": false + } + } + } +} +``` --- diff --git a/Server-Side-Encryption.md b/Server-Side-Encryption.md index 6e6368e..9c07c98 100644 --- a/Server-Side-Encryption.md +++ b/Server-Side-Encryption.md @@ -1,6 +1,6 @@ # Server-Side Encryption (SSE) -SeaweedFS supports AWS S3-compatible server-side encryption to protect data at rest. Two encryption methods are available, each offering different levels of control over encryption keys and management. +SeaweedFS supports AWS S3-compatible server-side encryption to protect data at rest. Three encryption methods are available, each offering different levels of control over encryption keys and management. ## Overview @@ -8,13 +8,14 @@ SeaweedFS supports AWS S3-compatible server-side encryption to protect data at r |------------------|----------------|----------| | **SSE-KMS** | External KMS providers | Enterprise key management, audit trails | | **SSE-C** | Customer-provided | Full customer control, regulatory compliance | +| **SSE-S3** | SeaweedFS-managed | Simple server-managed encryption, bucket defaults | ## Encryption Types ### SSE-KMS (Server-Side Encryption with Key Management Service) - **Best for**: Enterprise environments with external KMS infrastructure -- **Key management**: External providers (AWS KMS, Google Cloud KMS, OpenBao/Vault) -- **Features**: Centralized key management, detailed audit logs, per-bucket key assignment +- **Key management**: External providers (AWS KMS, Google Cloud KMS, OpenBao/Vault, Azure Key Vault [experimental]) +- **Features**: Centralized key management, detailed audit logs, per-bucket key assignment, optional Bucket Key optimization - **Configuration**: Requires KMS provider setup in S3 config - **Documentation**: [SSE-KMS Guide](Server-Side-Encryption-SSE-KMS.md) @@ -25,6 +26,12 @@ SeaweedFS supports AWS S3-compatible server-side encryption to protect data at r - **Configuration**: Keys provided via HTTP headers - **Documentation**: [SSE-C Guide](Server-Side-Encryption-SSE-C.md) +### SSE-S3 (Server-Managed Keys) +- **Best for**: Simple, fully managed encryption with minimal configuration +- **Key management**: SeaweedFS-managed keys with automatic key handling +- **Features**: Works with explicit `x-amz-server-side-encryption: AES256` and bucket default encryption; supports multipart uploads and range requests +- **Configuration**: Optional bucket-level default encryption via standard S3 bucket encryption API + ## Quick Start ### SSE-KMS (Enterprise) @@ -47,6 +54,13 @@ aws s3 cp file.txt s3://mybucket/file.txt \ --sse-c-key fileb://customer-key.bin ``` +### SSE-S3 (Server-Managed) +```bash +# Explicit SSE-S3 on upload (or configure bucket default encryption) +aws s3 cp file.txt s3://mybucket/file.txt \ + --server-side-encryption AES256 +``` + ## Configuration ### Basic Setup @@ -90,7 +104,7 @@ weed s3 -config=s3_kms_config.json - Multipart uploads with consistent encryption - Cross-encryption copy operations - Object metadata preservation -- Range requests (SSE-KMS only; SSE-C does not support range requests) +- Range requests for SSE-C, SSE-KMS, and SSE-S3 **AWS S3 Compatibility:** - Identical API behavior and headers @@ -99,10 +113,8 @@ weed s3 -config=s3_kms_config.json ## Implementation Notes -- **SSE-KMS**: Supports AWS KMS, Google Cloud KMS, OpenBao/Vault +- **SSE-KMS**: Supports AWS KMS, Google Cloud KMS, OpenBao/Vault; Azure Key Vault is available behind the `azurekms` build tag (experimental) - **SSE-C**: Full support with security best practices -- **Azure Key Vault**: Not yet implemented (planned) - -**Note**: SSE-S3 (server-managed keys) is currently in development and not yet production-ready. +- **SSE-S3**: Supported with SeaweedFS-managed keys and bucket default encryption For detailed setup guides, examples, and configuration options, see the individual encryption method documentation linked above. \ No newline at end of file From 5a9c70b86f715c3e83bf06ebfe8a616405205d67 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Thu, 11 Sep 2025 12:31:47 -0700 Subject: [PATCH 126/149] update s3 related --- AWS-CLI-with-SeaweedFS.md | 28 +++++++++++++++++++++++++++- Amazon-S3-API.md | 25 +++++++++++++++++-------- Docker-Compose-for-S3.md | 7 ++++++- S3-API-FAQ.md | 14 +++++++++++++- 4 files changed, 63 insertions(+), 11 deletions(-) diff --git a/AWS-CLI-with-SeaweedFS.md b/AWS-CLI-with-SeaweedFS.md index 548c299..77ff515 100644 --- a/AWS-CLI-with-SeaweedFS.md +++ b/AWS-CLI-with-SeaweedFS.md @@ -81,5 +81,31 @@ http://localhost:8333/newbucket/t.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Cre # access the url $ curl "http://localhost:8333/newbucket/t.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=some_access_key1%2F20200726%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200726T161749Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=e0cc153209e414ca8168661f57827aa03ab84e7041ef9270ff639bcc519d24f5" - ``` + +## Server-Side Encryption with AWS CLI + +### SSE-KMS +```bash +aws --endpoint-url http://localhost:8333 s3 cp file.txt s3://bucket/kms.txt \ + --sse aws:kms \ + --sse-kms-key-id "test-key-123" +``` + +### SSE-C +```bash +# Generate a 256-bit key +openssl rand -base64 32 > key.b64 +aws --endpoint-url http://localhost:8333 s3 cp file.txt s3://bucket/ssec.txt \ + --sse-c AES256 \ + --sse-c-key fileb://key.b64 +``` + +### SSE-S3 (Server-managed) +```bash +aws --endpoint-url http://localhost:8333 s3 cp file.txt s3://bucket/sse-s3.txt \ + --sse AES256 +``` + +## OIDC/JWT to S3 +For Keycloak and other OIDC providers, you can obtain a JWT and access S3 directly (or use STS to assume a role). See [[Keycloak Integration]]. diff --git a/Amazon-S3-API.md b/Amazon-S3-API.md index c8ad299..607ba11 100644 --- a/Amazon-S3-API.md +++ b/Amazon-S3-API.md @@ -56,11 +56,12 @@ To be sure, you can look at the function defined in the files `weed/s3api/s3api_ * DeleteObjectTagging // Server-Side Encryption (NEW) -* PutObject (with SSE-KMS, SSE-C) +* PutObject (with SSE-KMS, SSE-C, SSE-S3) * GetObject (with automatic decryption) * HeadObject (with encryption metadata) * CopyObject (with encryption/decryption) * Multipart uploads with encryption +* Bucket default encryption // Conditional Operations (NEW) * All object operations support conditional headers: @@ -123,12 +124,14 @@ Not included: | allows more than "/" as a delimiter | No | Yes | | Object Versioning | Yes | Yes | | MFA Delete for versioning | No | Yes | -| Server-Side Encryption (SSE-KMS) | Yes | Yes | -| Server-Side Encryption (SSE-C) | Yes | Yes | -| KMS Providers (Multi-cloud) | Yes | No | +| Server-Side Encryption (SSE-KMS) | Yes | Yes | +| Server-Side Encryption (SSE-C) | Yes | Yes | +| Server-Side Encryption (SSE-S3) | Yes | Yes | +| KMS Providers (Multi-cloud) | Yes | No | | Conditional Headers (All operations) | Yes | Yes | -| Range requests with SSE-KMS | Yes | Yes | -| Range requests with SSE-C | No | No | +| Range requests with SSE-KMS | Yes | Yes | +| Range requests with SSE-C | Yes | Yes | +| Range requests with SSE-S3 | Yes | Yes | ## Empty folders @@ -138,10 +141,11 @@ To be consistent with AWS S3, SeaweedFS tries to check whether the folder is emp # Server-Side Encryption -SeaweedFS supports AWS S3-compatible server-side encryption to protect your data at rest. Two encryption methods are available: +SeaweedFS supports AWS S3-compatible server-side encryption to protect your data at rest. Three encryption methods are available: -- **[SSE-KMS](Server-Side-Encryption-SSE-KMS)**: External Key Management Service providers (AWS KMS, Google Cloud KMS, OpenBao/Vault) +- **[SSE-KMS](Server-Side-Encryption-SSE-KMS)**: External Key Management Service providers (AWS KMS, Google Cloud KMS, OpenBao/Vault) - **[SSE-C](Server-Side-Encryption-SSE-C)**: Customer-provided encryption keys (maximum customer control) +- **SSE-S3**: SeaweedFS-managed server-side encryption (explicit `AES256` header or bucket default encryption) All encryption types support: - Automatic encryption/decryption @@ -153,6 +157,7 @@ All encryption types support: For detailed setup guides and examples, see: - **[Server-Side Encryption Overview](Server-Side-Encryption)** - **[SSE-KMS Guide](Server-Side-Encryption-SSE-KMS)** +- **[SSE-C Guide](Server-Side-Encryption-SSE-C)** ## Quick Examples @@ -162,6 +167,9 @@ aws s3 cp file.txt s3://mybucket/kms-encrypted.txt --server-side-encryption aws: # SSE-C (Customer-provided keys) aws s3 cp file.txt s3://mybucket/customer-encrypted.txt --sse-c AES256 --sse-c-key fileb://my-key.bin + +# SSE-S3 (Server-managed) +aws s3 cp file.txt s3://mybucket/server-encrypted.txt --server-side-encryption AES256 ``` # S3 Conditional Operations @@ -203,6 +211,7 @@ curl -X PUT -H "If-None-Match: *" -d "new content" "http://localhost:8333/mybuck By default, the access key and secret key to access `weed s3` is not authenticated. To enable credential based access, you can choose static or dynamic configuration: * **Dynamic Configuration**: setup auth with `s3.configure` in `weed shell` * **Static Configuration**: create a config.json file similar to the example below, and specify it via `weed s3 -config=config.json` +* **OIDC/JWT (Web Identity)**: for Keycloak and other OpenID providers, see [[Keycloak Integration]] for STS and JWT to S3 usage ## Dynamic Configuration diff --git a/Docker-Compose-for-S3.md b/Docker-Compose-for-S3.md index 387f4d8..5f36a64 100644 --- a/Docker-Compose-for-S3.md +++ b/Docker-Compose-for-S3.md @@ -39,4 +39,9 @@ services: }' > /etc/seaweedfs/config.json && \ weed server -s3 -s3.config /etc/seaweedfs/config.json" restart: unless-stopped -``` \ No newline at end of file +``` + +## Notes +- To enable advanced IAM (STS, OIDC providers), mount a JSON and add `-iam.config=/etc/seaweedfs/iam.json` to the S3 command. See [[Keycloak Integration]]. +- To use SSE-KMS (with OpenBao/Vault, AWS KMS, GCP KMS), mount KMS config (e.g. `s3_kms.json`) and start with `-config=/etc/seaweedfs/s3_kms.json`. See [[Server-Side-Encryption-SSE-KMS]]. +- SSE-S3 and bucket default encryption work without external KMS; see [[Server-Side-Encryption]]. \ No newline at end of file diff --git a/S3-API-FAQ.md b/S3-API-FAQ.md index 557b764..10db03c 100644 --- a/S3-API-FAQ.md +++ b/S3-API-FAQ.md @@ -53,7 +53,6 @@ rclone lsf seaweedfs:my-bucket/dir If the directory `dir` exists in `my-bucket`, the orphaned metadata will be cleaned up. Note that due to slight API usage differences, `rclone ls` does not trigger cleanup, but `rclone lsf` will. - ## Setting TTL It is possible to set a TTL for a specific directory using the S3 API. They are set using [`PutBucketLifecycleConfiguration`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html). @@ -112,6 +111,19 @@ To manage storage growth, you should: - Implement lifecycle policies to automatically clean up old versions - Use version-specific deletions for permanent removal when needed +## Does SeaweedFS support encrypted range requests? + +Yes. Range requests are supported for encrypted objects across all SSE modes: +- **SSE-KMS**: Supported +- **SSE-C**: Supported +- **SSE-S3**: Supported + +## Does SeaweedFS support bucket default encryption? + +Yes. Bucket default encryption is supported. You can configure bucket-level default encryption using the standard S3 bucket encryption API, and uploads without explicit encryption headers will be encrypted according to the bucket policy. Applies to SSE-KMS and SSE-S3. + +For setup guides, see [[Server-Side-Encryption]]. + ## Does SeaweedFS support S3 Object Lock? Yes! SeaweedFS provides comprehensive support for S3 Object Lock features, including: From a732d9c1ba88816cdf716205f8a09d1e42593b79 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Mon, 15 Sep 2025 21:19:52 -0700 Subject: [PATCH 127/149] more --- Amazon-S3-API.md | 8 ++++---- S3-API-FAQ.md | 4 ++-- Server-Side-Encryption-SSE-C.md | 6 +++--- Server-Side-Encryption-SSE-KMS.md | 5 ++--- Server-Side-Encryption.md | 30 ++++++++++++++++-------------- 5 files changed, 27 insertions(+), 26 deletions(-) diff --git a/Amazon-S3-API.md b/Amazon-S3-API.md index 607ba11..7bb1efc 100644 --- a/Amazon-S3-API.md +++ b/Amazon-S3-API.md @@ -141,11 +141,11 @@ To be consistent with AWS S3, SeaweedFS tries to check whether the folder is emp # Server-Side Encryption -SeaweedFS supports AWS S3-compatible server-side encryption to protect your data at rest. Three encryption methods are available: +Need encryption at rest? SeaweedFS speaks the same SSE dialects as Amazon S3, so your existing tools and SDKs just work. You can choose from three options: -- **[SSE-KMS](Server-Side-Encryption-SSE-KMS)**: External Key Management Service providers (AWS KMS, Google Cloud KMS, OpenBao/Vault) -- **[SSE-C](Server-Side-Encryption-SSE-C)**: Customer-provided encryption keys (maximum customer control) -- **SSE-S3**: SeaweedFS-managed server-side encryption (explicit `AES256` header or bucket default encryption) +- **[SSE-KMS](Server-Side-Encryption-SSE-KMS)**: Use an external KMS (AWS KMS, Google Cloud KMS, OpenBao/Vault) +- **[SSE-C](Server-Side-Encryption-SSE-C)**: Bring your own keys for maximum control +- **SSE-S3**: Let SeaweedFS manage keys (explicit `AES256` header or bucket default encryption) All encryption types support: - Automatic encryption/decryption diff --git a/S3-API-FAQ.md b/S3-API-FAQ.md index 10db03c..9ccd77b 100644 --- a/S3-API-FAQ.md +++ b/S3-API-FAQ.md @@ -113,14 +113,14 @@ To manage storage growth, you should: ## Does SeaweedFS support encrypted range requests? -Yes. Range requests are supported for encrypted objects across all SSE modes: +Yes. Range requests work just fine with encrypted objects across all SSE modes: - **SSE-KMS**: Supported - **SSE-C**: Supported - **SSE-S3**: Supported ## Does SeaweedFS support bucket default encryption? -Yes. Bucket default encryption is supported. You can configure bucket-level default encryption using the standard S3 bucket encryption API, and uploads without explicit encryption headers will be encrypted according to the bucket policy. Applies to SSE-KMS and SSE-S3. +Yes. You can set a bucket-level default encryption policy using the standard S3 bucket encryption API. Uploads without explicit encryption headers will follow the bucket policy. This applies to SSE-KMS and SSE-S3. For setup guides, see [[Server-Side-Encryption]]. diff --git a/Server-Side-Encryption-SSE-C.md b/Server-Side-Encryption-SSE-C.md index 429ea8c..60f2fd1 100644 --- a/Server-Side-Encryption-SSE-C.md +++ b/Server-Side-Encryption-SSE-C.md @@ -1,10 +1,10 @@ # Server-Side Encryption with Customer-provided Keys (SSE-C) -SeaweedFS supports AWS S3-compatible Server-Side Encryption with Customer-provided Keys (SSE-C), allowing clients to encrypt objects using their own encryption keys. +With SSE-C, you bring your own keys and SeaweedFS does the heavy lifting. Your app sends the key with each request, and we encrypt/decrypt on the server side—without ever storing your key. ## Overview -SSE-C enables client-side key management with server-side encryption: +SSE-C gives you client-side key management with server-side encryption: - **Client provides**: AES-256 encryption key and MD5 hash - **SeaweedFS handles**: Encryption/decryption operations transparently @@ -12,7 +12,7 @@ SSE-C enables client-side key management with server-side encryption: ## Required Headers -For all SSE-C operations, provide these headers: +For all SSE-C operations, include these headers: ```http X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256 diff --git a/Server-Side-Encryption-SSE-KMS.md b/Server-Side-Encryption-SSE-KMS.md index 3576867..633c4a9 100644 --- a/Server-Side-Encryption-SSE-KMS.md +++ b/Server-Side-Encryption-SSE-KMS.md @@ -1,7 +1,6 @@ # SSE-KMS: Server-Side Encryption with Key Management Service -SeaweedFS SSE-KMS supports multiple external Key Management Service providers for enterprise-grade encryption key management. This guide covers setup and configuration for AWS KMS, Google Cloud KMS, OpenBao/Vault. -and Azure Key Vault (experimental, build tag `azurekms`). +SeaweedFS works with your existing Key Management Service (KMS) so you can keep keys where they belong. This guide walks you through AWS KMS, Google Cloud KMS, and OpenBao/Vault. Azure Key Vault is also available as experimental (build tag `azurekms`). ## Supported KMS Providers @@ -16,7 +15,7 @@ and Azure Key Vault (experimental, build tag `azurekms`). ### 1. Configure SeaweedFS -KMS providers are configured in the S3 config JSON file: +Tell SeaweedFS about your KMS in the S3 config JSON file: ```json { diff --git a/Server-Side-Encryption.md b/Server-Side-Encryption.md index 9c07c98..316abd5 100644 --- a/Server-Side-Encryption.md +++ b/Server-Side-Encryption.md @@ -1,9 +1,11 @@ # Server-Side Encryption (SSE) -SeaweedFS supports AWS S3-compatible server-side encryption to protect data at rest. Three encryption methods are available, each offering different levels of control over encryption keys and management. +If you're using SeaweedFS with the S3 API, you can encrypt objects at rest without changing your apps. We support the same server-side encryption (SSE) options as Amazon S3, so you can pick the one that fits how you already manage keys. ## Overview +Use this quick guide to choose the right option: + | Encryption Type | Key Management | Use Case | |------------------|----------------|----------| | **SSE-KMS** | External KMS providers | Enterprise key management, audit trails | @@ -13,24 +15,24 @@ SeaweedFS supports AWS S3-compatible server-side encryption to protect data at r ## Encryption Types ### SSE-KMS (Server-Side Encryption with Key Management Service) -- **Best for**: Enterprise environments with external KMS infrastructure -- **Key management**: External providers (AWS KMS, Google Cloud KMS, OpenBao/Vault, Azure Key Vault [experimental]) -- **Features**: Centralized key management, detailed audit logs, per-bucket key assignment, optional Bucket Key optimization -- **Configuration**: Requires KMS provider setup in S3 config +- **Pick this if**: You already use a KMS and want strong audit trails +- **Keys live in**: External providers (AWS KMS, Google Cloud KMS, OpenBao/Vault, Azure Key Vault [experimental]) +- **Why teams like it**: Centralized key management, detailed audit logs, per-bucket key assignment, optional Bucket Key optimization +- **Configuration**: Requires KMS provider setup in the S3 config - **Documentation**: [SSE-KMS Guide](Server-Side-Encryption-SSE-KMS.md) ### SSE-C (Server-Side Encryption with Customer-Provided Keys) -- **Best for**: Maximum control over encryption keys, regulatory compliance -- **Key management**: Customer provides encryption keys with each request -- **Features**: No key storage on server, customer has full control +- **Pick this if**: You want to bring your own keys and keep full control +- **Keys live in**: Your application (sent per request) +- **Why teams like it**: No key storage on the server; maximum control for compliance-heavy environments - **Configuration**: Keys provided via HTTP headers - **Documentation**: [SSE-C Guide](Server-Side-Encryption-SSE-C.md) ### SSE-S3 (Server-Managed Keys) -- **Best for**: Simple, fully managed encryption with minimal configuration -- **Key management**: SeaweedFS-managed keys with automatic key handling -- **Features**: Works with explicit `x-amz-server-side-encryption: AES256` and bucket default encryption; supports multipart uploads and range requests -- **Configuration**: Optional bucket-level default encryption via standard S3 bucket encryption API +- **Pick this if**: You want simple, fully managed encryption with minimal setup +- **Keys live in**: SeaweedFS (we handle the key management for you) +- **Why teams like it**: Works with explicit `x-amz-server-side-encryption: AES256` and bucket default encryption; supports multipart uploads and range requests +- **Configuration**: Optional bucket-level default encryption via the standard S3 bucket encryption API ## Quick Start @@ -64,7 +66,7 @@ aws s3 cp file.txt s3://mybucket/file.txt \ ## Configuration ### Basic Setup -Configure KMS providers and IAM settings in the S3 config file: +Configure KMS providers and IAM settings in your S3 config file: ```json { @@ -117,4 +119,4 @@ weed s3 -config=s3_kms_config.json - **SSE-C**: Full support with security best practices - **SSE-S3**: Supported with SeaweedFS-managed keys and bucket default encryption -For detailed setup guides, examples, and configuration options, see the individual encryption method documentation linked above. \ No newline at end of file +For hands-on setup guides and examples, see the individual encryption method docs linked above. \ No newline at end of file From 299bcc30158698f71b6931d214d71143e988cd79 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Mon, 15 Sep 2025 22:24:39 -0700 Subject: [PATCH 128/149] SMQ --- Kafka-to-Kafka-Gateway-to-SMQ-to-SQL.md | 54 ++++++++++++ PostgreSQL-compatible-Server-weed-db.md | 106 +++++++++++++++++++++++ Pub-Sub-to-SMQ-to-SQL.md | 67 ++++++++++++++ Structured-Data-Lake-with-SMQ-and-SQL.md | 52 +++++++++++ _Sidebar.md | 4 + 5 files changed, 283 insertions(+) create mode 100644 Kafka-to-Kafka-Gateway-to-SMQ-to-SQL.md create mode 100644 PostgreSQL-compatible-Server-weed-db.md create mode 100644 Pub-Sub-to-SMQ-to-SQL.md create mode 100644 Structured-Data-Lake-with-SMQ-and-SQL.md diff --git a/Kafka-to-Kafka-Gateway-to-SMQ-to-SQL.md b/Kafka-to-Kafka-Gateway-to-SMQ-to-SQL.md new file mode 100644 index 0000000..b24897c --- /dev/null +++ b/Kafka-to-Kafka-Gateway-to-SMQ-to-SQL.md @@ -0,0 +1,54 @@ +# Kafka client → Kafka gateway → SMQ → SQL + +Bring your existing Kafka clients. Point them at the SeaweedFS Kafka gateway. Messages flow into Seaweed Message Queue (SMQ) for streaming, while SeaweedFS persists them into Parquet for SQL analytics. + +> See the end-to-end picture: [[Structured Data Lake with SMQ and SQL]]. + +## Why use the Kafka gateway + +- Keep your Kafka tooling and clients +- Scale stateless brokers and storage independently +- Get streaming + Parquet-based analytics without changing producers + +## Architecture + +``` +Kafka Clients <=> SeaweedFS Kafka Gateway <=> SMQ Brokers => Subscribers + \ + +--> SeaweedFS (Parquet) => SQL Engines +``` + +The gateway speaks the Kafka protocol to clients and maps topics/partitions, offsets, and consumer groups to SMQ semantics. + +## What stays the same + +- Kafka client libraries and tooling (producers/consumers) +- Topic/partition concepts +- Consumer groups and offsets + +## What you gain + +- Durable Parquet storage for batch analytics +- One pipeline for both streaming and SQL +- Simple, scalable operations (stateless brokers, disaggregated storage) + +## Getting started + +1) Start SMQ and the Kafka gateway (example ports): + +```bash +weed mq.broker -port=17777 -master=localhost:9333 +weed mq.agent -port=16777 -broker=localhost:17777 +weed mq.kafka -port=19092 -broker=localhost:17777 +``` + +2) Point your Kafka producer/consumer at `localhost:19092`. + +3) Query the resulting Parquet data with your SQL engine of choice (Trino, Spark, DuckDB, etc.). + +## Next steps + +- Central concepts: [[Structured Data Lake with SMQ and SQL]] +- SMQ overview: [[Seaweed Message Queue]] + + diff --git a/PostgreSQL-compatible-Server-weed-db.md b/PostgreSQL-compatible-Server-weed-db.md new file mode 100644 index 0000000..0c54481 --- /dev/null +++ b/PostgreSQL-compatible-Server-weed-db.md @@ -0,0 +1,106 @@ +# PostgreSQL-compatible Server (weed db) + +SeaweedFS exposes a PostgreSQL wire protocol server so you can connect with any PostgreSQL client (psql, JDBC/ODBC, BI tools) and run SQL over Seaweed Message Queue (SMQ) topics. It is read-only and designed for analytics: query live streams and Parquet-backed history without ETL. + +## What you get + +- PostgreSQL wire protocol compatibility (psql, JDBC/ODBC, BI tools) +- Query live + historical data (hybrid scanner: brokers + Parquet files) +- Simple ops: stateless server, MD5 auth, optional TLS +- Works alongside pub/sub and Kafka ingestion — one structured data lake + +## Start the server + +```bash +# Development (no auth) +weed db + +# Production (MD5 auth) +weed db -auth=md5 -users='{"admin":"secret","analyst":"readonly"}' -host=0.0.0.0 -port=5432 + +# Credentials from file (recommended) +echo '{"admin":"secret","analyst":"readonly"}' > /etc/seaweedfs/users.json +weed db -auth=md5 -users="@/etc/seaweedfs/users.json" + +# With TLS +auth_users='@/etc/seaweedfs/users.json' +weed db -auth=md5 -users="$auth_users" -tls-cert=/etc/ssl/server.crt -tls-key=/etc/ssl/server.key +``` + +Key options: +- `-auth`: `trust` (dev), `password`, `md5` (recommended) +- `-users`: inline JSON or `@/path/to/users.json` +- `-master`: SeaweedFS master (comma-separated for HA) +- `-max-connections`, `-idle-timeout`, `-database` +- `-tls-cert`, `-tls-key` for encrypted connections + +## Connect with clients + +```bash +# psql +PGPASSWORD=secret psql -h localhost -p 5432 -U admin -d default +psql "postgresql://admin:secret@localhost:5432/default" +``` + +```java +// JDBC +String url = "jdbc:postgresql://localhost:5432/default"; +Connection conn = DriverManager.getConnection(url, "admin", "secret"); +``` + +```python +# psycopg2 +import psycopg2 +conn = psycopg2.connect(host="localhost", port=5432, user="admin", password="secret", database="default") +``` + +BI tools: use the native PostgreSQL connector (host, port 5432, database `default`, user/password). + +## SQL you can run + +Schema/navigation: +```sql +SHOW DATABASES; -- MQ namespaces +USE my_namespace; -- switch database +SHOW TABLES; -- topics in namespace +DESCRIBE topic_name; -- schema +``` + +Queries (read-only): +```sql +SELECT * FROM events LIMIT 10; +SELECT * FROM events WHERE _ts >= '2025-01-01' LIMIT 1000; +SELECT COUNT(*) FROM events; -- fast-path optimized +SELECT MIN(timestamp), MAX(timestamp) FROM events; -- fast-path + +-- System columns available on all tables +SELECT _ts, _key, _source, * FROM events LIMIT 5; +``` + +Notes: +- `_ts` supports convenient string-to-timestamp parsing in WHERE clauses +- Live + Parquet results are merged transparently + +## Limitations (current) + +- Read-only: no INSERT/UPDATE/DELETE, no DDL (CREATE/DROP/ALTER) +- Not supported yet: `JOIN`, `ORDER BY`, `GROUP BY`, `HAVING`, window functions, subqueries + +## Operations and scaling + +- Stateless server; run multiple `weed db` instances behind a load balancer +- Use MD5 auth and TLS in production +- Tune `-max-connections` and `-idle-timeout` + +## How this fits your data lake + +- Producers (Kafka or pub/sub) write schematized messages into SMQ +- SeaweedFS persists to Parquet; brokers serve live data +- PostgreSQL-compatible server exposes SQL over both — one data lake + +## Related + +- [[Structured Data Lake with SMQ and SQL]] +- [[SQL Queries on Message Queue]] +- [[SQL Quick Reference]] +- [[Seaweed Message Queue]] diff --git a/Pub-Sub-to-SMQ-to-SQL.md b/Pub-Sub-to-SMQ-to-SQL.md new file mode 100644 index 0000000..f320aa6 --- /dev/null +++ b/Pub-Sub-to-SMQ-to-SQL.md @@ -0,0 +1,67 @@ +# Pub/Sub to SMQ to SQL: From Streams to Tables + +Seaweed Message Queue (SMQ) bridges your live pub/sub streams and your analytical world. You publish structured messages, SMQ brokers stream them in real time, and at the same time SeaweedFS persists them into Parquet files so you can query with SQL engines later. + +> New here? See the bigger picture: [[Structured Data Lake with SMQ and SQL]]. + +## Why this matters + +- Real-time and batch in one pipeline: stream to subscribers now, query with SQL later. +- Store once, use twice: messages land in Parquet (columnar, compressed), great for analytics. +- Stateless brokers, disaggregated storage: scale brokers independently of storage. + +## Architecture at a glance + +``` +Publishers => SMQ Agent (gRPC) => SMQ Brokers => Subscribers + \ + +--> SeaweedFS (Parquet) => SQL Engines +``` + +- **Publish**: Send structured messages with a schema. +- **Stream**: Subscribers process messages with consumer groups and offsets. +- **Persist**: Messages are compacted/organized into Parquet files in SeaweedFS. +- **Query**: Point your SQL engines to the Parquet location. + +## What you publish + +Messages are structured records defined by a schema. SMQ validates these on publish and keeps ordering guarantees per key while enabling high concurrency via a sliding window. + +## What you query + +Parquet files written by SMQ are queryable by your favorite SQL engines: + +- Trino/Presto +- Spark SQL +- DuckDB +- ClickHouse (via file table engines) + +Point them to the Parquet path in SeaweedFS and query away. + +## Quick start + +1) Start a broker and an agent: + +```bash +weed mq.broker -port=17777 -master=localhost:9333 +weed mq.agent -port=16777 -broker=localhost:17777 +``` + +2) Define a schema and publish: + +```go +type MyEvent struct { + Key []byte + UserId int64 + Action string +} +``` + +3) Subscribe in real time. Your consumers get the stream; your data team gets Parquet. + +## Where to go next + +- Central concepts: [[Structured Data Lake with SMQ and SQL]] +- Messaging basics: [[Seaweed Message Queue]] + + diff --git a/Structured-Data-Lake-with-SMQ-and-SQL.md b/Structured-Data-Lake-with-SMQ-and-SQL.md new file mode 100644 index 0000000..1d90948 --- /dev/null +++ b/Structured-Data-Lake-with-SMQ-and-SQL.md @@ -0,0 +1,52 @@ +# Structured Data Lake with SMQ and SQL + +SeaweedFS + Seaweed Message Queue (SMQ) gives you a unified pipeline: produce structured messages, process them in real time, and query the same data with SQL. Whether producers speak Kafka or a simple pub/sub gRPC API, they both write schematized messages into the same data lake. + +## Core ideas + +- Schematized messages can be queried directly with SQL (no ETL required) +- SMQ brokers are computation-only nodes and scale linearly with demand +- Structured data is written as messages and can be queried in real time +- Together, SeaweedFS + SMQ form a data lake for structured data (hot streams + Parquet) + +## Ingestion paths + +Two equivalent ways to ingest structured messages: + +- Kafka clients → [[Kafka to Kafka Gateway to SMQ to SQL]] +- Pub/Sub clients → [[Pub-Sub to SMQ to SQL]] + +Both end up with the same outcomes: live streams for subscribers and Parquet files in SeaweedFS for SQL engines. + +## Architecture + +``` +Producers (Kafka or Pub/Sub) ==> SMQ Brokers ==> Subscribers (real-time) + \ + +--> SeaweedFS (Parquet) ==> SQL Engines +``` + +## Querying the lake + +Point your SQL engines at the Parquet paths: + +- Trino/Presto +- Spark SQL +- DuckDB +- ClickHouse (file table engines) + +Examples are available in the ingestion pages. + +## Operate at scale + +- Scale SMQ brokers horizontally; they are stateless computation nodes +- Storage is disaggregated (SeaweedFS) for durable, efficient Parquet files + +## Learn more + +- Messaging basics: [[Seaweed Message Queue]] +- PostgreSQL-compatible server: [[PostgreSQL-compatible Server weed db]] +- Kafka ingestion: [[Kafka to Kafka Gateway to SMQ to SQL]] +- Pub/Sub ingestion: [[Pub-Sub to SMQ to SQL]] + + diff --git a/_Sidebar.md b/_Sidebar.md index 9d0656e..66b885a 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -115,9 +115,13 @@ * [[Filer Metadata Events]] ### Messaging +* [[Structured Data Lake with SMQ and SQL]] * [[Seaweed Message Queue]] * [[SQL Queries on Message Queue]] * [[SQL Quick Reference]] +* [[PostgreSQL-compatible Server weed db]] +* [[Pub-Sub to SMQ to SQL]] +* [[Kafka to Kafka Gateway to SMQ to SQL]] ### Use Cases * [[Use Cases]] From 9df018b074ded0570285acdb61f28c637287a0a5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Martin=20Stiborsk=C3=BD?= Date: Thu, 2 Oct 2025 09:09:41 +0200 Subject: [PATCH 129/149] fixed that value to be an integer, as expected --- S3-API-FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/S3-API-FAQ.md b/S3-API-FAQ.md index 9ccd77b..ebc60da 100644 --- a/S3-API-FAQ.md +++ b/S3-API-FAQ.md @@ -72,7 +72,7 @@ fs.configure -locationPrefix /buckets/f341868e-baff-4e20-896a-08bc148e32f9/my-d "Prefix": "my-directory-whose-files-will-expire-in-20-days" }, "Expiration": { - "Days": 20d + "Days": 20 } } ] From c1b39515e5a508da937a81399c75934868bd5e63 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Fri, 10 Oct 2025 18:24:48 -0700 Subject: [PATCH 130/149] Update Docker-Compose-for-S3.md --- Docker-Compose-for-S3.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/Docker-Compose-for-S3.md b/Docker-Compose-for-S3.md index 5f36a64..dadbf80 100644 --- a/Docker-Compose-for-S3.md +++ b/Docker-Compose-for-S3.md @@ -1,8 +1,6 @@ To use SeaweedFS S3 API, here is the simplest form: ``` -version: '3.8' - services: seaweedfs-s3: image: chrislusf/seaweedfs From 77f54bb7709f8eff63ff00dc1743fa0130dee98d Mon Sep 17 00:00:00 2001 From: nightcoffee Date: Mon, 20 Oct 2025 01:50:16 +0800 Subject: [PATCH 131/149] volume.fix.replication need -force option now --- Volume-Management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Volume-Management.md b/Volume-Management.md index 02804e9..5fb0f2d 100644 --- a/Volume-Management.md +++ b/Volume-Management.md @@ -14,7 +14,7 @@ scripts = """ ec.rebuild -force ec.balance -force volume.balance -force - volume.fix.replication + volume.fix.replication -force """ sleep_minutes = 17 # sleep minutes between each script execution From 4176f547f3b21b01b6834a66bf770da3fae54244 Mon Sep 17 00:00:00 2001 From: Mitch B Date: Thu, 23 Oct 2025 09:27:10 -0500 Subject: [PATCH 132/149] Add note about how one-volume-server-per-disk affects replication --- Production-Setup.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Production-Setup.md b/Production-Setup.md index a3266f5..58d07f0 100644 --- a/Production-Setup.md +++ b/Production-Setup.md @@ -110,6 +110,7 @@ Additional notes: * If the disk space is huge and there will be a lot of volumes, configure `-index=leveldb` to reduce memory load. * For busy volume servers, `-compactionMBps` can help to throttle the background jobs, e.g., compaction, balancing, encoding/decoding,etc. * After adding volume servers, there will not be data rebalancing. It is generally not a good idea to actively rebalance data, which cost network bandwidth and slows down other servers. Data are written to new servers after new volumes are created on them. You can use `weed shell` and run `volume.balance -force` to manually balance them. +* Multiple volume servers on the same physical host count as separate servers for replication purposes. So if you have two physical hosts with multiple volume servers each, replication `001` (one replica in the same rack) does not guarantee that each copy will be stored on different physical hosts. ## Check the object store setup From 83719e1cb5a3a12441f883688c1ea56c70e0d4fc Mon Sep 17 00:00:00 2001 From: Mitch B Date: Thu, 23 Oct 2025 09:50:55 -0500 Subject: [PATCH 133/149] Update filer defaultReplicaPlacement notes --- Production-Setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Production-Setup.md b/Production-Setup.md index 58d07f0..3b29042 100644 --- a/Production-Setup.md +++ b/Production-Setup.md @@ -141,7 +141,7 @@ weed filer -ip=xxx.xxx.xxx.xxx -master=ip1:9333,ip2:9333,ip3:9333 -dataCenter=dc ``` Additional notes: -* Both `weed filer` and `weed master` has option `-defaultReplicaPlacement`. `weed master` uses it for the object store, while `weed filer` uses it for files. The `weed filer` default setting is "000", and overwrites the one `weed master` has. +* Both `weed filer` and `weed master` has option `-defaultReplicaPlacement`. `weed master` uses it for the object store, while `weed filer` uses it for files. The `weed filer` setting will default to the value for `weed master`. * `-encryptVolumeData` option is when you need to encrypt the data on volume servers. See [[Filer Data Encryption]] ## Setup multiple filers From 2e8c786c3c6916d3645d23b77668d9157ab9dcb9 Mon Sep 17 00:00:00 2001 From: Mitch B Date: Thu, 23 Oct 2025 13:26:15 -0500 Subject: [PATCH 134/149] Fix typo "HUB" -> "HUP" --- Amazon-S3-API.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Amazon-S3-API.md b/Amazon-S3-API.md index 7bb1efc..8de6317 100644 --- a/Amazon-S3-API.md +++ b/Amazon-S3-API.md @@ -246,7 +246,7 @@ Output from above `s3.configure` command: ## Static Configuration -To enable credential based access, create a config.json file similar to the example below, and specify it via `weed s3 -config=config.json`. The config file can be re-read on the HUB signal without restarting the main process via `pkill -HUP weed` +To enable credential based access, create a config.json file similar to the example below, and specify it via `weed s3 -config=config.json`. The config file can be re-read on the HUP signal without restarting the main process via `pkill -HUP weed` You just need to create a user with all "Admin", "Read", "Write", "List", "Tagging" actions. You can create as many users as needed. Each user can have multiple credentials. From d3a5957a347c5924be4bf774306ee8b93c76f1e2 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Fri, 24 Oct 2025 13:04:58 -0700 Subject: [PATCH 135/149] outdated fix https://github.com/seaweedfs/seaweedfs/discussions/7373 --- Filer-Server-API.md | 1 - 1 file changed, 1 deletion(-) diff --git a/Filer-Server-API.md b/Filer-Server-API.md index ca63655..7e63d34 100644 --- a/Filer-Server-API.md +++ b/Filer-Server-API.md @@ -49,7 +49,6 @@ You can append to any HTTP API with &pretty=y to see a formatted json output. | resolveManifest | resolve manifest chunks | false | ### notice * It is recommended to add retries when writing to Filer. -* `AutoChunking` is not supported for method `PUT`. If the file length is greater than 256MB, only the leading 256MB in the `PUT` request will be saved. * When appending to a file, each append will create one chunk and added to the file metadata. If there are too many small appends, there could be too many chunks. So try to keep each append size reasonably big. Examples: From 3e907dd9961026aff0076b8ea8884d47a29525cb Mon Sep 17 00:00:00 2001 From: Mitch B Date: Mon, 27 Oct 2025 17:21:06 -0500 Subject: [PATCH 136/149] Fix typo "filler" -> "filer" --- weed-shell.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/weed-shell.md b/weed-shell.md index 0509c86..e7a62e1 100644 --- a/weed-shell.md +++ b/weed-shell.md @@ -120,7 +120,7 @@ volume 2480 fast-volume-1:8080 has 163827 entries, fast-volume-3:8080 missed 0 a volume 2480 fast-volume-3:8080 has 163827 entries, fast-volume-1:8080 missed 0 and partially deleted 0 entries ``` 2. `volume.fsck` -Search for files that are in the filler, but there are no chunks on the volume servers +Search for files that are in the filer, but there are no chunks on the volume servers ``` lock;volume.fsck -findMissingChunksInFiler -verifyNeedles -collection logs-data -volumeId 2480 -v checking directory /buckets/logs-data/2023-02-13 From 6c3c8a972e5c7453b80d589934715279d5053db1 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Tue, 28 Oct 2025 12:51:21 -0700 Subject: [PATCH 137/149] Update S3-CORS.md --- S3-CORS.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/S3-CORS.md b/S3-CORS.md index f50789f..98de3f6 100644 --- a/S3-CORS.md +++ b/S3-CORS.md @@ -37,6 +37,7 @@ A CORS configuration consists of one or more CORS rules. Each rule defines: - **ID** (optional): A unique identifier for the rule - **AllowedOrigin** (required): Specifies which origins are allowed to access the bucket - **AllowedMethod** (required): HTTP methods that are allowed (GET, PUT, POST, DELETE, HEAD) + - **Note**: Do NOT include `OPTIONS` in AllowedMethods - it is automatically handled for preflight requests - **AllowedHeader** (optional): Headers that are allowed in the actual request - **ExposeHeader** (optional): Headers that browsers can access from the response - **MaxAgeSeconds** (optional): How long browsers can cache the preflight response @@ -347,10 +348,11 @@ For actual requests (GET, POST, PUT, DELETE, etc.), SeaweedFS: ### Common Issues -1. **CORS policy error**: Check that your origin is listed in `AllowedOrigins` -2. **Method not allowed**: Ensure the HTTP method is in `AllowedMethods` -3. **Header blocked**: Add required headers to `AllowedHeaders` -4. **Preflight failure**: Verify all preflight requirements are met +1. **Invalid HTTP method: OPTIONS**: Do NOT include `OPTIONS` in `AllowedMethods` - it is automatically handled for CORS preflight requests. Only use: GET, PUT, POST, DELETE, HEAD +2. **CORS policy error**: Check that your origin is listed in `AllowedOrigins` +3. **Method not allowed**: Ensure the HTTP method is in `AllowedMethods` +4. **Header blocked**: Add required headers to `AllowedHeaders` +5. **Preflight failure**: Verify all preflight requirements are met ### Debugging Tips From 98dd76720581be06655a83773e5354be282f59fe Mon Sep 17 00:00:00 2001 From: chrislusf Date: Tue, 28 Oct 2025 12:59:38 -0700 Subject: [PATCH 138/149] add missing links --- _Sidebar.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/_Sidebar.md b/_Sidebar.md index 66b885a..bbadd6e 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -70,7 +70,9 @@ * [[S3 Credentials]] * [[Amazon S3 API]] * [[S3 Conditional Operations]] +* [[S3 CORS]] * [[S3 Object Lock and Retention]] +* [[S3 Object Versioning]] * [[AWS CLI with SeaweedFS]] * [[s3cmd with SeaweedFS]] * [[rclone with SeaweedFS]] From d4bdeb2ed45ba883792c904b02ad3ddda55cdccd Mon Sep 17 00:00:00 2001 From: chrislusf Date: Fri, 31 Oct 2025 17:13:46 -0700 Subject: [PATCH 139/149] Update S3-CORS.md --- S3-CORS.md | 38 +++++++++++++++++++++++++++++++++++++- 1 file changed, 37 insertions(+), 1 deletion(-) diff --git a/S3-CORS.md b/S3-CORS.md index 98de3f6..f2b1ee2 100644 --- a/S3-CORS.md +++ b/S3-CORS.md @@ -7,12 +7,48 @@ SeaweedFS supports S3-compatible Cross-Origin Resource Sharing (CORS) configurat CORS defines a way for web applications running at one domain to access resources at another domain. When a web application tries to access your S3 bucket from a different domain, the browser will first send a preflight OPTIONS request to check if the cross-origin request is allowed. SeaweedFS handles CORS through: +- **Global CORS configuration**: Server-wide default CORS settings - **Bucket-level CORS configuration**: Each bucket can have its own CORS rules - **Persistent storage**: CORS configurations are stored in bucket metadata - **Automatic header handling**: CORS middleware automatically applies appropriate headers - **Preflight request support**: Proper handling of OPTIONS requests -## CORS Configuration +## Quick Start: Global CORS Configuration + +The simplest way to enable CORS for all buckets is using the `-s3.allowedOrigins` parameter: + +```bash +# Allow all origins (useful for development) +weed server -s3 -s3.allowedOrigins=* + +# Allow specific origins +weed server -s3 -s3.allowedOrigins=https://app.example.com,https://admin.example.com + +# Docker Compose example +services: + seaweedfs: + image: chrislusf/seaweedfs:latest + command: "server -s3 -s3.allowedOrigins=*" +``` + +This global configuration: +- Works immediately without additional setup +- Applies to all buckets by default +- Can be overridden by bucket-level CORS configuration +- Supports GET, PUT, POST, DELETE, and HEAD methods +- Allows all headers (*) + +### CORS Configuration Priority + +SeaweedFS uses the following priority order: + +1. **Bucket-level CORS** (if configured via `aws s3api put-bucket-cors`) - highest priority +2. **Global CORS** (from `-s3.allowedOrigins` parameter) - fallback if no bucket config +3. **No CORS** (if neither is configured) - no CORS headers applied + +This means you can set a permissive global default and override it with stricter rules for specific buckets. + +## Advanced: Bucket-Level CORS Configuration ### Basic CORS Rule Structure From 3b61fc0b46bc7ca11c36809491b16e2199374357 Mon Sep 17 00:00:00 2001 From: santurak1 Date: Sun, 2 Nov 2025 12:55:09 +0100 Subject: [PATCH 140/149] Am the CEO/DIRECTOR/CREATOR --- Santuraki-jr-project.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 Santuraki-jr-project.md diff --git a/Santuraki-jr-project.md b/Santuraki-jr-project.md new file mode 100644 index 0000000..0c27339 --- /dev/null +++ b/Santuraki-jr-project.md @@ -0,0 +1 @@ +[text](url) \ No newline at end of file From 0e9254991f065058cb480e87d370d2b608786dbe Mon Sep 17 00:00:00 2001 From: chrislusf Date: Sun, 2 Nov 2025 12:42:23 -0800 Subject: [PATCH 141/149] Revert "Am the CEO/DIRECTOR/CREATOR" This reverts commit 3b61fc0b46bc7ca11c36809491b16e2199374357. --- Santuraki-jr-project.md | 1 - 1 file changed, 1 deletion(-) delete mode 100644 Santuraki-jr-project.md diff --git a/Santuraki-jr-project.md b/Santuraki-jr-project.md deleted file mode 100644 index 0c27339..0000000 --- a/Santuraki-jr-project.md +++ /dev/null @@ -1 +0,0 @@ -[text](url) \ No newline at end of file From 60ca2b1ac9e8fd78fd073edce91b13a565fa66e5 Mon Sep 17 00:00:00 2001 From: Mitch B Date: Wed, 5 Nov 2025 11:23:52 -0600 Subject: [PATCH 142/149] Add reload functionality to systemd filer service --- Server-Startup-Setup.md => Server-Startup-via-Systemd.md | 1 + 1 file changed, 1 insertion(+) rename Server-Startup-Setup.md => Server-Startup-via-Systemd.md (98%) diff --git a/Server-Startup-Setup.md b/Server-Startup-via-Systemd.md similarity index 98% rename from Server-Startup-Setup.md rename to Server-Startup-via-Systemd.md index 8c9e207..e764f08 100644 --- a/Server-Startup-Setup.md +++ b/Server-Startup-via-Systemd.md @@ -97,6 +97,7 @@ Type=simple User=root Group=root ExecStartPre=/bin/sleep 30 +ExecReload=/bin/kill -HUP $MAINPID ExecStart=/usr/local/bin/weed filer WorkingDirectory=/media/jmn/HDD/ From 3905fb89493e741505ef8f45efaddac8dc6ded79 Mon Sep 17 00:00:00 2001 From: Mitch B Date: Thu, 6 Nov 2025 12:43:43 -0600 Subject: [PATCH 143/149] Update sidebar page link --- _Sidebar.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_Sidebar.md b/_Sidebar.md index bbadd6e..22b1a79 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -18,7 +18,7 @@ * [[Store file with a Time To Live]] * [[Failover Master Server]] * [[Erasure coding for warm storage]] -* [[Server Startup Setup]] +* [[Server Startup via Systemd]] * [[Environment Variables]] ### Filer From 34c036bbc58e647b2b87fbf0bb055591ccae09d3 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 8 Nov 2025 00:57:58 -0800 Subject: [PATCH 144/149] Updated Independent Benchmarks (markdown) --- Independent-Benchmarks.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Independent-Benchmarks.md b/Independent-Benchmarks.md index 85ee54c..968bfbc 100644 --- a/Independent-Benchmarks.md +++ b/Independent-Benchmarks.md @@ -24,3 +24,6 @@ https://github.com/bingoohuang/blog/issues/57 # Benchmark SeaweedFS as a GlusterFS replacement https://github.com/seaweedfs/seaweedfs/wiki/Benchmark-SeaweedFS-as-a-GlusterFS-replacement + +# Benchmark Object Stores +https://it-notes.dragas.net/2025/11/06/self-hosting-your-mastodon-media-with-seaweedfs/ From 655ca795eed4ec2be1cb4ce89fcc02c6c87196d8 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Thu, 13 Nov 2025 00:05:55 -0800 Subject: [PATCH 145/149] arn:aws --- Keycloak-Integration.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Keycloak-Integration.md b/Keycloak-Integration.md index 645375e..38a551d 100644 --- a/Keycloak-Integration.md +++ b/Keycloak-Integration.md @@ -60,10 +60,10 @@ Minimal example with Keycloak OIDC provider, role mapping, roles, and policies: "scopes": ["openid", "profile", "email", "roles", "groups"], "roleMapping": { "rules": [ - { "claim": "groups", "value": "admins", "role": "arn:seaweed:iam::role/S3AdminRole" }, - { "claim": "groups", "value": "developers", "role": "arn:seaweed:iam::role/S3WriteRole" } + { "claim": "groups", "value": "admins", "role": "arn:aws:iam::role/S3AdminRole" }, + { "claim": "groups", "value": "developers", "role": "arn:aws:iam::role/S3WriteRole" } ], - "defaultRole": "arn:seaweed:iam::role/S3ReadOnlyRole" + "defaultRole": "arn:aws:iam::role/S3ReadOnlyRole" } } } @@ -100,7 +100,7 @@ Minimal example with Keycloak OIDC provider, role mapping, roles, and policies: "roles": [ { "roleName": "S3ReadOnlyRole", - "roleArn": "arn:seaweed:iam::role/S3ReadOnlyRole", + "roleArn": "arn:aws:iam::role/S3ReadOnlyRole", "attachedPolicies": ["S3ReadOnlyPolicy"], "trustPolicy": { "Version": "2012-10-17", @@ -118,7 +118,7 @@ Minimal example with Keycloak OIDC provider, role mapping, roles, and policies: }, { "roleName": "S3WriteRole", - "roleArn": "arn:seaweed:iam::role/S3WriteRole", + "roleArn": "arn:aws:iam::role/S3WriteRole", "attachedPolicies": ["S3WritePolicy"], "trustPolicy": { "Version": "2012-10-17", @@ -136,7 +136,7 @@ Minimal example with Keycloak OIDC provider, role mapping, roles, and policies: }, { "roleName": "S3AdminRole", - "roleArn": "arn:seaweed:iam::role/S3AdminRole", + "roleArn": "arn:aws:iam::role/S3AdminRole", "attachedPolicies": ["S3AdminPolicy"], "trustPolicy": { "Version": "2012-10-17", From 6e35b3dd3c7bc0930703cd355624b0fc4f97d3e9 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Thu, 13 Nov 2025 18:11:04 -0800 Subject: [PATCH 146/149] Update Environment-Variables.md --- Environment-Variables.md | 198 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 198 insertions(+) diff --git a/Environment-Variables.md b/Environment-Variables.md index 49e4672..d976fbf 100644 --- a/Environment-Variables.md +++ b/Environment-Variables.md @@ -13,6 +13,26 @@ weed master # Weed prefix For `v`, `logtostderr`, `stderrthreshold`, `vmoudle`, `options`, `logdir`, `alsologtostderr`, `log_backtrace_at` , and `config_dir` you have to use `WEED_` as prefix for environment variable like this `WEED_CONFIG_DIR=/tmp` +## Configuration File Settings +For configuration file settings (like filer stores, replication settings, etc.), you must use the `WEED_` prefix with dots (`.`) replaced by underscores (`_`). + +For example, the `filer.toml` configuration: +```toml +[redis2] +enabled = true +address = "localhost:6379" +password = "secret" +database = 0 +``` + +Becomes these environment variables: +```shell +WEED_REDIS2_ENABLED=true +WEED_REDIS2_ADDRESS=localhost:6379 +WEED_REDIS2_PASSWORD=secret +WEED_REDIS2_DATABASE=0 +``` + # S3 Admin Credentials For S3 API server authentication, see the dedicated **[S3 Credentials](S3-Credentials)** page which covers: - Configuration file setup (highest priority) @@ -69,3 +89,181 @@ services: depends_on: - filer ``` + +# Filer Metadata Store Configuration + +The filer supports multiple metadata storage backends. You can configure them using environment variables instead of a `filer.toml` file. + +## Redis Configuration + +### Basic Redis (`redis2`) +```yaml +version: '3.9' +services: + redis: + image: redis:7-alpine + ports: + - 6379:6379 + command: redis-server --requirepass your_password + + master: + image: chrislusf/seaweedfs:latest + ports: + - 9333:9333 + command: master + + volume: + image: chrislusf/seaweedfs:latest + ports: + - 8080:8080 + command: volume -mserver=master:9333 + depends_on: + - master + + filer: + image: chrislusf/seaweedfs:latest + ports: + - 8888:8888 + environment: + # Enable Redis as metadata store + - WEED_REDIS2_ENABLED=true + - WEED_REDIS2_ADDRESS=redis:6379 + - WEED_REDIS2_PASSWORD=your_password + - WEED_REDIS2_DATABASE=0 + # Optional: TLS configuration + - WEED_REDIS2_ENABLE_TLS=false + # Disable default leveldb2 + - WEED_LEVELDB2_ENABLED=false + command: filer -master=master:9333 + depends_on: + - master + - volume + - redis +``` + +### Redis Sentinel +```shell +WEED_REDIS2_SENTINEL_ENABLED=true +WEED_REDIS2_SENTINEL_ADDRESSES=sentinel1:26379,sentinel2:26379,sentinel3:26379 +WEED_REDIS2_SENTINEL_MASTERNAME=mymaster +WEED_REDIS2_SENTINEL_USERNAME= +WEED_REDIS2_SENTINEL_PASSWORD=secret +WEED_REDIS2_SENTINEL_DATABASE=0 +WEED_LEVELDB2_ENABLED=false +``` + +### Redis Cluster +```shell +WEED_REDIS_CLUSTER2_ENABLED=true +WEED_REDIS_CLUSTER2_ADDRESSES=redis1:6379,redis2:6379,redis3:6379 +WEED_REDIS_CLUSTER2_PASSWORD=secret +WEED_REDIS_CLUSTER2_READONLY=false +WEED_REDIS_CLUSTER2_ROUTEBYLATENCY=false +WEED_LEVELDB2_ENABLED=false +``` + +## MySQL/MariaDB Configuration +```yaml +version: '3.9' +services: + mysql: + image: mysql:8 + ports: + - 3306:3306 + environment: + - MYSQL_ROOT_PASSWORD=secret + - MYSQL_DATABASE=seaweedfs + - MYSQL_USER=seaweedfs + - MYSQL_PASSWORD=secret + + master: + image: chrislusf/seaweedfs:latest + ports: + - 9333:9333 + command: master + + volume: + image: chrislusf/seaweedfs:latest + ports: + - 8080:8080 + command: volume -mserver=master:9333 + depends_on: + - master + + filer: + image: chrislusf/seaweedfs:latest + ports: + - 8888:8888 + environment: + # MySQL configuration + - WEED_MYSQL_ENABLED=true + - WEED_MYSQL_HOSTNAME=mysql + - WEED_MYSQL_PORT=3306 + - WEED_MYSQL_DATABASE=seaweedfs + - WEED_MYSQL_USERNAME=seaweedfs + - WEED_MYSQL_PASSWORD=secret + - WEED_MYSQL_CONNECTION_MAX_IDLE=5 + - WEED_MYSQL_CONNECTION_MAX_OPEN=75 + - WEED_MYSQL_CONNECTION_MAX_LIFETIME_SECONDS=600 + - WEED_MYSQL_INTERPOLATEPARAMS=true + # Disable default leveldb2 + - WEED_LEVELDB2_ENABLED=false + command: filer -master=master:9333 + depends_on: + - master + - volume + - mysql +``` + +## PostgreSQL Configuration +```shell +WEED_POSTGRES_ENABLED=true +WEED_POSTGRES_HOSTNAME=postgres +WEED_POSTGRES_PORT=5432 +WEED_POSTGRES_DATABASE=seaweedfs +WEED_POSTGRES_USERNAME=seaweedfs +WEED_POSTGRES_PASSWORD=secret +WEED_POSTGRES_SSLMODE=disable +WEED_POSTGRES_CONNECTION_MAX_IDLE=5 +WEED_POSTGRES_CONNECTION_MAX_OPEN=75 +WEED_POSTGRES_CONNECTION_MAX_LIFETIME_SECONDS=600 +WEED_LEVELDB2_ENABLED=false +``` + +## MongoDB Configuration +```shell +WEED_MONGODB_ENABLED=true +WEED_MONGODB_URI=mongodb://mongodb:27017 +WEED_MONGODB_DATABASE=seaweedfs +WEED_MONGODB_USERNAME=seaweedfs +WEED_MONGODB_PASSWORD=secret +WEED_LEVELDB2_ENABLED=false +``` + +## Etcd Configuration +```shell +WEED_ETCD_ENABLED=true +WEED_ETCD_SERVERS=etcd1:2379,etcd2:2379,etcd3:2379 +WEED_ETCD_KEY_PREFIX=seaweedfs. +WEED_ETCD_TIMEOUT=3s +WEED_LEVELDB2_ENABLED=false +``` + +## Important Notes + +1. **Only one store can be enabled**: Make sure to disable the default `leveldb2` store when using an external metadata store: + ```shell + WEED_LEVELDB2_ENABLED=false + ``` + +2. **Available stores**: To see all available filer stores and their configuration options, run: + ```shell + weed scaffold -config=filer + ``` + +3. **Data migration**: Changing stores doesn't automatically migrate existing data. Apply these configurations to new installations or migrate data manually. + +4. **Array values**: For configuration options that accept arrays (like Redis cluster addresses), use comma-separated values: + ```shell + WEED_REDIS_CLUSTER2_ADDRESSES=host1:6379,host2:6379,host3:6379 + ``` From 8f3ef880c4db73fa40e9d8c5ef6cde296e20e3cb Mon Sep 17 00:00:00 2001 From: chrislusf Date: Tue, 25 Nov 2025 11:47:00 -0800 Subject: [PATCH 147/149] architecture --- S3-API-FAQ.md | 23 +++++++++++++++++++++++ Security-Configuration.md | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) diff --git a/S3-API-FAQ.md b/S3-API-FAQ.md index ebc60da..1728369 100644 --- a/S3-API-FAQ.md +++ b/S3-API-FAQ.md @@ -231,4 +231,27 @@ aws s3 presign s3://test-bucket/test-object --endpoint-url https://yourdomain.co For detailed configuration examples, see the [[S3-Nginx-Proxy]] documentation. +## TLS error: "client sent an HTTP request to an HTTPS server" + +SeaweedFS has two separate communication layers: +- **gRPC (Control Plane)**: Metadata operations - configured via `[grpc.*]` +- **HTTP/HTTPS (Data Plane)**: File data uploads/downloads - configured via `[https.*]` + +If you see this error when S3 API communicates with a TLS-enabled Filer, enable HTTPS client mode: + +```toml +[https.filer] +cert = "/path/to/filer.crt" +key = "/path/to/filer.key" +ca = "/path/to/ca.crt" + +[https.client] +enabled = true +cert = "/path/to/client.crt" +key = "/path/to/client.key" +ca = "/path/to/ca.crt" +``` + +The `[https.filer]` makes the Filer accept HTTPS, while `[https.client]` with `enabled = true` makes clients (S3 API, mount, etc.) use HTTPS for data operations. See [[Security-Configuration]] for details. + diff --git a/Security-Configuration.md b/Security-Configuration.md index 5b78e96..7593b88 100644 --- a/Security-Configuration.md +++ b/Security-Configuration.md @@ -1,5 +1,34 @@ +# Understanding Communication Layers + +SeaweedFS has two separate communication layers: + +## gRPC Communication (Control Plane) +Used for metadata operations and cluster coordination: +- S3 ↔ Filer: Getting bucket info, file metadata +- Filer ↔ Master: Getting volume assignments, cluster info +- Volume ↔ Master: Heartbeats, volume management +- **Configured via**: `[grpc.*]` sections in security.toml + +## HTTP/HTTPS Communication (Data Plane) +Used for actual file data uploads/downloads: +- S3/Filer/Client → Volume Server: Uploading and downloading actual file content +- **Configured via**: `[https.*]` sections in security.toml + +## Configuration Sections Summary + +| Configuration Block | Purpose | +|---------------------|---------| +| `[grpc.client]`, `[grpc.filer]`, `[grpc.master]`, `[grpc.volume]` | Secure gRPC communication for metadata/control plane | +| `[https.client]` | Client-side configuration - tells S3/Filer/etc to use HTTPS when communicating with servers | +| `[https.volume]` | Server-side configuration - makes Volume server accept HTTPS for data uploads | +| `[https.master]`, `[https.filer]` | Server-side configuration - makes Master/Filer accept HTTPS (web UI, HTTP APIs) | + +**Important**: If you enable `[https.filer]` for the Filer server, you must also set `[https.client]` with `enabled = true` so that S3 API uses HTTPS when communicating with the Filer. Otherwise, you'll get "client sent an HTTP request to an HTTPS server" errors. + +# Configuration + The first step is generating `security.toml` file via `weed scaffold -config=security`: ``` @@ -116,18 +145,21 @@ key = "" ca = "" # volume server https options +# Configure these if you want the volume server to accept HTTPS connections [https.volume] cert = "" key = "" ca = "" # master server https options +# Configure these if you want the master server to accept HTTPS connections [https.master] cert = "" key = "" ca = "" # filer server https options +# Configure these if you want the filer server to accept HTTPS connections [https.filer] cert = "" key = "" From d57ccb04ddbeeedd3e0c707f91a12430015d82ba Mon Sep 17 00:00:00 2001 From: chrislusf Date: Tue, 25 Nov 2025 11:59:12 -0800 Subject: [PATCH 148/149] update --- S3-API-FAQ.md | 2 +- Security-Configuration.md | 75 ++++++++++++++++----------------------- Security-Overview.md | 33 +++++++++++++++++ 3 files changed, 64 insertions(+), 46 deletions(-) diff --git a/S3-API-FAQ.md b/S3-API-FAQ.md index 1728369..d233eaa 100644 --- a/S3-API-FAQ.md +++ b/S3-API-FAQ.md @@ -252,6 +252,6 @@ key = "/path/to/client.key" ca = "/path/to/ca.crt" ``` -The `[https.filer]` makes the Filer accept HTTPS, while `[https.client]` with `enabled = true` makes clients (S3 API, mount, etc.) use HTTPS for data operations. See [[Security-Configuration]] for details. +The `[https.filer]` makes the Filer accept HTTPS, while `[https.client]` with `enabled = true` makes clients (S3 API, mount, etc.) use HTTPS for data operations. See [[Security-Overview]] for architecture details and [[Security-Configuration]] for configuration reference. diff --git a/Security-Configuration.md b/Security-Configuration.md index 7593b88..d7bb16e 100644 --- a/Security-Configuration.md +++ b/Security-Configuration.md @@ -1,33 +1,6 @@ -# Understanding Communication Layers - -SeaweedFS has two separate communication layers: - -## gRPC Communication (Control Plane) -Used for metadata operations and cluster coordination: -- S3 ↔ Filer: Getting bucket info, file metadata -- Filer ↔ Master: Getting volume assignments, cluster info -- Volume ↔ Master: Heartbeats, volume management -- **Configured via**: `[grpc.*]` sections in security.toml - -## HTTP/HTTPS Communication (Data Plane) -Used for actual file data uploads/downloads: -- S3/Filer/Client → Volume Server: Uploading and downloading actual file content -- **Configured via**: `[https.*]` sections in security.toml - -## Configuration Sections Summary - -| Configuration Block | Purpose | -|---------------------|---------| -| `[grpc.client]`, `[grpc.filer]`, `[grpc.master]`, `[grpc.volume]` | Secure gRPC communication for metadata/control plane | -| `[https.client]` | Client-side configuration - tells S3/Filer/etc to use HTTPS when communicating with servers | -| `[https.volume]` | Server-side configuration - makes Volume server accept HTTPS for data uploads | -| `[https.master]`, `[https.filer]` | Server-side configuration - makes Master/Filer accept HTTPS (web UI, HTTP APIs) | - -**Important**: If you enable `[https.filer]` for the Filer server, you must also set `[https.client]` with `enabled = true` so that S3 API uses HTTPS when communicating with the Filer. Otherwise, you'll get "client sent an HTTP request to an HTTPS server" errors. - -# Configuration +For an overview of SeaweedFS security architecture and communication layers, see [[Security-Overview]]. The first step is generating `security.toml` file via `weed scaffold -config=security`: @@ -92,29 +65,38 @@ expires_after_seconds = 10 # seconds key = "" expires_after_seconds = 10 # seconds -# all grpc tls authentications are mutual -# the values for the following ca, cert, and key are paths to the PERM files. -# the host name is not checked, so the PERM files can be shared. +# gRPC mTLS configuration +# All gRPC TLS authentications are mutual (mTLS) +# The values for ca, cert, and key are paths to the certificate/key files +# The host name is not checked, so the certificate files can be shared [grpc] ca = "" # Set wildcard domain for enable TLS authentication by common names allowed_wildcard_domain = "" # .mycompany.com +# Volume server gRPC options (server-side) +# Enables mTLS for incoming gRPC connections to volume server [grpc.volume] cert = "" key = "" allowed_commonNames = "" # comma-separated SSL certificate common names +# Master server gRPC options (server-side) +# Enables mTLS for incoming gRPC connections to master server [grpc.master] cert = "" key = "" allowed_commonNames = "" # comma-separated SSL certificate common names +# Filer server gRPC options (server-side) +# Enables mTLS for incoming gRPC connections to filer server [grpc.filer] cert = "" key = "" allowed_commonNames = "" # comma-separated SSL certificate common names +# S3 server gRPC options (server-side) +# Enables mTLS for incoming gRPC connections to S3 server [grpc.s3] cert = "" key = "" @@ -130,36 +112,39 @@ cert = "" key = "" allowed_commonNames = "" # comma-separated SSL certificate common names -# use this for any place needs a grpc client -# i.e., "weed backup|benchmark|filer.copy|filer.replicate|mount|s3|upload" +# gRPC client configuration for outgoing gRPC connections +# Used by clients (S3, mount, backup, benchmark, filer.copy, filer.replicate, upload, etc.) +# when connecting to any gRPC server (master, volume, filer) [grpc.client] cert = "" key = "" -# https client for master|volume|filer|etc connection -# It is necessary that the parameters [https.volume]|[https.master]|[https.filer] are set +# HTTPS client configuration for outgoing HTTP connections +# Used by S3, mount, filer.copy, backup, and other clients when communicating with master/volume/filer +# Set enabled=true to use HTTPS instead of HTTP for data operations (separate from gRPC) +# If [https.filer] or [https.volume] are enabled on servers, clients must have [https.client] enabled=true [https.client] -enabled = false -cert = "" -key = "" -ca = "" +enabled = false # Set to true to enable HTTPS for all outgoing HTTP client connections +cert = "" # Client certificate for mTLS (optional if server doesn't require client cert) +key = "" # Client key for mTLS (optional if server doesn't require client cert) +ca = "" # CA certificate to verify server certificates (required when enabled=true) -# volume server https options -# Configure these if you want the volume server to accept HTTPS connections +# Volume server HTTPS options (server-side) +# Enables HTTPS for incoming HTTP connections to volume server [https.volume] cert = "" key = "" ca = "" -# master server https options -# Configure these if you want the master server to accept HTTPS connections +# Master server HTTPS options (server-side) +# Enables HTTPS for incoming HTTP connections to master server (web UI, HTTP API) [https.master] cert = "" key = "" ca = "" -# filer server https options -# Configure these if you want the filer server to accept HTTPS connections +# Filer server HTTPS options (server-side) +# Enables HTTPS for incoming HTTP connections to filer server (web UI, HTTP API) [https.filer] cert = "" key = "" diff --git a/Security-Overview.md b/Security-Overview.md index f6aac39..04833e5 100644 --- a/Security-Overview.md +++ b/Security-Overview.md @@ -2,6 +2,39 @@ Since SeaweedFS is a distributed system with many volume servers, the volume servers have the risk of being changed without proper access control. We want to have the freedom to place a volume server anywhere we want, with the confidence that nobody can tamper the data. +## Understanding Communication Layers + +SeaweedFS has two separate communication layers that must be secured independently: + +### gRPC Communication (Control Plane) +Used for metadata operations and cluster coordination: +- S3 ↔ Filer: Getting bucket info, file metadata +- Filer ↔ Master: Getting volume assignments, cluster info +- Volume ↔ Master: Heartbeats, volume management +- **Configured via**: `[grpc.*]` sections in security.toml + +### HTTP/HTTPS Communication (Data Plane) +Used for actual file data uploads/downloads: +- S3/Filer/Client → Volume Server: Uploading and downloading actual file content +- S3 → Filer: File data operations (when S3 acts as a proxy) +- **Configured via**: `[https.*]` sections in security.toml + +### Configuration Sections Summary + +| Configuration Block | Purpose | +|---------------------|---------| +| `[grpc.client]` | Used by clients (S3, mount, filer.copy, backup, etc.) to connect to any gRPC server | +| `[grpc.filer]`, `[grpc.master]`, `[grpc.volume]`, `[grpc.s3]` | Server-side gRPC mTLS - secures gRPC endpoints | +| `[https.client]` | Client-side HTTPS configuration - tells clients (S3/Filer/etc.) to use HTTPS for HTTP connections | +| `[https.volume]`, `[https.master]`, `[https.filer]` | Server-side HTTPS configuration - makes servers accept HTTPS connections | + +**Important**: +- `[https.client]` with `enabled = true` is required for clients to use HTTPS when connecting to HTTPS-enabled servers +- If you enable `[https.filer]` on the Filer server, you must also enable `[https.client]` so that S3 API can connect via HTTPS +- Otherwise, you'll get "client sent an HTTP request to an HTTPS server" errors + +## Security Features + We will address the volume servers first. The following items are not covered, yet: - master server http REST services From 7fb862b04bf9926178e5ae8bb5f0952bca811406 Mon Sep 17 00:00:00 2001 From: Lapo Luchini Date: Wed, 26 Nov 2025 10:54:12 +0100 Subject: [PATCH 149/149] Update Maven URL. --- SeaweedFS-Java-Client.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SeaweedFS-Java-Client.md b/SeaweedFS-Java-Client.md index 517d8d4..1a88e7d 100644 --- a/SeaweedFS-Java-Client.md +++ b/SeaweedFS-Java-Client.md @@ -26,7 +26,7 @@ Maven ``` Or you can download the latest version from MavenCentral -https://mvnrepository.com/artifact/com.github.chrislusf/seaweedfs-client +https://mvnrepository.com/artifact/com.seaweedfs/seaweedfs-client ## Features