From 33814e2a4d53200170dcefb2d8634d568833ac42 Mon Sep 17 00:00:00 2001 From: Brian <16049740+bddrey@users.noreply.github.com> Date: Sat, 17 Jun 2023 22:26:16 -0400 Subject: [PATCH 01/11] Spelling error --- Replication.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Replication.md b/Replication.md index aba3bf4..bcef2d2 100644 --- a/Replication.md +++ b/Replication.md @@ -81,12 +81,12 @@ When a write is made to the filer, there is an additional step before step 1. an ## Fix replication -If one replica is missing, there are no automatic repair right away. This is to prevent over replication due to transient volume sever failures or disconnections. In stead, the volume will just become readonly. For any new writes, just assign a different file id to a different volume. +If one replica is missing, there are no automatic repair right away. This is to prevent over replication due to transient volume sever failures or disconnections. In stead, the volume will just become read-only. For any new writes, just assign a different file id to a different volume. To repair the missing replicas, you can use `volume.fix.replication` in `weed shell`. ### Replicate without deleting -In certain circumstances—like adding/removing/altering replication settings of volumes or servers—the best strategy is to only repair under-replicated volumes and not delete any while working on volume and server modifications, in this citation use the flag `noDelete`: +In certain circumstances—like adding/removing/altering replication settings of volumes or servers—the best strategy is to only repair under-replicated volumes and not delete any while working on volume and server modifications, in this situation use the flag `noDelete`: `volume.fix.replication -noDelete` From 90c2f3ac278239a3e3fbbf425fdd303a20bd8b11 Mon Sep 17 00:00:00 2001 From: Damiano Albani Date: Sat, 24 Jun 2023 14:06:22 +0200 Subject: [PATCH 02/11] Document rclone storage backend --- Cloud-Tier.md | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/Cloud-Tier.md b/Cloud-Tier.md index 4baf523..87e9c7a 100644 --- a/Cloud-Tier.md +++ b/Cloud-Tier.md @@ -50,9 +50,14 @@ Compared to direct S3 storage, this is both faster and cheaper! 1. Use `volume.tier.download` in `weed shell` to move volumes to the local cluster. ## Configuring Storage Backend -(Currently only s3 is developed. More is coming soon.) -Multiple s3 buckets are supported. Usually you just need to configure one backend. +The following storage backends are currently supported: +- S3 +- Rclone + +### S3 + +Multiple S3 buckets are supported. Usually you just need to configure one backend. ``` # The storage backends are configured on the master, inside master.toml @@ -75,6 +80,26 @@ Multiple s3 buckets are supported. Usually you just need to configure one backen ``` +### Rclone + +Here's an example of a configuration for an Rclone storage backend: + +``` +[storage.backend] + [storage.backend.rclone.default] + enabled = true + remote_name = "my-onedrive" +``` + +Where `my-remote` must correspond to what's configured in `~/.config/rclone/rclone.conf`: +``` +[my-onedrive] +type = onedrive +... +``` + +### Configuration + After changing the master config, you need to **restart the volume servers** so that they pull the updated configuration from the master. If you forget to do this, you'll get an error message about no storage backends being configured. From e65e776a3c597d6ee584a24c2327d8cf4ee0aa18 Mon Sep 17 00:00:00 2001 From: Damiano Albani Date: Sat, 24 Jun 2023 17:03:58 +0200 Subject: [PATCH 03/11] Updated Cloud Tier (markdown) --- Cloud-Tier.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/Cloud-Tier.md b/Cloud-Tier.md index 87e9c7a..b45b1a6 100644 --- a/Cloud-Tier.md +++ b/Cloud-Tier.md @@ -89,6 +89,7 @@ Here's an example of a configuration for an Rclone storage backend: [storage.backend.rclone.default] enabled = true remote_name = "my-onedrive" + key_template = "SeaweedFS/{{ slice . 0 2 }}/{{ slice . 2 4 }}/{{ slice . 4 }}" # optional ``` Where `my-remote` must correspond to what's configured in `~/.config/rclone/rclone.conf`: @@ -98,6 +99,16 @@ type = onedrive ... ``` +The optional `key_template` property makes it possible to choose how the individual volume files are eventually named in the storage backend. +The template has to follow the [text/template](https://pkg.go.dev/text/template) syntax, where `.` is a UUID string representing the volume ID. +In this example, files would be named as such in OneDrive: +``` +SeaweedFS/ + 5f/ + e5/ + 03d8-0aba-4e42-b1c8-6116bf862f71 +``` + ### Configuration After changing the master config, you need to **restart the volume servers** so that they pull the updated configuration From 3d28ca2d3bf82a1d13ed42844db75dced496cc85 Mon Sep 17 00:00:00 2001 From: "Amir H. Yeganemehr" Date: Wed, 5 Jul 2023 21:49:30 +0330 Subject: [PATCH 04/11] Add unix domain socket upstream --- S3-Nginx-Proxy.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/S3-Nginx-Proxy.md b/S3-Nginx-Proxy.md index 527563d..8ea16fa 100644 --- a/S3-Nginx-Proxy.md +++ b/S3-Nginx-Proxy.md @@ -5,6 +5,9 @@ It's a common concept to put a proxy in front of S3 that handles requests. Nginx ``` upstream seaweedfs { server localhost:8333 fail_timeout=0; keepalive 20;} +## Also you can use unix domain socket instead for better performance: +# upstream seaweedfs { server unix:/tmp/seaweedfs-s3-8333.sock; keepalive 20;} + server { listen 443 ssl; server_name ~^(?[^.]+).yourdomain.com; From ea2824ebeac3a660aacdfca01fd3bb651a786cc3 Mon Sep 17 00:00:00 2001 From: qs-wang <32129299+qs-wang@users.noreply.github.com> Date: Thu, 13 Jul 2023 09:35:43 +0800 Subject: [PATCH 05/11] Updated Filer Change Data Capture (markdown) --- Filer-Change-Data-Capture.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Filer-Change-Data-Capture.md b/Filer-Change-Data-Capture.md index e44e139..11f0f6f 100644 --- a/Filer-Change-Data-Capture.md +++ b/Filer-Change-Data-Capture.md @@ -104,6 +104,9 @@ This is based on Filer gRPC API. You should be able to easily implement it in yo https://github.com/seaweedfs/seaweedfs/blob/master/weed/pb/filer.proto#L52 +A Golang example: +https://github.com/tuxmart/seawolf + # Possible Use Cases This is basically stream processing or event processing for files. The possible use cases are all up to your imagination. From 58dc7dcea09adc6b3555588ead48e412a7d24de6 Mon Sep 17 00:00:00 2001 From: Glowry Date: Tue, 18 Jul 2023 16:12:50 +0800 Subject: [PATCH 06/11] Updated Tiered Storage (markdown) --- Tiered-Storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Tiered-Storage.md b/Tiered-Storage.md index 3b1ae70..69fda26 100644 --- a/Tiered-Storage.md +++ b/Tiered-Storage.md @@ -13,7 +13,7 @@ Since data that are hot, warm, and cold, it would be cost-efficient to place dat To write any data, the volume index needs to append one entry. To read any data, the volume index lookup is required. The volume index can be in memory mode or an LevelDB instance. The amount of index content is small while it is accessed frequently. -You can run `weed volume -dir.idx=/fast/disk/dir` or `weed server -volume.dir.idx=/vast/disk/dir` to ensure the volume index is located on a fast disk, e.g., a fast SSD mount. +You can run `weed volume -dir.idx=/fast/disk/dir` or `weed server -volume.dir.idx=/fast/disk/dir` to ensure the volume index is located on a fast disk, e.g., a fast SSD mount. If the volume server already has some existing data, you can just stop the volume server, move the `.idx` files to the index folder, and restart the volume server. From 72c7623a5ac2302bab7aa60da0ecb5cee315883f Mon Sep 17 00:00:00 2001 From: Glowry Date: Thu, 20 Jul 2023 16:39:44 +0800 Subject: [PATCH 07/11] Updated Cloud Drive Quick Setup (markdown) --- Cloud-Drive-Quick-Setup.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Cloud-Drive-Quick-Setup.md b/Cloud-Drive-Quick-Setup.md index 8e89cb0..0af1363 100644 --- a/Cloud-Drive-Quick-Setup.md +++ b/Cloud-Drive-Quick-Setup.md @@ -50,7 +50,8 @@ In `weed shell`: "s3SecretKey": "***", "s3Region": "us-east-2" } - +# For aliyun OSS +> remote.configure -name=s5 -type=aliyun -aliyun.access_key=xxx -aliyun.secret_key=yyy -aliyun.endpoint=http://oss-cn-wulanchabu-internal.aliyuncs.com -aliyun.region=oss-cn-wulanchabu -s3.storage_class=STANDARD -s3.support_tagging=false ``` # Mount Remote Storage From 280901325f67e98f92713fac12523ee99ad71df5 Mon Sep 17 00:00:00 2001 From: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.com> Date: Mon, 24 Jul 2023 15:39:05 +0500 Subject: [PATCH 08/11] Created rclone with SeaweedFS (markdown) --- rclone-with-SeaweedFS.md | 74 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 rclone-with-SeaweedFS.md diff --git a/rclone-with-SeaweedFS.md b/rclone-with-SeaweedFS.md new file mode 100644 index 0000000..3da659c --- /dev/null +++ b/rclone-with-SeaweedFS.md @@ -0,0 +1,74 @@ +### Installation +See https://rclone.org/install/ + +On mac: `brew install rclone` + +### Configuration +See https://rclone.org/s3/ + +Set config cat ~/.config/rclone/rclone.conf: +``` +[swfs] +type = s3 +provider = Other +access_key_id = any-key-id +secret_access_key = any-access-key +endpoint = http://localhost:8333 +upload_cutoff = 50Mi +chunk_size = 50Mi +force_path_style = true +``` + +### Execute commands + +copy files +``` +rclone --log-level INFO copy --checksum --fast-list /Users/kmlebedev/files swfs:/bucket-name/files +``` + + +## Client-side encryption + +### Installation local KMS API +``` +git clone https://github.com/kmlebedev/local-kms.git +cd local-kms +go install +``` + +run local-kms +``` +local-kms +INFO[2023-07-24 15:34:23.876] Local KMS Version Unknown (Commit Hash Unknown) +INFO[2023-07-24 15:34:23.992] No file found at path /init/seed.yaml; skipping seeding. +INFO[2023-07-24 15:34:23.992] Data will be stored in /tmp/local-kms +INFO[2023-07-24 15:34:23.992] Local KMS started on 0.0.0.0:8080 +``` + +create master key +``` +aws kms create-key --endpoint=http://localhost:8080 +{ + "KeyMetadata": { + "AWSAccountId": "111122223333", + "KeyId": "5beb0309-d1ec-45ea-895a-52bbecbc8bde", + "Arn": "arn:aws:kms:eu-west-2:111122223333:key/5beb0309-d1ec-45ea-895a-52bbecbc8bde", + "CreationDate": "2023-07-03T14:24:36+05:00", + "Enabled": true, + "KeyUsage": "ENCRYPT_DECRYPT", + "KeyState": "Enabled", + "Origin": "AWS_KMS", + "KeyManager": "CUSTOMER", + "CustomerMasterKeySpec": "SYMMETRIC_DEFAULT", + "KeySpec": "SYMMETRIC_DEFAULT", + "EncryptionAlgorithms": [ + "SYMMETRIC_DEFAULT" + ] + } +} +``` + +### Copy files with encryption +``` +rclone --log-level INFO copy --cse-kms-master-key-id 5beb0309-d1ec-45ea-895a-52bbecbc8bde --kms-endpoint http://localhost:8080 --ignore-size --checksum --fast-list /Users/kmlebedev/files swfs:/bucket-name/files +``` From 2aa6df1bb53af44159ee167fe4f01c4d0d7f9bfe Mon Sep 17 00:00:00 2001 From: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.com> Date: Mon, 24 Jul 2023 15:39:53 +0500 Subject: [PATCH 09/11] Updated _Sidebar (markdown) --- _Sidebar.md | 1 + 1 file changed, 1 insertion(+) diff --git a/_Sidebar.md b/_Sidebar.md index b476929..3aa8679 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -53,6 +53,7 @@ * [[Amazon S3 API]] * [[AWS CLI with SeaweedFS]] * [[s3cmd with SeaweedFS]] + * [[rclone with SeaweedFS]] * [[restic with SeaweedFS]] * [[nodejs with Seaweed S3]] * [[S3 API Benchmark]] From a40114f93b52de010f9af625d13a2f1ac49bee43 Mon Sep 17 00:00:00 2001 From: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.com> Date: Mon, 24 Jul 2023 16:04:14 +0500 Subject: [PATCH 10/11] Updated rclone with SeaweedFS (markdown) --- rclone-with-SeaweedFS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rclone-with-SeaweedFS.md b/rclone-with-SeaweedFS.md index 3da659c..97a1b8f 100644 --- a/rclone-with-SeaweedFS.md +++ b/rclone-with-SeaweedFS.md @@ -6,7 +6,7 @@ On mac: `brew install rclone` ### Configuration See https://rclone.org/s3/ -Set config cat ~/.config/rclone/rclone.conf: +Set config ~/.config/rclone/rclone.conf: ``` [swfs] type = s3 From 6472aeaccec8352bf347e7efe2f7bdedaf350ef7 Mon Sep 17 00:00:00 2001 From: cooper lee Date: Sun, 13 Aug 2023 04:30:55 +0800 Subject: [PATCH 11/11] add a Openresty client --- Client-Libraries.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Client-Libraries.md b/Client-Libraries.md index e5415fc..448a883 100644 --- a/Client-Libraries.md +++ b/Client-Libraries.md @@ -33,6 +33,7 @@ Name | Author | Language [Erlang SeaweedFS Client](https://github.com/Neurotec/seaweedfs.erl) | Neurotec | Erlang [Julia SeaweedFS Client](https://github.com/lawless-m/SeaweedFSClient.jl) | Lawless-m | Julia [Rust SeaweedFS Client](https://github.com/kerzeld/rusty_weed) | kerzeld | Rust +[Openresty Client](https://github.com/cooperlyt/lua-resty-seaweedfs) | Cooper | Lua ## GRPC APIs SeaweedFS uses GRPC internally. You can use them too. Just check https://github.com/seaweedfs/seaweedfs/tree/master/weed/pb for the proto files.