mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-20 13:30:46 +02:00
87332eb60b
* pb: add multipart concurrency fields to RemoteConf and tier move requests RemoteConf gains upload_concurrency/download_concurrency (0 = client default); VolumeTierMoveDatToRemote/FromRemote requests gain a concurrency field (0 = backend default). * remote storage: honor RemoteConf upload/download concurrency in s3 and azure clients s3 client: ReadFile passes conf download_concurrency to the downloader, WriteFile uses upload_concurrency for the uploader; previously hard-coded 1 upload / 5 download parts. 0 keeps defaults. Same for azure client. * storage: plumb concurrency through backend interface and tier upload/download BackendStorage.CopyFile/DownloadFile take a concurrency hint (<=0 = backend configured default); s3 backend reads upload_concurrency/download_concurrency from scaffold config with parseConcurrency fallback, rclone updated to the new signature. Tier move gRPC handlers forward the request concurrency to the backend. * shell: -upload_concurrency/-download_concurrency for remote.configure, -concurrent for volume.tier remote.configure exposes upload/download concurrency persisted into RemoteConf; volume.tier move/evict commands forward -concurrent to the tier move requests. Documented in master-cloud.toml scaffold. * test: cover concurrency propagation in remote tier integration test * remote.configure: merge existing config on partial update Load the stored RemoteConf before saving so a partial update (e.g. only -upload_concurrency) preserves credentials, endpoints, and type instead of replacing them with new-config defaults. Only treat a confirmed ErrNotFound as a new configuration; propagate all other load errors so a transient filer failure does not overwrite stored settings. On a type transition, reset backend-specific fields to the destination type's new-config defaults rather than inheriting the old backend's empty values. Bound configured concurrency to a sane maximum. * remote storage: honor configured download concurrency in S3 and Azure ReadFileWithConcurrency now resolves a zero request override against the client's configured download_concurrency (new downloadConcurrency() helpers), so the remote-mount/cache read path honors RemoteConf.DownloadConcurrency instead of the hard-coded default. Azure also clamps the resolved value to math.MaxUint16 regardless of whether the fallback was used, preventing uint16 wraparound when a configured value exceeds 65535. * shell: rename -concurrent to -concurrency and validate tier transfer bounds Rename the -concurrent flag to -concurrency across volume.tier.upload, volume.tier.download, and volume.tier.compact to match the proto field and RemoteConf field names. Add validateTierConcurrency to reject values that would wrap int32 or exceed a 1024 cap before constructing the request. * server: clamp tier move concurrency in gRPC handlers Add clampTierConcurrency to both VolumeTierMoveDatToRemote and VolumeTierMoveDatFromRemote handlers so a direct gRPC caller cannot spawn an unbounded number of network workers. * trim verbose comments added with concurrency feature Remove redundant doc comments on the backend interface, rclone backend, s3_backend parseConcurrency, and test helpers that restated the obvious. * remote.configure: apply type defaults before re-parse so explicit flags win applyTypeDefaults ran after the second flag parse, overwriting explicit destination flags (e.g. -s3.region=eu-west-1) with new-config defaults. Move the type-transition default reset before the re-parse so user-supplied flags override the destination defaults. * remote.configure: only treat explicit -type as a type transition The first parse defaults -type to s3, so a concurrency-only update on an existing non-S3 config captured requestedType=s3 and wrongly triggered a type transition, resetting the stored backend to S3. Use fs.Visit to detect whether -type was explicitly supplied; an omitted -type keeps the stored backend. --------- Co-authored-by: Jack Meredith <9480542+jackusm@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>
84 lines
2.4 KiB
Protocol Buffer
84 lines
2.4 KiB
Protocol Buffer
syntax = "proto3";
|
|
|
|
package remote_pb;
|
|
|
|
option go_package = "github.com/seaweedfs/seaweedfs/weed/pb/remote_pb";
|
|
option java_package = "seaweedfs.client";
|
|
option java_outer_classname = "FilerProto";
|
|
|
|
/////////////////////////
|
|
// Remote Storage related
|
|
/////////////////////////
|
|
message RemoteConf {
|
|
string type = 1;
|
|
string name = 2;
|
|
string s3_access_key = 4;
|
|
string s3_secret_key = 5;
|
|
string s3_region = 6;
|
|
string s3_endpoint = 7;
|
|
string s3_storage_class = 8;
|
|
bool s3_force_path_style = 9;
|
|
bool s3_support_tagging = 13;
|
|
bool s3_v4_signature = 11;
|
|
|
|
string gcs_google_application_credentials = 10;
|
|
string gcs_project_id = 12;
|
|
|
|
string azure_account_name = 15;
|
|
string azure_account_key = 16;
|
|
string azure_client_id = 17;
|
|
string azure_endpoint = 18;
|
|
|
|
string backblaze_key_id = 20;
|
|
string backblaze_application_key = 21;
|
|
string backblaze_endpoint = 22;
|
|
string backblaze_region = 23;
|
|
|
|
string aliyun_access_key = 25;
|
|
string aliyun_secret_key = 26;
|
|
string aliyun_endpoint = 27;
|
|
string aliyun_region = 28;
|
|
|
|
string tencent_secret_id = 30;
|
|
string tencent_secret_key = 31;
|
|
string tencent_endpoint = 32;
|
|
|
|
string baidu_access_key = 35;
|
|
string baidu_secret_key = 36;
|
|
string baidu_endpoint = 37;
|
|
string baidu_region = 38;
|
|
|
|
string wasabi_access_key = 40;
|
|
string wasabi_secret_key = 41;
|
|
string wasabi_endpoint = 42;
|
|
string wasabi_region = 43;
|
|
|
|
string filebase_access_key = 60;
|
|
string filebase_secret_key = 61;
|
|
string filebase_endpoint = 62;
|
|
|
|
string storj_access_key = 65;
|
|
string storj_secret_key = 66;
|
|
string storj_endpoint = 67;
|
|
|
|
string contabo_access_key = 68;
|
|
string contabo_secret_key = 69;
|
|
string contabo_endpoint = 70;
|
|
string contabo_region = 71;
|
|
|
|
uint32 upload_concurrency = 72; // multipart upload concurrency per file, 0 = client default (1 for S3, 16 for Azure)
|
|
uint32 download_concurrency = 73; // multipart download concurrency per read, 0 = client default (5 for S3, 16 for Azure)
|
|
}
|
|
|
|
message RemoteStorageMapping {
|
|
map<string,RemoteStorageLocation> mappings = 1;
|
|
string primary_bucket_storage_name = 2;
|
|
}
|
|
message RemoteStorageLocation {
|
|
string name = 1;
|
|
string bucket = 2;
|
|
string path = 3;
|
|
int32 listing_cache_ttl_seconds = 4; // 0 = disabled; >0 enables on-demand directory listing with this TTL in seconds
|
|
optional int32 cache_wait_ms = 5; // unset = size based default; 0 = read straight from the remote without caching
|
|
}
|