Commit Graph
39 Commits
Author SHA1 Message Date
LHHDZ 36b5b713ba fix deadlock caused by message chan blocked (#5639) 2024-06-03 07:42:40 -07:00
LHHDZ 3730772e05 refresh max volume count after adjust (#4662)
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-07-10 23:37:33 -07:00
LHHDZ 8e80ce9476 fix key corrupt when fs.configure copy path trie (#4353)
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-03-29 11:29:44 -07:00
LHHDZ d695119073 The filer does not set defaultReplication to the defaultReplication o… (#4315)
The filer does not set defaultReplication to the defaultReplication of the master (it is not necessary, because if the filer is not set, the configuration of the master will be used when calling Assign), otherwise, when the defaultReplication of the master is modified and all master nodes are restarted, the defaultReplication will not take effect (because the filer Keep the previous defaultReplication setting from the master)

Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-03-15 22:10:39 -07:00
LHHDZ db5515eada Fix s3api_object_list_handlers returning contents less than the specified limit when more data actually exists (#4240)
Fix when the stored data is actually enough but s3api_object_list_handlers returns less than the specified limit

Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-02-21 23:08:52 -08:00
LHHDZ f84c13dabd fix response not allowed (#4057) 2022-12-13 05:48:50 -08:00
LHHDZ bb289d7d69 sort tags by key (#4047)
Signed-off-by: changlin.shi <changlin.shi@ly.com>

Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-12-11 22:49:57 -08:00
LHHDZ 39f0246deb fix 'NoSuchUpload' when upload part (#3968) 2022-11-11 04:53:53 -08:00
LHHDZ a5b867af69 don't skip empty value at AssembleEntryWithAcp (#3855)
* add acl helper functionalities

Signed-off-by: changlin.shi <changlin.shi@ly.com>

* add tests

Signed-off-by: changlin.shi <changlin.shi@ly.com>

* remove 0 when create map

Signed-off-by: changlin.shi <changlin.shi@ly.com>

* delete when empty at `AssembleEntryWithAcp`

`PutBucketAcl/PutObjectAcl` allow request with empty grants, `AssembleEntryWithAcp`  shouldn't skip empty value

Signed-off-by: changlin.shi <changlin.shi@ly.com>

Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-10-13 22:50:44 -07:00
LHHDZ bf5e45b66a add acl helper functionalities (#3831) 2022-10-12 20:15:16 -07:00
LHHDZ d21e2f523d split ExtAcpKey to ExtAmzOwnerKey and ExtAmzAclKey to avoid unn… (#3824)
split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unnecessary `json.Unmarshal()` call

Signed-off-by: changlin.shi <changlin.shi@ly.com>

Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-10-11 20:14:14 -07:00
LHHDZ 3550692afc change s3_account.go package to avoid cycle dependency (#3813) 2022-10-10 06:44:29 -07:00
LHHDZ e9584d9661 add ownership rest apis (#3765) 2022-10-01 19:18:00 -07:00
LHHDZ 3de1e19780 s3: sync bucket info from filer (#3759) 2022-09-29 12:29:01 -07:00
LHHDZ e00a12b099 associate Account and Identity by accountId (#3754) 2022-09-28 13:25:59 -07:00
LHHDZ aacdcc4cad s3: add account (#3753)
associate `Account` and `Identity` by accountId
2022-09-28 12:45:18 -07:00
LHHDZ 6d2000e8da fix 'localFilerSocket' nil pointer (#3654) 2022-09-14 06:33:00 -07:00
LHHDZ bc629665de fix bug due to data racing on VidMap (#3606) 2022-09-05 20:00:16 -07:00
LHHDZ 84ec68e11a Add download speed limit support (#3408) 2022-08-05 01:16:42 -07:00
LHHDZ 994a2dec78 fix vid_map_test tests 2022-07-22 17:54:57 +08:00
LHHDZ 58c02d6429 Solve the problem that LookupFileId lookup urls is empty due to leader switching
The vidMap structure is modified to a linked list structure (the length is limited to 5). When the vidMap is reset, the current vidMap is added to the new vidMap as a cache node. When the query locations is empty, the cache node is searched to avoid problems when the master switches leaders.
2022-07-22 17:22:38 +08:00
LHHDZ c08f6d0ff7 File upload should succeed should return ETag instead of Etag
at `/go-1.18.1/src/net/textproto/header.go:39`, header is canonicalized by `textproto.CanonicalMIMEHeaderKey`
2022-07-14 19:45:01 +08:00
石昌林 392da4e038 Make the prompt information clearer 2022-07-13 11:39:45 +08:00
石昌林 ad5e8f68ec Check whether there is a duplicate accessKey when modifying iam 2022-07-13 11:36:11 +08:00
石昌林 200a862701 fixed volume xx not found caused by missing VolumeLocation events
When the requested master node is the leader, `VolumeLocation` or `ClusterNodeUpdate` may be returned here. If it is `VolumeLocation`, the update will be performed while resetting the vidMap, otherwise the event will be lost
2022-06-30 13:41:56 +08:00
石昌林 2b5b37231b Make sure that onChangeEvent is called successfully before updating lastTsNs 2022-06-29 23:57:07 +08:00
石昌林 e17e6ed542 Avoid synchronizing from scratch due to err 2022-06-29 23:31:04 +08:00
石昌林 e1b94eb6b9 fix error: concurrent map writes when add or remove cluster node 2022-06-28 10:34:59 +08:00
石昌林 15bfc60a88 Remove the collection query param of s3api and let the collection be determined by the filer 2022-06-27 13:47:53 +08:00
石昌林 f1ea906c13 Code comment optimization 2022-06-25 11:56:09 +08:00
石昌林 0c8e033e68 When the connection with the leader is disconnected, the vidMap should not be reset immediately, but should be reset after connecting to a new leader 2022-06-24 17:41:46 +08:00
石昌林 81f7f08708 Determine whether to preallocate according to the master configuration before executing copy volume 2022-06-20 21:12:44 +08:00
石昌林 be5c901bea Fix global counter key is incorrect & Add read lock when reading counter map to avoid memory problems 2022-06-20 19:16:53 +08:00
石昌林 bd63b305dd update go.sum
go.mod has been changed but go.sum has not been updated accordingly, causing the program to fail to start:
../../pkg/mod/google.golang.org/api@v0.83.0/option/option.go:12:2: missing go.sum entry for module providing package golang.org/x/oauth2 (imported by google.golang.org/api/option); to add:
	go get google.golang.org/api/option@v0.83.0
../../pkg/mod/google.golang.org/api@v0.83.0/option/option.go:13:2: missing go.sum entry for module providing package golang.org/x/oauth2/google (imported by cloud.google.com/go/storage); to add:
	go get cloud.google.com/go/storage@v1.22.1
../../pkg/mod/github.com/chrislusf/raft@v1.0.8/protobuf/seaweed_raft.pb.go:12:2: missing go.sum entry for module providing package google.golang.org/grpc (imported by github.com/chrislusf/seaweedfs/weed/command); to add:
	go get github.com/chrislusf/seaweedfs/weed/command
2022-06-20 14:35:04 +08:00
LHHDZ 8db9f13bc6 Merge branch 'master' into circuit_breaker 2022-06-20 13:42:18 +08:00
石昌林 9e036df356 remove go.uber.org/atomic 2022-06-20 12:35:29 +08:00
石昌林 3dd60529c5 some code optimizations 2022-06-17 19:07:39 +08:00
石昌林 37df209195 add some unit tests and some code optimizes 2022-06-17 17:11:18 +08:00
石昌林 78b3728169 add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply

# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply

# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply

# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply

# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
2022-06-15 21:07:55 +08:00