Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-10-14 10:08:35 +02:00
Code Issues Packages Projects Releases Wiki Activity
10,942 Commits 202 Branches 361 Tags
67a252ee8a0cbcf0f33cb7c94de21d4791ef1f39
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Konstantin Lebedev 67a252ee8a [master] refactor func ShouldGrowVolumes (#5884) 2024-09-04 08:16:44 -07:00
Konstantin Lebedev 3e6ca6e706 [webdav] status code 500 if internal error from filer (#5865) 2024-08-06 07:39:52 -07:00
Konstantin Lebedev ef4c555e90 [shell] volume.grow (#5855) 2024-08-05 09:22:17 -07:00
Konstantin Lebedev b2ffcdaab2 [master] do sync grow request only if absolutely necessary (#5821) 
* do sync grow request only if absolutely necessary
https://github.com/seaweedfs/seaweedfs/pull/5819

* remove check VolumeGrowStrategy Threshold on PickForWrite

* fix fmt.Errorf
 2024-07-30 13:21:35 -07:00
Konstantin Lebedev e1f9336688 [filer] fix panic filer on webdav.maxMB (#5822) 
fix panic filer on webdav.maxMB
 2024-07-27 13:40:22 -07:00
Konstantin Lebedev 67edf1d014 [master] Do Automatic Volume Grow in background (#5781) 
* Do Automatic Volume Grow in backgound

* pass lastGrowCount to master

* fix build

* fix count to uint64
 2024-07-16 08:03:40 -07:00
Konstantin Lebedev a53e406c99 [master] refactor HasGrowRequest to atomic bool (#5782) 
refactor HasGrowRequest to atomit bool
 2024-07-15 10:51:21 -07:00
Konstantin Lebedev 33964fa292 metrics stats of volume layout depends on the data center (#5775) 
stats volume layout depends on the data center
 2024-07-12 12:32:25 -07:00
Konstantin Lebedev 04f4b10884 fix: avoid timeout if datacenter does not exist in topology (#5772) 
* fix: avoid timeout if datacenter does not exist in topology

* fix: error msg

* fix: rm dublicate check

* fix: compare

* revert minor change
 2024-07-12 11:19:08 -07:00
Konstantin Lebedev 9c440d472c [s3] revert skip deletion error, since the error file was not found is already skipped on the side of the grpc function (#5760) 
* revert skip deletion error, since the error file was not found is already skipped on the side of the grpc function

* fix response error

* fix test_lifecycle_get

* Revert "fix test_lifecycle_get"

This reverts commit 8f991bdcf9.
 2024-07-10 08:47:43 -07:00
Konstantin Lebedev f77eee667d add s3test for sql (#5718) 
* add s3test for sql

* fix test test_bucket_listv2_delimiter_basic for s3

* fix action s3tests

* regen s3 api xsd

* rm minor s3 test test_bucket_listv2_fetchowner_defaultempty

* add docs

* without xmlns
 2024-07-04 11:00:41 -07:00
Konstantin Lebedev 5ffacbb6ea refactor all methods strings to const (#5726) 2024-07-01 01:00:39 -07:00
Konstantin Lebedev 34165013c8 chore: hot fix panic fs.verify (#5696) 2024-06-20 15:24:31 -07:00
Konstantin Lebedev 7988ee0805 [fs.verify] skip failed files if entry not found on filerStore (#5693) 2024-06-19 06:50:55 -07:00
Konstantin Lebedev ece018b1a9 [fs.verify] getmeta data from filer log (#5678) 
* initial

* fix: print entry path
 2024-06-18 22:56:47 -07:00
Konstantin Lebedev d389c5b27e fix: recreate index include deleted files (#5579) 
* fix: recreate index include deleted files
https://github.com/seaweedfs/seaweedfs/issues/5508

* fix: counting the number of files

* fix: log
 2024-05-12 11:31:34 -07:00
Konstantin Lebedev 6f9267ebdf fix: skip file size checking for volume with TTL (#5541) 
fix: skip file size checking for volume with tll
https://github.com/seaweedfs/seaweedfs/issues/5508
 2024-04-28 00:21:57 -07:00
Konstantin Lebedev 3d3ee04eb9 [s3] Put bucket lifecycle configuration (#5510) 2024-04-27 07:39:22 -07:00
Konstantin Lebedev 2f3fee9bb9 filer read chunk retry if status code 499 (#5528) 2024-04-24 06:51:05 -07:00
Konstantin Lebedev 33537ae29f [s3] fix s3 test_multipart_get_part (#5476) 
* try fix s3  test_multipart_get_part

* add passed s3 tests

* fix SeaweedFSUploadId

* rm spaces

* convert part request to range

* add passed s3 tests of multipart
 2024-04-14 10:41:32 -07:00
Konstantin Lebedev 5189a09de0 [volume] Reduce the number of buffers for uploading one chunk (#5458) 2024-04-11 04:47:21 -07:00
Konstantin Lebedev 3e25ed1b11 [s3] add s3 pass test_multipart_upload_size_too_small (#5475) 
* add s3 pass test_multipart_upload_size_too_small

* refactor metric names

* return ErrNoSuchUpload if empty parts

* fix test
 2024-04-07 11:52:35 -07:00
Konstantin Lebedev 35cba720a5 [s3] add s3 pass test_multipart_upload (#5474) 
add s3 pass test_multipart_upload
 2024-04-07 11:51:22 -07:00
Konstantin Lebedev d42a04cceb [s3] fix s3 test_multipart_resend_first_finishes_last (#5471) 
* try fix s3 test
https://github.com/seaweedfs/seaweedfs/pull/5466

* add error handler metrics

* refactor

* refactor multipartExt

* delete bad entry parts
 2024-04-06 10:56:39 -07:00
Konstantin Lebedev d5d8b8e2ae fix panic at isAllWritable (#5457) 
fix panic
https://github.com/seaweedfs/seaweedfs/issues/5456
 2024-04-02 09:06:19 -07:00
Konstantin Lebedev f9ec92dd83 [tests] add action s3 tests (#5427) 2024-03-29 07:57:27 -07:00
Konstantin Lebedev 9c1e0f5811 [master] grow volumes if no writable volumes in current dataCenter (#5434) 
* grow volumes if no writable volumes in current dataCenter
https://github.com/seaweedfs/seaweedfs/issues/3886

* fix tests with volume grow

* automatic volume grow one volume

* add ErrorChunkAssign metrics
 2024-03-29 00:38:27 -07:00
Konstantin Lebedev df40908e57 fix panic 5435 (#5436) 2024-03-28 16:17:59 -07:00
Konstantin Lebedev dc9568fc0d [master] add test for PickForWrite add metrics for volume layout (#5413) 2024-03-22 07:39:11 -07:00
Konstantin Lebedev 25643cfbc6 fix: panic: assignment to entry in nil map on S3Sink.CreateEntry (#5406) 2024-03-21 08:12:31 -07:00
Konstantin Lebedev d821cb3b18 fix: sync without dir /buckets/some/.uploads/hash_hash (#5402) 2024-03-20 12:54:29 -07:00
Konstantin Lebedev 54ee73287d [filer] replace an invalid methods in a metric with an invalid one (#5378) 
fix: replace an invalid methods in a metric with an invalid one for filer
 2024-03-14 08:19:09 -07:00
Konstantin Lebedev 170b63d6f8 [filer.backup] add param uploader_part_size for S3sink (#5352) 
* fix: install cronie

* chore: refactor configure S3Sink

* chore: refactor cinfig

* add filer-backup compose file

* fix: X-Amz-Meta-Mtime and resolve with comments

* fix: attr mtime

* fix: MaxUploadPartst is reduced to the maximum allowable

* fix: env and force set max MaxUploadParts

* fix: env WEED_SINK_S3_UPLOADER_PART_SIZE_MB
 2024-03-07 08:35:51 -08:00
Konstantin Lebedev 872b5161e6 [shell] fs log purge (#5344) 
* fix: install cronie

* chore: shell delete old logs

* Revert "fix: install cronie"

This reverts commit 920832b04b.

* fix: dayAgo

* fix: Nit: dayAgo => daysAgo
 2024-03-05 09:09:08 -08:00
Konstantin Lebedev 7187346cc1 avoid unexpected compact size (#5272) 
https://github.com/seaweedfs/seaweedfs/issues/5215
 2024-02-24 05:27:35 -08:00
Konstantin Lebedev 6181aa7594 fix: fs verify error counter (#5261) 2024-02-15 01:31:51 -08:00
Konstantin Lebedev 8fb978d6f6 [volume] requests that were canceled by the client are not an internal server problem(500 => 499) (#5304) 2024-02-14 14:18:56 -08:00
Konstantin Lebedev 0f8e76bbd6 fix: clean metric MasterReplicaPlacementMismatch for unregister volume (#5239) 2024-01-25 00:23:24 -08:00
Konstantin Lebedev 0b71fae2e3 fix: s3tests test_get_object_ifnonematch_good (#5237) 2024-01-24 11:05:55 -08:00
Konstantin Lebedev 4e9ea1e628 chore: filer healthz handler check filer store (#5208) 2024-01-17 07:46:37 -08:00
Konstantin Lebedev f9cf13fada [filer] avoid return http ok for not allowed methods (#5209) 2024-01-17 07:17:07 -08:00
Konstantin Lebedev 8d23e36c45 fix: doDeleteFiles deletes files (#5198) 2024-01-12 11:04:29 -08:00
Konstantin Lebedev 1169f94310 Fix filer sync set offset (#5197) 
* fix: compose 2mount with sync

* fix: DATA RACE
https://github.com/seaweedfs/seaweedfs/issues/5194
https://github.com/seaweedfs/seaweedfs/issues/5195
 2024-01-12 10:57:18 -08:00
Konstantin Lebedev a7fc723ae0 chore: add status code for request_total metrics (#5188) 2024-01-10 10:05:27 -08:00
Konstantin Lebedev b9d32d32e1 chore: filer sync add doDeleteFiles option for create only mode (#5166) 2024-01-06 10:02:16 -08:00
Konstantin Lebedev 3b62b338a5 chore: add maxMB option for webdav (#5165) 2024-01-05 11:10:20 -08:00
Konstantin Lebedev 4f6172f369 Fix missing VersionConfiguration node in get-bucket-versioning response (#5162) 
* fix: s3 response for get bucket version
https://github.com/seaweedfs/seaweedfs/issues/5155

* fix: s3 response for get bucket version
https://github.com/seaweedfs/seaweedfs/issues/5155
 2024-01-03 12:27:13 -08:00
Konstantin Lebedev 3e43778869 fix: webdav avoid create empty files (#5160) 2024-01-03 09:52:51 -08:00
Konstantin Lebedev b832ddd1ef fix: return etag with md5 in webdav responses (#5158) 2024-01-03 08:50:32 -08:00
Konstantin Lebedev 183352c796 shell meta load add concurrency (#4529) 
* fix: increase speed cmd fs meta load

* fix: add wg
 2023-12-28 11:54:37 -08:00
Konstantin Lebedev 3c9bcfb864 chore: add dsn for connection to mysql (#5060) 
* chore: add dsn for connection to mysql

* add comment

* new comment

* fix: validate dsn and adapt password
 2023-12-20 16:20:58 -08:00
Konstantin Lebedev 125ad8fe63 falls back to update only if error contains msg "duplicate entry" 
https://github.com/seaweedfs/seaweedfs/issues/5062
 2023-12-09 10:18:18 -08:00
Konstantin Lebedev 2b229e98ce fix: add doDeleteFile option for filer backup 2023-11-17 07:37:28 -08:00
Konstantin Lebedev e55fa68848 fix: excludeFileName is empty 2023-11-13 06:23:46 -08:00
Konstantin Lebedev 3c5295a1a6 filer backup add option for exclude file names that match the regexp to sync on filer 2023-11-13 06:23:46 -08:00
Konstantin Lebedev 5e41ab1370 s3 api add not implemented response for PutBucketVersioning 2023-11-13 06:22:23 -08:00
Konstantin Lebedev 7f6152f130 s3 api add default response for GetBucketVersioning 2023-11-13 06:22:23 -08:00
Konstantin Lebedev d693beea87 Replace an invalid method in a metric with an invalid one (#4987) 2023-11-06 10:51:47 -08:00
Konstantin LebedevKonstantin LebedevChris Lu
 		6c82d7ddfc [s3] do reload s3 static config (#4923) 
* do reload s3 config

* print error on reload s3 config

* print success msg

* Update weed/s3api/s3api_server.go

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-10-18 23:26:49 -07:00
Konstantin LebedevandKonstantin Lebedev 6175102ccf add make benchmark with WARP (#4914) 
add make benchmark

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-15 23:02:27 -07:00
Konstantin Lebedev ce9b822a8e golang up version to 1.21.3 (#4911) 2023-10-13 05:31:43 -07:00
Konstantin LebedevandKonstantin Lebedev 1cac5d983d fix: disallow file name too long when writing a file (#4881) 
* fix: disallow file name too long when writing a file

* bool LongerName to MaxFilenameLength

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-12 14:29:55 -07:00
Konstantin LebedevandKonstantin Lebedev edee91ef0e fix has changes false if deleted result size eq zero (#4909) 
* fix has changes false if deleted result size eq zero
https://github.com/seaweedfs/seaweedfs/issues/4171

* fix change respos status 202 to 304 if the file has already been deleted
https://github.com/seaweedfs/seaweedfs/issues/4171

* fix delete if set apply changes
https://github.com/seaweedfs/seaweedfs/issues/4171

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-12 08:15:20 -07:00
Konstantin LebedevandKonstantin Lebedev 2b3e39397e fix: skipping checking active volumes with the same number of files at the moment (#4893) 
* fix: skipping checking active volumes with the same number of files at the moment
 https://github.com/seaweedfs/seaweedfs/issues/4140

* refactor with comments
https://github.com/seaweedfs/seaweedfs/issues/4140

* add TestShouldSkipVolume

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-09 09:57:26 -07:00
Konstantin Lebedev dffe00a822 fix: logger place msg (#4880) 2023-10-02 08:29:09 -07:00
Konstantin LebedevChris LuKonstantin Lebedev
 		dd580190b4 fix: avoid deleting one replica without sync (#4875) 
* fix: avoid deleting one replica without sync
https://github.com/seaweedfs/seaweedfs/issues/4647

* Update weed/shell/command_volume_fix_replication.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* fix: revert this existing do option to positive

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-09-27 23:12:10 -07:00
Konstantin Lebedev 44906f1f3b fix: avoid error file name too long when writing a file (#4876) 2023-09-27 05:40:51 -07:00
Konstantin LebedevandKonstantin Lebedev df4ded758e fix: avoid deleting more than one replica (#4873) 
https://github.com/seaweedfs/seaweedfs/issues/4647

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-09-26 00:20:48 -07:00
Konstantin LebedevKonstantin LebedevChris Lu
 		f8b94cac0e [s3acl] Step1: move s3account.AccountManager into to iam.S3ApiConfiguration (#4859) 
* move s3account.AccountManager into to iam.S3ApiConfiguration and switch to Interface

https://github.com/seaweedfs/seaweedfs/issues/4519

* fix: test bucket acl default and
adjust the variable names

* fix: s3 api config test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-09-25 08:34:12 -07:00
Konstantin Lebedev a40888d343 fix: make dev_tls with volume https (#4866) 2023-09-25 06:47:38 -07:00
Konstantin LebedevandKonstantin Lebedev 750a0ba1b2 [iam] Replace action read/write to readAcp/writeAcp for handlers with acl (#4858) 
Replace action read/write to readAcp/writeAcp for handlers with acl query
 https://github.com/seaweedfs/seaweedfs/issues/4519

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-09-21 14:07:04 -07:00
Konstantin LebedevKonstantin LebedevChris Lu
 		a46f873edd [s3acl] Step 0: Put bucket ACL only responds success if the ACL is private. (#4856) 
* Passing test:
test_bucket_acl_default
test_bucket_acl_canned_private_to_private

https://github.com/seaweedfs/seaweedfs/issues/4519

* Update weed/s3api/s3api_bucket_handlers.go

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-09-21 08:20:05 -07:00
Konstantin LebedevandKonstantin Lebedev d8b424d123 [s3] optimization iam lookup for reducing algorithm complexity (#4857) 
optimization iam lookup for reducing algorithm complexity
https://github.com/seaweedfs/seaweedfs/issues/4519

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-09-21 08:19:11 -07:00
Konstantin LebedevandKonstantin Lebedev 2c076ebd96 avoid logging nil error (#4659) 
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-07-10 23:38:52 -07:00
Konstantin LebedevandKonstantin Lebedev 0515014149 chart with optional https port for s3 (#4507) 
https://github.com/seaweedfs/seaweedfs/pull/4482

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-20 09:43:20 -07:00
Konstantin Lebedev 1e22d5caf2 fix get file stats for IsEmpty (#4576) 2023-06-14 01:43:30 -07:00
Konstantin LebedevandKonstantin Lebedev 4dd890d4a2 optional https port for s3 (#4482) 
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-13 23:58:49 -07:00
Konstantin LebedevandKonstantin Lebedev 4527ead295 fix from comment delete volume is empty (#4573) 
* fix from coments
https://github.com/seaweedfs/seaweedfs/pull/4561

* fix tests

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-12 22:22:46 -07:00
Konstantin LebedevandKonstantin Lebedev 25535e9c36 Delete volume is empty (#4561) 
* use onlyEmpty for deleteVolume
https://github.com/seaweedfs/seaweedfs/issues/4559

* fix IsEmpty

* fix test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-12 10:42:44 -07:00
Konstantin Lebedev 5ee04d20fa Healthz check for deadlocks (#4558) 2023-06-09 09:42:48 -07:00
Konstantin LebedevandKonstantin Lebedev 5caff2c0f0 fix: deletes newly created volumes (#4551) 
https://github.com/seaweedfs/seaweedfs/issues/4531

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-06 09:08:27 -07:00
Konstantin LebedevandKonstantin Lebedev a0931be0c0 S3 TLS credentials Refreshing (#4506) 
* S3 TLS credentials Refreshing

* fix: logging

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-04 14:27:56 -07:00
Konstantin Lebedev d75a7b7f62 allow deleting only older empty dir without recursion (#4430) 2023-04-25 08:31:14 -07:00
Konstantin LebedevandKonstantin Lebedev 5614ad0000 fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter (#4399) 
* fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter

* fix list with delimiter and start token

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-04-12 16:53:49 -07:00
Konstantin Lebedev 88b2bbadd1 try fix s3test test_bucket_listv2_delimiter_prefix (#4396) 2023-04-12 08:47:20 -07:00
Konstantin LebedevandKonstantin Lebedev 095cdb0c7f fix test prefix is a directory (#4393) 
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-04-12 00:08:13 -07:00
Konstantin LebedevandKonstantin Lebedev 44ad07276c s3 fix get list of dir object key with slash suffix (#4391) 
* s3 fix get list of dir object key with slash suffix
https://github.com/seaweedfs/seaweedfs/issues/3086

* list only entry dir eq prefix

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-04-11 23:52:35 -07:00
Konstantin Lebedev 130bc3e668 s3 fix get fake dir object key (#4390) 2023-04-11 07:36:22 -07:00
Konstantin Lebedev de4545c28b add validate config for raft (#4332) 2023-03-21 06:36:13 -07:00
Konstantin LebedevKonstantin LebedevChris Lu
 		29d1312625 Fs.verify.concurrency (#4293) 
* init

* refactor

* refactor print msg

* param just concurrency

* fix race

* next fix race

* reset variables before each run

* clean

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-03-19 18:38:57 -07:00
Konstantin Lebedev e17429223e shell script unclean variables (#4298) 2023-03-13 07:16:31 -07:00
Konstantin Lebedev 569bc7b54d fs.verify needles status of file (#4060) 
* fs.verify needles status of file

* fix parse args

* refactor logging

* set modifyTimeAgoAtSec
 2022-12-21 15:54:17 -08:00
Konstantin Lebedev f59c3acd5f volume.list show volumes only from the specified dc/rack/dn (#4024) 2022-11-30 06:30:55 -08:00
Konstantin Lebedev 2b910d1cf8 avoid recursive deleting newly created empty directories (#4016) 2022-11-25 08:45:47 -08:00
Konstantin Lebedev 5431c445cd fix filer.remote.sync to azure with ContentType (#3949) 
* fix filer.remote.sync to azure with ContentType

* fix pass X-Amz-Meta to X-Ms-Meta
 2022-11-04 09:10:33 -07:00
Konstantin Lebedev 0999f9b7ff [volume.fsck] collect ids without cut off time for finding missing data from volumes (#3934) 
collect all file ids from the file without cut off time for finding missing data from volumes
 2022-10-31 11:38:12 -07:00
Konstantin LebedevandChris Lu a322ba042e [volume.fsck] param volumeId is comma separated the volume id (#3933) 
volume.fsck param volumeId is comma separated the volume id

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-10-31 11:36:26 -07:00
Konstantin Lebedev 4a48332248 refactor error contains already deleted (#3932) 2022-10-31 11:34:45 -07:00
Konstantin Lebedev c0deaa4948 [volume.fsck] check needles status from volume server (#3926) 
check needles status from volume server
 2022-10-31 11:33:04 -07:00
Konstantin Lebedev 409c9328de [master] avoid vacuum if not enough replica copies (#3924) 
avoid vacuum if not enough replica copies
 2022-10-30 20:34:19 -07:00