Github-Mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-08 15:41:15 +02:00
Code Issues Packages Projects Releases Wiki Activity
11,247 Commits 177 Branches 359 Tags
c5f21b2b01deb10a542455b95285860a53f1f4d0
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Konstantin Lebedev 0a4b1909a2 [shell] only apply the balancing for writable volumes (#6346) 2024-12-13 01:10:00 -08:00
Konstantin Lebedev 700b95304b [shell] volume.list show only writable volumes (#6338) 
* show only writable volumes

* fix import
 2024-12-11 09:06:15 -08:00
Konstantin Lebedev c37281735e volume.list avoid output empty data center and rack and disk info (#6341) 2024-12-11 09:03:16 -08:00
Konstantin Lebedev ff1392f7f4 [shell] use constant for hdd of type (#6337) 
use constant for hdd of type
 2024-12-10 08:43:59 -08:00
Konstantin Lebedev b65eb2ec45 [security] reload whiteList on http seerver (#6302) 
* reload whiteList

* white_list add to scaffold
 2024-12-02 10:38:10 -08:00
Konstantin Lebedev e2e97db917 [master] avoid timeout when assigning for main request with filter by DC or rack (#6291) 
* avoid timeout when assigning for main request with filter by DC or rack

https://github.com/seaweedfs/seaweedfs/issues/6290

* use constant NoWritableVolumes
 2024-11-26 08:33:31 -08:00
Konstantin Lebedev fec88e64eb [master] update LastLeaderChangeTime for hashicorp raft (#6292) 2024-11-26 08:02:45 -08:00
Konstantin Lebedev 8836fa19b6 use ShouldGrowVolumesByDcAndRack (#6280) 2024-11-25 09:30:37 -08:00
Konstantin Lebedev 167b50be88 fix missing register master metric MasterPickForWriteErrorCounter (#6277) 2024-11-25 08:59:11 -08:00
Konstantin Lebedev 7fa4e5c2a1 [master] Revert back automatic volume grow in case of compensate for crowded volumes (#6267) 
brought back the Automatic Volume Grow to compensate for crowded volumes
 2024-11-21 08:19:19 -08:00
Konstantin Lebedev a143c888e5 [shell] don't require lock when there are no changes for volume.fix.replication (#6266) 
* don't require lock when there are no changes

* revert takeAction
 2024-11-21 08:17:25 -08:00
Konstantin Lebedev a49d9e020c [master] avoid crowded more writable for auto grow (#6214) 
avoid crowded more writable
https://github.com/seaweedfs/seaweedfs/issues/6121
 2024-11-21 00:57:42 -08:00
Konstantin Lebedev 254ed8897e [shell] add noLock param for volume.move (#6261) 2024-11-20 08:35:24 -08:00
Konstantin Lebedev 9a5d3e7b31 [shell] add admin noLock for balance (#6209) 
add admin noLock for balance
 2024-11-05 19:09:22 -08:00
Konstantin Lebedev 5bddf0c085 [shell] volume.balance collect volume servers by dc rack node (#6191) 
* chore: balance by rack

* fix: rm check lock

* fix: selected racks

* fix: selected nodes

* fix: containts

* fix: one collectVolumeServersByDcRackNode

* fix: revert lock and add lock

* fix: panic test

* revert noLock
 2024-11-03 11:08:45 -08:00
Konstantin LebedevandChris Lu 44b275879b [s3] add {Get,Put,Delete}BucketTagging and PublicAccessBlock Handlers (#6088) 
* add {Get,Put,Delete}BucketTagging Handlers

* s3 add skip bucket PublicAccessBlock handlers

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2024-10-04 10:59:14 -07:00
Konstantin Lebedev a3556acd98 [s3] add skip bucket encryption handlers (#6091) 
s3 add skip bucket encryption handlers
 2024-10-03 08:10:43 -07:00
Konstantin Lebedev bae93e611d [filer] avoid 500 if table doesn't exist (#6075) 2024-09-27 07:15:54 -07:00
Konstantin Lebedev 6d4f5c94ac [volume] remove truncate idx file if size not healthy (#6043) 2024-09-19 01:04:08 -07:00
Konstantin Lebedev 2b13d2c083 [filer] fix return 204 for DELETE on entry not found (#6023) 
fix return 204 for DELETE on entry not found
 2024-09-16 23:11:03 -07:00
Konstantin Lebedev 34bbaa2cdd [master] process grow request with must grow (#5999) 
process grow request with must grow
 2024-09-09 23:45:02 -07:00
Konstantin Lebedev 15965f7c54 [shell] fix volume grow in shell (#5992) 
* fix volume grow in shell

* revert add Async

* check available volume space

* create a VolumeGrowRequest and remove unnecessary fields
 2024-09-09 11:42:56 -07:00
Konstantin Lebedev 16984ef07a [s3] reducing the number of metrics for 403 (#5961) 
reducing the number of metrics
 2024-09-04 11:06:54 -07:00
Konstantin Lebedev 67a252ee8a [master] refactor func ShouldGrowVolumes (#5884) 2024-09-04 08:16:44 -07:00
Konstantin Lebedev 3e6ca6e706 [webdav] status code 500 if internal error from filer (#5865) 2024-08-06 07:39:52 -07:00
Konstantin Lebedev ef4c555e90 [shell] volume.grow (#5855) 2024-08-05 09:22:17 -07:00
Konstantin Lebedev b2ffcdaab2 [master] do sync grow request only if absolutely necessary (#5821) 
* do sync grow request only if absolutely necessary
https://github.com/seaweedfs/seaweedfs/pull/5819

* remove check VolumeGrowStrategy Threshold on PickForWrite

* fix fmt.Errorf
 2024-07-30 13:21:35 -07:00
Konstantin Lebedev e1f9336688 [filer] fix panic filer on webdav.maxMB (#5822) 
fix panic filer on webdav.maxMB
 2024-07-27 13:40:22 -07:00
Konstantin Lebedev 67edf1d014 [master] Do Automatic Volume Grow in background (#5781) 
* Do Automatic Volume Grow in backgound

* pass lastGrowCount to master

* fix build

* fix count to uint64
 2024-07-16 08:03:40 -07:00
Konstantin Lebedev a53e406c99 [master] refactor HasGrowRequest to atomic bool (#5782) 
refactor HasGrowRequest to atomit bool
 2024-07-15 10:51:21 -07:00
Konstantin Lebedev 33964fa292 metrics stats of volume layout depends on the data center (#5775) 
stats volume layout depends on the data center
 2024-07-12 12:32:25 -07:00
Konstantin Lebedev 04f4b10884 fix: avoid timeout if datacenter does not exist in topology (#5772) 
* fix: avoid timeout if datacenter does not exist in topology

* fix: error msg

* fix: rm dublicate check

* fix: compare

* revert minor change
 2024-07-12 11:19:08 -07:00
Konstantin Lebedev 9c440d472c [s3] revert skip deletion error, since the error file was not found is already skipped on the side of the grpc function (#5760) 
* revert skip deletion error, since the error file was not found is already skipped on the side of the grpc function

* fix response error

* fix test_lifecycle_get

* Revert "fix test_lifecycle_get"

This reverts commit 8f991bdcf9.
 2024-07-10 08:47:43 -07:00
Konstantin Lebedev f77eee667d add s3test for sql (#5718) 
* add s3test for sql

* fix test test_bucket_listv2_delimiter_basic for s3

* fix action s3tests

* regen s3 api xsd

* rm minor s3 test test_bucket_listv2_fetchowner_defaultempty

* add docs

* without xmlns
 2024-07-04 11:00:41 -07:00
Konstantin Lebedev 5ffacbb6ea refactor all methods strings to const (#5726) 2024-07-01 01:00:39 -07:00
Konstantin Lebedev 34165013c8 chore: hot fix panic fs.verify (#5696) 2024-06-20 15:24:31 -07:00
Konstantin Lebedev 7988ee0805 [fs.verify] skip failed files if entry not found on filerStore (#5693) 2024-06-19 06:50:55 -07:00
Konstantin Lebedev ece018b1a9 [fs.verify] getmeta data from filer log (#5678) 
* initial

* fix: print entry path
 2024-06-18 22:56:47 -07:00
Konstantin Lebedev d389c5b27e fix: recreate index include deleted files (#5579) 
* fix: recreate index include deleted files
https://github.com/seaweedfs/seaweedfs/issues/5508

* fix: counting the number of files

* fix: log
 2024-05-12 11:31:34 -07:00
Konstantin Lebedev 6f9267ebdf fix: skip file size checking for volume with TTL (#5541) 
fix: skip file size checking for volume with tll
https://github.com/seaweedfs/seaweedfs/issues/5508
 2024-04-28 00:21:57 -07:00
Konstantin Lebedev 3d3ee04eb9 [s3] Put bucket lifecycle configuration (#5510) 2024-04-27 07:39:22 -07:00
Konstantin Lebedev 2f3fee9bb9 filer read chunk retry if status code 499 (#5528) 2024-04-24 06:51:05 -07:00
Konstantin Lebedev 33537ae29f [s3] fix s3 test_multipart_get_part (#5476) 
* try fix s3  test_multipart_get_part

* add passed s3 tests

* fix SeaweedFSUploadId

* rm spaces

* convert part request to range

* add passed s3 tests of multipart
 2024-04-14 10:41:32 -07:00
Konstantin Lebedev 5189a09de0 [volume] Reduce the number of buffers for uploading one chunk (#5458) 2024-04-11 04:47:21 -07:00
Konstantin Lebedev 3e25ed1b11 [s3] add s3 pass test_multipart_upload_size_too_small (#5475) 
* add s3 pass test_multipart_upload_size_too_small

* refactor metric names

* return ErrNoSuchUpload if empty parts

* fix test
 2024-04-07 11:52:35 -07:00
Konstantin Lebedev 35cba720a5 [s3] add s3 pass test_multipart_upload (#5474) 
add s3 pass test_multipart_upload
 2024-04-07 11:51:22 -07:00
Konstantin Lebedev d42a04cceb [s3] fix s3 test_multipart_resend_first_finishes_last (#5471) 
* try fix s3 test
https://github.com/seaweedfs/seaweedfs/pull/5466

* add error handler metrics

* refactor

* refactor multipartExt

* delete bad entry parts
 2024-04-06 10:56:39 -07:00
Konstantin Lebedev d5d8b8e2ae fix panic at isAllWritable (#5457) 
fix panic
https://github.com/seaweedfs/seaweedfs/issues/5456
 2024-04-02 09:06:19 -07:00
Konstantin Lebedev f9ec92dd83 [tests] add action s3 tests (#5427) 2024-03-29 07:57:27 -07:00
Konstantin Lebedev 9c1e0f5811 [master] grow volumes if no writable volumes in current dataCenter (#5434) 
* grow volumes if no writable volumes in current dataCenter
https://github.com/seaweedfs/seaweedfs/issues/3886

* fix tests with volume grow

* automatic volume grow one volume

* add ErrorChunkAssign metrics
 2024-03-29 00:38:27 -07:00
Konstantin Lebedev df40908e57 fix panic 5435 (#5436) 2024-03-28 16:17:59 -07:00
Konstantin Lebedev dc9568fc0d [master] add test for PickForWrite add metrics for volume layout (#5413) 2024-03-22 07:39:11 -07:00
Konstantin Lebedev 25643cfbc6 fix: panic: assignment to entry in nil map on S3Sink.CreateEntry (#5406) 2024-03-21 08:12:31 -07:00
Konstantin Lebedev d821cb3b18 fix: sync without dir /buckets/some/.uploads/hash_hash (#5402) 2024-03-20 12:54:29 -07:00
Konstantin Lebedev 54ee73287d [filer] replace an invalid methods in a metric with an invalid one (#5378) 
fix: replace an invalid methods in a metric with an invalid one for filer
 2024-03-14 08:19:09 -07:00
Konstantin Lebedev 170b63d6f8 [filer.backup] add param uploader_part_size for S3sink (#5352) 
* fix: install cronie

* chore: refactor configure S3Sink

* chore: refactor cinfig

* add filer-backup compose file

* fix: X-Amz-Meta-Mtime and resolve with comments

* fix: attr mtime

* fix: MaxUploadPartst is reduced to the maximum allowable

* fix: env and force set max MaxUploadParts

* fix: env WEED_SINK_S3_UPLOADER_PART_SIZE_MB
 2024-03-07 08:35:51 -08:00
Konstantin Lebedev 872b5161e6 [shell] fs log purge (#5344) 
* fix: install cronie

* chore: shell delete old logs

* Revert "fix: install cronie"

This reverts commit 920832b04b.

* fix: dayAgo

* fix: Nit: dayAgo => daysAgo
 2024-03-05 09:09:08 -08:00
Konstantin Lebedev 7187346cc1 avoid unexpected compact size (#5272) 
https://github.com/seaweedfs/seaweedfs/issues/5215
 2024-02-24 05:27:35 -08:00
Konstantin Lebedev 6181aa7594 fix: fs verify error counter (#5261) 2024-02-15 01:31:51 -08:00
Konstantin Lebedev 8fb978d6f6 [volume] requests that were canceled by the client are not an internal server problem(500 => 499) (#5304) 2024-02-14 14:18:56 -08:00
Konstantin Lebedev 0f8e76bbd6 fix: clean metric MasterReplicaPlacementMismatch for unregister volume (#5239) 2024-01-25 00:23:24 -08:00
Konstantin Lebedev 0b71fae2e3 fix: s3tests test_get_object_ifnonematch_good (#5237) 2024-01-24 11:05:55 -08:00
Konstantin Lebedev 4e9ea1e628 chore: filer healthz handler check filer store (#5208) 2024-01-17 07:46:37 -08:00
Konstantin Lebedev f9cf13fada [filer] avoid return http ok for not allowed methods (#5209) 2024-01-17 07:17:07 -08:00
Konstantin Lebedev 8d23e36c45 fix: doDeleteFiles deletes files (#5198) 2024-01-12 11:04:29 -08:00
Konstantin Lebedev 1169f94310 Fix filer sync set offset (#5197) 
* fix: compose 2mount with sync

* fix: DATA RACE
https://github.com/seaweedfs/seaweedfs/issues/5194
https://github.com/seaweedfs/seaweedfs/issues/5195
 2024-01-12 10:57:18 -08:00
Konstantin Lebedev a7fc723ae0 chore: add status code for request_total metrics (#5188) 2024-01-10 10:05:27 -08:00
Konstantin Lebedev b9d32d32e1 chore: filer sync add doDeleteFiles option for create only mode (#5166) 2024-01-06 10:02:16 -08:00
Konstantin Lebedev 3b62b338a5 chore: add maxMB option for webdav (#5165) 2024-01-05 11:10:20 -08:00
Konstantin Lebedev 4f6172f369 Fix missing VersionConfiguration node in get-bucket-versioning response (#5162) 
* fix: s3 response for get bucket version
https://github.com/seaweedfs/seaweedfs/issues/5155

* fix: s3 response for get bucket version
https://github.com/seaweedfs/seaweedfs/issues/5155
 2024-01-03 12:27:13 -08:00
Konstantin Lebedev 3e43778869 fix: webdav avoid create empty files (#5160) 2024-01-03 09:52:51 -08:00
Konstantin Lebedev b832ddd1ef fix: return etag with md5 in webdav responses (#5158) 2024-01-03 08:50:32 -08:00
Konstantin Lebedev 183352c796 shell meta load add concurrency (#4529) 
* fix: increase speed cmd fs meta load

* fix: add wg
 2023-12-28 11:54:37 -08:00
Konstantin Lebedev 3c9bcfb864 chore: add dsn for connection to mysql (#5060) 
* chore: add dsn for connection to mysql

* add comment

* new comment

* fix: validate dsn and adapt password
 2023-12-20 16:20:58 -08:00
Konstantin Lebedev 125ad8fe63 falls back to update only if error contains msg "duplicate entry" 
https://github.com/seaweedfs/seaweedfs/issues/5062
 2023-12-09 10:18:18 -08:00
Konstantin Lebedev dd78397fea logging PickForWrite error 
https://github.com/seaweedfs/seaweedfs/issues/3886
 2023-11-27 12:56:15 -08:00
Konstantin Lebedev 2b229e98ce fix: add doDeleteFile option for filer backup 2023-11-17 07:37:28 -08:00
Konstantin Lebedev e55fa68848 fix: excludeFileName is empty 2023-11-13 06:23:46 -08:00
Konstantin Lebedev 3c5295a1a6 filer backup add option for exclude file names that match the regexp to sync on filer 2023-11-13 06:23:46 -08:00
Konstantin Lebedev 5e41ab1370 s3 api add not implemented response for PutBucketVersioning 2023-11-13 06:22:23 -08:00
Konstantin Lebedev 7f6152f130 s3 api add default response for GetBucketVersioning 2023-11-13 06:22:23 -08:00
Konstantin Lebedev d693beea87 Replace an invalid method in a metric with an invalid one (#4987) 2023-11-06 10:51:47 -08:00
Konstantin LebedevKonstantin LebedevChris Lu
 		6c82d7ddfc [s3] do reload s3 static config (#4923) 
* do reload s3 config

* print error on reload s3 config

* print success msg

* Update weed/s3api/s3api_server.go

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-10-18 23:26:49 -07:00
Konstantin LebedevandKonstantin Lebedev 6175102ccf add make benchmark with WARP (#4914) 
add make benchmark

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-15 23:02:27 -07:00
Konstantin Lebedev ce9b822a8e golang up version to 1.21.3 (#4911) 2023-10-13 05:31:43 -07:00
Konstantin LebedevandKonstantin Lebedev 1cac5d983d fix: disallow file name too long when writing a file (#4881) 
* fix: disallow file name too long when writing a file

* bool LongerName to MaxFilenameLength

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-12 14:29:55 -07:00
Konstantin LebedevandKonstantin Lebedev edee91ef0e fix has changes false if deleted result size eq zero (#4909) 
* fix has changes false if deleted result size eq zero
https://github.com/seaweedfs/seaweedfs/issues/4171

* fix change respos status 202 to 304 if the file has already been deleted
https://github.com/seaweedfs/seaweedfs/issues/4171

* fix delete if set apply changes
https://github.com/seaweedfs/seaweedfs/issues/4171

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-12 08:15:20 -07:00
Konstantin LebedevandKonstantin Lebedev 2b3e39397e fix: skipping checking active volumes with the same number of files at the moment (#4893) 
* fix: skipping checking active volumes with the same number of files at the moment
 https://github.com/seaweedfs/seaweedfs/issues/4140

* refactor with comments
https://github.com/seaweedfs/seaweedfs/issues/4140

* add TestShouldSkipVolume

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-09 09:57:26 -07:00
Konstantin Lebedev dffe00a822 fix: logger place msg (#4880) 2023-10-02 08:29:09 -07:00
Konstantin LebedevChris LuKonstantin Lebedev
 		dd580190b4 fix: avoid deleting one replica without sync (#4875) 
* fix: avoid deleting one replica without sync
https://github.com/seaweedfs/seaweedfs/issues/4647

* Update weed/shell/command_volume_fix_replication.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* fix: revert this existing do option to positive

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-09-27 23:12:10 -07:00
Konstantin Lebedev 44906f1f3b fix: avoid error file name too long when writing a file (#4876) 2023-09-27 05:40:51 -07:00
Konstantin LebedevandKonstantin Lebedev df4ded758e fix: avoid deleting more than one replica (#4873) 
https://github.com/seaweedfs/seaweedfs/issues/4647

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-09-26 00:20:48 -07:00
Konstantin LebedevKonstantin LebedevChris Lu
 		f8b94cac0e [s3acl] Step1: move s3account.AccountManager into to iam.S3ApiConfiguration (#4859) 
* move s3account.AccountManager into to iam.S3ApiConfiguration and switch to Interface

https://github.com/seaweedfs/seaweedfs/issues/4519

* fix: test bucket acl default and
adjust the variable names

* fix: s3 api config test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-09-25 08:34:12 -07:00
Konstantin Lebedev a40888d343 fix: make dev_tls with volume https (#4866) 2023-09-25 06:47:38 -07:00
Konstantin LebedevandKonstantin Lebedev 750a0ba1b2 [iam] Replace action read/write to readAcp/writeAcp for handlers with acl (#4858) 
Replace action read/write to readAcp/writeAcp for handlers with acl query
 https://github.com/seaweedfs/seaweedfs/issues/4519

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-09-21 14:07:04 -07:00
Konstantin LebedevKonstantin LebedevChris Lu
 		a46f873edd [s3acl] Step 0: Put bucket ACL only responds success if the ACL is private. (#4856) 
* Passing test:
test_bucket_acl_default
test_bucket_acl_canned_private_to_private

https://github.com/seaweedfs/seaweedfs/issues/4519

* Update weed/s3api/s3api_bucket_handlers.go

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-09-21 08:20:05 -07:00
Konstantin LebedevandKonstantin Lebedev d8b424d123 [s3] optimization iam lookup for reducing algorithm complexity (#4857) 
optimization iam lookup for reducing algorithm complexity
https://github.com/seaweedfs/seaweedfs/issues/4519

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-09-21 08:19:11 -07:00
Konstantin LebedevandKonstantin Lebedev 2c076ebd96 avoid logging nil error (#4659) 
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-07-10 23:38:52 -07:00
Konstantin LebedevandKonstantin Lebedev 0515014149 chart with optional https port for s3 (#4507) 
https://github.com/seaweedfs/seaweedfs/pull/4482

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-20 09:43:20 -07:00
Konstantin Lebedev 1e22d5caf2 fix get file stats for IsEmpty (#4576) 2023-06-14 01:43:30 -07:00