Github-Mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-08 15:41:15 +02:00
Code Issues Packages Projects Releases Wiki Activity
9,740 Commits 177 Branches 359 Tags
f8b94cac0e91660ff4dc33bb458e973b4de6523b
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Konstantin LebedevKonstantin LebedevChris Lu
 		f8b94cac0e [s3acl] Step1: move s3account.AccountManager into to iam.S3ApiConfiguration (#4859) 
* move s3account.AccountManager into to iam.S3ApiConfiguration and switch to Interface

https://github.com/seaweedfs/seaweedfs/issues/4519

* fix: test bucket acl default and
adjust the variable names

* fix: s3 api config test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-09-25 08:34:12 -07:00
Konstantin Lebedev a40888d343 fix: make dev_tls with volume https (#4866) 2023-09-25 06:47:38 -07:00
Konstantin LebedevandKonstantin Lebedev 750a0ba1b2 [iam] Replace action read/write to readAcp/writeAcp for handlers with acl (#4858) 
Replace action read/write to readAcp/writeAcp for handlers with acl query
 https://github.com/seaweedfs/seaweedfs/issues/4519

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-09-21 14:07:04 -07:00
Konstantin LebedevKonstantin LebedevChris Lu
 		a46f873edd [s3acl] Step 0: Put bucket ACL only responds success if the ACL is private. (#4856) 
* Passing test:
test_bucket_acl_default
test_bucket_acl_canned_private_to_private

https://github.com/seaweedfs/seaweedfs/issues/4519

* Update weed/s3api/s3api_bucket_handlers.go

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-09-21 08:20:05 -07:00
Konstantin LebedevandKonstantin Lebedev d8b424d123 [s3] optimization iam lookup for reducing algorithm complexity (#4857) 
optimization iam lookup for reducing algorithm complexity
https://github.com/seaweedfs/seaweedfs/issues/4519

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-09-21 08:19:11 -07:00
Konstantin LebedevandKonstantin Lebedev 2c076ebd96 avoid logging nil error (#4659) 
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-07-10 23:38:52 -07:00
Konstantin LebedevandKonstantin Lebedev 0515014149 chart with optional https port for s3 (#4507) 
https://github.com/seaweedfs/seaweedfs/pull/4482

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-20 09:43:20 -07:00
Konstantin Lebedev 1e22d5caf2 fix get file stats for IsEmpty (#4576) 2023-06-14 01:43:30 -07:00
Konstantin LebedevandKonstantin Lebedev 4dd890d4a2 optional https port for s3 (#4482) 
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-13 23:58:49 -07:00
Konstantin LebedevandKonstantin Lebedev 4527ead295 fix from comment delete volume is empty (#4573) 
* fix from coments
https://github.com/seaweedfs/seaweedfs/pull/4561

* fix tests

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-12 22:22:46 -07:00
Konstantin LebedevandKonstantin Lebedev 25535e9c36 Delete volume is empty (#4561) 
* use onlyEmpty for deleteVolume
https://github.com/seaweedfs/seaweedfs/issues/4559

* fix IsEmpty

* fix test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-12 10:42:44 -07:00
Konstantin Lebedev 5ee04d20fa Healthz check for deadlocks (#4558) 2023-06-09 09:42:48 -07:00
Konstantin LebedevandKonstantin Lebedev 5caff2c0f0 fix: deletes newly created volumes (#4551) 
https://github.com/seaweedfs/seaweedfs/issues/4531

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-06 09:08:27 -07:00
Konstantin LebedevandKonstantin Lebedev a0931be0c0 S3 TLS credentials Refreshing (#4506) 
* S3 TLS credentials Refreshing

* fix: logging

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-04 14:27:56 -07:00
Konstantin Lebedev d75a7b7f62 allow deleting only older empty dir without recursion (#4430) 2023-04-25 08:31:14 -07:00
Konstantin LebedevandKonstantin Lebedev 5614ad0000 fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter (#4399) 
* fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter

* fix list with delimiter and start token

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-04-12 16:53:49 -07:00
Konstantin Lebedev 88b2bbadd1 try fix s3test test_bucket_listv2_delimiter_prefix (#4396) 2023-04-12 08:47:20 -07:00
Konstantin LebedevandKonstantin Lebedev 095cdb0c7f fix test prefix is a directory (#4393) 
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-04-12 00:08:13 -07:00
Konstantin LebedevandKonstantin Lebedev 44ad07276c s3 fix get list of dir object key with slash suffix (#4391) 
* s3 fix get list of dir object key with slash suffix
https://github.com/seaweedfs/seaweedfs/issues/3086

* list only entry dir eq prefix

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-04-11 23:52:35 -07:00
Konstantin Lebedev 130bc3e668 s3 fix get fake dir object key (#4390) 2023-04-11 07:36:22 -07:00
Konstantin Lebedev de4545c28b add validate config for raft (#4332) 2023-03-21 06:36:13 -07:00
Konstantin LebedevKonstantin LebedevChris Lu
 		29d1312625 Fs.verify.concurrency (#4293) 
* init

* refactor

* refactor print msg

* param just concurrency

* fix race

* next fix race

* reset variables before each run

* clean

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-03-19 18:38:57 -07:00
Konstantin Lebedev e17429223e shell script unclean variables (#4298) 2023-03-13 07:16:31 -07:00
Konstantin Lebedev 569bc7b54d fs.verify needles status of file (#4060) 
* fs.verify needles status of file

* fix parse args

* refactor logging

* set modifyTimeAgoAtSec
 2022-12-21 15:54:17 -08:00
Konstantin Lebedev f59c3acd5f volume.list show volumes only from the specified dc/rack/dn (#4024) 2022-11-30 06:30:55 -08:00
Konstantin Lebedev 2b910d1cf8 avoid recursive deleting newly created empty directories (#4016) 2022-11-25 08:45:47 -08:00
Konstantin Lebedev 5431c445cd fix filer.remote.sync to azure with ContentType (#3949) 
* fix filer.remote.sync to azure with ContentType

* fix pass X-Amz-Meta to X-Ms-Meta
 2022-11-04 09:10:33 -07:00
Konstantin Lebedev 0999f9b7ff [volume.fsck] collect ids without cut off time for finding missing data from volumes (#3934) 
collect all file ids from the file without cut off time for finding missing data from volumes
 2022-10-31 11:38:12 -07:00
Konstantin LebedevandChris Lu a322ba042e [volume.fsck] param volumeId is comma separated the volume id (#3933) 
volume.fsck param volumeId is comma separated the volume id

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-10-31 11:36:26 -07:00
Konstantin Lebedev 4a48332248 refactor error contains already deleted (#3932) 2022-10-31 11:34:45 -07:00
Konstantin Lebedev c0deaa4948 [volume.fsck] check needles status from volume server (#3926) 
check needles status from volume server
 2022-10-31 11:33:04 -07:00
Konstantin Lebedev 409c9328de [master] avoid vacuum if not enough replica copies (#3924) 
avoid vacuum if not enough replica copies
 2022-10-30 20:34:19 -07:00
Konstantin Lebedev bf8a9d2db1 [volume.chek.disk] sync of deletions the fix (#3923) 
* sync of deletions the fix

* avoid return if only partiallyDeletedNeedles

* refactor sync deletions
 2022-10-30 20:32:46 -07:00
Konstantin Lebedev 764d9cb105 [voluche.chek.disk] needles older than the cutoff time are not missing yet (#3922) 
needles older than the cutoff time are not missing yet

https://github.com/seaweedfs/seaweedfs/issues/3919
 2022-10-28 12:12:20 -07:00
Konstantin Lebedev f9f499b8d5 [fix] add param for ignore error (#3918) 2022-10-28 06:59:39 -07:00
Konstantin Lebedev c6f2899375 avoid fix missing of deleted needles (#3909) 2022-10-25 21:35:54 -07:00
Konstantin LebedevandChris Lu 6253058d9d ensure monotonic n.AppendAtNs in each place (#3880) 
https://github.com/seaweedfs/seaweedfs/issues/3852

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-10-23 18:24:52 -07:00
Konstantin Lebedev 452202f35b ReadNeedleMeta recovery panic occurred (#3883) 
ReadNeedleMeta panic occurred
https://github.com/seaweedfs/seaweedfs/issues/3882
 2022-10-23 14:00:28 -07:00
Konstantin Lebedev e20f0dbd2d avoid data race of TraverseBfs (#3856) 
* avoid data race of TraverseBfs

* close is enough
avoid panic
I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255
I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c]

goroutine 287 [running]:
github.com/seaweedfs/seaweedfs/weed/storage.(*SortedFileNeedleMap).Close(0x0)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load.func1()
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150
 2022-10-16 11:30:41 -07:00
Konstantin Lebedev ee38ab8581 avoid data race of LogBuffer isStopping (#3859) 2022-10-16 11:28:56 -07:00
Konstantin Lebedev 7836f7574e [volume.fsck] hotfix apply purging and add option verifyNeedle #3860 (#3861) 
* fix apply purging and add verifyNeedle

* common readSourceNeedleBlob

* use consts
 2022-10-15 20:38:46 -07:00
Konstantin Lebedev f476cf3403 fix volume_server.proto conflict (#3858) 
fix conflict

https://github.com/seaweedfs/seaweedfs/pull/3841
 2022-10-15 20:01:07 -07:00
Konstantin Lebedev 2f72103c83 avoid load volume file with BytesOffset mismatch (#3841) 
* avoid load volume file with BytesOffset mismatch

https://github.com/seaweedfs/seaweedfs/issues/2966

* set BytesOffset if has not VolumeInfoFile

* typos fail => failed

* exit if bytesOffset mismatch
 2022-10-14 00:18:09 -07:00
Konstantin Lebedev f19c9e3d9d Volume fsck by volume (#3851) 
* refactor

* refactor args verbose and writer

* refactor readFilerFileIdFile

* fix filter by collectMtime

* skip system log collection
 2022-10-13 23:30:30 -07:00
Konstantin Lebedev 1f7e52c63e vacuum metrics and force sync dst files (#3832) 2022-10-13 00:51:20 -07:00
Konstantin Lebedev 401315f337 master fix interruption through ctrl+c (#3834) 2022-10-12 07:18:40 -07:00
Konstantin Lebedev 5b28c3f728 revert disabling FSync for non Mac (#3814) 2022-10-10 07:28:02 -07:00
Konstantin Lebedev 5db25a8f2a metric shows who is currently blocking the cluster or not (#3799) 
* master_admin_lock Shows whether cluster is locked now or not
https://github.com/seaweedfs/seaweedfs/issues/3452

* fix metric MasterAdminLock
 2022-10-07 13:26:29 -07:00
Konstantin Lebedev a522507f95 configure raft metrics (#3798) 2022-10-07 04:20:34 -07:00
Konstantin Lebedev b9933d5589 master server graceful stop (#3797) 2022-10-06 09:30:30 -07:00
Konstantin Lebedev 36daa7709d show raft leader via shell (#3796) 2022-10-06 07:10:41 -07:00
Konstantin Lebedev ec46a34f33 avoid increasing the coordinates of metric labels (#3785) 2022-10-04 01:20:01 -07:00
Konstantin Lebedev 6fa3d0cc46 fix delete key panic of remote sync dir (#3770) 2022-10-01 02:33:47 -07:00
Konstantin Lebedev 301b678147 [volume] Add new volumes to HUP(reload) signal (#3755) 
Add new volumes to HUP(reload) signal
 2022-09-28 12:44:13 -07:00
Konstantin LebedevandChris Lu 7ae51d1ec6 fix set master options for single server (#3708) 
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-09-27 12:06:35 -07:00
Konstantin Lebedev faa6167b6b fs.meta.load load any dirs with prefix "important" (#3747) 
* fs.meta.load load any dirs with prefix "important"

* replace dirPattern to dirPrefix

* help dirPrefix
 2022-09-27 09:58:46 -07:00
Konstantin Lebedev 5d87ad72d8 mute log filer: no entry is found in filer store (#3707) 2022-09-16 09:05:38 -07:00
Konstantin Lebedev f8ef25099c Removing chunks on failed to write to replicas (#3591) 
* Removing chunks on failed to write to replicas
https://github.com/seaweedfs/seaweedfs/issues/3578

* put with in the util.Retry

* just purge on any errors
 2022-09-13 22:33:35 -07:00
Konstantin Lebedev 721c6197f9 skip deltaBeat if dn is zero (#3630) 
* skip deltaBeat
https://github.com/seaweedfs/seaweedfs/issues/3629

* fix GrpcPort

* skip url :0

* skip empty DataCenter or Rack

* skip empty heartbeat Ip

* dell msg add DataCenter

* comment todo

* fix
 2022-09-11 22:31:53 -07:00
Konstantin Lebedev b64674018a [sync] override amz storage class, None to delete (#3639) 
* override amz storage class, None to delete
https://github.com/seaweedfs/seaweedfs/issues/3636

* use empty string to delete

* without nil check
 2022-09-10 14:15:42 -07:00
Konstantin Lebedev 31d2f77ceb refactor https://github.com/seaweedfs/seaweedfs/pull/3616 (#3625) 2022-09-07 23:23:33 -07:00
Konstantin Lebedev 695e3a909c avoid data race on currentDiskUsage.maxVolumeCount (#3592) 
https://github.com/seaweedfs/seaweedfs/issues/3590
 2022-09-07 09:51:15 -07:00
Konstantin Lebedev cca45b02a2 avoid data race on calc freeVolumeSlotCount (#3594) 
https://github.com/seaweedfs/seaweedfs/issues/3593
 2022-09-07 09:50:17 -07:00
Konstantin Lebedev f4f6855c61 avoid SIGSEGV on localFilerSocket (#3609) 
https://github.com/seaweedfs/seaweedfs/issues/3607
 2022-09-06 08:20:23 -07:00
Konstantin Lebedev 3c3682fcce more log detail for upload err and deleting (#3577) 2022-09-06 08:19:13 -07:00
Konstantin Lebedev 01a25755c5 [filer] masterClient updates volume adjust log level (#3610) 2022-09-06 07:26:59 -07:00
Konstantin Lebedev 9b2b7d4f5a avoid data race on GoCDKPubSub.topic (#3596) 2022-09-05 07:57:44 -07:00
Konstantin Lebedev 916673ae24 do not print diskInfos with doVolumeCheckDisk in error log (#3598) 2022-09-05 07:49:26 -07:00
Konstantin Lebedev 5307cadb32 [s3] Replace code 500 to 499 for upload with ErrUnexpectedEOF (#3597) 2022-09-05 07:47:43 -07:00
Konstantin Lebedev 853880bd83 [filer] DATA RACE on signal_handling fixed (#3575) 2022-09-02 06:20:19 -07:00
Konstantin Lebedev c37d6fc01a avoid data race on grace.hooks (#3572) 
https://github.com/seaweedfs/seaweedfs/issues/3564
 2022-09-01 10:34:06 -07:00
Konstantin Lebedev 8c3040db81 avoid DATA RACE on S3Options.localFilerSocket (#3571) 
* avoid DATA RACE on S3Options.localFilerSocket
https://github.com/seaweedfs/seaweedfs/issues/3552

* copy localSocket
 2022-09-01 10:33:23 -07:00
Konstantin Lebedev 90d55cd179 avoid data race read fileChunksSize (#3570) 
https://github.com/seaweedfs/seaweedfs/issues/3557
 2022-09-01 10:31:57 -07:00
Konstantin Lebedev 82c2ce74db more error details in the log (#3568) 
* more error details in the log
https://github.com/seaweedfs/seaweedfs/issues/3567

* format message
 2022-09-01 09:04:15 -07:00
Konstantin Lebedev 803ca3c958 avoid data race on doSubscribeToOneFiler/ma.filer.UniqueFilerEpoch (#3566) 
https://github.com/seaweedfs/seaweedfs/issues/3565
 2022-09-01 09:01:47 -07:00
Konstantin Lebedev 77098de69a migrate from boltdb to bbolt (#3554) 2022-08-30 10:12:02 -07:00
Konstantin Lebedev ade94b0d0a avoid race conditions access to SuperBlock.Version (#3539) 
* avoid race conditions access to  SuperBlock.Version
https://github.com/seaweedfs/seaweedfs/issues/3515

* superBlockAccessLock replace to sync.Mutex
 2022-08-30 00:08:00 -07:00
Konstantin Lebedev e16dda88e4 avoid race conditions access to MasterClient.currentMaster (#3538) 
https://github.com/seaweedfs/seaweedfs/issues/3510
 2022-08-30 00:05:38 -07:00
Konstantin Lebedev 105702ebe0 avoid data race access to uploadReaderToChunks.uploadErr (#3550) 
avoid data race access to uploadErr
https://github.com/seaweedfs/seaweedfs/issues/3549
 2022-08-30 00:03:01 -07:00
Konstantin Lebedev 903a06a160 avoid data race access to startDelay (#3548) 2022-08-30 00:02:09 -07:00
Konstantin Lebedev 4966a3abc7 avoid race conditions access to growRequestCount (#3537) 
https://github.com/seaweedfs/seaweedfs/issues/3511
 2022-08-29 13:23:02 -07:00
Konstantin Lebedev e90ab4ac60 avoid race conditions for OnPeerUpdate (#3525) 
https://github.com/seaweedfs/seaweedfs/issues/3524
 2022-08-26 10:18:49 -07:00
Konstantin Lebedev 4f7a1f67cd avoid race conditions for diskLocation.MaxVolumeCount (#3526) 2022-08-26 08:41:42 -07:00
Konstantin Lebedev 913a5d0dd4 add target dev_race to MakeFile (#3522) 2022-08-26 01:28:03 -07:00
Konstantin Lebedev f7aeb06544 s3: report metadata if the directory is explicitly created (#3498) 
* replace mkdir to mkFile

* ContentLength must be zero

* revert mkDir

* Seaweedfs-Is-Directory-Key return metadata
 2022-08-24 00:15:44 -07:00
Konstantin Lebedev 6199d9b923 rand start garbageThreshold and reset metric ReplicaPlacementMismatch (#3396) 
* rand start garbageThreshold
reset metrics MasterReplicaPlacementMismatch

* vacuum through sleep
 2022-08-12 13:11:09 -07:00
Konstantin Lebedev d8c54b527e avoid Dockerfile Entrypoint: "file not found" (#3430) 
https://www.koskila.net/how-to-fix-exec-user-process-caused-no-such-file-or-directory-in-docker-on-windows/
 2022-08-11 15:07:44 -07:00
Konstantin Lebedev 4d4cd0948d avoid infinite loop WaitUntilConnected() (#3431) 
https://github.com/seaweedfs/seaweedfs/issues/3421
 2022-08-11 15:03:26 -07:00
Konstantin Lebedev 4d08393b7c filer prefer volume server in same data center (#3405) 
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
 2022-08-04 17:35:00 -07:00
Konstantin Lebedev 22181dd018 refactor FilerRequest metrics (#3402) 
* refactor FilerRequest metrics

* avoid double count proxy

* defer to
 2022-08-04 01:44:54 -07:00
Konstantin Lebedev 3ffa4ba91d clean 2022-08-02 01:30:38 +05:00
Konstantin Lebedev 2a5629dd28 align 2022-08-02 01:29:44 +05:00
Konstantin Lebedev df9cc31a0a refactor 2022-08-02 01:18:26 +05:00
Konstantin Lebedev 0a5d3c3cf6 rename to localWrite and replicatedWrite 2022-08-01 23:09:09 +05:00
Konstantin Lebedev fc65122766 rename to LoadAvg_1M 2022-08-01 21:32:21 +05:00
Konstantin Lebedev ab50f21ad9 Detailed metrics VolumeServerRequestHistogram for writing to disk and replication 2022-08-01 21:16:57 +05:00
Konstantin Lebedev 2f85e31f5c Merge branch 'master' into gentle_vacuum 2022-08-01 20:51:58 +05:00
Konstantin Lebedev 5209ebbeef remove percent 2022-08-01 20:40:38 +05:00
Konstantin Lebedev 78cbd8002f revert Sleep 2022-08-01 20:21:23 +05:00
Konstantin Lebedev cd5c7ad052 move to github.com/seaweedfs/seaweedfs 2022-08-01 16:36:32 +05:00