Github-Mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-08 15:41:15 +02:00
Code Issues Packages Projects Releases Wiki Activity
8,985 Commits 177 Branches 359 Tags
c0deaa494857eeb5bdb96d1a4fabbd3f825a741e
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Konstantin Lebedev c0deaa4948 [volume.fsck] check needles status from volume server (#3926) 
check needles status from volume server
 2022-10-31 11:33:04 -07:00
Konstantin Lebedev 409c9328de [master] avoid vacuum if not enough replica copies (#3924) 
avoid vacuum if not enough replica copies
 2022-10-30 20:34:19 -07:00
Konstantin Lebedev bf8a9d2db1 [volume.chek.disk] sync of deletions the fix (#3923) 
* sync of deletions the fix

* avoid return if only partiallyDeletedNeedles

* refactor sync deletions
 2022-10-30 20:32:46 -07:00
Konstantin Lebedev 764d9cb105 [voluche.chek.disk] needles older than the cutoff time are not missing yet (#3922) 
needles older than the cutoff time are not missing yet

https://github.com/seaweedfs/seaweedfs/issues/3919
 2022-10-28 12:12:20 -07:00
Konstantin Lebedev f9f499b8d5 [fix] add param for ignore error (#3918) 2022-10-28 06:59:39 -07:00
Konstantin Lebedev c6f2899375 avoid fix missing of deleted needles (#3909) 2022-10-25 21:35:54 -07:00
Konstantin LebedevandChris Lu 6253058d9d ensure monotonic n.AppendAtNs in each place (#3880) 
https://github.com/seaweedfs/seaweedfs/issues/3852

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-10-23 18:24:52 -07:00
Konstantin Lebedev 452202f35b ReadNeedleMeta recovery panic occurred (#3883) 
ReadNeedleMeta panic occurred
https://github.com/seaweedfs/seaweedfs/issues/3882
 2022-10-23 14:00:28 -07:00
Konstantin Lebedev e20f0dbd2d avoid data race of TraverseBfs (#3856) 
* avoid data race of TraverseBfs

* close is enough
avoid panic
I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255
I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c]

goroutine 287 [running]:
github.com/seaweedfs/seaweedfs/weed/storage.(*SortedFileNeedleMap).Close(0x0)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load.func1()
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150
 2022-10-16 11:30:41 -07:00
Konstantin Lebedev ee38ab8581 avoid data race of LogBuffer isStopping (#3859) 2022-10-16 11:28:56 -07:00
Konstantin Lebedev 7836f7574e [volume.fsck] hotfix apply purging and add option verifyNeedle #3860 (#3861) 
* fix apply purging and add verifyNeedle

* common readSourceNeedleBlob

* use consts
 2022-10-15 20:38:46 -07:00
Konstantin Lebedev f476cf3403 fix volume_server.proto conflict (#3858) 
fix conflict

https://github.com/seaweedfs/seaweedfs/pull/3841
 2022-10-15 20:01:07 -07:00
Konstantin Lebedev 2f72103c83 avoid load volume file with BytesOffset mismatch (#3841) 
* avoid load volume file with BytesOffset mismatch

https://github.com/seaweedfs/seaweedfs/issues/2966

* set BytesOffset if has not VolumeInfoFile

* typos fail => failed

* exit if bytesOffset mismatch
 2022-10-14 00:18:09 -07:00
Konstantin Lebedev f19c9e3d9d Volume fsck by volume (#3851) 
* refactor

* refactor args verbose and writer

* refactor readFilerFileIdFile

* fix filter by collectMtime

* skip system log collection
 2022-10-13 23:30:30 -07:00
Konstantin Lebedev 1f7e52c63e vacuum metrics and force sync dst files (#3832) 2022-10-13 00:51:20 -07:00
Konstantin Lebedev 401315f337 master fix interruption through ctrl+c (#3834) 2022-10-12 07:18:40 -07:00
Konstantin Lebedev 5b28c3f728 revert disabling FSync for non Mac (#3814) 2022-10-10 07:28:02 -07:00
Konstantin Lebedev 5db25a8f2a metric shows who is currently blocking the cluster or not (#3799) 
* master_admin_lock Shows whether cluster is locked now or not
https://github.com/seaweedfs/seaweedfs/issues/3452

* fix metric MasterAdminLock
 2022-10-07 13:26:29 -07:00
Konstantin Lebedev a522507f95 configure raft metrics (#3798) 2022-10-07 04:20:34 -07:00
Konstantin Lebedev b9933d5589 master server graceful stop (#3797) 2022-10-06 09:30:30 -07:00
Konstantin Lebedev 36daa7709d show raft leader via shell (#3796) 2022-10-06 07:10:41 -07:00
Konstantin Lebedev ec46a34f33 avoid increasing the coordinates of metric labels (#3785) 2022-10-04 01:20:01 -07:00
Konstantin Lebedev 6fa3d0cc46 fix delete key panic of remote sync dir (#3770) 2022-10-01 02:33:47 -07:00
Konstantin Lebedev 301b678147 [volume] Add new volumes to HUP(reload) signal (#3755) 
Add new volumes to HUP(reload) signal
 2022-09-28 12:44:13 -07:00
Konstantin LebedevandChris Lu 7ae51d1ec6 fix set master options for single server (#3708) 
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-09-27 12:06:35 -07:00
Konstantin Lebedev faa6167b6b fs.meta.load load any dirs with prefix "important" (#3747) 
* fs.meta.load load any dirs with prefix "important"

* replace dirPattern to dirPrefix

* help dirPrefix
 2022-09-27 09:58:46 -07:00
Konstantin Lebedev 5d87ad72d8 mute log filer: no entry is found in filer store (#3707) 2022-09-16 09:05:38 -07:00
Konstantin Lebedev f8ef25099c Removing chunks on failed to write to replicas (#3591) 
* Removing chunks on failed to write to replicas
https://github.com/seaweedfs/seaweedfs/issues/3578

* put with in the util.Retry

* just purge on any errors
 2022-09-13 22:33:35 -07:00
Konstantin Lebedev 721c6197f9 skip deltaBeat if dn is zero (#3630) 
* skip deltaBeat
https://github.com/seaweedfs/seaweedfs/issues/3629

* fix GrpcPort

* skip url :0

* skip empty DataCenter or Rack

* skip empty heartbeat Ip

* dell msg add DataCenter

* comment todo

* fix
 2022-09-11 22:31:53 -07:00
Konstantin Lebedev b64674018a [sync] override amz storage class, None to delete (#3639) 
* override amz storage class, None to delete
https://github.com/seaweedfs/seaweedfs/issues/3636

* use empty string to delete

* without nil check
 2022-09-10 14:15:42 -07:00
Konstantin Lebedev 31d2f77ceb refactor https://github.com/seaweedfs/seaweedfs/pull/3616 (#3625) 2022-09-07 23:23:33 -07:00
Konstantin Lebedev 695e3a909c avoid data race on currentDiskUsage.maxVolumeCount (#3592) 
https://github.com/seaweedfs/seaweedfs/issues/3590
 2022-09-07 09:51:15 -07:00
Konstantin Lebedev cca45b02a2 avoid data race on calc freeVolumeSlotCount (#3594) 
https://github.com/seaweedfs/seaweedfs/issues/3593
 2022-09-07 09:50:17 -07:00
Konstantin Lebedev f4f6855c61 avoid SIGSEGV on localFilerSocket (#3609) 
https://github.com/seaweedfs/seaweedfs/issues/3607
 2022-09-06 08:20:23 -07:00
Konstantin Lebedev 3c3682fcce more log detail for upload err and deleting (#3577) 2022-09-06 08:19:13 -07:00
Konstantin Lebedev 01a25755c5 [filer] masterClient updates volume adjust log level (#3610) 2022-09-06 07:26:59 -07:00
Konstantin Lebedev 9b2b7d4f5a avoid data race on GoCDKPubSub.topic (#3596) 2022-09-05 07:57:44 -07:00
Konstantin Lebedev 916673ae24 do not print diskInfos with doVolumeCheckDisk in error log (#3598) 2022-09-05 07:49:26 -07:00
Konstantin Lebedev 5307cadb32 [s3] Replace code 500 to 499 for upload with ErrUnexpectedEOF (#3597) 2022-09-05 07:47:43 -07:00
Konstantin Lebedev 853880bd83 [filer] DATA RACE on signal_handling fixed (#3575) 2022-09-02 06:20:19 -07:00
Konstantin Lebedev c37d6fc01a avoid data race on grace.hooks (#3572) 
https://github.com/seaweedfs/seaweedfs/issues/3564
 2022-09-01 10:34:06 -07:00
Konstantin Lebedev 8c3040db81 avoid DATA RACE on S3Options.localFilerSocket (#3571) 
* avoid DATA RACE on S3Options.localFilerSocket
https://github.com/seaweedfs/seaweedfs/issues/3552

* copy localSocket
 2022-09-01 10:33:23 -07:00
Konstantin Lebedev 90d55cd179 avoid data race read fileChunksSize (#3570) 
https://github.com/seaweedfs/seaweedfs/issues/3557
 2022-09-01 10:31:57 -07:00
Konstantin Lebedev 82c2ce74db more error details in the log (#3568) 
* more error details in the log
https://github.com/seaweedfs/seaweedfs/issues/3567

* format message
 2022-09-01 09:04:15 -07:00
Konstantin Lebedev 803ca3c958 avoid data race on doSubscribeToOneFiler/ma.filer.UniqueFilerEpoch (#3566) 
https://github.com/seaweedfs/seaweedfs/issues/3565
 2022-09-01 09:01:47 -07:00
Konstantin Lebedev 77098de69a migrate from boltdb to bbolt (#3554) 2022-08-30 10:12:02 -07:00
Konstantin Lebedev ade94b0d0a avoid race conditions access to SuperBlock.Version (#3539) 
* avoid race conditions access to  SuperBlock.Version
https://github.com/seaweedfs/seaweedfs/issues/3515

* superBlockAccessLock replace to sync.Mutex
 2022-08-30 00:08:00 -07:00
Konstantin Lebedev e16dda88e4 avoid race conditions access to MasterClient.currentMaster (#3538) 
https://github.com/seaweedfs/seaweedfs/issues/3510
 2022-08-30 00:05:38 -07:00
Konstantin Lebedev 105702ebe0 avoid data race access to uploadReaderToChunks.uploadErr (#3550) 
avoid data race access to uploadErr
https://github.com/seaweedfs/seaweedfs/issues/3549
 2022-08-30 00:03:01 -07:00
Konstantin Lebedev 903a06a160 avoid data race access to startDelay (#3548) 2022-08-30 00:02:09 -07:00
Konstantin Lebedev 4966a3abc7 avoid race conditions access to growRequestCount (#3537) 
https://github.com/seaweedfs/seaweedfs/issues/3511
 2022-08-29 13:23:02 -07:00
Konstantin Lebedev e90ab4ac60 avoid race conditions for OnPeerUpdate (#3525) 
https://github.com/seaweedfs/seaweedfs/issues/3524
 2022-08-26 10:18:49 -07:00
Konstantin Lebedev 4f7a1f67cd avoid race conditions for diskLocation.MaxVolumeCount (#3526) 2022-08-26 08:41:42 -07:00
Konstantin Lebedev 913a5d0dd4 add target dev_race to MakeFile (#3522) 2022-08-26 01:28:03 -07:00
Konstantin Lebedev f7aeb06544 s3: report metadata if the directory is explicitly created (#3498) 
* replace mkdir to mkFile

* ContentLength must be zero

* revert mkDir

* Seaweedfs-Is-Directory-Key return metadata
 2022-08-24 00:15:44 -07:00
Konstantin Lebedev 6199d9b923 rand start garbageThreshold and reset metric ReplicaPlacementMismatch (#3396) 
* rand start garbageThreshold
reset metrics MasterReplicaPlacementMismatch

* vacuum through sleep
 2022-08-12 13:11:09 -07:00
Konstantin Lebedev d8c54b527e avoid Dockerfile Entrypoint: "file not found" (#3430) 
https://www.koskila.net/how-to-fix-exec-user-process-caused-no-such-file-or-directory-in-docker-on-windows/
 2022-08-11 15:07:44 -07:00
Konstantin Lebedev 4d4cd0948d avoid infinite loop WaitUntilConnected() (#3431) 
https://github.com/seaweedfs/seaweedfs/issues/3421
 2022-08-11 15:03:26 -07:00
Konstantin Lebedev 4d08393b7c filer prefer volume server in same data center (#3405) 
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
 2022-08-04 17:35:00 -07:00
Konstantin Lebedev 22181dd018 refactor FilerRequest metrics (#3402) 
* refactor FilerRequest metrics

* avoid double count proxy

* defer to
 2022-08-04 01:44:54 -07:00
Konstantin Lebedev 3ffa4ba91d clean 2022-08-02 01:30:38 +05:00
Konstantin Lebedev 2a5629dd28 align 2022-08-02 01:29:44 +05:00
Konstantin Lebedev df9cc31a0a refactor 2022-08-02 01:18:26 +05:00
Konstantin Lebedev 0a5d3c3cf6 rename to localWrite and replicatedWrite 2022-08-01 23:09:09 +05:00
Konstantin Lebedev fc65122766 rename to LoadAvg_1M 2022-08-01 21:32:21 +05:00
Konstantin Lebedev ab50f21ad9 Detailed metrics VolumeServerRequestHistogram for writing to disk and replication 2022-08-01 21:16:57 +05:00
Konstantin Lebedev 2f85e31f5c Merge branch 'master' into gentle_vacuum 2022-08-01 20:51:58 +05:00
Konstantin Lebedev 5209ebbeef remove percent 2022-08-01 20:40:38 +05:00
Konstantin Lebedev 78cbd8002f revert Sleep 2022-08-01 20:21:23 +05:00
Konstantin Lebedev cd5c7ad052 move to github.com/seaweedfs/seaweedfs 2022-08-01 16:36:32 +05:00
Konstantin Lebedev 634bc97840 fix conflicts 2022-08-01 14:47:00 +05:00
Konstantin Lebedev 3c75479e2b Merge branch 'master' into gentle_vacuum 
# Conflicts:
#	weed/pb/messaging_pb/messaging.pb.go
#	weed/pb/messaging_pb/messaging_grpc.pb.go
#	weed/pb/s3_pb/s3.pb.go
#	weed/pb/volume_server_pb/volume_server.pb.go
#	weed/server/volume_grpc_vacuum.go
 2022-08-01 14:45:22 +05:00
Konstantin Lebedev c0d92f61a1 comment 2022-08-01 14:40:42 +05:00
Konstantin Lebedev 1d29f67c02 revert disk stats 2022-08-01 14:29:41 +05:00
Konstantin Lebedev a98f6d66a3 rollback over onPeerupdate implementation of automatic clean-up of failed servers in favor of synchronous ping 2022-08-01 12:51:41 +05:00
Konstantin Lebedev 2f0dda384d vacuum show LA 2022-07-29 11:59:33 +05:00
Konstantin Lebedev 7e09a548a6 exclude directories to sync on filer 2022-07-27 19:22:57 +05:00
Konstantin Lebedev c88ea31f62 fix RUnlock of unlocked RWMutex 2022-07-26 12:57:07 +05:00
Konstantin Lebedev da9d3e8f6c refactor 2022-07-26 11:56:45 +05:00
Konstantin Lebedev 046c3d5ad4 fix logic else brake 2022-07-26 11:47:11 +05:00
Konstantin Lebedev 3c42814b58 avoid deadlock 2022-07-21 17:15:10 +05:00
Konstantin Lebedev 03a0936556 Merge remote-tracking branch 'origin/issues/3083' into issues/3083 2022-07-21 15:51:33 +05:00
Konstantin Lebedev 93ca87b7cb use safe onPeerUpdateDoneCns 2022-07-21 15:51:14 +05:00
Konstantin Lebedev 3a1ef8837e Merge branch 'chrislusf:master' into issues/3083 2022-07-20 18:42:32 +05:00
Konstantin Lebedev 7875470e74 onPeerUpdateGoroutineCount use int32 2022-07-20 18:40:50 +05:00
Konstantin Lebedev c5189c343b remove ticker 
update the topology before each file
 2022-07-20 18:40:50 +05:00
Konstantin Lebedev e2d991d8d0 ticker.Stop 2022-07-20 18:40:50 +05:00
Konstantin Lebedev de4fcc0e2c sync update topologyInfo 2022-07-20 18:40:50 +05:00
Konstantin Lebedev fa88dff7ce update otherNodes 2022-07-20 18:40:50 +05:00
Konstantin Lebedev 72dca31cfa fix update topologyInfo 2022-07-20 18:40:50 +05:00
Konstantin Lebedev 884ffbafee clouse background update 2022-07-20 18:40:50 +05:00
Konstantin Lebedev 39eaf426f8 fix TestVolumeServerEvacuate 2022-07-20 18:40:50 +05:00
Konstantin Lebedev b5e5f6f55a update topologyInfo 2022-07-20 18:40:50 +05:00
Konstantin Lebedev 867269cdcf help rack 2022-07-20 18:40:50 +05:00
Konstantin Lebedev 6f764e1014 volume server evacuate from rack 2022-07-20 18:40:50 +05:00
Konstantin Lebedev ba0e3ce5fa volume server evacuate to target server 2022-07-20 18:40:50 +05:00
Konstantin Lebedev b6471ecd75 err msg with duplicated local subscription detected move to log level 1 
https://github.com/chrislusf/seaweedfs/issues/3320
 2022-07-20 18:40:50 +05:00
Konstantin Lebedev 6c390851e7 fix design 2022-07-20 18:08:12 +05:00
Konstantin Lebedev d3f7c09c03 remove ticker 
update the topology before each file
 2022-07-20 00:54:23 +05:00
Konstantin Lebedev 11e393dbe7 err msg with duplicated local subscription detected move to log level 1 
https://github.com/chrislusf/seaweedfs/issues/3320
 2022-07-20 00:45:13 +05:00