Github-Mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-15 11:00:51 +02:00
Code Issues Packages Projects Releases Wiki Activity
10,942 Commits 180 Branches 360 Tags
67a252ee8a0cbcf0f33cb7c94de21d4791ef1f39
Commit Graph
51 Commits
Author SHA1 Message Date
chrislu cb1ba0275a sort imports 2024-06-14 11:44:42 -07:00
chrislu 32dbb6a2ce Merge branch 'master' into mq 2024-06-14 11:42:05 -07:00
vadimartynov 8aae82dd71 Added context for the MasterClient's methods to avoid endless loops (#5628) 
* Added context for the MasterClient's methods to avoid endless loops

* Returned WithClient function. Added WithClientCustomGetMaster function

* Hid unused ctx arguments

* Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions

* Changed the context termination check in the tryConnectToMaster function

* Added a child context to the tryConnectToMaster function

* Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark
 2024-06-14 11:40:34 -07:00
chrislu 25b2850e7d refactor out FilerClientAccessor 2024-06-06 19:44:19 -07:00
chrislu edc87c16f9 balance subscribers 
need to ensure the offsets are persisted
 2024-05-27 17:30:16 -07:00
chrislu cdeaaf95b4 go fmt 2024-05-23 08:25:53 -07:00
chrislu 554ae09f82 rename 2024-05-21 10:02:07 -07:00
chrislu 67e6051585 rename Coordinator to SubCoordinator 2024-05-21 09:57:45 -07:00
chrislu d5abffa42c rename Balancer to PubBalancer 2024-05-21 09:56:30 -07:00
chrislu 3622196881 go fmt 2024-05-20 11:05:18 -07:00
chrislu 972e9faaa2 move initial assignment to rebalance logic 2024-05-14 09:09:36 -07:00
chrislu 6e5075e14e move read write topic config into filer client accessor 2024-05-14 08:50:17 -07:00
chrislu e7823ee967 retry connecting to broker leader 2024-03-09 12:56:49 -08:00
chrislu 62397f2371 adjust error 2024-03-07 10:47:50 -08:00
Chris Lu 4dc6681833 proxy requests to lock owner 2024-02-05 16:46:40 -08:00
Chris Lu f9131de43f fix lock TTL 2024-02-04 12:47:21 -08:00
Chris Lu 0a12301b3d avoid too large expiration time 2024-02-04 09:20:21 -08:00
chrislu d41792461c lock returns host and owner 2024-02-02 15:54:57 -08:00
chrislu d30150dde1 use a short-lived lock 2024-02-01 23:01:44 -08:00
chrislu ac50d8a822 re-lock if the lock owner is not found 2024-01-29 22:46:23 -08:00
chrislu 34839237ab refactor 2024-01-20 12:16:40 -08:00
chrislu 2828ccbb30 refactor 2024-01-20 11:41:11 -08:00
chrislu bc8d2a01cc create local topic partition from config on filer 2024-01-17 00:15:52 -08:00
chrislu ba73199174 reuse local partition 2024-01-15 21:22:41 -08:00
chrislu d51efddf5c flush to disk 
Need to do: read from disk
 2024-01-08 23:27:02 -08:00
chrislu 35869b5c80 subscriber can be notified of the assignment change when topic is just configured 
Next: Subscriber needs to read by the timestamp offset.
 2024-01-03 13:30:30 -08:00
Chris Lu 580940bf82 Merge accumulated changes related to message queue (#5098) 
* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* tracking topic=>broker

* merge

* comment
 2023-12-11 12:05:54 -08:00
chrislu de0b969b36 Revert "rename" 
This reverts commit 35b5264ab7.
 2023-10-12 20:28:11 -07:00
chrislu 35b5264ab7 rename 2023-10-11 21:44:56 -07:00
chrislu 0361c321b4 add CreateTopic API 2023-09-24 22:00:43 -07:00
chrislu b18112ae10 fix compilation bugs during merge 2023-09-19 14:08:17 -07:00
chrislu 4827425146 balancer works 2023-09-16 15:05:38 -07:00
chrislu 436d99443b receive broker stats 2023-09-14 23:49:05 -07:00
chrislu b771fefa37 Merge branch 'master' into sub 2023-09-13 00:03:47 -07:00
chrislu 4d6c18d86f pub sub initial tests 2023-08-27 17:50:59 -07:00
Nico D'Cotta 796b7508f3 Implement SRV lookups for filer (#4767) 2023-08-24 07:08:56 -07:00
chrislu 01d70c21f3 Squashed commit of the following: 
commit 32f4b1a13057d56b6de487cdb80ff7c205af01a6
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 22:52:19 2023 -0700

    fix compilation

commit e77ad33b7ca0423138fbae26a4433b60923a9588
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 22:46:44 2023 -0700

    pub

commit f431f30cc7ca277ca299e3cd118c05537fb9f5c3
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 13:27:39 2023 -0700

    fix generic type

commit 4e9dcb18293fd1e3e306e2dceb995dfd67a35e1d
Merge: 30f942580 16e3f2d52
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 12:47:14 2023 -0700

    Merge branch 'master' into pubsub

commit 30f942580ad1bb32ae94aade2e3a21ec3ab63e21
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 11:10:58 2023 -0700

    wip

commit f8b00980bc2f3879bb43decffd9a08d842f196f2
Author: chrislu <chris.lu@gmail.com>
Date:   Tue Jul 25 09:14:35 2023 -0700

    add design document

commit 08d2bebe42a26ebc39f1542f54d99e73620727dd
Author: chrislu <chris.lu@gmail.com>
Date:   Tue Jul 25 09:14:06 2023 -0700

    minor

commit bcfa7982b262a40fcdce6fc6613fad2ce07c13da
Author: chrislu <chris.lu@gmail.com>
Date:   Tue Jul 25 09:13:49 2023 -0700

    rename
 2023-08-20 22:53:05 -07:00
chrislu 81fdf3651b grpc connection to filer add sw-client-id header 2023-01-20 01:48:12 -08:00
Konstantin Lebedev e90ab4ac60 avoid race conditions for OnPeerUpdate (#3525) 
https://github.com/seaweedfs/seaweedfs/issues/3524
 2022-08-26 10:18:49 -07:00
askeipx 2e78a522ab remove old raft servers if they don't answer to pings for too long (#3398) 
* remove old raft servers if they don't answer to pings for too long

add ping durations as options

rename ping fields

fix some todos

get masters through masterclient

raft remove server from leader

use raft servers to ping them

CheckMastersAlive for hashicorp raft only

* prepare blocking ping

* pass waitForReady as param

* pass waitForReady through all functions

* waitForReady works

* refactor

* remove unneeded params

* rollback unneeded changes

* fix
 2022-08-23 23:18:21 -07:00
Konstantin Lebedev 4d08393b7c filer prefer volume server in same data center (#3405) 
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
 2022-08-04 17:35:00 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu aa4a22ad47 segment serde 2022-07-28 23:24:38 -07:00
chrislu 74f60f246f dynamically connect to a filer 2022-07-28 23:24:38 -07:00
chrislu 9f479aab98 allocate brokers to serve segments 2022-07-28 23:24:38 -07:00
chrislu 8060fdcac5 remove old code 2022-07-28 23:24:38 -07:00
chrislu f25e273e32 display data center and rack in cluster.ps 2022-07-28 23:22:52 -07:00
chrislu 68065128b8 add dc and rack 2022-07-28 23:22:51 -07:00
chrislu 87ccef72db cluster.ps can list brokers 2022-07-28 23:22:07 -07:00
chrislu ac66227798 renaming 2022-07-28 23:22:07 -07:00