mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-22 14:04:28 +02:00
Created Seaweed Message Queue (markdown)
1 parent
e6f831eedf
commit
d4453b5673
1 file changed
+23
@@ -0,0 +1,23 @@
|
||||
# Introduction
|
||||
|
||||
# Architecture
|
||||
|
||||
The structured messages are published to message brokers, persisted to SeaweedFS, and consumed by message subscribers. To simplify client libraries, a messaging agent with gRPC APIs is added for easier message publishing and subscribing.
|
||||
```
|
||||
Publishers => gRPC Publish APIs => Agent => Brokers => Agent => gRPC Subscribe APIs => Subscribers
|
||||
^
|
||||
|
|
||||
v
|
||||
SeaweedFS
|
||||
```
|
||||
|
||||
The Agent can be run either on the server side, or as a sidecar on each client.
|
||||
|
||||
# Features
|
||||
* Structured Messages: SeaweedFS is to store unstructured data files. Seaweed Message Queue is to store structured messages.
|
||||
* The messages stored on SeaweedFS can be converted into Parquet files, saving disk spaces with more efficient columnar compression.
|
||||
* The Parquet files can be consumed in streams via Seaweed messaging brokers.
|
||||
* The Parquet files can be read in batches directly via SeaweedFS.
|
||||
* Message published successfully are acknowledged, or receive an error if not successful.
|
||||
* Message consume offsets are tracked and persisted.
|
||||
* Consumer APIs can process messages in multi-threads while still ensuring serial processing the same message key.
|
||||
Reference in new issue
Block a user