Created Seaweed Message Queue (markdown)

Chris Lu committed 2025-03-10 21:25:35 -07:00
1 parent e6f831eedf
commit d4453b5673
1 file changed
+23
+23
@@ -0,0 +1,23 @@
# Introduction
# Architecture
The structured messages are published to message brokers, persisted to SeaweedFS, and consumed by message subscribers. To simplify client libraries, a messaging agent with gRPC APIs is added for easier message publishing and subscribing.
```
Publishers => gRPC Publish APIs => Agent => Brokers => Agent => gRPC Subscribe APIs => Subscribers
^
|
v
SeaweedFS
```
The Agent can be run either on the server side, or as a sidecar on each client.
# Features
* Structured Messages: SeaweedFS is to store unstructured data files. Seaweed Message Queue is to store structured messages.
* The messages stored on SeaweedFS can be converted into Parquet files, saving disk spaces with more efficient columnar compression.
* The Parquet files can be consumed in streams via Seaweed messaging brokers.
* The Parquet files can be read in batches directly via SeaweedFS.
* Message published successfully are acknowledged, or receive an error if not successful.
* Message consume offsets are tracked and persisted.
* Consumer APIs can process messages in multi-threads while still ensuring serial processing the same message key.