From d4453b56733f0ab82bdea072bacfd59d46689219 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Mon, 10 Mar 2025 21:25:35 -0700 Subject: [PATCH] Created Seaweed Message Queue (markdown) --- Seaweed-Message-Queue.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 Seaweed-Message-Queue.md diff --git a/Seaweed-Message-Queue.md b/Seaweed-Message-Queue.md new file mode 100644 index 0000000..a602293 --- /dev/null +++ b/Seaweed-Message-Queue.md @@ -0,0 +1,23 @@ +# Introduction + +# Architecture + +The structured messages are published to message brokers, persisted to SeaweedFS, and consumed by message subscribers. To simplify client libraries, a messaging agent with gRPC APIs is added for easier message publishing and subscribing. +``` + Publishers => gRPC Publish APIs => Agent => Brokers => Agent => gRPC Subscribe APIs => Subscribers + ^ + | + v + SeaweedFS +``` + +The Agent can be run either on the server side, or as a sidecar on each client. + +# Features +* Structured Messages: SeaweedFS is to store unstructured data files. Seaweed Message Queue is to store structured messages. +* The messages stored on SeaweedFS can be converted into Parquet files, saving disk spaces with more efficient columnar compression. +* The Parquet files can be consumed in streams via Seaweed messaging brokers. +* The Parquet files can be read in batches directly via SeaweedFS. +* Message published successfully are acknowledged, or receive an error if not successful. +* Message consume offsets are tracked and persisted. +* Consumer APIs can process messages in multi-threads while still ensuring serial processing the same message key.