diff --git a/Filer-Notification-Webhook.md b/Filer-Notification-Webhook.md index 15e9b2e..e889eea 100644 --- a/Filer-Notification-Webhook.md +++ b/Filer-Notification-Webhook.md @@ -67,10 +67,22 @@ If you're experiencing or expecting high event volumes, consider: ### Basic Setup Add the webhook configuration to your `notification.toml` file. This file should be placed in one of these locations (in descending priority): -- `./notification.toml` +- `./notification.toml` (the working directory of the `weed` process) - `$HOME/.seaweedfs/notification.toml` - `/etc/seaweedfs/notification.toml` +The file is read once at startup by any process that runs a filer: `weed filer`, `weed server -filer`, and `weed mini`. There is no command line flag for it — place the file where the filer can find it and restart. When the configuration is picked up, the filer logs: + +``` +Configure notification message queue for webhook +``` + +To generate a template with all options: + +```bash +weed scaffold -config=notification > notification.toml +``` + ### Minimal Configuration Example ```toml @@ -133,6 +145,46 @@ path_prefixes = ["/important", "/data"] | `event_types` | array | No | all | create, update, delete, rename | Filter events by type | | `path_prefixes` | array | No | all | - | Filter events by path prefix | +### One Endpoint, One Target + +Only one notification target can be enabled at a time — if two sections (e.g. `[notification.webhook]` and `[notification.kafka]`) are both `enabled = true`, the filer fails at startup. The webhook target itself supports a single endpoint: exactly one `[notification.webhook]` block with one `endpoint`. Named sub-tables such as `[notification.webhook.my_service]` are not additional endpoints; they are ignored. + +To route events to multiple services, point the webhook at one service that fans out, or use the Kafka/SQS/Pub-Sub targets and let each consumer subscribe. Use `event_types` and `path_prefixes` to narrow what the single endpoint receives. + +## Quick Test + +A minimal receiver that prints every event, using only the Python standard library: + +```python +from http.server import BaseHTTPRequestHandler, HTTPServer +import json + +class Handler(BaseHTTPRequestHandler): + def do_POST(self): + body = self.rfile.read(int(self.headers["Content-Length"])) + print(json.dumps(json.loads(body), indent=2)) + self.send_response(200) + self.end_headers() + +HTTPServer(("", 3000), Handler).serve_forever() +``` + +With this `notification.toml` in the directory where you start `weed`: + +```toml +[notification.webhook] +enabled = true +endpoint = "http://localhost:3000/webhook" +``` + +start a filer (`weed mini`, `weed server -filer`, or `weed filer`) and create a file: + +```bash +curl -F "file=@test.txt" "http://localhost:8888/data/" +``` + +The receiver prints a `create` event for `/data/test.txt`. + ## Event Types The webhook notification system supports four types of file system events: @@ -144,7 +196,7 @@ Triggered when a new file or directory is created. ### 2. Update Event Triggered when an existing file or directory is modified. -- **Detection**: Both `old_entry` and `new_entry` are present, no path change +- **Detection**: Both `old_entry` and `new_entry` are present, and neither the directory nor the file name changed - **Event Type**: `"update"` ### 3. Delete Event @@ -154,7 +206,7 @@ Triggered when a file or directory is deleted. ### 4. Rename Event Triggered when a file or directory is moved or renamed. -- **Detection**: Both `old_entry` and `new_entry` are present, and `new_parent_path` is specified +- **Detection**: Both `old_entry` and `new_entry` are present, and the directory or the file name changed - **Event Type**: `"rename"` ## Webhook Payload Format @@ -200,6 +252,8 @@ Triggered when a file or directory is moved or renamed. } ``` +Fields with empty values are omitted from the JSON: a `create` event has no `old_entry` key at all, a `delete` event has no `new_entry` key, and `delete_chunks` / `new_parent_path` only appear when set. Check for key presence rather than expecting explicit `null`s. `mtime` and `crtime` are unix time in seconds. The examples below show absent fields as `null` for readability. + ## Example Payloads ### Create File Event @@ -396,6 +450,8 @@ endpoint = "https://your-server.com/webhook" path_prefixes = ["/important", "/data"] ``` +Prefixes are plain string prefixes on the full path. For renames, the event matches if either the old or the new path matches a prefix. + ### Combined Filtering You can combine both filters: @@ -409,14 +465,18 @@ event_types = ["create", "delete"] path_prefixes = ["/important"] ``` +## Backfilling Existing Files + +Webhooks only fire for events that happen while the filer is running. To send events for files that already exist, run `fs.meta.notify` in `weed shell` — it walks a directory tree and publishes a `create` event for every entry to the configured notification target. Place `notification.toml` in the directory where you run `weed shell`. + ## Retry and Error Handling The webhook system includes robust error handling: 1. **Automatic Retries**: Failed requests are automatically retried up to `max_retries` times -2. **Exponential Backoff**: Retry delays increase exponentially from `backoff_seconds` to `max_backoff_seconds` -3. **Dead Letter Queue**: After exhausting retries, failed messages are logged for debugging -4. **Status Code Validation**: Only 2xx status codes are considered successful +2. **Exponential Backoff**: Retry delays double from `backoff_seconds` up to `max_backoff_seconds`, with up to 30% random jitter +3. **Dead Letter Logging**: After exhausting retries, failed messages are written to the filer log (`received dead letter message: ...`); they are not persisted or redelivered +4. **Status Code Validation**: Only 2xx status codes are considered successful; 3xx redirects are followed (the redirect target is cached for subsequent deliveries) and anything else is retried ### Example Retry Configuration @@ -429,11 +489,11 @@ backoff_seconds = 2 max_backoff_seconds = 60 ``` -This configuration will retry up to 5 times with delays: 2s, 4s, 8s, 16s, 32s. +This configuration will retry up to 5 times with delays around 2s, 4s, 8s, 16s, 32s (each with up to 30% jitter). ## Performance Tuning -**⚠️ Note**: While you can tune these parameters, remember that webhooks use a push model best suited for low to moderate traffic. For high-traffic scenarios (>1000 events/second sustained), consider using Kafka, message queues, or the pull-based metadata event logs instead. +**Note**: While you can tune these parameters, remember that webhooks use a push model best suited for low to moderate traffic. For high-traffic scenarios (>1000 events/second sustained), consider using Kafka, message queues, or the pull-based metadata event logs instead. ### Concurrent Workers @@ -493,7 +553,8 @@ Your webhook receiver should: 2. Validate request content type is `application/json` 3. Parse and validate JSON payload structure 4. Respond with 2xx status code for successful processing -5. Implement idempotency to handle potential duplicate events +5. Respond quickly — acknowledge first, then process asynchronously; a delivery worker is tied up until your endpoint responds or `timeout_seconds` elapses +6. Implement idempotency to handle potential duplicate events (delivery is at-least-once and may be out of order) ## Example Webhook Receiver (Node.js/Express)