webhook notification: document config loading, single endpoint, receiver contract, quick test

Filer-Notification-Webhook.md

@@ -67,10 +67,22 @@ If you're experiencing or expecting high event volumes, consider:
 ### Basic Setup
 
 Add the webhook configuration to your `notification.toml` file. This file should be placed in one of these locations (in descending priority):
-- `./notification.toml`
+- `./notification.toml` (the working directory of the `weed` process)
 - `$HOME/.seaweedfs/notification.toml`
 - `/etc/seaweedfs/notification.toml`
 
+The file is read once at startup by any process that runs a filer: `weed filer`, `weed server -filer`, and `weed mini`. There is no command line flag for it — place the file where the filer can find it and restart. When the configuration is picked up, the filer logs:
+
+```
+Configure notification message queue for webhook
+```
+
+To generate a template with all options:
+
+```bash
+weed scaffold -config=notification > notification.toml
+```
+
 ### Minimal Configuration Example
 
 ```toml
@@ -133,6 +145,46 @@ path_prefixes = ["/important", "/data"]
 | `event_types` | array | No | all | create, update, delete, rename | Filter events by type |
 | `path_prefixes` | array | No | all | - | Filter events by path prefix |
 
+### One Endpoint, One Target
+
+Only one notification target can be enabled at a time — if two sections (e.g. `[notification.webhook]` and `[notification.kafka]`) are both `enabled = true`, the filer fails at startup. The webhook target itself supports a single endpoint: exactly one `[notification.webhook]` block with one `endpoint`. Named sub-tables such as `[notification.webhook.my_service]` are not additional endpoints; they are ignored.
+
+To route events to multiple services, point the webhook at one service that fans out, or use the Kafka/SQS/Pub-Sub targets and let each consumer subscribe. Use `event_types` and `path_prefixes` to narrow what the single endpoint receives.
+
+## Quick Test
+
+A minimal receiver that prints every event, using only the Python standard library:
+
+```python
+from http.server import BaseHTTPRequestHandler, HTTPServer
+import json
+
+class Handler(BaseHTTPRequestHandler):
+    def do_POST(self):
+        body = self.rfile.read(int(self.headers["Content-Length"]))
+        print(json.dumps(json.loads(body), indent=2))
+        self.send_response(200)
+        self.end_headers()
+
+HTTPServer(("", 3000), Handler).serve_forever()
+```
+
+With this `notification.toml` in the directory where you start `weed`:
+
+```toml
+[notification.webhook]
+enabled = true
+endpoint = "http://localhost:3000/webhook"
+```
+
+start a filer (`weed mini`, `weed server -filer`, or `weed filer`) and create a file:
+
+```bash
+curl -F "file=@test.txt" "http://localhost:8888/data/"
+```
+
+The receiver prints a `create` event for `/data/test.txt`.
+
 ## Event Types
 
 The webhook notification system supports four types of file system events:
@@ -144,7 +196,7 @@ Triggered when a new file or directory is created.
 
 ### 2. Update Event
 Triggered when an existing file or directory is modified.
-- **Detection**: Both `old_entry` and `new_entry` are present, no path change
+- **Detection**: Both `old_entry` and `new_entry` are present, and neither the directory nor the file name changed
 - **Event Type**: `"update"`
 
 ### 3. Delete Event
@@ -154,7 +206,7 @@ Triggered when a file or directory is deleted.
 
 ### 4. Rename Event
 Triggered when a file or directory is moved or renamed.
-- **Detection**: Both `old_entry` and `new_entry` are present, and `new_parent_path` is specified
+- **Detection**: Both `old_entry` and `new_entry` are present, and the directory or the file name changed
 - **Event Type**: `"rename"`
 
 ## Webhook Payload Format
@@ -200,6 +252,8 @@ Triggered when a file or directory is moved or renamed.
 }
 ```
 
+Fields with empty values are omitted from the JSON: a `create` event has no `old_entry` key at all, a `delete` event has no `new_entry` key, and `delete_chunks` / `new_parent_path` only appear when set. Check for key presence rather than expecting explicit `null`s. `mtime` and `crtime` are unix time in seconds. The examples below show absent fields as `null` for readability.
+
 ## Example Payloads
 
 ### Create File Event
@@ -396,6 +450,8 @@ endpoint = "https://your-server.com/webhook"
 path_prefixes = ["/important", "/data"]
 ```
 
+Prefixes are plain string prefixes on the full path. For renames, the event matches if either the old or the new path matches a prefix.
+
 ### Combined Filtering
 
 You can combine both filters:
@@ -409,14 +465,18 @@ event_types = ["create", "delete"]
 path_prefixes = ["/important"]
 ```
 
+## Backfilling Existing Files
+
+Webhooks only fire for events that happen while the filer is running. To send events for files that already exist, run `fs.meta.notify` in `weed shell` — it walks a directory tree and publishes a `create` event for every entry to the configured notification target. Place `notification.toml` in the directory where you run `weed shell`.
+
 ## Retry and Error Handling
 
 The webhook system includes robust error handling:
 
 1. **Automatic Retries**: Failed requests are automatically retried up to `max_retries` times
-2. **Exponential Backoff**: Retry delays increase exponentially from `backoff_seconds` to `max_backoff_seconds`
-3. **Dead Letter Queue**: After exhausting retries, failed messages are logged for debugging
-4. **Status Code Validation**: Only 2xx status codes are considered successful
+2. **Exponential Backoff**: Retry delays double from `backoff_seconds` up to `max_backoff_seconds`, with up to 30% random jitter
+3. **Dead Letter Logging**: After exhausting retries, failed messages are written to the filer log (`received dead letter message: ...`); they are not persisted or redelivered
+4. **Status Code Validation**: Only 2xx status codes are considered successful; 3xx redirects are followed (the redirect target is cached for subsequent deliveries) and anything else is retried
 
 ### Example Retry Configuration
 
@@ -429,11 +489,11 @@ backoff_seconds = 2
 max_backoff_seconds = 60
 ```
 
-This configuration will retry up to 5 times with delays: 2s, 4s, 8s, 16s, 32s.
+This configuration will retry up to 5 times with delays around 2s, 4s, 8s, 16s, 32s (each with up to 30% jitter).
 
 ## Performance Tuning
 
-**⚠️ Note**: While you can tune these parameters, remember that webhooks use a push model best suited for low to moderate traffic. For high-traffic scenarios (>1000 events/second sustained), consider using Kafka, message queues, or the pull-based metadata event logs instead.
+**Note**: While you can tune these parameters, remember that webhooks use a push model best suited for low to moderate traffic. For high-traffic scenarios (>1000 events/second sustained), consider using Kafka, message queues, or the pull-based metadata event logs instead.
 
 ### Concurrent Workers
 
@@ -493,7 +553,8 @@ Your webhook receiver should:
 2. Validate request content type is `application/json`
 3. Parse and validate JSON payload structure
 4. Respond with 2xx status code for successful processing
-5. Implement idempotency to handle potential duplicate events
+5. Respond quickly — acknowledge first, then process asynchronously; a delivery worker is tied up until your endpoint responds or `timeout_seconds` elapses
+6. Implement idempotency to handle potential duplicate events (delivery is at-least-once and may be out of order)
 
 ## Example Webhook Receiver (Node.js/Express)