3 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandDevin	be081b49b7	notification: named webhook targets for per-path event routing (#11925) ... * notification: honor the config prefix in the multi-queue guard validateOneEnabledQueue checked "<name>.enabled" without the notification prefix, so the fatal-on-two-queues guard never saw the real keys and could never fire. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: support named webhook targets under [notification.webhook.<name>] The webhook queue only accepted one endpoint, so events could not be routed to different services by path prefix. Each enabled [notification.webhook.<name>] subsection now spins up its own target — own filter, http client, retry middleware and worker budget — published on a per-target topic so every matching endpoint gets each event exactly once. The flat [notification.webhook] section keeps working as the unnamed target, and named subsections alone now activate the queue. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: require enabled for the flat webhook target buildTargets included the flat section whenever an endpoint was set, so a disabled [notification.webhook] block — which the scaffold ships with a placeholder endpoint — would start receiving events as soon as any named target activated the queue. * notification: resolve named target enabled flags through config namedTargetPrefixes read the raw map value, which never sees environment variable overrides; GetBool resolves WEED_* env bindings, so a named target disabled in the file can now be switched on by WEED_NOTIFICATION_WEBHOOK_<NAME>_ENABLED. * notification: cancel the webhook queue only after all workers stop Each target's router goroutine deferred the shared cancel, so the first worker to exit tore down the queue context for every other target and the dead letter logger. * notification: isolate webhook targets behind per-target queues SendMessage published to each target's topic inline, so a subscriber channel that filled up blocked every later target and the filer's notification path with it. Each target now drains its own inbox through a forwarder goroutine; SendMessage fails fast on a full inbox, matching the "throw message" contract the filer already applies to errors. * notification: name the webhook target in failure and dead letter logs Every target dead-letters to the same topic, so without the name there is no way to tell which endpoint failed or needs a replay. * notification: assert delivery in the named targets test The test only exercised SendMessage; it now records requests on both endpoints and checks matching, non-matching, and unrouted keys reach the right servers exactly once. * notification: wait for the webhook router before accepting events * notification: apply backpressure instead of dropping a full webhook inbox * notification: bound queued and in-flight webhook deliveries per target * notification: release the pending token when shutdown races the enqueue A SendMessage that wins the pending select as the queue is canceling could block forever on a full, undrained inbox. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: stop waiting for a router that failed to start A router that exits before reaching Running() would leave initialize blocked on a context nothing had canceled yet. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: route webhook events past saturated targets A full pending reservation made SendMessage wait on the slow target before it ever reached the healthy ones after it, coupling every endpoint's latency to the most congested one. Targets that can take a reservation now get the event immediately; saturated targets defer to a second pass that still blocks for a slot, so backpressure is kept and no event is dropped. * notification: retry webhook deliveries before dead-lettering PoisonQueue must wrap the retry middleware: added after it, the poison middleware intercepts the first handler error, republishes to the dead letter topic, and returns nil — the retry chain never sees a failure, so max_retries/backoff settings were dead code and every delivery failed straight to dead letter on the first attempt. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 02:12:17 -07:00
Chris Lu	086ab3e28c	Fix webhook duplicate deliveries and POST to GET conversion (#7668) ... * Fix webhook duplicate deliveries and POST to GET conversion Fixes #7667 This commit addresses two critical issues with the webhook notification system: 1. Duplicate webhook deliveries based on worker count 2. POST requests being converted to GET when following redirects Issue 1: Multiple webhook deliveries ------------------------------------ Problem: The webhook queue was creating multiple handlers (one per worker) that all subscribed to the same topic. With Watermill's gochannel, each handler creates a separate subscription, and all subscriptions receive their own copy of every message, resulting in duplicate webhook calls equal to the worker count. Solution: Use a single handler instead of multiple handlers to ensure each webhook event is sent only once, regardless of worker configuration. Issue 2: POST to GET conversion with intelligent redirect handling ------------------------------------------------------------------ Problem: When webhook endpoints returned redirects (301/302/303), Go's default HTTP client would automatically follow them and convert POST requests to GET requests per HTTP specification. Solution: Implement intelligent redirect handling that: - Prevents automatic redirects to preserve POST method - Manually follows redirects by recreating POST requests - Caches the final redirect destination for performance - Invalidates cache and retries on failures (network or HTTP errors) - Provides automatic recovery from cached endpoint failures Benefits: - Webhooks are now sent exactly once per event - POST method is always preserved through redirects - Reduced latency through redirect destination caching - Automatic failover when cached destinations become unavailable - Thread-safe concurrent webhook delivery Testing: - Added TestQueueNoDuplicateWebhooks to verify single delivery - Added TestHttpClientFollowsRedirectAsPost for redirect handling - Added TestHttpClientUsesCachedRedirect for caching behavior - Added cache invalidation tests for error scenarios - All 18 webhook tests pass successfully * Address code review comments - Add maxWebhookRetryDepth constant to avoid magic number - Extract cache invalidation logic into invalidateCache() helper method - Fix redirect handling to properly follow redirects even on retry attempts - Remove misleading comment about nWorkers controlling handler parallelism - Fix test assertions to match actual execution flow - Remove trailing whitespace in test file All tests passing. * Refactor: use setFinalURL() instead of invalidateCache() Replace invalidateCache() with more explicit setFinalURL() function. This is cleaner as it makes the intent clear - we're setting the URL (either to a value or to empty string to clear it), rather than having a separate function just for clearing. No functional changes, all tests passing. * Add concurrent webhook delivery using nWorkers configuration Webhooks were previously sent sequentially (one-by-one), which could be a performance bottleneck for high-throughput scenarios. Now nWorkers configuration is properly used to control concurrent webhook delivery. Implementation: - Added semaphore channel (buffered to nWorkers capacity) - handleWebhook acquires semaphore slot before sending (blocks if at capacity) - Releases slot after webhook completes - Allows up to nWorkers concurrent webhook HTTP requests Benefits: - Improved throughput for slow webhook endpoints - nWorkers config now has actual purpose (was validated but unused) - Default 5 workers provides good balance - Configurable from 1-100 workers based on needs Example performance improvement: - Before: 500ms webhook latency = ~2 webhooks/sec max - After (5 workers): 500ms latency = ~10 webhooks/sec - After (10 workers): 500ms latency = ~20 webhooks/sec All tests passing. * Replace deprecated AddNoPublisherHandler with AddConsumerHandler AddNoPublisherHandler is deprecated in Watermill. Use AddConsumerHandler instead, which is the current recommended API for handlers that only consume messages without publishing. No functional changes, all tests passing. * Drain response bodies to enable HTTP connection reuse Added drainBody() calls in all code paths to ensure response bodies are consumed before returning. This is critical for HTTP keep-alive connection reuse. Without draining: - Connections are closed after each request - New TCP handshake + TLS handshake for every webhook - Higher latency and resource usage With draining: - Connections are reused via HTTP keep-alive - Significant performance improvement for repeated webhooks - Lower latency (no handshake overhead) - Reduced resource usage Implementation: - Added drainBody() helper that reads up to 1MB (prevents memory issues) - Drain on success path (line 161) - Drain on error responses before retry (lines 119, 152) - Drain on redirect responses before following (line 118) - Already had drainResponse() for network errors (line 99) All tests passing. * Use existing CloseResponse utility instead of custom drainBody Replaced custom drainBody() function with the existing util_http.CloseResponse() utility which is already used throughout the codebase. This provides: - Consistent behavior with rest of the codebase - Better logging (logs bytes drained via CountingReader) - Full body drainage (not limited to 1MB) - Cleaner code (no duplication) CloseResponse properly drains and closes the response body to enable HTTP keep-alive connection reuse. All tests passing. * Fix: Don't overwrite original error when draining response Before: err was being overwritten by drainResponse() result After: Use drainErr to avoid losing the original client.Do() error This was a subtle bug where if drainResponse() succeeded (returned nil), we would lose the original network error and potentially return a confusing error message. All tests passing. * Optimize HTTP client: reuse client and remove redundant timeout 1. Reuse single http.Client instance instead of creating new one per request - Reduces allocation overhead - More efficient for high-volume webhooks 2. Remove redundant timeout configuration - Before: timeout set on both context AND http.Client - After: timeout only on context (cleaner, context fires first anyway) Performance benefits: - Reduced GC pressure (fewer client allocations) - Better connection pooling (single transport instance) - Cleaner code (no redundancy) All tests passing.	2025-12-08 15:39:35 -08:00
Ibrahim Konsowa	d78aa3d2de	[Notifications] Improving webhook notifications (#6965) ... * worker setup * fix tests * start worker * graceful worker drain * retry queue * migrate queue to watermill * adding filters and improvements * add the event type to the webhook message * eliminating redundant JSON serialization * resolve review comments * trigger actions * fix tests * typo fixes * read max_backoff_seconds from config * add more context to the dead letter * close the http response on errors * drain the http response body in case not empty * eliminate exported typesπ	2025-07-15 10:49:37 -07:00