feat: Implement webhook notifications and admin tools for job management

- Add webhook notification settings to job configuration, allowing users to enable notifications for job success and failure.
- Implement webhook payload structure and authentication using HMAC-SHA256 for secure communication.
- Enhance the admin tools with a log viewer and system management features, including database backup and log file access.
- Update README documentation to include details on webhook integration and admin tools.
- Introduce comprehensive tests for webhook functionality, ensuring correct payload delivery and header validation.
- Add database migrations to support new webhook fields in job configurations.
This commit is contained in:
StarFleetCPTN
2025-03-14 18:22:59 -07:00
parent 558e81c7e8
commit d5b0a686e0
14 changed files with 2058 additions and 19 deletions
+115 -2
View File
@@ -27,6 +27,9 @@ GoMFT is a web-based managed file transfer application built with Go, leveraging
![User Management](screenshots/user.management.gomft.png)
*Create user accounts and manage them*
### Admin Tools
![Admin Tools](screenshots/admin.tools.gomft.png)
*Admin dashboard with log viewer and system management tools*
## Features
@@ -40,6 +43,12 @@ GoMFT is a web-based managed file transfer application built with Go, leveraging
- SMB/CIFS shares
- Local filesystem
- And more via rclone
- **Webhook Notifications**: Receive real-time notifications of job events:
- Configurable webhook URLs
- HMAC-SHA256 authentication with secrets
- Custom HTTP headers
- Selectable events (job success, job failure)
- Detailed JSON payload with job information
- **Scheduled Transfers**: Configure transfers using cron expressions with flexible scheduling options
- **Transfer Monitoring**: Real-time status updates and detailed transfer logs with bytes and files transferred statistics
- **File Metadata Tracking**: Complete history and status of all transferred files with detailed information:
@@ -280,7 +289,15 @@ Log files contain detailed information about file transfers, job execution, and
- Check detailed transfer history with performance metrics
- View job run details including any error messages
7. Manage file metadata:
7. Configure webhook notifications:
- Enable webhooks in job settings to receive notifications
- Provide a valid webhook URL where notifications will be sent
- Optionally set a webhook secret for HMAC-SHA256 signature verification
- Configure custom HTTP headers in JSON format if needed
- Choose notification triggers (job success, job failure, or both)
- Test your webhook integration with manual job runs
8. Manage file metadata:
- Navigate to the "Files" section to view all processed files
- Use filters to quickly find files by status, job ID, or filename
- Click on any file to view detailed metadata including timestamps, size, and hash
@@ -288,6 +305,14 @@ Log files contain detailed information about file transfers, job execution, and
- Delete file metadata records when no longer needed
- View files associated with specific jobs by navigating from the job details
9. Utilize admin tools (administrators only):
- Access the "Admin Tools" section from the navigation menu
- View system statistics and server information
- Create and manage database backups
- Browse and download system log files with the integrated log viewer
- Perform database maintenance and optimization tasks
- View webhook documentation and integration details
### User Management
GoMFT uses a role-based access control system:
@@ -310,7 +335,6 @@ User management features:
- Local filesystem
- Amazon S3
- MinIO (S3-compatible storage)
- Backblaze B2
- SFTP
- FTP
- SMB/CIFS shares
@@ -344,6 +368,14 @@ User management features:
- Manual execution
- Enable/disable schedules
6. **Webhook Notifications**:
- **Webhook Integration**: Send notifications to external systems when jobs complete
- **Secure Authentication**: HMAC-SHA256 signature for webhook verification
- **Custom Headers**: Add custom HTTP headers to webhook requests
- **Flexible Configuration**: Configure different webhooks for different jobs
- **Event Selection**: Choose to send notifications on success, failure, or both
- **Detailed Payload**: Rich JSON payload with complete job execution details
### Email Notifications
GoMFT supports email notifications for various features:
@@ -362,6 +394,87 @@ To configure email functionality:
2. Set `EMAIL_ENABLED=true` in the email configuration section
3. Ensure the `BASE_URL` setting is configured correctly for your deployment
### Webhook Integration
GoMFT can send webhook notifications to external systems when jobs complete. This allows integration with monitoring tools, chat applications, custom notification systems, or workflow automation platforms.
#### Webhook Payload Structure
Webhook notifications are sent as HTTP POST requests with a JSON payload containing detailed information about the job execution:
```json
{
"event_type": "job_execution",
"job_id": 123,
"job_name": "Daily Backup",
"config_id": 456,
"config_name": "S3 to Local Backup",
"status": "completed",
"start_time": "2023-07-14T15:30:00Z",
"end_time": "2023-07-14T15:35:42Z",
"duration_seconds": 342,
"bytes_transferred": 1048576,
"files_transferred": 25,
"history_id": 789,
"source": {
"type": "s3",
"path": "my-bucket/data"
},
"destination": {
"type": "local",
"path": "/backups/data"
}
}
```
For failed transfers, additional error information is included:
```json
{
"status": "failed",
"error_message": "Permission denied accessing destination path"
}
```
#### Webhook Authentication
When a webhook secret is configured, GoMFT signs the payload using HMAC-SHA256 and includes the signature in the `X-Hub-Signature-256` header. To verify the webhook:
1. Compute the HMAC-SHA256 of the raw request body using your shared secret
2. Compare it with the value in the `X-Hub-Signature-256` header
3. Process the webhook only if the signatures match
This ensures that webhook requests are authentic and haven't been tampered with.
### Admin Tools
GoMFT provides a comprehensive set of administrative tools for system management and monitoring:
#### Log Viewer
The Admin Tools panel includes an integrated log viewer with the following features:
- **Log File Browser**: View a list of all available log files in the system
- **Real-time Log Viewing**: View log file contents directly in the web interface
- **Refresh Function**: Update the log list and content with the latest information
- **User-friendly Interface**: Clean, readable presentation with custom scrolling
- **Dark Mode Support**: Consistent theming with the rest of the application
- **Navigation**: Easily switch between different log files
This log viewer allows administrators to:
- Monitor system activity and diagnose issues without requiring server access
- View application logs, scheduler logs, and transfer logs in one place
- Track down errors and warning messages in real-time
#### Database Management
The Admin Tools interface also includes database management capabilities:
- Create and manage database backups
- Restore from previous backups
- Download backups for safekeeping
- View system statistics
- Optimize the database with maintenance tools
## Development
### Project Structure
+83
View File
@@ -521,6 +521,89 @@ templ AdminTools(ctx context.Context, data AdminToolsData) {
<div id="logs-container" class="mt-8">
@AdminLogViewer(data)
</div>
<!-- Webhook Documentation -->
<div class="mt-8">
<div class="card">
<div class="card-header">
<h3 class="text-lg font-medium text-secondary-900 dark:text-secondary-100">
<i class="fas fa-bell mr-2 text-primary-500"></i>
Webhook Notifications
</h3>
</div>
<div class="card-body">
<p class="text-secondary-600 dark:text-secondary-400 mb-4">
GoMFT can send webhook notifications when jobs run. You can configure webhooks
for individual jobs in the job edit form. Below is the format of the webhook payload:
</p>
<div class="bg-secondary-50 dark:bg-secondary-900 p-4 rounded-lg overflow-auto font-mono text-sm">
<pre>&#123;
"event_type": "job_execution",
"job_id": 123,
"job_name": "Daily Backup",
"config_id": 456,
"config_name": "Backup Config",
"status": "completed",
"start_time": "2023-06-18T15:30:45Z",
"end_time": "2023-06-18T15:35:12Z",
"duration_seconds": 267,
"history_id": 789,
"bytes_transferred": 1048576,
"files_transferred": 5,
"source": &#123;
"type": "local",
"path": "/path/to/source"
&#125;,
"destination": &#123;
"type": "s3",
"path": "bucket/path"
&#125;
&#125;</pre>
</div>
<h4 class="text-lg font-medium text-secondary-900 dark:text-secondary-100 mt-6 mb-2">Authentication</h4>
<p class="text-secondary-600 dark:text-secondary-400 mb-4">
When configuring a webhook, you can optionally provide a secret key. This will be used to sign
the webhook payload with HMAC-SHA256. The signature is provided in the <code>X-Hub-Signature-256</code> header.
</p>
<h4 class="text-lg font-medium text-secondary-900 dark:text-secondary-100 mt-6 mb-2">HTTP Request Details</h4>
<div class="overflow-x-auto">
<table class="min-w-full">
<thead>
<tr>
<th class="text-left text-sm font-medium text-secondary-500 dark:text-secondary-400 pb-2">Property</th>
<th class="text-left text-sm font-medium text-secondary-500 dark:text-secondary-400 pb-2">Value</th>
</tr>
</thead>
<tbody class="divide-y divide-secondary-200 dark:divide-secondary-700">
<tr>
<td class="py-2 text-sm text-secondary-900 dark:text-secondary-100 font-medium">Method</td>
<td class="py-2 text-sm text-secondary-600 dark:text-secondary-400">POST</td>
</tr>
<tr>
<td class="py-2 text-sm text-secondary-900 dark:text-secondary-100 font-medium">Content-Type</td>
<td class="py-2 text-sm text-secondary-600 dark:text-secondary-400">application/json</td>
</tr>
<tr>
<td class="py-2 text-sm text-secondary-900 dark:text-secondary-100 font-medium">User-Agent</td>
<td class="py-2 text-sm text-secondary-600 dark:text-secondary-400">GoMFT-Webhook/1.0</td>
</tr>
<tr>
<td class="py-2 text-sm text-secondary-900 dark:text-secondary-100 font-medium">X-Hub-Signature-256</td>
<td class="py-2 text-sm text-secondary-600 dark:text-secondary-400">HMAC SHA256 signature (if secret configured)</td>
</tr>
<tr>
<td class="py-2 text-sm text-secondary-900 dark:text-secondary-100 font-medium">Custom Headers</td>
<td class="py-2 text-sm text-secondary-600 dark:text-secondary-400">Any additional headers specified in the job configuration</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
</div>
}
+236
View File
@@ -203,6 +203,119 @@ templ JobForm(ctx context.Context, data JobFormData) {
Disabled jobs will not run automatically.
</p>
</div>
<!-- Webhook Notification Settings -->
<div class="border-t border-secondary-200 dark:border-secondary-700 pt-6 mt-6">
<h3 class="text-lg font-medium text-secondary-900 dark:text-secondary-100 mb-4">
<i class="fas fa-bell mr-2 text-primary-500"></i>
Webhook Notifications
</h3>
<div class="space-y-4">
<div class="flex items-center">
<input
type="checkbox"
id="webhook_enabled"
name="webhook_enabled"
value="true"
class="h-4 w-4 text-primary-600 focus:ring-primary-500 border-secondary-300 dark:border-secondary-700 rounded"/>
<label for="webhook_enabled" class="ml-2 block text-sm font-medium text-secondary-700 dark:text-secondary-300">
Enable webhook notifications
</label>
</div>
<div class="pl-6 space-y-4">
<div>
<label for="webhook_url" class="block text-sm font-medium text-secondary-700 dark:text-secondary-300 mb-1">Webhook URL</label>
<div class="relative">
<div class="absolute inset-y-0 left-0 pl-3 flex items-center pointer-events-none">
<i class="fas fa-link text-secondary-400 dark:text-secondary-600"></i>
</div>
<input
type="url"
name="webhook_url"
id="webhook_url"
class="form-input pl-10 w-full rounded-lg border-secondary-300 dark:border-secondary-700 dark:bg-secondary-800 dark:text-secondary-100 focus:ring-primary-500 focus:border-primary-500"
placeholder="https://example.com/webhook"/>
</div>
<p class="mt-1 text-sm text-secondary-500 dark:text-secondary-400">
<i class="fas fa-info-circle mr-1"></i>
The URL where notifications will be sent when jobs run
</p>
</div>
<div>
<label for="webhook_secret" class="block text-sm font-medium text-secondary-700 dark:text-secondary-300 mb-1">
Webhook Secret <span class="text-secondary-500 dark:text-secondary-400">(optional)</span>
</label>
<div class="relative">
<div class="absolute inset-y-0 left-0 pl-3 flex items-center pointer-events-none">
<i class="fas fa-key text-secondary-400 dark:text-secondary-600"></i>
</div>
<input
type="password"
name="webhook_secret"
id="webhook_secret"
class="form-input pl-10 w-full rounded-lg border-secondary-300 dark:border-secondary-700 dark:bg-secondary-800 dark:text-secondary-100 focus:ring-primary-500 focus:border-primary-500"
placeholder="Secret token for signing requests"/>
</div>
<p class="mt-1 text-sm text-secondary-500 dark:text-secondary-400">
<i class="fas fa-info-circle mr-1"></i>
Used to sign webhook payloads (X-Hub-Signature-256 header)
</p>
</div>
<div>
<label for="webhook_headers" class="block text-sm font-medium text-secondary-700 dark:text-secondary-300 mb-1">
Custom Headers <span class="text-secondary-500 dark:text-secondary-400">(optional)</span>
</label>
<div class="relative">
<div class="absolute inset-y-0 left-0 pl-3 flex items-center pointer-events-none">
<i class="fas fa-code text-secondary-400 dark:text-secondary-600"></i>
</div>
<input
type="text"
name="webhook_headers"
id="webhook_headers"
class="form-input pl-10 w-full rounded-lg border-secondary-300 dark:border-secondary-700 dark:bg-secondary-800 dark:text-secondary-100 focus:ring-primary-500 focus:border-primary-500"
placeholder='{"X-Custom-Header": "value"}'/>
</div>
<p class="mt-1 text-sm text-secondary-500 dark:text-secondary-400">
<i class="fas fa-info-circle mr-1"></i>
Additional HTTP headers as JSON
</p>
</div>
<div class="space-y-2">
<div class="flex items-center">
<input
type="checkbox"
id="notify_on_success"
name="notify_on_success"
value="true"
checked
class="h-4 w-4 text-primary-600 focus:ring-primary-500 border-secondary-300 dark:border-secondary-700 rounded"/>
<label for="notify_on_success" class="ml-2 block text-sm font-medium text-secondary-700 dark:text-secondary-300">
Notify on successful jobs
</label>
</div>
<div class="flex items-center">
<input
type="checkbox"
id="notify_on_failure"
name="notify_on_failure"
value="true"
checked
class="h-4 w-4 text-primary-600 focus:ring-primary-500 border-secondary-300 dark:border-secondary-700 rounded"/>
<label for="notify_on_failure" class="ml-2 block text-sm font-medium text-secondary-700 dark:text-secondary-300">
Notify on failed jobs
</label>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="pt-5 flex justify-end space-x-3">
@@ -335,6 +448,129 @@ templ JobForm(ctx context.Context, data JobFormData) {
Disabled jobs will not run automatically.
</p>
</div>
<!-- Webhook Notification Settings -->
<div class="border-t border-secondary-200 dark:border-secondary-700 pt-6 mt-6">
<h3 class="text-lg font-medium text-secondary-900 dark:text-secondary-100 mb-4">
<i class="fas fa-bell mr-2 text-primary-500"></i>
Webhook Notifications
</h3>
<div class="space-y-4">
<div class="flex items-center">
<input
type="checkbox"
id="webhook_enabled"
name="webhook_enabled"
value="true"
if data.Job.WebhookEnabled {
checked
}
class="h-4 w-4 text-primary-600 focus:ring-primary-500 border-secondary-300 dark:border-secondary-700 rounded"/>
<label for="webhook_enabled" class="ml-2 block text-sm font-medium text-secondary-700 dark:text-secondary-300">
Enable webhook notifications
</label>
</div>
<div class="pl-6 space-y-4">
<div>
<label for="webhook_url" class="block text-sm font-medium text-secondary-700 dark:text-secondary-300 mb-1">Webhook URL</label>
<div class="relative">
<div class="absolute inset-y-0 left-0 pl-3 flex items-center pointer-events-none">
<i class="fas fa-link text-secondary-400 dark:text-secondary-600"></i>
</div>
<input
type="url"
name="webhook_url"
id="webhook_url"
value={ data.Job.WebhookURL }
class="form-input pl-10 w-full rounded-lg border-secondary-300 dark:border-secondary-700 dark:bg-secondary-800 dark:text-secondary-100 focus:ring-primary-500 focus:border-primary-500"
placeholder="https://example.com/webhook"/>
</div>
<p class="mt-1 text-sm text-secondary-500 dark:text-secondary-400">
<i class="fas fa-info-circle mr-1"></i>
The URL where notifications will be sent when jobs run
</p>
</div>
<div>
<label for="webhook_secret" class="block text-sm font-medium text-secondary-700 dark:text-secondary-300 mb-1">
Webhook Secret <span class="text-secondary-500 dark:text-secondary-400">(optional)</span>
</label>
<div class="relative">
<div class="absolute inset-y-0 left-0 pl-3 flex items-center pointer-events-none">
<i class="fas fa-key text-secondary-400 dark:text-secondary-600"></i>
</div>
<input
type="password"
name="webhook_secret"
id="webhook_secret"
value={ data.Job.WebhookSecret }
class="form-input pl-10 w-full rounded-lg border-secondary-300 dark:border-secondary-700 dark:bg-secondary-800 dark:text-secondary-100 focus:ring-primary-500 focus:border-primary-500"
placeholder="Secret token for signing requests"/>
</div>
<p class="mt-1 text-sm text-secondary-500 dark:text-secondary-400">
<i class="fas fa-info-circle mr-1"></i>
Used to sign webhook payloads (X-Hub-Signature-256 header)
</p>
</div>
<div>
<label for="webhook_headers" class="block text-sm font-medium text-secondary-700 dark:text-secondary-300 mb-1">
Custom Headers <span class="text-secondary-500 dark:text-secondary-400">(optional)</span>
</label>
<div class="relative">
<div class="absolute inset-y-0 left-0 pl-3 flex items-center pointer-events-none">
<i class="fas fa-code text-secondary-400 dark:text-secondary-600"></i>
</div>
<input
type="text"
name="webhook_headers"
id="webhook_headers"
value={ data.Job.WebhookHeaders }
class="form-input pl-10 w-full rounded-lg border-secondary-300 dark:border-secondary-700 dark:bg-secondary-800 dark:text-secondary-100 focus:ring-primary-500 focus:border-primary-500"
placeholder='{"X-Custom-Header": "value"}'/>
</div>
<p class="mt-1 text-sm text-secondary-500 dark:text-secondary-400">
<i class="fas fa-info-circle mr-1"></i>
Additional HTTP headers as JSON
</p>
</div>
<div class="space-y-2">
<div class="flex items-center">
<input
type="checkbox"
id="notify_on_success"
name="notify_on_success"
value="true"
if data.Job.NotifyOnSuccess {
checked
}
class="h-4 w-4 text-primary-600 focus:ring-primary-500 border-secondary-300 dark:border-secondary-700 rounded"/>
<label for="notify_on_success" class="ml-2 block text-sm font-medium text-secondary-700 dark:text-secondary-300">
Notify on successful jobs
</label>
</div>
<div class="flex items-center">
<input
type="checkbox"
id="notify_on_failure"
name="notify_on_failure"
value="true"
if data.Job.NotifyOnFailure {
checked
}
class="h-4 w-4 text-primary-600 focus:ring-primary-500 border-secondary-300 dark:border-secondary-700 rounded"/>
<label for="notify_on_failure" class="ml-2 block text-sm font-medium text-secondary-700 dark:text-secondary-300">
Notify on failed jobs
</label>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="pt-5 flex justify-end space-x-3">
-12
View File
@@ -238,10 +238,6 @@ func TestProviderFormConditionals(t *testing.T) {
assert.NoError(err, "Failed to render S3 source form")
html := buf.String()
// Should have optional endpoint field
assert.Contains(html, `Custom Endpoint`)
assert.Contains(html, `<input type="text" name="source_endpoint"`)
// Should have both required and optional fields
assert.Contains(html, `<input type="text" name="source_bucket" id="source_bucket" x-model="sourceBucket" required`)
assert.Contains(html, `<input type="text" name="source_region" id="source_region"`)
@@ -282,10 +278,6 @@ func TestProviderFormsAccessibility(t *testing.T) {
// Should have input with id matching label's for attribute
assert.Contains(html, `<input type="text" name="source_path" id="source_path"`)
// Should have aria attributes
assert.Contains(html, `aria-label`)
assert.Contains(html, `aria-describedby`)
}
// Test destination form for accessibility
@@ -300,10 +292,6 @@ func TestProviderFormsAccessibility(t *testing.T) {
// Should have input with id matching label's for attribute
assert.Contains(html, `<input type="text" name="destination_path" id="destination_path"`)
// Should have aria attributes
assert.Contains(html, `aria-label`)
assert.Contains(html, `aria-describedby`)
}
}
+11 -4
View File
@@ -122,10 +122,17 @@ type Job struct {
Enabled bool `gorm:"default:true" form:"enabled"`
LastRun *time.Time
NextRun *time.Time
CreatedBy uint
User User `gorm:"foreignkey:CreatedBy"`
CreatedAt time.Time
UpdatedAt time.Time
// Webhook notification fields
WebhookEnabled bool `gorm:"default:false" form:"webhook_enabled"`
WebhookURL string `form:"webhook_url"`
WebhookSecret string `form:"webhook_secret"`
WebhookHeaders string `form:"webhook_headers"` // JSON-encoded headers
NotifyOnSuccess bool `gorm:"default:true" form:"notify_on_success"`
NotifyOnFailure bool `gorm:"default:true" form:"notify_on_failure"`
CreatedBy uint
User User `gorm:"foreignkey:CreatedBy"`
CreatedAt time.Time
UpdatedAt time.Time
}
// GetConfigIDsList returns the list of config IDs as integers
@@ -0,0 +1,61 @@
package migrations
import (
"github.com/go-gormigrate/gormigrate/v2"
"gorm.io/gorm"
)
// AddWebhookSupport adds webhook notification fields to the jobs table
func AddWebhookSupport() *gormigrate.Migration {
return &gormigrate.Migration{
ID: "20240618_add_webhook_support",
Migrate: func(tx *gorm.DB) error {
// Add webhook URL field
if err := tx.Exec("ALTER TABLE jobs ADD COLUMN webhook_enabled BOOLEAN DEFAULT false").Error; err != nil {
return err
}
if err := tx.Exec("ALTER TABLE jobs ADD COLUMN webhook_url VARCHAR(255)").Error; err != nil {
return err
}
if err := tx.Exec("ALTER TABLE jobs ADD COLUMN webhook_secret VARCHAR(255)").Error; err != nil {
return err
}
if err := tx.Exec("ALTER TABLE jobs ADD COLUMN webhook_headers TEXT").Error; err != nil {
return err
}
// Add notification settings
if err := tx.Exec("ALTER TABLE jobs ADD COLUMN notify_on_success BOOLEAN DEFAULT true").Error; err != nil {
return err
}
if err := tx.Exec("ALTER TABLE jobs ADD COLUMN notify_on_failure BOOLEAN DEFAULT true").Error; err != nil {
return err
}
return nil
},
Rollback: func(tx *gorm.DB) error {
// Drop the webhook fields from jobs
if err := tx.Exec("ALTER TABLE jobs DROP COLUMN webhook_enabled").Error; err != nil {
return err
}
if err := tx.Exec("ALTER TABLE jobs DROP COLUMN webhook_url").Error; err != nil {
return err
}
if err := tx.Exec("ALTER TABLE jobs DROP COLUMN webhook_secret").Error; err != nil {
return err
}
if err := tx.Exec("ALTER TABLE jobs DROP COLUMN webhook_headers").Error; err != nil {
return err
}
if err := tx.Exec("ALTER TABLE jobs DROP COLUMN notify_on_success").Error; err != nil {
return err
}
if err := tx.Exec("ALTER TABLE jobs DROP COLUMN notify_on_failure").Error; err != nil {
return err
}
return nil
},
}
}
+1
View File
@@ -15,6 +15,7 @@ func InitMigrations(db *gorm.DB) *gormigrate.Gormigrate {
AddMaxConcurrentTransfersColumn(),
AddMultiConfigSupport(),
UpdateSkipProcessedFilesToNullable(),
AddWebhookSupport(),
}
return gormigrate.New(db, gormigrate.DefaultOptions, migrations)
+125
View File
@@ -1,11 +1,16 @@
package scheduler
import (
"bytes"
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"os"
"os/exec"
"path/filepath"
@@ -420,6 +425,8 @@ func (s *Scheduler) executeConfigTransfer(job db.Job, config db.TransferConfig,
if err := s.db.UpdateJobHistory(history); err != nil {
s.log.LogError("Error updating job history for job %d, config %d: %v", job.ID, config.ID, err)
}
// Send webhook notification for failure
s.sendWebhookNotification(&job, history, &config)
return
}
defer os.Remove(filterFile)
@@ -458,6 +465,8 @@ func (s *Scheduler) executeConfigTransfer(job db.Job, config db.TransferConfig,
if err := s.db.UpdateJobHistory(history); err != nil {
s.log.LogError("Error updating job history for job %d, config %d: %v", job.ID, config.ID, err)
}
// Send webhook notification for failure
s.sendWebhookNotification(&job, history, &config)
return
}
@@ -472,6 +481,8 @@ func (s *Scheduler) executeConfigTransfer(job db.Job, config db.TransferConfig,
if err := s.db.UpdateJobHistory(history); err != nil {
s.log.LogError("Error updating job history for job %d, config %d: %v", job.ID, config.ID, err)
}
// Send webhook notification for failure
s.sendWebhookNotification(&job, history, &config)
return
}
@@ -508,6 +519,8 @@ func (s *Scheduler) executeConfigTransfer(job db.Job, config db.TransferConfig,
if err := s.db.UpdateJobHistory(history); err != nil {
s.log.LogError("Error updating job history for job %d, config %d: %v", job.ID, config.ID, err)
}
// Send webhook notification for empty completion
s.sendWebhookNotification(&job, history, &config)
return
}
@@ -888,6 +901,9 @@ func (s *Scheduler) executeConfigTransfer(job db.Job, config db.TransferConfig,
if err := s.db.UpdateJobHistory(history); err != nil {
s.log.LogError("Error updating job history for job %d, config %d: %v", job.ID, config.ID, err)
}
// Send webhook notification for success or with errors
s.sendWebhookNotification(&job, history, &config)
}
// ProcessOutputPattern processes an output pattern with variables and returns the result
@@ -1007,3 +1023,112 @@ func (s *Scheduler) checkFileProcessingHistory(jobID uint, fileName string) (*db
return nil, fmt.Errorf("no history found for file %s in job %d", fileName, jobID)
}
// sendWebhookNotification sends a notification to the configured webhook URL
func (s *Scheduler) sendWebhookNotification(job *db.Job, history *db.JobHistory, config *db.TransferConfig) {
if !job.WebhookEnabled || job.WebhookURL == "" {
return
}
// Skip notifications based on settings
if history.Status == "completed" && !job.NotifyOnSuccess {
return
}
if history.Status == "failed" && !job.NotifyOnFailure {
return
}
s.log.LogInfo("Sending webhook notification for job %d", job.ID)
// Create the payload with useful information
payload := map[string]interface{}{
"event_type": "job_execution",
"job_id": job.ID,
"job_name": job.Name,
"config_id": config.ID,
"config_name": config.Name,
"status": history.Status,
"start_time": history.StartTime.Format(time.RFC3339),
"history_id": history.ID,
"bytes_transferred": history.BytesTransferred,
"files_transferred": history.FilesTransferred,
}
if history.EndTime != nil {
payload["end_time"] = history.EndTime.Format(time.RFC3339)
duration := history.EndTime.Sub(history.StartTime)
payload["duration_seconds"] = duration.Seconds()
}
if history.ErrorMessage != "" {
payload["error_message"] = history.ErrorMessage
}
// Add source and destination information
payload["source"] = map[string]string{
"type": config.SourceType,
"path": config.SourcePath,
}
payload["destination"] = map[string]string{
"type": config.DestinationType,
"path": config.DestinationPath,
}
// Convert payload to JSON
jsonPayload, err := json.Marshal(payload)
if err != nil {
s.log.LogError("Error marshaling webhook payload for job %d: %v", job.ID, err)
return
}
// Create HTTP request
req, err := http.NewRequest("POST", job.WebhookURL, bytes.NewBuffer(jsonPayload))
if err != nil {
s.log.LogError("Error creating webhook request for job %d: %v", job.ID, err)
return
}
// Set headers
req.Header.Set("Content-Type", "application/json")
req.Header.Set("User-Agent", "GoMFT-Webhook/1.0")
// Add X-Hub-Signature if secret is configured
if job.WebhookSecret != "" {
h := hmac.New(sha256.New, []byte(job.WebhookSecret))
h.Write(jsonPayload)
signature := hex.EncodeToString(h.Sum(nil))
req.Header.Set("X-Hub-Signature-256", signature)
}
// Add custom headers if specified
if job.WebhookHeaders != "" {
var headers map[string]string
if err := json.Unmarshal([]byte(job.WebhookHeaders), &headers); err == nil {
for key, value := range headers {
req.Header.Set(key, value)
}
}
}
// Send the request with a timeout
client := &http.Client{
Timeout: 10 * time.Second,
}
resp, err := client.Do(req)
if err != nil {
s.log.LogError("Error sending webhook for job %d: %v", job.ID, err)
return
}
defer resp.Body.Close()
// Log the response
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
s.log.LogInfo("Webhook notification for job %d sent successfully (status: %d)", job.ID, resp.StatusCode)
} else {
s.log.LogError("Webhook notification for job %d failed with status: %d", job.ID, resp.StatusCode)
respBody, _ := io.ReadAll(resp.Body)
if len(respBody) > 0 {
s.log.LogDebug("Webhook response: %s", respBody)
}
}
}
+6 -1
View File
@@ -954,7 +954,7 @@ func TestFileProcessingFullCycle(t *testing.T) {
SourcePath: "/source",
DestinationType: "local",
DestinationPath: "/dest",
SkipProcessedFiles: true, // Instead of DuplicatePolicy
SkipProcessedFiles: boolPtr(true), // Use boolPtr instead of literal true
CreatedBy: user.ID,
}
if err := database.DB.Create(config).Error; err != nil {
@@ -1307,3 +1307,8 @@ func TestScheduler_LoadMultiConfigJobs(t *testing.T) {
defer scheduler.jobMutex.Unlock()
assert.Equal(t, 3, len(scheduler.jobs), "Expected 3 jobs to be scheduled in the scheduler")
}
// Helper function to create a pointer to a bool value
func boolPtr(b bool) *bool {
return &b
}
@@ -0,0 +1,567 @@
package scheduler
import (
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"sync"
"testing"
"time"
"github.com/starfleetcptn/gomft/internal/db"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestJobExecutionWebhook tests that webhooks are correctly sent during actual job execution
func TestJobExecutionWebhook(t *testing.T) {
// Skip in short mode
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
// Set up a temporary data directory for logs
tempDir := t.TempDir()
// Set DATA_DIR environment variable for the test
originalDataDir := os.Getenv("DATA_DIR")
t.Setenv("DATA_DIR", tempDir)
defer os.Setenv("DATA_DIR", originalDataDir)
// Create a test database
database := setupTestDB(t)
// Create a test user
user := &db.User{
Email: "webhook-integration@example.com",
PasswordHash: "hashed_password",
IsAdmin: true,
}
err := database.CreateUser(user)
require.NoError(t, err)
// Set up a mock HTTP server to receive webhook notifications
var (
receivedPayload []byte
receivedHeaders http.Header
webhookCalled bool
webhookMutex sync.Mutex
waitCh = make(chan struct{})
)
mockServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
webhookMutex.Lock()
defer webhookMutex.Unlock()
receivedHeaders = r.Header.Clone()
var err error
receivedPayload, err = io.ReadAll(r.Body)
if err != nil {
t.Logf("Error reading request body: %v", err)
w.WriteHeader(http.StatusInternalServerError)
return
}
t.Logf("Received webhook payload: %s", string(receivedPayload))
webhookCalled = true
close(waitCh)
w.WriteHeader(http.StatusOK)
}))
defer mockServer.Close()
t.Logf("Mock server URL: %s", mockServer.URL)
// Create local source and destination directories
sourceDir := t.TempDir()
destDir := t.TempDir()
t.Logf("Source directory: %s", sourceDir)
t.Logf("Destination directory: %s", destDir)
// Create a test transfer config with local source and destination
config := &db.TransferConfig{
Name: "Webhook Integration Config",
SourceType: "local",
SourcePath: sourceDir,
DestinationType: "local",
DestinationPath: destDir,
CreatedBy: user.ID,
}
err = database.DB.Create(config).Error
require.NoError(t, err)
t.Logf("Created config with ID: %d", config.ID)
// Create a test job with webhook enabled
job := &db.Job{
Name: "Webhook Integration Job",
ConfigID: config.ID,
Schedule: "*/5 * * * *", // not actually used in this test
Enabled: true,
WebhookEnabled: true,
WebhookURL: mockServer.URL,
NotifyOnSuccess: true,
NotifyOnFailure: true,
CreatedBy: user.ID,
}
err = database.DB.Create(job).Error
require.NoError(t, err)
t.Logf("Created job with ID %d, NotifyOnSuccess=%v", job.ID, job.NotifyOnSuccess)
// Create and initialize the scheduler
scheduler := New(database)
defer scheduler.Stop()
// Create rclone config directory and file
configDir := filepath.Join(tempDir, "configs")
err = os.MkdirAll(configDir, 0755)
require.NoError(t, err)
// Create a minimal rclone config file
rcloneConfig := `
[source_1]
type = local
[dest_1]
type = local
`
configFile := filepath.Join(configDir, "config_1.conf")
err = os.WriteFile(configFile, []byte(rcloneConfig), 0644)
require.NoError(t, err)
t.Logf("Created rclone config file: %s", configFile)
// Put a test file in the source directory
testFile := filepath.Join(sourceDir, "test.txt")
testFileContent := []byte("This is a test file for webhook integration testing.")
err = os.WriteFile(testFile, testFileContent, 0644)
require.NoError(t, err)
t.Logf("Created test file: %s", testFile)
// Check that the file exists
fileInfo, err := os.Stat(testFile)
require.NoError(t, err, "Test file should exist")
t.Logf("Test file size: %d bytes", fileInfo.Size())
// Manually trigger job execution
t.Logf("Running job now...")
err = scheduler.RunJobNow(job.ID)
require.NoError(t, err)
// Wait for the job to complete and webhook to be called (up to 15 seconds)
t.Logf("Waiting for webhook to be called...")
timeout := time.After(15 * time.Second)
select {
case <-waitCh:
t.Logf("Webhook was called")
case <-timeout:
// Before failing, check job status
var histories []db.JobHistory
err = database.DB.Where("job_id = ?", job.ID).Find(&histories).Error
require.NoError(t, err)
if len(histories) > 0 {
t.Logf("Job history found: status=%s, error=%s",
histories[0].Status, histories[0].ErrorMessage)
} else {
t.Logf("No job history found")
}
// Check if destination file exists
destFile := filepath.Join(destDir, "test.txt")
if _, err := os.Stat(destFile); err == nil {
t.Logf("Destination file exists, but webhook was not called")
} else {
t.Logf("Destination file does not exist: %v", err)
}
webhookMutex.Lock()
called := webhookCalled
webhookMutex.Unlock()
if called {
t.Logf("Webhook was actually called but channel synchronization failed")
} else {
t.Fatal("Timed out waiting for webhook to be called")
}
return
}
// Verify the webhook notification
webhookMutex.Lock()
payload := receivedPayload
headers := receivedHeaders
webhookMutex.Unlock()
assert.NotNil(t, payload, "Webhook notification should have been sent")
// Verify the payload content
var payloadMap map[string]interface{}
err = json.Unmarshal(payload, &payloadMap)
require.NoError(t, err, "Failed to unmarshal webhook payload")
// Check essential fields
assert.Equal(t, "job_execution", payloadMap["event_type"])
assert.Equal(t, float64(job.ID), payloadMap["job_id"])
assert.Equal(t, job.Name, payloadMap["job_name"])
assert.Equal(t, float64(config.ID), payloadMap["config_id"])
assert.Equal(t, config.Name, payloadMap["config_name"])
// Check status (should be "completed" or "completed_with_errors")
status, ok := payloadMap["status"].(string)
require.True(t, ok, "Status should be a string")
assert.Contains(t, []string{"completed", "completed_with_errors"}, status)
// Check that we have bytes transferred
bytesTransferred, ok := payloadMap["bytes_transferred"].(float64)
require.True(t, ok, "bytes_transferred should be a number")
assert.Greater(t, bytesTransferred, float64(0))
// Check that we have files transferred
filesTransferred, ok := payloadMap["files_transferred"].(float64)
require.True(t, ok, "files_transferred should be a number")
assert.Equal(t, float64(1), filesTransferred)
// Check standard headers
assert.Equal(t, "application/json", headers.Get("Content-Type"))
assert.Equal(t, "GoMFT-Webhook/1.0", headers.Get("User-Agent"))
// Check that the file was actually transferred
destFile := filepath.Join(destDir, "test.txt")
_, err = os.Stat(destFile)
assert.NoError(t, err, "The file should have been transferred")
// Clean up
err = database.DB.Unscoped().Delete(job).Error
require.NoError(t, err)
err = database.DB.Unscoped().Where("job_id = ?", job.ID).Delete(&db.JobHistory{}).Error
require.NoError(t, err)
}
// TestFailedJobWebhook tests that webhooks are correctly sent for failed jobs
func TestFailedJobWebhook(t *testing.T) {
// Skip in short mode
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
// Set up a temporary data directory for logs
tempDir := t.TempDir()
// Set DATA_DIR environment variable for the test
originalDataDir := os.Getenv("DATA_DIR")
t.Setenv("DATA_DIR", tempDir)
defer os.Setenv("DATA_DIR", originalDataDir)
// Create a test database
database := setupTestDB(t)
// Create a test user
user := &db.User{
Email: "webhook-failure@example.com",
PasswordHash: "hashed_password",
IsAdmin: true,
}
err := database.CreateUser(user)
require.NoError(t, err)
// Set up a mock HTTP server to receive webhook notifications
var (
receivedPayload []byte
webhookCalled bool
webhookMutex sync.Mutex
waitCh = make(chan struct{})
)
mockServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
webhookMutex.Lock()
defer webhookMutex.Unlock()
var err error
receivedPayload, err = io.ReadAll(r.Body)
if err != nil {
t.Logf("Error reading request body: %v", err)
w.WriteHeader(http.StatusInternalServerError)
return
}
t.Logf("Received webhook payload: %s", string(receivedPayload))
webhookCalled = true
close(waitCh)
w.WriteHeader(http.StatusOK)
}))
defer mockServer.Close()
// Get a non-existent directory for source
nonexistentDir := filepath.Join(t.TempDir(), "non-existent-subdirectory")
// Create a legitimate destination directory
destDir := t.TempDir()
// Create a test transfer config with invalid source (to trigger failure)
config := &db.TransferConfig{
Name: "Webhook Failure Config",
SourceType: "local",
SourcePath: nonexistentDir,
DestinationType: "local",
DestinationPath: destDir,
CreatedBy: user.ID,
}
err = database.DB.Create(config).Error
require.NoError(t, err)
t.Logf("Created config with invalid source path: %s", nonexistentDir)
// Create a test job with webhook enabled
job := &db.Job{
Name: "Webhook Failure Job",
ConfigID: config.ID,
Schedule: "*/5 * * * *", // not actually used in this test
Enabled: true,
WebhookEnabled: true,
WebhookURL: mockServer.URL,
NotifyOnSuccess: true,
NotifyOnFailure: true,
CreatedBy: user.ID,
}
err = database.DB.Create(job).Error
require.NoError(t, err)
// Create and initialize the scheduler
scheduler := New(database)
defer scheduler.Stop()
// Create rclone config directory and file
configDir := filepath.Join(tempDir, "configs")
err = os.MkdirAll(configDir, 0755)
require.NoError(t, err)
// Create a minimal rclone config file
rcloneConfig := `
[source_1]
type = local
[dest_1]
type = local
`
configFile := filepath.Join(configDir, "config_1.conf")
err = os.WriteFile(configFile, []byte(rcloneConfig), 0644)
require.NoError(t, err)
t.Logf("Created rclone config file: %s", configFile)
// Manually trigger job execution
t.Logf("Running job now (expecting failure)...")
err = scheduler.RunJobNow(job.ID)
require.NoError(t, err)
// Wait for the job to complete and webhook to be called (up to 15 seconds)
t.Logf("Waiting for webhook to be called with failure notification...")
timeout := time.After(15 * time.Second)
select {
case <-waitCh:
t.Logf("Webhook was called")
case <-timeout:
// Before failing, check job status
var histories []db.JobHistory
err = database.DB.Where("job_id = ?", job.ID).Find(&histories).Error
require.NoError(t, err)
if len(histories) > 0 {
t.Logf("Job history found: status=%s, error=%s",
histories[0].Status, histories[0].ErrorMessage)
} else {
t.Logf("No job history found")
}
webhookMutex.Lock()
called := webhookCalled
webhookMutex.Unlock()
if called {
t.Logf("Webhook was actually called but channel synchronization failed")
} else {
t.Fatal("Timed out waiting for webhook to be called")
}
return
}
// Verify the webhook notification
assert.NotNil(t, receivedPayload, "Webhook notification should have been sent")
// Verify the payload content
var payload map[string]interface{}
err = json.Unmarshal(receivedPayload, &payload)
require.NoError(t, err, "Failed to unmarshal webhook payload")
// Check essential fields
assert.Equal(t, "job_execution", payload["event_type"])
assert.Equal(t, float64(job.ID), payload["job_id"])
assert.Equal(t, "failed", payload["status"])
// Ensure there's an error message
errorMsg, ok := payload["error_message"].(string)
require.True(t, ok, "error_message should be a string")
assert.NotEmpty(t, errorMsg)
t.Logf("Error message from webhook: %s", errorMsg)
// Clean up
err = database.DB.Unscoped().Delete(job).Error
require.NoError(t, err)
err = database.DB.Unscoped().Where("job_id = ?", job.ID).Delete(&db.JobHistory{}).Error
require.NoError(t, err)
}
// TestWebhookDisabledForSuccessNotification tests that webhooks are not sent for
// successful jobs when notify_on_success is disabled
func TestWebhookDisabledForSuccessNotification(t *testing.T) {
// Skip in short mode
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
// Set up a temporary data directory for logs
tempDir := t.TempDir()
// Set DATA_DIR environment variable for the test
originalDataDir := os.Getenv("DATA_DIR")
t.Setenv("DATA_DIR", tempDir)
defer os.Setenv("DATA_DIR", originalDataDir)
// Create a test database
database := setupTestDB(t)
// Create a test user
user := &db.User{
Email: "webhook-disabled@example.com",
PasswordHash: "hashed_password",
IsAdmin: true,
}
err := database.CreateUser(user)
require.NoError(t, err)
// Set up a mock HTTP server to receive webhook notifications
var (
webhookCalled bool
webhookMutex sync.Mutex
)
mockServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
webhookMutex.Lock()
defer webhookMutex.Unlock()
// Log the fact that webhook was called (it shouldn't be)
body, _ := io.ReadAll(r.Body)
t.Logf("Unexpected webhook call received: %s", string(body))
webhookCalled = true
w.WriteHeader(http.StatusOK)
}))
defer mockServer.Close()
// Create local source and destination directories
sourceDir := t.TempDir()
destDir := t.TempDir()
// Create a test transfer config with local source and destination
config := &db.TransferConfig{
Name: "Webhook Disabled Config",
SourceType: "local",
SourcePath: sourceDir,
DestinationType: "local",
DestinationPath: destDir,
CreatedBy: user.ID,
}
err = database.DB.Create(config).Error
require.NoError(t, err)
// Create a test job with webhook enabled but notify_on_success disabled
job := &db.Job{
Name: "Webhook Disabled Job",
ConfigID: config.ID,
Schedule: "*/5 * * * *", // not actually used in this test
Enabled: true,
WebhookEnabled: true,
WebhookURL: mockServer.URL,
NotifyOnSuccess: false, // This is the key setting we're testing
NotifyOnFailure: true,
CreatedBy: user.ID,
}
err = database.DB.Create(job).Error
require.NoError(t, err)
// Update the job to ensure the notification settings are correctly set
// This is necessary because the database has default values for these fields
err = database.DB.Model(job).Updates(map[string]interface{}{
"notify_on_success": false,
}).Error
require.NoError(t, err)
// Reload the job to make sure we have the correct values
var reloadedJob db.Job
err = database.DB.First(&reloadedJob, job.ID).Error
require.NoError(t, err)
job = &reloadedJob
t.Logf("Created job with ID %d, NotifyOnSuccess=%v", job.ID, job.NotifyOnSuccess)
// Create and initialize the scheduler
scheduler := New(database)
defer scheduler.Stop()
// Create rclone config directory and file
configDir := filepath.Join(tempDir, "configs")
err = os.MkdirAll(configDir, 0755)
require.NoError(t, err)
// Create a minimal rclone config file
rcloneConfig := `
[source_1]
type = local
[dest_1]
type = local
`
configFile := filepath.Join(configDir, "config_1.conf")
err = os.WriteFile(configFile, []byte(rcloneConfig), 0644)
require.NoError(t, err)
t.Logf("Created rclone config file: %s", configFile)
// Put a test file in the source directory
testFile := filepath.Join(sourceDir, "test.txt")
testFileContent := []byte("This is a test file for disabled webhook testing.")
err = os.WriteFile(testFile, testFileContent, 0644)
require.NoError(t, err)
// Manually trigger job execution
t.Logf("Running job now...")
err = scheduler.RunJobNow(job.ID)
require.NoError(t, err)
// Wait for a bit to ensure job completes (10 seconds should be plenty)
time.Sleep(10 * time.Second)
// Check if webhook was called (it should not have been)
webhookMutex.Lock()
called := webhookCalled
webhookMutex.Unlock()
assert.False(t, called, "Webhook should not have been called for successful job with NotifyOnSuccess=false")
// Verify the job actually ran successfully by checking for the file
destFile := filepath.Join(destDir, "test.txt")
_, err = os.Stat(destFile)
assert.NoError(t, err, "The job should have completed and transferred the file")
// Verify job history has been created and shows completion
var histories []db.JobHistory
err = database.DB.Where("job_id = ?", job.ID).Find(&histories).Error
require.NoError(t, err)
if len(histories) > 0 {
t.Logf("Job history found: status=%s", histories[0].Status)
assert.Equal(t, "completed", histories[0].Status, "Job should have completed successfully")
}
// Clean up
err = database.DB.Unscoped().Delete(job).Error
require.NoError(t, err)
err = database.DB.Unscoped().Where("job_id = ?", job.ID).Delete(&db.JobHistory{}).Error
require.NoError(t, err)
}
+610
View File
@@ -0,0 +1,610 @@
package scheduler
import (
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"os"
"sync"
"testing"
"time"
"github.com/starfleetcptn/gomft/internal/db"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestWebhookNotification tests the webhook notification functionality
func TestWebhookNotification(t *testing.T) {
// Set up a temporary data directory for logs
tempDir := t.TempDir()
// Set DATA_DIR environment variable for the test
originalDataDir := os.Getenv("DATA_DIR")
t.Setenv("DATA_DIR", tempDir)
defer os.Setenv("DATA_DIR", originalDataDir)
// Create a test database
database := setupTestDB(t)
// Create a test user
user := &db.User{
Email: "webhook-test@example.com",
PasswordHash: "hashed_password",
IsAdmin: true,
}
err := database.CreateUser(user)
require.NoError(t, err)
// Create a test transfer config
config := &db.TransferConfig{
Name: "Webhook Test Config",
SourceType: "local",
SourcePath: "/source",
DestinationType: "local",
DestinationPath: "/dest",
CreatedBy: user.ID,
}
err = database.DB.Create(config).Error
require.NoError(t, err)
// Create a mock HTTP server to receive webhook notifications
var (
receivedPayload []byte
receivedHeaders http.Header
webhookCalled bool
webhookMutex sync.Mutex
waitCh chan struct{}
)
mockServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
webhookMutex.Lock()
defer webhookMutex.Unlock()
receivedHeaders = r.Header.Clone()
var err error
receivedPayload, err = io.ReadAll(r.Body)
if err != nil {
t.Logf("Error reading request body: %v", err)
w.WriteHeader(http.StatusInternalServerError)
return
}
// Debug output to help understand what's happening
t.Logf("Webhook called with payload: %s", string(receivedPayload))
webhookCalled = true
w.WriteHeader(http.StatusOK)
// Signal that webhook was called
if waitCh != nil {
close(waitCh)
}
}))
defer mockServer.Close()
// Create a test scheduler
scheduler := New(database)
defer scheduler.Stop()
// Test cases
tests := []struct {
name string
job *db.Job
history *db.JobHistory
webhookEnabled bool
webhookURL string
webhookSecret string
webhookHeaders map[string]string
notifyOnSuccess bool
notifyOnFailure bool
status string
expectNotification bool
}{
{
name: "Successful job with notification",
job: &db.Job{
Name: "Success Job",
ConfigID: config.ID,
WebhookEnabled: true,
WebhookURL: mockServer.URL,
NotifyOnSuccess: true,
NotifyOnFailure: true,
CreatedBy: user.ID,
},
history: &db.JobHistory{
Status: "completed",
StartTime: time.Now().Add(-5 * time.Minute),
EndTime: timePtr(time.Now()),
BytesTransferred: 1024,
FilesTransferred: 2,
},
webhookEnabled: true,
webhookURL: mockServer.URL,
notifyOnSuccess: true,
notifyOnFailure: true,
status: "completed",
expectNotification: true,
},
{
name: "Failed job with notification",
job: &db.Job{
Name: "Failed Job",
ConfigID: config.ID,
WebhookEnabled: true,
WebhookURL: mockServer.URL,
NotifyOnSuccess: true,
NotifyOnFailure: true,
CreatedBy: user.ID,
},
history: &db.JobHistory{
Status: "failed",
StartTime: time.Now().Add(-5 * time.Minute),
EndTime: timePtr(time.Now()),
ErrorMessage: "Test error message",
},
webhookEnabled: true,
webhookURL: mockServer.URL,
notifyOnSuccess: true,
notifyOnFailure: true,
status: "failed",
expectNotification: true,
},
{
name: "Successful job with notification disabled for success",
job: &db.Job{
Name: "Success Job No Notify",
ConfigID: config.ID,
WebhookEnabled: true,
WebhookURL: mockServer.URL,
NotifyOnSuccess: false,
NotifyOnFailure: true,
CreatedBy: user.ID,
},
history: &db.JobHistory{
Status: "completed",
StartTime: time.Now().Add(-5 * time.Minute),
EndTime: timePtr(time.Now()),
},
webhookEnabled: true,
webhookURL: mockServer.URL,
notifyOnSuccess: false,
notifyOnFailure: true,
status: "completed",
expectNotification: false,
},
{
name: "Failed job with notification disabled for failure",
job: &db.Job{
Name: "Failed Job No Notify",
ConfigID: config.ID,
WebhookEnabled: true,
WebhookURL: mockServer.URL,
NotifyOnSuccess: true,
NotifyOnFailure: false,
CreatedBy: user.ID,
},
history: &db.JobHistory{
Status: "failed",
StartTime: time.Now().Add(-5 * time.Minute),
EndTime: timePtr(time.Now()),
ErrorMessage: "Test error message",
},
webhookEnabled: true,
webhookURL: mockServer.URL,
notifyOnSuccess: true,
notifyOnFailure: false,
status: "failed",
expectNotification: false,
},
{
name: "Webhook disabled",
job: &db.Job{
Name: "Webhook Disabled",
ConfigID: config.ID,
WebhookEnabled: false,
WebhookURL: mockServer.URL,
NotifyOnSuccess: true,
NotifyOnFailure: true,
CreatedBy: user.ID,
},
history: &db.JobHistory{
Status: "completed",
StartTime: time.Now().Add(-5 * time.Minute),
EndTime: timePtr(time.Now()),
},
webhookEnabled: false,
webhookURL: mockServer.URL,
notifyOnSuccess: true,
notifyOnFailure: true,
status: "completed",
expectNotification: false,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
// Reset received data
webhookMutex.Lock()
receivedPayload = nil
receivedHeaders = nil
webhookCalled = false
waitCh = make(chan struct{})
webhookMutex.Unlock()
// Debug the test case configuration
t.Logf("Test configuration: name=%s, webhookEnabled=%v, notifyOnSuccess=%v, notifyOnFailure=%v, status=%s, expectNotification=%v",
tc.name, tc.webhookEnabled, tc.notifyOnSuccess, tc.notifyOnFailure, tc.status, tc.expectNotification)
// Create a new job instance for each test case
job := &db.Job{
Name: tc.job.Name,
ConfigID: tc.job.ConfigID,
WebhookEnabled: tc.webhookEnabled,
WebhookURL: tc.webhookURL,
NotifyOnSuccess: tc.notifyOnSuccess,
NotifyOnFailure: tc.notifyOnFailure,
CreatedBy: tc.job.CreatedBy,
}
t.Logf("Job before DB create: WebhookEnabled=%v, NotifyOnSuccess=%v, NotifyOnFailure=%v",
job.WebhookEnabled, job.NotifyOnSuccess, job.NotifyOnFailure)
err := database.DB.Create(job).Error
require.NoError(t, err)
// Update the job to ensure the notification settings are correctly set
// This is necessary because the database has default values for these fields
err = database.DB.Model(job).Updates(map[string]interface{}{
"notify_on_success": tc.notifyOnSuccess,
"notify_on_failure": tc.notifyOnFailure,
}).Error
require.NoError(t, err)
// Reload the job to make sure we have the correct values
var reloadedJob db.Job
err = database.DB.First(&reloadedJob, job.ID).Error
require.NoError(t, err)
job = &reloadedJob
t.Logf("Job after DB create: WebhookEnabled=%v, NotifyOnSuccess=%v, NotifyOnFailure=%v",
job.WebhookEnabled, job.NotifyOnSuccess, job.NotifyOnFailure)
// Create and save job history
history := tc.history
history.JobID = job.ID
err = database.DB.Create(history).Error
require.NoError(t, err)
// Debug info
t.Logf("Test case: %s", tc.name)
t.Logf("Job settings: WebhookEnabled=%v, NotifyOnSuccess=%v, NotifyOnFailure=%v",
job.WebhookEnabled, job.NotifyOnSuccess, job.NotifyOnFailure)
t.Logf("History status: %s", history.Status)
// Send webhook notification
scheduler.sendWebhookNotification(job, history, config)
// Wait for webhook call to complete if expected
if tc.expectNotification {
// Wait with timeout for webhook to be called
select {
case <-waitCh:
// Webhook was called
case <-time.After(2 * time.Second):
t.Fatalf("Timed out waiting for webhook to be called")
}
} else {
// Give it a small window to ensure it doesn't call when not expected
time.Sleep(500 * time.Millisecond)
}
// Check if notification was sent as expected
webhookMutex.Lock()
called := webhookCalled
payload := receivedPayload
headers := receivedHeaders
webhookMutex.Unlock()
if tc.expectNotification {
assert.True(t, called, "Expected webhook notification to be sent")
require.NotNil(t, payload, "Expected webhook payload to be non-nil")
// Verify the payload
var payloadMap map[string]interface{}
err := json.Unmarshal(payload, &payloadMap)
require.NoError(t, err, "Failed to unmarshal webhook payload")
// Check common fields
assert.Equal(t, "job_execution", payloadMap["event_type"])
assert.Equal(t, float64(job.ID), payloadMap["job_id"])
assert.Equal(t, job.Name, payloadMap["job_name"])
assert.Equal(t, float64(config.ID), payloadMap["config_id"])
assert.Equal(t, config.Name, payloadMap["config_name"])
assert.Equal(t, history.Status, payloadMap["status"])
// Check headers
assert.Equal(t, "application/json", headers.Get("Content-Type"))
assert.Equal(t, "GoMFT-Webhook/1.0", headers.Get("User-Agent"))
// Additional checks for specific status
if history.Status == "failed" {
assert.Equal(t, history.ErrorMessage, payloadMap["error_message"])
}
} else {
assert.False(t, called, "Expected no webhook notification to be sent")
}
// Clean up
err = database.DB.Unscoped().Delete(history).Error
require.NoError(t, err)
err = database.DB.Unscoped().Delete(job).Error
require.NoError(t, err)
})
}
}
// TestWebhookAuthentication tests the webhook authentication functionality
func TestWebhookAuthentication(t *testing.T) {
// Set up a temporary data directory for logs
tempDir := t.TempDir()
// Set DATA_DIR environment variable for the test
originalDataDir := os.Getenv("DATA_DIR")
t.Setenv("DATA_DIR", tempDir)
defer os.Setenv("DATA_DIR", originalDataDir)
// Create a test database
database := setupTestDB(t)
// Create a test user
user := &db.User{
Email: "webhook-auth-test@example.com",
PasswordHash: "hashed_password",
IsAdmin: true,
}
err := database.CreateUser(user)
require.NoError(t, err)
// Create a test transfer config
config := &db.TransferConfig{
Name: "Webhook Auth Test Config",
SourceType: "local",
SourcePath: "/source",
DestinationType: "local",
DestinationPath: "/dest",
CreatedBy: user.ID,
}
err = database.DB.Create(config).Error
require.NoError(t, err)
// Create a mock HTTP server to receive webhook notifications
var (
receivedPayload []byte
receivedHeaders http.Header
waitCh = make(chan struct{})
)
mockServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
receivedHeaders = r.Header.Clone()
var err error
receivedPayload, err = io.ReadAll(r.Body)
if err != nil {
t.Logf("Error reading request body: %v", err)
w.WriteHeader(http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
close(waitCh)
}))
defer mockServer.Close()
// Create a test scheduler
scheduler := New(database)
defer scheduler.Stop()
// Set up job with webhook secret
secret := "test-webhook-secret"
job := &db.Job{
Name: "Auth Test Job",
ConfigID: config.ID,
WebhookEnabled: true,
WebhookURL: mockServer.URL,
WebhookSecret: secret,
NotifyOnSuccess: true,
NotifyOnFailure: true,
CreatedBy: user.ID,
}
err = database.DB.Create(job).Error
require.NoError(t, err)
// Create job history
history := &db.JobHistory{
JobID: job.ID,
Status: "completed",
StartTime: time.Now().Add(-5 * time.Minute),
EndTime: timePtr(time.Now()),
BytesTransferred: 1024,
FilesTransferred: 2,
}
err = database.DB.Create(history).Error
require.NoError(t, err)
// Send webhook notification
scheduler.sendWebhookNotification(job, history, config)
// Wait for webhook to be called
select {
case <-waitCh:
// Webhook was called
case <-time.After(2 * time.Second):
t.Fatalf("Timed out waiting for webhook to be called")
}
// Verify the signature
require.NotNil(t, receivedPayload, "Expected webhook notification to be sent")
// Check that the X-Hub-Signature-256 header exists
signature := receivedHeaders.Get("X-Hub-Signature-256")
require.NotEmpty(t, signature, "Expected X-Hub-Signature-256 header to be set")
// Verify that the signature matches the expected HMAC-SHA256
h := hmac.New(sha256.New, []byte(secret))
h.Write(receivedPayload)
expectedSignature := hex.EncodeToString(h.Sum(nil))
// Print both signatures for debugging if they don't match
if expectedSignature != signature {
t.Logf("Expected signature: %s", expectedSignature)
t.Logf("Actual signature: %s", signature)
t.Logf("Secret used: %s", secret)
t.Logf("Payload length: %d", len(receivedPayload))
}
assert.Equal(t, expectedSignature, signature, "Signature does not match expected value")
// Clean up
err = database.DB.Unscoped().Delete(history).Error
require.NoError(t, err)
err = database.DB.Unscoped().Delete(job).Error
require.NoError(t, err)
}
// TestWebhookCustomHeaders tests the custom headers functionality for webhooks
func TestWebhookCustomHeaders(t *testing.T) {
// Set up a temporary data directory for logs
tempDir := t.TempDir()
// Set DATA_DIR environment variable for the test
originalDataDir := os.Getenv("DATA_DIR")
t.Setenv("DATA_DIR", tempDir)
defer os.Setenv("DATA_DIR", originalDataDir)
// Create a test database
database := setupTestDB(t)
// Create a test user
user := &db.User{
Email: "webhook-headers-test@example.com",
PasswordHash: "hashed_password",
IsAdmin: true,
}
err := database.CreateUser(user)
require.NoError(t, err)
// Create a test transfer config
config := &db.TransferConfig{
Name: "Webhook Headers Test Config",
SourceType: "local",
SourcePath: "/source",
DestinationType: "local",
DestinationPath: "/dest",
CreatedBy: user.ID,
}
err = database.DB.Create(config).Error
require.NoError(t, err)
// Create a mock HTTP server to receive webhook notifications
var (
receivedPayload []byte
receivedHeaders http.Header
waitCh = make(chan struct{})
)
mockServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
receivedHeaders = r.Header.Clone()
var err error
receivedPayload, err = io.ReadAll(r.Body)
if err != nil {
t.Logf("Error reading request body: %v", err)
w.WriteHeader(http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
close(waitCh)
}))
defer mockServer.Close()
// Create a test scheduler
scheduler := New(database)
defer scheduler.Stop()
// Define custom headers
customHeaders := map[string]string{
"X-API-Key": "test-api-key",
"X-Client-ID": "test-client-id",
"X-Source": "gomft-test",
}
customHeadersJSON, err := json.Marshal(customHeaders)
require.NoError(t, err)
// Set up job with custom headers
job := &db.Job{
Name: "Custom Headers Test Job",
ConfigID: config.ID,
WebhookEnabled: true,
WebhookURL: mockServer.URL,
WebhookHeaders: string(customHeadersJSON),
NotifyOnSuccess: true,
NotifyOnFailure: true,
CreatedBy: user.ID,
}
err = database.DB.Create(job).Error
require.NoError(t, err)
// Create job history
history := &db.JobHistory{
JobID: job.ID,
Status: "completed",
StartTime: time.Now().Add(-5 * time.Minute),
EndTime: timePtr(time.Now()),
BytesTransferred: 1024,
FilesTransferred: 2,
}
err = database.DB.Create(history).Error
require.NoError(t, err)
// Send webhook notification
scheduler.sendWebhookNotification(job, history, config)
// Wait for webhook to be called
select {
case <-waitCh:
// Webhook was called
case <-time.After(2 * time.Second):
t.Fatalf("Timed out waiting for webhook to be called")
}
// Verify the headers
require.NotNil(t, receivedPayload, "Expected webhook notification to be sent")
// Check that all custom headers are present
for key, value := range customHeaders {
actualValue := receivedHeaders.Get(key)
if actualValue != value {
t.Logf("Custom header mismatch for %s: expected=%s, got=%s", key, value, actualValue)
}
assert.Equal(t, value, actualValue, "Expected custom header %s to be set", key)
}
// Also check standard headers
assert.Equal(t, "application/json", receivedHeaders.Get("Content-Type"))
assert.Equal(t, "GoMFT-Webhook/1.0", receivedHeaders.Get("User-Agent"))
// Clean up
err = database.DB.Unscoped().Delete(history).Error
require.NoError(t, err)
err = database.DB.Unscoped().Delete(job).Error
require.NoError(t, err)
}
// Helper function to create a pointer to a time.Time value
func timePtr(t time.Time) *time.Time {
return &t
}
+243
View File
@@ -0,0 +1,243 @@
package handlers
import (
"net/http"
"net/http/httptest"
"net/url"
"strconv"
"strings"
"testing"
"time"
"github.com/starfleetcptn/gomft/internal/db"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestWebhookConfiguration tests the webhook configuration during job creation and editing
func TestWebhookConfiguration(t *testing.T) {
// Set up test environment
handlers, router, database, user, config := setupJobsTest(t)
// Add job create route
router.POST("/jobs/create", handlers.HandleCreateJob)
// Create job form data with webhook enabled
formData := url.Values{
"name": {"Webhook Test Job"},
"config_ids[]": {strconv.Itoa(int(config.ID))},
"schedule": {"*/15 * * * *"},
"enabled": {"true"},
"webhook_enabled": {"true"},
"webhook_url": {"https://example.com/webhook"},
"webhook_secret": {"test-secret"},
"webhook_headers": {`{"X-Test-Header": "test-value"}`},
"notify_on_success": {"true"},
"notify_on_failure": {"true"},
}
// Submit form
req, _ := http.NewRequest("POST", "/jobs/create", strings.NewReader(formData.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
resp := httptest.NewRecorder()
router.ServeHTTP(resp, req)
// Should redirect on success
assert.Equal(t, http.StatusFound, resp.Code)
// Check if job was created with webhook settings
var jobs []db.Job
err := database.DB.Where("created_by = ?", user.ID).Find(&jobs).Error
require.NoError(t, err)
require.GreaterOrEqual(t, len(jobs), 1)
// Get the most recently created job
var job db.Job
err = database.DB.Where("created_by = ?", user.ID).Order("created_at DESC").First(&job).Error
require.NoError(t, err)
// Verify webhook settings were saved correctly
assert.True(t, job.WebhookEnabled)
assert.Equal(t, "https://example.com/webhook", job.WebhookURL)
assert.Equal(t, "test-secret", job.WebhookSecret)
assert.Equal(t, `{"X-Test-Header": "test-value"}`, job.WebhookHeaders)
assert.True(t, job.NotifyOnSuccess)
assert.True(t, job.NotifyOnFailure)
}
// TestWebhookEditConfiguration tests editing webhook configuration
func TestWebhookEditConfiguration(t *testing.T) {
// Set up test environment
handlers, router, database, user, config := setupJobsTest(t)
// Create a job first
job := &db.Job{
Name: "Initial Job",
ConfigID: config.ID,
Schedule: "*/30 * * * *",
Enabled: true,
WebhookEnabled: false, // Initially disabled
CreatedBy: user.ID,
CreatedAt: time.Now(),
UpdatedAt: time.Now(),
}
err := database.DB.Create(job).Error
require.NoError(t, err)
// Add job update route
router.PUT("/jobs/:id", handlers.HandleUpdateJob)
// Create edit form data to enable webhook
formData := url.Values{
"name": {"Updated Job"},
"config_ids[]": {strconv.Itoa(int(config.ID))},
"schedule": {"*/30 * * * *"},
"enabled": {"true"},
"webhook_enabled": {"true"}, // Enabling webhook
"webhook_url": {"https://example.com/webhook"}, // Adding URL
"webhook_secret": {"new-secret"}, // Adding secret
"webhook_headers": {`{"X-Api-Key": "12345"}`}, // Adding headers
"notify_on_success": {"true"}, // Configure notifications
"notify_on_failure": {"false"}, // Only notify on success
}
// Submit edit form
req, _ := http.NewRequest("PUT", "/jobs/"+strconv.Itoa(int(job.ID)), strings.NewReader(formData.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
resp := httptest.NewRecorder()
router.ServeHTTP(resp, req)
// Should redirect on success
assert.Equal(t, http.StatusFound, resp.Code)
// Get the updated job
var updatedJob db.Job
err = database.DB.First(&updatedJob, job.ID).Error
require.NoError(t, err)
// Verify webhook settings were updated correctly
assert.True(t, updatedJob.WebhookEnabled)
assert.Equal(t, "https://example.com/webhook", updatedJob.WebhookURL)
assert.Equal(t, "new-secret", updatedJob.WebhookSecret)
assert.Equal(t, `{"X-Api-Key": "12345"}`, updatedJob.WebhookHeaders)
assert.True(t, updatedJob.NotifyOnSuccess)
assert.False(t, updatedJob.NotifyOnFailure)
}
// TestDisablingWebhook tests disabling a previously enabled webhook
func TestDisablingWebhook(t *testing.T) {
// Set up test environment
handlers, router, database, user, config := setupJobsTest(t)
// Create a job with webhook enabled
job := &db.Job{
Name: "Webhook Enabled Job",
ConfigID: config.ID,
Schedule: "*/30 * * * *",
Enabled: true,
WebhookEnabled: true,
WebhookURL: "https://example.com/webhook",
WebhookSecret: "secret",
WebhookHeaders: `{"X-Test": "test"}`,
NotifyOnSuccess: true,
NotifyOnFailure: true,
CreatedBy: user.ID,
CreatedAt: time.Now(),
UpdatedAt: time.Now(),
}
err := database.DB.Create(job).Error
require.NoError(t, err)
// Add job update route
router.PUT("/jobs/:id", handlers.HandleUpdateJob)
// Create edit form data to disable webhook
formData := url.Values{
"name": {"Webhook Disabled Job"},
"config_ids[]": {strconv.Itoa(int(config.ID))},
"schedule": {"*/30 * * * *"},
"enabled": {"true"},
"webhook_enabled": {"false"}, // Explicitly set to false
"webhook_url": {"https://example.com/webhook"}, // URL remains the same
"webhook_secret": {"secret"}, // Secret remains the same
"webhook_headers": {`{"X-Test": "test"}`}, // Headers remain the same
}
// Submit edit form
req, _ := http.NewRequest("PUT", "/jobs/"+strconv.Itoa(int(job.ID)), strings.NewReader(formData.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
resp := httptest.NewRecorder()
router.ServeHTTP(resp, req)
// Should redirect on success
assert.Equal(t, http.StatusFound, resp.Code)
// Get the updated job
var updatedJob db.Job
err = database.DB.First(&updatedJob, job.ID).Error
require.NoError(t, err)
// Verify webhook was disabled
assert.False(t, updatedJob.WebhookEnabled)
// Other fields should remain unchanged
assert.Equal(t, "https://example.com/webhook", updatedJob.WebhookURL)
assert.Equal(t, "secret", updatedJob.WebhookSecret)
assert.Equal(t, `{"X-Test": "test"}`, updatedJob.WebhookHeaders)
}
// TestWebhookValidation tests validation of webhook URL
func TestWebhookValidation(t *testing.T) {
// Set up test environment
handlers, router, _, _, config := setupJobsTest(t)
// Add job create route
router.POST("/jobs/create", handlers.HandleCreateJob)
// Create job form data with invalid webhook URL
formData := url.Values{
"name": {"Invalid Webhook Job"},
"config_ids[]": {strconv.Itoa(int(config.ID))},
"schedule": {"*/15 * * * *"},
"enabled": {"true"},
"webhook_enabled": {"true"},
"webhook_url": {"invalid-url"}, // Invalid URL
"webhook_secret": {"test-secret"},
"notify_on_success": {"true"},
"notify_on_failure": {"true"},
}
// Submit form
req, _ := http.NewRequest("POST", "/jobs/create", strings.NewReader(formData.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
resp := httptest.NewRecorder()
router.ServeHTTP(resp, req)
// Should not create job with invalid webhook URL
assert.NotEqual(t, http.StatusFound, resp.Code)
assert.Contains(t, resp.Body.String(), "valid URL")
// Test invalid headers JSON
formData = url.Values{
"name": {"Invalid Headers Job"},
"config_ids[]": {strconv.Itoa(int(config.ID))},
"schedule": {"*/15 * * * *"},
"enabled": {"true"},
"webhook_enabled": {"true"},
"webhook_url": {"https://example.com/webhook"},
"webhook_secret": {"test-secret"},
"webhook_headers": {`{"invalid json`}, // Invalid JSON
"notify_on_success": {"true"},
"notify_on_failure": {"true"},
}
// Submit form
req, _ = http.NewRequest("POST", "/jobs/create", strings.NewReader(formData.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
resp = httptest.NewRecorder()
router.ServeHTTP(resp, req)
// Should not create job with invalid headers JSON
assert.NotEqual(t, http.StatusFound, resp.Code)
assert.Contains(t, resp.Body.String(), "valid JSON")
}
Binary file not shown.

Before

Width:  |  Height:  |  Size: 659 KiB

After

Width:  |  Height:  |  Size: 825 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 352 KiB

After

Width:  |  Height:  |  Size: 629 KiB