Github-Mirror/GoMFT
1
0
Fork 0
mirror of https://github.com/StarFleetCPTN/GoMFT.git synced 2026-09-08 15:41:20 +02:00
Code Issues Packages Projects Releases Wiki Activity

Add audit logging to storage providers

Browse Source
This commit is contained in:
StarFleetCPTN
2025-04-19 07:58:14 -07:00
parent 98e20ea843
commit 495b844c00
11 changed files with 706 additions and 988 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+14 -538
View File
@@ -58,12 +58,6 @@ GoMFT is a web-based managed file transfer application built with Go, leveraging
- Wasabi - Wasabi
- Local filesystem - Local filesystem
- And more via rclone - And more via rclone
- **Webhook Notifications**: Receive real-time notifications of job events:
- Configurable webhook URLs
- HMAC-SHA256 authentication with secrets
- Custom HTTP headers
- Selectable events (job success, job failure)
- Detailed JSON payload with job information
- **Multiple Notification Services**: Get job status updates through various notification channels: - **Multiple Notification Services**: Get job status updates through various notification channels:
- Email notifications with configurable SMTP settings - Email notifications with configurable SMTP settings
- Webhooks with authentication for custom integrations - Webhooks with authentication for custom integrations
@@ -135,9 +129,15 @@ cd gomft
2. Install dependencies: 2. Install dependencies:
```bash ```bash
go mod download go mod download
go install github.com/a-h/templ/cmd/templ@latest
``` ```
3. Build the application: 3. Generate template code:
```bash
templ generate
```
4. Build the application:
```bash ```bash
go build -o gomft go build -o gomft
``` ```
@@ -230,6 +230,7 @@ services:
- GOOGLE_CLIENT_ID=your_google_client_id - GOOGLE_CLIENT_ID=your_google_client_id
- GOOGLE_CLIENT_SECRET=your_google_client_secret - GOOGLE_CLIENT_SECRET=your_google_client_secret
- TOTP_ENCRYPTION_KEY=your_32_byte_encryption_key_here - TOTP_ENCRYPTION_KEY=your_32_byte_encryption_key_here
- GOMFT_ENCRYPTION_KEY=your_32_byte_encryption_key_here
- EMAIL_ENABLED=true - EMAIL_ENABLED=true
- EMAIL_HOST=smtp.example.com - EMAIL_HOST=smtp.example.com
- EMAIL_PORT=587 - EMAIL_PORT=587
@@ -255,539 +256,14 @@ docker-compose up -d
For more information and available tags, visit the [GoMFT Docker Hub page](https://hub.docker.com/r/starfleetcptn/gomft). For more information and available tags, visit the [GoMFT Docker Hub page](https://hub.docker.com/r/starfleetcptn/gomft).
--- Full documentation is available at [https://starfleetcptn.github.io/GoMFT/](https://starfleetcptn.github.io/GoMFT/).
## Configuration ## License
GoMFT uses an environment file located at `.env` in the root directory of the application. On first run, a default configuration will be created: [MIT License](LICENSE) - see the full license terms
```ini The GoMFT logo is licensed under the Creative Commons Attribution 4.0 International Public License.
# Basic configuration
SERVER_ADDRESS=:8080
DATA_DIR=/app/data
BACKUP_DIR=/app/backups
JWT_SECRET=change_this_to_a_secure_random_string
BASE_URL=http://localhost:8080
# Google OAuth configuration (optional, for built-in authentication) The gopher design is from https://github.com/egonelbre/gophers.
GOOGLE_CLIENT_ID=your_google_client_id
GOOGLE_CLIENT_SECRET=your_google_client_secret
# Email configuration The original Go gopher was designed by Renee French (http://reneefrench.blogspot.com/).
EMAIL_ENABLED=true
EMAIL_HOST=smtp.example.com
EMAIL_PORT=587
EMAIL_FROM_EMAIL=gomft@example.com
EMAIL_FROM_NAME=GoMFT
EMAIL_REPLY_TO=
EMAIL_ENABLE_TLS=true
EMAIL_REQUIRE_AUTH=true
EMAIL_USERNAME=smtp_username
EMAIL_PASSWORD=smtp_password
# Two-Factor Authentication configuration
TOTP_ENCRYPTION_KEY=your_32_byte_encryption_key_here
# UserID and GroupID
PUID=1000
PGID=1000
```
### Configuration Options
- `SERVER_ADDRESS`: The address and port to run the server on
- `DATA_DIR`: Directory for storing application data (database and configs)
- `BACKUP_DIR`: Directory for storing database backups
- `JWT_SECRET`: Secret key for JWT token generation
- `BASE_URL`: Base URL for generating links in emails (e.g., password reset links)
- Google OAuth configuration for built-in authentication:
- `GOOGLE_CLIENT_ID`: Your Google OAuth client ID
- `GOOGLE_CLIENT_SECRET`: Your Google OAuth client secret
- Email configuration settings for system notifications and password resets:
- `EMAIL_ENABLED`: Set to `true` to enable email functionality
- `EMAIL_HOST`: SMTP server hostname
- `EMAIL_PORT`: SMTP server port (usually 587 for TLS, 465 for SSL, or 25 for non-secure)
- `EMAIL_USERNAME`: Username for SMTP authentication
- `EMAIL_PASSWORD`: Password for SMTP authentication
- `EMAIL_FROM_EMAIL`: Email address used as sender
- `EMAIL_FROM_NAME`: Name displayed as the sender
- `EMAIL_REPLY_TO`: Optional reply-to email address
- `EMAIL_ENABLE_TLS`: Set to `true` to use TLS for secure email transmission
- `EMAIL_REQUIRE_AUTH`: Set to `true` to require authentication for SMTP connections, or `false` for servers that don't need authentication
- Two-Factor Authentication (2FA) configuration:
- `TOTP_ENCRYPTION_KEY`: Secret key used to encrypt/decrypt TOTP secrets (for 2FA)
- Should be exactly 32 bytes (characters) for optimal security
- If not set, a default development key will be used (not secure for production)
- If shorter than 32 bytes, it will be automatically padded (less secure)
- If longer than 32 bytes, it will be truncated to 32 bytes
- Example: `TOTP_ENCRYPTION_KEY=abcdefghijklmnopqrstuvwxyz123456`
- SSL/TLS Verification Control:
- `SKIP_SSL_VERIFY`: Set to `true` to disable SSL/TLS certificate verification for outgoing connections (e.g., webhooks, email). Use with caution, as this can expose connections to man-in-the-middle attacks. Defaults to `false` (verification enabled).
- Example: `SKIP_SSL_VERIFY=true`
### Logging Configuration
GoMFT provides configurable logging with rotation support through the following environment variables:
- `LOGS_DIR`: Directory where log files are stored (default: `./data/logs`)
- `LOG_MAX_SIZE`: Maximum size in megabytes for each log file before rotation (default: `10`)
- `LOG_MAX_BACKUPS`: Number of old log files to retain (default: `5`)
- `LOG_MAX_AGE`: Maximum number of days to retain old log files (default: `30`)
- `LOG_COMPRESS`: Whether to compress rotated log files (default: `true`)
- `LOG_LEVEL`: Controls verbosity level of logging (values: `error`, `info`, `debug`, default: `info`)
- `error`: Only show errors and critical issues
- `info`: Show errors and general operational information (default)
- `debug`: Show all messages including detailed debugging information
Log files contain detailed information about file transfers, job execution, and system operations, which can be useful for troubleshooting and auditing.
---
## Usage
1. Start the server:
```bash
./gomft
```
2. Access the web interface at `http://localhost:8080`
3. Log in with the default admin account:
- Email: `admin@example.com`
- Password: `admin`
- **Important**: Change this password immediately after first login
4. Create transfer configurations:
- Navigate to "Transfer Configs" section
- Configure source and destination locations with connection details
- Set file patterns and archive options as needed
- Configure performance settings:
- Set "Concurrent Transfers" slider to optimize throughput
- Use higher values (8-16) for many small files or fast networks
- Use lower values (1-4) for large files or limited bandwidth
- Consider source/destination system capabilities when setting
5. Create jobs using your configurations:
- Navigate to "Jobs" section
- Select an existing transfer config
- Use the visual schedule builder to set your timing preferences:
- Choose from common presets (hourly, daily, weekly, monthly)
- Customize with specific days, times, or intervals
- See a plain-language description of your schedule
- View upcoming run times on the interactive calendar
- Switch to advanced mode for direct cron expression input if needed
- Enable/disable jobs with a single click
6. Monitor transfers:
- View active and completed transfers on the Dashboard
- Check detailed transfer history with performance metrics
- View job run details including any error messages
7. Configure webhook notifications:
- Enable webhooks in job settings to receive notifications
- Provide a valid webhook URL where notifications will be sent
- Optionally set a webhook secret for HMAC-SHA256 signature verification
- Configure custom HTTP headers in JSON format if needed
- Choose notification triggers (job success, job failure, or both)
- Test your webhook integration with manual job runs
8. **Webhook Notifications**:
- **Webhook Integration**: Send notifications to external systems when jobs complete
- **Secure Authentication**: HMAC-SHA256 signature for webhook verification
- **Custom Headers**: Add custom HTTP headers to webhook requests
- **Flexible Configuration**: Configure different webhooks for different jobs
- **Event Selection**: Choose to send notifications on success, failure, or both
- **Detailed Payload**: Rich JSON payload with complete job execution details
9. **Multiple Notification Services**:
- **Pushbullet Integration**: Send notifications to your devices through Pushbullet
- Device targeting support for specific device delivery
- Customizable title and message templates
- API key-based authentication
- **Ntfy Integration**: Use public ntfy.sh or self-hosted ntfy server
- Topic-based routing of notifications
- Priority levels for different job events
- Optional username/password authentication for private servers
- Customizable title and message templates
- **Gotify Integration**: Send notifications to self-hosted Gotify servers
- Application token-based authentication
- Priority levels (1-10) for different notification importance
- Customizable title and message templates
- **Pushover Integration**: Professional notification delivery service
- Application and user key authentication
- Device targeting for selective delivery
- Sound selection for different notification types
- Priority levels from lowest to emergency
- Customizable title and message templates
- **Common Features**:
- Variable substitution in notification templates
- Job data access in templates (status, files, bytes, times)
- Event-based filtering (job start, completion, errors)
- Success/failure tracking for diagnostic purposes
10. Manage file metadata:
- Navigate to the "Files" section to view all processed files
- Use filters to quickly find files by status, job ID, or filename
- Click on any file to view detailed metadata including timestamps, size, and hash
- Use the advanced search page for complex queries with multiple criteria
- Delete file metadata records when no longer needed
- View files associated with specific jobs by navigating from the job details
11. Utilize admin tools (administrators only):
- Access the "Admin Tools" section from the navigation menu
- View system statistics and server information
- Create and manage database backups
- Browse and download system log files with the integrated log viewer
- Perform database maintenance and optimization tasks
- View webhook documentation and integration details
### User Management
GoMFT uses a role-based access control system with flexible authentication options:
- **Administrators**: Can create and manage users, access all features
- **Regular Users**: Can manage transfers and view history
#### Authentication Options
1. **Built-in Authentication**:
- Email/password login with secure password hashing
- JWT-based session management
- Password history tracking
- Account lockout protection
- Self-service password reset
2. **External Authentication Providers**:
- **Authentik Integration**:
- Enterprise-grade SSO capabilities
- Automatic user provisioning
- Role synchronization
- Group mapping support
- Secure token exchange
- **OpenID Connect (OIDC)**:
- Standard-compliant identity provider support
- Automatic user creation and updates
- Role mapping from OIDC claims
- Multiple provider support
- Secure token validation
- **OAuth2 Providers**:
- Google authentication
- GitHub integration
- Other OAuth2-compliant providers
- Custom provider configuration
- Automatic profile synchronization
3. **Security Features**:
- Secure password hashing with bcrypt
- JWT-based authentication with tokens
- Password history tracking prevents reuse
- Account lockout after failed attempts
- Two-factor authentication support
- Session management and timeout
- Secure token storage and handling
4. **User Profile Management**:
- Theme preferences (light/dark mode)
- Profile information updates
- Password change functionality
- Two-factor authentication setup
- External account linking
### Two-Factor Authentication (2FA) Implementation
#### Overview
This implementation adds TOTP-based (Time-based One-Time Password) two-factor authentication support to the application, compatible with standard authenticator apps like Google Authenticator, Authy, and others.
#### Features
- TOTP-based authentication (RFC 6238 compliant)
- QR code setup for easy enrollment
- Backup codes for account recovery
- Rate-limited verification attempts
- Secure secret storage
#### Database Changes
The following fields have been added to the `users` table:
- `two_factor_secret`: Stores the TOTP secret key
- `two_factor_enabled`: Boolean flag indicating if 2FA is enabled
- `backup_codes`: Stores recovery backup codes
#### Setup Process
1. Navigate to `/profile/2fa/setup`
2. Scan the displayed QR code with your authenticator app
3. Enter the verification code to confirm setup
4. Save your backup codes in a secure location
#### Login Flow
1. Enter email and password as usual
2. If 2FA is enabled:
- Enter the 6-digit code from your authenticator app
- Alternatively, use a backup code if you can't access your authenticator
#### Security Considerations
- The TOTP secrets are encrypted using AES-256-GCM
- You must set the `TOTP_ENCRYPTION_KEY` environment variable in production
- This key should be 32 bytes (characters) long and kept confidential
- Changing this key after users have set up 2FA will invalidate their existing 2FA configurations
- For high-security deployments, store this key in a secure vault and inject it at runtime
### Transfer Configuration Options
1. **Source/Destination Types**:
- Google Drive
- Google Photos
- Local filesystem
- Amazon S3
- MinIO (S3-compatible storage)
- NextCloud
- Backblaze B2
- Wasabi
- Hetzner Storage Box
- SFTP
- FTP
- SMB/CIFS shares
- And many more via rclone
2. **Connection Options**:
- Host/server addresses
- Authentication (username/password or key files)
- OAuth2 authentication for Google services
- Port configurations
- Cloud credentials (access keys, secret keys)
- Bucket and region settings
- Custom endpoints
- Custom rclone flags
3. **Google Photos Specific Options**:
- Read-only mode for safer operations
- Start year filter for historical photos
- Include/exclude archived media
- Album path configuration
- Built-in or custom OAuth authentication
4. **Google Drive Specific Options**:
- Folder ID for specific directory access
- Team/Shared Drive ID support
- Built-in or custom OAuth authentication
- Path-based navigation
5. **File Options**:
- File patterns for filtering (e.g., `*.txt`, `data_*.csv`)
- Output patterns for dynamic naming
- Archive options for transferred files
- Skip already processed files to avoid duplicates
- Concurrent file transfers (configurable per job)
6. **Performance Options**:
- **Multi-threaded File Transfers**: Process multiple files simultaneously for higher throughput
- Configurable concurrency level (1-32 concurrent transfers)
- Per-job concurrency settings to optimize for different storage types
- Automatic transfer queue management to prevent overloading systems
- Adaptive processing based on source/destination capabilities
7. **Schedule Options**:
- **Visual Schedule Builder**: Intuitive interface for setting schedule preferences
- **Natural Language Description**: Plain-language description of schedule
- **Interactive Calendar**: Visual representation of upcoming runs
- **Common Presets**: Hourly, daily, weekly, monthly schedules
- **Advanced Mode**: Manual cron expression input for complex schedules
- **Schedule Validation**: Preview and confirm schedule
- **Enable/Disable**: One-click enable/disable
- **Time Zone Support**: Accurate scheduling based on user's time zone
8. **Notification Options**:
- **Email Notifications**: Receive job status updates via email
- **Webhook Notifications**: Integration with external systems
- **Pushbullet**: Push notifications to your devices
- **Ntfy**: Simple push notifications via ntfy.sh
- **Gotify**: Self-hosted notification server integration
- **Pushover**: Professional notification service
- Configure event triggers (start, complete, error)
- Customize notification message templates
- Selective notification based on job status
### Email Notifications
GoMFT supports email notifications for various features:
- **Password Reset**: Users can request password reset links sent to their registered email
- **Styled Emails**: Professional HTML emails that match the application's design theme
- **Secure Tokens**: One-time use secure tokens with 15-minute expiration for enhanced security
- **Flexible Configuration**: Easily configure your SMTP server settings
- **Authentication Options**: Support for both authenticated and unauthenticated SMTP servers
- **TLS Support**: Secure communication with your SMTP server
- **Development Mode**: When emails are disabled, reset links are logged to the console
To configure email functionality:
1. Edit the `.env` file and provide your SMTP server details
2. Set `EMAIL_ENABLED=true` in the email configuration section
3. Ensure the `BASE_URL` setting is configured correctly for your deployment
### Webhook Integration
GoMFT can send webhook notifications to external systems when jobs complete. This allows integration with monitoring tools, chat applications, custom notification systems, or workflow automation platforms.
#### Webhook Payload Structure
Webhook notifications are sent as HTTP POST requests with a JSON payload containing detailed information about the job execution:
```json
{
"event_type": "job_execution",
"job_id": 123,
"job_name": "Daily Backup",
"config_id": 456,
"config_name": "S3 to Local Backup",
"status": "completed",
"start_time": "2023-07-14T15:30:00Z",
"end_time": "2023-07-14T15:35:42Z",
"duration_seconds": 342,
"bytes_transferred": 1048576,
"files_transferred": 25,
"history_id": 789,
"source": {
"type": "s3",
"path": "my-bucket/data"
},
"destination": {
"type": "local",
"path": "/backups/data"
}
}
```
For failed transfers, additional error information is included:
```json
{
"status": "failed",
"error_message": "Permission denied accessing destination path"
}
```
#### Webhook Authentication
When a webhook secret is configured, GoMFT signs the payload using HMAC-SHA256 and includes the signature in the `X-Hub-Signature-256` header. To verify the webhook:
1. Compute the HMAC-SHA256 of the raw request body using your shared secret
2. Compare it with the value in the `X-Hub-Signature-256` header
3. Process the webhook only if the signatures match
This ensures that webhook requests are authentic and haven't been tampered with.
### Admin Tools
GoMFT provides a comprehensive set of administrative tools for system management and monitoring:
#### Log Viewer
The Admin Tools panel includes an integrated log viewer with the following features:
- **Log File Browser**: View a list of all available log files in the system
- **Real-time Log Viewing**: View log file contents directly in the web interface
- **Refresh Function**: Update the log list and content with the latest information
- **User-friendly Interface**: Clean, readable presentation with custom scrolling
- **Dark Mode Support**: Consistent theming with the rest of the application
- **Navigation**: Easily switch between different log files
This log viewer allows administrators to:
- Monitor system activity and diagnose issues without requiring server access
- View application logs, scheduler logs, and transfer logs in one place
- Track down errors and warning messages in real-time
#### Database Management
The Admin Tools interface also includes database management capabilities:
- Create and manage database backups
- Restore from previous backups
- Download backups for safekeeping
- View system statistics
- Optimize the database with maintenance tools
---
## Development
### Project Structure
```
.
├── components/ # Templ components for UI
├── internal/
│ ├── api/ # REST API handlers
│ ├── auth/ # Authentication/authorization
│ ├── config/ # Configuration management
│ ├── db/ # Database models and operations
│ ├── email/ # Email service for notifications and password resets
│ ├── scheduler/ # Job scheduling and execution
│ └── web/ # Web interface handlers
├── static/ # Static assets
│ ├── css/
│ └── js/
└── main.go # Application entry point
```
### Technology Stack
- **Backend**: Go with Gin web framework
- **Frontend**: Templ for Go HTML components
- **UI Enhancement**: HTMX for dynamic interactions
- **Styling**: Tailwind CSS
- **Authentication**: JWT (JSON Web Tokens)
- **Database**: GORM with SQLite
- **File Transfer**: rclone
- **Deployment**: Docker containerization and traditional installation
### Building from Source
1. Install development dependencies:
```bash
go install github.com/cosmtrek/air@latest # Hot reload for development
go install github.com/a-h/templ/cmd/templ@latest # Templ template compiler
```
2. Generate template code:
```bash
templ generate
```
3. Run in development mode:
```bash
air
```
---
## Contributing
1. Fork the repository
2. Create a feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request
---
## Directory Structure
GoMFT uses the following directory structure:
- `/app/data`: Main application data directory
- Contains the SQLite database (`gomft.db`)
- Contains rclone configurations in `/app/data/configs`
- Contains log files in `/app/data/logs`
- `/app/backups`: Database backup directory
When using Docker, you should mount volumes to these locations:
```yaml
volumes:
- /host/path/data:/app/data # For all application data
- /host/path/backups:/app/backups # For database backups
```
These paths can be customized using the environment variables `
docs/docs/core-concepts/connections.md
+18 -2
View File
@@ -7,6 +7,8 @@ title: Connections
Connections in GoMFT are configurations that define how to access different storage systems. Before you can transfer files, you need to set up connections for your source and destination systems. Connections in GoMFT are configurations that define how to access different storage systems. Before you can transfer files, you need to set up connections for your source and destination systems.
> **Note**: GoMFT now supports the Storage Provider feature, which allows you to create reusable connection profiles with securely stored credentials. For detailed information, see the [Storage Providers](/docs/user-guides/storage-provider-guide) guide.
## Supported Connection Types ## Supported Connection Types
GoMFT leverages rclone as its transfer engine, supporting a wide range of storage systems: GoMFT leverages rclone as its transfer engine, supporting a wide range of storage systems:
@@ -74,20 +76,32 @@ Different connection types require different configuration fields. Here are some
GoMFT follows best practices for handling connection credentials: GoMFT follows best practices for handling connection credentials:
- **Encryption**: All sensitive credentials are encrypted at rest - **Encryption**: All sensitive credentials are encrypted at rest using AES-256 encryption
- **Access Control**: Connections are protected by user permissions - **Access Control**: Connections are protected by user permissions
- **Masked Values**: Passwords and secret keys are masked in the UI - **Masked Values**: Passwords and secret keys are masked in the UI
- **Key Management**: SSH keys and other credentials are securely stored - **Key Management**: SSH keys and other credentials are securely stored
- **Centralized Management**: With the Storage Provider feature, credentials can be managed in one place and reused across multiple transfers
## Managing Connections ## Managing Connections
You can manage connections either through traditional transfer configurations or using the new Storage Provider feature.
### Viewing Connections ### Viewing Connections
#### Traditional Connections
The **Transfer Configurations** page displays all configured connections with: The **Transfer Configurations** page displays all configured connections with:
- Configuration name - Configuration name
- Configuration type - Configuration type
- Last updated date - Last updated date
#### Storage Providers
Alternatively, you can use the new Storage Provider feature to manage your connections:
1. Navigate to the **Storage Providers** section in the left sidebar
2. View a list of all storage providers you have created
3. Each provider shows name, type, and creation date
### Editing Transfer Confirgurations ### Editing Transfer Confirgurations
To edit an existing connection: To edit an existing connection:
@@ -125,4 +139,6 @@ GoMFT includes a configuration testing feature to verify connectivity:
- **Use service accounts** rather than personal accounts when possible - **Use service accounts** rather than personal accounts when possible
- **Document connection details** in the description field - **Document connection details** in the description field
- **Use the minimal required permissions** for enhanced security - **Use the minimal required permissions** for enhanced security
- **Organize connections** using consistent naming conventions - **Organize connections** using consistent naming conventions
- **Use Storage Providers** for reusable connections across multiple transfers
- **Update credentials in one place** by using Storage Providers instead of updating each transfer individually
docs/docs/core-concepts/schedules.md
+34 -5
View File
@@ -157,12 +157,41 @@ When a schedule runs, GoMFT performs these actions:
GoMFT provides several ways to monitor your scheduled transfers: GoMFT provides several ways to monitor your scheduled transfers:
<!-- ### Schedule Calendar ### Transfer Calendar
View all scheduled transfers in a calendar view: The Transfer Calendar provides a visual overview of all your scheduled transfers:
1. Navigate to **Schedule Calendar** in the Schedules section
2. See all upcoming scheduled transfers in a monthly, weekly, or daily view 1. Navigate to **Transfer Calendar** in the sidebar
3. Click on any scheduled transfer to see details or edit it --> 2. View all scheduled transfers in a monthly, weekly, or daily view
3. Color-coded events indicate different transfer types or statuses
4. Hover over any event to see a summary of the transfer details
5. Click on any scheduled transfer to see full details or edit it
#### Calendar Views
- **Month View**: See all scheduled transfers for the entire month
- **Week View**: Focus on transfers scheduled for the current week
- **Day View**: Detailed timeline of transfers for a specific day
- **Agenda View**: List-based view of upcoming transfers
#### Calendar Features
- **Filtering**: Filter transfers by type, status, or associated connection
<!-- - **Search**: Find specific transfers by name or description -->
<!-- - **Export**: Export calendar events to iCal or CSV format -->
<!-- - **Drag and Drop**: Reschedule transfers by dragging them to a new time slot (requires appropriate permissions) -->
<!-- - **Conflict Detection**: Visual indicators for potentially overlapping transfers -->
<!-- #### Calendar Integration
You can subscribe to the transfer calendar using external calendar applications:
1. Click the **Calendar Subscription** button
2. Copy the provided iCal URL
3. Add the URL as a calendar subscription in applications like Google Calendar, Outlook, or Apple Calendar
4. Set the refresh frequency in your calendar application
> **Note**: The calendar subscription is read-only and requires authentication. Calendar subscriptions will only show transfers that the authenticated user has permission to view. -->
### Transfer History ### Transfer History
docs/docs/core-concepts/transfers.md
+23 -4
View File
@@ -7,6 +7,8 @@ title: Transfers
GoMFT's primary function is to manage file transfers between different storage systems. This page explains the transfer operations available in GoMFT and how to configure them. GoMFT's primary function is to manage file transfers between different storage systems. This page explains the transfer operations available in GoMFT and how to configure them.
> **Note**: GoMFT now supports the Storage Provider feature, which allows you to create reusable connection profiles for your transfers. For detailed information, see the [Storage Providers](/docs/user-guides/storage-provider-guide) guide.
## Transfer Types ## Transfer Types
GoMFT supports several types of transfer operations, each with different behaviors: GoMFT supports several types of transfer operations, each with different behaviors:
@@ -51,10 +53,23 @@ When creating a transfer in GoMFT, you need to configure the following elements:
- **Name**: A descriptive name for the transfer - **Name**: A descriptive name for the transfer
- **Description**: Optional details about the transfer's purpose - **Description**: Optional details about the transfer's purpose
- **Source**: The source connection and path - **Source**: Either a direct connection configuration or a Storage Provider
- **Destination**: The destination connection and path - **Destination**: Either a direct connection configuration or a Storage Provider
- **Transfer Type**: Copy, Sync, Move, or Bidirectional Sync - **Transfer Type**: Copy, Sync, Move, or Bidirectional Sync
#### Using Storage Providers
When creating a transfer, you can now select a Storage Provider instead of entering connection details directly:
1. In the Source or Destination section, select **Provider** from the dropdown
2. Choose from your available Storage Providers
3. Enter the path within the selected provider
This approach offers several benefits:
- Reuse the same provider across multiple transfers
- Update credentials in one place
- Enhanced security with AES-256 encryption for credentials
### Advanced Options ### Advanced Options
#### File Selection #### File Selection
@@ -147,12 +162,14 @@ When a transfer fails, GoMFT provides information to help identify the cause:
1. Check the error message in the transfer history 1. Check the error message in the transfer history
2. Review the detailed logs for the specific error 2. Review the detailed logs for the specific error
3. Common issues include: 3. For transfers using Storage Providers, you can test the provider connection directly from the Storage Providers section
4. Common issues include:
- Permission problems - Permission problems
- Network connectivity - Network connectivity
- Invalid credentials - Invalid credentials
- Path not found - Path not found
- Disk space issues - Disk space issues
- Expired tokens (for OAuth providers like OneDrive or Google Drive)
## Best Practices ## Best Practices
@@ -164,4 +181,6 @@ When a transfer fails, GoMFT provides information to help identify the cause:
- **Set bandwidth limits** to avoid network congestion during peak hours - **Set bandwidth limits** to avoid network congestion during peak hours
- **Schedule large transfers** during off-peak times - **Schedule large transfers** during off-peak times
- **Use notifications** to stay informed about transfer results - **Use notifications** to stay informed about transfer results
- **Regularly review logs** to identify potential issues - **Regularly review logs** to identify potential issues
- **Use Storage Providers** for reusable connections across multiple transfers
- **Convert existing transfers** to use Storage Providers for easier credential management
docs/docs/getting-started/configuration.md
+4 -1
View File
@@ -15,10 +15,11 @@ Environment variables are the primary way to configure GoMFT, especially when ru
| Variable | Description | Default | Example | | Variable | Description | Default | Example |
|----------|-------------|---------|---------| |----------|-------------|---------|---------|
| SERVER_ADDRESS | Server address and port | :8080 | `SERVER_ADDRESS=:9000` | | SERVER_ADDRESS | Server address and port | :8080 | `SERVER_ADDRESS=:8080` |
| DATA_DIR | Main data directory | ./data | `DATA_DIR=/app/data` | | DATA_DIR | Main data directory | ./data | `DATA_DIR=/app/data` |
| BACKUP_DIR | Directory for backups | ./backups | `BACKUP_DIR=/app/backups` | | BACKUP_DIR | Directory for backups | ./backups | `BACKUP_DIR=/app/backups` |
| JWT_SECRET | Secret for JWT tokens | change_this_to_a_secure_random_string | `JWT_SECRET=your-secure-secret-key` | | JWT_SECRET | Secret for JWT tokens | change_this_to_a_secure_random_string | `JWT_SECRET=your-secure-secret-key` |
| GOMFT_ENCRYPTION_KEY | Key used to encrypt sensitive data in the database | change_this_to_a_secure_random_string | `GOMFT_ENCRYPTION_KEY=your-secure-encryption-key` |
| BASE_URL | Base URL for GoMFT (used in email links) | http://localhost:8080 | `BASE_URL=https://gomft.example.com` | | BASE_URL | Base URL for GoMFT (used in email links) | http://localhost:8080 | `BASE_URL=https://gomft.example.com` |
| SKIP_SSL_VERIFY | Skip SSL verification for outgoing webhooks/notifications | false | `SKIP_SSL_VERIFY=false` | | SKIP_SSL_VERIFY | Skip SSL verification for outgoing webhooks/notifications | false | `SKIP_SSL_VERIFY=false` |
@@ -64,6 +65,7 @@ SERVER_ADDRESS=:8080
DATA_DIR=./data DATA_DIR=./data
BACKUP_DIR=./backups BACKUP_DIR=./backups
JWT_SECRET=change_this_to_a_secure_random_string JWT_SECRET=change_this_to_a_secure_random_string
GOMFT_ENCRYPTION_KEY=change_this_to_a_secure_random_string
BASE_URL=http://localhost:8080 BASE_URL=http://localhost:8080
SKIP_SSL_VERIFY=false SKIP_SSL_VERIFY=false
@@ -111,6 +113,7 @@ docker run -d \
-v /path/to/backups:/app/backups \ -v /path/to/backups:/app/backups \
-e SERVER_ADDRESS=:8080 \ -e SERVER_ADDRESS=:8080 \
-e JWT_SECRET=your-secure-secret \ -e JWT_SECRET=your-secure-secret \
-e GOMFT_ENCRYPTION_KEY=your-secure-encryption-key \
-e EMAIL_ENABLED=true \ -e EMAIL_ENABLED=true \
-e EMAIL_HOST=smtp.example.com \ -e EMAIL_HOST=smtp.example.com \
-e PUID=1000 \ -e PUID=1000 \
docs/docs/getting-started/installation.md
+15 -2
View File
@@ -136,6 +136,7 @@ For environments where Docker is not available or preferred, you can install GoM
- Go 1.20 or later - Go 1.20 or later
- Node.js 18 or later - Node.js 18 or later
- gcc (for building SQLite dependencies) - gcc (for building SQLite dependencies)
- templ (for generating template code)
### Building from Source ### Building from Source
@@ -158,13 +159,25 @@ npm install
npm run build npm run build
``` ```
4. Build the Go application: 4. Install templ if you haven't already:
```bash
go install github.com/a-h/templ/cmd/templ@latest
```
5. Generate templ templates:
```bash
templ generate
```
6. Build the Go application:
```bash ```bash
go build -o gomft go build -o gomft
``` ```
5. Run the application: 7. Run the application:
```bash ```bash
./gomft ./gomft
docs/docs/user-guides/storage-provider-guide.md
+326
View File
@@ -0,0 +1,326 @@
---
id: storage-provider-guide
title: Storage Provider Guide
sidebar_label: Storage Providers
description: Detailed instructions for using the Storage Provider feature in GoMFT
---
# Storage Provider User Guide
This guide provides detailed instructions for using the new Storage Provider feature in GoMFT.
## Table of Contents
1. [Introduction](#introduction)
2. [Managing Storage Providers](#managing-storage-providers)
- [Viewing Your Storage Providers](#viewing-your-storage-providers)
- [Creating a New Storage Provider](#creating-a-new-storage-provider)
- [Editing Storage Providers](#editing-storage-providers)
- [Testing Connections](#testing-connections)
- [Deleting Storage Providers](#deleting-storage-providers)
3. [Using Storage Providers in Transfers](#using-storage-providers-in-transfers)
- [Creating Transfers with Storage Providers](#creating-transfers-with-storage-providers)
- [Converting Existing Transfers](#converting-existing-transfers)
4. [Provider Type Reference](#provider-type-reference)
- [SFTP Configuration](#sftp-configuration)
- [S3 Configuration](#s3-configuration)
- [OneDrive Configuration](#onedrive-configuration)
- [Google Drive Configuration](#google-drive-configuration)
- [FTP Configuration](#ftp-configuration)
- [SMB Configuration](#smb-configuration)
5. [Troubleshooting](#troubleshooting)
- [Common Connection Issues](#common-connection-issues)
- [Error Messages](#error-messages)
6. [FAQ](#faq)
## Introduction
The Storage Provider feature allows you to securely store and manage credentials for various storage systems. Instead of entering connection details each time you create a transfer, you can now create reusable storage provider profiles. This approach offers several benefits:
- **Improved Security**: Credentials are stored securely using AES-256 encryption
- **Simplified Management**: Update credentials in one place instead of in each transfer
- **Easier Testing**: Test connections before creating transfers
- **Reusability**: Use the same provider for multiple transfers
## Managing Storage Providers
### Viewing Your Storage Providers
To view your storage providers:
1. Navigate to the **Storage Providers** section in the left sidebar
2. You'll see a list of all storage providers you have created
3. The list shows the provider name, type, and creation date
### Creating a New Storage Provider
To create a new storage provider:
1. From the Storage Providers page, click the **Add Provider** button
2. Enter a descriptive name for the provider
3. Select the provider type from the dropdown (SFTP, S3, OneDrive, etc.)
4. Fill in the required fields for the selected provider type
5. Click **Save** to create the provider or **Save & Test** to create and test the connection
#### Example: Creating an S3 Provider
1. Name: "Company AWS S3 Bucket"
2. Type: S3
3. Fill in the required fields:
- Access Key: Your AWS access key
- Secret Key: Your AWS secret key
- Region: e.g., us-west-2
- Bucket: Your bucket name
- Endpoint: Leave blank for AWS S3 or specify for S3-compatible services
4. Click **Save & Test**
### Editing Storage Providers
To edit an existing storage provider:
1. From the Storage Providers list, click the **Edit** button next to the provider
2. Update the fields as needed
3. For security reasons, sensitive fields (passwords, secret keys) appear empty
- Leave these fields empty to keep the existing values
- Enter new values only if you want to change them
4. Click **Save** to update the provider
### Testing Connections
Testing your storage provider connections ensures they're properly configured:
1. From the Storage Providers list, click the **Test** button next to the provider
2. Or when creating/editing a provider, use the **Save & Test** button
3. The system will attempt to connect using the provided credentials
4. You'll see a success message or an error with details about what went wrong
### Deleting Storage Providers
To delete a storage provider:
1. From the Storage Providers list, click the **Delete** button next to the provider
2. A confirmation dialog will appear
- If the provider is used in any transfers, you'll see a warning listing those transfers
- You cannot delete a provider that's in use without first updating those transfers
3. Confirm deletion if the provider is not in use
## Using Storage Providers in Transfers
### Creating Transfers with Storage Providers
To create a new transfer using storage providers:
1. Navigate to the **Transfers** section and click **New Transfer**
2. Fill in the transfer name and schedule as usual
3. In the Source section, select **Provider** and choose from the dropdown
- Only providers of appropriate types will be shown
- You'll see only providers you've created (unless you're an admin)
4. In the Destination section, also select a provider
5. Configure other transfer settings as needed (paths, file patterns, etc.)
6. Click **Save** to create the transfer
### Converting Existing Transfers
Existing transfers with embedded credentials can be converted to use storage providers:
1. Edit an existing transfer
2. In the Source section, click **Convert to Provider**
- This will create a new storage provider using the embedded credentials
- The provider will be named based on the transfer name
3. Do the same for the Destination section if needed
4. Click **Save** to update the transfer
## Provider Type Reference
### SFTP Configuration
Required fields:
- **Host**: The hostname or IP address of the SFTP server
- **Port**: Server port (usually 22)
- **Username**: Your SFTP username
- **Authentication Method**: Password or Key File
- **Password**: Your SFTP password (if using password authentication)
- **Key File**: Path to SSH private key file (if using key authentication)
Optional fields:
- **Key File Password**: Password for the key file (if the key is password-protected)
Example configuration:
```
Name: Company SFTP Server
Type: SFTP
Host: sftp.example.com
Port: 22
Username: user123
Authentication: Password
Password: ********
```
### S3 Configuration
Required fields:
- **Access Key**: Your S3 access key ID
- **Secret Key**: Your S3 secret access key
- **Bucket**: The S3 bucket name
Optional fields:
- **Region**: The AWS region (e.g., us-east-1)
- **Endpoint**: Server URL for S3-compatible services (leave blank for AWS S3)
Example configuration:
```
Name: Analytics Data Bucket
Type: S3
Access Key: AKIAIOSFODNN7EXAMPLE
Secret Key: ********
Region: us-west-2
Bucket: data-analytics-bucket
```
### OneDrive Configuration
Required fields:
- **Client ID**: Your Microsoft application client ID
- **Client Secret**: Your Microsoft application client secret
- **Refresh Token**: OAuth refresh token for authentication
Optional fields:
- **Drive ID**: Specific drive ID (for accessing shared or team drives)
Example configuration:
```
Name: Marketing OneDrive
Type: OneDrive
Client ID: 12345678-1234-1234-1234-123456789012
Client Secret: ********
Refresh Token: ********
```
### Google Drive Configuration
Required fields:
- **Client ID**: Your Google API client ID
- **Client Secret**: Your Google API client secret
- **Refresh Token**: OAuth refresh token for authentication
Optional fields:
- **Team Drive**: Team drive ID (for accessing shared drives)
Example configuration:
```
Name: Sales Team Drive
Type: Google Drive
Client ID: 123456789012-abcdefghijklmnopqrstuvwxyz.apps.googleusercontent.com
Client Secret: ********
Refresh Token: ********
Team Drive: 0ABCDEFGhijklMNOPQrstuvwxyz
```
### FTP Configuration
Required fields:
- **Host**: The hostname or IP address of the FTP server
- **Port**: Server port (usually 21)
- **Username**: Your FTP username
- **Password**: Your FTP password
Optional fields:
- **Passive Mode**: Enable/disable passive mode (default: enabled)
Example configuration:
```
Name: Legacy FTP Server
Type: FTP
Host: ftp.example.com
Port: 21
Username: ftpuser
Password: ********
Passive Mode: Enabled
```
### SMB Configuration
Required fields:
- **Host**: The hostname or IP address of the SMB/CIFS server
- **Share**: The share name
- **Username**: Your username
- **Password**: Your password
Optional fields:
- **Domain**: Windows domain (if applicable)
- **Port**: Server port (default: 445)
Example configuration:
```
Name: Finance Share
Type: SMB
Host: fileserver.example.com
Share: finance
Username: jsmith
Password: ********
Domain: EXAMPLE
```
## Troubleshooting
### Common Connection Issues
#### SFTP Connection Problems
- **Authentication Failed**: Verify username and password/key file
- **Host Not Found**: Check hostname and network connectivity
- **Permission Denied**: Ensure the user has proper permissions on the server
- **Connection Timeout**: Check firewall settings and server availability
#### S3 Connection Problems
- **Access Denied**: Verify access key, secret key, and bucket permissions
- **Invalid Region**: Ensure the region matches the bucket's region
- **No Such Bucket**: Verify the bucket name and existence
- **Endpoint Error**: For S3-compatible services, verify the endpoint URL
#### OAuth Provider Issues (OneDrive/Google Drive)
- **Invalid Client**: Verify client ID and secret
- **Token Expired**: Refresh tokens may need to be regenerated
- **Permission Scope**: Ensure the token has appropriate scopes for file access
- **Rate Limiting**: You may be making too many requests in a short period
### Error Messages
Common error messages and their solutions:
| Error Message | Possible Cause | Solution |
|---------------|----------------|----------|
| "Connection refused" | Server is not running or blocked by firewall | Check server status and firewall settings |
| "Authentication failed" | Incorrect credentials | Verify username/password or key file |
| "Invalid access key" | Incorrect or expired AWS credentials | Check your access key ID and regenerate if needed |
| "Permission denied" | Insufficient permissions | Check file/folder permissions on the server |
| "Connection timed out" | Network issue or server unavailable | Check network connectivity and server status |
| "No such file or directory" | Path does not exist | Verify the path exists on the server |
## FAQ
**Q: Can I use the same storage provider for multiple transfers?**
A: Yes, that's one of the main benefits. Create the provider once and use it in as many transfers as needed.
**Q: Can I see the passwords or secret keys I've stored?**
A: No, for security reasons, passwords and secret keys are never displayed after they're saved. You can update them, but you cannot view the existing values.
**Q: What happens if I need to update credentials?**
A: Edit the storage provider and enter the new credentials. All transfers using that provider will automatically use the updated credentials.
**Q: Are my credentials secure?**
A: Yes, all sensitive information is encrypted using AES-256 encryption before being stored in the database.
**Q: Can other users see my storage providers?**
A: No, each user can only see and use their own storage providers unless they have administrator privileges.
**Q: Can I export or import storage providers?**
A: Not currently. For security reasons, credential export is not supported.
**Q: What if I'm not sure if a provider is in use?**
A: When attempting to delete a provider, the system will show you all transfers that use it. You can also see usage information in the provider details.
**Q: Can I test a provider without creating a transfer?**
A: Yes, use the "Test" button on the provider list.
docs/sidebars.ts
+5
View File
@@ -29,6 +29,11 @@ const sidebars: SidebarsConfig = {
label: 'Core Concepts', label: 'Core Concepts',
items: ['core-concepts/transfers', 'core-concepts/connections', 'core-concepts/schedules', 'core-concepts/monitoring'], items: ['core-concepts/transfers', 'core-concepts/connections', 'core-concepts/schedules', 'core-concepts/monitoring'],
}, },
{
type: 'category',
label: 'User Guides',
items: ['user-guides/storage-provider-guide'],
},
{ {
type: 'category', type: 'category',
label: 'Advanced Features', label: 'Advanced Features',
internal/config/config.go
+57 -16
View File
@@ -1,6 +1,8 @@
package config package config
import ( import (
"crypto/rand"
"encoding/base64"
"os" "os"
"strconv" "strconv"
"strings" "strings"
@@ -9,14 +11,15 @@ import (
) )
type Config struct { type Config struct {
ServerAddress string `json:"server_address"` ServerAddress string `json:"server_address"`
DataDir string `json:"data_dir"` DataDir string `json:"data_dir"`
BackupDir string `json:"backup_dir"` BackupDir string `json:"backup_dir"`
JWTSecret string `json:"jwt_secret"` JWTSecret string `json:"jwt_secret"`
Email EmailConfig `json:"email"` Email EmailConfig `json:"email"`
BaseURL string `json:"base_url"` // Base URL for generating links in emails BaseURL string `json:"base_url"` // Base URL for generating links in emails
TOTPEncryptKey string `json:"totp_encrypt_key"` // Encryption key for TOTP secrets TOTPEncryptKey string `json:"totp_encrypt_key"` // Encryption key for TOTP secrets
SkipSSLVerify bool `json:"skip_ssl_verify"` // Skip SSL verification for outgoing webhooks/notifications GOMFTEncryptionKey string `json:"gomft_encryption_key"` // Encryption key for database
SkipSSLVerify bool `json:"skip_ssl_verify"` // Skip SSL verification for outgoing webhooks/notifications
} }
type EmailConfig struct { type EmailConfig struct {
@@ -33,15 +36,37 @@ type EmailConfig struct {
} }
func Load() (*Config, error) { func Load() (*Config, error) {
// Generate secure encryption keys
defaultTOTPKey, err := generateSecureKey()
if err != nil {
return nil, err
}
defaultGOMFTKey, err := generateSecureKey()
if err != nil {
return nil, err
}
// Only set the environment variables if they're not already defined
// This ensures user-provided keys take precedence
if os.Getenv("TOTP_ENCRYPTION_KEY") == "" {
os.Setenv("TOTP_ENCRYPTION_KEY", defaultTOTPKey)
}
if os.Getenv("GOMFT_ENCRYPTION_KEY") == "" {
os.Setenv("GOMFT_ENCRYPTION_KEY", defaultGOMFTKey)
}
// Default configuration // Default configuration
cfg := &Config{ cfg := &Config{
ServerAddress: ":8080", ServerAddress: ":8080",
DataDir: "./data", DataDir: "./data",
BackupDir: "./backups", BackupDir: "./backups",
JWTSecret: "change_this_to_a_secure_random_string", JWTSecret: "change_this_to_a_secure_random_string",
BaseURL: "http://localhost:8080", BaseURL: "http://localhost:8080",
TOTPEncryptKey: "this-is-a-dev-key-not-for-production!", // Default development key TOTPEncryptKey: defaultTOTPKey, // Secure randomly generated key
SkipSSLVerify: false, // Default to verifying SSL GOMFTEncryptionKey: defaultGOMFTKey, // Secure randomly generated key
SkipSSLVerify: false, // Default to verifying SSL
Email: EmailConfig{ Email: EmailConfig{
Enabled: false, Enabled: false,
Host: "smtp.example.com", Host: "smtp.example.com",
@@ -87,7 +112,9 @@ func Load() (*Config, error) {
if totpKey := os.Getenv("TOTP_ENCRYPTION_KEY"); totpKey != "" { if totpKey := os.Getenv("TOTP_ENCRYPTION_KEY"); totpKey != "" {
cfg.TOTPEncryptKey = totpKey cfg.TOTPEncryptKey = totpKey
} }
if gomftKey := os.Getenv("GOMFT_ENCRYPTION_KEY"); gomftKey != "" {
cfg.GOMFTEncryptionKey = gomftKey
}
// Email configuration // Email configuration
if emailEnabled := os.Getenv("EMAIL_ENABLED"); emailEnabled != "" { if emailEnabled := os.Getenv("EMAIL_ENABLED"); emailEnabled != "" {
cfg.Email.Enabled = strings.ToLower(emailEnabled) == "true" cfg.Email.Enabled = strings.ToLower(emailEnabled) == "true"
@@ -145,6 +172,7 @@ func Load() (*Config, error) {
"", "",
"# Two-Factor Authentication configuration", "# Two-Factor Authentication configuration",
"TOTP_ENCRYPTION_KEY=" + cfg.TOTPEncryptKey, "TOTP_ENCRYPTION_KEY=" + cfg.TOTPEncryptKey,
"GOMFT_ENCRYPTION_KEY=" + cfg.GOMFTEncryptionKey,
"", "",
"# Email configuration", "# Email configuration",
"EMAIL_ENABLED=" + strconv.FormatBool(cfg.Email.Enabled), "EMAIL_ENABLED=" + strconv.FormatBool(cfg.Email.Enabled),
@@ -171,3 +199,16 @@ func Load() (*Config, error) {
return cfg, nil return cfg, nil
} }
// generateSecureKey creates a cryptographically secure random key encoded as base64
func generateSecureKey() (string, error) {
// Generate 32 bytes of random data (256 bits)
bytes := make([]byte, 32)
_, err := rand.Read(bytes)
if err != nil {
return "", err
}
// Encode as base64
return base64.StdEncoding.EncodeToString(bytes), nil
}
internal/web/handlers/storage_provider_handlers.go
+210 -9
View File
@@ -99,9 +99,21 @@ func (h *Handlers) HandleCreateStorageProvider(c *gin.Context) {
// Set created by // Set created by
provider.CreatedBy = userID provider.CreatedBy = userID
// Start a transaction
tx := h.DB.Begin()
if tx.Error != nil {
ctx := components.CreateTemplateContext(c)
_ = components.StorageProviderForm(ctx, components.StorageProviderFormData{
Provider: &provider,
IsEdit: false,
Error: "Failed to begin transaction",
}).Render(ctx, c.Writer)
return
}
// Create provider in database // Create provider in database
err := h.DB.CreateStorageProvider(&provider) if err := tx.Create(&provider).Error; err != nil {
if err != nil { tx.Rollback()
ctx := components.CreateTemplateContext(c) ctx := components.CreateTemplateContext(c)
_ = components.StorageProviderForm(ctx, components.StorageProviderFormData{ _ = components.StorageProviderForm(ctx, components.StorageProviderFormData{
Provider: &provider, Provider: &provider,
@@ -111,6 +123,48 @@ func (h *Handlers) HandleCreateStorageProvider(c *gin.Context) {
return return
} }
// Create audit log
auditDetails := map[string]interface{}{
"name": provider.Name,
"type": provider.Type,
"host": provider.Host,
"port": provider.Port,
"username": provider.Username,
}
auditLog := db.AuditLog{
Action: "create",
EntityType: "storage_provider",
EntityID: provider.ID,
UserID: userID,
Details: auditDetails,
Timestamp: time.Now(),
}
if err := tx.Create(&auditLog).Error; err != nil {
tx.Rollback()
log.Printf("Error creating audit log: %v", err)
ctx := components.CreateTemplateContext(c)
_ = components.StorageProviderForm(ctx, components.StorageProviderFormData{
Provider: &provider,
IsEdit: false,
Error: "Failed to create audit log",
}).Render(ctx, c.Writer)
return
}
// Commit the transaction
if err := tx.Commit().Error; err != nil {
log.Printf("Error committing transaction: %v", err)
ctx := components.CreateTemplateContext(c)
_ = components.StorageProviderForm(ctx, components.StorageProviderFormData{
Provider: &provider,
IsEdit: false,
Error: "Failed to commit transaction",
}).Render(ctx, c.Writer)
return
}
// Test if requested // Test if requested
if c.PostForm("test") == "true" { if c.PostForm("test") == "true" {
// Create connector service // Create connector service
@@ -218,9 +272,51 @@ func (h *Handlers) HandleUpdateStorageProvider(c *gin.Context) {
provider.EncryptedRefreshToken = existingProvider.EncryptedRefreshToken provider.EncryptedRefreshToken = existingProvider.EncryptedRefreshToken
} }
// Start a transaction
tx := h.DB.Begin()
if tx.Error != nil {
ctx := components.CreateTemplateContext(c)
_ = components.StorageProviderForm(ctx, components.StorageProviderFormData{
Provider: &provider,
IsEdit: true,
Error: "Failed to begin transaction",
}).Render(ctx, c.Writer)
return
}
// Create audit log before update
auditDetails := map[string]interface{}{
"name": provider.Name,
"type": provider.Type,
"host": provider.Host,
"port": provider.Port,
"username": provider.Username,
}
auditLog := db.AuditLog{
Action: "update",
EntityType: "storage_provider",
EntityID: provider.ID,
UserID: userID,
Details: auditDetails,
Timestamp: time.Now(),
}
if err := tx.Create(&auditLog).Error; err != nil {
tx.Rollback()
log.Printf("Error creating audit log: %v", err)
ctx := components.CreateTemplateContext(c)
_ = components.StorageProviderForm(ctx, components.StorageProviderFormData{
Provider: &provider,
IsEdit: true,
Error: "Failed to create audit log",
}).Render(ctx, c.Writer)
return
}
// Update provider in database // Update provider in database
err = h.DB.UpdateStorageProvider(&provider) if err := tx.Save(&provider).Error; err != nil {
if err != nil { tx.Rollback()
ctx := components.CreateTemplateContext(c) ctx := components.CreateTemplateContext(c)
_ = components.StorageProviderForm(ctx, components.StorageProviderFormData{ _ = components.StorageProviderForm(ctx, components.StorageProviderFormData{
Provider: &provider, Provider: &provider,
@@ -230,6 +326,18 @@ func (h *Handlers) HandleUpdateStorageProvider(c *gin.Context) {
return return
} }
// Commit the transaction
if err := tx.Commit().Error; err != nil {
log.Printf("Error committing transaction: %v", err)
ctx := components.CreateTemplateContext(c)
_ = components.StorageProviderForm(ctx, components.StorageProviderFormData{
Provider: &provider,
IsEdit: true,
Error: "Failed to commit transaction",
}).Render(ctx, c.Writer)
return
}
// Test if requested // Test if requested
if c.PostForm("test") == "true" { if c.PostForm("test") == "true" {
// Create connector service // Create connector service
@@ -286,13 +394,67 @@ func (h *Handlers) HandleDeleteStorageProvider(c *gin.Context) {
return return
} }
// Get provider
var provider db.StorageProvider
if err := h.DB.First(&provider, id).Error; err != nil {
c.JSON(http.StatusNotFound, gin.H{"error": "Storage provider not found"})
return
}
// Check if user owns this provider
if provider.CreatedBy != c.GetUint("userID") {
// Check if user is admin
isAdmin, exists := c.Get("isAdmin")
if !exists || isAdmin != true {
c.JSON(http.StatusForbidden, gin.H{"error": "You do not have permission to delete this provider"})
return
}
}
// Start a transaction
tx := h.DB.Begin()
if tx.Error != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to begin transaction"})
return
}
// Create audit log before deletion
auditDetails := map[string]interface{}{
"name": provider.Name,
"type": provider.Type,
"host": provider.Host,
"port": provider.Port,
"username": provider.Username,
}
auditLog := db.AuditLog{
Action: "delete",
EntityType: "storage_provider",
EntityID: provider.ID,
UserID: c.GetUint("userID"),
Details: auditDetails,
Timestamp: time.Now(),
}
if err := tx.Create(&auditLog).Error; err != nil {
tx.Rollback()
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to create audit log"})
return
}
// Delete provider // Delete provider
err = h.DB.DeleteStorageProvider(uint(id)) if err := tx.Delete(&provider).Error; err != nil {
if err != nil { tx.Rollback()
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Failed to delete provider: %v", err)}) c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Failed to delete provider: %v", err)})
return return
} }
// Commit the transaction
if err := tx.Commit().Error; err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to commit transaction"})
return
}
c.Header("HX-Refresh", "true") c.Header("HX-Refresh", "true")
c.JSON(http.StatusOK, gin.H{"message": "Provider deleted successfully"}) c.JSON(http.StatusOK, gin.H{"message": "Provider deleted successfully"})
} }
@@ -376,13 +538,52 @@ func (h *Handlers) HandleDuplicateStorageProvider(c *gin.Context) {
duplicateProvider.UpdatedAt = time.Now() duplicateProvider.UpdatedAt = time.Now()
// Deep copy pointer fields here if any are added in the future // Deep copy pointer fields here if any are added in the future
// Save the duplicate // Start a transaction
err = h.DB.CreateStorageProvider(&duplicateProvider) tx := h.DB.Begin()
if err != nil { if tx.Error != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to begin transaction"})
return
}
// Create the duplicate in the database
if err := tx.Create(&duplicateProvider).Error; err != nil {
tx.Rollback()
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to create duplicate provider: " + err.Error()}) c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to create duplicate provider: " + err.Error()})
return return
} }
// Create audit log for duplication
auditDetails := map[string]interface{}{
"name": duplicateProvider.Name,
"type": duplicateProvider.Type,
"original_id": originalProvider.ID,
"original_name": originalProvider.Name,
"host": duplicateProvider.Host,
"port": duplicateProvider.Port,
"username": duplicateProvider.Username,
}
auditLog := db.AuditLog{
Action: "duplicate",
EntityType: "storage_provider",
EntityID: duplicateProvider.ID,
UserID: userID,
Details: auditDetails,
Timestamp: time.Now(),
}
if err := tx.Create(&auditLog).Error; err != nil {
tx.Rollback()
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to create audit log"})
return
}
// Commit the transaction
if err := tx.Commit().Error; err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to commit transaction"})
return
}
c.Header("HX-Refresh", "true") c.Header("HX-Refresh", "true")
c.JSON(http.StatusOK, gin.H{"message": "Provider duplicated successfully"}) c.JSON(http.StatusOK, gin.H{"message": "Provider duplicated successfully"})
} }
testing.md
-411
View File
@@ -1,411 +0,0 @@
# GoMFT Testing Guide
This document outlines the testing strategy and approaches for the GoMFT application.
## Testing Structure
The test suite is organized by components, following Go's standard pattern of placing test files alongside the code they test. For each package, we create corresponding `*_test.go` files.
## Test Types
### 1. Unit Tests
Unit tests focus on testing individual functions and components in isolation. Examples include:
- Configuration loading and validation
- Password hashing and validation
- JWT token generation and validation
- Database operations
### 2. Integration Tests
Integration tests verify that different components work together correctly. Examples include:
- Database operations that span multiple tables
- Authentication flows that involve multiple components
- File transfer operations that involve multiple services
### 3. API Tests
API tests verify HTTP endpoints and request handling. Examples include:
- Authentication endpoints
- CRUD operations on resources
- File transfer management endpoints
### 4. Webhook Tests
Webhook tests verify the correct functioning of the webhook notification system. Examples include:
- Webhook URL validation during job creation/update
- Webhook headers JSON validation
- Webhook delivery when jobs complete successfully
- Webhook delivery when jobs fail
- HMAC-SHA256 signature generation and verification
- Custom HTTP headers inclusion in webhook requests
### 5. Admin Tool Tests
Admin Tool tests verify the functionality of administrative interfaces. Examples include:
- Log Viewer functionality
- Database backup and restore operations
- System statistics reporting
- Maintenance functions (e.g., VACUUM)
## Testing Utilities
A central `testutils` package provides common utilities for testing:
- Database setup with in-memory SQLite
- Test user creation
- JWT token generation
- Configuration setup
## Running Tests
To run all tests:
```bash
go test ./...
```
To run tests for a specific package:
```bash
go test ./internal/db
```
To run a specific test:
```bash
go test ./internal/db -run TestUserCRUD
```
To see test coverage:
```bash
go test ./... -cover
```
For a detailed HTML coverage report:
```bash
go test ./... -coverprofile=coverage.out
go tool cover -html=coverage.out
```
## Mocking
For components that depend on external services or complex dependencies, we use mocking techniques:
- In-memory SQLite for database tests
- Mock schedulers for job scheduling tests
- Mock email services for email tests
- Mock HTTP servers for webhook receiver tests
- Mock file system for Log Viewer tests
### Webhook Testing Mocks
For webhook testing, implement the following mocks:
```go
// Example webhook receiver mock
func setupWebhookMock(t *testing.T) (string, chan []byte, chan http.Header) {
payloadCh := make(chan []byte, 1)
headersCh := make(chan http.Header, 1)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
payloadCh <- body
headersCh <- r.Header.Clone()
w.WriteHeader(http.StatusOK)
}))
t.Cleanup(func() {
server.Close()
})
return server.URL, payloadCh, headersCh
}
```
### Log Viewer Testing Mocks
For Log Viewer testing, implement file system mocks:
```go
// Example log file system mock
func setupLogFilesMock(t *testing.T) string {
tempDir := t.TempDir()
// Create sample log files
for i, content := range []string{
"INFO: Test log entry 1\nERROR: Test error\n",
"INFO: Test log entry 2\nWARN: Test warning\n",
} {
filename := fmt.Sprintf("test_log_%d.log", i)
err := os.WriteFile(filepath.Join(tempDir, filename), []byte(content), 0644)
require.NoError(t, err)
}
return tempDir
}
```
## Test Data
Test data should be created programmatically rather than relying on existing data in the database. This ensures tests are repeatable and isolated.
## Continuous Integration
Tests are automatically run as part of the CI pipeline to ensure code quality and prevent regressions.
## Example Tests
Here are examples of different types of tests:
### Configuration Test Example
```go
// See internal/config/config_test.go
func TestLoad(t *testing.T) {
// Test loading configuration from environment variables
}
```
### Database Test Example
```go
// See internal/db/db_test.go
func TestUserCRUD(t *testing.T) {
// Test creating, reading, updating, and deleting users
}
```
### HTTP Handler Test Example
```go
// See internal/web/handlers/basic_handlers_test.go
func TestHandleHome(t *testing.T) {
// Test handling home page requests
}
```
### Webhook Test Example
```go
// See internal/web/handlers/webhook_test.go
func TestWebhookValidation(t *testing.T) {
// Set up test environment
handlers, router, _, _, config := setupJobsTest(t)
// Add job create route
router.POST("/jobs/create", handlers.HandleCreateJob)
// Create job form data with invalid webhook URL
formData := url.Values{
"name": {"Invalid Webhook Job"},
"config_ids[]": {strconv.Itoa(int(config.ID))},
"schedule": {"*/15 * * * *"},
"enabled": {"true"},
"webhook_enabled": {"true"},
"webhook_url": {"invalid-url"}, // Invalid URL
"webhook_secret": {"test-secret"},
"notify_on_success": {"true"},
"notify_on_failure": {"true"},
}
// Submit form
req, _ := http.NewRequest("POST", "/jobs/create", strings.NewReader(formData.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
resp := httptest.NewRecorder()
router.ServeHTTP(resp, req)
// Should not create job with invalid webhook URL
assert.NotEqual(t, http.StatusFound, resp.Code)
assert.Contains(t, resp.Body.String(), "valid URL")
}
```
### Admin Tools Test Example
```go
// See internal/web/handlers/admin_handlers_test.go
func TestLogViewer(t *testing.T) {
// Set up test environment with mock log files
logDir := setupLogFilesMock(t)
t.Setenv("LOGS_DIR", logDir)
handlers, router, _ := setupAdminTest(t)
// Add log viewer route
router.GET("/admin/logs/view/:filename", handlers.HandleViewLogFile)
// Create request to view log file
req, _ := http.NewRequest("GET", "/admin/logs/view/test_log_0.log", nil)
resp := httptest.NewRecorder()
// Add admin user to context
ctx, _ := gin.CreateTestContext(resp)
ctx.Set("userID", uint(1))
ctx.Set("isAdmin", true)
req = req.WithContext(ctx)
// Serve request
router.ServeHTTP(resp, req)
// Verify response contains log content
assert.Equal(t, http.StatusOK, resp.Code)
assert.Contains(t, resp.Body.String(), "Test log entry 1")
assert.Contains(t, resp.Body.String(), "Test error")
}
```
## Test Best Practices
1. **Isolation**: Each test should be independent and not rely on the state of other tests.
2. **Coverage**: Aim for high test coverage, especially for critical components.
3. **Readability**: Tests should be easy to read and understand.
4. **Performance**: Tests should run quickly to enable fast feedback cycles.
5. **Maintainability**: Tests should be easy to maintain and update as the codebase evolves.
## Recent Testing Improvements
### Database Layer Testing
The database layer has seen significant improvements in test coverage. Key improvements include:
- Comprehensive CRUD operation tests
- Error handling tests for edge cases
- Transaction tests
- Tests for database initialization and migration
### Web Handlers Testing
#### File Metadata Handlers
We've implemented comprehensive tests for the file metadata handlers:
- `ListFileMetadata`
- `GetFileMetadataDetails`
- `GetFileMetadataForJob`
- `SearchFileMetadata`
- `DeleteFileMetadata`
- `HandleFileMetadataPartial`
- `HandleFileMetadataSearchPartial`
These tests cover:
- Authentication and authorization
- Pagination
- Filtering
- Error handling
- HTMX integration
#### Testing Challenges and Solutions
When testing web handlers, we encountered several challenges:
1. **Authentication**: Tests needed to simulate authenticated users with proper permissions.
2. **HTMX Integration**: Many handlers expect HTMX headers for proper functioning.
3. **HTML Response Validation**: Validating HTML responses can be brittle.
Solutions implemented:
- Created helper functions to set up authentication context
- Added HTMX headers to test requests
- Focused on verifying database state rather than HTML content
## Next Steps for Testing
### Web Handlers
The overall coverage for the web handlers package needs improvement. To improve this, we should focus on:
1. **Authentication Handlers**: Implement tests for login, logout, and registration handlers.
2. **Job Handlers**: Test job creation, modification, and deletion handlers.
3. **Configuration Handlers**: Test transfer configuration management handlers.
4. **Dashboard Handlers**: Test dashboard data retrieval handlers.
### API Layer
The API layer currently has minimal test coverage. We should implement tests for:
1. **API Authentication**: Test API token generation and validation.
2. **API Endpoints**: Test all REST API endpoints.
3. **Error Handling**: Test API error responses.
### Scheduler
The scheduler component needs tests for:
1. **Job Scheduling**: Test scheduling and execution of jobs.
2. **Error Handling**: Test error handling during job execution.
3. **Concurrency**: Test concurrent job execution.
### Performance Testing
Implement performance tests for critical operations:
1. **File Transfer**: Test large file transfer performance.
2. **Database Operations**: Test database performance under load.
3. **API Endpoints**: Test API endpoint performance.
### Webhook Testing
The webhook functionality requires comprehensive testing:
1. **Validation Tests**:
- Ensure invalid webhook URLs are rejected during job creation/updates
- Verify malformed JSON in webhook headers is detected and rejected
- Test validation edge cases (empty URLs, very long URLs, etc.)
2. **Notification Tests**:
- Verify webhooks are sent for successful job completion when configured
- Verify webhooks are sent for failed jobs when configured
- Confirm webhooks are not sent when the feature is disabled
- Test the conditional notification settings (notify on success, notify on failure)
3. **Security Tests**:
- Verify HMAC-SHA256 signatures are correctly generated
- Test signature verification process
- Ensure webhook secrets are securely handled
4. **Integration Tests**:
- Set up a mock webhook receiver to catch and validate payloads
- Test with various job types and configurations
- Verify all expected payload fields are present and accurate
### Admin Tools Testing
The Admin Tools interface, particularly the Log Viewer, requires testing:
1. **Log Viewer Tests**:
- Verify all log files are correctly listed and accessible
- Test the log file content display functionality
- Verify log download capability works correctly
- Test refresh functionality updates the log list and content
- Verify the viewer works correctly with various log file sizes
- Test compatibility with log rotation
2. **Database Management Tests**:
- Verify backup creation and listing functionality
- Test database restore capability
- Verify backup download functionality
- Test database optimization functions
3. **System Statistics Tests**:
- Verify accurate reporting of system metrics (database size, job counts, etc.)
- Test uptime calculation and display
## Conclusion
Continued focus on testing will ensure the reliability and maintainability of the GoMFT application. By systematically addressing each component, we can achieve high test coverage and confidence in the codebase.
The recent addition of webhook notification capabilities and admin tools, including the Log Viewer, has expanded the testing requirements. These new features involve various aspects of the system, from HTTP handling to file system operations, and require a comprehensive testing approach that considers:
1. **Functionality Testing**: Ensuring the basic functionality works as expected
2. **Edge Case Testing**: Handling invalid input and extreme conditions
3. **Integration Testing**: Verifying the components work together correctly
4. **Security Testing**: Validating security measures like HMAC signatures
By implementing the testing strategies outlined in this document, we can ensure that all components of the GoMFT system, including these newer features, maintain high quality and reliability.