Github-Mirror/GoMFT
1
0
Fork 0
mirror of https://github.com/StarFleetCPTN/GoMFT.git synced 2026-09-12 09:30:52 +02:00
Code Issues Packages Projects Releases Wiki Activity

Add audit logging to storage providers

Browse Source
This commit is contained in:
StarFleetCPTN
2025-04-19 07:58:14 -07:00
parent 98e20ea843
commit 495b844c00
11 changed files with 706 additions and 988 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/docs/core-concepts/connections.md
+18 -2
View File
@@ -7,6 +7,8 @@ title: Connections
Connections in GoMFT are configurations that define how to access different storage systems. Before you can transfer files, you need to set up connections for your source and destination systems.
> **Note**: GoMFT now supports the Storage Provider feature, which allows you to create reusable connection profiles with securely stored credentials. For detailed information, see the [Storage Providers](/docs/user-guides/storage-provider-guide) guide.
## Supported Connection Types
GoMFT leverages rclone as its transfer engine, supporting a wide range of storage systems:
@@ -74,20 +76,32 @@ Different connection types require different configuration fields. Here are some
GoMFT follows best practices for handling connection credentials:
- **Encryption**: All sensitive credentials are encrypted at rest
- **Encryption**: All sensitive credentials are encrypted at rest using AES-256 encryption
- **Access Control**: Connections are protected by user permissions
- **Masked Values**: Passwords and secret keys are masked in the UI
- **Key Management**: SSH keys and other credentials are securely stored
- **Centralized Management**: With the Storage Provider feature, credentials can be managed in one place and reused across multiple transfers
## Managing Connections
You can manage connections either through traditional transfer configurations or using the new Storage Provider feature.
### Viewing Connections
#### Traditional Connections
The **Transfer Configurations** page displays all configured connections with:
- Configuration name
- Configuration type
- Last updated date
#### Storage Providers
Alternatively, you can use the new Storage Provider feature to manage your connections:
1. Navigate to the **Storage Providers** section in the left sidebar
2. View a list of all storage providers you have created
3. Each provider shows name, type, and creation date
### Editing Transfer Confirgurations
To edit an existing connection:
@@ -125,4 +139,6 @@ GoMFT includes a configuration testing feature to verify connectivity:
- **Use service accounts** rather than personal accounts when possible
- **Document connection details** in the description field
- **Use the minimal required permissions** for enhanced security
- **Organize connections** using consistent naming conventions
- **Organize connections** using consistent naming conventions
- **Use Storage Providers** for reusable connections across multiple transfers
- **Update credentials in one place** by using Storage Providers instead of updating each transfer individually
docs/docs/core-concepts/schedules.md
+34 -5
View File
@@ -157,12 +157,41 @@ When a schedule runs, GoMFT performs these actions:
GoMFT provides several ways to monitor your scheduled transfers:
<!-- ### Schedule Calendar
### Transfer Calendar
View all scheduled transfers in a calendar view:
1. Navigate to **Schedule Calendar** in the Schedules section
2. See all upcoming scheduled transfers in a monthly, weekly, or daily view
3. Click on any scheduled transfer to see details or edit it -->
The Transfer Calendar provides a visual overview of all your scheduled transfers:
1. Navigate to **Transfer Calendar** in the sidebar
2. View all scheduled transfers in a monthly, weekly, or daily view
3. Color-coded events indicate different transfer types or statuses
4. Hover over any event to see a summary of the transfer details
5. Click on any scheduled transfer to see full details or edit it
#### Calendar Views
- **Month View**: See all scheduled transfers for the entire month
- **Week View**: Focus on transfers scheduled for the current week
- **Day View**: Detailed timeline of transfers for a specific day
- **Agenda View**: List-based view of upcoming transfers
#### Calendar Features
- **Filtering**: Filter transfers by type, status, or associated connection
<!-- - **Search**: Find specific transfers by name or description -->
<!-- - **Export**: Export calendar events to iCal or CSV format -->
<!-- - **Drag and Drop**: Reschedule transfers by dragging them to a new time slot (requires appropriate permissions) -->
<!-- - **Conflict Detection**: Visual indicators for potentially overlapping transfers -->
<!-- #### Calendar Integration
You can subscribe to the transfer calendar using external calendar applications:
1. Click the **Calendar Subscription** button
2. Copy the provided iCal URL
3. Add the URL as a calendar subscription in applications like Google Calendar, Outlook, or Apple Calendar
4. Set the refresh frequency in your calendar application
> **Note**: The calendar subscription is read-only and requires authentication. Calendar subscriptions will only show transfers that the authenticated user has permission to view. -->
### Transfer History
docs/docs/core-concepts/transfers.md
+23 -4
View File
@@ -7,6 +7,8 @@ title: Transfers
GoMFT's primary function is to manage file transfers between different storage systems. This page explains the transfer operations available in GoMFT and how to configure them.
> **Note**: GoMFT now supports the Storage Provider feature, which allows you to create reusable connection profiles for your transfers. For detailed information, see the [Storage Providers](/docs/user-guides/storage-provider-guide) guide.
## Transfer Types
GoMFT supports several types of transfer operations, each with different behaviors:
@@ -51,10 +53,23 @@ When creating a transfer in GoMFT, you need to configure the following elements:
- **Name**: A descriptive name for the transfer
- **Description**: Optional details about the transfer's purpose
- **Source**: The source connection and path
- **Destination**: The destination connection and path
- **Source**: Either a direct connection configuration or a Storage Provider
- **Destination**: Either a direct connection configuration or a Storage Provider
- **Transfer Type**: Copy, Sync, Move, or Bidirectional Sync
#### Using Storage Providers
When creating a transfer, you can now select a Storage Provider instead of entering connection details directly:
1. In the Source or Destination section, select **Provider** from the dropdown
2. Choose from your available Storage Providers
3. Enter the path within the selected provider
This approach offers several benefits:
- Reuse the same provider across multiple transfers
- Update credentials in one place
- Enhanced security with AES-256 encryption for credentials
### Advanced Options
#### File Selection
@@ -147,12 +162,14 @@ When a transfer fails, GoMFT provides information to help identify the cause:
1. Check the error message in the transfer history
2. Review the detailed logs for the specific error
3. Common issues include:
3. For transfers using Storage Providers, you can test the provider connection directly from the Storage Providers section
4. Common issues include:
- Permission problems
- Network connectivity
- Invalid credentials
- Path not found
- Disk space issues
- Expired tokens (for OAuth providers like OneDrive or Google Drive)
## Best Practices
@@ -164,4 +181,6 @@ When a transfer fails, GoMFT provides information to help identify the cause:
- **Set bandwidth limits** to avoid network congestion during peak hours
- **Schedule large transfers** during off-peak times
- **Use notifications** to stay informed about transfer results
- **Regularly review logs** to identify potential issues
- **Regularly review logs** to identify potential issues
- **Use Storage Providers** for reusable connections across multiple transfers
- **Convert existing transfers** to use Storage Providers for easier credential management
docs/docs/getting-started/configuration.md
+4 -1
View File
@@ -15,10 +15,11 @@ Environment variables are the primary way to configure GoMFT, especially when ru
| Variable | Description | Default | Example |
|----------|-------------|---------|---------|
| SERVER_ADDRESS | Server address and port | :8080 | `SERVER_ADDRESS=:9000` |
| SERVER_ADDRESS | Server address and port | :8080 | `SERVER_ADDRESS=:8080` |
| DATA_DIR | Main data directory | ./data | `DATA_DIR=/app/data` |
| BACKUP_DIR | Directory for backups | ./backups | `BACKUP_DIR=/app/backups` |
| JWT_SECRET | Secret for JWT tokens | change_this_to_a_secure_random_string | `JWT_SECRET=your-secure-secret-key` |
| GOMFT_ENCRYPTION_KEY | Key used to encrypt sensitive data in the database | change_this_to_a_secure_random_string | `GOMFT_ENCRYPTION_KEY=your-secure-encryption-key` |
| BASE_URL | Base URL for GoMFT (used in email links) | http://localhost:8080 | `BASE_URL=https://gomft.example.com` |
| SKIP_SSL_VERIFY | Skip SSL verification for outgoing webhooks/notifications | false | `SKIP_SSL_VERIFY=false` |
@@ -64,6 +65,7 @@ SERVER_ADDRESS=:8080
DATA_DIR=./data
BACKUP_DIR=./backups
JWT_SECRET=change_this_to_a_secure_random_string
GOMFT_ENCRYPTION_KEY=change_this_to_a_secure_random_string
BASE_URL=http://localhost:8080
SKIP_SSL_VERIFY=false
@@ -111,6 +113,7 @@ docker run -d \
-v /path/to/backups:/app/backups \
-e SERVER_ADDRESS=:8080 \
-e JWT_SECRET=your-secure-secret \
-e GOMFT_ENCRYPTION_KEY=your-secure-encryption-key \
-e EMAIL_ENABLED=true \
-e EMAIL_HOST=smtp.example.com \
-e PUID=1000 \
docs/docs/getting-started/installation.md
+15 -2
View File
@@ -136,6 +136,7 @@ For environments where Docker is not available or preferred, you can install GoM
- Go 1.20 or later
- Node.js 18 or later
- gcc (for building SQLite dependencies)
- templ (for generating template code)
### Building from Source
@@ -158,13 +159,25 @@ npm install
npm run build
```
4. Build the Go application:
4. Install templ if you haven't already:
```bash
go install github.com/a-h/templ/cmd/templ@latest
```
5. Generate templ templates:
```bash
templ generate
```
6. Build the Go application:
```bash
go build -o gomft
```
5. Run the application:
7. Run the application:
```bash
./gomft
docs/docs/user-guides/storage-provider-guide.md
+326
View File
@@ -0,0 +1,326 @@
---
id: storage-provider-guide
title: Storage Provider Guide
sidebar_label: Storage Providers
description: Detailed instructions for using the Storage Provider feature in GoMFT
---
# Storage Provider User Guide
This guide provides detailed instructions for using the new Storage Provider feature in GoMFT.
## Table of Contents
1. [Introduction](#introduction)
2. [Managing Storage Providers](#managing-storage-providers)
- [Viewing Your Storage Providers](#viewing-your-storage-providers)
- [Creating a New Storage Provider](#creating-a-new-storage-provider)
- [Editing Storage Providers](#editing-storage-providers)
- [Testing Connections](#testing-connections)
- [Deleting Storage Providers](#deleting-storage-providers)
3. [Using Storage Providers in Transfers](#using-storage-providers-in-transfers)
- [Creating Transfers with Storage Providers](#creating-transfers-with-storage-providers)
- [Converting Existing Transfers](#converting-existing-transfers)
4. [Provider Type Reference](#provider-type-reference)
- [SFTP Configuration](#sftp-configuration)
- [S3 Configuration](#s3-configuration)
- [OneDrive Configuration](#onedrive-configuration)
- [Google Drive Configuration](#google-drive-configuration)
- [FTP Configuration](#ftp-configuration)
- [SMB Configuration](#smb-configuration)
5. [Troubleshooting](#troubleshooting)
- [Common Connection Issues](#common-connection-issues)
- [Error Messages](#error-messages)
6. [FAQ](#faq)
## Introduction
The Storage Provider feature allows you to securely store and manage credentials for various storage systems. Instead of entering connection details each time you create a transfer, you can now create reusable storage provider profiles. This approach offers several benefits:
- **Improved Security**: Credentials are stored securely using AES-256 encryption
- **Simplified Management**: Update credentials in one place instead of in each transfer
- **Easier Testing**: Test connections before creating transfers
- **Reusability**: Use the same provider for multiple transfers
## Managing Storage Providers
### Viewing Your Storage Providers
To view your storage providers:
1. Navigate to the **Storage Providers** section in the left sidebar
2. You'll see a list of all storage providers you have created
3. The list shows the provider name, type, and creation date
### Creating a New Storage Provider
To create a new storage provider:
1. From the Storage Providers page, click the **Add Provider** button
2. Enter a descriptive name for the provider
3. Select the provider type from the dropdown (SFTP, S3, OneDrive, etc.)
4. Fill in the required fields for the selected provider type
5. Click **Save** to create the provider or **Save & Test** to create and test the connection
#### Example: Creating an S3 Provider
1. Name: "Company AWS S3 Bucket"
2. Type: S3
3. Fill in the required fields:
- Access Key: Your AWS access key
- Secret Key: Your AWS secret key
- Region: e.g., us-west-2
- Bucket: Your bucket name
- Endpoint: Leave blank for AWS S3 or specify for S3-compatible services
4. Click **Save & Test**
### Editing Storage Providers
To edit an existing storage provider:
1. From the Storage Providers list, click the **Edit** button next to the provider
2. Update the fields as needed
3. For security reasons, sensitive fields (passwords, secret keys) appear empty
- Leave these fields empty to keep the existing values
- Enter new values only if you want to change them
4. Click **Save** to update the provider
### Testing Connections
Testing your storage provider connections ensures they're properly configured:
1. From the Storage Providers list, click the **Test** button next to the provider
2. Or when creating/editing a provider, use the **Save & Test** button
3. The system will attempt to connect using the provided credentials
4. You'll see a success message or an error with details about what went wrong
### Deleting Storage Providers
To delete a storage provider:
1. From the Storage Providers list, click the **Delete** button next to the provider
2. A confirmation dialog will appear
- If the provider is used in any transfers, you'll see a warning listing those transfers
- You cannot delete a provider that's in use without first updating those transfers
3. Confirm deletion if the provider is not in use
## Using Storage Providers in Transfers
### Creating Transfers with Storage Providers
To create a new transfer using storage providers:
1. Navigate to the **Transfers** section and click **New Transfer**
2. Fill in the transfer name and schedule as usual
3. In the Source section, select **Provider** and choose from the dropdown
- Only providers of appropriate types will be shown
- You'll see only providers you've created (unless you're an admin)
4. In the Destination section, also select a provider
5. Configure other transfer settings as needed (paths, file patterns, etc.)
6. Click **Save** to create the transfer
### Converting Existing Transfers
Existing transfers with embedded credentials can be converted to use storage providers:
1. Edit an existing transfer
2. In the Source section, click **Convert to Provider**
- This will create a new storage provider using the embedded credentials
- The provider will be named based on the transfer name
3. Do the same for the Destination section if needed
4. Click **Save** to update the transfer
## Provider Type Reference
### SFTP Configuration
Required fields:
- **Host**: The hostname or IP address of the SFTP server
- **Port**: Server port (usually 22)
- **Username**: Your SFTP username
- **Authentication Method**: Password or Key File
- **Password**: Your SFTP password (if using password authentication)
- **Key File**: Path to SSH private key file (if using key authentication)
Optional fields:
- **Key File Password**: Password for the key file (if the key is password-protected)
Example configuration:
```
Name: Company SFTP Server
Type: SFTP
Host: sftp.example.com
Port: 22
Username: user123
Authentication: Password
Password: ********
```
### S3 Configuration
Required fields:
- **Access Key**: Your S3 access key ID
- **Secret Key**: Your S3 secret access key
- **Bucket**: The S3 bucket name
Optional fields:
- **Region**: The AWS region (e.g., us-east-1)
- **Endpoint**: Server URL for S3-compatible services (leave blank for AWS S3)
Example configuration:
```
Name: Analytics Data Bucket
Type: S3
Access Key: AKIAIOSFODNN7EXAMPLE
Secret Key: ********
Region: us-west-2
Bucket: data-analytics-bucket
```
### OneDrive Configuration
Required fields:
- **Client ID**: Your Microsoft application client ID
- **Client Secret**: Your Microsoft application client secret
- **Refresh Token**: OAuth refresh token for authentication
Optional fields:
- **Drive ID**: Specific drive ID (for accessing shared or team drives)
Example configuration:
```
Name: Marketing OneDrive
Type: OneDrive
Client ID: 12345678-1234-1234-1234-123456789012
Client Secret: ********
Refresh Token: ********
```
### Google Drive Configuration
Required fields:
- **Client ID**: Your Google API client ID
- **Client Secret**: Your Google API client secret
- **Refresh Token**: OAuth refresh token for authentication
Optional fields:
- **Team Drive**: Team drive ID (for accessing shared drives)
Example configuration:
```
Name: Sales Team Drive
Type: Google Drive
Client ID: 123456789012-abcdefghijklmnopqrstuvwxyz.apps.googleusercontent.com
Client Secret: ********
Refresh Token: ********
Team Drive: 0ABCDEFGhijklMNOPQrstuvwxyz
```
### FTP Configuration
Required fields:
- **Host**: The hostname or IP address of the FTP server
- **Port**: Server port (usually 21)
- **Username**: Your FTP username
- **Password**: Your FTP password
Optional fields:
- **Passive Mode**: Enable/disable passive mode (default: enabled)
Example configuration:
```
Name: Legacy FTP Server
Type: FTP
Host: ftp.example.com
Port: 21
Username: ftpuser
Password: ********
Passive Mode: Enabled
```
### SMB Configuration
Required fields:
- **Host**: The hostname or IP address of the SMB/CIFS server
- **Share**: The share name
- **Username**: Your username
- **Password**: Your password
Optional fields:
- **Domain**: Windows domain (if applicable)
- **Port**: Server port (default: 445)
Example configuration:
```
Name: Finance Share
Type: SMB
Host: fileserver.example.com
Share: finance
Username: jsmith
Password: ********
Domain: EXAMPLE
```
## Troubleshooting
### Common Connection Issues
#### SFTP Connection Problems
- **Authentication Failed**: Verify username and password/key file
- **Host Not Found**: Check hostname and network connectivity
- **Permission Denied**: Ensure the user has proper permissions on the server
- **Connection Timeout**: Check firewall settings and server availability
#### S3 Connection Problems
- **Access Denied**: Verify access key, secret key, and bucket permissions
- **Invalid Region**: Ensure the region matches the bucket's region
- **No Such Bucket**: Verify the bucket name and existence
- **Endpoint Error**: For S3-compatible services, verify the endpoint URL
#### OAuth Provider Issues (OneDrive/Google Drive)
- **Invalid Client**: Verify client ID and secret
- **Token Expired**: Refresh tokens may need to be regenerated
- **Permission Scope**: Ensure the token has appropriate scopes for file access
- **Rate Limiting**: You may be making too many requests in a short period
### Error Messages
Common error messages and their solutions:
| Error Message | Possible Cause | Solution |
|---------------|----------------|----------|
| "Connection refused" | Server is not running or blocked by firewall | Check server status and firewall settings |
| "Authentication failed" | Incorrect credentials | Verify username/password or key file |
| "Invalid access key" | Incorrect or expired AWS credentials | Check your access key ID and regenerate if needed |
| "Permission denied" | Insufficient permissions | Check file/folder permissions on the server |
| "Connection timed out" | Network issue or server unavailable | Check network connectivity and server status |
| "No such file or directory" | Path does not exist | Verify the path exists on the server |
## FAQ
**Q: Can I use the same storage provider for multiple transfers?**
A: Yes, that's one of the main benefits. Create the provider once and use it in as many transfers as needed.
**Q: Can I see the passwords or secret keys I've stored?**
A: No, for security reasons, passwords and secret keys are never displayed after they're saved. You can update them, but you cannot view the existing values.
**Q: What happens if I need to update credentials?**
A: Edit the storage provider and enter the new credentials. All transfers using that provider will automatically use the updated credentials.
**Q: Are my credentials secure?**
A: Yes, all sensitive information is encrypted using AES-256 encryption before being stored in the database.
**Q: Can other users see my storage providers?**
A: No, each user can only see and use their own storage providers unless they have administrator privileges.
**Q: Can I export or import storage providers?**
A: Not currently. For security reasons, credential export is not supported.
**Q: What if I'm not sure if a provider is in use?**
A: When attempting to delete a provider, the system will show you all transfers that use it. You can also see usage information in the provider details.
**Q: Can I test a provider without creating a transfer?**
A: Yes, use the "Test" button on the provider list.