diff --git a/Mount-Remote-Storage.md b/Mount-Remote-Storage.md index 631484a..3eae75f 100644 --- a/Mount-Remote-Storage.md +++ b/Mount-Remote-Storage.md @@ -82,22 +82,6 @@ For **on-demand, batch synchronization** of local-only files, use the `remote.co - You want to run periodic batch backups via cron - You need fine-grained control over which files to sync (using filters) -### Comparison: `filer.remote.sync` vs `remote.copy.local` - -| Feature | `weed filer.remote.sync` | `remote.copy.local` | -|---------|-------------------------|---------------------| -| **Type** | Long-running daemon process | One-time command | -| **Synchronization** | Continuous, real-time | On-demand, batch | -| **Trigger** | Automatic (on file changes) | Manual or cron-based | -| **File Selection** | All changes in directory | Selective (with filters) | -| **Use Case** | Active write workloads | Batch backups, recovery | -| **Setup** | Requires separate process | Run in weed shell | -| **Filters** | No (syncs everything) | Yes (`-include`, `-exclude`, size filters) | -| **Dry Run** | No | Yes (`-dryRun`) | -| **Force Update** | Automatic | Optional (`-forceUpdate`) | -| **Resource Usage** | Continuous (low overhead) | Temporary (during execution) | -| **Best For** | Production environments | Maintenance, recovery, scheduled backups | - ## Copy Local-Only Files to Remote @@ -130,27 +114,49 @@ The `remote.copy.local` command synchronizes local-only files to remote storage. ## Command Comparison -Here's a comparison of the three main remote storage commands: +Here's a comprehensive comparison of remote storage synchronization and caching commands: + +### Synchronization Commands (Local ↔ Remote) + +| Feature | `weed filer.remote.sync` | `remote.copy.local` | +|---------|-------------------------|---------------------| +| **Type** | Long-running daemon | One-time shell command | +| **Purpose** | Continuous sync of all changes | Upload local-only files to remote | +| **Data Flow** | Bidirectional (Local ↔ Remote) | Local → Remote | +| **Trigger** | Automatic (on file changes) | Manual or cron-based | +| **Synchronization** | Real-time, continuous | On-demand, batch | +| **File Selection** | All changes in directory | Local-only files (no remote entry) | +| **Filters** | No (syncs everything) | Yes (`-include`, `-exclude`, size filters) | +| **Dry Run** | No | Yes (`-dryRun`) | +| **Force Update** | Automatic | Optional (`-forceUpdate`) | +| **Setup** | Separate process | Run in weed shell | +| **Resource Usage** | Continuous (low overhead) | Temporary (during execution) | +| **Best For** | Production with active writes | Maintenance, recovery, scheduled backups | + +### Caching Commands (Remote ↔ Local Cache) + +| Feature | `remote.cache` | `remote.uncache` | +|---------|---------------|------------------| +| **Type** | One-time shell command | One-time shell command | +| **Purpose** | Download files from remote to local cache | Remove local cache, keep metadata | +| **Data Flow** | Remote → Local | Local → (delete) | +| **When to Use** | Warm up cache, pre-fetch data | Free up local storage | +| **File Selection** | Files in remote storage | Cached files | +| **Skips Files** | Already cached files | Files not in remote | +| **Filters** | `-include`, `-exclude`, `-maxSize`, `-maxAge` | `-include`, `-exclude`, `-minSize`, `-minAge` | +| **Dry Run** | `-dryRun` | `-dryRun` | +| **Use Case Example** | Cache frequently accessed files | Uncache old/large files to save space | -| Feature | `remote.cache` | `remote.uncache` | `remote.copy.local` | -|---------|---------------|------------------|---------------------| -| **Purpose** | Download files from remote to local cache | Remove local cache, keep metadata | Upload local-only files to remote | -| **Data Flow** | Remote → Local | Local → (delete) | Local → Remote | -| **When to Use** | Warm up cache, pre-fetch data | Free up local storage | Backup local files to remote | -| **File Selection** | Files in remote storage | Cached files | Local-only files (no remote entry) | -| **Skips Files** | Already cached files | Files not in remote | Files already in remote | -| **Force Option** | N/A | N/A | `-forceUpdate` to overwrite remote | -| **Common Filters** | `-include`, `-exclude`, `-maxSize`, `-maxAge` | `-include`, `-exclude`, `-minSize`, `-minAge` | `-include`, `-exclude`, `-minSize`, `-maxSize` | -| **Dry Run** | `-dryRun` | `-dryRun` | `-dryRun` | -| **Use Case Example** | Cache frequently accessed files | Uncache old/large files to save space | Upload newly created local files | ### Typical Workflow 1. **Mount remote storage**: `remote.mount -dir=/buckets/mybucket -remote=s3_1/bucket` -2. **Create local files**: Write files directly to `/buckets/mybucket/` -3. **Backup to remote**: `remote.copy.local -dir=/buckets/mybucket` (uploads local-only files) -4. **Free up space**: `remote.uncache -dir=/buckets/mybucket -minSize=10240000` (remove large files from cache) -5. **Warm up cache**: `remote.cache -dir=/buckets/mybucket -include=*.pdf` (pre-fetch PDFs) +2. **Start continuous sync** (optional): `weed filer.remote.sync -filer=localhost:8888 -dir=/buckets/mybucket` +3. **Create local files**: Write files directly to `/buckets/mybucket/` +4. **Batch backup** (if not using filer.remote.sync): `remote.copy.local -dir=/buckets/mybucket` +5. **Free up space**: `remote.uncache -dir=/buckets/mybucket -minSize=10240000` +6. **Warm up cache**: `remote.cache -dir=/buckets/mybucket -include=*.pdf` + ## Unmount a Remote Storage