From 5972a145493ef8fbed17aece51b9b6c7e06fb8d2 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Thu, 15 Jan 2026 01:00:37 -0800 Subject: [PATCH] docs: move remote.copy.local documentation to Mount-Remote-Storage page Moved the remote.copy.local command documentation and comparison table from Cache-Remote-Storage.md to Mount-Remote-Storage.md for better organization, as it's more related to mounting and syncing operations. --- Cache-Remote-Storage.md | 53 ----------------------------------------- Mount-Remote-Storage.md | 53 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+), 53 deletions(-) diff --git a/Cache-Remote-Storage.md b/Cache-Remote-Storage.md index ee550d8..263b662 100644 --- a/Cache-Remote-Storage.md +++ b/Cache-Remote-Storage.md @@ -46,59 +46,6 @@ In `weed shell`, checkout `remote.cache` and `remote.uncache`: ``` -## Copy Local-Only Files to Remote - -The `remote.copy.local` command synchronizes local-only files to remote storage. This is useful when you have files that were created locally and need to be backed up to remote storage. - -``` -> help remote.copy.local - remote.copy.local # copy local-only files to remote storage for mounted directories - - # copy all local-only files in a mounted directory to remote - remote.copy.local -dir=/xxx - - # copy with filters - remote.copy.local -dir=/xxx/some/sub/dir -include=*.pdf - remote.copy.local -dir=/xxx/some/sub/dir -exclude=*.txt - remote.copy.local -minSize=1024000 # copy files larger than 100K - remote.copy.local -maxSize=10240000 # copy files smaller than 10MB - - # force update even if remote file exists - remote.copy.local -dir=/xxx -forceUpdate=true - - # dry run to see what would be copied - remote.copy.local -dir=/xxx -dryRun=true - - This command only copies files that don't have remote entries yet. - Files that are already synchronized with remote storage are skipped. - - The actual data copying goes through volume servers in parallel. -``` - -## Command Comparison - -Here's a comparison of the three main remote storage commands: - -| Feature | `remote.cache` | `remote.uncache` | `remote.copy.local` | -|---------|---------------|------------------|---------------------| -| **Purpose** | Download files from remote to local cache | Remove local cache, keep metadata | Upload local-only files to remote | -| **Data Flow** | Remote → Local | Local → (delete) | Local → Remote | -| **When to Use** | Warm up cache, pre-fetch data | Free up local storage | Backup local files to remote | -| **File Selection** | Files in remote storage | Cached files | Local-only files (no remote entry) | -| **Skips Files** | Already cached files | Files not in remote | Files already in remote | -| **Force Option** | N/A | N/A | `-forceUpdate` to overwrite remote | -| **Common Filters** | `-include`, `-exclude`, `-maxSize`, `-maxAge` | `-include`, `-exclude`, `-minSize`, `-minAge` | `-include`, `-exclude`, `-minSize`, `-maxSize` | -| **Dry Run** | `-dryRun` | `-dryRun` | `-dryRun` | -| **Use Case Example** | Cache frequently accessed files | Uncache old/large files to save space | Upload newly created local files | - -### Typical Workflow - -1. **Mount remote storage**: `remote.mount -dir=/buckets/mybucket -remote=s3_1/bucket` -2. **Create local files**: Write files directly to `/buckets/mybucket/` -3. **Backup to remote**: `remote.copy.local -dir=/buckets/mybucket` (uploads local-only files) -4. **Free up space**: `remote.uncache -dir=/buckets/mybucket -minSize=10240000` (remove large files from cache) -5. **Warm up cache**: `remote.cache -dir=/buckets/mybucket -include=*.pdf` (pre-fetch PDFs) - ## Note If you want some more flexibility to decide which file to cache or uncache, please help to send a PR. diff --git a/Mount-Remote-Storage.md b/Mount-Remote-Storage.md index 5138fd5..5ee0428 100644 --- a/Mount-Remote-Storage.md +++ b/Mount-Remote-Storage.md @@ -63,6 +63,59 @@ weed filer.remote.sync -filer=: -dir=xxx ``` The process is designed to be worry-free. It should automatically resume if stopped, and can reconnect automatically. +## Copy Local-Only Files to Remote + +The `remote.copy.local` command synchronizes local-only files to remote storage. This is useful when you have files that were created locally and need to be backed up to remote storage. + +``` +> help remote.copy.local + remote.copy.local # copy local-only files to remote storage for mounted directories + + # copy all local-only files in a mounted directory to remote + remote.copy.local -dir=/xxx + + # copy with filters + remote.copy.local -dir=/xxx/some/sub/dir -include=*.pdf + remote.copy.local -dir=/xxx/some/sub/dir -exclude=*.txt + remote.copy.local -minSize=1024000 # copy files larger than 100K + remote.copy.local -maxSize=10240000 # copy files smaller than 10MB + + # force update even if remote file exists + remote.copy.local -dir=/xxx -forceUpdate=true + + # dry run to see what would be copied + remote.copy.local -dir=/xxx -dryRun=true + + This command only copies files that don't have remote entries yet. + Files that are already synchronized with remote storage are skipped. + + The actual data copying goes through volume servers in parallel. +``` + +## Command Comparison + +Here's a comparison of the three main remote storage commands: + +| Feature | `remote.cache` | `remote.uncache` | `remote.copy.local` | +|---------|---------------|------------------|---------------------| +| **Purpose** | Download files from remote to local cache | Remove local cache, keep metadata | Upload local-only files to remote | +| **Data Flow** | Remote → Local | Local → (delete) | Local → Remote | +| **When to Use** | Warm up cache, pre-fetch data | Free up local storage | Backup local files to remote | +| **File Selection** | Files in remote storage | Cached files | Local-only files (no remote entry) | +| **Skips Files** | Already cached files | Files not in remote | Files already in remote | +| **Force Option** | N/A | N/A | `-forceUpdate` to overwrite remote | +| **Common Filters** | `-include`, `-exclude`, `-maxSize`, `-maxAge` | `-include`, `-exclude`, `-minSize`, `-minAge` | `-include`, `-exclude`, `-minSize`, `-maxSize` | +| **Dry Run** | `-dryRun` | `-dryRun` | `-dryRun` | +| **Use Case Example** | Cache frequently accessed files | Uncache old/large files to save space | Upload newly created local files | + +### Typical Workflow + +1. **Mount remote storage**: `remote.mount -dir=/buckets/mybucket -remote=s3_1/bucket` +2. **Create local files**: Write files directly to `/buckets/mybucket/` +3. **Backup to remote**: `remote.copy.local -dir=/buckets/mybucket` (uploads local-only files) +4. **Free up space**: `remote.uncache -dir=/buckets/mybucket -minSize=10240000` (remove large files from cache) +5. **Warm up cache**: `remote.cache -dir=/buckets/mybucket -include=*.pdf` (pre-fetch PDFs) + ## Unmount a Remote Storage Similarly, running `remote.unmount -dir=xxx` can unmount a remote storage. However, this means all cached data and metadata will be deleted.