From b33efe06323e3b9b9770fe081a2abe233338deeb Mon Sep 17 00:00:00 2001 From: chrislusf Date: Mon, 11 Aug 2025 12:05:07 -0700 Subject: [PATCH] copy --- File-Operations-Quick-Reference.md | 172 +++++++++++++++++++++++++++++ Filer-Commands-and-Operations.md | 30 +++++ Filer-Server-API.md | 123 +++++++++++++++++++++ _Sidebar.md | 1 + 4 files changed, 326 insertions(+) create mode 100644 File-Operations-Quick-Reference.md diff --git a/File-Operations-Quick-Reference.md b/File-Operations-Quick-Reference.md new file mode 100644 index 0000000..62e2e8b --- /dev/null +++ b/File-Operations-Quick-Reference.md @@ -0,0 +1,172 @@ +# File Operations Quick Reference + +This page provides a quick reference for common file operations in SeaweedFS using the HTTP API. + +## Basic File Operations + +### Upload a File + +**Small files (direct upload):** +```bash +# Upload with PUT +curl -X PUT "http://localhost:8888/path/to/file.txt" -d "file content" + +# Upload with POST (multipart) +curl -X POST "http://localhost:8888/path/to/file.txt" -F "file=@localfile.txt" +``` + +### Download a File + +```bash +# Get file content +curl "http://localhost:8888/path/to/file.txt" + +# Download to local file +curl "http://localhost:8888/path/to/file.txt" -o localfile.txt +``` + +### Delete a File + +```bash +# Delete single file +curl -X DELETE "http://localhost:8888/path/to/file.txt" + +# Delete directory (recursive) +curl -X DELETE "http://localhost:8888/path/to/directory/?recursive=true" +``` + +## Advanced File Operations + +### Move/Rename Files + +```bash +# Move file to new location +curl -X POST "http://localhost:8888/new/path/file.txt?mv.from=/old/path/file.txt" + +# Rename file in same directory +curl -X POST "http://localhost:8888/path/to/newname.txt?mv.from=/path/to/oldname.txt" +``` + +### Copy Files ⭐ **New Feature** + +```bash +# Copy file (preserves original) +curl -X POST "http://localhost:8888/backup/file.txt?cp.from=/original/file.txt" + +# Copy with automatic name resolution +curl -X POST "http://localhost:8888/backup/?cp.from=/original/file.txt" +# Result: /backup/file.txt +``` + +### List Directory Contents + +```bash +# List directory +curl -H "Accept: application/json" "http://localhost:8888/path/to/directory/?pretty=y" + +# List with pagination +curl -H "Accept: application/json" "http://localhost:8888/path/?limit=10&lastFileName=somefile.txt" +``` + +### Create Directory + +```bash +# Create empty directory +curl -X POST "http://localhost:8888/path/to/new/directory/" +``` + +## File Metadata Operations + +### Get File Information + +```bash +# Get file metadata +curl -I "http://localhost:8888/path/to/file.txt" +``` + +### File Attributes and Tagging + +```bash +# Set custom attributes +curl -X PUT "http://localhost:8888/path/to/file.txt" \ + -H "Seaweed-Custom-Attribute: value" \ + -F "file=@localfile.txt" + +# Set TTL (time to live) +curl -X POST "http://localhost:8888/path/to/file.txt?ttl=3600" \ + -F "file=@localfile.txt" +``` + +## Response Codes + +| HTTP Code | Operation | Meaning | +| --------- | --------- | ------- | +| 200 OK | GET | File retrieved successfully | +| 201 Created | POST/PUT | File uploaded successfully | +| 204 No Content | DELETE, COPY, MOVE | Operation completed successfully | +| 400 Bad Request | Any | Invalid parameters or unsupported operation | +| 404 Not Found | GET, DELETE | File or directory not found | +| 409 Conflict | POST/PUT | File already exists (when using exclusive creation) | + +## Operation Comparison + +| Operation | Source File | Use Case | Performance | +| --------- | ----------- | -------- | ----------- | +| **Upload** | External → SeaweedFS | Add new files | Network dependent | +| **Download** | SeaweedFS → External | Retrieve files | Network dependent | +| **Move** (`mv.from`) | Deleted | Rename/relocate | Very fast (metadata only) | +| **Copy** (`cp.from`) | Preserved | Backup/duplicate | Depends on file size | +| **Delete** | Deleted | Remove files | Fast | + +## Best Practices + +### When to Use Copy vs Move + +**Use Copy (`cp.from`) when:** +- Creating backups before modifications +- Duplicating configuration templates +- Staging files for testing +- Need to preserve original file + +**Use Move (`mv.from`) when:** +- Renaming files +- Reorganizing file structure +- Moving files between directories +- Don't need original file + +### Performance Tips + +1. **Batch operations**: Group multiple operations when possible +2. **Small files**: Use direct PUT for files < 1MB +3. **Large files**: POST with multipart automatically chunks files +4. **Copy operations**: Server-side copy is much faster than download + upload +5. **Directory operations**: Plan directory structure to minimize reorganization + +### Error Handling + +```bash +# Check operation status +if curl -X POST "http://localhost:8888/dest?cp.from=/src" -w "%{http_code}" -o /dev/null -s | grep -q "204"; then + echo "Copy successful" +else + echo "Copy failed" +fi +``` + +## Related Documentation + +- [Filer Server API](Filer-Server-API.md) - Complete API reference +- [Filer Commands and Operations](Filer-Commands-and-Operations.md) - Command-line tools +- [Getting Started](Getting-Started.md) - Installation and setup +- [FAQ](FAQ.md) - Common questions and answers + +## Examples Repository + +For more complex examples and use cases, see: +- [SeaweedFS Examples](https://github.com/seaweedfs/seaweedfs/tree/master/examples) (if available) +- Community-contributed scripts and tools +- Language-specific client libraries + +--- + +💡 **Tip**: Use the `?pretty=y` parameter with JSON responses to get formatted output for easier reading during development and testing. diff --git a/Filer-Commands-and-Operations.md b/Filer-Commands-and-Operations.md index 68b2169..7085a08 100644 --- a/Filer-Commands-and-Operations.md +++ b/Filer-Commands-and-Operations.md @@ -23,6 +23,36 @@ The above `weed copy` command is very efficient. It will contact the master serv This put very little loads on filer and the master server. Data is only transmitted between the local machine and the volume server. +## Copy files within Filer + +SeaweedFS also supports copying files within the filer using the HTTP API. This is useful for creating backups, duplicates, or templates without downloading and re-uploading files. + +### HTTP API Copy + +```bash +# Copy a file to a new location +curl -X POST 'http://localhost:8888/path/to/destination?cp.from=/path/to/source' + +# Copy with automatic name resolution +curl -X POST 'http://localhost:8888/backup/?cp.from=/important/config.json' +# Creates: /backup/config.json +``` + +**Features:** +- **Efficient**: Server-side copy without client data transfer +- **Independent chunks**: Creates new chunk copies (not shared references) +- **Atomic operation**: Either succeeds completely or fails with no partial state +- **Preserves metadata**: File attributes, timestamps, and permissions maintained + +**Comparison of copy methods:** + +| Method | Use Case | Data Transfer | Performance | +| ------ | -------- | ------------- | ----------- | +| `weed filer.copy` | Local files → Filer | Client → Volume Server | Good for initial uploads | +| `cp.from` HTTP API | File → File within Filer | Volume Server → Volume Server | Excellent for server-side copies | + +For more details, see the [Filer Server API documentation](Filer-Server-API.md#copy-files). + ## Register a file on Filer As mentioned above, the (path, fileId, fileSize) can be registered on filer with this gRPC call. diff --git a/Filer-Server-API.md b/Filer-Server-API.md index fced80b..ca63655 100644 --- a/Filer-Server-API.md +++ b/Filer-Server-API.md @@ -237,6 +237,129 @@ Notice that the tag names follow http header key convention, with the first char | ---- | -- | -- | | mv.from | move from one file or directory to another location | Required field | +### Copy files + +SeaweedFS supports efficient file copying using the `cp.from` parameter. This operation creates a complete copy of a file while preserving the original file. + +#### Basic Usage +```bash +# Copy a file to the same directory with a new name +> curl -X POST 'http://localhost:8888/documents/report_backup.pdf?cp.from=/documents/report.pdf' + +# Copy a file to a different directory +> curl -X POST 'http://localhost:8888/backup/important.txt?cp.from=/projects/important.txt' + +# Copy with automatic name resolution (uses source filename) +> curl -X POST 'http://localhost:8888/backup/?cp.from=/projects/important.txt' +# Creates: /backup/important.txt +``` + +#### Parameters +| POST Parameter | Description | Default | +| ---- | -- | -- | +| cp.from | Source file path to copy from. Must be a valid file path. | Required field | + +#### How Copy Works + +**Small Files (< chunk size):** +- Content is stored directly in the filer metadata +- Copy operation duplicates the content bytes +- Very fast, only metadata operation required + +**Large Files (chunked):** +- File data is stored as chunks on volume servers +- Copy operation reads data from source chunks and writes to new chunks +- Creates independent chunk copies (not shared references) +- Preserves file integrity and allows independent deletion + +#### Examples + +**Copy a configuration file:** +```bash +curl -X POST 'http://localhost:8888/config/app.conf.backup?cp.from=/config/app.conf' +``` + +**Copy a large media file:** +```bash +curl -X POST 'http://localhost:8888/media/backup/video.mp4?cp.from=/media/original/video.mp4' +``` + +**Copy with path resolution:** +```bash +# If destination ends with /, uses source filename +curl -X POST 'http://localhost:8888/backup/?cp.from=/important/data.json' +# Result: /backup/data.json +``` + +#### Response Codes + +| HTTP Code | Description | +| ---- | -- | +| 204 No Content | Copy operation completed successfully | +| 400 Bad Request | Invalid source path, missing cp.from parameter, or directory copy attempt | +| 404 Not Found | Source file does not exist | +| 500 Internal Server Error | Volume server error or chunk copy failure | + +#### Performance Characteristics + +- **Small files**: Near-instant (metadata only) +- **Large files**: Proportional to file size (requires data transfer) +- **Network efficient**: Direct volume-to-volume transfer when possible +- **Atomic operation**: Either completes fully or fails with no partial state + +#### Limitations + +- **Directory copying**: Not supported (returns 400 error) +- **Cross-cluster copying**: Limited to same SeaweedFS cluster +- **Concurrent access**: Source file should not be modified during copy + +#### Error Examples + +**Attempting to copy a directory:** +```bash +curl -X POST 'http://localhost:8888/new_folder/?cp.from=/existing_folder/' +# Returns: 400 Bad Request - "directory copying not yet supported" +``` + +**Source file not found:** +```bash +curl -X POST 'http://localhost:8888/copy.txt?cp.from=/nonexistent.txt' +# Returns: 400 Bad Request - "failed to get src entry" +``` + +**Missing cp.from parameter:** +```bash +curl -X POST 'http://localhost:8888/copy.txt' +# Returns: Normal file upload behavior (not a copy operation) +``` + +#### Comparison with Move Operation + +| Operation | Source File | Use Case | Speed | +| ---- | ---- | ---- | ---- | +| `mv.from` | Deleted | Rename/relocate files | Very fast (metadata only) | +| `cp.from` | Preserved | Backup/duplicate files | Depends on file size | + +#### Best Practices + +1. **Backup workflows**: Use copy for creating backups before modifications +2. **Template files**: Copy configuration templates to create new instances +3. **Data migration**: Copy files before cross-cluster transfers +4. **Testing**: Copy production files to staging environments + +#### Shell Command Equivalent + +The SeaweedFS copy operation is similar to: +```bash +# SeaweedFS copy +curl -X POST 'http://localhost:8888/dst?cp.from=/src' + +# Unix equivalent +cp /src /dst +``` + +Note: Directory copying is not currently supported. Only individual files can be copied. + ### Create an empty folder Folders usually are created automatically when uploading a file. To create an empty file, you can use this: ``` diff --git a/_Sidebar.md b/_Sidebar.md index e3f307d..50a172e 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -24,6 +24,7 @@ ### Filer * [[Filer Setup]] * [[Directories and Files]] +* [[File Operations Quick Reference]] * [[Data Structure for Large Files]] * [[Filer Data Encryption]] * [[Filer Commands and Operations]]