Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-10-19 12:34:29 +02:00
Code Issues Packages Projects Releases Wiki Activity

docs: add combination reference table for metadata strategy and listing cache TTL

Chris Lu committed 2026-03-13 00:12:15 -07:00
1 parent 509bc6acdc
commit 8cbed1b461
1 file changed
+22 -34
Download Patch File Download Diff File Expand all files Collapse all files
Mount-Remote-Storage.md
+22 -34
@@ -41,50 +41,38 @@ With `remote.mount`, you can mount one bucket or any directory in the bucket.
## Metadata Strategies
The `-metadataStrategy` flag controls how metadata is cached when mounting:
Two flags control how metadata is handled:
### Eager (default)
- **`-metadataStrategy`**: `eager` (default) pulls all metadata at mount time; `lazy` skips the upfront pull and fetches file metadata on first access.
- **`-listingCacheTTL`**: `0` (default) disables on-demand directory listing; `>0` enables automatic directory listing from remote, cached for the specified number of seconds.
```
remote.mount -dir=/xxx -remote=cloud1/bucket
remote.mount -dir=/xxx -remote=cloud1/bucket -metadataStrategy=eager
```
These flags can be combined. Here are all four combinations:
Pulls down **all** metadata from the remote storage at mount time. All subsequent metadata operations (listing, stat, etc.) read from the local filer store with no remote API calls.
### Combination Reference
Best for: buckets with a manageable number of objects where you want instant local access.
| `-metadataStrategy` | `-listingCacheTTL` | Mount command | Behavior |
|---|---|---|---|
| `eager` | `0` | `remote.mount -dir=/xxx -remote=cloud1/bucket` | **Full upfront pull.** All metadata is downloaded at mount time. Listings and file access are instant. Use `remote.meta.sync` to refresh stale data. |
| `eager` | `>0` | `remote.mount -dir=/xxx -remote=cloud1/bucket -listingCacheTTL=300` | **Full upfront pull + auto-refreshing listings.** Same as eager, but directory listings automatically refresh from remote after the TTL expires. Keeps listings up-to-date when remote data changes over time without manual `remote.meta.sync`. |
| `lazy` | `0` | `remote.mount -dir=/xxx -remote=cloud1/bucket -metadataStrategy=lazy` | **Fully on-demand.** No upfront cost. File metadata is fetched on first access (stat, read). Directory listings are empty until files are accessed individually. Best for very large buckets where you access specific files by known paths. |
| `lazy` | `>0` | `remote.mount -dir=/xxx -remote=cloud1/bucket -metadataStrategy=lazy -listingCacheTTL=300` | **On-demand files + auto-populating listings.** No upfront cost. Directory listings automatically fetch from remote and cache for the TTL. Individual files are also fetchable on demand. Best for large buckets where you need both `ls` and direct file access without upfront cost. |
### Lazy
### Comparison
```
remote.mount -dir=/xxx -remote=cloud1/bucket -metadataStrategy=lazy
```
Skips the upfront metadata pull. Individual file metadata is fetched on demand when a file is first accessed (stat, read, etc.) and cached locally.
Best for: very large buckets where pulling all metadata upfront would be too slow or expensive.
### Lazy with Directory Listing Cache
```
remote.mount -dir=/xxx -remote=cloud1/bucket -metadataStrategy=lazy -listingCacheTTL=300
```
With `-listingCacheTTL=<seconds>`, directory listings (`ls`, S3 `ListObjects`) will automatically fetch from remote and cache results. The cached listing expires after the specified TTL, at which point the next listing triggers a fresh remote fetch.
This can also be used with eager mounts to keep listings up-to-date when remote data changes over time.
| Strategy | Upfront cost | File access | Directory listing | Stale data |
|----------|-------------|-------------|-------------------|------------|
| Eager | High (pulls all metadata) | Instant | Instant | Use `remote.meta.sync` |
| Lazy | None | First access fetches | Empty without `-listingCacheTTL` | TTL-based refresh |
| Lazy + listingCacheTTL | None | First access fetches | Auto-fetched, cached per TTL | Auto-refreshed |
| | Eager | Eager + TTL | Lazy | Lazy + TTL |
|---|---|---|---|---|
| **Mount time** | Slow (pulls all metadata) | Slow (pulls all metadata) | Instant | Instant |
| **`ls` / `ListObjects`** | Instant (local) | Instant, auto-refreshes | Empty | Auto-fetched from remote |
| **File stat/read** | Instant (local) | Instant (local) | First access fetches | First access fetches |
| **Stale listings** | Manual `remote.meta.sync` | Auto-refreshed per TTL | N/A (empty) | Auto-refreshed per TTL |
| **Remote API calls** | Only at mount + sync | Mount + periodic listing | Per file access | Per file access + periodic listing |
| **Best for** | Small/medium buckets | Buckets with external writes | Sparse access by known path | Large buckets, general use |
`remote.unmount` will drop all local metadata and cached file content.
### Repeatedly Update Metadata
Sometimes the data on the cloud changes and local metadata becomes stale. To `unmount` first and `mount` again can work but is costly, since all data has to be cached again.
For eager mounts without `-listingCacheTTL`, the data on the cloud may change and local metadata becomes stale. To `unmount` first and `mount` again can work but is costly, since all data has to be cached again.
To refresh the metadata changes, you can run this on the mounted directory or any sub directories, e.g.:
```
@@ -95,7 +83,7 @@ This will update local metadata accordingly and still keep file contents that ar
If the data on the cloud changes often, you can create a cronjob to run it. Or you can add this command to the admin scripts defined in `master.toml`, to run it regularly.
With `-listingCacheTTL`, directory listings refresh automatically, reducing the need for manual `remote.meta.sync` for listing purposes.
With `-listingCacheTTL`, directory listings refresh automatically, reducing the need for manual `remote.meta.sync`.
## Write Back Changes to Remote Storage