Add Migrate Between Filer Stores page

Chris Lu committed 2026-10-23 17:29:22 -07:00
1 parent 3ef22314d7
commit 061448f9cd
4 files changed
+80 -1

No files matched your search

+1 -1
@@ -32,7 +32,7 @@ The Filer Store persists all file metadata and directory information.
#### Switching between different Stores
It is easy to switch between different filer stores.
This is useful if you started with embedded DB and want to move to distributed one or in reverse.
This is useful if you started with embedded DB and want to move to distributed one or in reverse. See [[Migrate Between Filer Stores]] for all options, including online migration.
For example:
```sh
+76
@@ -0,0 +1,76 @@
How to move the filer metadata from one filer store to another, e.g. from embedded LevelDB to a distributed store such as Redis, MySQL or Cassandra, or the other way around.
Only metadata lives in the filer store: directory entries and the file id references pointing to chunks on the volume servers. The file content never moves, so a filer store migration is just a metadata copy plus a `filer.toml` change.
Pick one of the following options.
## Option 1: Offline migration with fs.meta.save / fs.meta.load
Simple and works between any two store types. Requires a short write stop.
1. Make sure no files are created or modified, then save all metadata:
```sh
weed shell
> fs.meta.save -o backup.meta.gz /
```
The output reports the number of directories and files saved.
2. Stop the filer.
3. Update `filer.toml` to the new store (see `weed scaffold -config=filer` for a template) and restart the filer.
4. Load the saved metadata:
```sh
weed shell
> fs.meta.load -concurrency=8 backup.meta.gz
```
* `-concurrency=N` parallelizes file entry creation. Caution: this may not work correctly with some stores such as Redis — use `-concurrency=1` there.
* `-dirPrefix=/buckets/important` loads only a subtree.
5. Verify:
```sh
weed shell
> fs.ls -l /
> fs.verify /
```
The saved `.meta` file also serves as a rollback backup.
## Option 2: Online migration with weed filer.meta.backup
`weed filer.meta.backup` continuously copies metadata changes into a different filer store — see [[Async Filer Metadata Backup]].
1. Create `backup_filer.toml` with the new store enabled, configured the same way as `filer.toml`.
2. Start the backup:
```sh
weed filer.meta.backup -config=/path/to/backup_filer.toml -filer="localhost:8888"
```
On the first run it traverses all existing metadata, then streams new changes. Progress is checkpointed in the new store, so it can be stopped and resumed at any time. `-restart` forces a fresh full copy. `-filerDir`, `-includePrefixes` and `-excludePrefixes` limit the scope.
3. Cut over once the backup has caught up (the log shows the timestamp it has progressed to):
* briefly stop writes to the filer
* point `filer.toml` at the new store and restart the filer
* resume writes
## Option 3: Add a filer on the new embedded store
Since 2.77, filers discover each other via the master and automatically synchronize metadata — see [[Filer Store Replication]]. If the old and new stores are both embedded stores (leveldb, rocksdb, sqlite), just start a second filer configured with the new store, wait for it to catch up, then shut down the old filer.
Caveats:
* This only works between embedded stores. A pair of a shared store (mysql, redis, ...) and an embedded store does not replicate, and neither do two different shared stores.
* Replicated embedded filers are eventually consistent.
* `weed filer.sync` does not work well with multiple `filer.store.id`s.
## Notes
* Only metadata is copied. All chunk references stay valid; nothing on the volume servers changes.
* Filer stores differ in atomic directory renaming, native TTL, and fast bucket deletion — check the [[Filer Stores]] table before picking the target store.
+2
@@ -1,3 +1,5 @@
This page is about copying existing files into SeaweedFS. To migrate the metadata from one filer store to another, see [[Migrate Between Filer Stores]].
If there are already a lot of files, how to move them to the SeaweedFS the most efficient way?
### 1. Setup Filer
+1
@@ -52,6 +52,7 @@
### Advanced Filer Configurations
* [[Migrate to Filer Store]]
* [[Migrate Between Filer Stores]]
* [[Add New Filer Store|Customize Filer Store]]
* [[Filer Store Replication]]
* [[Filer Active Active cross cluster continuous synchronization]]