From 061448f9cda233af5609ac0b0ca0960edaea2aa7 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Fri, 23 Oct 2026 17:29:22 -0700 Subject: [PATCH] Add Migrate Between Filer Stores page --- Filer-Stores.md | 2 +- Migrate-Between-Filer-Stores.md | 76 +++++++++++++++++++++++++++++++++ Migrate-to-Filer-Store.md | 2 + _Sidebar.md | 1 + 4 files changed, 80 insertions(+), 1 deletion(-) create mode 100644 Migrate-Between-Filer-Stores.md diff --git a/Filer-Stores.md b/Filer-Stores.md index 534f54f..a2dc15f 100644 --- a/Filer-Stores.md +++ b/Filer-Stores.md @@ -32,7 +32,7 @@ The Filer Store persists all file metadata and directory information. #### Switching between different Stores It is easy to switch between different filer stores. -This is useful if you started with embedded DB and want to move to distributed one or in reverse. +This is useful if you started with embedded DB and want to move to distributed one or in reverse. See [[Migrate Between Filer Stores]] for all options, including online migration. For example: ```sh diff --git a/Migrate-Between-Filer-Stores.md b/Migrate-Between-Filer-Stores.md new file mode 100644 index 0000000..601439b --- /dev/null +++ b/Migrate-Between-Filer-Stores.md @@ -0,0 +1,76 @@ +How to move the filer metadata from one filer store to another, e.g. from embedded LevelDB to a distributed store such as Redis, MySQL or Cassandra, or the other way around. + +Only metadata lives in the filer store: directory entries and the file id references pointing to chunks on the volume servers. The file content never moves, so a filer store migration is just a metadata copy plus a `filer.toml` change. + +Pick one of the following options. + +## Option 1: Offline migration with fs.meta.save / fs.meta.load + +Simple and works between any two store types. Requires a short write stop. + +1. Make sure no files are created or modified, then save all metadata: + + ```sh + weed shell + > fs.meta.save -o backup.meta.gz / + ``` + + The output reports the number of directories and files saved. + +2. Stop the filer. + +3. Update `filer.toml` to the new store (see `weed scaffold -config=filer` for a template) and restart the filer. + +4. Load the saved metadata: + + ```sh + weed shell + > fs.meta.load -concurrency=8 backup.meta.gz + ``` + + * `-concurrency=N` parallelizes file entry creation. Caution: this may not work correctly with some stores such as Redis — use `-concurrency=1` there. + * `-dirPrefix=/buckets/important` loads only a subtree. + +5. Verify: + + ```sh + weed shell + > fs.ls -l / + > fs.verify / + ``` + + The saved `.meta` file also serves as a rollback backup. + +## Option 2: Online migration with weed filer.meta.backup + +`weed filer.meta.backup` continuously copies metadata changes into a different filer store — see [[Async Filer Metadata Backup]]. + +1. Create `backup_filer.toml` with the new store enabled, configured the same way as `filer.toml`. + +2. Start the backup: + + ```sh + weed filer.meta.backup -config=/path/to/backup_filer.toml -filer="localhost:8888" + ``` + + On the first run it traverses all existing metadata, then streams new changes. Progress is checkpointed in the new store, so it can be stopped and resumed at any time. `-restart` forces a fresh full copy. `-filerDir`, `-includePrefixes` and `-excludePrefixes` limit the scope. + +3. Cut over once the backup has caught up (the log shows the timestamp it has progressed to): + * briefly stop writes to the filer + * point `filer.toml` at the new store and restart the filer + * resume writes + +## Option 3: Add a filer on the new embedded store + +Since 2.77, filers discover each other via the master and automatically synchronize metadata — see [[Filer Store Replication]]. If the old and new stores are both embedded stores (leveldb, rocksdb, sqlite), just start a second filer configured with the new store, wait for it to catch up, then shut down the old filer. + +Caveats: + +* This only works between embedded stores. A pair of a shared store (mysql, redis, ...) and an embedded store does not replicate, and neither do two different shared stores. +* Replicated embedded filers are eventually consistent. +* `weed filer.sync` does not work well with multiple `filer.store.id`s. + +## Notes + +* Only metadata is copied. All chunk references stay valid; nothing on the volume servers changes. +* Filer stores differ in atomic directory renaming, native TTL, and fast bucket deletion — check the [[Filer Stores]] table before picking the target store. diff --git a/Migrate-to-Filer-Store.md b/Migrate-to-Filer-Store.md index ed4075b..cfe00a4 100644 --- a/Migrate-to-Filer-Store.md +++ b/Migrate-to-Filer-Store.md @@ -1,3 +1,5 @@ +This page is about copying existing files into SeaweedFS. To migrate the metadata from one filer store to another, see [[Migrate Between Filer Stores]]. + If there are already a lot of files, how to move them to the SeaweedFS the most efficient way? ### 1. Setup Filer diff --git a/_Sidebar.md b/_Sidebar.md index 6ecb218..8f8048d 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -52,6 +52,7 @@ ### Advanced Filer Configurations * [[Migrate to Filer Store]] +* [[Migrate Between Filer Stores]] * [[Add New Filer Store|Customize Filer Store]] * [[Filer Store Replication]] * [[Filer Active Active cross cluster continuous synchronization]]