mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-24 15:00:43 +02:00
Add Migrate Between Filer Stores page
1 parent
3ef22314d7
commit
061448f9cd
4 files changed
+80
-1
No files matched your search
+1
-1
@@ -32,7 +32,7 @@ The Filer Store persists all file metadata and directory information.
|
||||
|
||||
#### Switching between different Stores
|
||||
It is easy to switch between different filer stores.
|
||||
This is useful if you started with embedded DB and want to move to distributed one or in reverse.
|
||||
This is useful if you started with embedded DB and want to move to distributed one or in reverse. See [[Migrate Between Filer Stores]] for all options, including online migration.
|
||||
|
||||
For example:
|
||||
```sh
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
How to move the filer metadata from one filer store to another, e.g. from embedded LevelDB to a distributed store such as Redis, MySQL or Cassandra, or the other way around.
|
||||
|
||||
Only metadata lives in the filer store: directory entries and the file id references pointing to chunks on the volume servers. The file content never moves, so a filer store migration is just a metadata copy plus a `filer.toml` change.
|
||||
|
||||
Pick one of the following options.
|
||||
|
||||
## Option 1: Offline migration with fs.meta.save / fs.meta.load
|
||||
|
||||
Simple and works between any two store types. Requires a short write stop.
|
||||
|
||||
1. Make sure no files are created or modified, then save all metadata:
|
||||
|
||||
```sh
|
||||
weed shell
|
||||
> fs.meta.save -o backup.meta.gz /
|
||||
```
|
||||
|
||||
The output reports the number of directories and files saved.
|
||||
|
||||
2. Stop the filer.
|
||||
|
||||
3. Update `filer.toml` to the new store (see `weed scaffold -config=filer` for a template) and restart the filer.
|
||||
|
||||
4. Load the saved metadata:
|
||||
|
||||
```sh
|
||||
weed shell
|
||||
> fs.meta.load -concurrency=8 backup.meta.gz
|
||||
```
|
||||
|
||||
* `-concurrency=N` parallelizes file entry creation. Caution: this may not work correctly with some stores such as Redis — use `-concurrency=1` there.
|
||||
* `-dirPrefix=/buckets/important` loads only a subtree.
|
||||
|
||||
5. Verify:
|
||||
|
||||
```sh
|
||||
weed shell
|
||||
> fs.ls -l /
|
||||
> fs.verify /
|
||||
```
|
||||
|
||||
The saved `.meta` file also serves as a rollback backup.
|
||||
|
||||
## Option 2: Online migration with weed filer.meta.backup
|
||||
|
||||
`weed filer.meta.backup` continuously copies metadata changes into a different filer store — see [[Async Filer Metadata Backup]].
|
||||
|
||||
1. Create `backup_filer.toml` with the new store enabled, configured the same way as `filer.toml`.
|
||||
|
||||
2. Start the backup:
|
||||
|
||||
```sh
|
||||
weed filer.meta.backup -config=/path/to/backup_filer.toml -filer="localhost:8888"
|
||||
```
|
||||
|
||||
On the first run it traverses all existing metadata, then streams new changes. Progress is checkpointed in the new store, so it can be stopped and resumed at any time. `-restart` forces a fresh full copy. `-filerDir`, `-includePrefixes` and `-excludePrefixes` limit the scope.
|
||||
|
||||
3. Cut over once the backup has caught up (the log shows the timestamp it has progressed to):
|
||||
* briefly stop writes to the filer
|
||||
* point `filer.toml` at the new store and restart the filer
|
||||
* resume writes
|
||||
|
||||
## Option 3: Add a filer on the new embedded store
|
||||
|
||||
Since 2.77, filers discover each other via the master and automatically synchronize metadata — see [[Filer Store Replication]]. If the old and new stores are both embedded stores (leveldb, rocksdb, sqlite), just start a second filer configured with the new store, wait for it to catch up, then shut down the old filer.
|
||||
|
||||
Caveats:
|
||||
|
||||
* This only works between embedded stores. A pair of a shared store (mysql, redis, ...) and an embedded store does not replicate, and neither do two different shared stores.
|
||||
* Replicated embedded filers are eventually consistent.
|
||||
* `weed filer.sync` does not work well with multiple `filer.store.id`s.
|
||||
|
||||
## Notes
|
||||
|
||||
* Only metadata is copied. All chunk references stay valid; nothing on the volume servers changes.
|
||||
* Filer stores differ in atomic directory renaming, native TTL, and fast bucket deletion — check the [[Filer Stores]] table before picking the target store.
|
||||
@@ -1,3 +1,5 @@
|
||||
This page is about copying existing files into SeaweedFS. To migrate the metadata from one filer store to another, see [[Migrate Between Filer Stores]].
|
||||
|
||||
If there are already a lot of files, how to move them to the SeaweedFS the most efficient way?
|
||||
|
||||
### 1. Setup Filer
|
||||
|
||||
+1
@@ -52,6 +52,7 @@
|
||||
|
||||
### Advanced Filer Configurations
|
||||
* [[Migrate to Filer Store]]
|
||||
* [[Migrate Between Filer Stores]]
|
||||
* [[Add New Filer Store|Customize Filer Store]]
|
||||
* [[Filer Store Replication]]
|
||||
* [[Filer Active Active cross cluster continuous synchronization]]
|
||||
|
||||
Reference in new issue
Block a user