Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-10-24 06:50:40 +02:00
Code Issues Packages Projects Releases Wiki Activity
Page: Migrate Between Filer Stores
Pages
A typical step‐by‐step example AWS CLI with SeaweedFS AWS IAM CLI Actual Users Admin UI OIDC Admin UI Amazon IAM API Amazon S3 API Applications Async Backup Async Filer Metadata Backup Async Replication to Cloud Async Replication to another Filer Azure Blob Storage Authentication Benchmark SeaweedFS as a GlusterFS replacement Benchmarks from jinleileiking Benchmarks Blob Store Architecture Cache Remote Storage Choosing a Filer Store Client Libraries Cloud Drive Architecture Cloud Drive Benefits Cloud Drive Quick Setup Cloud Monitoring Cloud Tier Cluster Plan Day 2 Operations Cluster Plan Inventory Reference Cluster Plan Workflow Components Configure Remote Storage Cryptography and FIPS Compliance Customize Filer Store Data Backup Data Structure for Large Files Deployment to Kubernetes and Minikube Deployment with seaweed up Directories and Files Distributed POSIX Locks Distributing AI Model Files for Multi GPU Loading Docker Compose for S3 Docker Image Registry with SeaweedFS Doris Iceberg Integration Dremio Iceberg Integration DuckDB Iceberg Integration DuckDB Lance Integration EC Bitrot Detection Environment Variables Erasure Coding for warm storage Error reporting to sentry FAQ FIO benchmark FUSE Mount Failover Master Server File Operations Quick Reference Filer Active Active cross cluster continuous synchronization Filer Cassandra Setup Filer Change Data Capture Filer Commands and Operations Filer Data Encryption Filer JWT Use Filer Metadata Events Filer Notification Webhook Filer Operation Serialization Filer Redis Setup Filer Server API Filer Setup Filer Store Replication Filer Stores Filer as a Key Large Value Store Gateway to Remote Object Storage Getting Started HDFS via S3 connector Hadoop Benchmark Hadoop Compatible File System Hardware Helm Chart Recipes Hobbyest Tinkerer scale on premises tutorial Home Iceberg REST Catalog API Iceberg Table Maintenance Image Gateway Independent Benchmarks Kafka to Kafka Gateway to SMQ to SQL Kubernetes Backups and Recovery with K8up Kubernetes ServiceAccount Authentication Lakekeeper Iceberg Integration Lance Maintenance Worker LanceDB Integration Large File Handling Load Command Line Options from a file Master Server API Migrate Between Filer Stores Migrate Maintenance Scripts to Admin Script Plugin Migrate to Filer Store Mount Remote Storage Mount on Windows Mount over the Internet OIDC Integration OLake Go Iceberg Integration Optimization for Many Small Buckets Optimization P2P reading in weed mount POSIX Compliance Path Specific Configuration Path Specific Filer Store Plugin Worker Scheduling PostgreSQL compatible Server weed db Production Setup Pub Sub to SMQ to SQL Quick Start with weed mini Replication RisingWave Iceberg Integration Run Blob Storage on Public Internet Run Trino on SeaweedFS Rust Volume Server S3 API Audit log S3 API Benchmark S3 API FAQ S3 Bucket Policies S3 Bucket Quota S3 CORS S3 Conditional Operations S3 Configuration S3 Credentials S3 Lifecycle Architecture S3 Lifecycle Monitoring S3 Lifecycle Operator Guide S3 Lifecycle Recipes S3 Lifecycle Troubleshooting S3 Lifecycle vs Volume TTL S3 Lifecycle S3 Nginx Proxy S3 Object Lock and Retention S3 Object Versioning S3 Policy Conditions S3 Policy Variables S3 Rate Limiting S3 RenameObject S3 Table Bucket Commands S3 Table Bucket S3 Tables Security SFTP Server SQL Queries on Message Queue SQL Quick Reference SRV Service Discovery Seaweed Message Queue SeaweedFS Architecture SeaweedFS Iceberg Catalog SeaweedFS Java Client SeaweedFS Lance Catalog SeaweedFS in Docker Swarm Security Configuration Security Overview Server Side Encryption SSE C Server Side Encryption SSE KMS Server Side Encryption Server Startup via Systemd Simplest S3 Bucket and User Setup Spark Iceberg Integration Spark Lance Integration Store file with a Time To Live Structured Data Lake with SMQ and SQL Super Large Directories Supported APIs vs Minio System Metrics TUS Resumable Uploads TensorFlow with SeaweedFS Tiered Storage Trino Iceberg Integration UrBackup with SeaweedFS Use Cases Volume Files Structure Volume Management Volume Server API WebDAV Words from SeaweedFS Users Worker fstab and systemd mount nodejs with Seaweed S3 rclone with SeaweedFS restic with SeaweedFS run HBase on SeaweedFS run Spark on SeaweedFS s3cmd with SeaweedFS seaweed up Command Reference weed shell
Clone
1
Migrate Between Filer Stores
Chris Lu edited this page 2026-10-23 17:29:22 -07:00
Table of Contents

How to move the filer metadata from one filer store to another, e.g. from embedded LevelDB to a distributed store such as Redis, MySQL or Cassandra, or the other way around.

Only metadata lives in the filer store: directory entries and the file id references pointing to chunks on the volume servers. The file content never moves, so a filer store migration is just a metadata copy plus a filer.toml change.

Pick one of the following options.

Option 1: Offline migration with fs.meta.save / fs.meta.load

Simple and works between any two store types. Requires a short write stop.

  1. Make sure no files are created or modified, then save all metadata:

    weed shell
> fs.meta.save -o backup.meta.gz /

    The output reports the number of directories and files saved.

  2. Stop the filer.

  3. Update filer.toml to the new store (see weed scaffold -config=filer for a template) and restart the filer.

  4. Load the saved metadata:

    weed shell
> fs.meta.load -concurrency=8 backup.meta.gz
    • -concurrency=N parallelizes file entry creation. Caution: this may not work correctly with some stores such as Redis — use -concurrency=1 there.
    • -dirPrefix=/buckets/important loads only a subtree.

  5. Verify:

    weed shell
> fs.ls -l /
> fs.verify /

    The saved .meta file also serves as a rollback backup.

Option 2: Online migration with weed filer.meta.backup

weed filer.meta.backup continuously copies metadata changes into a different filer store — see Async Filer Metadata Backup.

  1. Create backup_filer.toml with the new store enabled, configured the same way as filer.toml.

  2. Start the backup:

    weed filer.meta.backup -config=/path/to/backup_filer.toml -filer="localhost:8888"

    On the first run it traverses all existing metadata, then streams new changes. Progress is checkpointed in the new store, so it can be stopped and resumed at any time. -restart forces a fresh full copy. -filerDir, -includePrefixes and -excludePrefixes limit the scope.

  3. Cut over once the backup has caught up (the log shows the timestamp it has progressed to):

    • briefly stop writes to the filer
    • point filer.toml at the new store and restart the filer
    • resume writes

Option 3: Add a filer on the new embedded store

Since 2.77, filers discover each other via the master and automatically synchronize metadata — see Filer Store Replication. If the old and new stores are both embedded stores (leveldb, rocksdb, sqlite), just start a second filer configured with the new store, wait for it to catch up, then shut down the old filer.

Caveats:

  • This only works between embedded stores. A pair of a shared store (mysql, redis, ...) and an embedded store does not replicate, and neither do two different shared stores.
  • Replicated embedded filers are eventually consistent.
  • weed filer.sync does not work well with multiple filer.store.ids.

Notes

  • Only metadata is copied. All chunk references stay valid; nothing on the volume servers changes.
  • Filer stores differ in atomic directory renaming, native TTL, and fast bucket deletion — check the Filer Stores table before picking the target store.

Introduction

API

Configuration

Filer

Filer Stores

Management

Advanced Filer Configurations

FUSE Mount

WebDAV

SFTP Server

Cloud Drive

AWS S3 API

S3 Table Bucket

Iceberg Integrations

Lance Integrations

S3 Authentication & IAM

Server-Side Encryption

S3 Client Tools

Machine Learning

HDFS

Replication and Backup

Metadata Change Events

Messaging

Use Cases

Operations

Rust Volume Server

Advanced

Security

Misc Use Case Examples