mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-31 01:10:41 +01:00
Chris LuandDevin
816f9356b4
* volume.copy: reject only on file-count drift, warn on deleted-count Go warned and kept the copy on any count mismatch. The file count is exact on both sides of a stable source, so a mismatch means a genuinely bad copy and should reject; only the deleted count is a rebuild-time estimate that legitimately drifts. Match the Rust check_copy_counts, which warns solely on the counter that can drift. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.copy: stage the copy and validate before dropping the old replica The destination replica was deleted before the copy even started, so a copy that failed its checks — now including the file-count validation — left the server with no replica at all. The copy now writes under a copying/ staging dir the volume loader never scans, the file-size and count checks run on the staged .idx (LoadCompactNeedleMap gives the same counters the mount would), and only then is the existing replica deleted and the staged files renamed into place. A rejected or failed copy leaves the previous replica untouched, and the .note marker is no longer needed since partial staged files can never be loaded. * volume.copy: remove only the files a failed copy moved into place * volume.copy: keep the .note in-progress marker through the renames A crash between the staged-file renames left a loadable volume without its .vif (writable where the source was read-only) or a partial set the load scan could not find. Write the .note before the renames and land the .vif first so any partial state is discovered and swept, then drop the note before the mount. * volume.copy: serialize copies of the same volume Two copies of the same volume write the same staged copying/ file names and can pass the existing-volume check before the first one mounts. Fail a concurrent second copy as retryable, matching the Rust volume server's volume_copies_in_flight. * volume.copy: restore the replaced replica when the swap fails Once the staged copy validates, the outgoing replica's files are hard-linked into copying/ before DeleteVolume unlinks them. If a later rename or the mount then fails, the preserved files are renamed back and remounted, so a failed swap no longer leaves the node with no replica. The links are removed on success. * volume.copy: clear stale needle-map files around the swap A leftover .ldb/.sdx/.rdb at the destination base is trusted over the fresh .idx the mount loads, and a mount that fails leaves the backends it built. Clear them before the staged files land and again when a failed copy cleans up, gated on no live replica owning them. Also keep <base>.vif on cleanup while an .ecx index shares the basename. * volume.copy: skip the swap when the caller has gone A caller that left during the copy cannot use the result; bail before the swap deletes the replica it would replace. * volume.copy: discard a mounted copy when the result send fails A mounted copy was already announced to the master, so a later failure deletes it through the store to withdraw the announcement before the files go away, rather than unlinking files out from under a registered replica. * volume.copy: keep a vif-less copy discoverable on a split index dir With -dir.idx the staged .idx lands outside the data dir, so a copy whose source has no .vif leaves nothing the volume scan can find and the mount fails after the replica it replaced was already deleted. An empty .vif marks the basename; the loader treats it as absent and writes a real one on mount. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.copy: preserve the replaced replica on its own disk * volume.copy: restore the replaced replica's .vif past a kept incoming one * test: cover the rollback link/restore helpers * volume.copy: preserve the mounted replica's own files for swap rollback The swap backup linked files at the copy's target paths, which are wrong when the replica sits on another disk or keeps its .idx beside the .dat; a failed swap then restored nothing. Link the mounted volume's real paths and require every present file to link before the delete, since a partial backup cannot roll back. Re-check the store at swap time so a replica that mounted mid-copy is replaced rather than overwritten, and drop a .kept incoming .vif that occupies the path the replica's own .vif restores to. * volume.copy: clear a stale .idx a mount would prefer over the copy A load switches dirIdx to the data dir when an .idx sits beside the .dat, so a leftover <data>.idx shadows the fresh index the copy lands in the idx dir, and a surviving <idx>.idx fills in when the source had no index at all. Clearing only the derived backends still let a stale needle map reach the mount. * volume.copy: leave the shared copying/ dir for racing copies Copies of different volumes serialize per volume id but share the disk's copying/ dir; a finishing copy could rmdir it between another copy's MkdirAll and its first staged write, failing that copy with ENOENT. * volume.copy: fail replica preservation on stat errors other than not-found * volume.copy: refuse a mount while that volume is being copied A copy claims the volume id, then waits on the source with the replacement staged. A mount in that window loads the replica whose files the swap then clears; refuse it with Unavailable, retryable once the copy settles. Mirrors the same refusal on the Rust volume server. * volume.copy: refuse configure while that volume is being copied Configure unmounts and remounts; during a copy that already claimed the volume id the remount loads files the swap then unlinks. Refuse with Unavailable, retryable once the copy settles. Mirrors the Rust volume server's same refusal. * volume.copy: leave a still-standing replica's index on a rejected copy * volume.copy: preserve an unmounted replica's files for swap rollback * volume.copy: claim the in-flight id in mount and configure, not just check it * volume.copy: keep an EC-shared .vif when displacing an unmounted replica * volume.copy: restore replaced files over a surviving partial restoreReplacedVolumeFiles deleted a preserved link whenever the base name was still taken, on the assumption a landed file there belonged to the copy. When a moved-file removal fails, that left a surviving partial paired with the restored set's other files — mixed generations in one replica. A leftover at the basename now comes off first so the backup goes back, and the link is kept whenever it cannot go back. * volume.copy: link a repeated replaced path once so rollback cannot delete it * volume.copy: refuse allocation and staged adoption under the copy claim AllocateVolume and adoptStagedVolume still registered a volume while a copy of it held the destination: NewVolume loads the index before registration, and the copy then truncates or replaces files the live replica already opened, leaving it mounted on a stale index. Both paths now claim the volume id like VolumeMount and VolumeConfigure, so registration either wins the claim before the copy starts or sees it and refuses with Unavailable. * volume.copy: serialize reload loads with the copy claim A SIGHUP reload ran loadExistingVolume outside the copy's claim: its .note branch could unlink a copy's files mid-write, and its NewVolume+SetVolume could register a replica between the copy's recheck and its swap, leaving a live volume on a stale index. loadExistingVolume now consults a per-location hook the server installs: a vid whose claim is held is skipped for the next pass; otherwise the load takes the claim and its release drops it after registration, so whichever side claims the id first owns the files for the operation. * volume.copy: let a claim holder's own mount past the load guard VolumeMount, VolumeConfigure, VolumeCopy, and adoptStagedVolume all hold the per-volume claim while calling store.MountVolume, which reaches loadExistingVolume where the load guard saw the caller's own claim and skipped the disk: every mount failed with 'volume not found on disk'. Plumb claimHeld through MountVolume -> LoadVolume -> loadExistingVolume via the new MountVolumeClaimed entry point so an owner's own load proceeds while loads without the claim are still skipped. * volume.copy: keep the strict count check when only deletes drift Deletes append tombstone rows without moving the file count, so a copy whose source only deleted mid-flight still has an exactly comparable file count. Gate the strict check on file-count stability alone; the deleted-count drift is already a warning inside checkCopyCounts. * volume.copy: abort the swap when stale index files cannot be removed A surviving .ldb/.sdx/.idx is trusted over the copied .idx at mount, so a failed removal must stop the copy before the staged files land, not be swallowed. Mirrors the Rust side's remove_partial_files abort. * volume.copy: clear staged leftovers before starting a copy The staging names are fixed per volume, so a crashed copy can leave files under copying/; a retry whose new source lacks an optional file (.vif, or .dat on a remote volume) would rename the previous run's bytes into place. Clear them up front and refuse the copy when they cannot be removed. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>