Files
seaweedfs/weed/shell
f99cc6ca94 volume.fsck: batch and deduplicate filer entry deletions (#12480)
* volume.fsck: batch and deduplicate filer entry deletions

In volume.fsck with -findMissingChunksInFiler and -reallyDeleteFilerEntries, deleting orphaned filer entries was slow and CPU intensive due to tiny worker channel buffers (128), lack of batch processing, redundant HTTP DELETE requests for multi-chunk files, and lock contention on parent directory tracking.

Optimize by:
- Adding startPurgeBatchWorkers with 4096-buffered channels to prevent producer stalls and batch opportunistic deletes up to 64 items.
- Deduplicating missing chunks per file path during volume checks so that multi-chunk files are queued once.
- Processing batches with in-batch path deduplication and aggregating parent directories before updating purgedDirs in a single lock acquisition.
- Buffering temporary fid file writes (64KB) during the traversal to avoid millions of unbuffered syscalls.

* volume.fsck: report a failed write of the filer fid files

The fid files are written through a buffer now, so the last part of
each one reaches the disk only in the deferred Flush. Its error was
dropped: a full temp disk left a short fid file and the collection
still succeeded, and the volume check would then take the chunks
missing from it for orphans.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: retry a batched path whose earlier delete failed

A path was marked done before its delete ran, so a second copy of it in
the same batch was skipped after a transient failure and the entry
stayed. Only skip paths already deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: keep reporting a failed DELETE response read

The non-verbose path now discards the body without checking the copy,
so a connection cut after the headers went unreported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: keep one purge worker starter and one delete function

startPurgeWorkers and the single-path httpDelete were left only as
wrappers the tests called. Give their names to the batch versions and
adapt the tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 05:03:52 -08:00
..