filer: fold small chunks into manifests 256 at a time (#12462)

* filer: fold small chunks into manifests 256 at a time

Every metadata event carries the file's whole chunk list, so a file grown
by small appends re-sent each chunk record on every append until the
list reached ManifestBatch (10000). Chunks under 1 MiB now fold once 256
of them build up; full-size chunks still fold at ManifestBatch, so large
files keep their flat lists and read without a manifest fetch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: merge appended small chunks at the small-chunk fold count

The append path merges small chunks into full-size ones, but only at
1000 of them; the fold now hides small chunks in a manifest at 256, so
the merge would never see enough of them. Merging at the same count
keeps rewriting them into full chunks first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chris LuandClaude Opus 5.5 authored and GitHub committed 2026-11-06 13:41:02 -08:00
1 parent 728b476ba2
commit cdea7e79e0
4 files changed
+161 -13

No files matched your search

@@ -12,7 +12,9 @@ import (
"github.com/seaweedfs/seaweedfs/weed/stats"
)
const MergeChunkMinCount int = 1000
// MergeChunkMinCount matches the small-chunk fold so a merge rewrites them
// before MaybeManifestize hides them in a manifest.
const MergeChunkMinCount int = filer.SmallChunkManifestBatch
func (fs *FilerServer) maybeMergeChunks(ctx context.Context, so *operation.StorageOption, inputChunks []*filer_pb.FileChunk) (mergedChunks []*filer_pb.FileChunk, err error) {
// Don't merge SSE-encrypted chunks to preserve per-chunk metadata