* filer: stop unbounded metadata-log growth on append-only files
Two independent leaks make a filer metadata log grow far faster than the
data it describes. On a cluster whose live logs weigh a few hundred
megabytes, they produced tens of gigabytes of metadata per day.
1. Metadata events embed the whole old and new entry, chunk lists
included. An append-only file — a log or a journal written in small
pieces — therefore re-serializes every chunk it ever had on every
append, so the cost is quadratic in the number of chunks. A
2,700-chunk file emits a couple of megabytes per append. Cap the
carried chunk list at 256 entries and set
metadata_event_chunks_truncated so subscribers that need the content
read the entry from the filer instead of trusting the event.
Subscribers that only track names, attributes, or chunk deltas keep
working unchanged.
2. appendToFile uploads the chunk before committing the filer entry.
When the commit fails — store error, cancelled context, filer
shutting down — the chunk is stranded: no entry references it, so no
later delete can mark it, and neither vacuum nor fsck can see it
because both work from the entry index. The volume server keeps
counting those bytes as live data forever. Release the chunk when
the commit fails.
* filer: keep chunk lists in metadata events
Mount meta caches, peer filers replaying into their own store, and the
filer.sync, filer.backup and filer.meta.backup sinks all store the event's
entry as the file. Dropping the chunk list past a count left those copies
of a large file without content.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: release a metadata log chunk the lookup or commit leaves unreferenced
The flush retries a failed append with a new upload, so a store outage
left one stranded chunk per retry. A failed lookup strands the chunk too.
Use DeleteUncommittedChunks, as the other write paths do: the chunk was
never part of the file, so path chunk-deletion rules do not apply.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a metadata log chunk the failed create stored
CreateEntry inserts a new entry before its parent folders, so a parent
failure returns an error with the entry already referencing the chunk.
Release it only when the store shows no entry, or an entry without it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: count a metadata log piece the failed create stored as appended
The flush retry would append the same bytes after it, and a piece cut
inside a record then breaks replay of that segment. The retry takes the
update path, which does not create parent folders either, so returning
the error only duplicated the piece.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Martin Morez <martin@morez.org>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
