Commit Graph
2 Commits
Author SHA1 Message Date
Chris LuandClaude Opus 5.5 8e15110b4f filer: wake metadata subscribers only for directories they can match (#12669)
* log_buffer: let a subscriber skip wake-ups for unwanted partition keys

Every append woke every registered subscriber. SetSubscriberWakeFilter lets
a reader sleep through appends whose partition key it would filter out; it
still reads them on its next poll. Flushes and keyless appends wake everyone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer_pb: decide subscription matches from an event's directories

MetadataDirectoryMayMatchSubscription rules out events whose directory no
child path of can match the subscription, and ScanMetadataEventDirectories
reads an event's directory and new parent path without allocating.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: wake subscribers of a move's destination directory

Appends are keyed by the source directory, so a filtered subscriber of the
destination would otherwise only see the move on its next poll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: wake metadata subscribers only for directories they can match

Every metadata event woke and was scanned by every subscriber, so filer CPU
grew with the number of subscribers watching other paths. Readers now
register a directory wake filter, and the per-entry prefilter rules entries
out from their directories before the skeleton scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test that moves into a subscribed prefix pass the prefilter

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 20:53:41 -08:00
Chris Lu 594fc667d5 Cut per-subscriber replay decode and widen replay concurrency (#9917)
* Filter metadata events before unmarshaling them per subscriber

Every subscriber unmarshaled every log entry into a full event just to
run the path filter, and entries carry complete chunk lists, so a fleet
of path-filtered subscribers spends almost all replay CPU materializing
events it then discards. A shallow wire scan now extracts just the
directory, entry names and rename destination into a skeleton event,
feeds the same matcher, and skips the decode for entries the subscriber
cannot match. Any scan surprise (malformed bytes, merged duplicate
message fields) falls back to the full decode, and the unsynced-events
heartbeat keeps firing for skipped entries.

* Raise the legacy replay cap

The cap was sized when every replay pinned a private chunk reader per
source filer. Replays now share decoded chunks, so sixteen needlessly
serializes subscriber catch-up; the expensive part stays bounded by the
cache's load gate.

* Weight concurrent log-chunk loads by size

The flat eight-load gate let eight tiny chunks through as reluctantly as
eight full ones. Charge each load's chunk size against a 128MB in-flight
budget instead: small chunks decode wide open while full-size ones still
serialize enough to cap the transient peak. Oversized weights clamp to
the budget so they can always acquire.

* Propagate heartbeat send failures and reset the skip counter

A failed heartbeat send means the stream is gone, so end the replay
instead of scanning on. A delivered event also resets the skip counter,
keeping the heartbeat cadence relative to the last thing the client
actually received.

* Share the unsynced-events counter across the prefilter and delivery

Two independent counters could starve the heartbeat: alternating drops
reset each side before either reached its threshold. One shared counter
increments on every dropped entry, prefiltered or not, and only an
actual delivery resets it, restoring the original cadence exactly.

* Tighten comments

* Benchmark the subscription match paths

For a thousand-chunk event that the subscriber filters out, the shallow
scan matches in 10us and 9 allocations against 175us and 4031
allocations for the full decode.
2026-06-10 13:08:34 -07:00