8 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	f4ef37e752	filer.sync: resubscribe the metadata stream when a failure pins the offset (#11581) ... * filer sink: keep the gRPC status inside wrapped errors %v stringifies the status, so a peer teardown reported as Canceled ("the client connection is closing") reached IsTransientError as plain text and matched nothing: the sync job failed on the first attempt and pinned the offset. %w keeps the status reachable, so the retry runs on a fresh connection once the target is back. * pb: let a consumer drop the metadata stream to force a resubscribe A MetadataProcessor job that exhausts its retries pins the processed watermark so the event replays on the next subscribe — but nothing on the source stream notices a target-side failure, so the replay waited for an unrelated reconnect or a restart. The new Resubscribe channel cancels the stream's context; the Recv loop answers it with ErrResubscribe so the caller's retry loop resubscribes from GetResumeTsNs and replays the pinned events in order. * pb: stop the event retry loop once the stream context is done RetryUntil ignores context, so a subscriber parked on a failing offset write would keep retrying past a resubscribe signal until the sink came back. Stop retrying when the stream is being dropped so the resubscribe takes effect promptly. * filer.sync: signal resubscribe when a job failure pins the offset A job that exhausts its in-job retries leaves the event pinned behind oldestFailedTsNs, replayable only on a reconnect. Closing resubscribeCh on the first recorded failure lets the metadata follower drop the stream so the reconnect replays the pinned events instead of waiting for a process restart (#11572). * filer.sync: wire the resubscribe signal into the follow options filer.sync, filer.remote.sync, and the remote gateway bucket sync all run their subscription inside an outer retry loop, so ErrResubscribe resurfaces as a resubscribe from the persisted watermark. * filer.sync: wait for in-flight jobs before signaling resubscribe * remote sync: never resume past the saved offset when -timeAgo is set * filer.sync: drop events that arrive after the drain signals resubscribe * pb: interrupt the event retry backoff when the stream context ends * filer.sync: stop admitting once a failure pins, and count jobs per timestamp A pinned watermark only released once the processor went fully quiet, so a busy stream could starve the resubscribe — the failed event would wait for an unrelated reconnect anyway, the wait this mechanism exists to remove. The processor now latches stopped when a job fails: admission drops new events (they replay from the pinned watermark after the reconnect), a broadcast releases blocked waiters, and the resubscribe signals as soon as the jobs already in flight drain. A redelivery of an event still in the failure ledger may still run so its success shrinks the replay, but nothing starts once the signal has fired, or it would race the replay it asked for. Dropped events no longer inflate the received counters — an event counts only once admitted, and the replay's own admission counts it. While here: activeJobs keyed by TsNs collapsed events sharing a timestamp, so one completion could empty the map while a same-ts sibling was still running — letting the drain gate and the watermark outrun it. Jobs are now counted per timestamp, and the drain and lazy heap cleanup go through the counts.	2026-10-04 00:06:55 +08:00
3 people	562afa8ec9	filer: resume metadata subscriber from processed watermark on reconnect (#11574) ... * filer: resume metadata subscriber from processed watermark on reconnect * filer: take the reconnect position from GetResumeTsNs verbatim The callback is the subscriber's durable resume point; falling back to StartTsNs when it returns zero can resume from a cursor the log-chunk reader advanced past still-pending work. * filer: advance the stream cursor once a retried event recovers RetryForeverOnError resolves the failure inside handleErr, so returning without moving StartTsNs replays work the event already did when the stream reconnects before the next one arrives. * filer: let filtered-progress markers move the processed watermark A marker means the source examined everything up to its timestamp and skipped what did not match the subscription. With a resume callback the marker now reaches the consumer, and AddSyncJob advances the watermark to it once every earlier job finished and no failure pins the offset. Idle filtered stretches no longer rescan on every reconnect, while the guards keep the watermark behind pending or failed work. * filer: unpin the watermark once a failed event completes oldestFailedTsNs was only ever set, so a failure that a replay later fixed still held the resume offset, and every reconnect re-read the same backlog. Track outstanding failures in a set and recompute the pin when the failed event's job finally succeeds. * filer.remote.gateway: resume bucket sync from the processed watermark The bucket-sync subscriber runs the same MetadataProcessor queue as filer.remote.sync; give it the same GetResumeTsNs callback so a reconnect resumes from durably processed work, not the last seen event. * util: treat a peer-sent gRPC Canceled as transient A peer tearing down its end of the transport reports codes.Canceled ("the client connection is closing"), which IsTransientError used to reject: the sync job then failed on the first try and held the offset until a restart. Caller's own cancels are still excluded up front by errors.Is(err, context.Canceled), so only teardown-style statuses take the new branch. * fix: preserve filtered progress and distinguish caller cancellation * filer: bound the failed-event ledger past a persistent outage A destination rejecting every event grew failedTs by one entry per source event for the life of the processor. Past maxFailedSyncEvents the set now collapses to a sticky pin at the smallest failure seen, so the watermark still replays from the oldest failure while memory stays bounded; a restart re-derives the exact set. Also keep a resume-callback consumer's chunk-ref replay filter at the subscribe-time position instead of option.StartTsNs, so a resubscribe does not filter out events whose async processing is still pending. * filer: key the failed-event ledger by event, not just timestamp A success for one event cleared the pin recorded for a different event that shared its TsNs, letting the watermark pass an unresolved failure. The ledger now keys on the event's path identity, so recovery unblocks only the event that actually failed. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-10-03 20:57:50 +08:00
Chris Lu	2ef0e60aeb	filer.sync: export replication lag, event counters, and in-flight jobs (#11069) ... * filer.sync: count received, processed, and failed events and export in-flight jobs The metadata processor admits at most -concurrency jobs and blocks the subscription stream past that, so the backlog lives in the source filer's metadata log and cannot be counted here. What can be measured honestly: events read off the stream, replication outcomes, and worker saturation. in_flight_jobs pinned at the concurrency limit means the sync itself is the bottleneck; near zero means it is caught up or starved by the source. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * filer.sync: export replication lag in seconds Lag is now minus the freshest of the processed watermark and the last idle heartbeat: the watermark stops at the last real event, so a quiet caught-up stream would otherwise show phantom lag. A ticker drives the gauge because the offset callback only fires while events flow and freezes exactly when the workers are saturated. Until the first event or heartbeat the gauge stays unset rather than reporting lag against a zero or stale resume offset. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * filer.sync: track replicated data sizes alongside event counts An event count hides that 32 in-flight jobs can be 32 KB or 300 GB. Byte counters mirror the event counters, and in_flight_bytes pairs with in_flight_jobs. An event's size is the chunk delta - new chunks the old entry does not already have - so deletes, renames, and attribute-only updates count zero and byte rates reflect data movement, not metadata churn. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * grafana: chart the new filer.sync metrics The lag panel reads lag_seconds directly instead of deriving it from sync_offset, and the sync row gains event rate, throughput, and the in-flight jobs and bytes gauges. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * filer.sync: a pinned failure keeps showing as lag An idle heartbeat means the stream is consumed, not that every event replicated. While a permanent failure pins the watermark, letting the heartbeat advance lag_seconds or the sync_offset gauge would report a caught-up stream with an unreplicated event in it, so both now ignore heartbeats until a restart replays the failure. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * filer.sync: in-flight gauges survive subscription retries A subscription retry builds a new processor sharing the gauge children while the old processor's jobs may still be draining, so setting the gauge from either side's local count clobbers the other. Each job now increments and decrements for itself, keeping the total truthful across generations. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw	2026-09-01 10:10:30 -07:00
Chris Lu	652273301e	filer sync: do not advance the sync offset past a failed event (#10424) ... * util: retry transient errors, not just the ones containing "transport" util.Retry only retried when the error string contained "transport", so a plain "read: connection reset by peer" from S3 got zero retries. Classify the error instead: net timeouts, connection resets, and the throttling and overload replies S3 and gRPC return are all worth another attempt, while a cancelled or expired context is not. * filer sync: hold the sync offset behind a failed event A sync job that returned an error was logged and forgotten, and the watermark advanced past it anyway. The offset is the durable resume point, so the event was never replayed: for filer.remote.sync that left the file present locally, absent on the remote, with no RemoteEntry and nothing to retry it. Pin the watermark at the oldest failed event. Later events keep flowing, but the persisted offset stays behind the failure, so a restart replays it.	2026-07-24 10:32:14 -07:00
Chris Lu	c9623007a2	fix(filer.sync): keep sync_offset fresh through filtered-event markers (#9733) ... On a read-only watched path the idle heartbeat keeps sync_offset fresh, but a busy source filer still emits a MaxUnsyncedEvents marker after many filtered events. The marker has a non-nil but empty EventNotification, so the client routed it to the event path, where it advanced no real watermark yet drove offsetFunc to republish the stale processed watermark — regressing the gauge between heartbeats and spiking the derived lag every time a filtered-event burst landed. Route the empty marker through OnIdleHeartbeat like the idle heartbeat so its fresh timestamp keeps the gauge current; it still advances the in-stream resume cursor.	2026-05-28 23:29:59 -07:00
Chris Lu	c2f5db3a02	perf(filer.sync): don't serialize descendants behind dir attribute updates (#9079) ... * perf(filer.sync): don't serialize descendants behind dir attribute updates The MetadataProcessor treated every in-flight directory job as a subtree barrier: any active dir job at /foo forced all file events under /foo to wait, and because the admit loop runs on the single stream.Recv() goroutine, a stalled descendant also stalled the whole gRPC stream. For large directories this turned every attribute-only dir event (mtime / xattr / chmod bumps) into a full-subtree pinch point. Classify dir jobs as barrier (create / delete / rename) vs non-barrier (filer_pb.IsUpdate on a directory — same parent and same name, i.e. an in-place attribute update). Only barrier dirs block descendants and get blocked by ancestor barrier dirs. Non-barrier dir updates still bump the ancestor descendantCount, so an incoming barrier dir on an ancestor still waits for them — preserving the "delete /a waits for in-flight /a/b update" safety. Tests cover the loosened cases and the preserved barriers: non-barrier update doesn't block a file descendant, barrier create still does, barrier delete still waits for in-flight descendants, and a barrier ancestor still waits for a non-barrier descendant update. * fix(filer.sync): serialize same-path barrier dir jobs against concurrent ops Review (Gemini) flagged that pathConflicts had latent same-path gaps that predated this PR but deserve fixing alongside the dir-conflict loosening: two barrier dir jobs at the same path could run concurrently (e.g. create /a and delete /a), and a file job at the same path as an in-flight barrier dir wasn't blocked either. Tighten pathConflicts so that: - an active barrier dir at p blocks every incoming job at p (file, barrier dir, or non-barrier attribute update) — same-path promotions, renames, and delete/create collisions must serialize; - an active file at p blocks incoming files and barrier dirs at p; - non-barrier dir updates at the same path still overlap with each other (attribute bumps are last-writer-wins, intentional). TestDirVsDirConflict and TestFileUnderActiveDirConflict flip their "same path does not conflict" assertions to match. New TestSamePathBarrierSerialization covers all five same-path cases explicitly. * fix(filer.sync): serialize incoming barrier dir against same-path non-barrier update Bug introduced by the previous same-path tightening commit and caught in review (CodeRabbit, critical): a kindNonBarrierDir at /dir1 was not indexed at its own path, so a later kindBarrierDir at /dir1 saw neither activeBarrierDirPaths["/dir1"] nor descendantCount["/dir1"] (the latter only counts strict descendants) and was admitted concurrently with the in-flight attribute update. That violated the "barrier at p serializes all work at p" rule. Track non-barrier dir jobs in a new activeNonBarrierDirPaths map and check it only from the incoming-barrier-dir branch of pathConflicts. The map is deliberately invisible to the ancestor check, so non-barrier updates still don't serialize file descendants — the loosening this PR is about stays intact. Regression test added in TestSamePathBarrierSerialization covers both the admission conflict and the index cleanup on job completion.	2026-04-14 18:34:05 -07:00
Chris Lu	e648c76bcf	go fmt	2026-04-10 17:31:14 -07:00
Chris Lu	29bdbb3c48	filer.sync: replace O(n) conflict check with O(depth) index lookups (#8772) ... * filer.sync: replace O(n) conflict check with O(depth) index lookups The MetadataProcessor.conflictsWith() scanned all active jobs linearly for every new event dispatch. At high concurrency (256-1024), this O(n) scan under the activeJobsLock became a bottleneck that throttled the event dispatch pipeline, negating the benefit of higher -concurrency values. Replace the linear scan with three index maps: - activeFilePaths: O(1) exact file path lookup - activeDirPaths: O(1) directory path lookup per ancestor - descendantCount: O(1) check for active jobs under a directory Conflict check is now O(depth) where depth is the path depth (typically 3-6 levels), constant regardless of active job count. Benchmark confirms ~81ns per check whether there are 32 or 1024 active jobs. Also replace the O(n) watermark scan with minActiveTs tracking so non-oldest job completions are O(1). Ref: #8771 * filer.sync: replace O(n) watermark rescan with min-heap lazy deletion Address review feedback: - Replace minActiveTs O(n) rescan with a tsMinHeap using lazy deletion. Each TsNs is pushed once and popped once, giving O(log n) amortized watermark tracking regardless of completion order. - Fix benchmark to consume conflictsWith result via package-level sink variable to prevent compiler elision. The watermark advancement semantics (conservative, sets to completing job's TsNs) are unchanged from the original code. This is intentionally safe for idempotent replay on restart.	2026-03-25 15:43:25 -07:00