filer.sync: resubscribe the metadata stream when a failure pins the offset (#11581)

* filer sink: keep the gRPC status inside wrapped errors

%v stringifies the status, so a peer teardown reported as Canceled ("the
client connection is closing") reached IsTransientError as plain text and
matched nothing: the sync job failed on the first attempt and pinned the
offset. %w keeps the status reachable, so the retry runs on a fresh
connection once the target is back.

* pb: let a consumer drop the metadata stream to force a resubscribe

A MetadataProcessor job that exhausts its retries pins the processed
watermark so the event replays on the next subscribe — but nothing on the
source stream notices a target-side failure, so the replay waited for an
unrelated reconnect or a restart. The new Resubscribe channel cancels the
stream's context; the Recv loop answers it with ErrResubscribe so the
caller's retry loop resubscribes from GetResumeTsNs and replays the pinned
events in order.

* pb: stop the event retry loop once the stream context is done

RetryUntil ignores context, so a subscriber parked on a failing offset
write would keep retrying past a resubscribe signal until the sink came
back. Stop retrying when the stream is being dropped so the resubscribe
takes effect promptly.

* filer.sync: signal resubscribe when a job failure pins the offset

A job that exhausts its in-job retries leaves the event pinned behind oldestFailedTsNs, replayable only on a reconnect. Closing resubscribeCh on the first recorded failure lets the metadata follower drop the stream so the reconnect replays the pinned events instead of waiting for a process restart (#11572).

* filer.sync: wire the resubscribe signal into the follow options

filer.sync, filer.remote.sync, and the remote gateway bucket sync all run their subscription inside an outer retry loop, so ErrResubscribe resurfaces as a resubscribe from the persisted watermark.

* filer.sync: wait for in-flight jobs before signaling resubscribe

* remote sync: never resume past the saved offset when -timeAgo is set

* filer.sync: drop events that arrive after the drain signals resubscribe

* pb: interrupt the event retry backoff when the stream context ends

* filer.sync: stop admitting once a failure pins, and count jobs per timestamp

A pinned watermark only released once the processor went fully quiet, so a busy stream could starve the resubscribe — the failed event would wait for an unrelated reconnect anyway, the wait this mechanism exists to remove. The processor now latches stopped when a job fails: admission drops new events (they replay from the pinned watermark after the reconnect), a broadcast releases blocked waiters, and the resubscribe signals as soon as the jobs already in flight drain. A redelivery of an event still in the failure ledger may still run so its success shrinks the replay, but nothing starts once the signal has fired, or it would race the replay it asked for.

Dropped events no longer inflate the received counters — an event counts only once admitted, and the replay's own admission counts it.

While here: activeJobs keyed by TsNs collapsed events sharing a timestamp, so one completion could empty the map while a same-ts sibling was still running — letting the drain gate and the watermark outrun it. Jobs are now counted per timestamp, and the drain and lazy heap cleanup go through the counts.

weed/pb/filer_pb_tail.go

@@ -2,6 +2,7 @@ package pb
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"io"
 	"time"
@@ -49,8 +50,18 @@ type MetadataFollowOption struct {
 	// durably processed watermark while StartTsNs keeps tracking positions the
 	// stream has merely seen.
 	GetResumeTsNs func() int64
+	// Resubscribe, when closed, drops the stream with ErrResubscribe so the
+	// caller's reconnect loop resubscribes from GetResumeTsNs and replays the
+	// events still pinning the processed watermark. Target-side job failures
+	// never surface on this stream, so without it a pinned event waits for an
+	// unrelated source-stream reconnect (or a restart) to be replayed.
+	Resubscribe <-chan struct{}
 }
 
+// ErrResubscribe ends a metadata follow when the consumer asked for the
+// stream to drop so the reconnect replays what the resume watermark pins.
+var ErrResubscribe = errors.New("resubscribe to replay events behind the failed offset")
+
 type ProcessMetadataFunc func(resp *filer_pb.SubscribeMetadataResponse) error
 
 func FollowMetadata(filerAddress ServerAddress, grpcDialOption grpc.DialOption, option *MetadataFollowOption, processEventFn ProcessMetadataFunc) error {
@@ -98,6 +109,16 @@ func makeSubscribeMetadataFunc(option *MetadataFollowOption, processEventFn Proc
 			return fmt.Errorf("subscribe: %w", err)
 		}
 
+		if option.Resubscribe != nil {
+			go func() {
+				select {
+				case <-option.Resubscribe:
+					cancel()
+				case <-ctx.Done():
+				}
+			}()
+		}
+
 		handleErr := func(resp *filer_pb.SubscribeMetadataResponse, err error) {
 			switch option.EventErrorType {
 			case TrivialOnError:
@@ -105,12 +126,21 @@ func makeSubscribeMetadataFunc(option *MetadataFollowOption, processEventFn Proc
 			case FatalOnError:
 				glog.Fatalf("process %v: %v", resp, err)
 			case RetryForeverOnError:
-				util.RetryUntil("followMetaUpdates", func() error {
-					return processEventFn(resp)
-				}, func(err error) bool {
-					glog.Errorf("process %v: %v", resp, err)
-					return true
-				})
+				waitTime := time.Second
+				for ctx.Err() == nil {
+					if err := processEventFn(resp); err == nil {
+						break
+					} else {
+						glog.Errorf("process %v: %v", resp, err)
+					}
+					select {
+					case <-ctx.Done():
+					case <-time.After(waitTime):
+					}
+					if waitTime < util.RetryWaitTime {
+						waitTime += waitTime / 2
+					}
+				}
 			case DontLogError:
 				// pass
 			default:
@@ -200,6 +230,11 @@ func makeSubscribeMetadataFunc(option *MetadataFollowOption, processEventFn Proc
 				return drainPendingRefs()
 			}
 			if listenErr != nil {
+				select {
+				case <-option.Resubscribe:
+					return ErrResubscribe
+				default:
+				}
 				return listenErr
 			}
 

weed/pb/filer_pb_tail_test.go

@@ -2,6 +2,7 @@ package pb
 
 import (
 	"context"
+	"errors"
 	"io"
 	"sync"
 	"sync/atomic"
@@ -437,3 +438,52 @@ func TestFilerSyncReconnectReadsWatermarkEachSubscribe(t *testing.T) {
 		t.Fatalf("expected subscribes at 100 then 300, got %v", sinceNs)
 	}
 }
+
+// blockingFilerClient returns a stream whose Recv parks until the subscribe
+// context ends, standing in for a quiet source stream during a sink outage.
+type blockingFilerClient struct {
+	filer_pb.SeaweedFilerClient
+}
+
+func (c *blockingFilerClient) SubscribeMetadata(ctx context.Context, in *filer_pb.SubscribeMetadataRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[filer_pb.SubscribeMetadataResponse], error) {
+	return &blockingSubscribeStream{ctx: ctx}, nil
+}
+
+type blockingSubscribeStream struct {
+	grpc.ClientStream
+	ctx context.Context
+}
+
+func (s *blockingSubscribeStream) Recv() (*filer_pb.SubscribeMetadataResponse, error) {
+	<-s.ctx.Done()
+	return nil, s.ctx.Err()
+}
+
+// A consumer that pins an event asks for the stream to drop so the reconnect
+// replays it; the follower must end with ErrResubscribe, not sit on Recv.
+func TestFilerSyncResubscribeSignalEndsStream(t *testing.T) {
+	resubscribe := make(chan struct{})
+	option := &MetadataFollowOption{
+		ClientName:     "syncFrom_A_To_B",
+		EventErrorType: DontLogError,
+		Resubscribe:    resubscribe,
+	}
+	fn := makeSubscribeMetadataFunc(option, func(resp *filer_pb.SubscribeMetadataResponse) error {
+		return nil
+	})
+	done := make(chan error, 1)
+	go func() {
+		done <- fn(&blockingFilerClient{})
+	}()
+
+	close(resubscribe)
+
+	select {
+	case err := <-done:
+		if !errors.Is(err, ErrResubscribe) {
+			t.Fatalf("expected ErrResubscribe, got %v", err)
+		}
+	case <-time.After(5 * time.Second):
+		t.Fatal("follower did not end after the resubscribe signal")
+	}
+}