mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-18 17:55:26 +01:00
91eaabf3715b34be2ccf8a7eccd4d342ca62517e
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
0f0f3811bf |
filer: serve caught-up aggregated subscribers from memory, not a held disk pass (#12674)
* filer: serve caught-up aggregated subscribers from memory, not a held disk pass An aggregated SubscribeMetadata stream that does not take metadata chunks (filer.sync, filer.backup, filer.meta.backup, filer.meta.tail, the filer.remote.* syncs, the S3 gateway's subscriptions, and mounts that predate chunk support) re-ran the persisted-log pass whenever its cursor moved: after every memory hold at the delivery watermark and on every 2 s re-probe. The pass holds at the peers' flush low-watermark. Peers flush on their own one-minute schedules, so on a cluster that keeps writing some peer has always flushed past a caught-up cursor while the slowest has not: the pass held at its first entry, the loop came back to it, and memory was never read. The stream got each change only once the slowest peer's flush covered it, in bursts up to a flush interval late, and every pass listed and re-read the newest log files. When the aggregated ring was already filling at the cursor and evicted nothing after it, memory holds everything the persisted logs could add, since every peer's log reaches the ring through the aggregator. Skip the pass and read memory, held at the delivery watermark as before. The skipped pass proves nothing past the cursor, and the next-day scan for an empty pass does not apply. An empty ring, a ring that starts past the cursor, and a cursor memory just refused still take the pass. Chunk subscribers already skip it for a caught-up cursor: their ref listing stops below the flush hold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: keep a new peer's pre-start history on the disk pass The ring only receives a peer's events from the start of its subscription. A peer followed for the first time after this filer started (a filer added to the cluster, a restarted pod with a new address, or the late-arriving peers bootstrap) starts at its notice time, so what it logged before then is only in the persisted logs. A cursor below that start passed the ring check and skipped the disk pass, losing those events for subscribers that do not take metadata chunks. Record the newest start of a peer subscription that had no resume point and send cursors below it through the disk pass as before. Peers that rejoin with a resume point (a restart under the same address) do not move it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
8d3cb2ed2d |
filer: resume a rejoining peer's meta subscription at its last position (#12279)
* filer: resume a rejoining peer's meta subscription at its last position OnPeerUpdate starts every peer add at the update's arrival time. For filers sharing one store there is no persisted offset to correct it, so a peer that drops out and rejoins (a graceful restart) resubscribes from "now" and the events it logged while its stream was down never reach the aggregated buffer - filer.backup and other meta subscribers silently skip them. Track each peer's delivered-through position (the same timestamp domain as the stream's events and idle-heartbeat claims) and rewind the start of a rejoining subscription to it. First-seen peers keep the given start, and non-shared peers still resume from their persisted offset. * filer: keep the re-add resume point behind a replay-floored event The session's replay floor already pins the persisted offset and the reconnect resume point so an event that outlived its retry budget is redelivered on resubscribe; cap peerResumeTsNs to the same position on session exit so the new re-add resume cannot skip past it either. * filer: stop test aggregators' log-buffer workers * filer: cover peer rejoin resubscribing at the resume position * filer: drop the unreachable replay-floor cap on the resume point Only a peer with its own store can floor an event, and that peer's subscribe start always comes from its persisted offset, never from the re-add resume point. The cap also ran in the old session's defer, after the re-add had already read the value. * filer: say why a re-add may resume from an idle heartbeat The heartbeat comments promised a reconnect never resumes from one, while the re-add resume point does. Spell out the difference and the fence that makes the local-stream claim safe to resume from. * filer: forget a removed peer's resume point after an hour The map held an entry for every address ever seen; with pod IPs that is one per restart. The removal mark now outlives the watermark grace and takes the resume point with it once the retention passes. * filer: advance the resume point only for tracked peers Same rule as the watermarks: a straggler from a dropped session must not recreate the entry. * filer: set the test aggregator's subscribe loop instead of a nil fallback |
||
|
|
6b379badbb |
filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM (#12104)
* filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM Each log_buffer.NewLogBuffer preallocates five 8 MiB windows and spawns loopFlush/loopInterval goroutines that keep them reachable. The weed/filer tests built ~70 filers and aggregators that almost never called ShutdownLogBuffer, pinning ~2.8 GB plus live churn, and the linux/386 job ran out of its ~4 GiB address space inside newSealedBuffers. - MetaAggregator.Shutdown now retires MetaLogBuffer too: the "aggr" buffer's loops previously outlived the aggregator. - newTestFiler and newTestFilerWithStubStore register t.Cleanup shutdowns; the one-off filer/aggregator literals defer theirs. * filer: wait for the aggregated meta-log buffer to stop in Shutdown ShutdownLogBuffer only signals the stop; loopFlush and its pooled windows exit asynchronously. Match Filer.Shutdown's local-buffer sequence so the aggregator returns with the buffer's memory already released. * filer: cancel test filers' flush context before buffer shutdown The stub-store helpers build real NewFiler instances, so the queued shutdown flush retries appendToFile every ~737ms under flushCtx, which nothing ever cancelled — the buffer and its pool stayed alive past each test. Cancelling first lets the flush drop the window and WaitForShutdown confirms loopFlush exited. * filer: wait for test log buffers to drain after shutdown The stub-flush buffers cannot hang, but without WaitForShutdown their loopFlush goroutines release the pooled windows asynchronously — the memory this PR means to free is still counted as live while the next test allocates. |
||
|
|
603a1221a7 |
filer: converge peer metadata replication on restart (#11698)
* filer: skip replayed peer events older than the local entry Replay deleted and re-inserted every received event unconditionally, so a restarted filer that rewound its subscription offset re-applied stale peer writes over newer local state (values reverted, deleted entries came back and pointed at already-vacuumed chunks). Compare the event's TsNs against the local entry's Mtime and skip per-path, so a move event can still remove its source while keeping a newer destination. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: add replay last-writer-wins regression tests Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: stop peer subscriptions and flush offsets on shutdown Filer.Shutdown closed the store while subscription goroutines were still mid-stream; their deferred offset save was never guaranteed to run, so a restart rewound each peer's stream back to the last periodic flush and replayed stale events. MetaAggregator.Shutdown now closes every peer's stop channel and waits for the loops so the saved offset covers what was replayed, and the reconnect sleep is interruptible by the stop channel. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: test that shutdown stops and waits for peer loops Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: order replay by persisted per-path commit timestamps Review feedback found the mtime comparison still lets stale replay win: a local delete removes the entry it would have been compared against, an entry mtime is client-controllable, and the check ran unlocked so a local write could land between it and the store mutation. - Record the event timestamp per mutated path in the store KV on every emitted mutation, plus a delete record for sources of deletes and renames. The record survives the entry, so a replayed create older than a local delete no longer resurrects it; ancestor delete records cover children swept without per-child events. - Replay compares the event timestamp against that record under the same per-path entry locks local RPC mutations hold, falling back to entry mtime only for entries written before records existed. Lookup errors now abort the event instead of being treated as an absent entry. - MetaAggregator.Shutdown marks the aggregator stopped under peerChansLock so a master update cannot start a subscription that outlives the wait, and waits for the loops without the five-second cap now that every blocking stage — signature RPC, bootstrap traversal, retry sleeps, and per-event processing — cancels on the stop channel. * filer: pin peer offset at failed replay, fix commit-record context and key size A replayed event that exhausted its retries (or was interrupted by shutdown) was still followed by advancing the peer watermark, so the persisted offset skipped the unapplied event forever. replicateMetadataChange now reports the outcome: errPeerSubscriptionStopped aborts the subscription so the deferred save keeps the last completed event, and an exhausted retry records a barrier the persisted offset and reconnect resume point cannot pass while the stream keeps flowing. Commit records also failed on two fronts: NotifyUpdateEvent runs after CommitTransaction with the same context, so SQL stores wrote MpTs/MpDl through an already-committed *sql.Tx; and metaPathKey embedded the whole path, overflowing the 766-char name column SQL KV stores use, which the KvPut fallback update then silently dropped. The writes now use a fresh context and a sha256-hashed fixed-size key. * filer: lock HTTP delete handler and replay against ancestor paths The HTTP DeleteHandler called DeleteEntryMetaAndData without the shared entry lock, so a peer replay could still interleave with a local delete and its commit record. It now holds the same lock the gRPC DeleteEntry handler does. Replay also locked only the exact event paths, but its ordering check consults ancestor delete records: a local recursive delete holds only the ancestor's lock, so a stale child create could check before the tombstone landed and then reinsert. lockReplayPaths now locks every ancestor of both event paths in one lexicographic order — parents sort before children, as the rename handler's two-path order already does — while never locking the root, which cannot be deleted and would serialize all replays. * filer: record replay commit before the replacement insert If the same-path update DeleteEntry succeeded but InsertEntry failed, the path sat deleted with no commit record until the retry arrived, and a stale peer create could resurrect it in the gap. The commit record now lands immediately after the delete — without a delete record, since an update is not a delete — and again after the successful insert. * filer: lock replay ancestors shared, not exclusive Exclusive locks on every ancestor serialized all replays under a shared parent — unrelated objects in the same bucket could no longer apply in parallel. Ancestors only need to exclude an exclusive holder (a local delete or rename of the ancestor), so they now take shared locks; a path the event changes stays exclusive even when it is also the other path's ancestor. * filer: order peer bootstrap on the sender's commit timestamp traversePeerMetadata compared localCommitTsNs — a commit-clock value — against the entry's mtime, which a client can set freely: a peer change committed recently with an old mtime was skipped, and a stale entry mtime-stamped in the future overwrote a newer local commit. TraverseBfsMetadataResponse now carries the sender's per-path commit timestamp (recorded MpTs record, entry mtime as the unrecorded fallback). The bootstrap applies an entry only when its commit is newer than the local one and records the applied timestamp, so replayed events order against it — closing the gap between bootstrap content and the event stream. A peer without the field sends 0 and keeps the mtime fallback, preserving rolling-upgrade behavior. * filer: keep failed commit records pending until they land notePathChange ran after the mutation committed and only logged a KvPut failure: a delete whose MpTs/MpDl record never landed left no tombstone, so a stale replayed create could resurrect the path while the store was unreachable. Failed records now stay in metaTsPending and a repair pass retries them until they land. localCommitTsNs, deletedAfterTsNs and CommitTsNs consult the pending record, so ordering holds in the window between the committed mutation and the durable record. A per-path flag bounds the repair to one goroutine, and respawns cover merges landing between its last pass and the flag release. * filer: keep only unwritten work in the pending commit record The pending record merged every obligation into one timestamp and flag, and a later successful write left it untouched: a repair pass could then write an older timestamp over a newer one, and a merged delete inherited the newer change's time, which the ancestor check reads as a newer delete. The record now holds a change and a delete timestamp of its own, both discharged under the record's lock alongside the store writes that cover them, so a repair writes only what is still owed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: fail traversal on a commit-record read error CommitTsNs answered a store read failure with the entry mtime, a value clients can set arbitrarily far ahead; a peer bootstrapping on it would record the minted time and reject real updates below it. The lookup now returns the error so the traversal restarts instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
3a61debaa5 |
filer: rebuild peer metadata subscriptions after a master reconnect (#10648)
* filer: keep the existing peer subscription on a repeated add A cluster node add for a peer that is already followed restarted the subscription, dropping the metadata events between the two runs. * master: tell a connecting client the current cluster membership Cluster node updates are only broadcast to the clients connected at that moment. A filer that lost its master stream while a peer came back never learned about the peer, and stopped replicating its metadata for good. * test: a filer joining the master learns about the filers already there * test: a filer resubscribes to a peer that registered while it was disconnected Runs the reported sequence against real processes: filer2 leaves, filer1 is paused and its master stream is broken, filer2 registers again, and filer1 has to replicate from it after reconnecting. |