mini: lift the post-mint allocation hold on first inventory report (#12061)

* mini: lift the post-mint allocation hold on first inventory report

DeferAllocation armed a fixed warmup window whenever the leader minted a
fresh topology id, so the first write on a brand-new weed mini blocked
for the whole window even though no prior cluster state could exist.
The window still has to cover a rebuilt master's veteran servers, which
report on their first post-reconnect heartbeat.

Mini has exactly one possible reporter — the embedded volume server,
whose first heartbeat carries the full inventory — so hold allocation
only until that report is applied. The bound raise and topology-id
freeze checks run while the heartbeat is processed, so a wiped master
state still raises the bound or trips the freeze before the hold lifts.

Standalone weed master and weed server keep the fixed window: volume
servers outside the process can still be on their way.

* master: apply queued topology-id reports before opening the allocation gate

SetTopologyInitPending(false) cleared the pending flag before replaying
topology ids stashed while the local id was still being minted. A
NextVolumeId landing in the drain window saw no gate and no freeze yet.
The post-mint allocation hold used to cover the gap by delaying the
winner past the freeze; once the hold can release on first inventory,
the gate must stay closed until the queued checks have run.

* mini: gate the first-report release on single-master mode

releaseAllocOnFirstReport assumes the embedded volume server is the only
possible reporter of prior cluster state. A mini started with
-master.peers joins other masters, whose own volume servers can still be
on their way when the first report lands, so keep the regular
quiet-pulse hold there.

* master: check reported topology id on empty full reports

A HasNoVolumes heartbeat carries the same LastSeenTopologyId a populated
list does, but the mismatch freeze only ran for non-empty reports. An
EC-only veteran could therefore release the post-mint allocation hold
through NoteInventoryApplied while its stale topology id never reached
CheckReportedTopologyId.

* master: drain the topology-id stash inside the init-gate lock

CheckReportedTopologyId only held reportedTopologyIdsLock while stashing,
so a report in flight during SetTopologyInitPending(false) could land its
freeze after topologyInitPending cleared, letting a NextVolumeId slip
through the gap. The whole check now runs under the lock, and the gate is
released inside the same critical section as the drain.

* mini, master: pin and enforce the single volume-server reporter

isSingleMasterMode only checks master peers; SendHeartbeat still accepted
any volume server, so a late foreign report could carry prior cluster
state after the first-report release dropped the hold. Mini now pins the
embedded volume server's address and the master refuses registration from
any other reporter while ReleaseAllocOnFirstReport is set.

* master: admit the volume-server reporter before applying heartbeat state

* test: cover SendHeartbeat admission ordering

A refused reporter must not freeze growth or raise the id bound, and the
pinned reporter's stale topology id still must.

* mini: re-derive the single-reporter release on every run

The package-level master options survive in-process reruns, so a
multi-master mini following a single-master one must not inherit the
first-report release or a stale reporter pin.

* master: admit addressless first heartbeats under the reporter pin

An unregistered heartbeat with no Ip skipped the single-reporter
admission check but still reached the seen-bound raise and the
topology-id freeze before the delta-heartbeat skip. Admission now runs
for every unregistered heartbeat; an empty address can never equal the
pin.
This commit is contained in:
Chris Lu authored and GitHub committed 2026-10-25 09:25:39 -07:00
1 parent 88b91cccba
commit 53834117a6
7 files changed
+316 -16

No files matched your search

+20 -9
View File
@@ -82,6 +82,15 @@ type MasterOptions struct {
// is cancelled. Used by integration tests and by weed mini; nil for
// standalone weed master.
shutdownCtx context.Context
// releaseAllocOnFirstReport lifts the post-mint allocation hold as soon as
// one full volume-server inventory has been applied. weed mini sets it in
// single-master mode, with singleReporterAddr pinning the embedded volume
// server as the only admitted reporter: any other volume server is refused,
// so the warmup window after a fresh topology id is dead time.
// Standalone weed master / weed server can face late-arriving veterans and
// must keep the window.
releaseAllocOnFirstReport bool
singleReporterAddr string
}
func init() {
@@ -488,14 +497,16 @@ func (m *MasterOptions) toMasterOption(whiteList []string) *weed_server.MasterOp
VacuumIntervalSeconds: *m.vacuumIntervalSeconds,
VacuumDeleteEmptyAfterSeconds: *m.vacuumDeleteEmptyAfterSeconds,
// PulseSeconds: *m.pulseSeconds,
DefaultReplicaPlacement: *m.defaultReplication,
GarbageThreshold: *m.garbageThreshold,
WhiteList: whiteList,
DisableHttp: *m.disableHttp,
MetricsAddress: *m.metricsAddress,
MetricsIntervalSec: *m.metricsIntervalSec,
TelemetryUrl: *m.telemetryUrl,
TelemetryEnabled: *m.telemetryEnabled,
SingleMaster: isSingleMasterMode(*m.peers),
DefaultReplicaPlacement: *m.defaultReplication,
GarbageThreshold: *m.garbageThreshold,
WhiteList: whiteList,
DisableHttp: *m.disableHttp,
MetricsAddress: *m.metricsAddress,
MetricsIntervalSec: *m.metricsIntervalSec,
TelemetryUrl: *m.telemetryUrl,
TelemetryEnabled: *m.telemetryEnabled,
SingleMaster: isSingleMasterMode(*m.peers),
ReleaseAllocOnFirstReport: m.releaseAllocOnFirstReport,
SingleVolumeReporter: m.singleReporterAddr,
}
}
+16
View File
@@ -1564,6 +1564,22 @@ func runMini(cmd *Command, args []string) bool {
// tear those services down too. On Ctrl+C they rely on their own
// OnInterrupt hooks (see grace's LIFO ordering).
miniMasterOptions.shutdownCtx = MiniClusterCtx
// In single-master mode mini's embedded volume server is the only reporter
// of prior cluster state; its first heartbeat already carries the full
// inventory, so the post-mint allocation hold ends there instead of
// running the whole warmup window on every fresh start. The master pins
// the embedded address and refuses any other volume server, keeping the
// single-reporter assumption true. Multi-master minis can field
// late-reporting volume servers and keep the regular hold.
// Both fields are re-derived on every run: the package-level options
// survive in-process reruns, so a multi-master mini after a
// single-master one must not inherit the release or the stale pin.
singleMaster := isSingleMasterMode(*miniMasterOptions.peers)
miniMasterOptions.releaseAllocOnFirstReport = singleMaster
miniMasterOptions.singleReporterAddr = ""
if singleMaster {
miniMasterOptions.singleReporterAddr = util.JoinHostPort(*miniIp, *miniOptions.v.port)
}
miniOptions.v.shutdownCtx = MiniClusterCtx
miniFilerOptions.shutdownCtx = MiniClusterCtx
// Mini is a small/dev setup with short-lived RPCs; cap the filer's