mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-25 14:30:43 +01:00
689ea6ea3e2bb1beeb87f53cfbf1b6747f16c17a
487
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandDevin
|
5e01caa522 |
master: cap parallel vacuums per rack via master.vacuum.max_parallel_per_rack (#11935)
* master: cap parallel vacuums per rack via -maxParallelVacuumPerRack Hosts that run one volume server per disk under a single rack label could not bound host-level vacuum I/O: maxParallelVacuumPerServer limits each process, so every disk on the machine compacted at once. A second quota map keyed on rack id debits alongside the per-server one, so at most N volume copies compact in parallel within a rack. Defaults to 0 (no limit); the flag is wired on master, server, mini and master.follower. * master: resolve a data node's rack key once per vacuum sweep A volume server that unregisters mid-sweep loses its parent node, so the quota credit path calling dn.GetRack() panicked and crashed the master. Rack names are also local to their data center, so same-named racks in different centers shared one quota bucket. Each node's rack key is now captured when the limiters are built and carries the data center id, so checks, debits and credits never touch the live parent pointer. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: reserve vacuum quota per needed copy count, not per copy Each copy of a volume checked the shared server/rack quota independently, so several same-rack replicas all passed against one remaining slot and the debits drove the bucket negative. Needs are now counted per server and per rack first; a volume whose own copies exceed a cap reserves the whole bucket once it is free instead of pending forever. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master.follower: register -maxParallelVacuumPerRack as a flag It was only a fixed default, so passing the flag on the command line failed flag parsing. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: key the rack vacuum quota by rack node identity Joining data center and rack names with "/" still let two different racks share one bucket when the names themselves contained slashes. Key the limiter by the rack node instead, which is unique per rack regardless of naming. * master: keep the tiered-replica vacuum test through the rebase * master: document 0 as no limit for -maxParallelVacuumPerServer The clamped per-server need makes a non-positive cap mean "no limit", matching the new rack flag; say so in the flag help. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: key the rack-key cache by data node identity Node ids are only unique within a rack, so two servers that share an id in different racks would have reused the first server's cached rack and escaped the per-rack vacuum cap. * master: read the per-rack vacuum cap from master.toml -maxParallelVacuumPerRack becomes the master.max_parallel_vacuum_per_rack config key, so the new master limit is set the same way as other master options instead of a flag wired into every command that embeds a master. * master: let master.max_parallel_vacuum_per_server override the flag The released -maxParallelVacuumPerServer flag keeps working, but a value in master.toml now wins when set, matching how master.mdir already overrides -mdir. * master: group the vacuum caps under master.vacuum in master.toml Rename master.max_parallel_vacuum_per_rack to master.vacuum.max_parallel_per_rack and master.max_parallel_vacuum_per_server to master.vacuum.max_parallel_per_server, so the keys sit in one subsystem table like master.volume_growth and master.maintenance. * master: deprecate -maxParallelVacuumPerServer for master.toml The flag still works - and is overridden by master.vacuum.max_parallel_per_server when set - but the help text now points operators at the config key, matching the codebase style of help-text-only deprecation. * topology: read node parent atomically UnlinkChildNode clears a data node's parent under the rack's lock while readers like the vacuum sweep's rack capture reach it with no shared lock, so a torn interface read could crash the master. Store the parent in an atomic.Pointer and nil-guard the detached walk in DataNode.GetTopology. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
168789d5f3 |
master: cap writable volumes per collection via -maxWritableVolumes (#11923)
* master: cap writable volumes per collection via -maxWritableVolumes Some deployments serialize disk writes (e.g. an Unraid array where concurrent writes to multiple disks collapse throughput). With more than one writable volume in a collection, the master's weighted pick spreads assignments over all of them, so several disks get written at once. -maxWritableVolumes=N narrows the pick candidates to the N fullest writable volumes, so a collection's writes concentrate on one (or few) volumes at a time. 0 keeps the current spread-everywhere behavior. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: give master.follower a maxWritableVolumes default * master: cap writable volumes inside the requested location * master: pick the capped volumes without sorting the writable list * master: assert the capped candidate set instead of random pick outcomes * master: describe -maxWritableVolumes as a per-layout cap --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
ac319fe318 |
volume: never compact a volume tiered to remote storage (#11895)
* volume: never compact a volume tiered to remote storage A tiered volume's data backend is the remote object, but compaction reads the local .dat path. When tiering removed the local .dat, every vacuum of a tiered volume with garbage fails with "no such file or directory", and the master's sweep and the worker's vacuum task keep selecting it. When the local .dat was kept, the compaction succeeds against the stale copy and the commit reloads the volume to read the remote object (old layout) through the compacted index, so live needles stop reading. Mirror the Rust volume server (#11545): - CompactByIndex and CompactByVolumeData refuse a volume with a remote file before creating .cpd/.cpx ("volume N is tiered to remote storage, cannot compact"). - CommitCompact refuses under the data-file locks when the volume became tiered during the copy, and removes .cpd/.cpx. - Store.CheckCompactVolume reports no garbage for a tiered volume, so the master's vacuum and volume.vacuum do not select it. - The worker's vacuum detection skips a volume with a remote copy, as balance and EC detection already do. Tests (each fails on master): TestCompactRefusesTieredVolume, TestCommitCompactRefusesVolumeTieredDuringCopy (on master the commit succeeds and a surviving needle no longer reads), TestPickPrimaryReplicaSkipsTieredVolumes. * volume: skip tiered replicas in vacuum selection, keep their real ratio Reporting a zero garbage ratio for a tiered volume hid it from volume.tier.compact, whose preflight reads that ratio while the volume is still remote, and did not exclude it from a vacuum with a zero threshold. Report the real ratio again, as the Rust volume server does, and have the master's vacuum check skip a replica whose volume info names remote storage, whatever the threshold or request. When CommitCompact refuses a volume tiered during the copy, also close the temporary needle map and remove .cpldb: nothing else would, since no later vacuum selects the volume. The tiered-compaction test no longer depends on random needle sizes reaching a 0.3 garbage ratio. * volume tier: do not attach a copy of a .dat a compaction replaced The upload copies the .dat, then records the remote file in the .vif and switches the backend. A compaction commit in between swaps in a compacted .dat and index; the upload then served the old-layout remote object through the new index, so live needles stopped reading. CommitCompact's tiered check cannot see this: the volume is not tiered until the switch. AttachRemoteDatFile records the remote file and switches the backend under the data-file locks, and refuses when the backend is no longer the file that was copied. The upload then removes the unused remote copy. * volume tier: keep a committed .vif entry when only the dir fsync fails AttachRemoteDatFile rolled back the in-memory remote entry on any SaveVolumeInfo error. A NotCrashDurableError comes after the rename, so the .vif on disk already names the copy; with the entry gone, the upload handler deleted that object, and a reload would serve from a missing remote file. Keep the entry and switch to the copy as the .vif says, and still return the error so the caller keeps the local .dat. * vacuum worker: filter tiered replicas instead of skipping the volume A partially tiered volume is the leftover of an interrupted volume.tier.upload; skipping the whole volume left garbage on its local replicas unreclaimed. Per-replica filtering matches the master's batchVacuumVolumeCheck and is safe: a tiered replica refuses both compaction and commit. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
4d06742664 |
Detect and quarantine duplicate volume ids (#11763) (#11828)
* master: detect and quarantine duplicate volume ids across layouts (#11763) * shell: volume.check.duplicate_ids audit command (#11763) * master: persist duplicate-vid quarantine across heartbeats (#11763) * master: quarantine all copies of a duplicated volume id Near-simultaneous registrations let each side quarantine only the other, so arrival order decided which copy stayed writable. A confirmed cross-layout collision means either copy could be the impostor, so flag every colliding copy read-only regardless of registration order. * topology: return a copy from VolumeLayout.Lookup Callers iterating the returned slice raced with registrations and removals replacing the location list in place. * master: release duplicate-vid quarantine when the conflict resolves A cross-node volume.move briefly shows the same id on two nodes under different disk layouts; permanent quarantine would pin a valid target read-only. Keep the first-reported state per quarantined copy and lift the marks once the conflicting copy unregisters, so only a persistent collision stays read-only. * shell: include ec shards in volume.check.duplicate_ids Ec-only volumes share the global id space. Compare shard collections against regular ones while allowing shards and regular copies of the same encoded volume. * master: keep quarantine release current and cover all removal paths Track the latest server-reported read-only flag while quarantined so a release restores the server s true state, and run the release check on collection deletion and data-node removal too, not only explicit volume unregistration. * shell: do not flag ec-only volumes as duplicates An ec-only volume has no regular copy, so its shard collections had nothing to compare against and always read as foreign. * master: quarantine every conflicting layout at the layout level (#11763) - collect all conflicting layouts, not just the first, so a vid spanning three layouts does not leave one writable - mark quarantine on the location list only: folding it into the stored report made the volume digest permanently disagree with the server and forced a full volume list every heartbeat - isAllWritable now honors the location read-only mark so a quarantined copy stays out of writables through resyncs and re-registration - re-check the conflict before keeping marks, and do not clear a node's quarantine while it still holds the vid in another layout - log the full layout keys on both sides of the conflict * master: push duplicate-vid quarantine to the volume server (#11763) A quarantined copy stayed writable on the volume server itself for a client holding an existing file id, since the master-only mark never reached the server's local write path. Quarantine now also calls VolumeMarkReadonly (can_delete so expiry still drains), is re-pushed if the server comes back reporting it writable, and VolumeMarkWritable runs when the conflict resolves and the server had reported writable. * master: retry unenforced quarantine marks and restore server writability on release (#11763) Dial the advertised grpc port, return mark errors, and keep a quarantine entry until the writable push succeeds so a released copy does not stay read-only on the server. * master: treat rp-only layout differences as an in-flight replication change (#11763) volume.configure.replication rolls through nodes one at a time, so a volume legitimately reports the old and new replica placement while it settles. Layouts differing only in rp within the same collection are a supported transition, not a reused id. * shell: group ec shards by collection in the duplicate id audit (#11763) EC shards of one volume legitimately span disk types when DiskTypePrefer spills, so grouping by disk type reports healthy volumes as duplicates. |
||
|
Chris Lu
|
2874bd05fb |
Freeze volume growth on topology id mismatch (#11763) (#11820)
* pb: ResumeVolumeGrowth rpc for operator-driven growth resume (#11763) * master: freeze volume growth on topology id mismatch (#11763) * shell: volume.growth.resume command (#11763) * master: replicate the growth freeze and gate it on veteran heartbeats (#11763) * master: block allocation while the topology id is being minted DeferAllocation only armed after the topology id persisted, so growth requests arriving during init ran ahead of the warmup window. Gate NextVolumeId on an init-pending flag spanning the whole mint. * master: stash reported topology ids until the local id is minted (#11763) * drain stashed topology id reports when the id arrives outside the mint path * persist the growth-freeze flag in the mirror file and raft snapshots * log hashicorp raft marshal and apply errors in applyGrowthFreeze |
||
|
Chris Lu
|
0b3ed4502f |
Design + master mirror of max volume id outside raft (#11763) (#11818)
* design: durable volume-id allocation boundary (#11763) * master: mirror max volume id to a file outside raft (#11763) The allocation bound lived only in raft state, so a master that lost it rebuilt the bound from whatever volume servers were connected and could reissue ids held by offline servers. NextVolumeId now persists the bound to <mdir>/max_volume_id before the raft apply, and both raft implementations persist on apply and recovery so follower mirrors stay current. The file also carries the topology id, which survives raft state cleanup the same way the snapshot recovery did. * master: serialize and fully sync the max volume id mirror (#11763) * master: rewrite the mirror when the topology id changes at the same bound A same-bound raft apply carrying a new topology id was skipped by the lastPersistedMaxVolumeId check, leaving the mirrored topology id empty until the next allocation. Track the mirrored topology id too and rewrite when either value differs. * drop the design doc from the PR stack |
||
|
|
3c6700c310 |
volume balance: stop planning moves onto disk types a server does not have (#11762)
* fix(topology): capacity queries no longer create empty disk-type entries AvailableSpaceFor and CapacityFor called getOrCreateDisk, so asking a node whether it had room for a disk type it lacks created a zeroed DiskUsageCounts entry for that type. Volume growth walks every data center, rack and data node, and ToDataNodeInfo publishes diskUsages, so every node ended up advertising an empty disk (MaxVolumeCount 0, DiskId 0) of every type ever grown: HDD-only servers showed an "ssd" disk and SSD-only servers an "hdd" disk. Read the counters through a non-creating getDisk; a missing entry answers 0, the same numbers as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * fix(admin): don't let an empty disk-type entry shadow a node's real disk UpdateTopology keys disks by node and disk id. The master published an empty entry (no capacity, nothing on it, DiskId 0) for every disk type a capacity check had asked about, and it collided with the node's real disk 0; map iteration order decided which survived. An SSD-only node intermittently showed an empty HDD disk, and an HDD node lost its disk -- and with it its volumes from the volume index, hence "Could not find volume N on source server" in balance detection. IsShadowedEmptyDiskEntry matches an empty entry whose disk id another DiskInfo of the node, one with capacity or data, already uses. UpdateTopology and CountTopologyResources skip it. A lone empty entry is still an empty disk of unknown capacity and is kept, as GetDisksWithEffectiveCapacity expects. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * fix(balance): don't seed balance candidates from shadowed empty disk entries detectForDiskType registered every node with a DiskInfos entry of the type as a zero-volume candidate. An empty entry the master made up for a type the node lacks made that node the least-loaded destination, and its zero max switched the whole disk type to raw-count balancing. The volume server then refused every move: "no space left hdd" / "no space left ssd". Skip entries IsShadowedEmptyDiskEntry matches when seeding candidates. A server that holds volumes of the type but reports no max still enters through the metrics and still triggers the raw-count fallback. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * fix(admin,balance): ignore made-up disk-type entries at any disk id The made-up entry is published with DiskId 0, but it need not collide with a real disk: when location 0 is unavailable (not reported) the node's usable disk is location 1. IsShadowedEmptyDiskEntry required a collision, so such an entry survived both UpdateTopology -- where the unknown-capacity fallback offers it to planners -- and balance detection, which planned moves onto it again. Its disk id cannot tell it apart: a DiskInfo with no volumes, no EC shards and no per-disk max is published with DiskId 0 whichever disk it stands for. IsPhantomDiskTypeEntry (renamed, since nothing needs to be shadowed now) matches an empty entry on any node that has a disk of another type with capacity or data. A real disk looks the same only while it offers no volume slot, when it can take no volume either. A node whose entries are all empty keeps them as disks of unknown capacity, as before. Reported by review on #11762. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * test(balance): assert where the real disks land in the empty-entry test Check through GetNodeDisks that each server's real disk sits at the subtest's disk id before planning. The id-1 case already landed on physical disk 1 (SplitByPhysicalDisk folds all-zero record ids onto the entry's DiskId), so this only makes the setup explicit; on the first revision of the fix it shows the real disk at 1 beside the made-up entry at 0. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * fix(topology,balance,ec): don't publish or trust usage-only disk-type entries ToDataNodeInfo started from the raw disk-usage counters, so a node that was asked for capacity of a disk type it lacks kept advertising an empty DiskInfo for that type even though no such disk was ever reported. Skip usage entries that have no disk child and hold all-zero counters. The same stale entries still misled planners that read DiskInfos directly: the balance plugin's move precondition treated a phantom entry as proof a target has that disk type, and the EC balancer counted it as a matching disk. Filter them with IsPhantomDiskTypeEntry. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
492e700d1f |
Volume grow avg fill threshold (#11725)
* master: add conditional grow by average fill ratio to VolumeGrow gRPC Callers that pre-grow volumes on fill level currently have to list all volumes of a collection and sum sizes themselves, which is expensive on clusters with millions of volumes. The master already tracks each writable volume's effective size in memory, so it can answer the question directly. Add grow_avg_fill_threshold to VolumeGrowRequest: when set (e.g. 0.6), growth only happens once the layout's writable volumes have reached that average fill ratio. A layout with no writable volumes skips the check, since that is the case growth exists to fix. VolumeGrowResponse now reports the observed avg_fill and writable_count whether or not growth was triggered. * master: unit tests for WritableAvgFill and FindVolumeLayout Covers the average-fill computation (mixed fills, no writables, a writable volume without a size-tracking entry yet) and that FindVolumeLayout resolves existing layouts without creating missing collections. * address review: report avg_fill and writable_count for every VolumeGrow The stats were only read when grow_avg_fill_threshold was set, so a request without a threshold got AvgFill=0 and WritableCount=0 even when writable volumes existed. Read the layout stats unconditionally and use the threshold only to decide whether growth happens. * master: scope the grow fill check to the requested target A VolumeGrow aimed at a data center, rack or node must not skip growth because fill elsewhere keeps the layout average below the threshold. Measure only writables serving the target; a target with none still grows. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
41260a55f6 |
master: record writableVolumeCount on HTTP /dir/assign like gRPC Assign (#11716)
* master: extract the periodic per-layout grow decision Move the must-grow and crowded-step decision of the periodic growth scan in ProcessGrowRequest into periodicGrowCount, a pure function of the layout's lastGrowCount, writable and crowded counts, so it can be tested without running the scan loop or growing real volumes. No behavior change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: record writableVolumeCount on HTTP /dir/assign The periodic growth scan (#5781) keeps a layout supplied ahead of demand only when it knows how many writable volumes clients want: lastGrowCount, which gRPC Assign records with SetLastGrowCount. HTTP /dir/assign parsed writableVolumeCount but only used it to size the emergency grow when no writable volume was left, so the scan always saw lastGrowCount 0 for HTTP-only writers. Since #5821 removed the per-assign threshold grow, such a layout only grew by the rack-aware step of 2, and only once every writable volume was crowded. When a batch of volumes filled in lockstep, those 2 took every write and could fill before the next scan, leaving no writable volumes until an assign-triggered emergency grow landed, with 503s for every other assign meanwhile. Record writableVolumeCount on HTTP assigns as gRPC Assign does. Clients that do not send it are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: reject unknown rack/dataNode before recording grow targets An assign naming a rack or data node outside the topology can never be served, but lastGrowCount was recorded first. The periodic growth scan drops the requested placement, so it kept topping the layout up with volumes the requester could not use. Both assign paths now validate the placement like they already validate the data center. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: scope placement existence checks to the requested path A rack in another data center or a data node under another rack passed validation yet could never serve the request, and the grow count was still recorded. The rack and data-node lookups now take the requested parents, matching the filters the placement engine applies. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
0ca484c354 |
vacuum: bound master vacuum RPCs with phase deadlines (#11579)
* vacuum: bound the commit RPC with a phase deadline VacuumVolumeCommit ran on context.Background(), so a volume server that keeps the call pending would hold the topology-wide vacuum guard forever and every later sweep would be skipped. Give the call a deadline scaled like the existing phase waits (one minute per GB of the volume size limit) so a stalled commit ends as an error instead of blocking the sweep; the timeout is a var so tests can shrink it. * vacuum: bound the replica status probe with a phase deadline The VolumeStatus call on replicas that were not compacted also ran on context.Background(), so a stalled replica could pin the sweep the same way a stalled commit can. Give it the same per-phase deadline. * vacuum: bound the cleanup RPC with a phase deadline VacuumVolumeCleanup also ran on context.Background(); a stalled server would keep the sweep worker and the shared vacuum guard pending forever. Give it the same per-phase deadline. * vacuum: let the check and compact phase waits cancel their RPCs The coordinator wait timers fired while the check and compact calls still ran on context.Background(), so the sweep gave up but the RPC goroutine stayed until the server answered, and a compact stream kept writing on the server. Share one deadline context between the wait and the calls so an expired wait actually cancels them. * vacuum: test that a stalled volume server releases the vacuum guard A fake volume server keeps one vacuum-phase RPC pending until the client context is cancelled. Before the phase deadlines, Vacuum never returned and vacuumLockCounter stayed held; now each phase cancels on its deadline and the guard is free for the next request. * volume: stop compaction at the next needle when the client cancels The progress callback only noticed a gone client when a 128 MiB report failed to send, so an aborted VacuumVolumeCompact kept copying for up to a whole interval while the master had already moved on to cleanup. Check the stream context on every needle, the same early return the Rust volume server does with tx.is_closed(). * vacuum: assert the stalled phase RPC is cancelled, not just bypassed The check and compact coordinator waits already returned on timeout before the deadlines existed, so a regression that put the calls back on context.Background() would pass unnoticed. Wait for the fake server to report that the phase RPC context ended. * vacuum: give the stalled-RPC test room to reach the handler The 50ms phase budget starts before goroutine scheduling and the gRPC dial, so a busy test host could expire it before the fake server saw the call. Raise the override to 250ms; the test still finishes in about a second. * vacuum: describe the phase deadline as scaled, not per-GB The formula keeps the exact expression the check and compact waits already used (floor plus one at 1 GiB granularity); it is a backstop, not a per-GB SLO. |
||
|
ihnokim
|
11e8c4c288 |
master: follow heartbeat read-only changes in the layout's replica flag (#11527)
A replica's read-only flag in the volume layout only moved on registration and on volume.mark. A change that arrived in the regular heartbeat updated the node's record, which the writable list follows, but not the layout flag, which the vacuum sweep reads. So the sweep kept trying volumes on a disk that had gone read-only while the server ran, and after a restart it skipped volumes that had since become writable again until the next restart (issue #11516). Apply the reported state to the flag for every changed volume. Only the flag: the writable list stays with EnsureCorrectWritables and its capacity guards. |
||
|
Chris LuandDevin
|
0978e7f833 |
vacuum: keep disk-full read-only volumes reclaimable (#11519)
* storage/topology: keep disk-full read-only volumes vacuumable The vacuum sweep skipped every read-only replica, so a volume that went read-only because its disk filled could never reclaim its garbage — the exact situation compaction exists for. The volume server now reports disk_space_low in VacuumVolumeCheckResponse, and the sweep skips a read-only replica only when the flag is clear. An explicit volumeId vacuum is unaffected: it already bypassed the read-only rule. The field takes number 4: 2 and 3 are downstream-allocated for tombstone retention, keeping the wire merge clean. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * storage: measure vacuum free space against live bytes The pre-compaction space check required the current .dat + .idx size free, which includes the garbage being reclaimed — on a nearly full disk that estimate can never fit, so the volume stayed garbage-bound forever. Measure against the estimated compacted output instead: superblock plus live index entries plus live content bytes, with the existing ten percent buffer unchanged. Mirrors the same check in the Rust volume server. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vacuum: count per-needle framing in the compacted-size estimate The live-bytes estimate covered each live needle's content and index entry but not its .dat framing (header, checksum, timestamp, padding — ~32 bytes on version 3). For small-needle volumes that is more than the 10% headroom, so a disk with space between the estimate and the real output still ran out mid-compaction. Rust side mirrors the same formula. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * storage: report disk_space_low only when it is the sole read-only cause Review feedback (ihnokim, greptile, devin): a volume read-only for low disk space AND an operator mark or I/O quarantine was still eligible for the automatic sweep, rewriting a copy meant to stay protected. The flag now reports only the benign sole-cause case in both servers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: fail closed when the read-only lookup misses in the sweep A heartbeat can drop the volume from the DataNode cache between the location-list copy and VacuumVolumeCheck; a lookup error previously skipped the read-only check entirely. Review feedback (coderabbit). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yi111andChris Lu
|
150a69fe11 |
master: make volume capacity reservation timeout configurable (#11426) (#11497)
* master: make volume capacity reservation timeout configurable (#11426) * master: expire reservations on reads, fix int timeout units - AvailableSpaceForReservation now expires reservations too: a node that is full of reservations is filtered out before TryReserveCapacity can clean them, which stranded expired capacity indefinitely. - Drop TryReserveCapacityWithTimeout: a per-call timeout lets one caller expire another's live reservations, and the Node interface stays stable for implementations outside this tree. - parseReservationTimeout no longer routes integer values through GetDuration, which read them as nanoseconds; bare numbers are seconds. The 5m fallback is now the shared DefaultReservationTimeout. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
a261f90e18 |
vacuum: let the sweep release volumes that stay empty and quiet (#11477)
* vacuum: let the sweep release volumes that stay empty and quiet Vacuuming reclaims bytes but not slots: a fully emptied volume stays registered to its collection forever, and since growth is gated only on slot count a store at 99% free disk can still refuse writes to other collections (#11429). volume.deleteEmpty exists but is manual-only. With -vacuumDeleteEmptyAfterSeconds (or master.vacuumDeleteEmptyAfterSeconds under weed server/mini; default 0, off) the automatic sweep now deletes replica copies that have stayed empty and quiet for that long, the same rule volume.deleteEmpty applies on demand: remote-backed copies are skipped, and every delete carries the volume server's onlyEmpty / onlyGarbage guards so a copy written since the last report is refused rather than removed. Copies that still hold data or were written recently stay; only a volume whose every copy is deleted leaves the sweep's work map, sparing a compaction of bytes that are all deleted. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vacuum: harden empty-volume sweep against partial and racing deletes Review follow-up on #11477: - delete a volume only when every replica copy is a verifiable empty-and-quiet candidate; deleting the empty copy of a volume whose sibling holds live files would silently cut its replica count (greptile P1). - drain the volume out of the writable list before deleting, the same drain the compact pass uses, so PickForWrite stops assigning it and pending writes settle (devin). - bound the VolumeDelete RPC so one stalled server cannot hold the vacuum lock indefinitely (greptile P1, reusing allocateVolumeTimeout). The vid2location panic scenario raised in review does not exist: VolumeLocationList methods are nil-receiver safe and a missing vid just fails enoughCopies, so a partially deleted volume skips compaction instead of crashing the sweep. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vacuum: unregister deleted empty replicas and prune the sweep list A successful VolumeDelete only updates the volume server; the master still tracked the replica and kept it in the sweep's location list for the compaction pass (coderabbit on #11477). Unregister the replica right after its delete succeeds and drop it from the sweep copy, so a partially deleted volume only compacts copies that still exist. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vacuum: pin deleting volumes out of the writable list across heartbeats Review follow-up on #11477 (greptile): DrainAndRemoveFromWritable only removed the volume once; a heartbeat landing between the drain and the replica deletes re-evaluated writability and re-added it, so a client write could reach a replica whose siblings were already gone and leave the volume under-replicated when the last copy refused its onlyEmpty delete. MarkDeleting records the vid in deletingVolumes — checked inside setVolumeWritable so heartbeat, capacity-recovery, and admin re-add paths all hold it out — and UnmarkDeleting releases it once the sweep finishes the copy pass. A partially deleted volume's surviving replicas then return to writable through the normal heartbeat path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vacuum: restore writability when a sweep delete survives Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
56d2f05ccd |
topology: wake the vacuum dispatcher when a worker frees quota (#11436)
* topology: wake the vacuum dispatcher when a worker frees quota The dispatch loop slept a fixed 10s whenever every pending volume was waiting for a per-server quota slot, so a sweep took volumes x 10s regardless of how fast the compactions were. Workers now signal on a buffered channel after crediting quota; the dispatcher waits on it with the 10s sleep kept only as a timeout. * master: add -vacuumIntervalSeconds to tune the automatic sweep interval The 14-minute base interval was a literal inside the refresh loop while every neighbouring vacuum knob was already a flag. Defaults to 840s, unchanged. * topology: keep the 14 minute floor on the vacuum interval A zero-valued MasterOption or a negative -vacuumIntervalSeconds left the sweep sleeping only its jitter, so treat non-positive intervals as the previous default. |
||
|
hsdfatandChris Lu
|
a93a1ab2eb |
fix(volume): return an error instead of 201 when a write lands on no volume (#11397)
* fix(volume): return an error instead of 201 when a write lands on no volume ReplicatedWrite only writes locally when this server holds the volume. For a volume id no server holds, the master lookup returns no locations, so the write went nowhere and the upload still got 201 Created. The same happened for a type=replicate write to a server without the volume, so the primary, or the S3 chunk fan-out, counted a replica that was never written. A server without the volume still forwards the write to the replicas the master lists. When there is nothing to forward to, fail with "volume N not found on host:port". PostHandler returns that as 500, the status the Rust volume server already returns here, and uploaders re-assign on 5xx. Fixes #6609 * volume: reuse Store.HasVolume, drop issue ref from test comment --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
5d5ea18287 |
topology: fix fatal concurrent map read/write on VolumeLayout.crowded (#11216)
SetVolumeCrowded mutated the crowded map under accessLock.RLock(), while GetWritableVolumeCount reads the same map under RLock() on the Assign hot path. Two concurrent RLock holders with one writing and one reading the map triggers a fatal "concurrent map read and map write" that kills the master process (unrecoverable, bypasses recover). Take the write lock in SetVolumeCrowded instead. This event path is a low-frequency single consumer driven by the crowded-volume event loop, and every other mutation of crowded already holds Lock(); setVolumeCrowded takes no nested locks, so there is no deadlock path. The hot readers (GetWritableVolumeCount, CloneWritableVolumes) keep using RLock. Adds a -race regression test that fails (race detected) on the old RLock and passes with the write lock. Fixes #11211 |
||
|
Chris Lu
|
15e4da65f7 |
volume: avoid read-only replica write targets (#11195)
* master: carry replica read-only state in volume lookups * volume: refresh writable replica targets * volume: preserve read-only replicas for deletes * master: propagate read-only delete capability * volume: target delete-capable replicas * volume: honor configured HTTPS for replica deletes * volume: reject insecure delete authorization forwarding * master: broadcast delete capability changes * volume: align Rust replica routing * http: protect credentialed replica redirects * master: preserve digest compatibility for delete capability * volume: propagate read-only state in short heartbeats * volume: report changed short volume state * http: guard TLS client redirects * master: announce mounted volume read-only state * volume: replace changed identity deltas * master: replace incremental volume layouts in order * master: keep moved volume lookup available * volume: announce read-only mounts |
||
|
yanglongwei
|
eb717199d0 |
master: delete replica_placement_mismatch labels when volumes leave topology (#11062)
* master: delete replica_placement_mismatch labels when volumes leave topology Fixes #10804. Setting the gauge to 0 left stale Prometheus time series that grew unbounded with volume churn; remove the label set on unregister instead. * master: delete replica_placement_mismatch only after last placement leaves Unconditional DeleteLabelValues on UnRegisterVolumeLayout dropped the series while other data nodes still held the volume, hiding under-replication until the next collect cycle. Delete only when Lookup is empty, and cover the two-copy case in a regression test. |
||
|
Chris LuandDevin
|
cc281dabc9 |
master: keep new volumes and writes off servers in maintenance mode (#11147)
* master: keep new volumes and writes off servers in maintenance mode The master recorded a volume server's maintenance flag from the heartbeat but never consulted it. A server in maintenance (#7977) is being drained, yet the master kept creating volumes on it whenever it had free slots and kept handing out its volumes for writes. Nothing on the volume server blocks plain HTTP uploads either, so "read-only mode" was only a name. Volume growth: a data node in maintenance mode reports zero free slots through AvailableSpaceFor, which takes it out of every candidate list, feasibility count and capacity reservation. Its slots still roll up into its rack and data center, so the random offset drawn from those totals for an other-rack or other-DC replica could land in space the walk then skips and fail with "No free volume slot found!" while siblings had room; the walk now folds the offset into the space that is actually eligible. This also covers the pre-existing case of an over-committed sibling. Assignment: a replica on a server in maintenance mode is treated like a read-only replica in isAllWritable, so its volume leaves the writable list and returns when the flag clears. Topology.SetDataNodeMaintenanceMode re-evaluates the node's volumes on every change, since heartbeats are digest-based and a full volume list may not follow for a long time. Reads and lookups are untouched. The flag moves to an atomic so the assign and growth paths can read it without the node lock. Heartbeat: the Go volume server sent its state only when it changed, so a master elected while a server sat in maintenance never learned about it. The state now rides along on every heartbeat, as the Rust server already does; the master's compare is an atomic swap, and only a change does work. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: hold maintenance mode through vacuum commit and mark-writable SetVolumeAvailable and SetVolumeWritable put a volume back on the writable list on the replica count alone. A vacuum that started before the server entered maintenance, or a vacuum worker's mark-writable arriving after it, handed the volume back to assignment with a replica on the draining server. Heartbeats carry only changed volumes, so nothing re-evaluated it until the volume itself changed. Apply isAllWritable on both paths, the same test EnsureCorrectWritables uses. Also pin that re-evaluating a volume a concurrent disconnect already removed from its layout is a no-op. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: record a server's read-only notification on its node before judging the volume A volume server notifies the master the moment it flips a volume between read-only and writable, ahead of the heartbeat that repeats the flag. The layout only set its per-location flag, so isAllWritable, which reads the node's heartbeat copy, still saw the old value: a mark-writable was withheld until the next heartbeat, and a re-evaluation landing between a mark-readonly and its heartbeat put the volume back on the writable list. Record the flag on the node's volume first. AddOrUpdateVolume keeps the digest and the active volume count in step, so the heartbeat that follows finds nothing to change. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: a read-only mark does not confirm a provisional volume DataNode.SetVolumeReadOnly went through Disk.AddOrUpdateVolume, which treats its input as a server report and so ended the grace period that keeps a just-grown volume safe from a full report collected before the grow. A volume marked read-only before its first report could then be removed by that stale report. Give Disk a SetVolumeReadOnly that flips the flag and keeps the digest and active volume count in step without touching volumeAddedAt. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
7620e96171 |
expose whether a volume replica is backed by remote storage, and prefer local replicas (#11105)
* expose whether a volume replica is backed by remote storage
Volume locations returned by lookups do not indicate whether a replica
has been tiered to remote storage. Readers cannot distinguish a local
replica from a remote-backed one, so they may hit a remote-backed
replica first even when a local replica is available.
Add DataInRemote to the lookup location message, populate it from the
master's volume info, and carry it through the wdclient vid map so
clients can prefer local replicas when resolving chunk locations.
* wdclient: prefer local volume replicas over remote-tier replicas on lookup
LookupFileIdWithFallback (and the publicUrl variant in FilerClient)
didn't honor the DataInRemote flag when shuffling URLs, so the
DataInRemote patch only took effect in LookupVolumeServerUrl. Apply
the same ReorderToFront(localUrls) to sameDcUrls/otherDcUrls so
non-remote replicas stay at the front, matching the existing vidMap
convention.
* wdclient: propagate DataInRemote across tier transitions on existing replicas
When a volume is tiered to remote storage or a remote-backed replica is
restored locally, the cached DataInRemote on the same volume-server URL
stayed at its old value because two pieces of state never updated:
* master_grpc_server.go only split newVolumes and (already-tracked) volumes
into NewVids vs RemoteVids. ChangedVolumes went straight to NewVids, so
the broadcast announced the re-classified volume as a fresh arrival and
the client had no way to tell whether its existing cache was stale.
* vid_map.addLocationToMap early-returned when an entry already had the
same URL. A tier transition reports the same URL with DataInRemote
flipped, so the cached entry stayed at the old classification.
Wire both sides together: ChangedVolumes now go through the same IsRemote
split as newVolumes, and addLocationToMap replaces the existing entry in
place when the URL matches but DataInRemote has changed. The server
reference key only depends on URL/grpc port, so the refcount does not
move across the flip.
Adds vid_map_remote_transition_test.go covering the local->remote and
remote->local paths so the in-place update and the cache-key stability
are pinned by tests.
* wdclient: prefer local replicas across data-center boundaries
The previous local-first ordering hoisted local URLs to the front of each
data-center bucket separately, then concatenated same-DC before other-DC.
That meant a same-DC remote replica could still be tried before an
other-DC local replica even though the local one would answer cheaply.
Reorder once across the full candidate list: concatenate same-DC and
other-DC first, then ReorderToFront pulls every local replica to the very
front while preserving the DC preference inside each tier. Apply the same
ordering in all four lookup paths so the cached vidMap, the
LookupFileIdWithFallback provider path, FilerClient.GetLookupFileIdFunction
(PublicUrl-preferred variant), and the deprecated filer.LookupFn all agree:
- weed/wdclient/vid_map.go (LookupVolumeServerUrl)
- weed/wdclient/vidmap_client.go (LookupFileIdWithFallback)
- weed/wdclient/filer_client.go (LookupFileId)
- weed/filer/reader_at.go (LookupFn)
Strengthen the existing local-first tests: vidmap_client_localfirst_test
now asserts both endpoints are present (not just the local one is first),
and slice_test asserts an exact match instead of accepting two orderings.
Add TestLookupFileIdWithFallbackGlobalLocalFirst to pin the cross-DC
ordering invariant: any local replica (same or other DC) precedes every
remote-tier replica; within each tier DC1 precedes DC2.
Add docstrings to ToVolumeLocations, ReorderToFront, LookupVolumeServerUrl,
LookupFileId, GetVidLocations, GetLocations, LookupFileIdWithFallback, and
updateVidMap so the touched lookup paths are described in one place.
* topology: broadcast tier transitions on existing replicas
When a volume replica is tiered to remote storage or restored locally, the
wdclient's cached DataInRemote went stale: every connected client kept
preferring a remote-backed replica over a freshly restored local one, or
demoted a freshly tiered remote replica. The fix in commit
|
||
|
Chris Lu
|
5f787a25c3 |
master: survive a volume layout deleted twice (#11098)
* master: survive a layout deleted twice Two volume servers dropping the last replica of volumes that share a layout both find it empty and both delete it. The loser's lookup misses, and the single-value type assertion on the result crashed the master before the caller could look at the found flag. Claude-Session: https://claude.ai/code/session_01WmX6Rchx298NQksHDXg7sk * master: remove a layout and read it back in one step DeleteVolumeLayout looked the layout up and then deleted it, so two deleters could each release the lookup ownership of the same layout, or one could find nothing to release at all. Have the map hand back what it removed. Claude-Session: https://claude.ai/code/session_01WmX6Rchx298NQksHDXg7sk |
||
|
Chris Lu
|
8f2daad338 |
topology: mirror the writable volume list in a set (#11076)
Membership was a linear scan over a slice, and ensureCorrectWritables runs it for every volume on every heartbeat, so the master's steady-state cost per volume server is quadratic in that server's volume count. BenchmarkSyncDataNodeRegistration, median of 3: 1000 volumes 565.7us -> 535.0us -5.4% 100000 volumes 1.665s -> 55.3ms -96.7% Allocations are unchanged at both sizes, so the difference is the scan. Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1 |
||
|
Chris Lu
|
9d5525e747 |
master: keep periodic volume growth to the data centers a layout lives in (#11060)
* master: keep the periodic growth scan to data centers hosting the layout The rack-aware scan planned growth for every data center in the topology, so a collection pinned to one DC (fs.configure -dataCenter) sprouted volumes in all the others within one scan cycle. Plan only for data centers already hosting the layout's volumes; an empty DC gets its volumes from the DC-constrained assign that first asks for them. The lastGrowCount divisor likewise counts only the racks the scan can plan for. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 * master: pin periodic must-grow growth to a single-DC layout's data center The must-grow and crowded paths of the periodic loop grow with no DataCenter, so even with the scan fixed a pinned collection's volumes could still land in any DC once lastGrowCount demands more writables. Stamp the grow request with the layout's data center when its volumes all live in one; layouts spanning DCs keep unconstrained growth. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 * master: never pin growth of a cross-DC-replicated layout A layout whose replication spans data centers cannot legitimately live in one DC; observing a single hosting DC there means the other DCs are down. Do not encode that outage as a placement constraint. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 * master: bound the hosting-DC walk by the answer it needs listVolumeDataCenters walked every location of the layout under accessLock — ~190ms for a million volumes, twice per layout per cycle, stalling assigns behind the read lock. Stop once enough distinct DCs answer the caller's question: two for the single-DC check, the topology's DC count for the scan. A spanning million-volume layout now finishes in microseconds; only a layout truly confined to fewer DCs still pays a full walk, the same cost class as the under-replication count this loop already takes each cycle. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 |
||
|
Chris Lu
|
0c59c0fb05 |
master: scope the startup capacity shed to a truly empty topology (#11058)
* master: scope the startup capacity shed to a truly empty topology The retryable "no volume server capacity registered yet" shed checked capacity for the requested disk type, so a cluster serving only other media -- where that capacity will never register -- shed every assign until the client's deadline instead of failing fast. An unsteered write to such a cluster hung for its full HTTP deadline and surfaced "context deadline exceeded" in place of "No writable volumes". Shed only while no disk type has any registered capacity, and name the unserved medium in the fast failure. Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R * master: name the unserved medium for every fail-fast caller The diagnostic sat in the growth-initiator block, so a follower joining an in-flight growth and a growth-disabled master failed the same way with only the generic pick error. Wrap at the fail-fast break instead, which every caller reaches, and cover all three paths in the test. Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R |
||
|
Guang Jiong LouandChris Lu
|
f740210235 |
get volume topology info without volume details (#11036)
* get volume topology info without volume details Signed-off-by: lou <alex1988@outlook.com> * master: rename VolumeListRequest.without_volumes to topology_only The field shapes the reply rather than selecting volumes, and it leaves out the ec shards too, which the old name denied. Match the message's *_only style and say what a master that predates the field does with it. Claude-Session: https://claude.ai/code/session_01QHnaNRgxnjzZsiz7WTFML5 * master: refuse topology_only combined with a volume selector A topology_only request that also names a collection or volume ids contradicts itself, and answering either half in silence surprises the caller. Answer InvalidArgument from both VolumeList and its stream, before the stream sends its header. Claude-Session: https://claude.ai/code/session_01QHnaNRgxnjzZsiz7WTFML5 --------- Signed-off-by: lou <alex1988@outlook.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
d3b8030a69 |
master: shed assigns retryably until volume servers register capacity (#11032)
An assign arriving before any volume server has heartbeated saw zero available space and failed outright with a plain error no client retries, so the first write to a fresh bucket answered 500 while the cluster was still starting. Distinguish a topology with no registered capacity from a genuinely full one: fail fast only when registered capacity is exhausted, and shed ResourceExhausted otherwise so the client's retry budget rides out the startup window. Claude-Session: https://claude.ai/code/session_018G9kWFgy8BaBAEkYV3YL9n |
||
|
Guang Jiong LouandChris Lu
|
93666c90e9 |
filter by volume ids (#10983)
* filter by volume ids * master: carry the volume ids VolumeList asks about in one repeated field One id and a list of them ask the same question, so field 2 holds the list rather than standing beside a second field that supersedes it. Claude-Session: https://claude.ai/code/session_011qAmAdhrYvnzGkw7A9N4mP --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
a02c0024e5 |
master: cap the reported capacity at what the disks hold (#10960)
* master: cap the reported capacity at what the disks hold Statistics reported max volume count times the volume size limit, which is how many volumes the cluster is allowed to place, not how much space it has. A cluster given far more slots than its disks can fill reported a capacity it could never reach -- 65536 slots at 30GB read as 1.9PB on a 460GB disk -- and the number never moved, since writing data changes neither the slot count nor the size limit. The volume servers already report each filesystem's total and free bytes in their heartbeats, so bound the answer by what they say is left. * mount: keep the last known sizes when filer statistics fails A failed Statistics call returned before df's answer was filled in, so a mount whose filer or master was briefly unreachable reported an empty filesystem rather than the sizes it already had. * master: drop the disk ceiling when a volume server does not report A cluster part way through an upgrade has volume servers that predate the disk bytes in the heartbeat. Summing only the ones that answered left the quiet server's free space out of the total, and the server holding the room is exactly the one that could make the cluster read as full. Answer with the disks only when every one of them reported. |
||
|
|
ef4c9d9178 |
filter volume by local or remote storage name (#10946)
* filter volume by local or remote storage name Signed-off-by: lou <alex1988@outlook.com> * fix SelectsEverything Signed-off-by: lou <alex1988@outlook.com> * keep the proto sync out of this change The branch copied weed/pb/*.proto over their seaweed-volume and Java counterparts and regenerated every .pb.go with a different protoc and protoc-gen-go-grpc. DiskStatus.error arriving that way broke the Rust build, and the rest is toolchain churn in files this change has nothing to say about. --------- Signed-off-by: lou <alex1988@outlook.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
b77431c142 |
master: stop hintless small-file assigns from marking volumes full (#10944)
* master: estimate a hintless assign's size from the volume's average file size An assign that carries no dataSize hint charged a flat 1MB per file id against the volume's effective size. A small-file workload overpays by orders of magnitude: bulk-writing 4KB files marks volumes holding a few hundred MB of real data as crowded and then full, so the master grows unnecessary volumes and, once every volume is spuriously full, fails all assigns. Estimate from the volume's own average file size instead, and keep the 1MB fallback only for volumes with no history. * master: decay pending assign sizes for volumes gone quiet The decay that corrects pending assign estimates runs only when a heartbeat reports the volume, and a heartbeat only reports a volume whose content changed. A volume held out of the writable list takes no writes, so once inflated estimates mark every volume full, nothing is ever reported again, nothing decays, and the cluster refuses all writes until a restart. Run the decay from the master's periodic loop for volumes no heartbeat has reported within two pulses, feeding the last reported size back through the same path an unchanged heartbeat would take. * master: trim the comments on the assign size estimate * master: keep the periodic decay out of the replica-dedup window UpdateVolumeSize ignores a report arriving within two seconds of the last one, so replicas of the same volume do not each halve the pending estimate. The periodic decay went through the same path and stamped that window, so a real heartbeat landing right behind it was dropped along with its reported size and compact revision. Only a volume whose content changed is reported at all, so nothing would send that size again and the master kept a stale one. Let the dedup window belong to volume server reports alone. * master: let the decay read the size record under the lock it mutates The periodic decay picked its volumes under a read lock and replayed them under a write one, carrying the size it had read across the gap. A heartbeat landing in between was rolled back: the replay wrote the older size and compact revision over the fresh ones, and a compaction report lost that way is never resent, since only a volume whose content changed is reported. The decay has no size of its own to contribute, so it now reads the record under the same lock it mutates. * master: let a heartbeat that beat the decay stand for the cycle The decay chooses its volumes under a read lock and applies them under a write one. A heartbeat landing in that gap already did the halving the cycle owed, so applying the decay on top of it halved twice and forgot pending bytes the volume has not written yet - the double-halving the replica-dedup window exists to prevent. Both callers now give way to a report already handled for this cycle; only a real report still advances lastUpdateTime, so a quiet volume keeps decaying every pulse. * master: keep genuinely full volumes out of the decay pass A volume the disk really did fill keeps its fullSince set for good, so it was selected every pulse for a decay that cannot help it: UpdateVolumeSize refuses to recover a volume whose reported size is at the limit, and replaying a size that cannot move leaves the record as it found it. Full and quiet is the ordinary resting state of a cluster, so this was most of the pass, taking the layout write lock away from the heartbeats to do nothing. On a million tracked volumes with a hundredth of them phantom-full it costs ten thousand write locks a pulse instead of a million. * master: put the stale-replay test back on the path it guards Giving the decay the dedup window left this test short-circuiting there, so it no longer reached the locked read it was written for and passed with that read removed. Age the record past the window, which is the only case where reading it under the lock is what saves the report. |
||
|
Chris Lu
|
9c8d3b6a81 |
ec: refund the cleared leftover shards' slots in the encode source health check (#10903)
* erasure_coding: one home for the shard-count to volume-slots conversion * ec: refund the cleared leftover shards' slots in the encode source health check |
||
|
Chris Lu
|
35d53a20f6 |
master: let the leader admit a master that starts with no raft state (#10865)
* master: answer with the leader raft already knows Topo.Leader() backs off for up to 20 seconds waiting for an election. Callers that a health probe or a client is blocked on cannot afford that: /cluster/status, /cluster/healthz and /readyz all sit past the probe timeout of both the helm chart and the operator, so a master that is still joining looks dead rather than joining, and the kubelet restarts it. informNewLeader and SendHeartbeat hold the client on a master that cannot serve it, exactly when it should move on to find the one that can. Answer these from MaybeLeader instead, which reports what raft knows right now. MaybeLeader takes over the "am I the leader myself" fallback that Leader() used to apply on top of it, so one non-blocking call is still correct; Leader() keeps the backoff for callers that must wait. * master: let the leader admit a master that starts with no raft state Neither raft implementation lets a server outside the configuration campaign: goraft's promotable() requires a non-empty log, and hashicorp rejects vote requests from a candidate that is not in its configuration. A master that comes up with fresh state therefore cannot elect itself in — the leader has to pull it in. Nothing did. The peer list is static, rendered from the replica count, so scaling it up leaves the sitting leader running the old list with no idea the new masters exist. Under goraft they wait forever. Under hashicorp they are worse off: each bootstraps a cluster of its own from the new list, and two of them form a quorum next to the live leader, with their own TopologyId. That is the split brain SetTopologyId kills a master over. Admit the peer where it registers instead. Only the leader gets past the IsLeader check in KeepConnected, and a joining master's client lands there, so that is the moment it joins. The broadcast OnPeerUpdate rides on is not enough on its own: it only reaches masters already connected, which is why a leader that came up first missed both newcomers. RaftAddServer grew a goraft branch on the way, so cluster.raft.add stops silently doing nothing on the default raft, and RaftRemoveServer with it. Bootstrapping is now one call for both implementations, made only after the peers confirm nobody has a leader, and retried until this master is in rather than checked once and dropped. * master: do not evict a peer that is still in -peers The hashicorp leader drops a master from the raft configuration as soon as it stops answering pings. A master that is merely restarting answers nothing, so an ordinary bounce shrinks the quorum behind the operator's back — and then races its own return: the master comes back, registers, gets re-admitted, and the eviction lands after it. A randomized start/stop walk lands on it. Two of three masters running, the leader evicts the one that just went down, the restart re-adds it, the removal commits late and takes the leader's own leadership with it. What is left is a two-server configuration whose other half is down, and a running master that nobody will ask for a vote — no quorum, no way back until the third master returns. -peers is what declares membership. updatePeers already reconciles the configuration against it on every leadership change, and an operator who really means to drop a master can say so with cluster.raft.remove, so keep the eviction for masters that are no longer listed at all. * test: bounce masters at random and hold the election to it Twelve rounds of stopping or starting a random master, on both raft implementations, checking the two things an election must never get wrong: two masters claiming leadership at once, and a quorum that comes back without agreeing on one. The cluster's identity has to survive the whole walk, since a master that re-mints a TopologyId is the split brain SetTopologyId kills its peers over. The seed is random and logged, so a failure names the walk that reproduces it. Below a quorum the walk moves straight on. A master that has lost its quorum cannot commit anything, and goraft only checks whether it still has one on an election-timeout ticker, after its peers have been quiet for a full timeout — measured taking over 30 seconds to step down. That direction belongs to TestTwoMastersDownAndRestart, which was giving it ten seconds and would have started failing on a slower machine; it now waits on that behaviour explicitly rather than sleeping twice and hoping. WaitForTopologyId returns the id it waited for. Reading it separately raced the leader applying the raft entry that carries it, which shows up as an empty id right after an election rather than as a wrong one. |
||
|
Chris Lu
|
3bd218e030 |
volume: cut idle memory at high volume counts (#10861)
* volume: start a volume's batch write worker on first use Mounting a volume started a goroutine parked on a 128-slot channel, plus the 128-entry batch slice it had already allocated. That is around 6.7KB per volume the server pays whether or not the volume ever takes a write: 7231 bytes per mounted volume, of which 4101 is goroutine stack. Only a write that asks for fsync ever reaches the worker, and a remote-tiered or read-only volume never can. Create the channel and its goroutine on the first such request instead, and let a write arriving after Destroy fall back to the inline path rather than queue onto a worker that has gone. Measured over 20000 mounted volumes: 7231 -> 1269 bytes each. * volume: update the heartbeat report state in place Every heartbeat built a second map of what it was about to tell the master, holding a freshly allocated short information message per volume, then swapped it in over the old one -- and computed departures through a third map of the live volume ids. A server holding 2M volumes rebuilt all three every VolumePulsePeriod for a report that usually says nothing. Number the heartbeats instead and mark the entry already held with the pass that found the copy, so a quiet volume costs a map lookup and no allocation. Departures are the entries a pass did not mark; the live-id map is now built only when there are some, sized to them. Measured over 10000 mounted volumes: 436 -> 196 bytes allocated per volume per heartbeat. * volume: fill one volume information message per heartbeat, not per volume The heartbeat built a message for every volume held so it could hash it, then dropped all but the few it had something to say about. At 2M volumes that is 2M messages allocated every VolumePulsePeriod to send almost none of them. Fill a message the caller supplies instead, and replace it only when the heartbeat keeps it, so a server with nothing to report fills the same one all the way through. Measured over 10000 mounted volumes: 196 -> 4 bytes allocated per volume per heartbeat, and a heartbeat runs a third faster. * volume: drop the per-volume trace from the heartbeat's status read glog.V(4).Infof evaluates its arguments whether or not the verbosity is on, so every volume boxed its id into a fresh interface slice on every heartbeat: 759 of the 773 allocations a 1000-volume heartbeat made, for a line that at this scale would print millions of unreadable rows. Measured over 1000 mounted volumes: 4776 -> 1792 bytes and 759 -> 14 allocations per heartbeat, which no longer grows with the volume count. * seaweed-volume: mirror the in-place heartbeat report state Same change as the Go volume server: number the heartbeats and mark the entry already held with the pass that found the copy, instead of building a second map of hashes and swapping it in. The volume snapshot must leave the reporting state as it found it, so it keeps asking through changed() while a real heartbeat marks through record(). * volume: refuse writes to a closed volume instead of dereferencing nil Close and Destroy leave the needle map and data backend nil, but a caller that already holds the volume can still reach the write path, where both are used unguarded: a write racing a volume deletion took the server down. syncDelete has always checked; syncWrite and the batch worker had not. Reachable before this series and now also from the inline fallback a durable write takes when the worker has gone. * seaweed-volume: guard the report state with one mutex, as Go does The full-list flag and the generation that answers it have to move together. Split across separate atomics they cannot: a request landing between begin's two reads returns full == false with the generation it just raised, and one landing between commit's read and its clear is marked answered by a heartbeat that carried no list. Either way the resend is dropped. Neither is reachable today -- every caller reaches this through the store's RwLock, the flag setters under a read lock and the heartbeat build under a write lock, so they cannot interleave. The type should not depend on that being true two files away, and Go holds a single mutex over exactly these fields. * test: build the servers under test to match the harness's offset size The mixed Go/Rust suites run both servers against one dataset, so both have to agree on the offset width. They did not: the harness built Go with no tags, 4-byte offsets, while the Rust crate defaults to its 5bytes feature, and the Rust server then refused the .vif the Go server had just written -- "bytes_offset mismatch: found 4, expected 5". Build each side to match the offset size the test binary itself was compiled with, so a plain `go test` and one with -tags 5BytesOffset both get a matched pair. |
||
|
|
f7c4636d22 |
topology: refresh oversized mark on every heartbeat (#10829)
* topology: refresh oversized mark on every heartbeat The oversized flag on a volume location was only set when the volume was registered (RegisterVolume). A volume that later grew past the size limit kept its stale "not oversized" mark, so the heartbeat path (ensureCorrectWritables) kept re-adding it to the writable list while RecordAssign removed it on every assign - a writable/unwritable flip loop that let writes continue past the limit and made vacuum race in-flight writes. Refresh the mark from each heartbeat's reported size in both heartbeat paths (ApplyVolumeChanges and SyncDataNodeRegistration), mirroring what RegisterVolume already did at registration time. A volume that grew past the limit now stays unwritable, and one that shrank back clears the mark and can recover. * topology: order heartbeat writable correction after decay and honor cooldown Review feedback (Greptile, CodeRabbit) on the oversized-mark refresh: 1. Greptile: clearing the oversized mark before EnsureCorrectWritables let the delay-unaware helper re-add a just-compacted volume to writables, bypassing capacityRecoveryDelay. ensureCorrectWritables now checks fullSince and skips the re-add while the cooldown is pending, so a volume removed for capacity only recovers through UpdateVolumeSize's heartbeat recovery path. 2. CodeRabbit: in the full-heartbeat path the mark was refreshed after the writable correction, so a newly oversized volume stayed writable for an extra heartbeat cycle. The standalone changedVolumes loop is merged into the volumeInfos loop and EnsureCorrectWritables now runs after UpdateOversizedState + UpdateVolumeSize in both heartbeat paths, using the freshly refreshed mark. 3. TestHandlingVolumeServerHeartbeat used a size (254320) that is past the test's volumeSizeLimit (32768); it only passed because the stale mark hid the oversized state. Sized down to 30000 to keep testing the add/remove flow, and added TestEnsureCorrectWritablesHonorsRecoveryCooldown covering the cooldown window and the recovery after it. * topology: do not restore a still-crowded volume after the cooldown Greptile review: after capacityRecoveryDelay elapses, ensureCorrectWritables could restore a volume whose effective size is still past the crowded threshold. UpdateVolumeSize refuses the recovery (effectiveSize > crowded threshold -> setVolumeCrowded + return false), but the cooldown check in ensureCorrectWritables only looked at fullSince, so once the delay passed it re-added the volume even though capacity tracking still considers it crowded. Check the crowded mark before re-adding: a volume UpdateVolumeSize just marked crowded must not be restored here, otherwise assignments resume while the volume is still flagged for growth. Adds TestEnsureCorrectWritablesDoesNotRestoreCrowdedVolume: effectiveSize decays to 10500 (past the 9000 crowded threshold) after a report of 8000, and ensureCorrectWritables keeps the volume unwritable past the cooldown. * ci: trigger re-run of flaky FUSE jobs * topology: gate the writable restore on the limit, not on crowded A crowded volume is above the growth threshold, not full, and is normally writable. Refusing to restore one locks it out for good: nothing writes to a volume that is not writable, so its size can never fall back under the threshold. Gate on the same size the assign path uses to remove it. * topology: let only the heartbeat refresh set the oversized mark Registration also set it, from whatever VolumeInfo it was handed. The incremental path builds that from a short heartbeat message, which carries no size, so every arrival announcement cleared the mark and handed the volume back to the writable list until the next full report. * topology: use the re-resolved layout after a dropped one is replaced A layout dropped with its collection makes RegisterVolume refuse, and the full heartbeat then re-registered against a fresh layout but kept applying the size, oversized and writable updates to the dropped one. --------- Co-authored-by: hzsunchao <hzsunchao@corp.netease.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
9575032b4c |
volume: forward fsync=true to replicas in ReplicatedWrite (#10805)
* volume: forward fsync=true to replicas in ReplicatedWrite When a write request carries fsync=true, only the primary volume server flushed to disk: the replica fan-out URL in ReplicatedWrite only carried type/ttl/ts/cm, so replicas always wrote without fsync even when the client explicitly requested a durable write. Forward the fsync request parameter to the replica volume servers so a durable write means every replica has flushed to disk, not just the primary. Replicas without fsync are untouched (zero behavior change). * storage: flush a durable write inline while stopping The fsync flag on the write path really selects the async batch worker, and it was switched off once the store is stopping. So a fsync=true write landing during the pre-stop drain got acked without ever being flushed - and now that ReplicatedWrite forwards fsync, that covers replicas too. Flush it inline instead of queueing it. The drain keeps accepting writes, which is the whole point of preStopSeconds, and the ack still means the .dat is on disk. If the fsync fails, the append comes back off the .dat and the needle map goes back to what it pointed at before, so nothing resolves to an offset past the truncated end. * storage: make the store's stopping flag atomic SetStopping runs on the signal handler goroutine while the write and vacuum paths read the flag, so every read of it was racy. Nothing about the shutdown ordering changes; only the flag itself is now safe to read. * topology: check the errors the replication test was dropping The mock replica ignored its response write and the mock master ignored whatever Serve returned, so a broken mock would have shown up as a confusing timeout rather than a failure. Also drops the explicit listener close: grpc.Server.Stop already closes the listener it was given. --------- Co-authored-by: hzsunchao <hzsunchao@corp.netease.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
980471c818 |
storage: count a volume's needles in uint32 (#10718)
FileCount and DeleteCount were int, so each cost a word on every replica the master holds. A volume caps at 30GB on a 4-byte-offset build and 8TB on a 5-byte one, and neither holds 4.29 billion needles. That takes VolumeInfo from 120 bytes to 112, which is its own size class rather than rounding up into the 128 one, so a replica costs 135.7 bytes in the map instead of 151.7 -- about 25MB across the 1.6M replicas in a cluster the size of the one this came from. Counts are narrowed where they are read rather than assigned across, so a report claiming more than a volume can hold pins at the ceiling instead of wrapping to a small number. |
||
|
Chris Lu
|
753cb8cda8 |
master: stop copying the cluster to name it (#10700)
* topology: name a node's volumes without copying them ToVolumeLocations reads a volume id off every volume in the cluster, and got there through GetVolumes, which copies a whole storage.VolumeInfo per volume to be read for four bytes of it. Every client that connects asks for this. At 800k volumes the walk goes from 94.6MB to 16.0MB, which is the ids themselves. * master: log why a client send failed, not what was sent The message names every volume on a newly connected node, so a client going away had the master format a protobuf that size into text -- through the one log level that is always on. The error is the part worth having. |
||
|
Chris Lu
|
5e9b7833ee |
topology: keep the volume map's values out of its slots (#10680)
Go stores a map value inline once it fits in 128 bytes, and indirectly above that. Shrinking storage.VolumeInfo to 120 bytes therefore moved 1.6M of them into the map's own slots, which are allocated to capacity rather than to occupancy, and the map grew by 149MB even though each volume got smaller. Hold pointers, so the size of a volume record stops deciding how it is stored. Updates are written through the pointer already there, so a heartbeat that changes a volume allocates nothing, as it did when the struct was above the threshold and Go was doing this itself. At 800k volume ids across 3 servers the resident topology goes from 705.3MB to 546.6MB, with heartbeat cost unchanged at 111.6MB. |
||
|
Chris Lu
|
52d74df4d1 |
clients: stream the volume listings that ask for everything (#10679)
* master: stream volume listings A listing of 800k volumes is 36MB on the wire but 305MB as messages, and the master built all of it, then held it while grpc encoded it. Two of those at once is most of a small master's heap, and the maintenance scanner asks every 30 minutes. The topology goes out first, listing nothing, then its volumes in batches, so the master holds a batch rather than a cluster: 341MB of live heap for one listing becomes 4.4MB. It allocates much the same either way -- what changes is how much of it has to be live at once, which is what sets the heap ceiling. Batches are built under their disk's lock and sent outside it, so a slow reader stalls the stream rather than the topology. They therefore do not share one instant, which a single listing did not either: it takes each disk's lock in turn, so a volume moving during either can be seen twice or not at all. The client helper hides which kind of master answered: one too old for the stream is asked the old way and its reply cut into the same batches. Either way the topology handed over lists no volumes, so a caller cannot come to depend on finding them there. * admin: stream the listing the maintenance scan reads It asks for every volume in the cluster every 30 minutes. Reassembling it client-side keeps the scan identical -- ActiveTopology splits disks by the disk ids on the volumes, so it needs them in the topology -- while the master no longer builds the whole reply to send it. * topology: report a disk id that does not depend on map order A topology disk that fronts several physical disks took its reported id from whichever volume the map yielded first, so two listings of an unchanged disk could disagree. Take the smallest instead. * topology: test that a streamed listing rebuilds to the whole one The callers that stream now rebuild the listing from a topology sent without volumes plus the batches after it, so that has to come out the same as being sent it whole, at every batch size and under a filter. * clients: stream the volume listings that ask for everything The dashboard's list and export pages, the collection and ec shard pages, the topology view, the worker metrics and two shell commands each asked the master to build all 800k volumes into one reply. They read the same listing as before, rebuilt on their side, so the master no longer holds it. The three that already ask for one volume or one collection stay as they are: their replies are small, and streaming one costs a round trip to say so. |
||
|
Chris Lu
|
46ce8cbe84 |
master: stream volume listings (#10676)
* master: stream volume listings A listing of 800k volumes is 36MB on the wire but 305MB as messages, and the master built all of it, then held it while grpc encoded it. Two of those at once is most of a small master's heap, and the maintenance scanner asks every 30 minutes. The topology goes out first, listing nothing, then its volumes in batches, so the master holds a batch rather than a cluster: 341MB of live heap for one listing becomes 4.4MB. It allocates much the same either way -- what changes is how much of it has to be live at once, which is what sets the heap ceiling. Batches are built under their disk's lock and sent outside it, so a slow reader stalls the stream rather than the topology. They therefore do not share one instant, which a single listing did not either: it takes each disk's lock in turn, so a volume moving during either can be seen twice or not at all. The client helper hides which kind of master answered: one too old for the stream is asked the old way and its reply cut into the same batches. Either way the topology handed over lists no volumes, so a caller cannot come to depend on finding them there. * admin: stream the listing the maintenance scan reads It asks for every volume in the cluster every 30 minutes. Reassembling it client-side keeps the scan identical -- ActiveTopology splits disks by the disk ids on the volumes, so it needs them in the topology -- while the master no longer builds the whole reply to send it. |
||
|
Chris Lu
|
98f9e67b4d |
topology: provisional volume update must not erase the reported disk id (#10687)
Volume growth registers a provisional record before it can know which directory the server chose, while the server's own report -- pushed during the AllocateVolume RPC -- carries the real disk id. The merge is last-writer-wins, so whichever lands second sticks, and fresh volumes nondeterministically show disk 0 on multi-dir servers. Keep the reported disk id when the provisional update carries none, before the report digest is computed so the stored record stays consistent with what the server keeps reporting. Claude-Session: https://claude.ai/code/session_01QdTEEPbg4MtcoEGwqbgtZC |
||
|
Chris Lu
|
3911e4c548 | master: keep a racing registration out of a dying collection (#10677) | ||
|
Chris Lu
|
a2ff9cca27 |
master: let VolumeList ask for the volumes it wants (#10674)
* master: let VolumeList ask for the volumes it wants The request carried nothing, so every caller was answered with the whole cluster. A dashboard opening one volume's page, or a capacity probe adding up one bucket, was served all 800k of them and threw away the rest -- and the master built every one of those messages first. The topology, its disks and their counters are still reported in full: a caller reading free space or replica placement needs the cluster whichever volumes it asked about. Only what is listed under a disk is selected, ec shards included. An empty collection and a zero volume id take everything, the way volume.list already reads its own -collectionPattern and -volumeId, so a caller that forgets to narrow is answered too much rather than answered wrongly. That leaves the default collection unnameable, since it is the one the empty string names, so it gets a field of its own. An older client sends none of it and is answered exactly as before. * admin: ask the master for the volume the page is showing A volume's detail page was pulling every volume in the cluster to find one and its replicas, and discarding the rest. * admin: ask the master for the ec volume the page is showing Same as the volume detail page: one volume's shards were found by pulling every ec shard in the cluster. * s3: ask the master for the bucket's own collection The SOSAPI capacity probe summed one collection's volumes out of a listing of every volume in the cluster. Cluster capacity still comes out the same: it is read from the disk counters, which a filtered listing reports in full. * topology: read the disk usage counters atomically They are written with atomic.AddInt64 from heartbeats but were read plainly by the two listings and by FreeSpace, and the map they sit in was iterated without the lock its neighbour takes. Under -race a listing concurrent with a heartbeat trips on both. |
||
|
Chris Lu
|
f09e8345c6 |
storage: stop keeping the remote storage key on the master (#10672)
A master decides nothing from it. Every caller that read it was asking whether a volume is remote, which the backend name answers, and the value itself is reported on demand by the server holding the volume, through the volume info in ReadVolumeFileStatus. It is also the one string here that cannot be shared: unique per volume, so unlike the collection and backend names it carries its own characters for every volume a master tracks. VolumeInfo goes from 136 bytes to 120. 800k volumes registered from a heartbeat that has been over the wire go from 214 to 163 B/volume when tiered. The volume server's own status page keeps showing the key, now read from the volume it holds rather than relayed through a master, which is also where the other volume server implementation reads it. The heartbeat digest drops it on the same grounds: a change to something the master does not hold cannot make its copy stale. Both implementations and their shared vectors move together, and the field-coverage test now names what is deliberately not retained rather than being loosened. |
||
|
Chris Lu
|
7d6c55dedb |
topology: build the volume list without copying the volume map first (#10668)
* topology: build the volume list without copying the volume map first ToDiskInfo copied every VolumeInfo on the disk into a fresh slice, walked it to build a protobuf message for each, and threw the copy away. The copy was as large as the messages it produced. Building them straight from the map holds the disk's read lock for the walk rather than just the copy, so a heartbeat updating that disk waits for it. It is a read lock on a call that is now infrequent, against an allocation of the same size as the response. ToTopologyInfo over 550k volumes 193617502 B/op -> 110011017 B/op, and faster for not making the copy. * trim the comments on this change to the parts that are not evident |
||
|
Chris Lu
|
567052bfb6 |
s3: take bucket sizes from the master's summary (#10664)
* pb: ask the master what each collection holds Callers tracking usage were sent every volume in the cluster to add up themselves, which is the master's largest single allocation. * topology: summarise what each collection holds One pass over the topology, allocating per collection rather than per volume. Regular volumes count once each for logical totals and once per replica for physical, taken from the lookup index, which is already keyed by volume and so needs no set of seen ids. Ec shards are node-local so their sizes sum, while the file and delete counts describe the volume and resolve once every holder has been seen. Replicas of one volume disagree while a write is landing or a heartbeat is late. Walking a full listing took whichever replica the map iteration reached first, so the answer moved between runs; this takes the largest, which is stable and never reports usage below what some replica already holds. * s3: take bucket sizes from the master's summary The bucket size metrics pulled the whole volume list once a minute and added it up, which cost the master 184.6MB of allocation and 17.8MB on the wire for six numbers per collection. VolumeList over 550k volumes 184.6 MB allocated, 17.8 MB on the wire CollectionStatistics 176 bytes allocated, 47 bytes on the wire The aggregation moves to the master with it, so the cases the removed tests covered are now asserted against it directly. * topology: count the replica holding the most live data Quotas are enforced on size less deletions, and the replica with the biggest raw size can be the one that has deleted the most. Counting it reported a bucket smaller than it is and would leave one writable over its quota, which is the opposite of what picking the largest was meant to guarantee. * topology: cap a volume's deletions at what it holds Live usage is read as a collection's size less its deletions, so a volume reporting more deleted bytes than it has cancels live bytes belonging to other volumes in the same bucket and reports it smaller than it is. Replica selection already floored that volume's own live size at zero; the totals have to agree with it. |
||
|
Chris Lu
|
67b0cc0706 |
topology: keep per-node volume state with the location it describes (#10654)
* topology: keep per-node volume state with the location it describes The read-only and oversized indexes were maps from volume id to a list of the nodes reporting that state -- the same key space the lookup index already holds, kept a second and third time. Nothing ever asked which nodes; both are only ever asked whether any node does. So the state rides on the location list as a bit per entry, and the two indexes go. Removing a location shifts the bits with it, and a node replacing another at the same address inherits its slot, since that is what happens to the location too. 800k volumes, 90% read-only: readonly index 40.0MB -> 0, lookup index 42.5MB -> 48.6MB for the bits, 33.9MB net. * topology: rebuild the location flags when stale entries are dropped Refresh rebuilds the location list, so leaving the flags alone left bits describing whoever moved into the dropped entries' place. * topology: assert the refreshed flag survived, not just that it moved Clearing the mask rather than rebuilding it would have passed: the check that the flag lands on the right location is done by clearing it, which an already empty mask satisfies. |
||
|
Chris Lu
|
2dc59c9b51 |
topology: track volume size only where writes can land (#10653)
* topology: track volume size only where writes can land Size tracking decays pending assignment estimates so the master does not overfill a volume before heartbeats catch up. Nothing is ever assigned to a read-only volume, so an entry for one can never be consulted -- and in a tiered cluster that is most of them, which made this the volume layout's largest cost. A volume held out of the writable list for capacity is not read-only and keeps its entry: that entry is what enforces the recovery delay. 800k volumes, 90% read-only: sizeTracking 79.1MB -> 8.4MB, and the crowded set falls out with it because a read-only volume no longer reaches the threshold check at all. * topology: decide size tracking per volume, not per reporting replica A volume is unwritable if any replica is read-only, so asking the replica whose heartbeat happened to arrive made the answer depend on arrival order: a writable replica reporting after a read-only one put the tracking back. Ask the volume instead, which also drops the caller-supplied flag and the churn it caused. The crowded entry goes with the tracking, since leaving it behind would only move the memory this releases. Costs a map lookup per replica on a full-list heartbeat, about 19ms per 100k volumes and no allocations, on a path that is now rare. |
||
|
Chris Lu
|
a2ffc7aadf |
heartbeat: keep the master current through collection churn (#10657)
* heartbeat: name departed volumes in delta heartbeats * master: release the lookup index with a deleted collection * master: keep a fresh grow safe from the report that raced it * volume: name the volumes a deleted collection took with it Deleting a collection left the master to work out what went by omission from the next full volume list, which it no longer gets: heartbeats carry the whole list only when the master asks for it. The volumes a bucket's churn creates and destroys between two of those requests are never named in either direction, so the master keeps counting their slots as occupied and a cluster that creates and drops collections quickly runs its free-slot accounting dry -- assigns fail with no free volumes left while the disk holds a handful of volumes. The destroy path already knows exactly which volumes it removed, so send them down the same channel every other deletion uses. * rust: name the volumes a deleted collection took with it Mirrors the Go volume server. The notify path derives its deltas by diffing snapshots, so a collection delete that does not wake it is invisible until the master next asks for the whole list. |