FileCount and DeleteCount were int, so each cost a word on every replica the
master holds. A volume caps at 30GB on a 4-byte-offset build and 8TB on a
5-byte one, and neither holds 4.29 billion needles.
That takes VolumeInfo from 120 bytes to 112, which is its own size class rather
than rounding up into the 128 one, so a replica costs 135.7 bytes in the map
instead of 151.7 -- about 25MB across the 1.6M replicas in a cluster the size
of the one this came from.
Counts are narrowed where they are read rather than assigned across, so a
report claiming more than a volume can hold pins at the ceiling instead of
wrapping to a small number.
* topology: keep per-node volume state with the location it describes
The read-only and oversized indexes were maps from volume id to a list of the
nodes reporting that state -- the same key space the lookup index already
holds, kept a second and third time. Nothing ever asked which nodes; both are
only ever asked whether any node does.
So the state rides on the location list as a bit per entry, and the two indexes
go. Removing a location shifts the bits with it, and a node replacing another
at the same address inherits its slot, since that is what happens to the
location too.
800k volumes, 90% read-only: readonly index 40.0MB -> 0, lookup index
42.5MB -> 48.6MB for the bits, 33.9MB net.
* topology: rebuild the location flags when stale entries are dropped
Refresh rebuilds the location list, so leaving the flags alone left bits
describing whoever moved into the dropped entries' place.
* topology: assert the refreshed flag survived, not just that it moved
Clearing the mask rather than rebuilding it would have passed: the check that
the flag lands on the right location is done by clearing it, which an already
empty mask satisfies.
VolumeLocationList.Stats subtracts the deleted figures from the totals to
report live size and needle count. Both deleted figures are maintained as
counters independent of the totals they come off, so either can transiently
exceed its total, and neither subtraction was clamped.
Unclamped, the size wraps to ~16 EB. The count is signed so it merely goes
negative, but VolumeLayout.Stats converts it with uint64(fileCount), which
turns it into ~1.8e19 just the same. Either one swamps the cluster totals
behind /dir/status, /vol/status and Topology.CollectionVolumeStats.
commandFsMergeVolumes.getVolumeSize had the same unclamped subtraction, where
a wrapped size reads as a volume far too large to join any merge plan.
Clamped to zero, matching the guards already in CollectionInfo.LogicalSize
and the admin server's logical-size accumulator.
* topology: digest the volumes a master believes each node holds
A volume server resends its whole volume list every heartbeat because that list
is the only way the master can notice a volume that vanished without a delta.
A digest gives the master the same guarantee without the list: the two ends
agree iff the master's copy is current.
VolumeInfo.ReportHash covers every field of VolumeInformationMessage, so a
change the hash misses is a change the master would never hear about. Both ends
run it over the same converted VolumeInfo, so they cannot drift apart.
Disk keeps the xor of its volumes' hashes, which is order-independent and its
own inverse, so add, update and remove each stay O(1) and the running value
needs no per-volume storage.
Nothing reads the digest yet; the heartbeat protocol change comes next.
* topology: test that a changed-volumes-only heartbeat reconciles
The digest is not a change detector -- in a live cluster some volumes always
have changed. It answers whether the master holds what the volume server holds
once the heartbeat's own changes are applied, so reporting three volumes out of
fifty has to reconcile while a volume lost without a delta must not.
* topology: digest the lookup index too, not just the disk maps
The reported digest answers whether the master holds what the volume server
holds. It cannot answer whether the master can serve those volumes: the disk
map and the lookup index are maintained separately, and a disconnect racing a
reconnect drops a volume from the index while leaving it on the node. The
server's report is identical either way, so a digest built from the disk maps
alone matches while the volume answers 'volume id not found'.
Track a second digest over volume ids on both sides of that split, so the
master can see its own indexes disagree without the volume server's help, and
without the O(volumes) scan the full heartbeat currently relies on.
* topology: exclude nodes reporting a duplicate volume id from the digest
A volume id can end up mounted on two disks of one server -- a stale twin
re-attached after a disk repair, which the store handles rather than rejects.
The server reports both copies with different disk ids, but the master keys
volumes by id alone within a disk type and keeps only the last one. Its digest
can then never equal the server's, and no amount of resending the full list
would fix it.
Detect it from the report itself, where deduplicating the ids already tells us
the count, and mark the node. A marked node has to keep sending full lists;
representing both copies is a separate question, and nesting the volume map by
disk id would cost more memory than the digest saves.
* topology: move the lookup digest with the entry, not the node passed in
Two volume servers can hold one address: GetOrCreateDataNode keys on the id a
server reports and refuses to merge a new id onto an address an older node
still claims, while the lookup list keys on address alone. Registering the
second server therefore displaces the first from the entry, and unregistering
through either removes whichever node the entry named.
Crediting the node handed to Set and Remove instead of the one actually
displaced or removed left the digest on the wrong node. A displaced node went
on reporting a consistent index while it could no longer serve the volume,
which is exactly the silent unavailability the digest exists to catch.
Set and Remove now return the node they displaced and removed, so ownership
can be transferred rather than assumed.
Currently the file size of only one volume location is taken into
account in the stats. This commit multiplies the disk usages by the
amount of nodes holding a replica of the volume.
This will yield the expected amount of disk usage and matches the
total size calculations from before.
Chris Lu