The existing multi-disk EC integration test asserts on shard counts. Counting
cannot tell a healthy volume from one a repair reassembled out of the wrong
inputs — both have fourteen shards. This drives the real shell commands
(ec.encode, ec.balance, ec.rebuild) against a live three-node, four-disk
cluster and reads the stored bytes back after every step, so a rebuild that
produced fourteen plausible-but-wrong shards fails here.
An 8 KB random payload is stored, then encoded, balanced, damaged (two shard
files removed and the servers restarted so the master relearns the reduced
set from disk), and rebuilt. The rebuild output matches the shape of the
support case that motivated this — "rebuildOneEcVolume", "missing shard N.0",
"copied N.1 from ..." — and the payload is verified identical after each of
upload, encode, balance, shard loss, and rebuild.
Two ordering facts the test pins, both of which cost real debugging time:
ec.rebuild is driven by the master's topology, not disk truth, so shards must
be relearned (via restart) before a repair can target the right set; and the
shell lock is dropped when the restart disconnects the master, so it has to be
retaken before the rebuild.
FileCount and DeleteCount were int, so each cost a word on every replica the
master holds. A volume caps at 30GB on a 4-byte-offset build and 8TB on a
5-byte one, and neither holds 4.29 billion needles.
That takes VolumeInfo from 120 bytes to 112, which is its own size class rather
than rounding up into the 128 one, so a replica costs 135.7 bytes in the map
instead of 151.7 -- about 25MB across the 1.6M replicas in a cluster the size
of the one this came from.
Counts are narrowed where they are read rather than assigned across, so a
report claiming more than a volume can hold pins at the ceiling instead of
wrapping to a small number.
* filer: stop silently dropping metadata replay failures from peers
When two filers do not share a store (e.g. one leveldb3 per pod), each
subscribes to its peers' metadata streams and replays their events
locally (meta_aggregator.go's maybeReplicateMetadataChange, wired into
doSubscribeToOneFiler). A failed Replay() was logged and then treated
as done anyway: processEventFn always returned nil regardless of the
replay outcome, and processOne advanced lastTsNs unconditionally. The
offset is the only record of subscription progress, so a dropped event
was gone for good - no retry, and nothing else ever observed it.
An entry that fails to replay this way diverges from its peer
permanently. This is how a bucket's quota (entry.Quota, carried on
peer events like everything else - see entry_codec.go's EqualEntry
comparing Quota, and FromPbEntry copying it in entry.go) can end up
different across filers indefinitely: one replay hiccup on one filer,
and its enforcement and any metric reading its own store diverges from
the others' with no signal anything went wrong.
Fix: replicateMetadataChange now retries a failure with util.Retry,
which already distinguishes transient errors (timeouts, connection
resets, throttling, ...) from everything else and bounds the backoff.
That covers the common case - a busy store, a blip talking to a
remote-backed backend - without changing behavior when replay
succeeds. An error that is not transient, or outlives the retry
budget, is not retried further: propagating it so the offset never
advances would stall this peer's entire stream behind one event that
may never replay, which is worse than the one entry staying stale.
Instead it is skipped, loudly - counted in a new
stats.FilerMetaAggregatorReplayFailures metric and logged at error
level - so the divergence is discoverable instead of silent.
Tested: go build ./... and go test ./weed/filer/... ./weed/server/...
Added meta_aggregator_replay_test.go: one test fails against the old
one-shot Replay call (a single transient failure is never retried, so
the store never converges) and passes with the fix; a second covers a
permanently-failing event completing quickly and being counted instead
of retried forever.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* filer: keep the test quota constant int64 for 32-bit builds
An untyped shift constant passed to t.Fatalf's ...any defaults to int and
overflows on 32-bit, failing go vet there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* filer: name the diverged entry in the give-up log line
event.Directory is only the parent (typically /buckets), so for any
directory with more than one child the previous log line could not say
which entry failed to replay - the exact thing the change exists to
make discoverable. Name comes from NewEntry, falling back to OldEntry
for deletes; both getters are nil-safe.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore(filer): trim metadata replay comments to the non-obvious why
Compress the added comments on replicateMetadataChange and its tests down to
the reasoning a maintainer cannot get from the code: why a retry-exhausted
failure is skipped rather than propagated, what the old one-shot Replay body
did that the test pins, and why the quota constant is typed int64. Drops
deployment-specific narration and restatement of the code. No behaviour change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore: restore load-bearing clauses trimmed in the comment pass
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* filer: document and test the multi-step DeleteEntry replay hazard
CodeRabbit flagged that FilerStoreWrapper.DeleteEntry skips the delete
once FindEntry reports the path already gone, and that the redis store
families remove the primary key before parent-directory membership.
Chained together, a delete that fails between those two steps is
retried as a no-op: the stale membership is never revisited, and
replicateMetadataChange now reports overall success for it without
incrementing FilerMetaAggregatorReplayFailures, whereas before this PR
every such failure was unconditionally logged. The underlying store
inconsistency is pre-existing (a single non-retried Replay already
leaves the same stale membership behind); what retry adds is that this
one case no longer surfaces it.
Making Replay atomic or teaching every store to repair secondary
mutations on retry is out of scope here. Instead: document the hazard
at Replay, filerstore_wrapper.go's DeleteEntry, and
replicateMetadataChange, and add a test against the real
FilerStoreWrapper (not a strawman) that pins down the current,
documented behavior.
* filer: trim replay retry comments and tests
Drop the comment-only hunks documenting the pre-existing DeleteEntry
partial-failure hazard, the test that asserted that hazard still exists,
and the second hand-rolled fake store. Reuse stubFilerStore for the two
retry tests.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* admin: make paths relative
* admin: make filer browser link and nav path checks prefix-relative
* admin: add isCurrentPath and currentPathStartsWith helpers
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* benchmark tool for mounted filesystems
* ci: on-demand mount benchmark, native WinFsp vs rclone plus a Linux reference
* windows mount: let the Windows cache manager cache file data
WinFsp only turns the cache manager on for a file when FileInfoTimeout
is infinite; at any finite value every application read and write is a
synchronous trip into the mount process at whatever size the application
issued. Metadata events already reach FspFileSystemNotify, which purges
a changed file's cached pages and attributes, so an infinite timeout
stays coherent. The dir listing, volume info and EA timeouts are pinned
to one second so they do not silently inherit the infinity.
* windows mount: cache resolved paths and attributes in the adapter
WinFsp addresses every operation by path and has no FORGET, so the
adapter walked the whole path through Lookup on each one, and in a
directory the filer has not listed yet every walk was a filer round
trip; nothing played the part of the kernel's dentry and attribute
caches. The path cache owns one lookup reference per entry the way the
kernel holds one until FORGET, serves attribute reads for files without
an open handle, and is purged by the mount's own mutations and by
metadata events, with the timeout as backstop.
* windows mount: keep a closed file's attributes cached
Open steals the path's cache entry for its handle and Release returned
the reference with a purge, so the stat that follows every copied file
walked to the filer again. Reading the handle's final attributes before
it goes away and moving the reference back into the cache serves that
stat locally, the way the kernel's attribute cache does after a close.
Only if the path still names that inode, though: WinFsp reports the
path the handle opened with, and after a delete-on-close or a rename
caching it would resurrect an entry that is gone.
* windows mount: persist entries at create, and let the flush stay at close
WinFsp posts the cleanup and close that carry the flush after
CloseHandle has returned, so deferring the filer entry to the flush let
everything that reads through the filer race an unflushed close: a
listing missed just-written files, and a directory rename moved a
directory on the filer before its newest child existed there, leaving
the straggler flush to recreate the child under the dead path.
Flush-at-cleanup is not the answer either: it makes every handle's
cleanup flush, and those flushes race the unlinks of delete-on-close,
re-inserting the entry the unlink just removed. Persisting the entry at
create takes the ordering question away.
* mount: flush written pages before a truncate shrinks past them
The shrink trims chunks, but written pages that have not become chunks
yet are invisible to it, so the next flush wrote them back and the file
grew again, resurrecting the truncated bytes. Windows hits this on
every write-then-shrink because its flush runs after CloseHandle, but
the gap is platform-neutral.
* mount: order a file's unlink against its in-flight flush
Unlink set the handle's deleted flag bare, so a flush already past its
own check of that flag wrote the entry back right after the delete
removed it, and a delete-on-close file outlived its last handle. The
flag is now set under the handle's flush lock and re-checked under it,
so a flush either completes before the delete or sees the flag and
skips. An eagerly created handle also starts clean: the dirty mark
existed to make the deferred filer create happen at flush, and eager
creates have nothing to flush.
* shell: volume.move restores source writability when aborted after the copy phase
* shell: volume.move removes the incomplete target copy when aborted before the source delete
* shell: give each abort cleanup RPC its own timeout
* filer: TUS creation accepts Upload-Concat partial uploads
* filer: TUS final uploads concatenate completed partials
* filer: TUS concatenation tests
* filer: consumed marker pins TUS chunk ownership on completion
* filer: TUS session delete decides chunk ownership after removing the session info
* filer: TUS completion persists the consumed marker before creating the entry
* filer: TUS completion re-verifies the session after persisting the consumed marker
* filer: serialize TUS session ownership transitions per filer
* filer: surface failed TUS consumed-marker rollbacks
* s3: a null object wins over a rescan when the latest-version pointer is absent
The read path already resolves an absent pointer this way; the listing-path
counterpart scanned .versions/ first and could surface an old version or
delete marker over the current suspended-versioning null object.
* s3: dedup a key against its .versions sibling in suspended buckets too
A suspended bucket keeps its .versions directories, so a suspended-versioning
null object and its .versions sibling emitted the same key twice.
* s3: retract a null object from the listing when a delete marker shadows it
Deleting a key whose null version predates versioning leaves the base-path
entry in place and records the delete marker under <key>.versions. The
listing appended the base-path entry and relied on the .versions sibling to
replace it, but a delete-marker current version emitted nothing, so the
deleted key stayed visible to ListObjects while GET and HEAD returned 404.
* s3: keep a key's .versions sibling on the same page as the key
When the page quota ran out between a base-path entry and its .versions
directory, the page ended with the stale entry and the next page skipped the
directory as a marker echo, so the replacement or retraction never happened.
* s3: the null version is not latest when the .versions pointer names a newer one
ListObjectVersions stamped IsLatest on every base-path null object, so a key
deleted after enabling versioning reported IsLatest on both the delete marker
and the null version.
* s3: test listing after a pre-versioning null object is delete-marked
* s3: find a key's earlier page entry by scan, not by adjacency
A key such as k.bak sorts between k and k.versions, so the entry a .versions
sibling replaces or retracts is not always the last one on the page. Scan
back through the page for the key, and insert a late resolution in sorted
position instead of at the end.
* s3: settle trailing null objects by lookup when a page fills
The quota can run out while keys still sit between a null object and its
.versions sibling, and the sibling-adjacent page-boundary exception never
fires for those. Track the trailing null objects whose sibling has not been
ruled out and look each one up before declaring the page full; a retraction
reopens the quota.
* s3: do not resolve a .versions sibling its page has already moved past
A page resuming from a marker inside the base key's extension region has
already listed and settled the base null object on an earlier page, so
resolving the .versions directory again re-emitted the key.
* s3: test listing with keys between a null object and its .versions sibling
* s3: pick the newer of the null object and the scanned versions
Making the null object win outright whenever the pointer is absent misread
multi-filer pointer lag: version files replicate ahead of the pointer, and a
key overwritten or delete-marked after pre-versioning days would list its
stale null again. The suspended-versioning write that legitimately makes the
null current is also the newer entry, so mtime tells the two apart.
* s3: a delete-marked null object no longer keeps its prefix alive
The hidden-entries probe took any plain file as proof of a listable key, but
a null object shadowed by its .versions sibling's delete marker is not one.
Hold plain files pending until the sibling settles them either way.
* s3: settle an evicted pending null instead of dropping it
Nested keys like k, k!, k!! can hold more pending nulls than the cap. A
silently evicted one could close the page unsettled, and the resume skip
would then keep the stale entry for good.
* s3: test deleted-prefix hiding and the pending-null cap
* s3: cover the reported '!' intervening key with a live version
* s3: an unstamped same-second version outranks the null object
Second-resolution mtimes cannot order same-second writes, so the tie went to
the stale null when the pointer lagged. The suspended write that makes a null
current stamps the version it displaces before clearing the pointer, so the
stamp is the authoritative signal and a tie without it goes to the version.
* s3: a pointer-less versions listing still checks what replicated
ListObjectVersions took a missing pointer as proof the null object is latest,
but under pointer lag the sibling can already hold newer replicated versions
or markers. Apply the same nullObjectWins rule as the listing recovery.
* s3: a failed null-object settlement fails the listing
Every getEntry error read as a missing sibling, so a transient filer error at
a page boundary committed the unsettled null and the next page skipped its
sibling for good. Only a definitive not-found means the null is live; other
failures are retained on eviction and fail the request at page close.
* s3: retract a CommonPrefix whose only backers were delete-marked nulls
The directory probe settles this for the / delimiter, but any other delimiter
derives prefixes from base-path keys directly, and a prefix built solely from
null objects survived their delete markers. Count the unsettled null backers
behind the newest prefix and retract it when the last one settles as a marker;
a live resolution or any listable contributor confirms the prefix instead.
* s3: test custom-delimiter prefix retraction
* s3: an explicit signal marks the null object current, not the demotion stamp
The NoncurrentSinceNs stamp survives promotion: delete the version that
demoted another and the promoted one is current yet still stamped, so a
lagging replica would resurrect the stale null. A suspended-versioning write
now records Seaweed-X-Amz-Null-Version-Is-Latest on the .versions directory
when it clears the pointer, every pointer update removes it, and the
recovery paths trust the signal instead of the stamp.
* s3: a filer failover retry rebuilds the listing page from scratch
The failover wrapper reruns the callback on another filer after a transport
error, and the partially built page, spent quota, and advanced marker leaked
into the retry, which could then return a stale or duplicated page as
success.
* s3: only a prefix's own backers can debit it
A delete marker for a version-only key (no base object) derived the same
prefix as its neighbors and decremented backing it never contributed,
retracting a prefix that a live null object still backed. Track backers by
key so settlement is idempotent and only debits what was counted.
* s3: test a version-only marker against a null-backed prefix
* s3: a pointer recompute clears the null-current signal
The routed finalize for delete markers, COPY, and multipart rewrites the
.versions pointer through RECOMPUTE_LATEST, which left a suspended-era
null-current signal in place. Version files never carry the signal, so
mapping it in CopyExtended deletes it whenever the pointer recomputes.
* s3: the pointer outranks the null-current signal in the versions listing
The signal check guarded the pointer check, so a stale signal a recompute
had not cleared yet would have let the null claim IsLatest alongside the
pointed-at version.
* topology: name a node's volumes without copying them
ToVolumeLocations reads a volume id off every volume in the cluster, and got
there through GetVolumes, which copies a whole storage.VolumeInfo per volume to
be read for four bytes of it. Every client that connects asks for this.
At 800k volumes the walk goes from 94.6MB to 16.0MB, which is the ids
themselves.
* master: log why a client send failed, not what was sent
The message names every volume on a newly connected node, so a client going
away had the master format a protobuf that size into text -- through the one
log level that is always on. The error is the part worth having.
Go stores a map value inline once it fits in 128 bytes, and indirectly above
that. Shrinking storage.VolumeInfo to 120 bytes therefore moved 1.6M of them
into the map's own slots, which are allocated to capacity rather than to
occupancy, and the map grew by 149MB even though each volume got smaller.
Hold pointers, so the size of a volume record stops deciding how it is stored.
Updates are written through the pointer already there, so a heartbeat that
changes a volume allocates nothing, as it did when the struct was above the
threshold and Go was doing this itself.
At 800k volume ids across 3 servers the resident topology goes from 705.3MB to
546.6MB, with heartbeat cost unchanged at 111.6MB.
* master: stream volume listings
A listing of 800k volumes is 36MB on the wire but 305MB as messages, and the
master built all of it, then held it while grpc encoded it. Two of those at
once is most of a small master's heap, and the maintenance scanner asks every
30 minutes.
The topology goes out first, listing nothing, then its volumes in batches, so
the master holds a batch rather than a cluster: 341MB of live heap for one
listing becomes 4.4MB. It allocates much the same either way -- what changes is
how much of it has to be live at once, which is what sets the heap ceiling.
Batches are built under their disk's lock and sent outside it, so a slow reader
stalls the stream rather than the topology. They therefore do not share one
instant, which a single listing did not either: it takes each disk's lock in
turn, so a volume moving during either can be seen twice or not at all.
The client helper hides which kind of master answered: one too old for the
stream is asked the old way and its reply cut into the same batches. Either way
the topology handed over lists no volumes, so a caller cannot come to depend on
finding them there.
* admin: stream the listing the maintenance scan reads
It asks for every volume in the cluster every 30 minutes. Reassembling it
client-side keeps the scan identical -- ActiveTopology splits disks by the
disk ids on the volumes, so it needs them in the topology -- while the master
no longer builds the whole reply to send it.
* topology: report a disk id that does not depend on map order
A topology disk that fronts several physical disks took its reported id from
whichever volume the map yielded first, so two listings of an unchanged disk
could disagree. Take the smallest instead.
* topology: test that a streamed listing rebuilds to the whole one
The callers that stream now rebuild the listing from a topology sent without
volumes plus the batches after it, so that has to come out the same as being
sent it whole, at every batch size and under a filter.
* clients: stream the volume listings that ask for everything
The dashboard's list and export pages, the collection and ec shard pages, the
topology view, the worker metrics and two shell commands each asked the master
to build all 800k volumes into one reply. They read the same listing as before,
rebuilt on their side, so the master no longer holds it.
The three that already ask for one volume or one collection stay as they are:
their replies are small, and streaming one costs a round trip to say so.
* master: stream volume listings
A listing of 800k volumes is 36MB on the wire but 305MB as messages, and the
master built all of it, then held it while grpc encoded it. Two of those at
once is most of a small master's heap, and the maintenance scanner asks every
30 minutes.
The topology goes out first, listing nothing, then its volumes in batches, so
the master holds a batch rather than a cluster: 341MB of live heap for one
listing becomes 4.4MB. It allocates much the same either way -- what changes is
how much of it has to be live at once, which is what sets the heap ceiling.
Batches are built under their disk's lock and sent outside it, so a slow reader
stalls the stream rather than the topology. They therefore do not share one
instant, which a single listing did not either: it takes each disk's lock in
turn, so a volume moving during either can be seen twice or not at all.
The client helper hides which kind of master answered: one too old for the
stream is asked the old way and its reply cut into the same batches. Either way
the topology handed over lists no volumes, so a caller cannot come to depend on
finding them there.
* admin: stream the listing the maintenance scan reads
It asks for every volume in the cluster every 30 minutes. Reassembling it
client-side keeps the scan identical -- ActiveTopology splits disks by the
disk ids on the volumes, so it needs them in the topology -- while the master
no longer builds the whole reply to send it.
Volume growth registers a provisional record before it can know which
directory the server chose, while the server's own report -- pushed
during the AllocateVolume RPC -- carries the real disk id. The merge is
last-writer-wins, so whichever lands second sticks, and fresh volumes
nondeterministically show disk 0 on multi-dir servers. Keep the reported
disk id when the provisional update carries none, before the report
digest is computed so the stored record stays consistent with what the
server keeps reporting.
Claude-Session: https://claude.ai/code/session_01QdTEEPbg4MtcoEGwqbgtZC
The volume server names the directory index in every
VolumeShortInformationMessage, but NewVolumeInfoFromShort dropped it, so
volumes registered through the incremental new-volume path showed
disk_id 0 at the master until a full report -- misreporting multi-dir
servers in volume.list and the per-physical-disk topology views.
Claude-Session: https://claude.ai/code/session_01QdTEEPbg4MtcoEGwqbgtZC
* volume: decode IPv6 transition addresses in the remote-endpoint guard
checkBlockedIP normalized only ::ffff: mapped IPv4, so NAT64 (64:ff9b::/96),
6to4 (2002::/16), Teredo (2001:0000::/32), and IPv4-compatible (::/96) addresses
that embed an internal IPv4 (loopback, 169.254.169.254, RFC 1918) passed the
endpoint guard even though the plain IPv4 forms are refused. Extract the
embedded IPv4 from those forms and re-check it against the deny list, which
covers both the up-front validation and the dial-time guard. Mirrored in the
Rust volume server.
* volume: require the full NAT64 well-known prefix before decoding
Only 64:ff9b::/96 carries the embedded IPv4 in the low 32 bits, so also require
bytes 4-11 to be zero before treating an address as NAT64; other 64:ff9b:
prefixes place the IPv4 elsewhere and are left untouched. Add public-target
coverage for 6to4, Teredo, and IPv4-compatible so every decoder is exercised on
both a blocked and an allowed destination. Mirrored in the Rust volume server.
* master: let VolumeList ask for the volumes it wants
The request carried nothing, so every caller was answered with the whole
cluster. A dashboard opening one volume's page, or a capacity probe adding up
one bucket, was served all 800k of them and threw away the rest -- and the
master built every one of those messages first.
The topology, its disks and their counters are still reported in full: a caller
reading free space or replica placement needs the cluster whichever volumes it
asked about. Only what is listed under a disk is selected, ec shards included.
An empty collection and a zero volume id take everything, the way volume.list
already reads its own -collectionPattern and -volumeId, so a caller that
forgets to narrow is answered too much rather than answered wrongly. That
leaves the default collection unnameable, since it is the one the empty string
names, so it gets a field of its own.
An older client sends none of it and is answered exactly as before.
* admin: ask the master for the volume the page is showing
A volume's detail page was pulling every volume in the cluster to find one and
its replicas, and discarding the rest.
* admin: ask the master for the ec volume the page is showing
Same as the volume detail page: one volume's shards were found by pulling every
ec shard in the cluster.
* s3: ask the master for the bucket's own collection
The SOSAPI capacity probe summed one collection's volumes out of a listing of
every volume in the cluster. Cluster capacity still comes out the same: it is
read from the disk counters, which a filtered listing reports in full.
* topology: read the disk usage counters atomically
They are written with atomic.AddInt64 from heartbeats but were read plainly by
the two listings and by FreeSpace, and the map they sit in was iterated without
the lock its neighbour takes. Under -race a listing concurrent with a heartbeat
trips on both.
A master decides nothing from it. Every caller that read it was asking whether
a volume is remote, which the backend name answers, and the value itself is
reported on demand by the server holding the volume, through the volume info in
ReadVolumeFileStatus.
It is also the one string here that cannot be shared: unique per volume, so
unlike the collection and backend names it carries its own characters for every
volume a master tracks.
VolumeInfo goes from 136 bytes to 120. 800k volumes registered from a heartbeat
that has been over the wire go from 214 to 163 B/volume when tiered.
The volume server's own status page keeps showing the key, now read from the
volume it holds rather than relayed through a master, which is also where the
other volume server implementation reads it.
The heartbeat digest drops it on the same grounds: a change to something the
master does not hold cannot make its copy stale. Both implementations and their
shared vectors move together, and the field-coverage test now names what is
deliberately not retained rather than being loosened.
* storage: order VolumeInfo by alignment
The struct is held for every volume replica in the cluster, so the padding the
compiler inserts is multiplied by however many volumes a master tracks. Two
one-byte fields each sat at the head of a word and left the rest of it empty,
which was ten of the eighteen wasted bytes.
Grouping by size rather than by meaning takes the struct from 152 bytes to 136,
and the map holding them shrinks with it, since a Go map's slack scales with
the size of the value.
800k volumes registered from a heartbeat that has been over the wire:
211 -> 195 B/volume, 214 -> 198 tiered.
* trim the comments on this change to the parts that are not evident
* topology: build the volume list without copying the volume map first
ToDiskInfo copied every VolumeInfo on the disk into a fresh slice, walked it to
build a protobuf message for each, and threw the copy away. The copy was as
large as the messages it produced.
Building them straight from the map holds the disk's read lock for the walk
rather than just the copy, so a heartbeat updating that disk waits for it. It
is a read lock on a call that is now infrequent, against an allocation of the
same size as the response.
ToTopologyInfo over 550k volumes 193617502 B/op -> 110011017 B/op, and faster
for not making the copy.
* trim the comments on this change to the parts that are not evident
* ec.balance: add a -volumeIds filter
Collection scope is often too broad for maintenance. -volumeIds narrows the
plan to the given ec volume ids by leaving every other volume out of the
topology handed to the planner, so no phase, dedup included, can plan against
them. Ids with no ec shard in the selected collection, dataCenter and disk type
are rejected rather than silently skipped.
* ec.encode: key the orphan sweep without narrowing the volume id
int is 32-bit on 32-bit builds, so int(vid) wraps for volume ids above
MaxInt32. Format the id as the uint32 it is.
* pb: ask the master what each collection holds
Callers tracking usage were sent every volume in the cluster to add up
themselves, which is the master's largest single allocation.
* topology: summarise what each collection holds
One pass over the topology, allocating per collection rather than per volume.
Regular volumes count once each for logical totals and once per replica for
physical, taken from the lookup index, which is already keyed by volume and so
needs no set of seen ids. Ec shards are node-local so their sizes sum, while
the file and delete counts describe the volume and resolve once every holder
has been seen.
Replicas of one volume disagree while a write is landing or a heartbeat is
late. Walking a full listing took whichever replica the map iteration reached
first, so the answer moved between runs; this takes the largest, which is
stable and never reports usage below what some replica already holds.
* s3: take bucket sizes from the master's summary
The bucket size metrics pulled the whole volume list once a minute and added it
up, which cost the master 184.6MB of allocation and 17.8MB on the wire for six
numbers per collection.
VolumeList over 550k volumes 184.6 MB allocated, 17.8 MB on the wire
CollectionStatistics 176 bytes allocated, 47 bytes on the wire
The aggregation moves to the master with it, so the cases the removed tests
covered are now asserted against it directly.
* topology: count the replica holding the most live data
Quotas are enforced on size less deletions, and the replica with the biggest
raw size can be the one that has deleted the most. Counting it reported a
bucket smaller than it is and would leave one writable over its quota, which is
the opposite of what picking the largest was meant to guarantee.
* topology: cap a volume's deletions at what it holds
Live usage is read as a collection's size less its deletions, so a volume
reporting more deleted bytes than it has cancels live bytes belonging to other
volumes in the same bucket and reports it smaller than it is. Replica selection
already floored that volume's own live size at zero; the totals have to agree
with it.
* storage: share the volume strings a cluster repeats
Decoding a heartbeat allocates a fresh string for the collection, disk type and
remote backend of every volume, and a master holding a million volumes then
holds a million copies of the same handful of names.
Not the remote storage key, which is unique per volume: interning that would
fill the table rather than share anything.
800k volumes registered from a heartbeat that has actually been over the wire:
227 -> 211 B/volume, and 238 -> 214 when the volumes are tiered, since the
backend name shares too.
* storage: hold the interned strings rather than let them be collected
unique.Make clears its entries by weak reference, and its canonical value does
not survive a collection even while a caller still holds the string it handed
back -- so a volume reported later would get a second copy of a name the rest
of the cluster already shares. With only changed volumes reported, most are
interned once and never again, so that is the common case rather than a corner.
The table therefore only grows, which is why it stays restricted to values
drawn from a small set. Ten thousand collections keep a few hundred kilobytes.
* iceberg: stamp a default name mapping on new tables
* iceberg: repair non-compliant manifests at commit
* s3tables: verify ClickHouse writes read back through PyIceberg
* iceberg: carry the manifest-list content into repaired manifests
* iceberg: refresh the default name mapping on schema evolution
* iceberg: merge historical names into the refreshed name mapping
* iceberg: never fail a commit on repair fallout
* iceberg: harden manifest repair against writer dialects
* s3tables: keep PyIceberg reader stderr out of row data
* iceberg: keep name mappings unambiguous across field id reassignment
* iceberg: align existing manifest content metadata with the list entry
* s3: add the RenameObject endpoint
PUT /{bucket}/{key}?renameObject with x-amz-rename-source moves an object
through the filer's AtomicRenameEntry, so no bytes are read or rewritten and
the ETag, tags and SSE keys travel with the entry.
Only unversioned buckets: a versioned rename would have to rebuild the
.versions chain, and AWS offers RenameObject on directory buckets, which
cannot be versioned. The source arrives in a header, so it is authorized
separately for read and delete; both keys are locked, in key order, across the
precondition checks and the move.
* s3: let a matched source ETag precondition settle its date precondition
RFC 7232 has an ETag precondition outrank the date precondition on its own
side, and AWS documents the same for CopyObject: a matching
x-amz-copy-source-if-match with a failing x-amz-copy-source-if-unmodified-since
copies rather than returning 412. The source check evaluated all four headers in
sequence, so the date header could still veto a decided ETag match.
validateConditionalHeadersForReads already applies this precedence; the source
path now matches it.
* s3: cover a rename source named as a directory without a trailing slash
Renaming a directory would move a whole subtree, so it has to stay a missing
key whether or not the caller wrote the trailing slash.
* s3: accept a bare object key as the RenameObject source
AWS spells x-amz-rename-source both ways. Its CLI, Java and Rust examples pass
the bare source key, and only a second CLI example and the boto3 conditional
example pass bucket/key; the API reference's own example is a bare key too. The
header was read as bucket/key only, so the form AWS leads with was rejected with
InvalidArgument and the endpoint was unusable as documented.
A value is now read as a literal key first — the only reading that can never
name the wrong object — and as bucket-qualified second, when it carries the
request's own bucket and the literal key does not exist. That costs one extra
lookup only for a source that starts with the bucket's own name.
Another bucket's name in the source is no longer a distinct error: RenameObject
moves within one bucket, so it is simply part of a key this bucket does not
hold, and it reports NoSuchKey.
* s3: only a proven absence picks the other reading of a rename source
A source that resolves to a directory is not a miss to fall through on: the
literal path is still what the caller named, so answering for it beats renaming
a different object under the bucket-qualified reading. With a directory at
bucket/source.txt and an object at source.txt, a rename naming the former moved
the latter.
A failed lookup is not a proof of absence either, so a blip can no longer
redirect a rename to the other reading.
* topology: keep per-node volume state with the location it describes
The read-only and oversized indexes were maps from volume id to a list of the
nodes reporting that state -- the same key space the lookup index already
holds, kept a second and third time. Nothing ever asked which nodes; both are
only ever asked whether any node does.
So the state rides on the location list as a bit per entry, and the two indexes
go. Removing a location shifts the bits with it, and a node replacing another
at the same address inherits its slot, since that is what happens to the
location too.
800k volumes, 90% read-only: readonly index 40.0MB -> 0, lookup index
42.5MB -> 48.6MB for the bits, 33.9MB net.
* topology: rebuild the location flags when stale entries are dropped
Refresh rebuilds the location list, so leaving the flags alone left bits
describing whoever moved into the dropped entries' place.
* topology: assert the refreshed flag survived, not just that it moved
Clearing the mask rather than rebuilding it would have passed: the check that
the flag lands on the right location is done by clearing it, which an already
empty mask satisfies.
* topology: track volume size only where writes can land
Size tracking decays pending assignment estimates so the master does not
overfill a volume before heartbeats catch up. Nothing is ever assigned to a
read-only volume, so an entry for one can never be consulted -- and in a tiered
cluster that is most of them, which made this the volume layout's largest cost.
A volume held out of the writable list for capacity is not read-only and keeps
its entry: that entry is what enforces the recovery delay.
800k volumes, 90% read-only: sizeTracking 79.1MB -> 8.4MB, and the crowded set
falls out with it because a read-only volume no longer reaches the threshold
check at all.
* topology: decide size tracking per volume, not per reporting replica
A volume is unwritable if any replica is read-only, so asking the replica whose
heartbeat happened to arrive made the answer depend on arrival order: a
writable replica reporting after a read-only one put the tracking back.
Ask the volume instead, which also drops the caller-supplied flag and the churn
it caused. The crowded entry goes with the tracking, since leaving it behind
would only move the memory this releases.
Costs a map lookup per replica on a full-list heartbeat, about 19ms per 100k
volumes and no allocations, on a path that is now rare.
* s3api: fix ListObjectsV2 dropping objects under a partial prefix
ListObjectsV2 with a prefix that names only part of a directory name -
"data/a" matching both "data/a/..." and "data/ab/..." - dropped objects
when paginating with a small max-keys, reporting IsTruncated=false before
all matching keys had been returned.
On resume, normalizePrefixMarker took the listing directory from the
continuation marker ("data/a/") and discarded the "a" name prefix, so once
the marker's subtree was exhausted the sibling directories that also
matched the prefix were never listed.
Resolve the listing directory and name prefix from the prefix instead of
the marker, so siblings sharing the name prefix are still listed after the
marker's subtree.
Fixes#10652
* s3api: cover a sibling-directory marker in the partial prefix tests
* s3api: resume a truncated delimited listing from the emitted CommonPrefix
Rebuilding the continuation token from the request dir plus the listing prefix
assumes that prefix names a directory. With a partial name prefix like "data/a"
it produced "data/a/ab/" for the sibling prefix "data/ab/", so the next page
resumed in the wrong subtree and repeated it forever.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* heartbeat: name departed volumes in delta heartbeats
* master: release the lookup index with a deleted collection
* master: keep a fresh grow safe from the report that raced it
* volume: name the volumes a deleted collection took with it
Deleting a collection left the master to work out what went by omission from
the next full volume list, which it no longer gets: heartbeats carry the whole
list only when the master asks for it. The volumes a bucket's churn creates and
destroys between two of those requests are never named in either direction, so
the master keeps counting their slots as occupied and a cluster that creates
and drops collections quickly runs its free-slot accounting dry -- assigns fail
with no free volumes left while the disk holds a handful of volumes.
The destroy path already knows exactly which volumes it removed, so send them
down the same channel every other deletion uses.
* rust: name the volumes a deleted collection took with it
Mirrors the Go volume server. The notify path derives its deltas by diffing
snapshots, so a collection delete that does not wake it is invisible until the
master next asks for the whole list.
* topology: mark a volume crowded only if it can take writes
Crowding asks for more room to write into, and the writable-volume refresh loop
marked anything past the threshold regardless of whether writes could land
there. Growth already discounts those by intersecting the crowded set with the
writable list, so the entries changed no decision and only took space -- in a
tiered cluster, one for nearly every volume.
* topology: wait for the crowded-volume collector before reading what it saw
Closing the stop channel does not order the collector's writes against the
test's reads.
* topology: drop the sleep from the crowded-volume test
The channels are unbuffered, so every send has been received by the time the
sweep returns, and waiting for the collector covers the recording. The sleep
only suggested the result turned on timing.
* filer: keep the existing peer subscription on a repeated add
A cluster node add for a peer that is already followed restarted the
subscription, dropping the metadata events between the two runs.
* master: tell a connecting client the current cluster membership
Cluster node updates are only broadcast to the clients connected at that
moment. A filer that lost its master stream while a peer came back never
learned about the peer, and stopped replicating its metadata for good.
* test: a filer joining the master learns about the filers already there
* test: a filer resubscribes to a peer that registered while it was disconnected
Runs the reported sequence against real processes: filer2 leaves, filer1
is paused and its master stream is broken, filer2 registers again, and
filer1 has to replicate from it after reconnecting.
* volume: validate the file extension in CopyFile and ReceiveFile
CopyFile and ReceiveFile build an on-disk path from the client-supplied
Ext. Both are intentionally ungated for cluster-internal peers, so a
value like "/../../x" is joined onto the volume directory and, once
path-cleaned, resolves outside it -- an EC-shard receive can then write,
and CopyFile read, anywhere the process can reach.
Constrain Ext to a real suffix (a leading dot followed by alphanumerics)
before it is used to build any path, so it can no longer carry a
separator or a parent reference.
* test: use an alphanumeric missing-file extension in the copy variants
The not-found and stop-offset-zero cases used ".definitely-missing" as a
deliberately absent source. The extension is now validated, and the hyphen
makes it invalid, so switch to ".missing" -- still a nonexistent file, but a
real extension shape.
* volume: validate the collection in CopyFile and ReceiveFile
The client-supplied Collection is folded into the on-disk path as
"<collection>_<vid>" by VolumeFileName and EcShardBaseFileName, both joined
with path.Join / util.Join. A Collection carrying a separator, e.g.
"../../x", therefore path-cleans to a target outside the volume directory,
the same escape the extension check just closed. Reject a collection that is
a bare parent reference or holds a separator; ordinary names ('.', '-' and
all) still pass.
* filer: drain pending log chunk refs when the metadata stream ends
In metadata chunks mode the server sends log file refs in responses of their
own, and the client can only read them once it knows the run of refs is over.
That was inferred solely from the arrival of a normal event, so refs still
pending when the stream ended were dropped: the subscription returned no
events and no error.
A follower never noticed, because it runs forever and a live event always
arrives to close the run. A bounded subscription — StopTsNs set, range already
in the past — can receive nothing but refs and then EOF, and silently reports
that nothing happened. For anything auditing a path that is the worst possible
answer, since an empty result is indistinguishable from a quiet period.
Drain on EOF as well as at the transition point.
* filer: add filer.meta.scan to audit one directory's change history
Reconstructing what happened to a path means replaying the metadata log, and
filer.meta.tail is built for watching rather than auditing: it follows forever
unless given a stop, prints multi-line JSON, and takes ranges only as durations
before now, so an incident timestamp has to be converted by hand.
Its -pattern also cannot find a versioned object. A versioned key is stored as
<key>.versions/v_<id>, so the events carry the names "<key>.versions" and
"v_<id>" and a pattern of the object's own name matches neither — the search
comes back empty while the object is being written continuously.
filer.meta.scan prints one line per change, stops at the end of the range,
accepts absolute -since/-until with an explicit -tz, and reports versioned
writes against the object key with the version id alongside, so -name matches
the key a client would ask for. Delete markers are labelled as such rather than
appearing as zero-length writes, and pointer flips on the .versions container
are distinguished from writes of object data.
* filer.meta.scan: read persisted log chunks from the volume servers
Reading a range through the filer makes it decode every log entry in that
range and filter each one, so the cost lands on the filer and does not shrink
when the prefix is narrow — only the bytes on the wire do. On a cluster whose
metadata log is dense that is the expensive part of a scan, and it is charged
to the process least able to spare it.
Enable metadata chunks mode: the filer hands out log chunk ids and the scan
reads them from the volume servers itself. ReadLogFileRefs re-applies the same
path filter client-side, so the output is unchanged — verified identical to
the filer-read path over the same range, including after a restart drops the
in-memory buffer and the data must come off disk.
Direct read needs a route to the volume servers that the filer does not, so a
failure before anything has been printed retries through the filer; retrying
after partial output would duplicate lines. -directRead=false forces it.
* filer.meta.scan: confirm an empty direct-read result through the filer
An audit that returns nothing is read as "nothing happened here", so it is the
one answer that must not be produced by a bug. Direct read has more ways to
come back empty than the filer path does — it needs a route to the volume
servers, and it depends on the ref-drain contract holding.
When direct read yields no changes, re-run through the filer before reporting
it, and warn if the two disagree. Re-running is safe only because nothing was
printed; after partial output a replay would duplicate lines instead, so that
case reports the error rather than retrying.
In metadata chunks mode the server sends log file refs in responses of their
own, and the client can only read them once it knows the run of refs is over.
That was inferred solely from the arrival of a normal event, so refs still
pending when the stream ended were dropped: the subscription returned no
events and no error.
A follower never noticed, because it runs forever and a live event always
arrives to close the run. A bounded subscription — StopTsNs set, range already
in the past — can receive nothing but refs and then EOF, and silently reports
that nothing happened. For anything auditing a path that is the worst possible
answer, since an empty result is indistinguishable from a quiet period.
Drain on EOF as well as at the transition point.
* master: evict a dead peer via the local raft handle
OnPeerUpdate only runs on the leader, and the AddVoter branch right above
mutates the local raft directly. The remove branch instead dialed our own
RaftRemoveServer back over gRPC. Drop the self-dial and remove the peer
through the local handle, matching the add path. This also leaves operator
tooling as the only caller of the RaftRemoveServer RPC.
* master: require whitelist auth for raft membership RPCs
RaftAddServer, RaftRemoveServer and RaftLeadershipTransfer rewrite raft
quorum but had no caller check beyond "am I the leader". Any client that
could reach the master gRPC port could add an unreachable phantom voter
and stall the write path.
Gate the three on the admin whitelist, mirroring the volume server's
checkGrpcAdminAuth. With no whitelist configured the guard allows every
caller, so default and single-master deployments are unaffected;
operators who set -whiteList get these RPCs locked down to it. The
leader's own dead-peer eviction no longer dials these RPCs, so the only
remaining callers are operator tooling.
Chris Lu