mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-20 13:30:46 +02:00
6dd83c823d4bbc50a0e5eb503b208ffe9ab59d93
1138
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Lu
|
65b9ae7704 |
master: keep disk_id when registering volumes from incremental heartbeats (#10686)
The volume server names the directory index in every VolumeShortInformationMessage, but NewVolumeInfoFromShort dropped it, so volumes registered through the incremental new-volume path showed disk_id 0 at the master until a full report -- misreporting multi-dir servers in volume.list and the per-physical-disk topology views. Claude-Session: https://claude.ai/code/session_01QdTEEPbg4MtcoEGwqbgtZC |
||
|
Chris Lu
|
f09e8345c6 |
storage: stop keeping the remote storage key on the master (#10672)
A master decides nothing from it. Every caller that read it was asking whether a volume is remote, which the backend name answers, and the value itself is reported on demand by the server holding the volume, through the volume info in ReadVolumeFileStatus. It is also the one string here that cannot be shared: unique per volume, so unlike the collection and backend names it carries its own characters for every volume a master tracks. VolumeInfo goes from 136 bytes to 120. 800k volumes registered from a heartbeat that has been over the wire go from 214 to 163 B/volume when tiered. The volume server's own status page keeps showing the key, now read from the volume it holds rather than relayed through a master, which is also where the other volume server implementation reads it. The heartbeat digest drops it on the same grounds: a change to something the master does not hold cannot make its copy stale. Both implementations and their shared vectors move together, and the field-coverage test now names what is deliberately not retained rather than being loosened. |
||
|
Chris Lu
|
0f7a64c596 |
storage: order VolumeInfo by alignment (#10669)
* storage: order VolumeInfo by alignment The struct is held for every volume replica in the cluster, so the padding the compiler inserts is multiplied by however many volumes a master tracks. Two one-byte fields each sat at the head of a word and left the rest of it empty, which was ten of the eighteen wasted bytes. Grouping by size rather than by meaning takes the struct from 152 bytes to 136, and the map holding them shrinks with it, since a Go map's slack scales with the size of the value. 800k volumes registered from a heartbeat that has been over the wire: 211 -> 195 B/volume, 214 -> 198 tiered. * trim the comments on this change to the parts that are not evident |
||
|
Chris Lu
|
38db7e1493 |
storage: share the volume strings a cluster repeats (#10665)
* storage: share the volume strings a cluster repeats Decoding a heartbeat allocates a fresh string for the collection, disk type and remote backend of every volume, and a master holding a million volumes then holds a million copies of the same handful of names. Not the remote storage key, which is unique per volume: interning that would fill the table rather than share anything. 800k volumes registered from a heartbeat that has actually been over the wire: 227 -> 211 B/volume, and 238 -> 214 when the volumes are tiered, since the backend name shares too. * storage: hold the interned strings rather than let them be collected unique.Make clears its entries by weak reference, and its canonical value does not survive a collection even while a caller still holds the string it handed back -- so a volume reported later would get a second copy of a name the rest of the cluster already shares. With only changed volumes reported, most are interned once and never again, so that is the common case rather than a corner. The table therefore only grows, which is why it stays restricted to values drawn from a small set. Ten thousand collections keep a few hundred kilobytes. |
||
|
Chris Lu
|
a2ffc7aadf |
heartbeat: keep the master current through collection churn (#10657)
* heartbeat: name departed volumes in delta heartbeats * master: release the lookup index with a deleted collection * master: keep a fresh grow safe from the report that raced it * volume: name the volumes a deleted collection took with it Deleting a collection left the master to work out what went by omission from the next full volume list, which it no longer gets: heartbeats carry the whole list only when the master asks for it. The volumes a bucket's churn creates and destroys between two of those requests are never named in either direction, so the master keeps counting their slots as occupied and a cluster that creates and drops collections quickly runs its free-slot accounting dry -- assigns fail with no free volumes left while the disk holds a handful of volumes. The destroy path already knows exactly which volumes it removed, so send them down the same channel every other deletion uses. * rust: name the volumes a deleted collection took with it Mirrors the Go volume server. The notify path derives its deltas by diffing snapshots, so a collection delete that does not wake it is invisible until the master next asks for the whole list. |
||
|
Mohit Talniya
|
5b9236c76d | storage: fix corrupted leveldb detection in DoOffsetLoading (#10650) | ||
|
Chris Lu
|
ce7d388639 |
heartbeat: send only the volumes that changed (#10640)
* pb: let a heartbeat carry only the volumes that changed A partial list cannot travel in volumes: a master that did not understand it would read the absences as deletions. So changes get their own field, used only once the master has said it compares digests and can tell when it has fallen behind. * master: apply the volumes a heartbeat reports as changed Only the named volumes are touched. A full report says the server holds exactly these; a changed report says nothing about the ones it leaves out, so absence must not read as removal. Also advertises that the master compares digests, which is what lets a server stop sending its whole list. Advertising it once per connection means a server reconnecting to a master that does not is back to full lists straight away. * volume: send only the volumes that changed once the master accepts them The whole list goes on every heartbeat until the master says it compares digests, and again whenever it asks, so a master that cannot tell when it has fallen behind never has to. has_no_volumes stays derived from a full list alone. Deriving it from what a heartbeat happens to carry would make a quiet one read as a server that had lost every volume, and the master would drop them all. The digest still covers every volume held rather than the ones sent, which is what lets the master confirm that applying the changes left it current. Reporting state is per-connection: a server that reconnects, or reaches a different master, starts again from the full list. * volume: let the zero reporting state stand for having told no master anything A Store built as a literal, which tests do, left the reporting state nil and panicked on the first heartbeat. As a value its zero form already means nothing has been reported to anyone, which is exactly the state that sends the whole list. * rust: send only the volumes that changed once the master accepts them Mirrors the Go volume server, with one hazard the Go side does not have: mount and unmount deltas here are derived by diffing successive heartbeats, so a heartbeat that carries a partial list would report every volume it left out as unmounted. Collecting now returns the full set alongside the message, and every site that diffs uses that rather than what went on the wire. * volume: do not let a full-list request be lost to the heartbeat it raced The request arrived while a heartbeat was already being built as a delta, and committing that heartbeat cleared it, so the master waited for another digest mismatch before asking again. Count the requests and clear only the one the heartbeat answered. * rust: stop marking volumes reported by a heartbeat that is thrown away The state-notify path collected a heartbeat only to diff its volume list, then sent a delta message of its own and dropped the one it had collected. Once collecting recorded what the master had been told, every mount or unmount silently marked the changed volumes as sent, and the master learned of them only after a digest mismatch. Snapshotting no longer records anything, and no longer expires ec volumes whose deletion that path was already discarding. * master: announce only the volumes a change actually brought Every changed volume was broadcast as a new location. Volumes grow constantly and growth moves no location, so on a busy cluster that told every connected client about volumes it could already reach, filling bounded broadcast queues and pushing out the topology updates that matter. * master: ask for the full list when only one can repair the master Delta heartbeats stop the full report, and with it the only thing that re-registers a volume the lookup index lost. The volume server cannot see that divergence and its digest cannot show it, so the master now checks its own two indexes agree and asks for the list when they do not. A node reporting one volume id twice is kept on full lists for the same reason rather than merely skipped: its digest can never be verified, so nothing else would tell the master what it had stopped holding. * master: keep the volume options on every heartbeat response A volume server takes them from whatever response arrives, and preallocate is a bare bool with no way to tell off from unmentioned. A response sent to ask for the volume list therefore turned preallocation off until the server reconnected. Responses sent mid-stream now start from the configured options rather than being built field by field. * master: announce a volume the lookup index had lost Repairing the index makes the volume servable again, but clients were told it went when the node dropped out and nothing told them otherwise: the disk map still held it, so it did not count as an arrival. Reaching the lookup index is what makes a volume servable, so recovering an entry there is an arrival as far as clients are concerned, on both the full report and the changed-volume path. |
||
|
Chris Lu
|
6d08b08f37 |
heartbeat: carry a volume digest and verify it (#10627)
* pb: carry a volume digest on the heartbeat The full volume list is the only way a master notices a volume that vanished without a delta, so it cannot simply be dropped. A digest gives the same guarantee without the list, and a way back to the list when they disagree. The digest has explicit presence: a server holding no volumes reports 0, which has to stay distinguishable from a server that does not compute one at all. * volume: report a digest of the volumes each heartbeat carries Digests exactly what goes on the wire: volumes skipped as quarantined, phantom or expired are absent from both the list and the digest, so the master compares against the same set the server meant to report. Runs the master's own hash over the master's own conversion of the message, so the two ends cannot drift into disagreeing about a field. * master: check the reported volume digest and ask for the list on a mismatch Compared after everything the heartbeat carried has been applied, so agreement means the master is current rather than that nothing changed. Servers reporting no digest are untouched, and a mismatch on a heartbeat that already carried the full list is reported rather than answered: there is nothing further to ask for, so asking again would loop. Nodes reporting one volume id twice are skipped for the same reason. * rust: report the heartbeat volume digest Mirrors the Go volume server. The master compares this against a digest it computes itself, so the hash has to agree byte for byte across the two implementations, not merely be a hash of the same fields: report_hash_vectors pins it against values generated by the Go side, and the ttl and replica placement narrowing the master applies when it decodes a message is applied here too rather than assumed away. A drift there would not corrupt anything, but every volume server on this implementation would report a digest the master can never match and fall back to sending its whole volume list forever, which is the cost the digest exists to avoid. * master: pin what the digest check does to each kind of report The upgrade story rests on these: a server that reports no digest is never asked for anything, so the two sides can be upgraded in either order, and a disagreement that resending cannot fix is reported rather than re-asked, so it cannot loop. * topology: enumerate the digest coverage test from the message The list of fields was written out by hand, so a field added to VolumeInformationMessage later would fall outside the digest while the test went on passing, and a change to it would never reach the master. Walk the message descriptor instead. Some fields are narrowed or normalised on the way into VolumeInfo, so the smallest change to the wire value can land back on the stored one; the test offers several values per field and asks only that some change is visible. |
||
|
Chris Lu
|
553bc5ab90 |
topology: digest the volumes a master believes each node holds (#10619)
* topology: digest the volumes a master believes each node holds A volume server resends its whole volume list every heartbeat because that list is the only way the master can notice a volume that vanished without a delta. A digest gives the master the same guarantee without the list: the two ends agree iff the master's copy is current. VolumeInfo.ReportHash covers every field of VolumeInformationMessage, so a change the hash misses is a change the master would never hear about. Both ends run it over the same converted VolumeInfo, so they cannot drift apart. Disk keeps the xor of its volumes' hashes, which is order-independent and its own inverse, so add, update and remove each stay O(1) and the running value needs no per-volume storage. Nothing reads the digest yet; the heartbeat protocol change comes next. * topology: test that a changed-volumes-only heartbeat reconciles The digest is not a change detector -- in a live cluster some volumes always have changed. It answers whether the master holds what the volume server holds once the heartbeat's own changes are applied, so reporting three volumes out of fifty has to reconcile while a volume lost without a delta must not. * topology: digest the lookup index too, not just the disk maps The reported digest answers whether the master holds what the volume server holds. It cannot answer whether the master can serve those volumes: the disk map and the lookup index are maintained separately, and a disconnect racing a reconnect drops a volume from the index while leaving it on the node. The server's report is identical either way, so a digest built from the disk maps alone matches while the volume answers 'volume id not found'. Track a second digest over volume ids on both sides of that split, so the master can see its own indexes disagree without the volume server's help, and without the O(volumes) scan the full heartbeat currently relies on. * topology: exclude nodes reporting a duplicate volume id from the digest A volume id can end up mounted on two disks of one server -- a stale twin re-attached after a disk repair, which the store handles rather than rejects. The server reports both copies with different disk ids, but the master keys volumes by id alone within a disk type and keeps only the last one. Its digest can then never equal the server's, and no amount of resending the full list would fix it. Detect it from the report itself, where deduplicating the ids already tells us the count, and mark the node. A marked node has to keep sending full lists; representing both copies is a separate question, and nesting the volume map by disk id would cost more memory than the digest saves. * topology: move the lookup digest with the entry, not the node passed in Two volume servers can hold one address: GetOrCreateDataNode keys on the id a server reports and refuses to merge a new id onto an address an older node still claims, while the lookup list keys on address alone. Registering the second server therefore displaces the first from the entry, and unregistering through either removes whichever node the entry named. Crediting the node handed to Set and Remove instead of the one actually displaced or removed left the digest on the wrong node. A displaced node went on reporting a consistent index while it could no longer serve the volume, which is exactly the silent unavailability the digest exists to catch. Set and Remove now return the node they displaced and removed, so ownership can be transferred rather than assumed. |
||
|
Chris Lu
|
ee54fd6c08 |
perf(weed/storage/super_block): intern the byte-encoded replica placements (#10610)
NewReplicaPlacementFromByte formatted the byte with fmt.Sprintf and parsed the result back, allocating a string and a ReplicaPlacement every call. The master calls it once per volume in every heartbeat, and keeps the pointer for the lifetime of the volume, so a cluster with 1.6M volume replicas carries 1.6M of these where a handful of distinct values exist. The table is a flat pointer-free array, so it costs 6KB of static data and no heap objects however few placements a cluster actually uses. A byte only ever decodes to a valid placement, so the table is complete and the error return stays nil. BenchmarkSyncDataNodeRegistration/100000Volumes 500601 allocs/op -> 300589 allocs/op |
||
|
Chris Lu
|
33c36fc7a3 |
perf(weed/storage/needle): intern the stored ttl values (#10611)
The master decodes a TTL per volume in every heartbeat and keeps it for the volume's lifetime, so a cluster using TTLs carries one two-byte object per volume replica where at most 256 counts times 7 units exist. Share them, and decode the uint32 form directly instead of staging it through a byte slice. Clusters that set no TTL are unaffected; that path already returned the shared EMPTY_TTL. BenchmarkSyncDataNodeRegistration/100000Volumes, volumes carrying a ttl 600600 allocs/op -> 500597 allocs/op |
||
|
Chris Lu
|
505049a4de |
volume: skip directory fsync on Windows, report a failed makeupDiff (#10572)
* volume: skip directory fsync on Windows * ci: run the windows jobs for the whole vacuum path Both windows jobs start the same weed mini cluster, so both exercise the volume server's vacuum path, but only one of them watched a single file in it. Cover the compact, reconcile and load files in both. * volume: report a failed makeupDiff instead of discarding it The cleanup removes assigned to the same err the makeupDiff failure was held in, so an aborted compaction returned nil once both removes succeeded. The master then recorded the vacuum as committed and the volume reloaded against the discarded generation. * volume: correct the fsyncDir comments after the windows skip Both comments described the old shape, where windows fell through to a sync whose error was swallowed. * volume: keep the makeupDiff failure ahead of its cleanup errors A failed remove of .cpd/.cpx outranked the failure that abandoned the compaction, so the caller saw the cleanup error instead of the cause. Log it and return the original, matching the Rust do_commit_compact. A leftover temp file is rolled back by reconcile on the next start. |
||
|
Chris Lu
|
312cfe5ae1 |
Fix volume.merge corrupting every needle it copies (#10565)
* Give volume.merge the needle size the target actually indexes by needleBlobFromNeedle returned the size Append reports, which is Size(n.DataSize) - payload bytes only. The .dat header, the needle map and WriteNeedleBlobRequest.Size all use n.Size, which additionally covers the flags, name, mime and lastModified fields. Every needle volume.merge copied therefore landed with a too-small size. The target indexed it at that length, so every later read failed the header check in ReadBytes with a size mismatch, and on v3 the fresh AppendAtNs stamp landed NeedleHeaderSize+DataSize+NeedleChecksumSize into the blob - exactly on the flags byte - overwriting flags, name size, mime size and the first mime bytes with the top of a timestamp. Needles came back with flags 0x18, no name, no mime and a phantom TTL parsed from two arbitrary timestamp bytes; the ones that decoded as expired 404 and vacuum would drop them. Since merge rebuilds every replica from the merged copy, no clean replica survives. Return n.Size, which Append fills in as it serializes, matching what the normal write path stores via nm.Put. * Reject needle blobs whose size disagrees with their own header WriteNeedleBlob trusts the caller's size for two destructive things: it is what goes into the needle map, and it is where the v3 AppendAtNs stamp is written inside the caller's buffer. A caller passing the payload-only DataSize convention corrupts both, and nothing surfaces until the needle is read back - by which point every replica may already have been rebuilt from it. Parse the blob's own header and refuse the write when the two disagree. Mirrored in the Rust volume server. |
||
|
Mohit Talniya
|
0815ad78f6 |
fix(volume): persist the leveldb needle map watermark at batch boundaries (#10557)
levelDbWrite persists the replay watermark when its updateWatermark
argument is true. Put and Delete passed "watermark == 0", which is true
on exactly the writes that carry no checkpoint and false on the batch
boundary that carries one. The two cases were inverted:
recordCount % watermarkBatchSize != 0 -> watermark 0, flag true
-> re-persists a zero on 9999 of every 10000 writes
recordCount % watermarkBatchSize == 0 -> watermark N, flag false
-> drops the only value worth saving
The stored watermark therefore never left 0. Recovery stayed correct,
because replaying .idx from offset 0 is a superset of replaying from N
and replay is idempotent, so this never surfaced as a failure. It only
meant generateLevelDbFile walked the entire index on every rebuild, and
every needle write paid a second leveldb Put to rewrite the same zero.
Pass "watermark != 0" so the boundary write checkpoints and the writes
in between leave the key alone.
Verified on a 25000-needle volume: the stored watermark now reads 20000
instead of 0, and a rebuild replays 5000 entries instead of 25000.
The new test drives a full batch of Puts and a full batch of Deletes to
cover both call sites.
|
||
|
Chris Lu
|
de00091765 |
test: random needles always carry at least one byte (#10523)
A zero-data needle lands in .dat as a size-0 record, byte-identical to a delete marker, so scans that walk .dat count it as deleted. Once in 1024 writes newRandomNeedle produced one, and the idx-head repair then skipped a row TestRepairIdxHeadTombstones_ReadOnlyVolume expected back. |
||
|
Lisandro Pin
|
fa9f471f56 |
EC scrubbing: list shards for needles failing scrubs in the result output. (#10510)
This allows to pinpoint failures to a subset of shards, which can then be bisected and potentially reconstructed. |
||
|
Chris Lu
|
4dc1b70b2f |
test: pin that a .vif replication outranks the superblock (#10499)
* test: pin that a .vif replication outranks the superblock Store.ConfigureVolume rewrites the .vif and never the replica-placement byte in the .dat, so that byte keeps whatever the volume was created with for good. readSuperBlock reads it and then overrides it from the .vif, which is what makes a replication change take effect and survive a remount. Invert that and every replication change silently reverts on the next mount, while the .vif on disk still records what the operator asked for -- a durability setting quietly going back to its old value, with nothing to indicate it. Worth pinning rather than reading off the code, because the field beside it resolves the other way: version takes the superblock over the .vif. Two fields, one function, opposite precedence, each a line to invert wrongly. Covers the empty case too, since a .vif that declares no replication has to leave the superblock standing or a volume whose replication was never configured would be forced to whatever the zero value parses as. * test: drop the unreachable nil check on MaybeLoadVolumeInfo It initialises the returned pointer before the existence check and every return is naked, so it never yields nil. Guarding against it implied a contract the callee does not have. |
||
|
Chris Lu
|
13176b4edd |
volume: recover .idx rows overwritten by tiered deletes (#10474)
* volume: recover .idx rows overwritten by tiered deletes A delete on a read-only volume backed by a remote tier used to write its tombstone row at .idx offset 0 rather than appending it, so each delete overwrote one more row at the front and lost the Put rows indexing the first needles in .dat. Those needles 404 even though .dat still holds them, and rebuilding .idx with weed fix means stopping the server and pulling the whole .dat back from the tier. The damage has a fingerprint -- .idx opening with a run of offset-0 tombstones, which a healthy .idx never does -- and .idx and .dat grow in lockstep, so the lost rows indexed exactly the first N .dat records. Detect it at load and re-derive them from a header-only walk over the head of .dat, cheap even against a remote tier, appending only the keys the .idx no longer names. * rust volume: mirror the .idx head tombstone recovery Port the Go detection and repair: an .idx opening with a run of offset-0 tombstones lost the Put rows indexing the first needles in .dat, so re-derive them at load from a header-only walk over the head of .dat and append the keys the .idx no longer names. * volume: put recovered .idx rows back in front instead of appending Appending left the offset-0 tombstone run at the head, so every later load re-walked .idx to the tail to notice the volume was already recovered, and the rows for the head of .dat sat past the .dat-tail row -- costing CheckVolumeDataIntegrity its O(1) path and breaking the ascending append order BinarySearchByAppendAtNs assumes. Rewrite .idx as the recovered rows followed by its current contents, through a temp file and a rename. .idx is back in .dat append order, so a later load stops after reading one row. * volume: keep the .idx mode when the repair replaces it The recovery renames a fresh temp file over .idx, so a fixed 0644 (Go) or whatever the umask allows (Rust) would silently widen an index an operator had locked down. Carry the mode off the file being replaced. |
||
|
Chris Lu
|
fee3fcb55a |
mount: report data sizes to df with -df.logical (#10459)
df on a mount shows the space the cluster gives up to the data: every replica of a regular volume, every shard of an ec one. That is the honest answer for capacity planning, but it is not the question a user asks when they want to know how much of their data is stored. Add -df.logical. The master reports the logical sizes alongside the raw ones: one replica per regular volume, the data shards of each ec volume counted once. Free space is divided by the copies the requested replication makes, so used plus available stays the amount of data the mount can still write, and it comes off the cluster-wide usage rather than one collection's, since capacity is cluster-wide too. Statistics through a filer resolves an unset replication to the filer's default rather than the master's, matching where the writes it is sizing for actually land. The flag governs the quota check too, so a mount has one notion of how much it is using. A filer that predates the new fields sends zeros, and the mount keeps reporting the raw sizes. |
||
|
Chris Lu
|
be81b9d5d7 |
volume: fix EC decode/reconstruct index locality under -dir.idx (#10442)
* volume: fix EC decode/reconstruct index locality under -dir.idx EC->replicated decode failed under -dir.idx and on multi-disk with "volume not found on disk". The reconstruct rebuilds the .dat on the data disk but the on-demand VolumeMount scans only the data directory, matching on .idx/.vif; with the rebuilt .idx off in the index directory it matched the volume's leftover EC .vif and skipped the volume as EC metadata. - Resolve the EC .ecx local-first: prefer the copy co-located with the shards over the shared -dir.idx copy, with a non-empty preference so a 0-byte local stub still yields to a valid sibling (the cross-disk fallback). - Co-locate the rebuilt .idx with the .dat at the end of the reconstruct so the mount finds it; sweep .ecx/.ecj from both the data and index directories on Destroy so a stale copy cannot re-mount as a phantom EC volume. - Add VolumeConsolidateIndex: once the EC shards are deleted, unmount, move the .idx/.sdx from the data disk back to the -dir.idx directory (copy fallback across filesystems), and remount. A no-op without -dir.idx. * volume: tests for EC index locality (local-first .ecx, sweep, consolidate) - NewEcVolume prefers a non-empty local .ecx over the shared index dir, and a 0-byte local stub yields to a non-empty shared copy (the #9212 fallback). - Destroy sweeps .ecx/.ecj from both the data and index directories. - ConsolidateVolumeIndex moves a co-located index back to the -dir.idx dir and keeps the volume mounted; no-op without a separate index dir. - RenameOrCopyFile moves a file and drops the source. * volume: relocate the decoded index in place, without a read gap ConsolidateVolumeIndex previously unmounted the volume, moved the index, and remounted it. Between the EC-shard delete and the remount the volume had neither a normal nor an EC form mounted, so a read landing in that window got a not-found (or was proxied away). Move the index in place instead: RelocateIndexTo takes the data-file write lock, closes the needle map and data backend, moves the .idx (and derived .sdx), then retargets dirIdx and reloads — the same close-swap-load CommitCompact uses. The volume never leaves the mounted set, so a concurrent read blocks briefly on the lock rather than failing. The test now writes a needle before consolidating and reads it back after, proving the in-place reload keeps the volume serving. * volume: address review — maintenance guard, no orphan on copy failure - VolumeConsolidateIndex now rejects the request under maintenance mode, like VolumeConfigure and the other mutating volume RPCs. - RenameOrCopyFile rolls the cross-device copy back when the source cannot be removed, so a failed move never leaves two divergent copies (the loader would keep the data-dir one while the idx-dir orphan goes stale). - RelocateIndexTo logs a failed reopen-after-failed-move instead of swallowing it, since that leaves the volume unusable until the next load. |
||
|
Chris Lu
|
2d9227747a |
volume: reject needle blob writes to read-only volumes (#10435)
* volume: reject needle blob writes to read-only volumes WriteNeedleBlob appends the blob to .dat and only then calls nm.Put. On a read-only volume the needle map is a SortedFileNeedleMap whose Put always fails, so the append is never indexed and never rolled back. Nothing upstream stops this: volume.check.disk picks its targets from the master's cached topology, which goes stale the moment a volume server marks a replica read-only itself — a failed data integrity check at load, or an EIO quarantine. Each sync attempt then grows the .dat of a replica that is supposed to be frozen by one unindexed needle, and reports it as "invalid argument", the bare os.ErrInvalid the needle map returns. Check IsReadOnly before touching .dat, same as the upload path does. * volume: say which needle and volume failed to index An index write that fails surfaced as a bare errno with no volume, no needle and no file — "invalid argument" for a read-only needle map, or a plain ENOSPC when .idx lives on its own filesystem via -dir.idx. Both were logged at V(4), so by default the operator saw only the errno the client got back. |
||
|
Yaroslav HalchenkoandClaude Code 2.1.217 / Claude Opus 4.7
|
490379bff3 |
Add codespell support with configuration and typo fixes (#10393)
* Add GitHub Actions workflow for codespell on master * Add rudimentary codespell config * Tune codespell config: skip generated code, ignore camelCase, whitelist domain terms Add camelCase/PascalCase regex to ignore common Go/Rust/JS identifiers like allLocations, publishErr, ReadInside, FlushInterval. Also skip templ-generated *_templ.go files, and whitelist a handful of short/domain-specific words (visibles, fo, te, ser, bject, unparseable, keep-alives, tread, anc, ue) that show up as false positives across the tree. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix ambiguous typos and protect false positives Fixes typos that codespell reports with multiple candidate suggestions (so `codespell -w` cannot auto-apply them), plus one inline pragma and one config entry to protect legitimate identifiers. Manual fixes (single correct answer chosen from context): - pattens -> patterns (5x) in filer/upload/shell flag help strings - finded -> found (2x) in tarantool storage.lua comment - spacify -> specify (2x) in helm chart values.yaml comment - wether -> whether in skiplist.go docstring - simpe -> simple in mq schema test case name False-positive protection: - Add `//codespell:ignore` next to `source GET's` (possessive of HTTP verb) in s3api_object_handlers_copy_stream.go - Whitelist `auther` in .codespellrc — it's a local variable meaning "authenticator" in weed/security/tls.go, not a typo of "author". Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Extend codespell ignore list: .git-meta path and thirdparty groupId Also skip `.git-meta` (scratch dir for commit messages that may contain typo words verbatim) and whitelist `thirdparty` — it appears as the literal Maven groupId `org.apache.hadoop.thirdparty` in hdfs3 poms and cannot be renamed. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [DATALAD RUNCMD] Fix non-ambiguous typos with codespell -w Auto-applied fixes to the 44 remaining single-suggestion typos across docs, comments, log messages, tests, config, and one Java pom. === Do not change lines below === { "chain": [], "cmd": "uvx codespell -w", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [], "pwd": "." } ^^^ Do not change lines above ^^^ * Revert breaking codespell fixes; whitelist unknwon and atleast Two of the auto-applied `codespell -w` fixes were false positives that would break the build/tests: - go.mod: `github.com/unknwon/goconfig` is a real Go module path — the upstream author's GitHub handle is literally `unknwon`. Renaming to `unknown` would fail dependency resolution. - test/benchmark/fuse_db/bin/{sqlite_verify.py,run_mysql.sh,run_sqlite.sh}: `atleast` is a literal CLI mode value (a string constant compared and passed as a positional argument). Rewriting to `at least` splits it into two arguments and breaks the mode check. Reverted those files and whitelisted both words in .codespellrc so future runs won't re-suggest the same broken fixes. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris Lu
|
5a54beac80 |
EC decode: read shards with the encode-time block layout (#10385)
* erasure_coding: WriteDatFile takes the encode-time dat size for the shard block layout * volume server: derive EC decode layout from the encode-time dat size, not the live extent * erasure_coding: test decode after tail deletions shrink the live extent below a large-block row * seaweed-volume: write_dat_file_from_shards takes the encode-time dat size for the shard block layout * seaweed-volume: derive EC decode layout from the encode-time dat size, not the live extent * seaweed-volume: test decode after tail deletions shrink the live extent below a large-block row * erasure_coding: reject decoding with no data shards * worker: record the encode-time dat size in the .vif * erasure_coding: fall back to the shard-derived layout only when the encode-time dat size is missing * erasure_coding: reject an ambiguous shard-derived block layout * seaweed-volume: fall back to the shard-derived layout only when the encode-time dat size is missing * seaweed-volume: reject an ambiguous shard-derived block layout |
||
|
Chris Lu
|
1f8d0a9ccf |
volume server: fill in ModifiedAtSecond on the /status volume list (#10351)
Only the heartbeat path read the .dat mtime; collectStatForOneVolume left the field at 0, so /status consumers could not tell how long a volume had been idle. |
||
|
|
8a71327324 |
fix: report short S3 ReaderAt reads (#10345)
* fix: report short S3 ReaderAt reads Problem: S3BackendStorageFile.ReadAt returned a short buffer with a nil error, hiding truncated remote data. Root cause: Every terminal io.EOF was cleared regardless of how many bytes were read. Fix: Clear io.EOF only when the requested buffer was completely filled. Validation: go test ./weed/storage/backend/... Co-authored-by: Codex <noreply@openai.com> * fix: validate S3 ReaderAt requests Co-authored-by: Codex <noreply@openai.com> * s3 ReadAt: simplify negative offset error * s3 ReadAt: stop reading once the buffer is full Skips the extra Read that only collected the terminal EOF, and bounds the loop if the server ignores the Range header and returns more data than requested, where Read on an empty slice can spin forever. * s3 ReadAt: cover full reads that arrive with io.EOF --------- Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
8bff3b3213 |
fix(volume): reject overflowing needle ID deltas (#10342)
* fix: reject overflowing needle ID deltas Problem: Parsing a file ID with a delta can wrap a valid maximum needle ID back to zero without returning an error. Root cause: Needle.ParsePath added the parsed uint64 delta without checking whether the sum exceeded the needle ID range. Fix: Compare the delta with the remaining uint64 capacity before addition and return a contextual overflow error when it does not fit. Validation: go test ./weed/storage/needle -run ^TestNeedleParsePathRejectsDeltaOverflow$ -count=1; go test ./weed/storage/needle -count=1; git diff --check 10cdaf381875492a2c752d1038797e96ff18208f..HEAD Co-authored-by: Codex <noreply@openai.com> * fix: propagate needle ID delta parse errors Co-authored-by: Codex <noreply@openai.com> * print the needle id in hex in the delta overflow error * batch delete: keep processing after a cookie mismatch * rust volume: reject overflowing needle id deltas --------- Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
10cdaf3818 |
Introduce weed shell command ec.check.replication. (#10328)
* Introduce weed shell command `ec.check.replication`.
This command performs a quick check of EC volume shard replication, reporting
volumes whose shards are over- or under-replicated. Each volume is checked
against its own data+parity ratio, obtained via
erasure_coding.EcShardsVolume{Data,Parity}Shards, so builds that derive the EC
ratio per volume report custom ratios correctly.
The name follows the shell's convention (cluster.check, volume.check.disk); the
closest normal-volume counterpart is volume.fix.replication.
* shell: ec.check.replication reports mixed under+over-replication in both lists
|
||
|
Chris Lu
|
6f816c955d |
volume.fsck: fix orphan purge against the rust volume server (#10289)
* rust volume: accept odd-length needle id hex in file ids Go formats the needle id with strconv.FormatUint and parses it back with strconv.ParseUint, neither of which pads to an even number of hex digits. hex::decode rejected such file ids with "Odd number of digits", so volume.fsck could not purge orphans from a rust volume server. Parse the needle id and cookie with from_str_radix, matching Go's ParseNeedleId and ParseCookie. * storage: emit even-length needle id hex in NeedleId.FileId volume.fsck and volume.check_disk build purge file ids here with unpadded FormatUint hex, while every other fid formatter strips whole leading zero bytes. Pad to even length so the output matches the canonical fid format and strict hex parsers accept it. |
||
|
Chris Lu
|
0ae1fdcad2 |
volume: reload a remote-tiered volume without re-entering the data lock (#10266)
LoadRemoteFile now takes dataFileAccessLock (so a live tier upload does not race the heartbeat's DataBackend read). But load() also runs it, and CommitCompact calls load() while already holding that lock, so reloading a remote-tiered volume during a compaction commit re-enters the non-reentrant lock and deadlocks. Split the locked bodies out: swapDataBackendLocked and loadRemoteFileLocked assume the caller holds dataFileAccessLock. load() uses loadRemoteFileLocked; the public LoadRemoteFile keeps taking the lock for the live tier-upload handler that does not hold it. |
||
|
Chris Lu
|
254c2a1024 |
volume: clear remote flag when tiering a volume back to local (#10262)
VolumeTierMoveDatFromRemote downloads the .dat, trims the .vif, and swaps the data backend to the local file, but left hasRemoteFile set. The volume.tier.download command masks this by unmounting and remounting right after, which reloads the flag from the trimmed .vif — but in the window before the remount the in-memory flag is wrong: doDeleteRequest would skip appending the tombstone to the freshly local .dat, and the phantom-.dat guard stays disabled. Give SwapDataBackend a hasRemoteFile argument so the backend swap and the flag move together under one lock, and route both tier directions through it: the tier-down download passes false, LoadRemoteFile passes true. The flag can no longer disagree with the live backend. |
||
|
Chris Lu
|
2d2fdeac3d |
volume: keep tier-uploaded volume reporting to master after volume.tier.upload (#10259)
volume: keep tier-uploaded volume reporting to master A live volume.tier.upload removes the local .dat and swaps the data backend to remote, but v.hasRemoteFile was only ever set when a volume is loaded from disk, so the running volume kept it false. The phantom .dat guard then saw fileCount>0, !HasRemoteFile, and a missing .dat, and stopped reporting the volume to the master. The volume vanished from the topology even though the upload succeeded and the data was in cloud storage. Set hasRemoteFile in LoadRemoteFile, the single point where a volume's backend becomes remote, so it is true both on disk-scan load and after an in-process tier upload. Route the backend reassignment through SwapDataBackend so it happens under dataFileAccessLock, closing the old backend and never racing the heartbeat's concurrent DataBackend read. Make the field atomic since the heartbeat now reads it concurrently with the tier-upload handler that writes it. |
||
|
qzhello
|
1d8a6e832c | fix(ec): detect truncated .ecx instead of treating it as clean EOF (#10217) | ||
|
Chris Lu
|
b4a99b996d |
feat(ec): EC bitrot CHECKSUM scrub on the Rust volume server (#10154)
* proto: add EC bitrot checksum messages + CHECKSUM scrub mode Mirror weed/pb/volume_server.proto byte-for-byte (field numbers + types) so the .ecsum sidecar payload is wire-identical across the Go and Rust binaries: EcBitrotProtection / EcShardChecksums / ChecksumAlgorithm, VolumeScrubMode.CHECKSUM=4, and VolumeEcShardsCopyRequest.copy_ecsum_file. No code uses them yet — the .ecsum format, producer, mount-load, copy, and scrub land in following commits. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): port the .ecsum bitrot checksum module ec_bitrot.rs mirrors weed/storage/erasure_coding/ec_bitrot.go: the .ecsum sidecar format (14-byte big-endian ECSU header + CRC32C over a prost-serialized EcBitrotProtection payload), the per-shard per-block CRC32C producer (ShardChecksumBuilder), save/load with payload self-integrity, manifest validation, status resolution, and verify_shard_file_blocks for the CHECKSUM scrub. A byte-exact test pins the serialized bytes against the Go reference's identical constant so a format drift in either binary fails loudly. Producer wiring (encode/vacuum), mount-load, copy, and the mode-4 dispatch land in following commits. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * test(ec): pin .ecsum sidecar bytes for cross-binary interop Deterministic EcBitrotProtection -> exact on-disk bytes, asserted against a canonical constant on BOTH sides (this test and ec_bitrot.rs), so a format drift in either binary fails its own suite rather than silently desyncing a Go-written .ecsum from a Rust-written one. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): write the .ecsum bitrot sidecar during EC encode write_ec_files now feeds each shard's bytes through a per-shard ShardChecksumBuilder as it writes them, then persists the generation-0 sidecar (<base>.ecsum) alongside the shards — mirroring weed's WriteEcFiles + SaveBitrotSidecar. Best-effort: a failed sidecar write leaves the generation unprotected rather than failing the encode. A test confirms the produced sidecar validates and its per-block CRCs match every on-disk shard. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): load the .ecsum at mount + EcVolume::checksum_scrub EcVolume now loads and validates its generation-0 .ecsum sidecar at mount, caching the parsed protection + BitrotStatus (Off/On/Invalid), and exposes bitrot_protection() mirroring Go's EcVolume.BitrotProtection(). checksum_scrub() verifies every locally-held shard's raw bytes against the sidecar block CRCs — the only path that exercises cold parity shards — reporting mismatched shards without mutating anything; a wholesale mismatch beyond parity is flagged as a suspect sidecar rather than mass shard corruption. Mirrors Go's ChecksumScrub. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(scrub): dispatch EC CHECKSUM (mode 4) to checksum_scrub Accept VolumeScrubMode.CHECKSUM=4 and route it to EcVolume::checksum_scrub, accumulating blocks scanned + mismatched shards into the scrub response, plus the CHECKSUM scrub-mode metric label. Read-only bitrot verification over local shards. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): copy the .ecsum sidecar during VolumeEcShardsCopy Honor copy_ecsum_file: when set, copy the generation-0 .ecsum alongside the shards so protection travels with them, mirroring Go's non-2PC copy path. Tolerant of a missing source (empty stream) — the 0-byte file is dropped so mount sees no sidecar (protection off) rather than a truncated/invalid one. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): remove the .ecsum sidecars when destroying an EC volume remove_ec_volume_files now clears <base>.ecsum (and any versioned .ecsum.v<N>) from the data and idx dirs, so a vid reuse can't load a stale sidecar. Mirrors Go's removeBitrotSidecars. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * style(ec): align bitrot comments and test setup for merge-cleanliness Match the shared bitrot code (write_ec_files, encode_one_batch, checksum_scrub, the encode sidecar test) to the canonical wording/layout so the volume-server Rust port stays line-aligned across trees, keeping periodic merges conflict-free. No behavior change. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo |
||
|
Chris Lu
|
bea1357d38 |
ec: skip physically near-full disks when placing EC shards (#10167)
EC placement scored destinations purely by free EC shard slots (derived from maxVolumeCount) and shard counts, blind to real disk fullness — the same defect as volume balancing. A disk that is physically full but still shows free EC slots kept being chosen, and EC shard bytes are captured by statfs free space yet not by any slot accounting, so the slot math is exactly the metric that can't see EC fullness. Treat a disk at/above 90% physical usage as having zero free EC slots at snapshot-build time, so every existing freeSlots>0 placement predicate excludes it. Applied in all three snapshot builders (shell countFreeShardSlots, the shared ecbalancer FromActiveTopology, and the worker ec_balance buildBalancerTopology) via the shared balancer.DiskTooFullAfter gate. Servers not reporting disk bytes fall back to slot-only behavior. ec.rebuild recovery is left ungated so shard recovery can still complete onto fuller disks. |
||
|
Chris Lu
|
77bf2a3ab0 |
volume.balance: gate on real physical disk usage (fixes #10160) (#10162)
* shell: add volume.balance -byDiskUsage to balance by actual data The default balancer ranks servers by slot density, dividing used volumes by MaxVolumeCount. When MaxVolumeCount is configured higher than the disk can hold, a physically near-full server looks nearly empty and gets picked as the move target, so balancing drains less-full servers onto an already-full one. -byDiskUsage ranks servers by the actual data they hold (sum of volume sizes) instead, so the fullest-by-data server is treated as full and balancing drains it. It assumes comparable disk sizes per disk type and still respects each server's free volume slots. Default behavior is unchanged. * plumb physical disk usage into topology, gate volume.balance on it Volume servers now report each disk's filesystem total/free bytes in the heartbeat, and the master stores them in DiskInfo. volume.balance uses them to skip any move target whose disk is already near full (-maxDiskUsagePercent, default 90), so an over-configured maxVolumeCount can no longer make a physically full server look empty and get drained onto. The gate judges each server against its own disk, so heterogeneous disk sizes are fine; servers that do not report bytes fall back to slot-only behavior. Rust seaweed-volume mirrors the heartbeat reporting. * admin: report real physical disk capacity when volume servers provide it The dashboard estimated server capacity as maxVolumeCount * volumeSizeLimit, which overstates it when maxVolumeCount is set higher than the disk holds. Prefer the filesystem capacity now reported per disk, falling back to the estimate for servers that do not report it. * worker: gate automatic balance on physical disk fullness too The maintenance balance worker selects the least slot-utilized server as the move destination, so an over-configured maxVolumeCount makes a physically full server look empty and get drained onto — the same defect as the shell command. Now that DiskInfo carries real disk bytes, skip any destination whose disk is at/above 90% used (per server, against its own disk); a full server can still be a source. When every candidate destination is full, create no tasks. Servers that do not report disk bytes are not gated. * balance: share the physical-disk-fullness gate between shell and worker The shell volume.balance command and the maintenance balance worker each grew their own copy of the disk-fullness gate (targetDiskTooFull / destinationDiskTooFull) and a maxDiskUsagePercent=90 constant. Pull both into weed/topology/balancer (DiskTooFullAfter + DefaultMaxDiskUsagePercent) so the policy has one home and the two balancers can't drift. * balance: harden the physical-disk gate Guard against a nil DiskInfo in the byte/slot lookups. Let a zero disk-capacity report clear previously stored bytes (0 means "not reported" for bytes, unlike maxVolumeCount), so a server that stops reporting falls back to slot-only instead of trusting stale capacity. In the worker, charge each planned move's bytes to its destination within a detection cycle so the gate sees a target fill up rather than only its heartbeat-time free space. Note the per-location capacity summing assumes one location per filesystem (the used ratio the gate relies on stays correct regardless; absolute capacity can over-report). |
||
|
Chris Lu
|
41d6c821ba |
feat(topology): report empty disks (per-disk type + capacity in heartbeat) (#10166)
* fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk
DiskId 0 doubles as the first physical disk (Locations[0]) and the
protobuf "unset" default. SplitByPhysicalDisk folded every DiskId-0
record onto the aggregate DiskId whenever that was non-zero, so on a
multi-disk node the first disk's volumes merged into whichever disk
held volumes[0]: the node reported one fewer disk, the sibling showed
~2x volumes, and per-disk max was smeared across the survivors. This
surfaced as cluster.status and volume.list undercounting disks.
Only treat 0 as unset when no record carries a non-zero DiskId; with a
mix, 0 is a real disk and keeps its own entry.
* fix(admin): resolve physical disk 0 in active-topology indexes
rebuildIndexes re-derived each volume/EC record's physical disk id with
the same "DiskId 0 means unset" heuristic SplitByPhysicalDisk used, so
the two agreed only by sharing the bug. Now that SplitByPhysicalDisk
keeps disk 0 distinct, the duplicated heuristic would fold disk-0 records
onto a sibling while at.disks kept them on disk 0; GetVolumeLocations and
GetECShardLocations then matched no record and silently dropped every
volume and EC shard on the first disk, starving balance and EC tasks.
Build the indexes from the same SplitByPhysicalDisk reconstruction that
builds at.disks, so the keys always resolve. One source of truth instead
of a parallel normalize.
* fix(ec): allow physical disk 0 as preferred EC shard target
pickBestDiskOnNode gated its result on bestDiskId != 0, but 0 is both a
valid physical disk and the uint32 zero value, so a best-scoring disk 0
was discarded and the non-matching fallback returned instead. Gate on
bestScore.
* test(admin): cover EC-shard index resolution for physical disk 0
rebuildIndexes builds ecShardIndex the same way as volumeIndex; pin the EC
path too so a shard on disk 0 keeps resolving via GetECShardLocations.
* proto: per-disk type/capacity in DiskTag, DiskInfo.physical_disks
DiskTag gains type + max_volume_count so the heartbeat can describe every
physical disk, including ones holding no volumes or EC shards. DiskInfo
gains physical_disks so the master can hand the full per-type disk set to
per-physical-disk consumers.
* feat(volume): report each physical disk's type and capacity
CollectHeartbeat fills DiskTag.type and the per-disk effective max for
every location, so the master can account for disks that hold no volumes
or EC shards yet. Rust heartbeat mirrors it.
* feat(master): surface empty disks in the per-physical-disk view
The master records each disk's type and max from DiskTags and lists them
on DiskInfo.physical_disks per type, including disks with no volumes or
EC shards. SplitByPhysicalDisk enumerates that full set and gives each
disk its exact max, so cluster.status, volume.list and the admin
topology count and can target empty disks. Without physical_disks the
even-split fallback is unchanged.
* fix(master): clamp per-disk free at zero for over-allocated disks
In the exact-max path FreeVolumeCount could go negative when a disk holds
more volumes than its max; a negative would reduce the node's summed free
and block placement on healthy disks. Clamp at 0.
* fix(master): rebuild disk tags fresh each heartbeat
DiskTags is the full authoritative per-disk list every heartbeat, so
rebuild dn.diskTags from scratch like dn.diskBackends; merging left stale
entries for removed disks.
* fix(master): keep zero-capacity disks in physical_disks
A disk reporting max 0 (an unavailable disk) is a valid physical disk,
not a signal to drop it. List every disk of the type, but only emit
physical_disks when the node reports real per-disk capacity, so an older
server sending all zeros still falls back to the aggregate split.
* test(volume): cover disk-space-low per-disk max in heartbeat
Assert DiskTag.max_volume_count follows the used-slots override when a
location is low on space, matching the per-type max_volume_counts.
* chore: trim comments on the empty-disk change
Drop narration; keep only the non-obvious why (disk-0 sentinel, exact-max
free clamp, EC slots not subtracted, all-zeros fallback).
* refactor(master): merge per-disk tags and capacity into one map
diskTags and diskBackends were parallel maps keyed by the same DiskId and
filled together from DiskTags. Fold them into one diskMetas map of
{tags, type, max}.
* refactor(proto): per-disk max as a map keyed by disk id
physical_disks was a repeated {disk_id, max_volume_count} whose fields
duplicated DiskInfo's own disk_id/max_volume_count. A map<uint32,int64>
keyed by disk id expresses "max per disk" directly, drops the extra
PhysicalDiskInfo message, and the consumer reads it as the disk set.
* docs(proto): note DiskInfo.disk_id's two meanings
Identity on a per-physical-disk DiskInfo (from SplitByPhysicalDisk),
representative fallback on the type-keyed aggregate.
|
||
|
Lisandro Pin
|
cac83bb4a8 |
Fix scrubbing of deleted needles on EC volumes. (#10130)
EC volumes do not propagate deletions to all shard indexes, so it is possible to run scrubbing on a volume where a deleted needle is still present in the index, or a needle deleted from the index is still present on the volume. On either scenario, scrubbing will fail due to size mismatch errors. This PR reworks the scrubbing logic so needle size mismatches are ignored in such scenarios. Scrubbing can still be forced to check deleted needles (f.ex. to discover index inconsistencies); this option will be exposed in RPCs and `weed shell` on a follow-up PR. |
||
|
Chris Lu
|
acbb6f7550 |
fix(scrub): don't flag offset-0 logical tombstones in volume scrub (#10148)
* fix(scrub): don't flag offset-0 logical tombstones in volume scrub A remote-tier delete records a tombstone at .idx offset 0 with no physical .dat bytes. Full scrub double-flagged a healthy remote-tiered volume with deletes: scrubVolumeData counted the tombstone's GetActualSize(-1)=32 toward totalRead (want > physical .dat), and CheckIndexFile treated it as occupying [0,31] and flagged the first live needle as overlapping. Skip offset-0 logical tombstones from both the size reconcile and the overlap check; they are still counted for the index-size check. Local deletes (offset != 0) are unaffected. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * fix(scrub): mirror offset-0 logical tombstone handling into Rust Same fix as the Go volume_checking.go + idx/check.go change: Volume::scrub skips offset-0 logical tombstones from total_read, and check_index_file excludes them from the overlap check (still counted for the index-size check). Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo |
||
|
Chris Lu
|
c9f2ef9ef7 |
fix(ec): suppress deleted-needle size mismatch in EC LOCAL scrub (#10147)
* fix(ec): suppress deleted-needle size mismatch in EC LOCAL scrub EcVolume.ScrubLocal reassembles each fully-local needle and ReadBytes-checks it, but appended every error unconditionally. A needle the .ecx still reports live while its reassembled on-disk header carries size 0 (delete state disagrees between index and header) is not corruption — the LOCAL twin of the #10130 fix for the FULL path. Suppress the ErrorSizeMismatch in that case; genuine (non-zero) size mismatches and CRC/tail errors are still reported. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * fix(ec): mirror EC LOCAL scrub deleted-needle suppression into Rust Same suppression as the Go EcVolume.ScrubLocal change: a NeedleError::SizeMismatch whose on-disk header size is 0 against a live index entry is a delete-state disagreement, not corruption. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo |
||
|
Chris Lu
|
2efc0e1656 |
ec: recover EC shards whose .ecx index lives only on a peer server (#10108)
* ec: recover EC shards whose .ecx index lives only on a peer server A volume server that boots with EC shard files on disk but no .ecx index on any local disk cannot mount the shards, so the master never learns about them. ec.rebuild works off master-registered shards, so it sees the volume as short and gives up even though the shard data is intact. Add an operator-triggered recovery: VolumeEcShardsMount gains a recover_missing_index flag that makes the volume server fetch the missing .ecx (plus .ecj/.vif) from a peer holding it and mount the on-disk shards. ec.rebuild runs this across the cluster before planning, so orphaned shards register and the rebuild sees the true shard set. .ecx is an immutable encode-time index, identical on every holder. .ecj is a per-holder deletion journal that differs across holders, so the recovered node adopts the source peer's deletion view, like a balanced or rebuilt shard does. * ec: mirror missing-index recovery into the Rust volume server Port the #10104 recovery to seaweed-volume so the Rust volume server self-heals the same layout: EC shards on disk with the .ecx index only on a peer. Adds collect_ec_volumes_missing_index / mount_recovered_ec_shards to the store, recover_missing_ec_indexes (master LookupEcVolume + peer CopyFile fetch + mount) to the server, and the recover_missing_index flag on VolumeEcShardsMount. .ecx is the immutable encode-time index, identical on every holder. .ecj is a per-holder deletion journal, so the recovered node adopts the source peer's deletion view, matching the Go path. |
||
|
Chris Lu
|
bc257fe72e |
volume: detect phantom volumes held open as deleted FDs (#10011)
* volume: detect phantom volumes held open as deleted FDs
Add disk-file validation in heartbeat collection to prevent reporting
phantom volumes that exist in memory but are deleted from disk. This
unblocks re-replication when files are unlinked while the volume server
holds them open via file descriptors.
Cache disk checks per-volume with 30-second TTL to avoid syscall overhead.
Implement in both Go and Rust volume servers.
* volume: make last_disk_check_ns field public for heartbeat access
* volume: only check for phantom volumes when size > 0
Skip phantom volume detection for zero-size volumes (e.g., test volumes).
Phantom volumes only occur when disk files are deleted while the process
holds them open via FDs - which requires the volume to have had actual data.
Test volumes with zero size should not trigger disk file existence checks.
* volume: only check for phantom volumes when size > 0
Skip phantom volume detection for zero-size volumes (e.g., test volumes).
Phantom volumes only occur when disk files are deleted while the process
holds them open via FDs - which requires the volume to have had actual data.
Test volumes with zero size should not trigger disk file existence checks.
* volume: only check for phantom volumes if file_count > 0
Use file_count as the indicator for whether a volume held actual data,
rather than volume size. Phantom volumes only occur when a volume that
had files is deleted while the process holds open file descriptors.
Test volumes with no file count won't trigger the phantom detection check.
* volume: stat the .dat with its extension when detecting phantom volumes
DataFileName()/IndexFileName() return the extensionless base path, so os.Stat
saw every volume's files as missing and dropped it from the heartbeat, leaving
the master with no locations and breaking deletes/lookups. Stat FileName(".dat")
instead, skip remote-tiered volumes whose .dat lives in cloud storage, and
re-check a missing file every heartbeat rather than caching the negative.
|
||
|
Chris Lu
|
5e8152b81c |
storage: register tier backends at the binary composition root (#9989)
The s3 and rclone tiered-storage backends were registered via blank imports in weed/storage (volume_tier.go and volume_info/volume_info.go). That forced every library consumer of weed/storage -- weed/shell, and through it external tools -- to link aws-sdk-go and, under the rclone build tag, the full rclone backend set and its cloud-storage SDKs, even though those consumers never tier volumes. Move the registrations into a new weed/storage/backend/all aggregator and blank-import it once from weed/command, the binary's composition root. The weed binary still registers both backends; weed/storage and its library consumers no longer pull the backend SDKs into their dependency graph. |
||
|
Chris Lu
|
33df4fe2c4 |
storage: nil-safe ReplicaPlacement.String()
Guard a nil receiver like TTL.String() already does, so formatting a zero-value VolumeInfo can't depend on fmt's panic recovery. |
||
|
Chris Lu
|
9c10d64ae9 |
shell: show remote storage name/key in volume.list output (#9987)
VolumeInfo.String() dropped RemoteStorageName/RemoteStorageKey, which are useful when debugging volume tiering. Append them when the volume is remote. |
||
|
Chris Lu
|
7e608c877a |
refactor(ec_balance): make the balance planner per-volume ratio-capable (#9960)
* refactor(ec_balance): make the balance planner per-volume ratio-capable Thread a per-volume EC ratio through the balance planner: Plan resolves each volume's data/parity from a new Options.VolumeRatio (falling back to the collection Ratio, then the build default, when it reports 0), and keys the global phase's ratio maps by volume instead of collection. The shell and worker balance paths build the per-volume lookup from each shard's heartbeat via the new ecbalancer.VolumeShardRatio. In OSS this is behavior-preserving: VolumeShardRatio returns 0 because the per-volume data_shards/parity_shards heartbeat fields are an enterprise feature, so every volume falls back to the collection ratio -- the existing standard-scheme behavior. The refactor keeps the shared planner in sync with the enterprise fork, which overrides VolumeShardRatio to classify and spread a mixed-ratio collection by each volume's own data/parity split. * perf(ec_balance): hoist the collection ratio out of the per-volume loop The collection ratio is constant for every volume in a collection, so resolve it once per collection instead of per volume; a custom Ratio func may do map lookups or locking. Addresses a review comment. |
||
|
Chris Lu
|
138220b961 |
fix(ec): recover EC shards with the volume's own ratio, not the build default (#9958)
* fix(ec): recover EC shards with the volume's own ratio, not the build default recoverOneRemoteEcShardInterval rebuilt a missing shard with a hardcoded 10+4 Reed-Solomon matrix (and counted sufficiency / iterated shards against the 10+4 constants). For a custom-ratio volume (e.g. 9+3) that reconstructs with the wrong matrix and corrupts the recovered bytes, and cachedLookupEcShardLocations could wrongly reject a degraded but recoverable custom-ratio read. Use the volume's own ECContext (loaded from its .vif) for the encoder, the shard-iteration bound, and the data-shard sufficiency checks. In OSS the ratio is always 10+4 so this is a no-op; it brings the Go volume server in line with the Rust one, which already reconstructs with the volume's ratio. * fix(ec): close data races in the EC read-recovery path Address review: the freshness check in cachedLookupEcShardLocations read ecVolume.ShardLocations / ShardLocationsRefreshTime without the lock while recover goroutines mutate them via forgetShardId -- snapshot both under ShardLocationsLock.RLock(). The recover goroutines also wrote the shared is_deleted return concurrently -- collect it via an atomic and fold it in after they join. Also size availableShards/missingShards by the volume's ECContext ratio rather than the 10+4 constants. |
||
|
Chris Lu
|
ef5fee6c28 |
fix(storage): delete/unmount every copy of a duplicate volume id (#9954)
* fix(storage): delete and unmount every copy of a duplicate volume id NewStore has no cross-disk duplicate guard (unlike the Rust volume server, which refuses to start in that state), so a stale twin of a volume id can mount on a second disk after a disk repair. DeleteVolume and UnmountVolume returned after the first matching disk, leaving the twin to survive and re-register as the volume's content. Walk every disk and act on all copies, emitting one heartbeat delta per copy. * fix(storage): surface partial delete/unmount failures across duplicate copies Address review: if removing one copy of a duplicate volume id fails with a real error (disk IO, permissions), the loop logged it and could still return success once another copy was removed -- leaving the stale copy to re-register, the exact divergence this guards against. DeleteVolume and UnmountVolume now accumulate such errors and return them (still attempting every disk), so a copy left behind is never reported as success. Add a DeleteVolume duplicate-copies regression test. |
||
|
Chris Lu
|
284796c7b6 |
fix(ec): fence stale-worker EC shard cleanup by encode generation (#9953)
* feat(ec): add encode_ts_ns to the EC task params, shard-unmount, and shard-delete RPCs The generation fence for stale EC-worker cleanup needs the encode generation on three messages: ErasureCodingTaskParams (admin issues it), VolumeEcShardsUnmountRequest, and VolumeEcShardsDeleteRequest (the worker carries it to the volume server). Additive fields only; 0 preserves the existing unfenced behavior. Mirror the two volume-server fields in the Rust volume server's proto copy. * feat(ec): issue the EC encode generation from the admin and carry it on the worker Stamp each EC proposal's encode_ts_ns from the admin's per-cycle DetectionSequence (a single-clock value) so generations are globally ordered even though detection runs on a rotating worker. The worker writes that generation into the distributed .vif and passes it on its shard unmount/delete RPCs; it falls back to a local timestamp for the .vif only on the unfenced legacy/shell path (keeping the read guard on). * fix(ec): fence the stale-worker EC shard unmount and teardown by generation A reaped-but-still-running EC worker's cleanupStaleEcShards issued a generation-blind unmount + full teardown that could unmount and then overwrite a newer run's live shards on a shared node. Both RPCs now carry the encode generation: the volume server unmounts/deletes a disk only when its .vif generation is strictly older than the request, and preserves a same-or-newer generation, a generation-0 (recovered or pre-upgrade) volume, and an unreadable .vif. Unload is per-disk, never node-wide. Request generation 0 keeps the blanket teardown for the shell pre-encode cleanup and pre-upgrade callers. Mirrored in the Rust volume server. * test(ec): cover the generation-fenced teardown and unmount End-to-end volume-server tests: a fenced FullTeardown wipes a strictly- older generation, preserves a newer one, preserves a generation-0 volume, and blanket-wipes on request generation 0; the gen-aware unmount preserves a same-or-newer mounted generation; and the .vif generation reader handles present/absent/no-config cases. * test(ec): pin the fenced .vif==teardown generation and the unreadable-.vif preserve A fenced run must stamp the admin generation verbatim into the .vif so it matches the generation sent on the teardown RPCs; add a regression test that sets the task generation and asserts the .vif carries it exactly. Also cover the present-but-unparseable .vif case (reads as generation 0, preserved) and correct the readEcGenerationTsNs docstring accordingly. * fix(ec): surface EC full-teardown filesystem errors in the Rust volume server remove_ec_volume_files(_full_teardown) discarded every fs::remove_file error, so a teardown that failed on permissions or a full disk still returned full_teardown_done=true and left stale artifacts to collide with the next encode. Return io::Result, ignore NotFound, propagate the first real error, and have the teardown RPC surface it -- matching the Go contract. The best-effort reconcile/load-cleanup callers keep ignoring it. * refactor(ec): reuse the EC volume lookup on unmount and short-circuit the gen read Address review: the Rust unmount fence reuses the ec_vol it already fetched instead of a second find_ec_volume; the Go .vif generation reader breaks out of the data/idx loop early when the two dirs are the same. |
||
|
Chris Lu
|
da243b9423 |
fix(ec): group orphan-source completeness by encode generation (topology encode_ts_ns) (#9952)
* feat(ec): carry the encode generation through the topology heartbeat Add encode_ts_ns (field 14) to VolumeEcShardInformationMessage and populate it from each EC volume's .vif identity. The volume server emits it on the full and incremental heartbeats; the master stores it on EcVolumeInfo and re-emits it via GetTopologyInfo, so the admin/worker layer can see which encode run produced each shard set. Field 14 avoids the enterprise fork's reserved 10-13. Mirror the proto field and both heartbeat emit sites in the Rust volume server. * fix(ec): group orphan-source shard completeness by encode generation countExistingEcShardsForVolume ORed EcIndexBits across every disk, so two interrupted encode runs whose shard sets overlap unioned into a false-complete set -- triggering the orphaned-source delete while no single generation was actually complete. Group shards by encode_ts_ns and return the largest single generation's count, so the trigger fires only when one run holds the full set. Shards from pre-upgrade servers (encode_ts_ns==0) form their own bucket. The heartbeat carries one encode_ts_ns per (volume, disk), so this separates generations on different disks; same-disk mixing is prevented upstream by the pre-encode artifact wipe and the cross-run read guard. * fix(ec): guard against a nil Ec shard info entry in the generation count Defensive: a manually-constructed or corrupted topology could carry a nil entry in EcShardInfos. Skip it rather than dereference. * fix(ec): carry the encode generation on the EC shard unmount delta The mount delta sets EncodeTsNs; the unmount deletion delta left it 0. Populate it from the Ec volume before unloading so both incremental deltas are consistent (the Rust volume server already does this via its snapshot diff). |
||
|
Chris Lu
|
26754fca4d |
fix(ec): don't fabricate a stub .vif when mounting an EC volume (#9951)
When an EC volume's .vif was missing, NewEcVolume wrote a stub holding only the version. That stub implies the default 10+4 ratio with DatFileSize=0 and no encode identity, which the custom-ratio resolver and the startup credibility checks then read as an authoritative config -- masking the real ratio of a custom-ratio volume and defeating the byte-exact .vif gate. Mount with in-memory defaults instead and leave the real .vif to the encoder or a recovery tool. The Rust volume server already behaves this way. |