* shell: dial scrubbed volume servers on their registered gRPC port
volume.scrub, ec.scrub and volume.server.state built each data node's
address from its HTTP address alone, so a volume server whose gRPC port
is not the HTTP port + 10000 was dialed on the wrong port.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: resolve -node volume servers to their registered gRPC port
An explicit -node/-nodes list was dialed on HTTP port + 10000 too.
Match each entry against the topology and take the data node's gRPC
port; an address the master does not know is used as given.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep a gRPC port given in -node
A node given as host:port.grpcPort was rewritten to the registered gRPC
port; only a bare host:port takes it from the topology.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: dial -node volume servers when the topology is unavailable
An explicit node list did not need the master before; keep using it as
given if the volume list cannot be fetched. Selecting every node still
requires the topology.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: bound the topology lookup for an explicit -node list
With the master down, volumeServer.state -nodes waited on the master
forever instead of reaching the given volume servers. Give the lookup a
30s deadline and fall back to the addresses as given.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: resolve volume server addresses from a topology-only listing
Only the data nodes are needed to find each server's gRPC port, so ask
the master for the topology without its volumes and ec shards. A large
cluster no longer risks running past the -node lookup deadline and
falling back to the HTTP port + 10000.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: skip the topology lookup when every -node names its gRPC port
A host:port.grpcPort entry is dialed as given, so a list made only of
those does not need the master and no longer waits on it during an
outage.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Introduce a new `READS` scrub mode.
`READS` performs a full volume scrub but, unlike `FULL`, it will attempt to
reconstruct data for missing/damaged shard intervals from other shards in the cluster
when necessary.
The goal of this check is to ensure that EC volume contents _are readable by Seaweed_
even on a degraded storage state, by exercising parity data which is not read in `FULL`
mode. This is useful not only to validate data is user-readable, but also to detect potential
parity shard issues which may be difficult to pinpoint otherwise - particularly for older
volumes lacking sidecar data, and hence unaffected by `CHECKSUM` scrubs.
For regular volumes, this operation is equivalent to `FULL`.
Example:
```
> ec.shard.unmount --volumeId=1 --shardId=0,3,11 --delete --apply
Live shard topology for volume ID 1 (14 shards):
0@10.200.18.89:9001
1@10.200.18.89:9002
2@10.200.18.89:9003
3@10.200.18.89:9004
4@10.200.18.89:9005
5@10.200.18.89:9006
6@10.200.18.89:9007
7@10.200.18.89:9008
8@10.200.18.89:9009
9@10.200.18.89:9013
10@10.200.18.89:9010
11@10.200.18.89:9011
12@10.200.18.89:9012
13@10.200.18.89:9020
Will unmount + delete 3 shard(s):
0@10.200.18.89:9001
3@10.200.18.89:9004
11@10.200.18.89:9011
Unmounting shard 0@10.200.18.89:9001 for volume ID 1...
Deleting shard 0@10.200.18.89:9001 for volume ID 1...
Unmounting shard 3@10.200.18.89:9004 for volume ID 1...
Deleting shard 3@10.200.18.89:9004 for volume ID 1...
Unmounting shard 11@10.200.18.89:9011 for volume ID 1...
Deleting shard 11@10.200.18.89:9011 for volume ID 1...
All done!
> ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=full
using FULL mode
Scrubbing 10.200.18.89:9002 (1/1)...
Scrubbed 6 EC files and 1 volumes on 1 nodes
Got scrub failures on 1 EC volumes and 1 EC shards :(
Affected volumes: 10.200.18.89:9002:1
Affected shards: 10.200.18.89:9002:1:0
> ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=reads
using READS mode
Scrubbing 10.200.18.89:9002 (1/1)...
Scrubbed 6 EC files and 1 volumes on 1 nodes
```
* ec: report the shards a READS scrub had to rebuild
A READS scrub that recovers an interval was recording nothing, so a volume
missing three shards came back clean and nobody repaired it. The unreadable
shard is now recorded before the rebuild is attempted: READS reports the same
broken shards as FULL and differs only in whether the needles themselves
failed, which is the signal worth having - shards are gone, data is still
there.
forceDeletedNeedlesCheck now applies to READS as well, in the shell and in the
RPC guard: it runs the same needle walk as FULL.
Regenerated the proto instead of hand-editing it, so the pancis typo (which
protoc-gen-go-grpc emits into eight other files here) and the header whitespace
stay as generated.
Mirrors into the Rust volume server, which also now honors
force_deleted_needles_check rather than hardcoding it off.
Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq
* ec: answer a deleted needle from a READS rebuild as deleted
#11020 gave the Rust recovery a deleted flag alongside its bytes, and it
answers a deleted needle with no bytes at all. The READS scrub appended that
empty answer, which does not compile against the new signature and, once it
did, would leave the needle short and report the size mismatch as damage.
Zero-fill the interval instead, the way the direct read beside it already
does: the assembled needle then reaches read_bytes as the delete-state
mismatch the walk already tolerates. Go takes the same branch off the flag
its recovery returns, rather than discarding it.
Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq
---------
Co-authored-by: Lisandro Pin <lisandro.pin@proton.ch>
On volumes failing scrubs, the detail output can get very verbose, which makes
reading results difficult. Most users won't care about this information to
begin with - just whether or not volumes pass scrub tests.
This MR gates the display of scrub result details behind a `--details` flag.
* Give the `ScrubVolume()` RPC an option to flag found broken volumes as read-only.
Also exposes this option in the shell `volume.scrub` command.
* Remove redundant test in `TestVolumeMarkReadonlyWritableErrorPaths`.
417051bb slightly rearranges the logic for `VolumeMarkReadonly()` and `VolumeMarkWritable()`,
so calling them for invalid volume IDs will actually yield that error, instead of checking
maintnenance mode first.
