7 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	37bf1cd91d	volume: validate copy/tail source addresses before dialing (#11390) ... * pb: stop exiting the process on malformed server addresses ServerToGrpcAddress and GrpcAddressToServerAddress called glog.Fatalf when hostAndPort could not parse the port, which os.Exit(255)ed the whole process. A caller-supplied copy or tail source address reached this path synchronously in the serving goroutine, so one anonymous VolumeCopy with a non-numeric port terminated the volume server. Log the parse error and return the input unchanged instead: the dial or request that consumes the address then fails as an ordinary error. * volume: validate copy and tail source addresses before dialing VolumeCopy, VolumeEcShardsCopy and VolumeTailReceiver dial a caller-supplied source address (SourceDataNode / SourceVolumeServer) with no endpoint validation, so an anonymous caller could aim the volume server at loopback, link-local (cloud metadata) or other unintended destinations and read dial behavior back as a connectivity oracle. Apply the same peer-target deny list FetchAndWriteNeedle uses for replica targets: the source must be a bare host:port whose host is not loopback, link-local or unspecified; cluster peers stay reachable on private networks, and -volume.allowUntrustedRemoteEndpoints opts out. The loopback-using copy tests set the flag to keep exercising the copy path in process. * rust volume: validate copy and tail source addresses before dialing Mirror the Go guard on the Rust volume server: volume_copy, volume_ec_shards_copy and volume_tail_receiver dial a caller-supplied source address, so run it through validate_replica_target first (bare host:port; no loopback, link-local or unspecified hosts; private peers stay allowed). --volume.allowUntrustedRemoteEndpoints opts out; the test fixture and the Rust test-cluster launcher set it so loopback sources in tests keep working. * volume: pin validated copy/tail source addresses at dial time validateReplicaTarget resolves the source hostname once, but the gRPC client resolved it again at connect, leaving a DNS-rebinding window for hostname sources. The copy and tail source dials now run through the same guardedDialerPolicy the remote-storage path uses, so every resolved address is re-checked against the replica deny list (private peers allowed) immediately before the TCP connect. guardedDialerPolicy also moves to util.OutboundDialContext so the guarded path keeps the -ip.bind source binding the default gRPC dialer had. The Rust volume server mirrors this with connect_guarded, a tonic connector that resolves, re-checks each address, and connects to the first passing IP; handlers use it whenever the untrusted-endpoint opt-out is off. A handler-level test now exercises the enabled validation branches for all three source-taking RPCs. * pb: return empty server address for malformed grpc addresses GrpcAddressToServerAddress used to return the unparseable input on a hostAndPort failure, so a malformed raft address (e.g. "host:abc") flowed into admin dashboard master maps unchanged. Return an empty string instead, skip empty conversions at the two raft-cluster merge sites, and drop the now-stale comment about the fatal exit the earlier commit removed. * test: opt erasure-coding loopback clusters out of the remote endpoint guard The erasure-coding suites drive VolumeEcShardsCopy / VolumeCopy between volume servers bound to 127.0.0.1, which the copy/tail source guard now rejects by default. Pass -volume.allowUntrustedRemoteEndpoints to the test volume launches, matching what the volume_server framework harnesses already do. * admin: only claim fallback master leadership on an empty raft response A nonempty RaftListClusterServers response whose entries were all rejected left masterMap empty, so the fallback marked the reachable current master as leader the same way a genuinely empty (non-raft) response does. Track whether the successful response returned zero servers and only promote the fallback master then.	2026-09-18 12:55:47 -07:00
Chris Lu	f4bad510c9	test/ec: pin rack in seedAndSpread volume.grow to stop silent no-ops (#11299) ... seedAndSpread() calls `volume.grow -dataNode X` without pinning the rack. The master's grow picks the rack by weighted-random when -rack is unset, and only one of the three racks holds the requested data node, so an unpinned grow lands on the wrong rack two times out of three. The VolumeGrow RPC swallows the "No matching data node" failure for non-cache collections, so those grows count as success without creating a volume. The per-server cap (maxGrowsPerServer=4) is then exhausted by silent no-ops before the volumes ever spread, and seedAndSpread times out with "volumes never spread across >=2 disks on all 3 nodes". Pin -dataCenter dc1 and -rack rack{i} alongside -dataNode so every grow reaches the target node. This removes the timing-sensitive assumption that made TestECVacuumDuplicateShardClaimAcrossDisks flaky.	2026-09-13 21:05:55 -07:00
Chris Lu	5d8a463b3e	test/ec: fix EC interruption matrix slot exhaustion (#11295) ... * test/ec: fix EC interruption matrix slot exhaustion The EC integration test cluster (test/erasure_coding/chaos_lifecycle_test.go) configured each disk with -max 4 and the seedAndSpread spread loop fired volume.grow -count 4 every 2 s with no per-server cap. Because the master topology lags the volume.grow writes, the loop re-fired before the prior grow was visible, over-filling disks to capacity. A full disk leaves zero free EC shard slots (failing the cluster-wide capacity check with "no free ec shard slots") and drops the source disk below the encode's FreeVolumeCount >= 2 health check (failing with "no healthy replicas"), which aborted ec.encode before any phase marker printed and made every encode scenario in TestECInterruptionMatrix fail. Three changes to the test cluster: 1. Raise -max from 4 to 8 per disk so the source disk always retains FreeVolumeCount >= 2 for ec.encode's 14-shard generation (2 volume-slot equivalents) even after the spread loop and multiple encodes. 2. Switch the spread loop from -count 4 to -count 1 so each grow lands exactly one volume on the volume server's least-loaded disk, giving deterministic cross-disk spreading instead of relying on a single multi-volume grow to fan out. 3. Cap grows per server at 4 so heartbeat lag cannot run away and over-fill disks before the master registers the prior grow. 4. Pass -minFreeSpace 0 so the test is not falsely gated by the physical disk's free-space percentage on the host running CI (the EC shard slot calculation separately enforces a 90 % disk-usage cap via balancer.DiskTooFullAfter, which already guards against an over-set maxVolumeCount on a physically full disk). Verified locally by running TestECInterruptionMatrix twice (all encode, decode, and balance scenarios pass, including the previously failing encode@Deletingoriginalvolumes). * test/ec: only count successful grows toward the spread cap A failed volume.grow (e.g. a transient collectTopologyInfo or VolumeGrow RPC error) would otherwise consume one of the four permitted attempts without creating any volume, exhausting the retry budget and leaving the loop to only poll until the Eventually timeout. Increment the per-server counter only when commandGrow.Do returns nil.	2026-09-13 14:41:45 -07:00
Chris Lu	902a12fd6f	wdclient: bound the wait for a master leader by the caller's context (#11002) ... * wdclient: bound the wait for a master leader by the caller's context WithClient waited on GetMaster with context.Background(), so a caller that arrived while no master leader was known parked in a 200ms poll loop until one appeared, whatever deadline it had already set on the RPC. Each retry above it then left another goroutine in the same wait. Take the context in WithClient and WithClientCustomGetMaster and hand it to GetMaster, and stop the retry loop once it is done. The dial keeps context.Background(): fn brings its own RPC context, so a cancellation seen here cannot be attributed to the shared connection. Call sites pass whatever they hold: the request context in the filer's CollectionList, DeleteCollection and Statistics handlers and in the credential store's propagation, the operation context in the shell's s3.bucket.delete and the kafka gateway's broker and filer discovery, and context.Background() where there is none - the shell commands, the admin dashboard wrapper, and the exclusive locker's initial lease. The locker's release keeps its own uncancelled context so a slow unlock cannot turn into a ghost lock. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * wdclient: test that WithClient gives up with the caller's context Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * wdclient: cut the master retry backoff short when the caller gives up util.Retry sleeps unconditionally between attempts, so a transient error arriving just before the caller's deadline still cost it a full backoff step. Use the context-aware util.RetryWithBackoff, the same helper the volume lookup in this file already uses. Two call sites went with it: the shell's lock-holder lookup builds its three second bound before WithClient so it also covers finding the leader, as its comment already promised, and the filer's post-delete collection cleanup goes back to an uncancelled context - the entry is already gone, so a caller that hung up must not leave the collection behind. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * wdclient: test that a cancel during backoff ends the retry Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU	2026-08-27 22:27:45 -07:00
Chris Lu	76a1983c86	test: re-lock and retry every chaos command, not just the balance (#10770) ... The harness kills shells mid-command, and the master releases the dead session's lock only when it notices the connection is gone. That cleanup lands after the harness has already re-acquired the lock, so it can clear the lock this run holds and the next command refuses with need to run "lock" first to continue recoverInterruptedBalance answered that the way an operator would -- run lock again and retry -- but the encode and decode recoveries called shellCommand once and required success, so the same reap failed the run outright. Move the retry into shellCommand: the reap can land during any command that follows a kill, not only a balance.	2026-08-15 14:13:37 -07:00
Chris Lu	1c926e8fac	test: systematic EC interruption verification — exhaustive model check + deterministic kill matrix (#10764) ... * ec: bounded-exhaustive model check of the volume lifecycle The randomized chaos harness samples the state space; this enumerates it. The lifecycle is a state machine whose steps mirror the pipelines in this package, and the checker explores every schedule within the bound: a crash at every step boundary, an error return running the rollback (itself crashable at every step), a volume-server restart applying the startup reconciliation rules in every quiescent state, and the prescribed restart-based recovery from every crashed state. Checked in every reachable state: durability (a readable copy always exists), at most one generation mounted, and — a property the sweep discipline turns out to guarantee — at most one generation's files on disk. From every quiescent state the recovery must converge to a clean volume. Runs in well under a second. * test: deterministic EC interruption matrix Enumerate every phase of every interruptible EC operation and kill a real weed shell exactly when the phase announces itself on the command output, instead of at a random moment: four encode phases, four decode phases, and the balance's move phase (set up with -rebalance=false so a move is guaranteed). Each scenario prepares its precondition, kills at the marker, runs the prescribed recovery, and verifies every stored byte still reads back identical. The interruption recoveries move out of the randomized ops into shared chaosRun helpers both drivers use. * test: make the randomized EC chaos walk opt-in The systematic layers — the interruption matrix and the lifecycle model check — carry the CI coverage deterministically; the randomized walk stays for exploratory runs, behind EC_CHAOS_SEED. * ci: bound the EC integration suite by the job budget, not go test's default The suite with the interruption matrix runs close to the default 10m binary timeout on slower runners. * test: require every interruption-matrix marker to appear A marker that never prints means a pipeline refactor renamed or dropped the progress line; silently degenerating into a no-interruption run would let CI pass without exercising the boundary the scenario names. Also recheck the marker channel after the wait: a shell that prints and exits at once makes both channels ready, and select picking the exit case must not report a printed marker as missed.	2026-08-14 17:45:11 -07:00
Chris Lu	602746f51d	test: EC lifecycle chaos harness, with four fixes it found (#10763) ... * ec: let the encode's balance see a migrating volume's shards across disk-type buckets Shard generation writes beside the source .dat, so a cross-tier encode (source on hdd, -diskType=ssd) leaves the fresh shards in the source disk-type bucket. The encode's internal balance ingested only the target bucket, saw no shards, and planned no moves; the spread guard then correctly aborted the encode (and before that guard existed, the shards silently stayed clumped on the generation host in the wrong tier). EcBalance now takes the encode batch as migratingVolumeIds and ingests those volumes' shards from every bucket, while everything else keeps the bucket filter so a plain ec.balance never drags deliberately tiered shards onto another disk type. The in-memory model delete also becomes bucket-agnostic: a node holds a given shard in exactly one bucket, and a bucket-scoped delete missed cross-bucket moves in the dry-run model. * volume: decode reads shard 0 from its resolved path, not the EC volume's base dir On a multi-disk server a volume's shards can sit on several disks; the store registers each shard with its own path and CollectEcShards resolves them, but FindDatFileSize derived the .ec00 path from the EcVolume's base directory. When shard 0 lived on a sibling disk, VolumeEcShardsToVolume failed with 'open ...ec00: no such file or directory' and ec.decode aborted. * ec: decode re-copies shards the topology claims but the target does not hold An interrupted earlier decode or balance can leave the master believing the decode target holds a shard whose file never landed: the mount registered but the partial copy was cleaned, or the file was swept. The collect step took the topology's word for it, excluded the shard from the copy set, and the decode failed with 'missing shard'. Probe the target's live inventory (VolumeEcShardsInfo) and treat anything it cannot serve as still-to-copy. * ec: decode discovers shards across disk-type buckets Shards sit wherever encode generation and balance left them: a cross-tier encode leaves them in the source disk-type bucket, a partial migration straddles buckets. ec.decode scoped its shard discovery to the -diskType bucket and reported a decodable volume as having no shards at all. Union across buckets, the way the encode's shard verification already does. * test: EC chaos lifecycle harness Randomized, seeded sequences of the EC lifecycle against a live cluster in the production-shaped layout: multiple data disks per server, a separate -dir.idx directory so .ecx/.ecj sidecars are shared across disks, and a tagged ssd tier. Operations cover encode (hdd and ssd targets), balance, shard damage plus rebuild, decode, re-encode, deletes, scrub, tier moves, crash-restarts, sidecar fault injections (a data-dir .vif pushed into the shared idx dir; a stale-generation shard planted beside a newer encode), and interruptions: a real weed shell subprocess killed mid-encode, mid-decode, and mid-balance, with the recovery re-run required to converge. One invariant holds after every step: every stored byte reads back identical and every deleted needle stays deleted. EC_CHAOS_SEED and EC_CHAOS_STEPS make runs reproducible and scalable. A known gap is tolerated and logged rather than fixed here: a shard mounted on two disks of one node (orphan adoption after an interrupted copy) is invisible to ec.balance's dedup and unaddressable by ec.shard.unmount's shard@address form, so no cleanup path exists yet. * test: fail payload-corruption checks on the test goroutine t.Fatalf inside require.Eventually's condition runs on the poller's goroutine, where Goexit kills only that goroutine and the corruption message can be lost behind a generic timeout. Record the mismatch, end the polling, and fail on the test goroutine. Also assert the full shard count in the cross-bucket decode-discovery test.	2026-08-14 17:26:54 -07:00