feat(shell): Support batched EC encode and multi-volume selection in ec.encode (#10030)

* fix(shell): correct volume.list -writable filter unit and comparison

* fix(shell): correct volume.list -writable filter unit and comparison

* chore(shell): fix typo in EC shard helper param names

* fix(shell): use exact match for volume.balance -racks/-nodes filter

The old strings.Contains-based filter quietly included any id that was a
  substring of the user-supplied flag value (e.g. -racks=rack10 also matched
  rack1). Replace it with an exact-match set parsed from the comma-separated
  flag value, and add regression tests for both -racks and -nodes paths.

  Also fix a small typo in the "remote storage" error returned by
  maybeMoveOneVolume.

* fix(shell): use exact match for volume.balance -racks/-nodes filter

The old strings.Contains-based filter quietly included any id that was a
  substring of the user-supplied flag value (e.g. -racks=rack10 also matched
  rack1). Replace it with an exact-match set parsed from the comma-separated
  flag value, and add regression tests for both -racks and -nodes paths.

  Also fix a small typo in the "remote storage" error returned by
  maybeMoveOneVolume.

* refactor(shell): drop nil sentinel in splitCSVSet, use len() in callers

* feat(shell): support batched EC encode and multi-volume selection

Add -volumeIds (comma-separated) and -batchSize flags to ec.encode.

When -batchSize > 0, volumes are processed in independent batches, each
committed separately: encode -> rebalance -> verify -> delete originals.
This bounds the working set and lets source volumes be reclaimed without
waiting for the entire set to finish, at the cost of per-batch rebalancing.
Because each batch deletes its originals, a failure in a later batch is
unrecoverable for already-completed batches.

To let the single-volume, multi-volume, and collection paths share one
per-batch routine, the re-balance scope is now always derived from the
volumes actually selected for encoding (collectCollectionsForVolumeIds),
rather than every collection matching the -collection regex. Practical
effect: with -collection, a collection that matches the pattern but
contributes no encodable volumes is no longer re-balanced as a side effect.
The -volumeId path is unchanged; -batchSize=0 (default) preserves the
original single-pass flow.

The per-batch routine reuses the existing assertEncodableRegularVolumes
guard, doEcEncode skipped-node handling, and verifyEcShardsBeforeDelete
retry loop. The capacity pre-flight check takes the already-fetched
topology instead of issuing another VolumeList to the master per batch.

Also clarify the -collection flag description to note it accepts a regex
pattern, matching the existing command help.

-volumeId and -volumeIds are mutually exclusive; ids in -volumeIds are
validated and de-duplicated.
This commit is contained in:
qzhello
2026-06-22 01:22:20 -07:00
committed by GitHub
parent eb93976166
commit 4f9393889c
2 changed files with 164 additions and 47 deletions
+24
View File
@@ -246,3 +246,27 @@ func TestEcEncodeNodeCountCheck(t *testing.T) {
willProceed = forceChanges || nodeCount >= minNodeCount
assert.True(t, willProceed, "Should proceed with -force even with %d nodes", nodeCount)
}
func TestParseEcEncodeVolumeIds(t *testing.T) {
vids, err := parseEcEncodeVolumeIds("101, 102,101, 103")
assert.NoError(t, err)
assert.Equal(t, []needle.VolumeId{101, 102, 103}, vids)
_, err = parseEcEncodeVolumeIds("101,abc")
assert.Error(t, err)
_, err = parseEcEncodeVolumeIds(" , ")
assert.Error(t, err)
}
func TestChunkEcEncodeVolumeIds(t *testing.T) {
vids := []needle.VolumeId{101, 102, 103, 104, 105}
assert.Equal(t, [][]needle.VolumeId{
{101, 102},
{103, 104},
{105},
}, chunkEcEncodeVolumeIds(vids, 2))
assert.Equal(t, [][]needle.VolumeId{vids}, chunkEcEncodeVolumeIds(vids, 0))
}