mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-20 13:30:46 +02:00
feat(shell): Support batched EC encode and multi-volume selection in ec.encode (#10030)
* fix(shell): correct volume.list -writable filter unit and comparison * fix(shell): correct volume.list -writable filter unit and comparison * chore(shell): fix typo in EC shard helper param names * fix(shell): use exact match for volume.balance -racks/-nodes filter The old strings.Contains-based filter quietly included any id that was a substring of the user-supplied flag value (e.g. -racks=rack10 also matched rack1). Replace it with an exact-match set parsed from the comma-separated flag value, and add regression tests for both -racks and -nodes paths. Also fix a small typo in the "remote storage" error returned by maybeMoveOneVolume. * fix(shell): use exact match for volume.balance -racks/-nodes filter The old strings.Contains-based filter quietly included any id that was a substring of the user-supplied flag value (e.g. -racks=rack10 also matched rack1). Replace it with an exact-match set parsed from the comma-separated flag value, and add regression tests for both -racks and -nodes paths. Also fix a small typo in the "remote storage" error returned by maybeMoveOneVolume. * refactor(shell): drop nil sentinel in splitCSVSet, use len() in callers * feat(shell): support batched EC encode and multi-volume selection Add -volumeIds (comma-separated) and -batchSize flags to ec.encode. When -batchSize > 0, volumes are processed in independent batches, each committed separately: encode -> rebalance -> verify -> delete originals. This bounds the working set and lets source volumes be reclaimed without waiting for the entire set to finish, at the cost of per-batch rebalancing. Because each batch deletes its originals, a failure in a later batch is unrecoverable for already-completed batches. To let the single-volume, multi-volume, and collection paths share one per-batch routine, the re-balance scope is now always derived from the volumes actually selected for encoding (collectCollectionsForVolumeIds), rather than every collection matching the -collection regex. Practical effect: with -collection, a collection that matches the pattern but contributes no encodable volumes is no longer re-balanced as a side effect. The -volumeId path is unchanged; -batchSize=0 (default) preserves the original single-pass flow. The per-batch routine reuses the existing assertEncodableRegularVolumes guard, doEcEncode skipped-node handling, and verifyEcShardsBeforeDelete retry loop. The capacity pre-flight check takes the already-fetched topology instead of issuing another VolumeList to the master per batch. Also clarify the -collection flag description to note it accepts a regex pattern, matching the existing command help. -volumeId and -volumeIds are mutually exclusive; ids in -volumeIds are validated and de-duplicated.
This commit is contained in:
@@ -246,3 +246,27 @@ func TestEcEncodeNodeCountCheck(t *testing.T) {
|
||||
willProceed = forceChanges || nodeCount >= minNodeCount
|
||||
assert.True(t, willProceed, "Should proceed with -force even with %d nodes", nodeCount)
|
||||
}
|
||||
|
||||
func TestParseEcEncodeVolumeIds(t *testing.T) {
|
||||
vids, err := parseEcEncodeVolumeIds("101, 102,101, 103")
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, []needle.VolumeId{101, 102, 103}, vids)
|
||||
|
||||
_, err = parseEcEncodeVolumeIds("101,abc")
|
||||
assert.Error(t, err)
|
||||
|
||||
_, err = parseEcEncodeVolumeIds(" , ")
|
||||
assert.Error(t, err)
|
||||
}
|
||||
|
||||
func TestChunkEcEncodeVolumeIds(t *testing.T) {
|
||||
vids := []needle.VolumeId{101, 102, 103, 104, 105}
|
||||
|
||||
assert.Equal(t, [][]needle.VolumeId{
|
||||
{101, 102},
|
||||
{103, 104},
|
||||
{105},
|
||||
}, chunkEcEncodeVolumeIds(vids, 2))
|
||||
|
||||
assert.Equal(t, [][]needle.VolumeId{vids}, chunkEcEncodeVolumeIds(vids, 0))
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user