shell: make every -collection flag parse the same way (#11794)

* shell: make every -collection flag parse the same way

The filter vocabulary (comma-separated names, wildcards, regexes,
"_default" for the empty-named collection) already existed in
wildcard.CompileCollectionMatcher, but several commands still did their
own exact-match comparison or sent the raw flag value to servers.

fs.mergeVolumes used a literal string compare, so -collection=_default
silently produced an empty merge plan; it now uses the shared matcher
while keeping "*" as the match-all default. volume.fsck and volume.mark
filter through the matcher too, and volume.vacuum resolves the filter
client-side into per-collection and per-volume requests, since no
Collection value scopes the VacuumVolume RPC to the unnamed collection
alone.

Commands that put one concrete collection name on the wire normalize
"_default" to "" (volume.grow, fs.configure, mq.topic.compact,
collection.delete, ec.rebuild, ec.balance), and the tier -volumeId
commands resolve the volume's actual collection from topology first so
patterns and named-collection requests pass the volume server's
collection check.

* shell: spell the default-collection keyword via CollectionDefault

* shell: address review findings on collection filter handling

volume.fsck keeps a set of every volume the master knows before any
filtering, so a filer entry whose volume was excluded by -collection or
-volumeId is skipped instead of reported "volume not found" and deleted
by -reallyDeleteFilerEntries. The collection filter also no longer
scopes the filer walk to /buckets/<collection>: fs.configure can bind a
collection to any path prefix, and a narrower walk could mark live
chunks as orphans under -reallyDeleteFromVolume.

ec.rebuild and ec.balance resolve -collection through the new
wildcard.ExpandCollectionFilter, which passes literal names through even
when the master does not list them — ec.rebuild must be able to recover
a collection whose shards all lost their index and never registered.

fs.configure and mq.topic.compact now reject _default outright: an empty
Collection means "unset" in both the path-rule merge and the filer's
storage-option fallback chain, so the keyword would have been silently
ignored.

The unnamed-collection vacuum sweep keeps the master's low-disk-space
exception: a volume whose every replica is read-only is probed with
VacuumVolumeCheck and stays eligible when a replica reports
DiskSpaceLow, instead of being dropped outright.
This commit is contained in:
Chris Lu authored and GitHub committed 2026-10-16 21:07:51 -07:00
1 parent 268424a5e1
commit b0ab4c4e1e
23 files changed
+593 -71

No files matched your search

+17 -7
View File
@@ -89,7 +89,7 @@ func (c *commandVolumeTierUpload) Do(args []string, commandEnv *CommandEnv, writ
tierCommand := flag.NewFlagSet(c.Name(), flag.ContinueOnError)
volumeId := tierCommand.Int("volumeId", 0, "the volume id")
collection := tierCommand.String("collection", "", "the collection name")
collection := tierCommand.String("collection", "", "comma-separated collection names, wildcards, or regex patterns; empty matches the collection with no name")
fullPercentage := tierCommand.Float64("fullPercent", 95, "the volume reaches the percentage of max volume size")
quietPeriod := tierCommand.Duration("quietFor", 24*time.Hour, "select volumes without no writes for this period")
dest := tierCommand.String("dest", "", "the target tier name")
@@ -137,20 +137,30 @@ func (c *commandVolumeTierUpload) Do(args []string, commandEnv *CommandEnv, writ
return nil
}
func doVolumeTierUpload(commandEnv *CommandEnv, writer io.Writer, collection string, vid needle.VolumeId, dest string, keepLocalDatFile bool, concurrency int) (err error) {
func doVolumeTierUpload(commandEnv *CommandEnv, writer io.Writer, collectionPattern string, vid needle.VolumeId, dest string, keepLocalDatFile bool, concurrency int) (err error) {
// find volume location
topoInfo, _, err := collectTopologyInfo(commandEnv, 0)
if err != nil {
return fmt.Errorf("collect topology info: %v", err)
}
// The volume server verifies the request's collection against the volume's
// own, so the pattern must resolve to the actual collection here.
collection, found, err := findVolumeCollection(topoInfo, vid, collectionPattern)
if err != nil {
return err
}
if !found {
if collectionPattern == "" {
return fmt.Errorf("volume %d not found", vid)
}
return fmt.Errorf("volume %d not found in collections matching %s", vid, collectionPattern)
}
existingLocations := collectVolumeTierUploadLocations(topoInfo, vid, collection, writer)
if len(existingLocations) == 0 {
if collection == "" {
return fmt.Errorf("volume %d not found", vid)
}
return fmt.Errorf("volume %d not found in collection %s", vid, collection)
return fmt.Errorf("volume %d has no replicas in collection %s", vid, collection)
}
_, err = markVolumeReplicasWritable(context.Background(), commandEnv.option.GrpcDialOption, vid, existingLocations, false, false)
@@ -192,7 +202,7 @@ func collectVolumeTierUploadLocations(topoInfo *master_pb.TopologyInfo, vid need
eachDataNode(topoInfo, func(dc DataCenterId, rack RackId, dn *master_pb.DataNodeInfo) {
for _, disk := range dn.DiskInfos {
for _, vi := range disk.VolumeInfos {
if needle.VolumeId(vi.Id) == vid && (collection == "" || vi.Collection == collection) {
if needle.VolumeId(vi.Id) == vid && vi.Collection == collection {
fmt.Fprintf(writer, "find volume %d from Url:%s, GrpcPort:%d, DC:%s\n", vid, dn.Id, dn.GrpcPort, string(dc))
loc := wdclient.Location{
Url: dn.Id,