shell: parse every collection filter the same way (#10955)

* worker: move the collection filter parser into weed/util/wildcard

The parser sits beside the volume-list filtering it was written for, in
weed/plugin/worker, which imports weed/shell — so the shell commands that
parse the same filter three other ways can never call it. Move it down to
weed/util/wildcard, next to the comma-separated wildcard helper it already
replaced, leaving the behavior unchanged.

* shell: parse every collection filter the same way

The shell parsed a collection filter three ways: compileCollectionPattern
compiled one regex for ec.encode, ec.decode, volume.balance and the tier
commands; volume.list and volume.deleteEmpty matched a single wildcard; and
volume.tier.move, volume.fix.replication and volume.configure.replication
called filepath.Match on their own. None of them took a list, so
"ec.encode -collection=a,b" selected nothing, the same way the admin UI did.

They all go through the shared matcher now: a comma-separated list of names,
"*" and "?" wildcards, "_default" for the collection with no name, and regex
entries. The one thing that stays per-command is what an empty value means -
every collection for -collectionPattern, the unnamed collection for the ec
and tier -collection flag - so compileCollectionPattern keeps that mapping.

The matchers are compiled once per command instead of once per volume, and a
regex entry now has to match the whole name unless it anchors itself, so
-collection=bucket no longer picks up mybucket2.

* shell: keep dots in collection names, and commas inside a regex

A dot no longer marks an entry as a regex, so a collection named "my.bucket"
matches itself and not "my-bucket" - the difference decides which volumes
volume.deleteEmpty and volume.tier.move touch. A dot still counts when it is
quantified, so "bucket.*" stays a prefix regex.

The comma split also leaves alone the commas inside a character class or a
repetition count, so "bucket[0-9]{1,3}" stays one entry instead of becoming
two broken fragments.

* shell: let a regex entry match its own spelling

A collection named after regex syntax, say "logs(2024)", was unreachable:
the entry compiled to a pattern that matches "logs2024" instead. Match the
entry verbatim as well, so naming a collection always selects it, whatever
characters it holds.

* shell: reject a collection filter that names no collection

A value of "," parsed to no entries and then matched every collection, so a
typo widened ec.encode or volume.deleteEmpty to the whole cluster. Only a
genuinely empty filter means "all collections"; anything else has to name one.

* shell: keep commas inside a regex group out of the entry split

The split already left alone the commas inside a character class or a
repetition count, but not the ones inside a group, so "bucket(foo,bar)"
was cut into two fragments that no longer compile.

* shell: cover escaping a collection name that is not a regex

A name like "logs(2024" does not parse as a regex on its own; escaping it,
"logs\(2024", reaches it. Pin that so the escape hatch does not regress.

* shell: split entries only on commas inside a closed regex construct

An unmatched "{" or "[" made the splitter swallow every comma after it, so
"foo{bar,videos" became one entry that matches neither collection - the
silent no-op this filter work exists to remove. A construct now has to close
before its commas stop separating entries.

* shell: skip character classes while scanning a regex group

A ")" inside a class is a literal, so "(a[)],b)" ended its group early and
split into two fragments that no longer compile.

* shell: cover escaping a comma inside a collection name

A comma separates entries, so a name holding one is reached by escaping it.

* shell: follow the regexp parser when scanning a character class

A "]" leading a class is a member of it, and a POSIX class such as
"[:alpha:]" carries its own "]", so stopping at the first one cut a valid
filter like "(a[]),],b)" into fragments and rejected it.
This commit is contained in:
Chris Lu authored and GitHub committed 2026-08-25 18:03:52 -07:00
1 parent 368b2035b2
commit 627b5e9d59
31 files changed
+506 -292

No files matched your search

+7 -15
View File
@@ -29,7 +29,7 @@ func init() {
}
type commandVolumeFixReplication struct {
collectionPattern *string
collectionMatcher *wildcard.CollectionMatcher
// TODO: move parameter flags here so we don't shuffle them around via function calls.
}
@@ -71,7 +71,7 @@ func (c *commandVolumeFixReplication) HasTag(tag CommandTag) bool {
func (c *commandVolumeFixReplication) Do(args []string, commandEnv *CommandEnv, writer io.Writer) (err error) {
volFixReplicationCommand := flag.NewFlagSet(c.Name(), flag.ContinueOnError)
c.collectionPattern = volFixReplicationCommand.String("collectionPattern", "", "match with wildcard characters '*' and '?'")
collectionPattern := volFixReplicationCommand.String("collectionPattern", "", "comma-separated collection names, with '*' and '?' wildcards; empty matches all")
applyChanges := volFixReplicationCommand.Bool("apply", false, "apply the fix")
// TODO: remove this alias
applyChangesAlias := volFixReplicationCommand.Bool("force", false, "apply the fix (alias for -apply)")
@@ -87,6 +87,10 @@ func (c *commandVolumeFixReplication) Do(args []string, commandEnv *CommandEnv,
return nil
}
if c.collectionMatcher, err = wildcard.CompileCollectionMatcher(*collectionPattern); err != nil {
return err
}
handleDeprecatedForceFlag(writer, volFixReplicationCommand, applyChangesAlias, applyChanges)
infoAboutSimulationMode(writer, *applyChanges, "-apply")
commandEnv.noLock = !*applyChanges
@@ -127,7 +131,7 @@ func (c *commandVolumeFixReplication) Do(args []string, commandEnv *CommandEnv,
replica := replicas[0]
// Filter here so the termination counter matches what gets fixed; else -apply loops forever.
if !c.matchCollectionPattern(replica.info.Collection) {
if !c.collectionMatcher.Matches(replica.info.Collection) {
continue
}
@@ -330,18 +334,6 @@ func checkOneVolume(a *VolumeReplica, b *VolumeReplica, writer io.Writer, comman
return
}
// matchCollectionPattern reports whether collection matches -collectionPattern:
// empty matches everything, CollectionDefault matches the unnamed collection.
func (c *commandVolumeFixReplication) matchCollectionPattern(collection string) bool {
if *c.collectionPattern == "" {
return true
}
if *c.collectionPattern == CollectionDefault {
return collection == ""
}
return wildcard.MatchesWildcard(*c.collectionPattern, collection)
}
// deleteOneVolume trims one replica from each of the given volumes, and
// reports how many replicas it actually deleted.
func (c *commandVolumeFixReplication) deleteOneVolume(commandEnv *CommandEnv, writer io.Writer, applyChanges bool, doCheck bool, volumeIds []uint32, volumeReplicas map[uint32][]*VolumeReplica, selectOneVolumeFn SelectOneVolumeFunc) (deleted int, err error) {