filer: keep S3 list order byte-lexicographic regardless of SQL name column collation (#9824)

* mysql: keep S3 list order byte-lexicographic regardless of name column collation

ORDER BY name and the name > ? pagination predicate follow the column
collation, so a case-insensitive filemeta.name (e.g. utf8mb3_general_ci)
returns S3 keys out of byte order and breaks clients that merge two sorted
listings.

Detect the live name collation at startup; only when it isn't binary, wrap
the list comparison, prefix, and ORDER BY in BINARY name so order and
pagination stay consistent. Correctly configured utf8mb4_bin tables keep
their indexed range scan unchanged, and the operator gets a warning to
convert the column.

* postgres: keep S3 list order byte-lexicographic regardless of name column collation

ORDER BY name and the name > $n pagination predicate follow the column or
database collation, so a locale-aware filemeta.name (e.g. the en_US.UTF-8
database default) returns S3 keys out of byte order and breaks clients that
merge two sorted listings.

Detect the live name collation at startup; only when it isn't byte-ordered,
wrap the list comparison, prefix, and ORDER BY in COLLATE "C" so order and
pagination stay consistent. A byte-ordered (C/POSIX/C.UTF-8) column keeps its
indexed range scan unchanged, and the operator gets a warning to declare the
column COLLATE "C".
This commit is contained in:
Chris Lu authored and GitHub committed 2026-06-04 14:33:41 -07:00
1 parent 8c2d9f466f
commit a24f4844d3
10 files changed
+222 -8

No files matched your search

+4 -1
View File
@@ -68,11 +68,12 @@ func (store *PostgresStore2) initialize(createTable, upsertQuery string, enableU
} else if upsertQuery == "" {
upsertQuery = postgres.DefaultUpsertQuery
}
store.SqlGenerator = &postgres.SqlGenPostgres{
gen := &postgres.SqlGenPostgres{
CreateTableSqlTemplate: createTable,
DropTableSqlTemplate: `drop table "%s"`,
UpsertQueryTemplate: upsertQuery,
}
store.SqlGenerator = gen
// pgx-optimized connection string with better timeouts and connection handling
sqlUrl := "connect_timeout=30"
@@ -126,5 +127,7 @@ func (store *PostgresStore2) initialize(createTable, upsertQuery string, enableU
return fmt.Errorf("init table %s: %v", abstract_sql.DEFAULT_TABLE, err)
}
postgres.ConfigureListOrdering(store.DB, gen)
return nil
}