4 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	cb9fcd39d2	filer/postgres: create filemeta table on startup via createTable config (#11229) ... * filer/postgres: create default filemeta table on startup The postgres filer store hardcoded CreateTableSqlTemplate to empty and never created the filemeta table, unlike postgres2/mysql2/sqlite which all create it during Initialize. Users had to create the table manually or the filer would crash loop with "relation filemeta does not exist". Read the createTable config option (same as postgres2), default to DefaultCreateTableQuery when unset, and execute CREATE TABLE IF NOT EXISTS on the default table after the connection pool is established. SupportBucketTable stays false so per-bucket table creation remains a no-op; only the shared filemeta table is created, via a direct ExecContext since AbstractSqlStore.CreateTable short-circuits without bucket support. * filer/postgres: accept boolean createTable = true/false viper reads a TOML boolean as the string "true"/"false" via GetString, so createTable = true was being used as a SQL template and failed. Add ResolveCreateTableQuery to normalize the value: true and empty select the default template, false disables table creation, anything else is a custom template. Both postgres and postgres2 now use it, and both skip the CREATE TABLE call when the resolved template is empty. * scaffold: document createTable option for postgres filer store Replace the commented-out CREATE TABLE SQL in the [postgres] scaffold with a createTable config hint, matching the [postgres2] section. Users no longer need to manually create the filemeta table before starting the filer. * filer/postgres: make createTable opt-in for postgres, keep postgres2 default The previous commit defaulted postgres to create the filemeta table even when createTable was unset, which could break existing deployments whose DB user lacks CREATE TABLE privileges. ResolveCreateTableQuery now returns empty for an unset value so postgres only creates the table when createTable is explicitly true or a custom template — preserving the prior no-DDL behaviour for existing configurations. postgres2 keeps its existing always-create default: it defaults an empty resolved value to DefaultCreateTableQuery, and only skips when createTable is explicitly false. * filer/postgres2: simplify createTable handling, document all modes Drop the false opt-out from postgres2 — it only skipped the default table while per-bucket CreateTable still ran, leaving restricted DB roles broken on bucket access. postgres2 now accepts true the same way (defaulting to DefaultCreateTableQuery) and keeps its existing always-create behaviour for every other value, matching the original semantics. The scaffold comment now documents true/false/custom for the postgres section so users know false (or unset) is the backward-compatible default. * filer/postgres2: normalize false via ResolveCreateTableQuery postgres2 only handled "" and "true", leaving createTable = false as the literal string "false" which CreateTable then executed as invalid SQL. Route it through ResolveCreateTableQuery (which maps false to empty) and default the empty result to DefaultCreateTableQuery, so false is treated the same as unset for the bucket-aware store. * filer/postgres2: honor createTable = false for default table postgres2 treated false the same as unset and always created the default filemeta table, failing startup for restricted DB roles that explicitly opted out. Track the original false value before ResolveCreateTableQuery collapses it to empty, and skip the default CreateTable call when set. Per-bucket table creation is unaffected — it is a runtime requirement of the bucket-aware store. Users who need to suppress all DDL should use the postgres (non-bucket) store with createTable unset. * filer/postgres2: disable bucket tables when createTable = false Setting SupportBucketTable = false when createTable is explicitly false makes AbstractSqlStore.CreateTable a no-op (it already returns nil when SupportBucketTable is false), so neither the default filemeta table nor per-bucket tables are created. The template stays empty and no DDL runs, honouring the opt-out for restricted DB roles. All data routes to the pre-provisioned filemeta table, matching the postgres (non-bucket) store. * filer: suppress DDL without disabling bucket routing Setting SupportBucketTable = false when createTable = false also disabled per-bucket routing, hiding objects in pre-provisioned per-bucket tables. Keep SupportBucketTable true and instead skip the CREATE TABLE execution when the resolved template is empty. GetSqlCreateTable now returns empty for both postgres and mysql SQL generators when CreateTableSqlTemplate is empty, and AbstractSqlStore.CreateTable skips the ExecContext call when the SQL is empty. This preserves bucket routing while suppressing all DDL for users who explicitly set createTable = false and pre-provision their tables. * filer: add SkipDDL to suppress CREATE and DROP without disabling routing createTable = false with SupportBucketTable = true preserved bucket routing but deleteTable still executed DROP TABLE on bucket deletion, dropping externally managed tables. CanDropWholeBucket also returned true, so the S3 layer tried whole-table drops instead of row-by-row deletes. Add a SkipDDL flag to AbstractSqlStore, independent of SupportBucketTable. CreateTable and deleteTable both skip when SkipDDL is set, and CanDropWholeBucket returns false so bucket deletion falls back to row-by-row metadata deletes. postgres2 sets SkipDDL when createTable is explicitly false — bucket routing is preserved, no DDL runs. * filer: fall back to row-by-row delete when CanDropWholeBucket is false DeleteFolderChildren took the whole-table drop path whenever the path was a bucket root, even when SkipDDL made deleteTable a no-op. The no-op returned nil, the caller returned early, and rows inserted after the recursive enumeration survived the bucket deletion. Gate the whole-table drop on CanDropWholeBucket so the row-by-row DeleteFolderChildren SQL runs when SkipDDL is set, removing all metadata without issuing DROP TABLE.	2026-09-08 16:17:02 -07:00
Chris Lu	873705baf9	filer: default the filemeta CREATE TABLE for postgres2/mysql2 when createTable is unset (#10232) ... * postgres2: default the filemeta CREATE TABLE when createTable is unset An empty createTable rendered through fmt.Sprintf produced %!(EXTRA string=filemeta), which Postgres rejected with a syntax error at init. Fall back to a working template so a minimal config bootstraps. * mysql2: default the filemeta CREATE TABLE when createTable is unset Same empty-template failure the postgres2 path had: an unset createTable rendered to %!(EXTRA string=filemeta) and MySQL rejected it at init. Fall back to a working template.	2026-07-05 10:17:40 -07:00
Chris Lu	a24f4844d3	filer: keep S3 list order byte-lexicographic regardless of SQL name column collation (#9824) ... * mysql: keep S3 list order byte-lexicographic regardless of name column collation ORDER BY name and the name > ? pagination predicate follow the column collation, so a case-insensitive filemeta.name (e.g. utf8mb3_general_ci) returns S3 keys out of byte order and breaks clients that merge two sorted listings. Detect the live name collation at startup; only when it isn't binary, wrap the list comparison, prefix, and ORDER BY in BINARY name so order and pagination stay consistent. Correctly configured utf8mb4_bin tables keep their indexed range scan unchanged, and the operator gets a warning to convert the column. * postgres: keep S3 list order byte-lexicographic regardless of name column collation ORDER BY name and the name > $n pagination predicate follow the column or database collation, so a locale-aware filemeta.name (e.g. the en_US.UTF-8 database default) returns S3 keys out of byte order and breaks clients that merge two sorted listings. Detect the live name collation at startup; only when it isn't byte-ordered, wrap the list comparison, prefix, and ORDER BY in COLLATE "C" so order and pagination stay consistent. A byte-ordered (C/POSIX/C.UTF-8) column keeps its indexed range scan unchanged, and the operator gets a warning to declare the column COLLATE "C".	2026-06-04 14:33:41 -07:00
Chris Lu	21b4b81edb	fix(filer/postgres): default to ON CONFLICT upsert to keep tx alive (#9709) ... * fix(filer/postgres): default to ON CONFLICT upsert to keep tx alive A KvPut from the inode-index secondary write could fail with 23505 (duplicate key) inside a rename's transaction, after which the next statement returned 25P02 and rename surfaced to FUSE as EIO. Default the postgres upsert query when enableUpsert=true so INSERTs are idempotent; the enableUpsert=false escape hatch is preserved for non-PG-compatible backends. * fix(filer/mysql): default to ON DUPLICATE KEY UPDATE upsert Same shape as the postgres default: when enableUpsert=true but no upsertQuery is configured, install a sensible default so the inode-index KvPut does not waste a duplicate-key roundtrip on every entry write. Uses the VALUES() form so the default works on MariaDB and MySQL >=5.7; the MySQL 8.0.19 row-alias form is left to explicit config. * fix(filer): default enableUpsert=true for sql stores The default-template fallback only kicks in when enableUpsert=true, so minimal configs that omit the flag entirely were still exposed. Default it on for postgres/postgres2/mysql/mysql2; an explicit false in filer.toml still wins because SetDefault only fills absent keys.	2026-05-27 12:23:30 -07:00