* filer/postgres: create default filemeta table on startup
The postgres filer store hardcoded CreateTableSqlTemplate to empty and
never created the filemeta table, unlike postgres2/mysql2/sqlite which
all create it during Initialize. Users had to create the table manually
or the filer would crash loop with "relation filemeta does not exist".
Read the createTable config option (same as postgres2), default to
DefaultCreateTableQuery when unset, and execute CREATE TABLE IF NOT EXISTS
on the default table after the connection pool is established.
SupportBucketTable stays false so per-bucket table creation remains a
no-op; only the shared filemeta table is created, via a direct ExecContext
since AbstractSqlStore.CreateTable short-circuits without bucket support.
* filer/postgres: accept boolean createTable = true/false
viper reads a TOML boolean as the string "true"/"false" via GetString, so
createTable = true was being used as a SQL template and failed. Add
ResolveCreateTableQuery to normalize the value: true and empty select the
default template, false disables table creation, anything else is a custom
template. Both postgres and postgres2 now use it, and both skip the CREATE
TABLE call when the resolved template is empty.
* scaffold: document createTable option for postgres filer store
Replace the commented-out CREATE TABLE SQL in the [postgres] scaffold with
a createTable config hint, matching the [postgres2] section. Users no longer
need to manually create the filemeta table before starting the filer.
* filer/postgres: make createTable opt-in for postgres, keep postgres2 default
The previous commit defaulted postgres to create the filemeta table even
when createTable was unset, which could break existing deployments whose
DB user lacks CREATE TABLE privileges. ResolveCreateTableQuery now returns
empty for an unset value so postgres only creates the table when
createTable is explicitly true or a custom template — preserving the
prior no-DDL behaviour for existing configurations.
postgres2 keeps its existing always-create default: it defaults an empty
resolved value to DefaultCreateTableQuery, and only skips when
createTable is explicitly false.
* filer/postgres2: simplify createTable handling, document all modes
Drop the false opt-out from postgres2 — it only skipped the default table
while per-bucket CreateTable still ran, leaving restricted DB roles broken
on bucket access. postgres2 now accepts true the same way (defaulting to
DefaultCreateTableQuery) and keeps its existing always-create behaviour
for every other value, matching the original semantics.
The scaffold comment now documents true/false/custom for the postgres
section so users know false (or unset) is the backward-compatible default.
* filer/postgres2: normalize false via ResolveCreateTableQuery
postgres2 only handled "" and "true", leaving createTable = false as the
literal string "false" which CreateTable then executed as invalid SQL.
Route it through ResolveCreateTableQuery (which maps false to empty) and
default the empty result to DefaultCreateTableQuery, so false is treated
the same as unset for the bucket-aware store.
* filer/postgres2: honor createTable = false for default table
postgres2 treated false the same as unset and always created the default
filemeta table, failing startup for restricted DB roles that explicitly
opted out. Track the original false value before ResolveCreateTableQuery
collapses it to empty, and skip the default CreateTable call when set.
Per-bucket table creation is unaffected — it is a runtime requirement of
the bucket-aware store. Users who need to suppress all DDL should use the
postgres (non-bucket) store with createTable unset.
* filer/postgres2: disable bucket tables when createTable = false
Setting SupportBucketTable = false when createTable is explicitly false
makes AbstractSqlStore.CreateTable a no-op (it already returns nil when
SupportBucketTable is false), so neither the default filemeta table nor
per-bucket tables are created. The template stays empty and no DDL runs,
honouring the opt-out for restricted DB roles. All data routes to the
pre-provisioned filemeta table, matching the postgres (non-bucket) store.
* filer: suppress DDL without disabling bucket routing
Setting SupportBucketTable = false when createTable = false also disabled
per-bucket routing, hiding objects in pre-provisioned per-bucket tables.
Keep SupportBucketTable true and instead skip the CREATE TABLE execution
when the resolved template is empty.
GetSqlCreateTable now returns empty for both postgres and mysql SQL
generators when CreateTableSqlTemplate is empty, and
AbstractSqlStore.CreateTable skips the ExecContext call when the SQL is
empty. This preserves bucket routing while suppressing all DDL for users
who explicitly set createTable = false and pre-provision their tables.
* filer: add SkipDDL to suppress CREATE and DROP without disabling routing
createTable = false with SupportBucketTable = true preserved bucket
routing but deleteTable still executed DROP TABLE on bucket deletion,
dropping externally managed tables. CanDropWholeBucket also returned
true, so the S3 layer tried whole-table drops instead of row-by-row
deletes.
Add a SkipDDL flag to AbstractSqlStore, independent of SupportBucketTable.
CreateTable and deleteTable both skip when SkipDDL is set, and
CanDropWholeBucket returns false so bucket deletion falls back to
row-by-row metadata deletes. postgres2 sets SkipDDL when createTable is
explicitly false — bucket routing is preserved, no DDL runs.
* filer: fall back to row-by-row delete when CanDropWholeBucket is false
DeleteFolderChildren took the whole-table drop path whenever the path was
a bucket root, even when SkipDDL made deleteTable a no-op. The no-op
returned nil, the caller returned early, and rows inserted after the
recursive enumeration survived the bucket deletion.
Gate the whole-table drop on CanDropWholeBucket so the row-by-row
DeleteFolderChildren SQL runs when SkipDDL is set, removing all metadata
without issuing DROP TABLE.
* filer: give the SQL stores' key-value reads their own connections
A listing holds the connection its rows are on for the whole iteration, and
FilerStoreWrapper calls maybeReadHardLink -> KvGet from inside that iteration,
so a hard-linked entry needs a second connection while the first is still busy.
Out of one bounded pool that is a deadlock: the listings fill the pool and then
wait for a connection none of them will release, and the wrapper's
context.WithoutCancel leaves the waiters without a deadline, so the filer stays
wedged rather than erroring.
The sqlite store shows it at its sharpest -- it allows a single connection, so
one listing over one hard-linked entry never returns. On postgres with
connection_max_open = 50, 60 concurrent listings over hard-linked entries made
no progress at all.
Key-value reads now run on their own pool, carved out of connection_max_open
rather than added to it, so the operator's cap still bounds what the store opens
against the database. An unbounded pool keeps a single pool: nothing can wait
there. sqlite's single connection becomes two, one per pool, and its writes get
a busy timeout so a write that meets the reader waits instead of failing.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* sqlite: keep both pools on one database, whatever the dbFile spells
A dbFile that already carries URI options got a second "?" appended, which the
driver reads as part of the preceding option value, and a bare :memory: is
private to each connection, so the key-value pool would open its own empty
database and every key-value operation would fail on a missing filemeta.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* sqlite: assert the busy timeout on the in-memory DSN too
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* filer: SQL store pool defaults survive concurrent walks (idle == open == 50, lifetime 300s)
The code defaults for the four SQL stores were connection_max_idle=2 with NO
default for connection_max_open (unlimited) or lifetime, while the scaffold
filer.toml documents 10/50/300 -- so an env-configured or minimal-toml filer got
the worst possible pool. Under a concurrent listing burst (s3.lifecycle.run-shard
walks 16 shards in parallel) every operation released above the 2 idle slots
closes its TCP connection, so the walk opens a fresh connection per operation
until the filer exhausts its ephemeral ports:
list /buckets/... : failed to connect ... dial tcp ...:5432:
connect: cannot assign requested address
Measured on a production filer: 0 -> 28k TIME_WAIT with only ~1.3k concurrent,
and in the minimal docker-compose reproduction (2000-dir bucket, port range
narrowed to 400): the whole range in TIME_WAIT with only ~12 ESTABLISHED.
Default all three knobs, with idle == open so released connections are kept and
reused: idle connections only accumulate up to the actual peak concurrency and
connection_max_lifetime_seconds recycles them, so a quiet deployment holds
nothing extra. An explicit 0 still disables the caps as before. The scaffold's
connection_max_idle moves 10 -> 50 to match.
With this change the same reproduction completes all 16 shards with the default
configuration (TIME_WAIT peak 19 vs the whole port range).
* filer: trim the SQL pool default comments
One line of the non-obvious why is enough; the rest narrated the code.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* filer: leave the SQL stores' connection_max_open unset
A listing holds its connection for the whole row iteration while its callback
runs another query -- FilerStoreWrapper.maybeReadHardLink does a KvGet per
hard-linked entry -- so every concurrent listing needs two connections from the
same pool. With a default cap, listings past the cap wedge: 60 concurrent
listings over hard-linked entries made no progress at all against a 50
connection pool, and the wrapper's context.WithoutCancel leaves the waiters
without a deadline.
The idle pool is what fixes the connection churn: idle 50 with an unbounded
max_open holds the same 14 postgres sessions across a 16-way listing burst that
opened 455 with idle 2.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Concurrent bucket deletion across multiple filer replicas races on the
per-bucket DROP TABLE. The first replica drops the table; the rest hit
an undefined-table error (postgres 42P01, mysql 1051) which propagates
out of DeleteFolderChildren and panics the filer. On restart the same
pending DROP re-runs and the filer crash-loops.
Make the drop idempotent. Same defect class in all SQL backends, so fix
postgres, postgres2, mysql, mysql2, and sqlite together.
* mysql: keep S3 list order byte-lexicographic regardless of name column collation
ORDER BY name and the name > ? pagination predicate follow the column
collation, so a case-insensitive filemeta.name (e.g. utf8mb3_general_ci)
returns S3 keys out of byte order and breaks clients that merge two sorted
listings.
Detect the live name collation at startup; only when it isn't binary, wrap
the list comparison, prefix, and ORDER BY in BINARY name so order and
pagination stay consistent. Correctly configured utf8mb4_bin tables keep
their indexed range scan unchanged, and the operator gets a warning to
convert the column.
* postgres: keep S3 list order byte-lexicographic regardless of name column collation
ORDER BY name and the name > $n pagination predicate follow the column or
database collation, so a locale-aware filemeta.name (e.g. the en_US.UTF-8
database default) returns S3 keys out of byte order and breaks clients that
merge two sorted listings.
Detect the live name collation at startup; only when it isn't byte-ordered,
wrap the list comparison, prefix, and ORDER BY in COLLATE "C" so order and
pagination stay consistent. A byte-ordered (C/POSIX/C.UTF-8) column keeps its
indexed range scan unchanged, and the operator gets a warning to declare the
column COLLATE "C".
* fix(filer/postgres): default to ON CONFLICT upsert to keep tx alive
A KvPut from the inode-index secondary write could fail with 23505
(duplicate key) inside a rename's transaction, after which the next
statement returned 25P02 and rename surfaced to FUSE as EIO. Default
the postgres upsert query when enableUpsert=true so INSERTs are
idempotent; the enableUpsert=false escape hatch is preserved for
non-PG-compatible backends.
* fix(filer/mysql): default to ON DUPLICATE KEY UPDATE upsert
Same shape as the postgres default: when enableUpsert=true but no
upsertQuery is configured, install a sensible default so the
inode-index KvPut does not waste a duplicate-key roundtrip on every
entry write. Uses the VALUES() form so the default works on MariaDB
and MySQL >=5.7; the MySQL 8.0.19 row-alias form is left to explicit
config.
* fix(filer): default enableUpsert=true for sql stores
The default-template fallback only kicks in when enableUpsert=true,
so minimal configs that omit the flag entirely were still exposed.
Default it on for postgres/postgres2/mysql/mysql2; an explicit false
in filer.toml still wins because SetDefault only fills absent keys.
* fix(filer): don't disable the SQL idle connection pool when unconfigured
The mysql/mysql2/postgres stores called SetMaxIdleConns(maxIdle)
unconditionally, so an unset connection_max_idle (0) actively kept zero
idle connections - every query opened and closed a fresh connection
instead of reusing the pool.
Only apply the value when it's set; otherwise leave database/sql's
default idle pool of 2 in place.
* comments: shorten idle-pool note
* fix(filer): default the SQL idle pool via config, keep explicit 0 honored
Apply the idle-pool default at the config layer with SetDefault instead of
guarding the SetMaxIdleConns call. An absent connection_max_idle now reads
back as 2 (pool stays on), while an explicit 0 flows through to
SetMaxIdleConns(0) so operators can still disable idle pooling on purpose.
* fix(filer/postgres): use pgx v5 API for PgBouncer simple protocol
In pgx/v5 the `prefer_simple_protocol` DSN parameter was removed, so
appending it to the connection string caused PgBouncer/PostgreSQL to
reject it as an unknown startup parameter:
FATAL: unsupported startup parameter: prefer_simple_protocol (SQLSTATE 08P01)
Parse the DSN with pgx.ParseConfig and, when pgbouncer_compatible is
set, configure DefaultQueryExecMode = QueryExecModeSimpleProtocol and
disable the statement/description caches. Register the config via
stdlib.RegisterConnConfig before sql.Open.
Fixes#9005
* refactor(filer/postgres): extract shared OpenPGXDB helper with cleanup
Extract the pgx v5 ParseConfig/RegisterConnConfig/sql.Open/Ping logic
into a shared postgres.OpenPGXDB helper used by both postgres and
postgres2 filer stores, eliminating ~60 lines of duplication.
The helper also unregisters the conn config via stdlib.UnregisterConnConfig
on every failure path (sql.Open error, Ping error) so we do not leak
entries in stdlib's global connection config map when initialization
fails.
* refactor(filer/postgres): use stdlib.OpenDB to avoid conn config leak
Switch OpenPGXDB from RegisterConnConfig + sql.Open("pgx", connStr) to
stdlib.OpenDB(*connConfig). The former leaks an entry in stdlib's global
conn config map on every successful initialization; stdlib.OpenDB takes
the config directly and keeps no global registration.
Addresses CodeRabbit review feedback on #9010.
Chris Lu