* filer: give the SQL stores' key-value reads their own connections
A listing holds the connection its rows are on for the whole iteration, and
FilerStoreWrapper calls maybeReadHardLink -> KvGet from inside that iteration,
so a hard-linked entry needs a second connection while the first is still busy.
Out of one bounded pool that is a deadlock: the listings fill the pool and then
wait for a connection none of them will release, and the wrapper's
context.WithoutCancel leaves the waiters without a deadline, so the filer stays
wedged rather than erroring.
The sqlite store shows it at its sharpest -- it allows a single connection, so
one listing over one hard-linked entry never returns. On postgres with
connection_max_open = 50, 60 concurrent listings over hard-linked entries made
no progress at all.
Key-value reads now run on their own pool, carved out of connection_max_open
rather than added to it, so the operator's cap still bounds what the store opens
against the database. An unbounded pool keeps a single pool: nothing can wait
there. sqlite's single connection becomes two, one per pool, and its writes get
a busy timeout so a write that meets the reader waits instead of failing.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* sqlite: keep both pools on one database, whatever the dbFile spells
A dbFile that already carries URI options got a second "?" appended, which the
driver reads as part of the preceding option value, and a bare :memory: is
private to each connection, so the key-value pool would open its own empty
database and every key-value operation would fail on a missing filemeta.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* sqlite: assert the busy timeout on the in-memory DSN too
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* filer: SQL store pool defaults survive concurrent walks (idle == open == 50, lifetime 300s)
The code defaults for the four SQL stores were connection_max_idle=2 with NO
default for connection_max_open (unlimited) or lifetime, while the scaffold
filer.toml documents 10/50/300 -- so an env-configured or minimal-toml filer got
the worst possible pool. Under a concurrent listing burst (s3.lifecycle.run-shard
walks 16 shards in parallel) every operation released above the 2 idle slots
closes its TCP connection, so the walk opens a fresh connection per operation
until the filer exhausts its ephemeral ports:
list /buckets/... : failed to connect ... dial tcp ...:5432:
connect: cannot assign requested address
Measured on a production filer: 0 -> 28k TIME_WAIT with only ~1.3k concurrent,
and in the minimal docker-compose reproduction (2000-dir bucket, port range
narrowed to 400): the whole range in TIME_WAIT with only ~12 ESTABLISHED.
Default all three knobs, with idle == open so released connections are kept and
reused: idle connections only accumulate up to the actual peak concurrency and
connection_max_lifetime_seconds recycles them, so a quiet deployment holds
nothing extra. An explicit 0 still disables the caps as before. The scaffold's
connection_max_idle moves 10 -> 50 to match.
With this change the same reproduction completes all 16 shards with the default
configuration (TIME_WAIT peak 19 vs the whole port range).
* filer: trim the SQL pool default comments
One line of the non-obvious why is enough; the rest narrated the code.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
* filer: leave the SQL stores' connection_max_open unset
A listing holds its connection for the whole row iteration while its callback
runs another query -- FilerStoreWrapper.maybeReadHardLink does a KvGet per
hard-linked entry -- so every concurrent listing needs two connections from the
same pool. With a default cap, listings past the cap wedge: 60 concurrent
listings over hard-linked entries made no progress at all against a 50
connection pool, and the wrapper's context.WithoutCancel leaves the waiters
without a deadline.
The idle pool is what fixes the connection churn: idle 50 with an unbounded
max_open holds the same 14 postgres sessions across a 16-way listing burst that
opened 455 with idle 2.
Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Concurrent bucket deletion across multiple filer replicas races on the
per-bucket DROP TABLE. The first replica drops the table; the rest hit
an undefined-table error (postgres 42P01, mysql 1051) which propagates
out of DeleteFolderChildren and panics the filer. On restart the same
pending DROP re-runs and the filer crash-loops.
Make the drop idempotent. Same defect class in all SQL backends, so fix
postgres, postgres2, mysql, mysql2, and sqlite together.
* mysql: keep S3 list order byte-lexicographic regardless of name column collation
ORDER BY name and the name > ? pagination predicate follow the column
collation, so a case-insensitive filemeta.name (e.g. utf8mb3_general_ci)
returns S3 keys out of byte order and breaks clients that merge two sorted
listings.
Detect the live name collation at startup; only when it isn't binary, wrap
the list comparison, prefix, and ORDER BY in BINARY name so order and
pagination stay consistent. Correctly configured utf8mb4_bin tables keep
their indexed range scan unchanged, and the operator gets a warning to
convert the column.
* postgres: keep S3 list order byte-lexicographic regardless of name column collation
ORDER BY name and the name > $n pagination predicate follow the column or
database collation, so a locale-aware filemeta.name (e.g. the en_US.UTF-8
database default) returns S3 keys out of byte order and breaks clients that
merge two sorted listings.
Detect the live name collation at startup; only when it isn't byte-ordered,
wrap the list comparison, prefix, and ORDER BY in COLLATE "C" so order and
pagination stay consistent. A byte-ordered (C/POSIX/C.UTF-8) column keeps its
indexed range scan unchanged, and the operator gets a warning to declare the
column COLLATE "C".
* fix(filer/postgres): default to ON CONFLICT upsert to keep tx alive
A KvPut from the inode-index secondary write could fail with 23505
(duplicate key) inside a rename's transaction, after which the next
statement returned 25P02 and rename surfaced to FUSE as EIO. Default
the postgres upsert query when enableUpsert=true so INSERTs are
idempotent; the enableUpsert=false escape hatch is preserved for
non-PG-compatible backends.
* fix(filer/mysql): default to ON DUPLICATE KEY UPDATE upsert
Same shape as the postgres default: when enableUpsert=true but no
upsertQuery is configured, install a sensible default so the
inode-index KvPut does not waste a duplicate-key roundtrip on every
entry write. Uses the VALUES() form so the default works on MariaDB
and MySQL >=5.7; the MySQL 8.0.19 row-alias form is left to explicit
config.
* fix(filer): default enableUpsert=true for sql stores
The default-template fallback only kicks in when enableUpsert=true,
so minimal configs that omit the flag entirely were still exposed.
Default it on for postgres/postgres2/mysql/mysql2; an explicit false
in filer.toml still wins because SetDefault only fills absent keys.
* fix(filer): don't disable the SQL idle connection pool when unconfigured
The mysql/mysql2/postgres stores called SetMaxIdleConns(maxIdle)
unconditionally, so an unset connection_max_idle (0) actively kept zero
idle connections - every query opened and closed a fresh connection
instead of reusing the pool.
Only apply the value when it's set; otherwise leave database/sql's
default idle pool of 2 in place.
* comments: shorten idle-pool note
* fix(filer): default the SQL idle pool via config, keep explicit 0 honored
Apply the idle-pool default at the config layer with SetDefault instead of
guarding the SetMaxIdleConns call. An absent connection_max_idle now reads
back as 2 (pool stays on), while an explicit 0 flows through to
SetMaxIdleConns(0) so operators can still disable idle pooling on purpose.
* fix(filer/postgres): use pgx v5 API for PgBouncer simple protocol
In pgx/v5 the `prefer_simple_protocol` DSN parameter was removed, so
appending it to the connection string caused PgBouncer/PostgreSQL to
reject it as an unknown startup parameter:
FATAL: unsupported startup parameter: prefer_simple_protocol (SQLSTATE 08P01)
Parse the DSN with pgx.ParseConfig and, when pgbouncer_compatible is
set, configure DefaultQueryExecMode = QueryExecModeSimpleProtocol and
disable the statement/description caches. Register the config via
stdlib.RegisterConnConfig before sql.Open.
Fixes#9005
* refactor(filer/postgres): extract shared OpenPGXDB helper with cleanup
Extract the pgx v5 ParseConfig/RegisterConnConfig/sql.Open/Ping logic
into a shared postgres.OpenPGXDB helper used by both postgres and
postgres2 filer stores, eliminating ~60 lines of duplication.
The helper also unregisters the conn config via stdlib.UnregisterConnConfig
on every failure path (sql.Open error, Ping error) so we do not leak
entries in stdlib's global connection config map when initialization
fails.
* refactor(filer/postgres): use stdlib.OpenDB to avoid conn config leak
Switch OpenPGXDB from RegisterConnConfig + sql.Open("pgx", connStr) to
stdlib.OpenDB(*connConfig). The former leaks an entry in stdlib's global
conn config map on every successful initialization; stdlib.OpenDB takes
the config directly and keeps no global registration.
Addresses CodeRabbit review feedback on #9010.
Chris Lu