filer: a listing over a hard link no longer deadlocks a bounded SQL pool (#11118)

* filer: give the SQL stores' key-value reads their own connections

A listing holds the connection its rows are on for the whole iteration, and
FilerStoreWrapper calls maybeReadHardLink -> KvGet from inside that iteration,
so a hard-linked entry needs a second connection while the first is still busy.
Out of one bounded pool that is a deadlock: the listings fill the pool and then
wait for a connection none of them will release, and the wrapper's
context.WithoutCancel leaves the waiters without a deadline, so the filer stays
wedged rather than erroring.

The sqlite store shows it at its sharpest -- it allows a single connection, so
one listing over one hard-linked entry never returns. On postgres with
connection_max_open = 50, 60 concurrent listings over hard-linked entries made
no progress at all.

Key-value reads now run on their own pool, carved out of connection_max_open
rather than added to it, so the operator's cap still bounds what the store opens
against the database. An unbounded pool keeps a single pool: nothing can wait
there. sqlite's single connection becomes two, one per pool, and its writes get
a busy timeout so a write that meets the reader waits instead of failing.

Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t

* sqlite: keep both pools on one database, whatever the dbFile spells

A dbFile that already carries URI options got a second "?" appended, which the
driver reads as part of the preceding option value, and a bare :memory: is
private to each connection, so the key-value pool would open its own empty
database and every key-value operation would fail on a missing filemeta.

Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t

* sqlite: assert the busy timeout on the in-memory DSN too

Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
This commit is contained in:
Chris Lu authored and GitHub committed 2026-09-02 23:45:45 -07:00
1 parent 9b61289293
commit 9f6efc8b53
9 files changed
+228 -35

No files matched your search

+6 -1
View File
@@ -9,6 +9,7 @@ package postgres2
import (
"context"
"database/sql"
"fmt"
"strconv"
@@ -128,7 +129,11 @@ func (store *PostgresStore2) initialize(createTable, upsertQuery string, enableU
if openErr != nil {
return openErr
}
store.DB = db
if err = store.UseConnectionPools(db, func() (*sql.DB, error) {
return postgres.OpenPGXDB(sqlUrl, adaptedSqlUrl, pgbouncerCompatible, maxIdle, maxOpen, maxLifetimeSeconds)
}, maxIdle, maxOpen, maxLifetimeSeconds); err != nil {
return err
}
if err = store.CreateTable(context.Background(), abstract_sql.DEFAULT_TABLE); err != nil {
return fmt.Errorf("init table %s: %v", abstract_sql.DEFAULT_TABLE, err)