filer: a listing over a hard link no longer deadlocks a bounded SQL pool (#11118)

* filer: give the SQL stores' key-value reads their own connections

A listing holds the connection its rows are on for the whole iteration, and
FilerStoreWrapper calls maybeReadHardLink -> KvGet from inside that iteration,
so a hard-linked entry needs a second connection while the first is still busy.
Out of one bounded pool that is a deadlock: the listings fill the pool and then
wait for a connection none of them will release, and the wrapper's
context.WithoutCancel leaves the waiters without a deadline, so the filer stays
wedged rather than erroring.

The sqlite store shows it at its sharpest -- it allows a single connection, so
one listing over one hard-linked entry never returns. On postgres with
connection_max_open = 50, 60 concurrent listings over hard-linked entries made
no progress at all.

Key-value reads now run on their own pool, carved out of connection_max_open
rather than added to it, so the operator's cap still bounds what the store opens
against the database. An unbounded pool keeps a single pool: nothing can wait
there. sqlite's single connection becomes two, one per pool, and its writes get
a busy timeout so a write that meets the reader waits instead of failing.

Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t

* sqlite: keep both pools on one database, whatever the dbFile spells

A dbFile that already carries URI options got a second "?" appended, which the
driver reads as part of the preceding option value, and a bare :memory: is
private to each connection, so the key-value pool would open its own empty
database and every key-value operation would fail on a missing filemeta.

Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t

* sqlite: assert the busy timeout on the in-memory DSN too

Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
This commit is contained in:
Chris Lu authored and GitHub committed 2026-09-02 23:45:45 -07:00
1 parent 9b61289293
commit 9f6efc8b53
9 files changed
+228 -35

No files matched your search

+28 -7
View File
@@ -8,6 +8,8 @@ import (
"context"
"database/sql"
"fmt"
"strings"
"time"
"github.com/seaweedfs/seaweedfs/weed/filer"
"github.com/seaweedfs/seaweedfs/weed/filer/abstract_sql"
@@ -49,6 +51,21 @@ func (store *SqliteStore) Initialize(configuration util.Configuration, prefix st
)
}
// sqliteDSN keeps the store's two pools on one database and lets a writer that
// meets the other pool's reader wait for it instead of failing outright. A bare
// :memory: is private to each connection, so it has to be named and shared.
func sqliteDSN(dbFile string) string {
dsn := dbFile
if dsn == ":memory:" {
dsn = fmt.Sprintf("file:seaweedfs%d?mode=memory&cache=shared", time.Now().UnixNano())
}
separator := "?"
if strings.Contains(dsn, "?") {
separator = "&"
}
return dsn + separator + "_pragma=busy_timeout(10000)"
}
func (store *SqliteStore) initialize(dbFile, createTable, upsertQuery string) (err error) {
store.SupportBucketTable = true
@@ -58,21 +75,25 @@ func (store *SqliteStore) initialize(dbFile, createTable, upsertQuery string) (e
UpsertQueryTemplate: upsertQuery,
}
var dbErr error
store.DB, dbErr = sql.Open("sqlite", dbFile)
dsn := sqliteDSN(dbFile)
db, dbErr := sql.Open("sqlite", dsn)
if dbErr != nil {
if store.DB != nil {
store.DB.Close()
store.DB = nil
if db != nil {
db.Close()
}
return fmt.Errorf("can not connect to %s error:%v", dbFile, dbErr)
}
if err = store.DB.Ping(); err != nil {
if err = db.Ping(); err != nil {
return fmt.Errorf("connect to %s error:%v", dbFile, err)
}
store.DB.SetMaxOpenConns(1)
if err = store.UseConnectionPools(db, func() (*sql.DB, error) {
return sql.Open("sqlite", dsn)
}, 1, 1, 0); err != nil {
return err
}
if err = store.CreateTable(context.Background(), abstract_sql.DEFAULT_TABLE); err != nil {
return fmt.Errorf("init table %s: %v", abstract_sql.DEFAULT_TABLE, err)