13 Commits
Author SHA1 Message Date
Chris Lu 9f6efc8b53 filer: a listing over a hard link no longer deadlocks a bounded SQL pool (#11118)
* filer: give the SQL stores' key-value reads their own connections

A listing holds the connection its rows are on for the whole iteration, and
FilerStoreWrapper calls maybeReadHardLink -> KvGet from inside that iteration,
so a hard-linked entry needs a second connection while the first is still busy.
Out of one bounded pool that is a deadlock: the listings fill the pool and then
wait for a connection none of them will release, and the wrapper's
context.WithoutCancel leaves the waiters without a deadline, so the filer stays
wedged rather than erroring.

The sqlite store shows it at its sharpest -- it allows a single connection, so
one listing over one hard-linked entry never returns. On postgres with
connection_max_open = 50, 60 concurrent listings over hard-linked entries made
no progress at all.

Key-value reads now run on their own pool, carved out of connection_max_open
rather than added to it, so the operator's cap still bounds what the store opens
against the database. An unbounded pool keeps a single pool: nothing can wait
there. sqlite's single connection becomes two, one per pool, and its writes get
a busy timeout so a write that meets the reader waits instead of failing.

Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t

* sqlite: keep both pools on one database, whatever the dbFile spells

A dbFile that already carries URI options got a second "?" appended, which the
driver reads as part of the preceding option value, and a bare :memory: is
private to each connection, so the key-value pool would open its own empty
database and every key-value operation would fail on a missing filemeta.

Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t

* sqlite: assert the busy timeout on the in-memory DSN too

Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
2026-09-02 23:45:45 -07:00
Chris Lu 803c5a8dca fix(filer): use DROP TABLE IF EXISTS in SQL stores (#10158)
Concurrent bucket deletion across multiple filer replicas races on the
per-bucket DROP TABLE. The first replica drops the table; the rest hit
an undefined-table error (postgres 42P01, mysql 1051) which propagates
out of DeleteFolderChildren and panics the filer. On restart the same
pending DROP re-runs and the filer crash-loops.

Make the drop idempotent. Same defect class in all SQL backends, so fix
postgres, postgres2, mysql, mysql2, and sqlite together.
2026-06-30 10:38:51 -07:00
Chris Lu e4b70c2521 go fix 2026-02-20 18:42:00 -08:00
chrislu cea73ac008 serialize sqlite operations
fix https://github.com/seaweedfs/seaweedfs/issues/3827
2022-10-12 00:03:27 -07:00
chrislu dff85e9c71 fix error handling 2022-10-12 00:03:14 -07:00
chrislu 21c0587900 go fmt 2022-09-14 23:06:44 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu 4aa39ef33f conditionally compile sqlite 2022-04-21 01:22:38 -07:00
Chris Lu 7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
陈杨文 6f683e6572 also failed on mips64
alpine support mips64, will try to build on next version
2021-06-30 04:36:00 +08:00
Chris Lu 41db292332 skip s390 ppc64le
due to https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/22703
2021-06-29 10:06:38 -07:00
Chris Lu 67dd094b35 sqlite: does not support non-linux/darwin/windows 2021-05-25 17:36:15 -07:00
Chris Lu 064269bb57 filer: Support sqlite as filer meta store 2021-05-23 23:58:28 -07:00