Commit Graph
14 Commits
Author SHA1 Message Date
Chris Lu a5ccefb661 filer: escape LIKE wildcards in SQL list prefixes (#11757)
* filer: escape LIKE wildcards in SQL list prefixes

ListDirectoryPrefixedEntries bound prefix+"%" straight into a LIKE
clause, so '_' and '%' in object names (e.g. Langfuse keys like
call_03_...) acted as wildcards: wrong results and a ~40x slower
range scan because (dirhash, name) can no longer be narrowed.

Escape '!'/'%'/'_' in the prefix and declare ESCAPE '!' in the
generated list SQL so every abstract_sql backend (mysql, mysql2,
postgres, postgres2, sqlite) matches literally with identical
semantics. '!' is used instead of '\' because SQLite shares
SqlGenMysql but would reject the MySQL-style '\\' literal, and
'\' would still need escaping itself.

* filer: escape LIKE wildcards in YDB list prefixes

Same wildcard-in-prefix bug as the abstract_sql backends: YQL LIKE
treated '_' and '%' in the prefix as wildcards. Apply the shared
LikePattern helper and declare ESCAPE '!' (YQL allows any escape
char except '%', '_' and '\').

* test: cover literal '_'/'%'/'!' prefixes in sqlite filer listing

End-to-end regression test through SqliteStore against a real
database; runs in CI under the sqlite build tag.
2026-10-15 16:51:15 -07:00
Chris Lu 9f6efc8b53 filer: a listing over a hard link no longer deadlocks a bounded SQL pool (#11118)
* filer: give the SQL stores' key-value reads their own connections

A listing holds the connection its rows are on for the whole iteration, and
FilerStoreWrapper calls maybeReadHardLink -> KvGet from inside that iteration,
so a hard-linked entry needs a second connection while the first is still busy.
Out of one bounded pool that is a deadlock: the listings fill the pool and then
wait for a connection none of them will release, and the wrapper's
context.WithoutCancel leaves the waiters without a deadline, so the filer stays
wedged rather than erroring.

The sqlite store shows it at its sharpest -- it allows a single connection, so
one listing over one hard-linked entry never returns. On postgres with
connection_max_open = 50, 60 concurrent listings over hard-linked entries made
no progress at all.

Key-value reads now run on their own pool, carved out of connection_max_open
rather than added to it, so the operator's cap still bounds what the store opens
against the database. An unbounded pool keeps a single pool: nothing can wait
there. sqlite's single connection becomes two, one per pool, and its writes get
a busy timeout so a write that meets the reader waits instead of failing.

Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t

* sqlite: keep both pools on one database, whatever the dbFile spells

A dbFile that already carries URI options got a second "?" appended, which the
driver reads as part of the preceding option value, and a bare :memory: is
private to each connection, so the key-value pool would open its own empty
database and every key-value operation would fail on a missing filemeta.

Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t

* sqlite: assert the busy timeout on the in-memory DSN too

Claude-Session: https://claude.ai/code/session_018DWwctzD4T2DmnczPRM47t
2026-09-02 23:45:45 -07:00
Chris Lu 803c5a8dca fix(filer): use DROP TABLE IF EXISTS in SQL stores (#10158)
Concurrent bucket deletion across multiple filer replicas races on the
per-bucket DROP TABLE. The first replica drops the table; the rest hit
an undefined-table error (postgres 42P01, mysql 1051) which propagates
out of DeleteFolderChildren and panics the filer. On restart the same
pending DROP re-runs and the filer crash-loops.

Make the drop idempotent. Same defect class in all SQL backends, so fix
postgres, postgres2, mysql, mysql2, and sqlite together.
2026-06-30 10:38:51 -07:00
Chris Lu e4b70c2521 go fix 2026-02-20 18:42:00 -08:00
chrislu cea73ac008 serialize sqlite operations
fix https://github.com/seaweedfs/seaweedfs/issues/3827
2022-10-12 00:03:27 -07:00
chrislu dff85e9c71 fix error handling 2022-10-12 00:03:14 -07:00
chrislu 21c0587900 go fmt 2022-09-14 23:06:44 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu 4aa39ef33f conditionally compile sqlite 2022-04-21 01:22:38 -07:00
Chris Lu 7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
陈杨文 6f683e6572 also failed on mips64
alpine support mips64, will try to build on next version
2021-06-30 04:36:00 +08:00
Chris Lu 41db292332 skip s390 ppc64le
due to https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/22703
2021-06-29 10:06:38 -07:00
Chris Lu 67dd094b35 sqlite: does not support non-linux/darwin/windows 2021-05-25 17:36:15 -07:00
Chris Lu 064269bb57 filer: Support sqlite as filer meta store 2021-05-23 23:58:28 -07:00