mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 15:41:15 +02:00
Chris Lu
9e34426a56
* lance: a maintenance job that sorts a table by its declared fields Lance appends fragments in write order and has no notion of a sorted table, so nothing but a rewrite establishes one, and nothing but another rewrite restores it once rows have been appended. lance_sort reads the order from the dataset's own configuration, falls back to the worker's, and rewrites the table in it. The spec and the marker live in crates/sort rather than in the job, because weed/worker/tasks/iceberg sorts too: two jobs that disagreed about what "id desc nulls-first" means would be two features wearing one name. The sort spills. lance builds its DataFusion runtime with a FairSpillPool and a disk manager, but only when LanceExecutionOptions::use_spilling is set, and that struct derives Default over a plain bool — so Scanner::try_into_stream, which fills its options with ..Default::default(), is precisely the path that does not spill. The job builds the plan with create_plan and executes it with spilling on and the operator's memory budget. The marker rides in the same commit as the data: Operation::Overwrite is the one operation carrying config values alongside fragments, so a sorted table and the record of its sorting cannot disagree. It records the version the sort read, not the one it wrote, which is not knowable while the marker is being assembled. Detection treats anything committed after the sort's own commit as data the sort did not produce — row counts alone cannot see a rewrite that leaves the count where it was, and such a table would look sorted forever. Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y * lance: identify a sorted table by the files it wrote, not its version Review found three ways the version-based marker misjudges a table, and they share a cause: the version a sort produces is not knowable while the marker is being assembled, so the marker recorded the version it read and detection inferred the rest. A commit that rebases past a conflict lands on a different number, and the inference then reads a rewrite into an ordinary table — a full re-sort, and its indices, for nothing. Data file names do not have that problem. They are chosen before the commit, so the commit can carry them, and they do not change with the version it lands on. The marker now records how many files the sort wrote and a digest of their names, and detection asks whether the table still holds them: the same files means untouched, the same files followed by more means appended, anything else means the data was replaced. That also closes the hole the row threshold left. A replacement that grew the table by fewer rows than min_unsorted_rows read as sorted, however many rows had actually moved; the threshold now applies only where the sorted files are still in place, which is what it was for. A marker without a row count is stale rather than a zero to compare against, and deletes stop forcing a re-sort — they write a deletion file beside the data rather than rewriting it, and removing rows does not unsort the ones that remain. Sort fields are also compared exactly rather than case-folded. Arrow schemas are case-sensitive, so `id` and `ID` are two columns, and folding them together rejected a valid order. Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y * lance: count the rows appended after a sort, not the table's net growth Review found that rows deleted from the sorted fragments hide appended rows one for one: the threshold compared the live row count against the count recorded at sort time, so 800 deletions and 300 appends read as a table that shrank, and a table where deletions keep pace with appends stays "sorted" with an unsorted tail forever. The fragments say it directly. The marker already records how many fragments the sort wrote, so the ones after that prefix are exactly what arrived since, and the manifest carries each fragment's live row count — physical rows less its deletions. Counting those is the arithmetic the threshold was always meant to do, and it needs no row count from the marker at all. A fragment whose length the manifest does not record cannot be counted, and a table that cannot be judged is one to sort rather than one to leave alone forever, so an uncountable appended fragment reads as stale. Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y
29 lines
972 B
TOML
29 lines
972 B
TOML
# Rust plugin workers for SeaweedFS.
|
|
#
|
|
# `core` is the plugin.proto contract and nothing else; a worker crate beside it
|
|
# supplies job handlers and a binary. Adding a worker means adding a member here,
|
|
# not touching the protocol. `sort` is neither: it is the sort specification the
|
|
# sorting jobs share, so that two of them cannot drift on what an order means.
|
|
[workspace]
|
|
resolver = "2"
|
|
members = ["crates/core", "crates/lance", "crates/sort"]
|
|
|
|
[workspace.package]
|
|
version = "0.1.0"
|
|
edition = "2021"
|
|
|
|
[workspace.dependencies]
|
|
anyhow = "1"
|
|
async-trait = "0.1"
|
|
prost = "0.13"
|
|
prost-types = "0.13"
|
|
tokio = { version = "1", features = ["full"] }
|
|
tokio-stream = "0.1"
|
|
tonic = { version = "0.12", features = ["tls"] }
|
|
# Already in the tree via tonic; named here so the metrics server can use them.
|
|
axum = "0.7"
|
|
prometheus = { version = "0.13", default-features = false }
|
|
tonic-build = "0.12"
|
|
tracing = "0.1"
|
|
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|