2 Commits

Include renames

Author	SHA1	Message	Date
3 people	adaf3534fa	rust: clippy-clean both crates and adopt the std APIs the 1.91 MSRV allows (#11312) ... * rust: apply clippy --fix to both crates The mechanical part of a clippy sweep: `cargo clippy --all-targets --fix` on seaweed-volume and the seaweed-worker workspace, hand-reviewed. Both manifests declare their MSRV (1.91.1 and 1.94.1), so every suggestion clippy applied is within it: the collapsible_if sites become let chains (1.88, edition 2024), `% n == 0` becomes is_multiple_of (1.87), chunks_exact with a constant becomes as_chunks (1.88), repeat().take() becomes repeat_n (1.82), and io::Error::new(Other, ..) becomes io::Error::other (1.74). The rest is redundant clones, borrows, casts, closures and field names. Nothing here changes behaviour. The three let_and_return sites in needle_map.rs and store_ec.rs deserve a note: the `let result = ..; result` shape was a deliberate edition-2021 workaround to drop a redb guard before the table it borrows. Edition 2024 drops tail-expression temporaries before locals, which is why clippy now flags it, and the two comments that described the workaround say so instead. Manual edits on top of the tool output: the blocks clippy rewrote are re-indented the way rustfmt lays them out (only those blocks — the crate is not rustfmt-clean and a whole-crate fmt would bury this diff), the blank lines let_and_return left behind are removed, and the CRC legacy_value test compares against a literal worked out from the original shift formula rather than restating rotate_right. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust: clear the clippy warnings --fix cannot apply, and say why the rest stay Hand fixes for the lints clippy only reports. Behaviour is unchanged throughout; each rewrite is the one clippy names. - needless_range_loop (7): index loops over shard vectors become iterator loops. Where the old code indexed `v[..n]` the new loop iterates `v[..n]` so an undersized vector still panics the same way. - field_reassign_with_default (6): struct literals with `..Default`. - redundant_pattern_matching (3): `if let Err(_) = guard.check()` becomes `.is_err()`, which also releases the read guard at the end of the condition instead of at the end of the block. - manual_strip (2), manual_checked_ops, format_in_format_args, redundant_locals, wrong_self_convention (to_vif takes self by value, so it is into_vif; CompactEntry is Copy, so to_needle_value takes self). - type_complexity (2): `OrphanShardLoad` and `RawNeedleEntry` name two tuples that were spelled out inline. - new_without_default: CompactNeedleMap gets a Default that calls new(). - suspicious_open_options: a test helper spells out `.truncate(false)`, which is what `.create(true).write(true)` already did. What stays, and the attribute that says so: - too_many_arguments (10): `#[expect]` on each function. Folding 8–15 parameters into a struct is a design change, not a lint fix. - await_holding_lock / readonly_write_lock: one test holds the store write guard across a sleep on purpose, as a barrier that parks the copy task at the mount block. `#[expect(.., reason = ..)]` records it. - module_inception: needle/needle.rs mirrors the Go package layout. Two lints become crate-wide policy in `[lints.clippy]`, with the reason next to each: result_large_err, because every RPC path returns tonic::Status (176 bytes) and boxing it would change every handler signature; and needless_update, because `..Default::default()` on a protobuf message literal is what lets a proto gain a field without touching every constructor (all 11 sites are pb messages). The worker workspace gets the same table and its members opt in with `lints.workspace = true`; its generated plugin.rs also allows large_enum_variant on prost's oneof enums. Both crates are now clean under `cargo clippy --all-targets -- -D warnings`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust volume: use the std APIs the 1.91 MSRV already pays for The crate declares rust-version 1.91.1, so a few things the code still worked around are plain std now. All of them come from the 1.85–1.91 release notes; nothing here needs a newer toolchain than the manifest already requires. - std::sync::LazyLock (1.80) replaces the lazy_static! block in metrics.rs, and the lazy_static dependency goes. Every use site reads the same through Deref, so no caller changes. - Duration::from_mins / from_hours (1.91) replace `from_secs(v * 60)` and `from_secs(v * 3600)` in the option parser and the shard-location refresh TTLs. One difference for the parser: an absurd count that overflows u64 seconds now panics in release builds too, where the multiplication used to wrap. - Result::flatten (1.89) replaces `.and_then(|r| r)` on the replication join handle. - OsStr::display (1.87) replaces `to_string_lossy()` where the name was only being formatted; the output is byte-identical. - `#[allow]` becomes `#[expect]` (1.81) on the suppressions that are meant to be permanent, so a suppression that stops being needed becomes a warning rather than lingering. Doing that found four that already had: dead_code on ChunkManifest, base_name and last_io_error, and too_many_arguments on read_from_data_shards, which is down to seven parameters. Those attributes are deleted. The three allows that depend on cfg (a unix-only mutation, a linux-only field set, a profiling-only parameter) stay as allow, because expect would be unfulfilled on the other platforms. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * ci: add a commented-out clippy step to both Rust workflows Both crates are warning-free under `cargo clippy --all-targets -D warnings` now. Whether that becomes a gate is a policy call, so the step is present but commented out; uncommenting it is the whole change. The comment points at the `[lints.clippy]` table where crate-wide exceptions are recorded, so the gate does not become a reason to sprinkle allows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust volume: guard parse_duration against overflow panics Duration::from_mins/from_hours panic when the count overflows u64 seconds. Use checked_mul so an oversized CLI value falls back to the parser default instead of crashing volume startup. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-09-14 11:29:29 -07:00
Chris Lu	9e34426a56	lance: a maintenance job that sorts a table by its declared fields (#11113) ... * lance: a maintenance job that sorts a table by its declared fields Lance appends fragments in write order and has no notion of a sorted table, so nothing but a rewrite establishes one, and nothing but another rewrite restores it once rows have been appended. lance_sort reads the order from the dataset's own configuration, falls back to the worker's, and rewrites the table in it. The spec and the marker live in crates/sort rather than in the job, because weed/worker/tasks/iceberg sorts too: two jobs that disagreed about what "id desc nulls-first" means would be two features wearing one name. The sort spills. lance builds its DataFusion runtime with a FairSpillPool and a disk manager, but only when LanceExecutionOptions::use_spilling is set, and that struct derives Default over a plain bool — so Scanner::try_into_stream, which fills its options with ..Default::default(), is precisely the path that does not spill. The job builds the plan with create_plan and executes it with spilling on and the operator's memory budget. The marker rides in the same commit as the data: Operation::Overwrite is the one operation carrying config values alongside fragments, so a sorted table and the record of its sorting cannot disagree. It records the version the sort read, not the one it wrote, which is not knowable while the marker is being assembled. Detection treats anything committed after the sort's own commit as data the sort did not produce — row counts alone cannot see a rewrite that leaves the count where it was, and such a table would look sorted forever. Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y * lance: identify a sorted table by the files it wrote, not its version Review found three ways the version-based marker misjudges a table, and they share a cause: the version a sort produces is not knowable while the marker is being assembled, so the marker recorded the version it read and detection inferred the rest. A commit that rebases past a conflict lands on a different number, and the inference then reads a rewrite into an ordinary table — a full re-sort, and its indices, for nothing. Data file names do not have that problem. They are chosen before the commit, so the commit can carry them, and they do not change with the version it lands on. The marker now records how many files the sort wrote and a digest of their names, and detection asks whether the table still holds them: the same files means untouched, the same files followed by more means appended, anything else means the data was replaced. That also closes the hole the row threshold left. A replacement that grew the table by fewer rows than min_unsorted_rows read as sorted, however many rows had actually moved; the threshold now applies only where the sorted files are still in place, which is what it was for. A marker without a row count is stale rather than a zero to compare against, and deletes stop forcing a re-sort — they write a deletion file beside the data rather than rewriting it, and removing rows does not unsort the ones that remain. Sort fields are also compared exactly rather than case-folded. Arrow schemas are case-sensitive, so `id` and `ID` are two columns, and folding them together rejected a valid order. Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y * lance: count the rows appended after a sort, not the table's net growth Review found that rows deleted from the sorted fragments hide appended rows one for one: the threshold compared the live row count against the count recorded at sort time, so 800 deletions and 300 appends read as a table that shrank, and a table where deletions keep pace with appends stays "sorted" with an unsorted tail forever. The fragments say it directly. The marker already records how many fragments the sort wrote, so the ones after that prefix are exactly what arrived since, and the manifest carries each fragment's live row count — physical rows less its deletions. Counting those is the arithmetic the threshold was always meant to do, and it needs no row count from the marker at all. A fragment whose length the manifest does not record cannot be counted, and a table that cannot be judged is one to sort rather than one to leave alone forever, so an uncountable appended fragment reads as stale. Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y	2026-09-02 21:25:50 -07:00