2 Commits

Include renames

Author	SHA1	Message	Date
3 people	516e251f9e	rust volume: move the crate to edition 2024 (#11244) ... * rust volume: move the crate to edition 2024 Edition 2024 turns three things in this crate into hard errors, and changes drop order in a further 34 places without changing compilation. The compiler errors are fixed here; the silent changes were audited against `RUSTFLAGS='-W rust-2024-compatibility' cargo check --all-targets` output captured before the flip, since edition 2024 stops reporting them. `std::env::set_var`/`remove_var` are unsafe as of 2024 because they race with concurrent readers. All six call sites are safe by construction rather than by assertion, and the SAFETY comments say why: the build script runs single-threaded before anything else in the process, and every test reaching the `config.rs` helpers holds `process_state_lock()` for the duration. The two `ref` bindings in handlers.rs sit in patterns that already borrow implicitly, so removing the modifier leaves both bindings at `&String`. On the 34 drop-order sites: no lock guard's scope is extended anywhere, and `volume.rs` has none. Most are moved-from `Option`/`Result` husks — `if let Some(v) = map.remove(&k)`, `while let Some(m) = stream.next().await` — where the value is moved into the binding and the temporary has nothing left to drop; where closing order actually matters these paths already call `v.close()`, `ec_vol.destroy()` or `drop(writer)` explicitly. Two sites get strictly better ordering: the metrics read guard in `run_metrics_push_loop` shrinks to the end of its initializer block (it never crossed an `.await` either way), and an EC test now closes the volume's descriptors before the `TempDir` removes the directory. No `rust-version` is declared. Edition 2024 needs rustc 1.85, but that is not the binding constraint — the dependency tree already requires 1.91.1 through the `aws-sdk-s3`/`aws-smithy-*` family, so `cargo +1.85 check` fails on the deps regardless. CI builds on `dtolnay/rust-toolchain@stable`. `vendor/reed-solomon-erasure` is a separate package and keeps edition 2021. Cargo.lock is unchanged despite edition 2024 implying resolver 3. Verified: `cargo test` 551 passed / 0 failed, `cargo test --no-default-features` 550 passed / 0 failed (the two feature sets produce an identical migration site list), `cargo build --release` clean. No automated test covers shutdown ordering, so the channel and runtime sites in `main.rs`, `write_queue.rs` and `grpc_server.rs` were read individually. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC * rust volume: address edition-2024 review feedback Three fixes from review of the edition bump. Serialize the two environment-reading tests. The SAFETY comments on the `env::set_var`/`remove_var` helpers claim every test touching the environment holds `process_state_lock()`, but `test_resolve_config_defaults_dir_to_platform_temp_dir` and `test_resolve_config_index_accepts_redb_and_leveldb_aliases` called `resolve_config` — which reads HOME/USERPROFILE, SEAWEED_WRITE_QUEUE and the WEED_* set — without taking it. `set_var` is unsafe precisely because a concurrent *reader* is UB, not only a concurrent writer, so the comment was overclaiming. An audit of the module found exactly these two; every other environment-touching test already held the lock. The race predates edition 2024, which only made the requirement explicit. Declare `rust-version = "1.91.1"`. The edition needs 1.85, but that was never the binding constraint: `cargo +1.90 check --all-targets` fails on the `aws-sdk-s3`/`aws-smithy-*` family, and 1.91.1 checks clean. Declaring the verified floor turns a wall of per-dependency errors into one clear message. Cargo.lock is unchanged despite this making the resolver MSRV-aware. Update the README, which advertised "Rust 1.75+ (2021 edition)". 1.75 was already stale before this branch — the tree has needed 1.91 for a while. Verified: `cargo test` 551 passed / 0 failed, `cargo test --no-default-features` 550 passed / 0 failed, `cargo build --release` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC * rust volume: state the exact MSRV patch release in the README The README said "Rust 1.91+", which reads as 1.91.0 and is wrong by one patch release: `cargo +1.91.0 check --all-targets` fails on the aws-sdk-s3 family, `cargo +1.91.1` passes. Say 1.91.1+, matching `rust-version` in Cargo.toml, and call out that the patch component is load-bearing so nobody installs 1.91.0 and hits the same wall. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC * rust worker: move the workspace to edition 2024 Moves the seaweed-worker workspace (core, lance, sort) from edition 2021 to 2024, the same migration seaweed-volume just got in this branch. Edition 2024 turns exactly one thing in this workspace into a hard error. The baseline came from RUSTFLAGS='-W rust-2024-compatibility' cargo check --all-targets, run before the flip; unlike seaweed-volume's 34 silent + 8 hard sites, the worker reports only the one hard site and no tail_expr_drop_order or if_let_rescope sites at all. The worker is a much smaller crate and none of its expressions hold a guard or temporary whose drop order the edition changes, so there is nothing to audit on the silent side. Fixed (1 site): std::env::set_var is unsafe as of 2024 because it races with concurrent readers. The single call is in crates/core/build.rs, which sets PROTOC from protoc_bin_vendored the way seaweed-volume's build script does. A build script's main runs single-threaded before anything else in the process, so no other thread can be reading the environment concurrently; the SAFETY comment says so. There are no config.rs-style test helpers here -- the worker's tests do not mutate the environment -- so unlike the volume crate there are no process_state_lock() callers to audit. No redundant ref bindings to clean up: a grep for ref across the three crates finds none. MSRV: rust-version = "1.94.1", verified rather than inferred. Edition 2024 only needs 1.85, but the dependency tree needs more: lance's aws feature pulls in a newer cut of the same aws-sdk-*/aws-smithy-* family that sets seaweed-volume's 1.91.1 floor, and that newer cut requires 1.94.1. cargo +1.94.0 check --all-targets fails on that family; cargo +1.94.1 check --all-targets is clean. The worker's floor is therefore higher than the volume's, and moves with lance and the AWS SDK rather than with the edition. CI builds on dtolnay/rust-toolchain@stable, so nothing changes there. The edition is set once in [workspace.package] and inherited by each member via edition.workspace = true; rust-version is added the same way. The workspace keeps its explicit resolver = "2" -- edition 2024 would default to resolver 3, but the pin is deliberate and Cargo.lock is unchanged by this commit either way. The README gains a "Requires Rust 1.94.1+ (2024 edition)" line in its Building section, matching the one seaweed-volume's README now carries, and calling out that the patch release is load-bearing (1.94.0 does not build) so nobody installs 1.94.0 and hits the same wall. Verification: * cargo check --all-targets -- clean, zero warnings (default toolchain 1.97) * cargo +1.94.1 check --all-targets -- clean * cargo +1.94.0 check --all-targets -- fails on the AWS SDK, as claimed * cargo test --all-targets -- 40 passed, 0 failed (core 13, sort 11, lance lib 3, lance bin 2, compaction 6, lifecycle 1, sort integration 4) * Cargo.lock unchanged Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-09-09 10:54:06 -07:00
Chris Lu	9e34426a56	lance: a maintenance job that sorts a table by its declared fields (#11113) ... * lance: a maintenance job that sorts a table by its declared fields Lance appends fragments in write order and has no notion of a sorted table, so nothing but a rewrite establishes one, and nothing but another rewrite restores it once rows have been appended. lance_sort reads the order from the dataset's own configuration, falls back to the worker's, and rewrites the table in it. The spec and the marker live in crates/sort rather than in the job, because weed/worker/tasks/iceberg sorts too: two jobs that disagreed about what "id desc nulls-first" means would be two features wearing one name. The sort spills. lance builds its DataFusion runtime with a FairSpillPool and a disk manager, but only when LanceExecutionOptions::use_spilling is set, and that struct derives Default over a plain bool — so Scanner::try_into_stream, which fills its options with ..Default::default(), is precisely the path that does not spill. The job builds the plan with create_plan and executes it with spilling on and the operator's memory budget. The marker rides in the same commit as the data: Operation::Overwrite is the one operation carrying config values alongside fragments, so a sorted table and the record of its sorting cannot disagree. It records the version the sort read, not the one it wrote, which is not knowable while the marker is being assembled. Detection treats anything committed after the sort's own commit as data the sort did not produce — row counts alone cannot see a rewrite that leaves the count where it was, and such a table would look sorted forever. Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y * lance: identify a sorted table by the files it wrote, not its version Review found three ways the version-based marker misjudges a table, and they share a cause: the version a sort produces is not knowable while the marker is being assembled, so the marker recorded the version it read and detection inferred the rest. A commit that rebases past a conflict lands on a different number, and the inference then reads a rewrite into an ordinary table — a full re-sort, and its indices, for nothing. Data file names do not have that problem. They are chosen before the commit, so the commit can carry them, and they do not change with the version it lands on. The marker now records how many files the sort wrote and a digest of their names, and detection asks whether the table still holds them: the same files means untouched, the same files followed by more means appended, anything else means the data was replaced. That also closes the hole the row threshold left. A replacement that grew the table by fewer rows than min_unsorted_rows read as sorted, however many rows had actually moved; the threshold now applies only where the sorted files are still in place, which is what it was for. A marker without a row count is stale rather than a zero to compare against, and deletes stop forcing a re-sort — they write a deletion file beside the data rather than rewriting it, and removing rows does not unsort the ones that remain. Sort fields are also compared exactly rather than case-folded. Arrow schemas are case-sensitive, so `id` and `ID` are two columns, and folding them together rejected a valid order. Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y * lance: count the rows appended after a sort, not the table's net growth Review found that rows deleted from the sorted fragments hide appended rows one for one: the threshold compared the live row count against the count recorded at sort time, so 800 deletions and 300 appends read as a table that shrank, and a table where deletions keep pace with appends stays "sorted" with an unsorted tail forever. The fragments say it directly. The marker already records how many fragments the sort wrote, so the ones after that prefix are exactly what arrived since, and the manifest carries each fragment's live row count — physical rows less its deletions. Counting those is the arithmetic the threshold was always meant to do, and it needs no row count from the marker at all. A fragment whose length the manifest does not record cannot be counted, and a table that cannot be judged is one to sort rather than one to leave alone forever, so an uncountable appended fragment reads as stale. Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y	2026-09-02 21:25:50 -07:00