2 Commits

Include renames

Author	SHA1	Message	Date
3 people	166af06a2b	rust: cargo fmt both crates, with a commented-out fmt --check CI step (#11329) ... * rust: migrate seaweed-volume and seaweed-worker to tonic 0.14 / prost 0.14 tonic 0.14 boxes the contents of tonic::Status, which is what made every RPC path trip clippy's result_large_err; the allow for that lint goes in the next commit. The prost codec moved out of tonic into tonic-prost and tonic-prost-build, so both build scripts now call tonic_prost_build::configure() and both crates depend on tonic-prost for the generated code. The `tls` feature was split into a per-backend feature; `tls-aws-lc` is the same backend both crates already install through rustls::crypto::aws_lc_rs. tonic 0.14 depends on axum 0.8 and tower 0.5, which would have left a second axum and a second tower in each tree next to the 0.7 / 0.4 the crates named themselves. Bumping them keeps one copy of each: axum 0.8 only changes the path-parameter syntax for the routes here (`/:vid` -> `/{vid}`, `/*path` -> `/{*path}`), tower 0.5 needs the `util` feature named explicitly for ServiceExt::oneshot (it used to arrive through tonic's feature unification), and tower-http 0.6 is the matching release. Lock files move only through cargo's own resolution for the new versions; no other dependency was refreshed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust: drop the result_large_err allow now that tonic::Status is boxed tonic 0.14 stores Status behind a Box, so Result<_, Status> is no longer a large-Err type and clippy has nothing to say about it. Both crates pass `cargo clippy --all-targets -- -D warnings` without the allow (seaweed-volume in both feature sets), so the policy entry and its comment go. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: drop the unused headers argument of try_expand_chunk_manifest The parameter was already named `_headers`; nothing in the body reads it. With it gone the function is under clippy's argument threshold and the expect goes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: pass EC peer reads an EcInterval instead of ten arguments fetch_one_interval, read_remote_ec_shard_interval, do_read_remote_ec_shard_interval and recover_one_remote_ec_shard_interval all took the same (vid, needle_id, shard_id, shard_offset, size, expected_encode_ts_ns) tuple, and the two that reconstruct also took the location map with the data/parity counts. Those are now EcInterval (Copy) and EcShardMap (a borrow of the map plus the counts). The fan-out inside recovery builds its per-shard request with `EcInterval { shard_id: sid, ..iv }`, which is the one place the old argument list was easy to get wrong. Bodies destructure at the top, so the code below the signatures is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: give the EC encoder an EcEncodeLayout and an EncodeRun encode_dat_file took the Reed-Solomon shape and three block sizes as five loose integers; they are now one Copy struct, EcEncodeLayout, which is what Go calls ECContext. The per-row and per-batch helpers took the same six sinks and the offsets; they become methods on EncodeRun, which owns the borrows for one run, so each call names only the offset and block size that vary. The byte-level work is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: describe a .dat rebuild with DatRebuild instead of nine arguments write_dat_file_from_shards, its _with_dirs twin and the private write_dat_file were three layers over one nine-argument signature. One public function now takes a DatRebuild, whose shard_dirs is None when every shard sits beside the .dat and Some(dirs) for the cross-disk reconciled layout. The field docs carry what the function doc used to say about the encode-time size and the block layout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: split copy_file_from_source's fifteen arguments into two structs CopyFileSpec is the per-file request (what to ask the source for, where it lands, whether its bytes count as progress); CopyProgress is the sender, throttler and report state that all three files of one VolumeCopy share, held by &mut across the calls. The three production call sites now read as the .dat/.idx/.vif literals they are, instead of positional trues and falses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: create volumes from a VolumeSpec Volume::new, DiskLocation::create_volume and Store::add_volume each took the same five-value tail of Go's NewVolume argument list: collection, replica placement, TTL, preallocation and needle version. That tail is now VolumeSpec, a Copy struct whose Default is what almost every test wanted anyway (empty collection, no replication, no TTL, no preallocation, current version), so most of the 104 call sites shrink to `&VolumeSpec::default()` or name the one field they set. The id, directories, index kind and disk type stay positional because they differ at every site. Two imports that only test modules use moved into those modules, and DiskLocation no longer imports ReplicaPlacement. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-worker: run cargo fmt Layout only; no token in the workspace changes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: run cargo fmt Layout only; no token in the crate changes. Every earlier Rust PR here formatted only the blocks it touched so as not to drown its diff in this one, and this commit is that debt paid in a single place. rustfmt needed two passes to settle one block in handlers.rs; the committed form is the fixed point, so `cargo fmt --check` is clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * ci: add a commented-out cargo fmt --check step to both Rust workflows Same shape as the commented clippy step from #11312: the check is written out so that making formatting a gate is a one-line uncomment, and whether to do that stays a maintainer call. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2026-09-15 09:29:22 -07:00
Chris Lu	0dfaa103d0	test: take a table through its whole life, for Iceberg and Lance (#10862) ... * lance worker: share the integration tests' scaffolding The recorder that keeps what a handler sent, the config builder and the storage-option fallback all lived inside compaction.rs, so a second test binary would have had to copy them. They move to tests/common. The fallback now reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_ENDPOINT_URL from the environment, defaulting to what it used before. A harness can then point these tests at a gateway that checks what it is given rather than one that accepts anything. * lance worker: maintain one named table, for a harness to drive Compacts and cleans up whatever WEED_LANCE_TABLE names, through the handlers' own detect-then-execute path: a proposal the worker would not have made is not one worth running. The existing tests seed the tables they check. This one deliberately does not, so a harness that has already written a table and knows what is in it can have the real handlers maintain it and then read it back. * test: take a table through its whole life, for Iceberg and Lance Created in the catalog, filled by a real client, maintained by the worker, read again, dropped. The step nothing was checking is the read after maintenance: compaction once rewrote every dictionary-encoded column onto a single value and shipped, because the maintenance tests were thorough about sequence numbers, manifest entries and metadata versions and none of them opened the parquet file the worker had just written. So the assertion is a tally - row count, the cardinality of each dictionary-encoded column, and an md5 over whole rows - taken before maintenance and again after, required to be equal. The cardinalities name the failure that happened; the digest catches a rewrite that keeps every column's cardinality and hands the values to the wrong rows. A compaction that merged nothing fails rather than passes, or the read afterwards is checking a file the worker never wrote. The Iceberg half runs two clients. DuckDB is the one the corruption was reported against and the only one here that writes the deprecated PLAIN_DICTIONARY encoding, which parquet-go normalizes away on write, so a Go writer cannot produce it. PyIceberg writes the modern spelling. Pinning parquet-go back to v0.30.1 fails the DuckDB half and passes the PyIceberg one, which is why both are here. Lance maintenance lives in the Rust worker, so it runs there where cargo is installed and through the two lance calls those handlers wrap where it is not. WEED_LANCE_MAINTENANCE picks one instead of letting the test guess. * ci: run the table lifecycle tests CI maintains the Lance table through the lance library rather than the worker: a cold build of the lance crate costs more than the glue it would be checking, and the worker's own tests cover its handlers. The suite drives the Iceberg maintenance worker, so a change to it now triggers this workflow too. * test: let the lifecycle harness fail instead of skipping Setup failures all exited zero, so a cluster that would not come up, or a port allocation that lost, reported a green run for code nothing had executed. That is the failure mode this whole directory exists to close, and it was in the harness itself. Only a checkout without a weed binary skips now, and it runs the tests so each one says so rather than the package quietly passing. Everything else fails. The filer existence probe gets a deadline while I am here: it ran without one, so an unresponsive filer would hang the suite past every timeout the clients have. * test: make the lifecycle checks check what they claim to Three of them could pass without having looked. The DuckDB skip matched "syntax error", "not implemented" and "Failed to load" anywhere in the output, in any phase. A parse error in the SQL this test generates, or a refusal from our own catalog, would have taken the only coverage of the PLAIN_DICTIONARY encoding out of CI and left it green. It now matches the extension failing to install, and only in the phase that installs it. Everything past LOAD is ours and fails. The digests covered id, category and value. Compaction rewrites the whole row, so a defect confined to ts, or to a Lance vector, changed nothing either side of maintenance. Every persisted column goes in now, ts as microseconds so no timezone sits between the two runs. The Lance drop check caught every exception as proof the dataset was gone. pylance turns credential and transport failures into the same ValueError, so it only accepts the message that means not found. * docs: say up front which maintenance path the Lance half takes The opening summary said the worker maintains both tables. It maintains the Iceberg one always and the Lance one only where cargo is installed, which is not what CI does.	2026-08-21 15:16:11 -07:00