mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-14 18:18:35 +02:00
516e251f9e
* rust volume: move the crate to edition 2024 Edition 2024 turns three things in this crate into hard errors, and changes drop order in a further 34 places without changing compilation. The compiler errors are fixed here; the silent changes were audited against `RUSTFLAGS='-W rust-2024-compatibility' cargo check --all-targets` output captured before the flip, since edition 2024 stops reporting them. `std::env::set_var`/`remove_var` are unsafe as of 2024 because they race with concurrent readers. All six call sites are safe by construction rather than by assertion, and the SAFETY comments say why: the build script runs single-threaded before anything else in the process, and every test reaching the `config.rs` helpers holds `process_state_lock()` for the duration. The two `ref` bindings in handlers.rs sit in patterns that already borrow implicitly, so removing the modifier leaves both bindings at `&String`. On the 34 drop-order sites: no lock guard's scope is extended anywhere, and `volume.rs` has none. Most are moved-from `Option`/`Result` husks — `if let Some(v) = map.remove(&k)`, `while let Some(m) = stream.next().await` — where the value is moved into the binding and the temporary has nothing left to drop; where closing order actually matters these paths already call `v.close()`, `ec_vol.destroy()` or `drop(writer)` explicitly. Two sites get strictly better ordering: the metrics read guard in `run_metrics_push_loop` shrinks to the end of its initializer block (it never crossed an `.await` either way), and an EC test now closes the volume's descriptors before the `TempDir` removes the directory. No `rust-version` is declared. Edition 2024 needs rustc 1.85, but that is not the binding constraint — the dependency tree already requires 1.91.1 through the `aws-sdk-s3`/`aws-smithy-*` family, so `cargo +1.85 check` fails on the deps regardless. CI builds on `dtolnay/rust-toolchain@stable`. `vendor/reed-solomon-erasure` is a separate package and keeps edition 2021. Cargo.lock is unchanged despite edition 2024 implying resolver 3. Verified: `cargo test` 551 passed / 0 failed, `cargo test --no-default-features` 550 passed / 0 failed (the two feature sets produce an identical migration site list), `cargo build --release` clean. No automated test covers shutdown ordering, so the channel and runtime sites in `main.rs`, `write_queue.rs` and `grpc_server.rs` were read individually. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC * rust volume: address edition-2024 review feedback Three fixes from review of the edition bump. Serialize the two environment-reading tests. The SAFETY comments on the `env::set_var`/`remove_var` helpers claim every test touching the environment holds `process_state_lock()`, but `test_resolve_config_defaults_dir_to_platform_temp_dir` and `test_resolve_config_index_accepts_redb_and_leveldb_aliases` called `resolve_config` — which reads HOME/USERPROFILE, SEAWEED_WRITE_QUEUE and the WEED_* set — without taking it. `set_var` is unsafe precisely because a concurrent *reader* is UB, not only a concurrent writer, so the comment was overclaiming. An audit of the module found exactly these two; every other environment-touching test already held the lock. The race predates edition 2024, which only made the requirement explicit. Declare `rust-version = "1.91.1"`. The edition needs 1.85, but that was never the binding constraint: `cargo +1.90 check --all-targets` fails on the `aws-sdk-s3`/`aws-smithy-*` family, and 1.91.1 checks clean. Declaring the verified floor turns a wall of per-dependency errors into one clear message. Cargo.lock is unchanged despite this making the resolver MSRV-aware. Update the README, which advertised "Rust 1.75+ (2021 edition)". 1.75 was already stale before this branch — the tree has needed 1.91 for a while. Verified: `cargo test` 551 passed / 0 failed, `cargo test --no-default-features` 550 passed / 0 failed, `cargo build --release` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC * rust volume: state the exact MSRV patch release in the README The README said "Rust 1.91+", which reads as 1.91.0 and is wrong by one patch release: `cargo +1.91.0 check --all-targets` fails on the aws-sdk-s3 family, `cargo +1.91.1` passes. Say 1.91.1+, matching `rust-version` in Cargo.toml, and call out that the patch component is load-bearing so nobody installs 1.91.0 and hits the same wall. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC * rust worker: move the workspace to edition 2024 Moves the seaweed-worker workspace (core, lance, sort) from edition 2021 to 2024, the same migration seaweed-volume just got in this branch. Edition 2024 turns exactly one thing in this workspace into a hard error. The baseline came from RUSTFLAGS='-W rust-2024-compatibility' cargo check --all-targets, run before the flip; unlike seaweed-volume's 34 silent + 8 hard sites, the worker reports only the one hard site and no tail_expr_drop_order or if_let_rescope sites at all. The worker is a much smaller crate and none of its expressions hold a guard or temporary whose drop order the edition changes, so there is nothing to audit on the silent side. Fixed (1 site): std::env::set_var is unsafe as of 2024 because it races with concurrent readers. The single call is in crates/core/build.rs, which sets PROTOC from protoc_bin_vendored the way seaweed-volume's build script does. A build script's main runs single-threaded before anything else in the process, so no other thread can be reading the environment concurrently; the SAFETY comment says so. There are no config.rs-style test helpers here -- the worker's tests do not mutate the environment -- so unlike the volume crate there are no process_state_lock() callers to audit. No redundant ref bindings to clean up: a grep for ref across the three crates finds none. MSRV: rust-version = "1.94.1", verified rather than inferred. Edition 2024 only needs 1.85, but the dependency tree needs more: lance's aws feature pulls in a newer cut of the same aws-sdk-*/aws-smithy-* family that sets seaweed-volume's 1.91.1 floor, and that newer cut requires 1.94.1. cargo +1.94.0 check --all-targets fails on that family; cargo +1.94.1 check --all-targets is clean. The worker's floor is therefore higher than the volume's, and moves with lance and the AWS SDK rather than with the edition. CI builds on dtolnay/rust-toolchain@stable, so nothing changes there. The edition is set once in [workspace.package] and inherited by each member via edition.workspace = true; rust-version is added the same way. The workspace keeps its explicit resolver = "2" -- edition 2024 would default to resolver 3, but the pin is deliberate and Cargo.lock is unchanged by this commit either way. The README gains a "Requires Rust 1.94.1+ (2024 edition)" line in its Building section, matching the one seaweed-volume's README now carries, and calling out that the patch release is load-bearing (1.94.0 does not build) so nobody installs 1.94.0 and hits the same wall. Verification: * cargo check --all-targets -- clean, zero warnings (default toolchain 1.97) * cargo +1.94.1 check --all-targets -- clean * cargo +1.94.0 check --all-targets -- fails on the AWS SDK, as claimed * cargo test --all-targets -- 40 passed, 0 failed (core 13, sort 11, lance lib 3, lance bin 2, compaction 6, lifecycle 1, sort integration 4) * Cargo.lock unchanged Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>
153 lines
4.0 KiB
TOML
153 lines
4.0 KiB
TOML
[package]
|
|
name = "weed-volume"
|
|
version = "0.1.0"
|
|
edition = "2024"
|
|
# The edition needs 1.85; the dependency tree needs more. Verified with
|
|
# `cargo +1.91.1 check --all-targets` (1.90 fails on the AWS SDK).
|
|
rust-version = "1.91.1"
|
|
description = "SeaweedFS Volume Server — Rust implementation"
|
|
|
|
[lib]
|
|
name = "seaweed_volume"
|
|
|
|
[[bin]]
|
|
name = "weed-volume"
|
|
path = "src/main.rs"
|
|
|
|
[features]
|
|
# Default: 5-byte offsets (8TB max volume size), matching production Go builds (-tags 5BytesOffset).
|
|
# Disable with --no-default-features for 4-byte offsets (32GB max volume size).
|
|
default = ["5bytes"]
|
|
5bytes = []
|
|
# Unstable redb cursor bulk-load for full_rebuild. Off in production.
|
|
# Pulls redb's experimental_cursor (and therefore experimental-api-5).
|
|
redb-experimental-cursor = ["redb/experimental_cursor"]
|
|
|
|
[dependencies]
|
|
# Async runtime
|
|
tokio = { version = "1", features = ["full"] }
|
|
tokio-stream = { version = "0.1", features = ["net"] }
|
|
tokio-io-timeout = "1"
|
|
|
|
# gRPC + protobuf
|
|
tonic = { version = "0.12", features = ["tls"] }
|
|
tonic-reflection = "0.12"
|
|
prost = "0.13"
|
|
prost-types = "0.13"
|
|
|
|
# HTTP server
|
|
axum = { version = "0.7", features = ["multipart"] }
|
|
http-body = "1"
|
|
hyper = { version = "1", features = ["full"] }
|
|
hyper-util = { version = "0.1", features = ["tokio", "service", "server-auto", "http1", "http2"] }
|
|
tower = "0.4"
|
|
tower-http = { version = "0.5", features = ["cors", "trace"] }
|
|
|
|
# CLI
|
|
clap = { version = "4", features = ["derive"] }
|
|
|
|
# Metrics
|
|
prometheus = { version = "0.13", default-features = false, features = ["process"] }
|
|
lazy_static = "1"
|
|
|
|
# JWT
|
|
jsonwebtoken = { version = "10", features = ["rust_crypto"] }
|
|
|
|
# TLS
|
|
rustls = "0.23"
|
|
tokio-rustls = "0.26"
|
|
rustls-pemfile = "2"
|
|
|
|
# LevelDB (via RocksDB for better Rust support)
|
|
# Using rusty-leveldb for pure Rust LevelDB
|
|
rusty-leveldb = "3"
|
|
|
|
# Disk-backed needle map (alternative to in-memory HashMap)
|
|
redb = "4"
|
|
|
|
# Reed-Solomon erasure coding
|
|
reed-solomon-erasure = "6"
|
|
|
|
# Logging
|
|
tracing = "0.1"
|
|
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
|
# Config
|
|
toml = "0.8"
|
|
serde = { version = "1", features = ["derive"] }
|
|
serde_json = "1"
|
|
serde_urlencoded = "0.7"
|
|
|
|
# CRC32 — using Castagnoli polynomial (CRC32-C), matching Go's crc32.Castagnoli
|
|
crc32c = "0.6"
|
|
crc32fast = "1"
|
|
|
|
# xxhash64 - must match Go's cespare/xxhash for the heartbeat volume digest
|
|
xxhash-rust = { version = "0.8", features = ["xxh64"] }
|
|
|
|
# Memory-mapped files
|
|
memmap2 = "0.9"
|
|
|
|
# UUID
|
|
uuid = { version = "1", features = ["v4"] }
|
|
|
|
# HTTP client (for proxying, remote fetch)
|
|
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "stream", "multipart", "json"] }
|
|
|
|
# Content hashing
|
|
md-5 = "0.10"
|
|
base64 = "0.22"
|
|
|
|
# Compression
|
|
flate2 = "1"
|
|
|
|
# Image processing
|
|
image = { version = "0.25", default-features = false, features = ["png", "jpeg", "gif", "webp"] }
|
|
kamadak-exif = "0.5"
|
|
|
|
# Multipart form-data parsing
|
|
multer = "3"
|
|
|
|
# MIME type guessing from file extensions
|
|
mime_guess = "2"
|
|
|
|
# Misc
|
|
bytes = "1"
|
|
rand = "0.10"
|
|
chrono = "0.4"
|
|
hex = "0.4"
|
|
parking_lot = "0.12"
|
|
dashmap = "6"
|
|
thiserror = "1"
|
|
anyhow = "1"
|
|
async-trait = "0.1"
|
|
futures = "0.3"
|
|
async-stream = "0.3"
|
|
x509-parser = "0.16"
|
|
|
|
# Disk space checking
|
|
sysinfo = "0.31"
|
|
libc = "0.2"
|
|
|
|
# AWS S3 SDK (for remote storage backends)
|
|
aws-config = { version = "1", features = ["behavior-version-latest"] }
|
|
aws-sdk-s3 = { version = "1.125.0", default-features = false, features = ["sigv4a", "http-1x", "default-https-client", "rt-tokio"] }
|
|
aws-credential-types = "1"
|
|
aws-types = "1"
|
|
|
|
# pprof is Unix-only (requires libc/nix APIs not available on Windows)
|
|
[target.'cfg(unix)'.dependencies]
|
|
pprof = { version = "0.15", features = ["prost-codec"] }
|
|
|
|
[dev-dependencies]
|
|
tempfile = "3"
|
|
|
|
[build-dependencies]
|
|
tonic-build = "0.12"
|
|
# Ships protoc with the build so neither CI nor a developer needs a system
|
|
# install, and so the version is pinned rather than whatever the platform's
|
|
# package manager happens to carry.
|
|
protoc-bin-vendored = "3"
|
|
|
|
[patch.crates-io]
|
|
reed-solomon-erasure = { path = "vendor/reed-solomon-erasure" }
|