Files
seaweedfs/seaweed-volume/src/main.rs
T
Chris LuandClaude Opus 5.5 862913d199 volume, ec_balance: clean up failed EC shard copies; stop re-proposing empty-index volumes (#12718)
* volume: remove files a failed VolumeEcShardsCopy wrote

The shard files were copied before the .ecx, so a source holding a
0-byte index failed the request only after its shards had landed, and
nothing removed them: every retried move stranded another unmounted
shard on a new disk.

Copy the .ecx first, staged beside its target, so a bad index fails
before any shard lands; rename it into place only after the shards and
sidecars have copied, so a failed copy never truncates or replaces an
.ecx the destination already has. On any error, remove exactly the
files this call created; files already on the disk (an existing
holder's shards, index or journal) are left alone. Go and Rust volume
servers behave the same.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* ec_balance: skip EC volumes whose every holder reports an empty .ecx

A shard copy refuses a 0-byte .ecx, so detection proposed the same
failing move of such a volume every cycle. Leave volumes with no
holder reporting a needle out of planning; their shards still count
toward disk occupancy.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* volume: serialize VolumeEcShardsCopy per volume

Overlapping copies of one vid shared the staged .ecx, and a failed copy's
cleanup removed shards that a concurrent copy had already committed and
mounted. Hold a per-vid EC lifecycle lock across the whole copy and its
cleanup, named as in the enterprise build (Store.EcLifecycleLock /
ec_lifecycle_lock).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: overwrite an existing .ecx in place when committing a copied index

A mounted EcVolume reads its index through an open handle, so renaming
the staged copy over it left the reader on the replaced inode: tombstones
the source had folded into its .ecx never reached the mounted holder.
An index already on the disk is now overwritten in place, as before the
staging change; a fresh destination still gets the rename. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stage replacements of shards a VolumeEcShardsCopy finds on disk

The shard transfer opened an existing destination shard with O_TRUNC and
removed it on a stream or write error, so a failed retry deleted a shard
this disk already held. A shard already present is now copied to
<shard>.copying and renamed into place only after every other file has
copied; a mounted copy is reopened so it serves the new file. Fresh
shards still write in place and are removed on failure. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stage the .vif and .ecsum a VolumeEcShardsCopy brings until commit

The sidecars landed as soon as they transferred, ahead of the deferred
.ecx commit, so a failure after them left a new .vif beside the index
the failed copy kept. They are now staged as .copying and renamed into
place with the shards and index once every transfer has succeeded; a
source without one still stages nothing. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: run a Rust VolumeEcShardsCopy to completion when its caller goes away

tonic drops a cancelled handler future at its current await, which
released the EC lifecycle lock while file operations the copy had started
(a cleanup remove_file on the blocking pool, a create or rename) could
still land; a retry taking the lock could then lose its files to them.
The copy now runs in a spawned task that owns the lock, as Go's handler
runs to completion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fail the sidecar copy tests with an unreadable source .ecsum

A directory at the source path reads as "not found" on volume servers
that skip non-regular EC files, which turned the failure into a no-op.
A mode-000 file fails the transfer on every implementation; root and
Windows, which can read it, skip the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: copy each shard a VolumeEcShardsCopy lists once

A shard listed twice and already held here was staged twice; the first
commit rename consumed the staged file, so the second failed the request
after the shard was replaced, and every retry failed the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: wait out an in-flight VolumeEcShardsCopy before mounting its shards

A shard the copy had already landed could be mounted while the copy was
still running; the copy's failure cleanup then removed the file but left
the shard registered and announced. MountEcShards takes the per-volume
lifecycle lock, as the enterprise build already does, and the Rust mount
handler takes the same lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fail a VolumeEcShardsCopy whose replaced mounted shard cannot be reopened

The copy reported success while the mounted shard kept serving the
replaced file. The error lets the caller retry the copy instead of
treating the shard as delivered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check which transfer fails in the EC copy cleanup tests

An earlier failure, before any shard or sidecar landed, would leave the
destination unchanged and pass every assertion without exercising the
cleanup. The sidecar test also skips on Windows, where a mode-000 file
stays readable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fail the sidecar transfer in the EC copy test by injection

A mode-000 file is still readable to root and on Windows, so the test
skipped there. The source now returns an error for the .ecsum request
only, and the test checks that it served .vif before that failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the EC lifecycle lock for a blanket full teardown

A teardown acknowledged while a detached copy of the same volume was
still waiting on its source let that copy recreate the index and shards
afterwards. Go and Rust now wait for the copy, as the enterprise build
does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep the request id on a Rust EC shard copy

The copy now runs in a spawned task, outside the gRPC layer's request id
scope, so its CopyFile calls to the source went out with a new id. Carry
the caller's id into the task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: unmount a replaced EC shard whose reopen fails

The failed copy left the mount reading the replaced file's inode, so the
destination kept serving the old shard. Unmount it, which also reports
the removal to the master, so a retry copies and mounts it again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: make the Rust EC copy tests fail for the right reason

The sidecar test now serves a directory as the source .ecsum, which root
cannot read either, so it no longer returns early there. The cancel test
holds the store until the handler's join reports the cancellation, so a
copy that finished before the abort can no longer pass it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stage every shard a VolumeEcShardsCopy transfers

A shard missing here was written in place with a cleanup guard armed
before its transfer started, so a failed copy removed a shard that
ReceiveFile wrote meanwhile. Every shard is now staged, and only one
that did not exist when it is moved into place is removed on failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the EC lifecycle lock for VolumeEcShardsRebuild

A rebuild folds the deletion journal into .ecx. Run during a copy of the
same volume, the copy's index then replaced the folded tombstones. Go
and Rust take the lock the enterprise build already takes here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check the staged .vif bytes in the EC sidecar failure test

A source without the optional .vif also returns success, so the served
extension list could not prove the transfer. The test now reads the
staged .vif when the .ecsum request fails and compares it with the
source's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: unmount a failed reopen's shard on the copy's disk only

UnmountEcShards walks every disk, so a reopen failure on the destination
also unmounted a healthy copy of the same shard on a sibling disk. The
copy now unloads the shard from its own disk and reports that disk's
removal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the EC lifecycle lock for an ordinary EC shard delete

Shards a copy has staged are invisible to the delete's last-shard check,
so deleting the only canonical shard removed the index the copy was
about to commit its shards against. Go and Rust deletes now wait for an
in-flight copy of the same volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: leave a mounted shard's copied file in place on copy failure

A shard mounted on the copy's disk whose file was missing got the copied
file reopened into its mount, then that file was unlinked by a later
failure's cleanup while the mount kept serving it. Only an unmounted
shard's new file is removed now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: skip repeated shard ids with a seen set

Scanning the request prefix for each id made a request with many
repeats quadratic while the copy held the volume's lifecycle lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: make the Go EC copy and lock tests observe their effects

The fake source counts requests per extension, so the duplicate-id test
proves .ec05 is fetched once, and returns fixture errors instead of
failing the test from its handler goroutine. The staged .vif reaches
the test through a channel. The mount and teardown lock tests seed real
files and check that nothing happened while the lock was held and that
the operation took effect after it was released.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: poll the handler directly in the Rust EC lifecycle lock tests

A spawned handler that had not run when the 200 ms timer fired passed
the blocking check without ever reaching the lock. The tests now poll
the pinned handler future themselves, which guarantees it is waiting on
the lock, and cover the ordinary shard delete as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: observe the lifecycle-lock waiter in EC copy lock tests

The mount, teardown, shard-delete and rebuild tests slept 200 ms while
holding the lock, so a handler scheduled only after the unlock passed
without ever taking it. Wait until the handler's goroutine is blocked on
the mutex instead; each test now fails within its timeout if the handler
skips the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check the shard is staged before the failure-path write in a copy test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: recover a missing EC index under the volume's lifecycle lock

Missing-index recovery fetched the index and then ran a store-wide mount
of every orphan shard, without the per-volume lifecycle lock. It could
mount shards an in-flight VolumeEcShardsCopy of another volume had just
renamed in, and that copy's failure cleanup then unlinked a mounted shard.
Each recovered volume now holds its lifecycle lock across a re-check, the
fetch and a mount restricted to that volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): recover a missing EC index under the volume's lifecycle lock

Mirror of the Go change: recover_missing_ec_indexes takes the volume's
ec_lifecycle_lock, re-checks that the index is still missing, fetches it,
and mounts only that volume's shards through mount_recovered_ec_shards_only,
so an in-flight VolumeEcShardsCopy of another volume never has its
renamed-in shards mounted from under it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: mount a volume whose index a copy installed during recovery

Recovery re-checks under the lifecycle lock that the index is still
missing. When a copy installed it in the meantime, the re-check skipped
the mount too, and the volume's other shards stayed unmounted because
later scans no longer list it. Skip only the fetch and still mount the
volume's orphan shards under the lock, in Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fetch a recovered EC index into the disk the locked re-check found

The re-check under the lifecycle lock only kept a yes/no answer, so the
fetch still wrote to the destination from the first scan, which may be a
disk that has since gone away. Fetch into the record the re-check returned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: mount a copy-installed EC index even when no peer is known

Recovery skipped the locked step when the master was unreachable, the
lookup failed or no peer was listed. A copy that installed the index after
discovery then left the volume's other shards unmounted, and later scans
no longer list it. Always run the locked re-check, and give up for lack of
peers only when the index is still missing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: grow the stack buffer before scanning for a lock waiter

A full buffer can cut off the handler goroutine's stack, so the test would
report a handler that is waiting as one that never waited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): fetch a recovered EC index into the disk the locked re-check found

Mirror of the Go change: fetch into the record the re-check under the
lifecycle lock returned instead of the destination from the first scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): mount a copy-installed EC index even when no peer is known

Mirror of the Go change: a failed lookup or an empty peer list no longer
skips the locked re-check, so a volume whose index a copy installed after
discovery still gets its orphan shards mounted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): run missing-index recovery to completion when the mount RPC is cancelled

Dropping the mount handler mid-recovery released the volume's lifecycle
lock while the .ecj merge it had started on a blocking thread was still
landing, so a teardown could remove the volume before that late write.
Run recovery on its own task, as the copy handler does, so the lock is
held until it finishes; Go's handler already runs it to completion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: skip other volumes before stat in a single-volume EC scan

Recovery re-checks and mounts each volume with its own scans, so a
request with many orphans repeated whole-store discovery once per volume.
The index-owner and orphan-shard scans now take the same volume filter
and drop other volumes' files by name, before the per-file stat, and the
locked re-check scans only its own volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): skip other volumes before stat in a single-volume EC scan

Mirror of the Go change: the index-owner and orphan-shard scans take the
volume filter and drop other volumes' files by name before the per-file
stat, so recovery's per-volume re-check and mount no longer repeat
whole-store discovery under the store lock for every orphan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): announce shards recovered after the mount RPC was cancelled

Recovery now outlives a cancelled mount RPC, but the heartbeat nudge
stayed in the handler, so shards it mounted after the caller left waited
for the next full EC heartbeat. Nudge it from the recovery task, as Go's
loads announce each recovered shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: match the collection as well as the volume id in the recovery re-check

Discovery keys orphans by (collection, vid), but the locked re-check
matched the vid only. Two collections with orphans of the same vid could
both resolve to the first record, so the other's index was never fetched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): match the collection as well as the volume id in the recovery re-check

Mirror of the Go change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:48:21 -08:00

1352 lines
52 KiB
Rust

use std::sync::{Arc, RwLock};
use tracing::{error, info, warn};
use seaweed_volume::config::{self, VolumeServerConfig};
use seaweed_volume::metrics;
use seaweed_volume::pb::volume_server_pb::volume_server_server::VolumeServerServer;
use seaweed_volume::security::tls::{
GrpcClientAuthPolicy, TlsPolicy, build_rustls_server_config,
build_rustls_server_config_with_grpc_client_auth, install_default_crypto_provider,
};
use seaweed_volume::security::{Guard, SigningKey};
#[cfg(unix)]
use seaweed_volume::server::debug::build_debug_router;
use seaweed_volume::server::grpc_client::{
GRPC_INITIAL_WINDOW_SIZE, GRPC_KEEPALIVE_INTERVAL, GRPC_KEEPALIVE_TIMEOUT,
GRPC_MAX_MESSAGE_SIZE, load_outgoing_grpc_tls,
};
use seaweed_volume::server::grpc_server::VolumeGrpcService;
#[cfg(unix)]
use seaweed_volume::server::profiling::CpuProfileSession;
use seaweed_volume::server::request_id::GrpcRequestIdLayer;
use seaweed_volume::server::volume_server::{
RuntimeMetricsConfig, VolumeServerState, build_metrics_router,
};
use seaweed_volume::server::write_queue::{self, WriteQueue};
use seaweed_volume::storage::store::Store;
use seaweed_volume::storage::types::DiskType;
use tokio_rustls::TlsAcceptor;
#[cfg(unix)]
type CpuProfileParam = Option<CpuProfileSession>;
#[cfg(not(unix))]
type CpuProfileParam = Option<()>;
// The two settings that only make sense for the inbound server. The rest of
// this server's HTTP/2 tuning — keepalive, window sizes, message size — is
// imported from `server::grpc_client` above, which is also what the outgoing
// clients dial with, so the two directions cannot drift apart.
const GRPC_MAX_HEADER_LIST_SIZE: u32 = 8 * 1024 * 1024;
const GRPC_MAX_CONCURRENT_STREAMS: u32 = 1000;
fn main() {
// Before anything allocates: stop glibc from training its mmap threshold
// upward on our large EC buffers and turning them into heap it never
// returns. See seaweed_volume::malloc_tuning for the measurements.
let malloc_tuning = seaweed_volume::malloc_tuning::pin_mmap_threshold();
install_default_crypto_provider();
// Initialize tracing
tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env()
.unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")),
)
.init();
let config = config::parse_cli();
seaweed_volume::server::server_stats::init_process_start();
#[cfg(unix)]
let cpu_profile = match CpuProfileSession::start(&config) {
Ok(session) => session,
Err(e) => {
error!("{}", e);
std::process::exit(1);
}
};
#[cfg(not(unix))]
let cpu_profile: Option<()> = None;
info!(
"SeaweedFS Volume Server (Rust) v{}",
seaweed_volume::version::full_version()
);
match malloc_tuning {
seaweed_volume::malloc_tuning::MallocTuning::Pinned(bytes) => {
info!("pinned glibc M_MMAP_THRESHOLD to {} bytes", bytes)
}
seaweed_volume::malloc_tuning::MallocTuning::DeferredToEnv => info!(
"an allocator mmap-threshold override ({}) is set; leaving glibc's mmap threshold to the environment",
seaweed_volume::malloc_tuning::MMAP_THRESHOLD_ENV
),
seaweed_volume::malloc_tuning::MallocTuning::Failed => {
warn!("mallopt(M_MMAP_THRESHOLD) failed; large freed buffers may stay resident")
}
seaweed_volume::malloc_tuning::MallocTuning::NotApplicable => {}
}
// Register Prometheus metrics
metrics::register_metrics();
// Build the tokio runtime and run the async entry point
let rt = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.expect("Failed to build tokio runtime");
if let Err(e) = rt.block_on(run(config, cpu_profile)) {
error!("Volume server failed: {}", e);
std::process::exit(1);
}
}
/// Mirrors Go's rejectHttpsDowngrade: a redirect must not downgrade a request
/// carrying data and auth headers from https to plaintext.
fn redirect_is_https_downgrade(previous: &[reqwest::Url], next: &reqwest::Url) -> bool {
matches!(previous.first(), Some(url) if url.scheme() == "https") && next.scheme() != "https"
}
fn reject_https_downgrade(attempt: reqwest::redirect::Attempt) -> reqwest::redirect::Action {
if attempt.previous().len() >= 10 {
attempt.error("redirect limit reached")
} else if redirect_is_https_downgrade(attempt.previous(), attempt.url()) {
attempt.error("redirect downgraded https to plaintext")
} else {
attempt.follow()
}
}
/// A peer that never answers a SYN (silent partition) must not stall new
/// outgoing connections; requests set their own overall bounds as needed.
const OUTGOING_CONNECT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
/// Base of every outgoing HTTP client: redirect policy and connect bound.
fn outgoing_client_builder(redirect_policy: reqwest::redirect::Policy) -> reqwest::ClientBuilder {
reqwest::Client::builder()
.redirect(redirect_policy)
.connect_timeout(OUTGOING_CONNECT_TIMEOUT)
}
fn build_outgoing_http_client(
config: &VolumeServerConfig,
) -> Result<(reqwest::Client, String), Box<dyn std::error::Error>> {
let scheme = if config.https_client_enabled {
"https"
} else {
"http"
};
let redirect_policy = reqwest::redirect::Policy::custom(reject_https_downgrade);
if !config.https_client_enabled {
return Ok((
outgoing_client_builder(redirect_policy).build()?,
scheme.to_string(),
));
}
let mut builder = outgoing_client_builder(redirect_policy);
if !config.https_client_ca_file.is_empty() {
let ca_pem = std::fs::read(&config.https_client_ca_file).map_err(|e| {
format!(
"Failed to read HTTPS client CA file '{}': {}",
config.https_client_ca_file, e
)
})?;
let cert = reqwest::Certificate::from_pem(&ca_pem).map_err(|e| {
format!(
"Failed to parse HTTPS client CA PEM '{}': {}",
config.https_client_ca_file, e
)
})?;
builder = builder.add_root_certificate(cert);
}
match (
config.https_client_cert_file.is_empty(),
config.https_client_key_file.is_empty(),
) {
(true, true) => {}
(false, false) => {
let cert_pem = std::fs::read(&config.https_client_cert_file).map_err(|e| {
format!(
"Failed to read HTTPS client cert file '{}': {}",
config.https_client_cert_file, e
)
})?;
let key_pem = std::fs::read(&config.https_client_key_file).map_err(|e| {
format!(
"Failed to read HTTPS client key file '{}': {}",
config.https_client_key_file, e
)
})?;
let mut identity_pem = cert_pem;
if !identity_pem.ends_with(b"\n") {
identity_pem.push(b'\n');
}
identity_pem.extend_from_slice(&key_pem);
let identity = reqwest::Identity::from_pem(&identity_pem).map_err(|e| {
format!(
"Failed to parse HTTPS client identity '{}'+ '{}': {}",
config.https_client_cert_file, config.https_client_key_file, e
)
})?;
builder = builder.identity(identity);
}
_ => {
return Err(format!(
"HTTPS client requires both cert and key, got cert='{}' key='{}'",
config.https_client_cert_file, config.https_client_key_file
)
.into());
}
}
Ok((builder.build()?, scheme.to_string()))
}
fn tls_policy_is_configured(policy: &TlsPolicy) -> bool {
!policy.min_version.is_empty()
|| !policy.max_version.is_empty()
|| !policy.cipher_suites.is_empty()
}
fn effective_http_tls_policy(ca_path: &str, configured_policy: &TlsPolicy) -> TlsPolicy {
if ca_path.is_empty() {
TlsPolicy::default()
} else {
configured_policy.clone()
}
}
fn build_grpc_server_tls_acceptor(
cert_path: &str,
key_path: &str,
ca_path: &str,
tls_policy: &TlsPolicy,
allowed_wildcard_domain: &str,
allowed_common_names: &[String],
) -> Result<Option<TlsAcceptor>, Box<dyn std::error::Error>> {
if cert_path.is_empty() && key_path.is_empty() {
return Ok(None);
}
if cert_path.is_empty() || key_path.is_empty() {
return Err("grpc.volume.cert and grpc.volume.key must both be set".into());
}
if ca_path.is_empty() {
return Err("grpc.volume cert and key are set but grpc.ca is not".into());
}
let client_auth_policy = GrpcClientAuthPolicy {
allowed_common_names: allowed_common_names.to_vec(),
allowed_wildcard_domain: allowed_wildcard_domain.to_string(),
};
let mut server_config = build_rustls_server_config_with_grpc_client_auth(
cert_path,
key_path,
ca_path,
tls_policy,
&client_auth_policy,
)?;
server_config.alpn_protocols = vec![b"h2".to_vec()];
Ok(Some(TlsAcceptor::from(Arc::new(server_config))))
}
fn build_http_server_tls_acceptor(
config: &VolumeServerConfig,
) -> Result<Option<TlsAcceptor>, Box<dyn std::error::Error>> {
if config.https_cert_file.is_empty() || config.https_key_file.is_empty() {
return Ok(None);
}
let effective_policy = effective_http_tls_policy(&config.https_ca_file, &config.tls_policy);
let tls_config = match build_rustls_server_config(
&config.https_cert_file,
&config.https_key_file,
&config.https_ca_file,
&effective_policy,
) {
Ok(tls_config) => tls_config,
Err(e)
if !config.https_ca_file.is_empty() && tls_policy_is_configured(&config.tls_policy) =>
{
warn!(
"Failed to apply HTTP TLS policy '{}', falling back to default rustls policy",
e
);
build_rustls_server_config(
&config.https_cert_file,
&config.https_key_file,
&config.https_ca_file,
&TlsPolicy::default(),
)?
}
Err(e) => return Err(e.into()),
};
Ok(Some(TlsAcceptor::from(Arc::new(tls_config))))
}
fn build_grpc_server_builder() -> tonic::transport::Server {
tonic::transport::Server::builder()
.http2_keepalive_interval(Some(GRPC_KEEPALIVE_INTERVAL))
.http2_keepalive_timeout(Some(GRPC_KEEPALIVE_TIMEOUT))
.max_concurrent_streams(Some(GRPC_MAX_CONCURRENT_STREAMS))
.initial_stream_window_size(Some(GRPC_INITIAL_WINDOW_SIZE))
.initial_connection_window_size(Some(GRPC_INITIAL_WINDOW_SIZE))
.http2_max_header_list_size(Some(GRPC_MAX_HEADER_LIST_SIZE))
}
fn build_volume_grpc_service(
grpc_service: VolumeGrpcService,
) -> VolumeServerServer<VolumeGrpcService> {
VolumeServerServer::new(grpc_service)
.max_decoding_message_size(GRPC_MAX_MESSAGE_SIZE)
.max_encoding_message_size(GRPC_MAX_MESSAGE_SIZE)
}
fn apply_idle_timeout<S>(
stream: S,
idle_timeout: std::time::Duration,
) -> std::pin::Pin<Box<tokio_io_timeout::TimeoutStream<S>>>
where
S: tokio::io::AsyncRead + tokio::io::AsyncWrite,
{
let mut stream = tokio_io_timeout::TimeoutStream::new(stream);
if !idle_timeout.is_zero() {
stream.set_read_timeout(Some(idle_timeout));
stream.set_write_timeout(Some(idle_timeout));
}
Box::pin(stream)
}
async fn run(
config: VolumeServerConfig,
#[allow(unused_variables)] cpu_profile: CpuProfileParam,
) -> Result<(), Box<dyn std::error::Error>> {
// Initialize the store
let mut store = Store::new(config.index_type);
store.id = config.id.clone();
store.ip = config.ip.clone();
store.port = config.port;
store.grpc_port = config.grpc_port;
store.public_url = config.public_url.clone();
store.data_center = config.data_center.clone();
store.rack = config.rack.clone();
// Shared between the gRPC claims and the disk locations' load scans so
// a reload cannot sweep files an in-flight VolumeCopy still writes.
let volume_copies_in_flight =
std::sync::Arc::new(std::sync::Mutex::new(std::collections::HashSet::new()));
store.set_volume_load_guard(volume_copies_in_flight.clone());
// Build shared state
let guard = Guard::new(
&config.white_list,
SigningKey(config.jwt_signing_key.clone()),
config.jwt_signing_expires_seconds,
SigningKey(config.jwt_read_signing_key.clone()),
config.jwt_read_signing_expires_seconds,
);
let master_url = config.masters.first().cloned().unwrap_or_default();
// Defensive-copy the configured seed masters before freezing the lookup
// set, so any later mutation of config.masters cannot desync them.
let master_urls: Vec<String> = config.masters.clone();
let seed_master_set = VolumeServerState::build_seed_master_set(&master_urls);
let self_url = format!("{}:{}", config.ip, config.port);
let (http_client, outgoing_http_scheme) = build_outgoing_http_client(&config)?;
let outgoing_grpc_tls = load_outgoing_grpc_tls(&config)?;
let security_file = config.security_file.clone();
let cli_white_list = config.white_list.clone();
let state = Arc::new(VolumeServerState {
store: RwLock::new(store),
guard: RwLock::new(guard),
is_stopping: RwLock::new(false),
maintenance: std::sync::atomic::AtomicBool::new(false),
state_version: std::sync::atomic::AtomicU32::new(0),
concurrent_upload_limit: config.concurrent_upload_limit,
concurrent_download_limit: config.concurrent_download_limit,
inflight_upload_data_timeout: config.inflight_upload_data_timeout,
inflight_download_data_timeout: config.inflight_download_data_timeout,
inflight_upload_bytes: std::sync::atomic::AtomicI64::new(0),
inflight_download_bytes: std::sync::atomic::AtomicI64::new(0),
upload_notify: tokio::sync::Notify::new(),
download_notify: tokio::sync::Notify::new(),
data_center: config.data_center.clone(),
rack: config.rack.clone(),
file_size_limit_bytes: config.file_size_limit_bytes,
maintenance_byte_per_second: config.maintenance_byte_per_second,
vacuum_commit_slots: if config.max_parallel_vacuum_commits > 0 {
Some(tokio::sync::Semaphore::new(
config.max_parallel_vacuum_commits,
))
} else {
None
},
vacuum_commits_waiting: std::sync::atomic::AtomicU32::new(0),
// Go sets isHeartbeating: true unconditionally at startup
is_heartbeating: std::sync::atomic::AtomicBool::new(true),
has_master: !config.masters.is_empty(),
pre_stop_seconds: config.pre_stop_seconds,
volume_state_notify: tokio::sync::Notify::new(),
write_queue: std::sync::OnceLock::new(),
read_mode: config.read_mode,
allow_untrusted_remote_endpoints: config.allow_untrusted_remote_endpoints,
master_url,
master_urls,
seed_master_set,
current_master_url: tokio::sync::RwLock::new(String::new()),
self_url,
http_client,
outgoing_http_scheme,
outgoing_grpc_tls,
metrics_runtime: std::sync::RwLock::new(RuntimeMetricsConfig::default()),
metrics_notify: tokio::sync::Notify::new(),
fix_jpg_orientation: config.fix_jpg_orientation,
has_slow_read: config.has_slow_read,
read_buffer_size_bytes: (config.read_buffer_size_mb.max(1) as usize) * 1024 * 1024,
security_file,
cli_white_list,
state_file_path: if config.folders.is_empty() {
String::new()
} else {
std::path::Path::new(&config.folders[0])
.join("state.pb")
.to_string_lossy()
.into_owned()
},
ec_lifecycle_locks: std::sync::Mutex::new(std::collections::HashMap::new()),
ec_decodes_in_flight: std::sync::Mutex::new(std::collections::HashSet::new()),
volume_copies_in_flight,
ec_decode_tail: std::sync::Mutex::new(std::collections::HashSet::new()),
ec_decode_tail_notify: tokio::sync::Notify::new(),
});
// Load persisted state from disk if it exists (matches Go's State.Load on startup)
if let Some(saved) =
seaweed_volume::server::grpc_server::load_state_file(&state.state_file_path)
{
state
.maintenance
.store(saved.maintenance, std::sync::atomic::Ordering::Relaxed);
state
.state_version
.store(saved.version, std::sync::atomic::Ordering::Relaxed);
}
if !config.masters.is_empty() {
let hb_config = seaweed_volume::server::heartbeat::HeartbeatConfig {
ip: config.ip.clone(),
port: config.port,
grpc_port: config.grpc_port,
public_url: config.public_url.clone(),
data_center: config.data_center.clone(),
rack: config.rack.clone(),
master_addresses: config.masters.clone(),
pulse_seconds: 5,
};
seaweed_volume::server::heartbeat::prime_master_configuration(&hb_config, &state).await;
}
{
let mut store = state.store.write().unwrap();
store.store_dirs = config.folders.clone();
for (i, dir) in config.folders.iter().enumerate() {
let idx_dir = if config.idx_folder.is_empty() {
dir.as_str()
} else {
config.idx_folder.as_str()
};
let max_volumes = config.folder_max_limits[i];
let disk_type = DiskType::from_string(&config.disk_types[i]);
let tags = config.folder_tags.get(i).cloned().unwrap_or_default();
info!(
"Adding storage location: {} (max_volumes={}, disk_type={:?})",
dir, max_volumes, disk_type
);
let min_free_space = config.min_free_spaces[i].clone();
store
.add_location(dir, idx_dir, max_volumes, disk_type, min_free_space, tags)
.map_err(|e| format!("Failed to add storage location {}: {}", dir, e))?;
}
}
// The write queue is always on: durable (fsync) uploads go through it so
// a batch shares one .dat/.idx sync, the way Go's per-volume async writer
// group-commits. Non-fsync writes stay on the synchronous path.
let min_bytes = usize::try_from(config.fsync_min_batch_bytes)
.map_err(|_| "fsyncMinBatchBytes exceeds this platform's maximum usize value")?;
let wq = WriteQueue::new(
state.clone(),
128,
write_queue::SyncBatchOption {
min_bytes,
min_requests: config.fsync_min_batch_requests,
wait: config.fsync_batch_wait,
},
);
let _ = state.write_queue.set(wq);
// Set initial metric gauges for concurrent limits and max volumes
metrics::CONCURRENT_UPLOAD_LIMIT.set(state.concurrent_upload_limit);
metrics::CONCURRENT_DOWNLOAD_LIMIT.set(state.concurrent_download_limit);
{
let store = state.store.read().unwrap();
let mut max_vols: i64 = 0;
for loc in &store.locations {
max_vols += loc
.max_volume_count
.load(std::sync::atomic::Ordering::Relaxed) as i64;
}
metrics::MAX_VOLUMES.set(max_vols);
}
// Run initial disk space check
{
let store = state.store.read().unwrap();
for loc in &store.locations {
loc.check_disk_space();
}
}
// Spawn background disk space monitor (checks every 60 seconds)
{
let monitor_state = state.clone();
tokio::spawn(async move {
let mut interval = tokio::time::interval(std::time::Duration::from_secs(60));
interval.tick().await; // skip the first immediate tick
loop {
interval.tick().await;
let store = monitor_state.store.read().unwrap();
for loc in &store.locations {
loc.check_disk_space();
}
}
});
}
// Build HTTP routers
#[allow(unused_mut)]
let mut admin_router = seaweed_volume::server::volume_server::build_admin_router_with_ui(
state.clone(),
config.ui_enabled,
);
#[cfg(unix)]
if config.pprof {
admin_router = admin_router.merge(build_debug_router());
}
let admin_addr = format!("{}:{}", config.bind_ip, config.port);
let public_port = config.public_port;
let needs_public = public_port != config.port;
let http_idle_timeout = std::time::Duration::from_secs(config.idle_timeout as u64);
let grpc_addr = format!("{}:{}", config.bind_ip, config.grpc_port);
let grpc_tls_acceptor = build_grpc_server_tls_acceptor(
&config.grpc_cert_file,
&config.grpc_key_file,
&config.grpc_ca_file,
&config.tls_policy,
&config.grpc_allowed_wildcard_domain,
&config.grpc_volume_allowed_common_names,
)?;
info!("Starting HTTP server on {}", admin_addr);
info!("Starting gRPC server on {}", grpc_addr);
if needs_public {
info!(
"Starting public HTTP server on {}:{}",
config.bind_ip, public_port
);
}
// Set up graceful shutdown via SIGINT/SIGTERM using broadcast channel
let (shutdown_tx, _) = tokio::sync::broadcast::channel::<()>(1);
let state_shutdown = state.clone();
let shutdown_tx_clone = shutdown_tx.clone();
tokio::spawn(async move {
let ctrl_c = tokio::signal::ctrl_c();
#[cfg(unix)]
{
let mut sigterm =
tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())
.expect("Failed to install SIGTERM handler");
tokio::select! {
_ = ctrl_c => { info!("Received SIGINT, shutting down..."); }
_ = sigterm.recv() => { info!("Received SIGTERM, shutting down..."); }
}
}
#[cfg(not(unix))]
{
ctrl_c.await.ok();
info!("Received shutdown signal...");
}
*state_shutdown.is_stopping.write().unwrap() = true;
// Wake heartbeat loop immediately so it sends deregister heartbeat
// before the pre_stop delay (matches Go: StopHeartbeat() closes stopChan
// before sleeping preStopSeconds)
state_shutdown.volume_state_notify.notify_one();
// Graceful drain: wait pre_stop_seconds before shutting down servers
let pre_stop = state_shutdown.pre_stop_seconds;
if pre_stop > 0 {
info!("Pre-stop: waiting {} seconds before shutdown...", pre_stop);
tokio::time::sleep(std::time::Duration::from_secs(pre_stop as u64)).await;
}
let _ = shutdown_tx_clone.send(());
});
// Set up SIGHUP handler for config reload (mirrors Go's grace.OnReload)
#[cfg(unix)]
{
let state_reload = state.clone();
tokio::spawn(async move {
let mut sighup = tokio::signal::unix::signal(tokio::signal::unix::SignalKind::hangup())
.expect("Failed to install SIGHUP handler");
loop {
sighup.recv().await;
info!("Received SIGHUP, reloading...");
// 1. Load new volumes from disk (Go's LoadNewVolumes)
{
info!("Loading new volume ids...");
let mut store = state_reload.store.write().unwrap();
store.load_new_volumes();
}
// 2. Reload security config (Go's Reload)
{
info!("Reloading security config...");
let sec = config::parse_security_config(&state_reload.security_file);
let mut whitelist = state_reload.cli_white_list.clone();
whitelist.extend(sec.guard_white_list.iter().cloned());
let mut guard = state_reload.guard.write().unwrap();
guard.update_whitelist(&whitelist);
guard.update_signing_keys(
SigningKey(sec.jwt_signing_key),
sec.jwt_signing_expires,
SigningKey(sec.jwt_read_signing_key),
sec.jwt_read_signing_expires,
);
}
// Trigger heartbeat to report new volumes
state_reload.volume_state_notify.notify_one();
info!("SIGHUP reload complete");
}
});
}
// Build optional TLS acceptor for HTTPS
let https_tls_acceptor =
if !config.https_cert_file.is_empty() && !config.https_key_file.is_empty() {
info!(
"TLS enabled for HTTP server (cert={}, key={})",
config.https_cert_file, config.https_key_file
);
build_http_server_tls_acceptor(&config)?
} else {
None
};
// Spawn all servers concurrently
let admin_listener = tokio::net::TcpListener::bind(&admin_addr)
.await
.unwrap_or_else(|e| panic!("Failed to bind HTTP to {}: {}", admin_addr, e));
let scheme = if https_tls_acceptor.is_some() {
"HTTPS"
} else {
"HTTP"
};
info!("{} server listening on {}", scheme, admin_addr);
let http_handle = if let Some(tls_acceptor) = https_tls_acceptor.clone() {
let mut shutdown_rx = shutdown_tx.subscribe();
tokio::spawn(async move {
serve_https(
admin_listener,
admin_router,
tls_acceptor,
http_idle_timeout,
async move {
let _ = shutdown_rx.recv().await;
},
)
.await;
})
} else {
let mut shutdown_rx = shutdown_tx.subscribe();
tokio::spawn(async move {
serve_http(
admin_listener,
admin_router,
http_idle_timeout,
async move {
let _ = shutdown_rx.recv().await;
},
)
.await;
})
};
// Bind the gRPC listener before spawning to propagate bind errors at startup.
let grpc_listener = tokio::net::TcpListener::bind(&grpc_addr)
.await
.unwrap_or_else(|e| panic!("Failed to bind gRPC to {}: {}", grpc_addr, e));
let grpc_local_addr = grpc_listener
.local_addr()
.unwrap_or_else(|e| panic!("Failed to get gRPC local addr: {}", e));
let grpc_handle = {
let grpc_state = state.clone();
let grpc_tls_acceptor = grpc_tls_acceptor.clone();
let mut shutdown_rx = shutdown_tx.subscribe();
let shutdown_tx_grpc = shutdown_tx.clone();
tokio::spawn(async move {
let grpc_service = VolumeGrpcService {
state: grpc_state.clone(),
};
let reflection_v1 = tonic_reflection::server::Builder::configure()
.register_encoded_file_descriptor_set(seaweed_volume::pb::FILE_DESCRIPTOR_SET)
.build_v1()
.expect("Failed to build gRPC reflection v1 service");
let reflection_v1alpha = tonic_reflection::server::Builder::configure()
.register_encoded_file_descriptor_set(seaweed_volume::pb::FILE_DESCRIPTOR_SET)
.build_v1alpha()
.expect("Failed to build gRPC reflection v1alpha service");
let result = if let Some(tls_acceptor) = grpc_tls_acceptor {
let incoming =
grpc_tls_incoming(grpc_listener, tls_acceptor, GRPC_TLS_HANDSHAKE_TIMEOUT);
info!("gRPC server listening on {} (TLS enabled)", grpc_local_addr);
build_grpc_server_builder()
.layer(GrpcRequestIdLayer)
.add_service(reflection_v1)
.add_service(reflection_v1alpha)
.add_service(build_volume_grpc_service(grpc_service))
.serve_with_incoming_shutdown(incoming, async move {
let _ = shutdown_rx.recv().await;
})
.await
} else {
let incoming = tokio_stream::wrappers::TcpListenerStream::new(grpc_listener);
info!("gRPC server listening on {}", grpc_local_addr);
build_grpc_server_builder()
.layer(GrpcRequestIdLayer)
.add_service(reflection_v1)
.add_service(reflection_v1alpha)
.add_service(build_volume_grpc_service(grpc_service))
.serve_with_incoming_shutdown(incoming, async move {
let _ = shutdown_rx.recv().await;
})
.await
};
if let Err(ref e) = result {
error!("gRPC server error: {}", e);
let _ = shutdown_tx_grpc.send(());
}
result
})
};
// Spawn heartbeat to master (if master addresses are configured)
let heartbeat_handle = {
let master_addrs = config.masters.clone();
if !master_addrs.is_empty() {
let hb_config = seaweed_volume::server::heartbeat::HeartbeatConfig {
ip: config.ip.clone(),
port: config.port,
grpc_port: config.grpc_port,
public_url: config.public_url.clone(),
data_center: config.data_center.clone(),
rack: config.rack.clone(),
master_addresses: master_addrs.clone(),
pulse_seconds: 5,
};
let hb_shutdown = shutdown_tx.subscribe();
let hb_state = state.clone();
info!("Will send heartbeats to master: {:?}", master_addrs);
Some(tokio::spawn(async move {
seaweed_volume::server::heartbeat::run_heartbeat_with_state(
hb_config,
hb_state,
hb_shutdown,
)
.await;
}))
} else {
None
}
};
let public_handle = if needs_public {
let public_router =
seaweed_volume::server::volume_server::build_public_router(state.clone());
let public_addr = format!("{}:{}", config.bind_ip, public_port);
let listener = tokio::net::TcpListener::bind(&public_addr)
.await
.unwrap_or_else(|e| panic!("Failed to bind public HTTP to {}: {}", public_addr, e));
info!("Public HTTP server listening on {}", public_addr);
let mut shutdown_rx = shutdown_tx.subscribe();
Some(tokio::spawn(async move {
serve_http(listener, public_router, http_idle_timeout, async move {
let _ = shutdown_rx.recv().await;
})
.await;
}))
} else {
None
};
let metrics_handle = if config.metrics_port > 0 {
let metrics_router = build_metrics_router();
let metrics_addr = format!("{}:{}", config.metrics_ip, config.metrics_port);
info!("Metrics server listening on {}", metrics_addr);
let listener = tokio::net::TcpListener::bind(&metrics_addr)
.await
.unwrap_or_else(|e| panic!("Failed to bind metrics HTTP to {}: {}", metrics_addr, e));
let mut shutdown_rx = shutdown_tx.subscribe();
Some(tokio::spawn(async move {
if let Err(e) = axum::serve(listener, metrics_router)
.with_graceful_shutdown(async move {
let _ = shutdown_rx.recv().await;
})
.await
{
error!("Metrics HTTP server error: {}", e);
}
}))
} else {
None
};
#[cfg(unix)]
let debug_handle = if config.debug {
let debug_addr = format!("0.0.0.0:{}", config.debug_port);
info!("Debug pprof server listening on {}", debug_addr);
let listener = tokio::net::TcpListener::bind(&debug_addr)
.await
.unwrap_or_else(|e| panic!("Failed to bind debug HTTP to {}: {}", debug_addr, e));
let debug_router = build_debug_router();
let mut shutdown_rx = shutdown_tx.subscribe();
Some(tokio::spawn(async move {
if let Err(e) = axum::serve(listener, debug_router)
.with_graceful_shutdown(async move {
let _ = shutdown_rx.recv().await;
})
.await
{
error!("Debug HTTP server error: {}", e);
}
}))
} else {
None
};
#[cfg(not(unix))]
let debug_handle: Option<tokio::task::JoinHandle<()>> = None;
let metrics_push_handle = {
let push_state = state.clone();
let push_instance = format!("{}:{}", config.ip, config.port);
let push_shutdown = shutdown_tx.subscribe();
Some(tokio::spawn(async move {
run_metrics_push_loop(push_state, push_instance, push_shutdown).await;
}))
};
// Whichever server stops first takes the other down, as a serve failure
// is fatal for the Go volume server.
let mut server_err: Option<String> = None;
let mut grpc_handle = grpc_handle;
let mut http_handle = http_handle;
let (http_result, grpc_result) = tokio::select! {
r = &mut grpc_handle => {
let _ = shutdown_tx.send(());
(http_handle.await, r)
}
r = &mut http_handle => {
let _ = shutdown_tx.send(());
(r, grpc_handle.await)
}
};
if let Err(e) = http_result {
let msg = format!("HTTP task panicked: {}", e);
error!("{}", msg);
server_err = Some(msg);
}
match grpc_result {
Ok(Ok(())) => {}
Ok(Err(e)) => {
let msg = format!("gRPC server exited with error: {}", e);
error!("{}", msg);
server_err = Some(msg);
}
Err(e) => {
let msg = format!("gRPC task panicked: {}", e);
error!("{}", msg);
server_err = Some(msg);
}
}
if let Some(h) = public_handle {
let _ = h.await;
}
if let Some(h) = metrics_handle {
let _ = h.await;
}
if let Some(h) = debug_handle {
let _ = h.await;
}
if let Some(h) = heartbeat_handle {
let _ = h.await;
}
if let Some(h) = metrics_push_handle {
let _ = h.await;
}
// Close all volumes (flush and release file handles) matching Go's Shutdown()
state.store.write().unwrap().close();
#[cfg(unix)]
if let Some(cpu_profile) = cpu_profile {
cpu_profile.finish().map_err(std::io::Error::other)?;
}
if let Some(err_msg) = server_err {
return Err(std::io::Error::other(err_msg).into());
}
info!("Volume server stopped.");
Ok(())
}
async fn run_metrics_push_loop(
state: Arc<VolumeServerState>,
instance: String,
mut shutdown_rx: tokio::sync::broadcast::Receiver<()>,
) {
loop {
let push_cfg = { state.metrics_runtime.read().unwrap().push_gateway.clone() };
if push_cfg.address.is_empty() || push_cfg.interval_seconds == 0 {
tokio::select! {
_ = state.metrics_notify.notified() => continue,
_ = shutdown_rx.recv() => return,
}
}
if let Err(e) = metrics::push_metrics_once(
&state.http_client,
&push_cfg.address,
"volumeServer",
&instance,
)
.await
{
info!("could not push metrics to {}: {}", push_cfg.address, e);
}
let interval = std::time::Duration::from_secs(push_cfg.interval_seconds.max(1) as u64);
tokio::select! {
_ = tokio::time::sleep(interval) => {}
_ = state.metrics_notify.notified() => {}
_ = shutdown_rx.recv() => return,
}
}
}
/// grpc-go's default ConnectionTimeout, which bounds a Go server's handshake.
const GRPC_TLS_HANDSHAKE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(120);
/// Each handshake runs on its own task, so a peer that connects and stays
/// silent holds up no other connection, as with a Go gRPC server.
fn grpc_tls_incoming(
listener: tokio::net::TcpListener,
tls_acceptor: TlsAcceptor,
handshake_timeout: std::time::Duration,
) -> impl tokio_stream::Stream<
Item = Result<tokio_rustls::server::TlsStream<tokio::net::TcpStream>, std::io::Error>,
> {
let (tx, rx) = tokio::sync::mpsc::channel(64);
tokio::spawn(async move {
let mut retry_delay = std::time::Duration::ZERO;
loop {
let accepted = tokio::select! {
_ = tx.closed() => return,
accepted = listener.accept() => accepted,
};
let (tcp_stream, remote_addr) = match accepted {
Ok(accepted) => {
retry_delay = std::time::Duration::ZERO;
accepted
}
Err(e) => {
// Back off and retry like grpc-go, so running out of file
// descriptors does not close the gRPC port for good.
retry_delay = (retry_delay * 2).clamp(
std::time::Duration::from_millis(5),
std::time::Duration::from_secs(1),
);
tracing::warn!("gRPC accept error: {}; retrying in {:?}", e, retry_delay);
tokio::select! {
_ = tx.closed() => return,
_ = tokio::time::sleep(retry_delay) => continue,
}
}
};
let tls_acceptor = tls_acceptor.clone();
let tx = tx.clone();
tokio::spawn(async move {
match tokio::time::timeout(handshake_timeout, tls_acceptor.accept(tcp_stream)).await
{
Ok(Ok(tls_stream)) => {
let _ = tx.send(Ok(tls_stream)).await;
}
Ok(Err(e)) => {
tracing::debug!("gRPC TLS handshake failed from {}: {}", remote_addr, e);
}
Err(_) => {
tracing::debug!("gRPC TLS handshake from {} timed out", remote_addr);
}
}
});
}
});
tokio_stream::wrappers::ReceiverStream::new(rx)
}
/// Serve an axum Router over TLS using tokio-rustls.
/// Accepts TCP connections, performs TLS handshake, then serves HTTP over the encrypted stream.
async fn serve_http<F>(
tcp_listener: tokio::net::TcpListener,
app: axum::Router,
idle_timeout: std::time::Duration,
shutdown_signal: F,
) where
F: std::future::Future<Output = ()> + Send + 'static,
{
use hyper_util::rt::{TokioExecutor, TokioIo};
use hyper_util::server::conn::auto::Builder as HttpBuilder;
use hyper_util::service::TowerToHyperService;
use tower::Service;
let mut make_svc = app.into_make_service_with_connect_info::<std::net::SocketAddr>();
tokio::pin!(shutdown_signal);
loop {
tokio::select! {
_ = &mut shutdown_signal => {
info!("HTTP server shutting down");
break;
}
result = tcp_listener.accept() => {
match result {
Ok((tcp_stream, remote_addr)) => {
let tower_svc = make_svc.call(remote_addr).await.expect("infallible");
let hyper_svc = TowerToHyperService::new(tower_svc);
tokio::spawn(async move {
let io = TokioIo::new(apply_idle_timeout(tcp_stream, idle_timeout));
let builder = HttpBuilder::new(TokioExecutor::new());
if let Err(e) = builder.serve_connection(io, hyper_svc).await {
tracing::debug!("HTTP connection error: {}", e);
}
});
}
Err(e) => {
error!("Failed to accept TCP connection: {}", e);
}
}
}
}
}
}
async fn serve_https<F>(
tcp_listener: tokio::net::TcpListener,
app: axum::Router,
tls_acceptor: TlsAcceptor,
idle_timeout: std::time::Duration,
shutdown_signal: F,
) where
F: std::future::Future<Output = ()> + Send + 'static,
{
use hyper_util::rt::{TokioExecutor, TokioIo};
use hyper_util::server::conn::auto::Builder as HttpBuilder;
use hyper_util::service::TowerToHyperService;
use tower::Service;
let mut make_svc = app.into_make_service_with_connect_info::<std::net::SocketAddr>();
tokio::pin!(shutdown_signal);
loop {
tokio::select! {
_ = &mut shutdown_signal => {
info!("HTTPS server shutting down");
break;
}
result = tcp_listener.accept() => {
match result {
Ok((tcp_stream, remote_addr)) => {
let tls_acceptor = tls_acceptor.clone();
let tower_svc = make_svc.call(remote_addr).await.expect("infallible");
let hyper_svc = TowerToHyperService::new(tower_svc);
tokio::spawn(async move {
match tls_acceptor.accept(tcp_stream).await {
Ok(tls_stream) => {
let io = TokioIo::new(apply_idle_timeout(tls_stream, idle_timeout));
let builder = HttpBuilder::new(TokioExecutor::new());
if let Err(e) = builder.serve_connection(io, hyper_svc).await {
tracing::debug!("HTTPS connection error: {}", e);
}
}
Err(e) => {
tracing::debug!("TLS handshake failed: {}", e);
}
}
});
}
Err(e) => {
error!("Failed to accept TCP connection: {}", e);
}
}
}
}
}
}
#[cfg(test)]
mod tests {
use super::{
GRPC_TLS_HANDSHAKE_TIMEOUT, build_grpc_server_tls_acceptor, effective_http_tls_policy,
grpc_tls_incoming, tls_policy_is_configured,
};
use seaweed_volume::security::tls::TlsPolicy;
/// A connect that never finishes fails at the 5 s bound. It stalls in the
/// resolver, which the connect bound also covers, because that needs no
/// routing: a real unanswered SYN depends on the host's network.
#[tokio::test(start_paused = true)]
async fn outgoing_client_bounds_a_stalled_connect() {
struct Stalled;
impl reqwest::dns::Resolve for Stalled {
fn resolve(&self, _: reqwest::dns::Name) -> reqwest::dns::Resolving {
Box::pin(std::future::pending())
}
}
let client = super::outgoing_client_builder(reqwest::redirect::Policy::none())
.no_proxy()
.dns_resolver(std::sync::Arc::new(Stalled))
.build()
.unwrap();
let started = tokio::time::Instant::now();
let sent = tokio::time::timeout(
std::time::Duration::from_secs(600),
client.get("http://silent-peer.invalid:8080/status").send(),
)
.await
.expect("a stalled connect must not outlive its bound");
assert!(sent.is_err(), "a stalled connect cannot answer");
assert_eq!(started.elapsed(), super::OUTGOING_CONNECT_TIMEOUT);
}
fn write_pem(dir: &tempfile::TempDir, name: &str, body: &str) -> String {
let path = dir.path().join(name);
std::fs::write(&path, body).unwrap();
path.to_string_lossy().into_owned()
}
// A CA and a leaf it signed for localhost, usable on either side.
const TEST_CA_PEM: &str = "-----BEGIN CERTIFICATE-----\nMIIBhzCCAS2gAwIBAgIUWQMpep+B+ShfbwEfk20OGMkBgpAwCgYIKoZIzj0EAwIw\nGTEXMBUGA1UEAwwOdm9sdW1lLXRlc3QtY2EwHhcNMjYxMTAzMTgzOTI2WhcNMzYx\nMDMxMTgzOTI2WjAZMRcwFQYDVQQDDA52b2x1bWUtdGVzdC1jYTBZMBMGByqGSM49\nAgEGCCqGSM49AwEHA0IABD8L549bGTiFShKDcA2UFhz2MH9wxVuFCWCfJ7OVYM2x\nweY/5uCLjBRAnFx5FuWnaNtdTWkImXBSguHe861foCajUzBRMB0GA1UdDgQWBBQH\nOSZ93m5jmlLJpS5cFzH2hicEdjAfBgNVHSMEGDAWgBQHOSZ93m5jmlLJpS5cFzH2\nhicEdjAPBgNVHRMBAf8EBTADAQH/MAoGCCqGSM49BAMCA0gAMEUCIEetcimwxiDU\n8RS0b5uaev3c/kbnOT1nOV6n2wdtplZ4AiEAlWDH77I9ssy+yqXArkdiEadePIl5\nUyXg6crXqITwUeE=\n-----END CERTIFICATE-----\n";
const TEST_LEAF_PEM: &str = "-----BEGIN CERTIFICATE-----\nMIIBuTCCAV+gAwIBAgIUC4xhlzsyNZi0XHzyCwV/xquGQLkwCgYIKoZIzj0EAwIw\nGTEXMBUGA1UEAwwOdm9sdW1lLXRlc3QtY2EwHhcNMjYxMTAzMTgzOTI2WhcNMzYx\nMDMxMTgzOTI2WjAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwWTATBgcqhkjOPQIBBggq\nhkjOPQMBBwNCAASKt0T/i1pt7Aw3DloYFRoaX8+w/reR3tIEaMo911/fIvQhqkg4\nLPCXKocq0KNw3Na3HnZeU9Ls+FLoSHynBsyIo4GJMIGGMBoGA1UdEQQTMBGCCWxv\nY2FsaG9zdIcEfwAAATAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwCQYD\nVR0TBAIwADAdBgNVHQ4EFgQUp8jNqBiE0UeYRkCny2In6SyKEKkwHwYDVR0jBBgw\nFoAUBzkmfd5uY5pSyaUuXBcx9oYnBHYwCgYIKoZIzj0EAwIDSAAwRQIhAIzNfGpS\n+rIxr3eGInfD+0DzQxSuqfcQZTkV2eGkebcuAiAIV73Y7wKg7D08qfmmrH/TaAZB\nm2wr/1assOVDaw8dMg==\n-----END CERTIFICATE-----\n";
const TEST_LEAF_KEY_PEM: &str = "-----BEGIN PRIVATE KEY-----\nMIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgmMs2vKPN5ymhYOXl\nztUUiSWAcmKVVWxveajzfxIwqBehRANCAASKt0T/i1pt7Aw3DloYFRoaX8+w/reR\n3tIEaMo911/fIvQhqkg4LPCXKocq0KNw3Na3HnZeU9Ls+FLoSHynBsyI\n-----END PRIVATE KEY-----\n";
fn test_grpc_tls_acceptor(dir: &tempfile::TempDir) -> tokio_rustls::TlsAcceptor {
seaweed_volume::security::tls::install_default_crypto_provider();
let ca = write_pem(dir, "ca.pem", TEST_CA_PEM);
let cert = write_pem(dir, "leaf.pem", TEST_LEAF_PEM);
let key = write_pem(dir, "leaf.key", TEST_LEAF_KEY_PEM);
build_grpc_server_tls_acceptor(&cert, &key, &ca, &TlsPolicy::default(), "", &[])
.unwrap()
.unwrap()
}
#[tokio::test]
async fn test_grpc_tls_incoming_serves_past_a_silent_peer() {
use tokio_stream::StreamExt;
let dir = tempfile::tempdir().unwrap();
let acceptor = test_grpc_tls_acceptor(&dir);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let mut incoming = Box::pin(grpc_tls_incoming(
listener,
acceptor,
GRPC_TLS_HANDSHAKE_TIMEOUT,
));
let _silent = tokio::net::TcpStream::connect(addr).await.unwrap();
let mut roots = rustls::RootCertStore::empty();
for c in rustls_pemfile::certs(&mut TEST_CA_PEM.as_bytes()) {
roots.add(c.unwrap()).unwrap();
}
let config = rustls::ClientConfig::builder()
.with_root_certificates(roots)
.with_client_auth_cert(
rustls_pemfile::certs(&mut TEST_LEAF_PEM.as_bytes())
.map(|c| c.unwrap())
.collect(),
rustls_pemfile::private_key(&mut TEST_LEAF_KEY_PEM.as_bytes())
.unwrap()
.unwrap(),
)
.unwrap();
let client = tokio::spawn(async move {
let tcp = tokio::net::TcpStream::connect(addr).await.unwrap();
tokio_rustls::TlsConnector::from(std::sync::Arc::new(config))
.connect("localhost".try_into().unwrap(), tcp)
.await
});
let served = tokio::time::timeout(std::time::Duration::from_secs(5), incoming.next()).await;
assert!(
matches!(served, Ok(Some(Ok(_)))),
"the handshake waited behind a silent peer"
);
client.await.unwrap().unwrap();
}
#[tokio::test]
async fn test_grpc_tls_incoming_closes_a_silent_peer_after_the_handshake_timeout() {
use tokio::io::AsyncReadExt;
let dir = tempfile::tempdir().unwrap();
let acceptor = test_grpc_tls_acceptor(&dir);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let _incoming =
grpc_tls_incoming(listener, acceptor, std::time::Duration::from_millis(100));
let mut silent = tokio::net::TcpStream::connect(addr).await.unwrap();
let mut buf = [0u8; 1];
let read =
tokio::time::timeout(std::time::Duration::from_secs(5), silent.read(&mut buf)).await;
assert!(
matches!(read, Ok(Ok(0)) | Ok(Err(_))),
"the silent peer stayed connected past the handshake timeout"
);
}
#[test]
fn test_grpc_server_tls_requires_ca() {
let dir = tempfile::tempdir().unwrap();
let cert = write_pem(
&dir,
"server.crt",
"-----BEGIN CERTIFICATE-----\nZmFrZQ==\n-----END CERTIFICATE-----\n",
);
let key = write_pem(
&dir,
"server.key",
"-----BEGIN PRIVATE KEY-----\nZmFrZQ==\n-----END PRIVATE KEY-----\n",
);
assert!(
build_grpc_server_tls_acceptor(&cert, &key, "", &TlsPolicy::default(), "", &[])
.is_err()
);
}
#[test]
fn test_grpc_server_tls_errors_when_files_are_missing() {
assert!(
build_grpc_server_tls_acceptor(
"/missing/server.crt",
"/missing/server.key",
"/missing/ca.crt",
&TlsPolicy::default(),
"",
&[],
)
.is_err()
);
}
#[test]
fn test_grpc_server_tls_errors_on_unsupported_tls_policy() {
let dir = tempfile::tempdir().unwrap();
let cert = write_pem(
&dir,
"server.crt",
"-----BEGIN CERTIFICATE-----\nZmFrZQ==\n-----END CERTIFICATE-----\n",
);
let key = write_pem(
&dir,
"server.key",
"-----BEGIN PRIVATE KEY-----\nZmFrZQ==\n-----END PRIVATE KEY-----\n",
);
let ca = write_pem(
&dir,
"ca.crt",
"-----BEGIN CERTIFICATE-----\nZmFrZQ==\n-----END CERTIFICATE-----\n",
);
assert!(
build_grpc_server_tls_acceptor(
&cert,
&key,
&ca,
&TlsPolicy {
min_version: "TLS 1.0".to_string(),
max_version: "TLS 1.1".to_string(),
cipher_suites: String::new(),
},
"",
&[],
)
.is_err()
);
}
#[test]
fn test_effective_http_tls_policy_ignores_tls_policy_without_ca() {
let configured = TlsPolicy {
min_version: "TLS 1.3".to_string(),
max_version: "TLS 1.3".to_string(),
cipher_suites: "TLS_AES_128_GCM_SHA256".to_string(),
};
assert_eq!(
effective_http_tls_policy("", &configured),
TlsPolicy::default()
);
assert_eq!(
effective_http_tls_policy("/etc/seaweedfs/http-ca.pem", &configured),
configured
);
}
#[test]
fn test_tls_policy_is_configured_detects_non_empty_fields() {
assert!(!tls_policy_is_configured(&TlsPolicy::default()));
assert!(tls_policy_is_configured(&TlsPolicy {
min_version: "TLS 1.2".to_string(),
max_version: String::new(),
cipher_suites: String::new(),
}));
}
#[test]
fn test_redirect_is_https_downgrade() {
use super::redirect_is_https_downgrade;
let https = reqwest::Url::parse("https://peer:8443/1?type=replicate").unwrap();
let http = reqwest::Url::parse("http://other:8080/1?type=replicate").unwrap();
assert!(redirect_is_https_downgrade(&[https.clone()], &http));
assert!(!redirect_is_https_downgrade(&[http.clone()], &https));
assert!(!redirect_is_https_downgrade(&[http.clone()], &http));
assert!(!redirect_is_https_downgrade(&[https.clone()], &https));
assert!(!redirect_is_https_downgrade(&[], &http));
// Only the origin scheme counts (Go's via[0]): a chain that started
// plaintext loses nothing by returning to plaintext.
assert!(!redirect_is_https_downgrade(&[http.clone(), https.clone()], &http));
}
}