mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-11 00:50:43 +02:00
rust volume: stop glibc retaining freed EC buffers as unreturnable heap (#11255)
* rust volume: stop glibc retaining freed EC buffers as unreturnable heap
A Rust volume server doing EC work accumulates hundreds of MB of resident
anonymous memory that it never gives back, and under a hard cgroup
MemoryMax that ends in an OOM kill while most of the resident set is
free-but-unreturned.
It is not a leak. glibc serves allocations >= M_MMAP_THRESHOLD with mmap
and munmaps them on free, but the threshold is ADAPTIVE: freeing an
mmap'd block raises it toward that block's size, up to 32 MiB. EC
reconstruction and needle reassembly allocate large short-lived buffers,
so the first few train the threshold upward and every later buffer is
carved from the heap instead. Heap pages only return to the OS from the
top of the arena, so they stay resident for the life of the process --
reusable, but anonymous, and anonymous pages cannot be reclaimed under
pressure the way page cache can. The retained footprint is exactly the
headroom a burst of maintenance work needs.
Measured on a 17-node cluster (EC 10+4, --index=redb), one node, two
identical `ec.scrub -mode full` rounds over 10912 EC files each, same
unit restarted with and without a pinned threshold:
baseline round 1 round 2 60s idle
default (adaptive) 10 MB 84 MB 88 MB 88 MB
pinned threshold 10 MB 13 MB 14 MB 14 MB
78 MB retained versus 4 MB for identical work. On heavier mixed scrub
workloads the same effect reached ~600 MB per volume server against a
3 GiB cap, and restarting the process was the only way to release it.
Calling mallopt(M_MMAP_THRESHOLD, ...) sets the threshold and disables
the dynamic adjustment. Pin it to glibc's own default rather than
inventing a value: the goal is to stop the adaptation, not to second-guess
the default. MALLOC_MMAP_THRESHOLD_ still wins if an operator sets it,
glibc-only, and a failed mallopt is logged rather than fatal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFr2v4BUqrXdgj4LEUAwVF
* Address PR review: validate env overrides, honour GLIBC_TUNABLES, fix non-glibc test compile
Three review-bot findings on seaweed-volume/src/malloc_tuning.rs:
1. (CodeRabbit) The test used cfg!(...), which keeps both branches in
compilation. On non-glibc targets DEFAULT_MMAP_THRESHOLD is undefined,
so the test failed to compile. Split into #[cfg]-gated tests so each
branch only references items defined for that target.
2. (Greptile) MALLOC_MMAP_THRESHOLD_ was checked by presence only. An
empty or non-numeric value makes glibc ignore the override while we still
skipped mallopt, leaving the adaptive threshold enabled -- exactly the
behaviour this module exists to prevent. Now we defer only when the value
is non-empty and parses as an integer; otherwise we fall through to
pinning.
3. (Codex) The modern GLIBC_TUNABLES=glibc.malloc.mmap_threshold=... tunable
was missed, so mallopt could overwrite an operator's explicit tunable. Now
we detect that tunable (with the same validation) and defer to it.
The override check moved into the glibc-gated inner function, so off glibc
pin_mmap_threshold() always reports NotApplicable regardless of any
allocator env vars that happen to be set. The startup log for DeferredToEnv
is reworded to cover both override sources. Added tests for the override
parsers and the off-glibc no-op.
* Address round-2 review: match glibc's actual override parsing
Three follow-up review-bot findings after the first round of fixes, all
rooted in our validation not matching how glibc actually parses the
overrides:
1. (Greptile, P1) parse::<i64>() accepted negative values like "-1" and
returned DeferredToEnv, but glibc's threshold is unsigned and rejects
negatives — so we skipped mallopt while glibc also ignored the override,
leaving the adaptive threshold enabled. Now we reject negatives and
zero.
2. (Devin, BUG) glibc parses thresholds as unsigned (strtoul for tunables,
atoi for the legacy var). Values above i64::MAX are valid for glibc but
were rejected by parse::<i64>(), so we pinned 128 KiB over the operator's
explicit setting. Now we parse as u64, accepting the full unsigned range.
3. (CodeRabbit, Major) Two issues in usable_glibc_tunable_threshold:
a. A malformed sibling entry (e.g. glibc.malloc.check=2=2:...) makes
glibc reject the entire GLIBC_TUNABLES string, but our per-entry scan
still returned true for the valid-looking mmap_threshold entry. Now
we validate every entry (exactly one '=') before accepting any.
b. Hex values (0x20000) are accepted by glibc's strtoul but were rejected
by parse::<i64>(). Now parse_strtoul_threshold handles 0x-prefixed hex.
MALLOC_MMAP_THRESHOLD_ stays decimal-only (atoi), matching glibc.
Added regression tests for negatives, zero, >i64::MAX, hex tunables, and
malformed mixed GLIBC_TUNABLES entries. Verified: clippy clean and tests
pass on macOS (non-glibc); glibc-gated code type-checks for
x86_64-unknown-linux-gnu.
* Address round-3 review: match glibc's actual override parsing
Three follow-up review-bot findings (Greptile P1, Devin BUG, CodeRabbit
Major) all on the same issue: the round-2 fix rejected negative and zero
override values, but glibc actually accepts them.
Verified against the glibc source (malloc/malloc.c, malloc/arena.c,
elf/dl-tunables.c, elf/dl-misc.c):
- do_set_mmap_threshold(size_t value) does NO clamping — it just sets
mp_.mmap_threshold = value and mp_.no_dyn_threshold = 1.
- MALLOC_MMAP_THRESHOLD_: glibc calls atoi(value) then mallopt, which
always sets the threshold and disables dynamic adjustment — even for
empty, negative, or non-numeric values (atoi returns 0). So ANY
presence of the variable means the operator's override is in effect.
Reverted to presence-only check for the legacy variable. The round-1
Greptile comment claiming glibc "cannot apply the override" for
empty/malformed values was incorrect.
- GLIBC_TUNABLES: glibc parses values with _dl_strtoul (elf/dl-misc.c),
which accepts decimal, 0x hex, 0 octal, an optional sign (negatives
wrap to unsigned long), and requires the entire value consumed
(tunable_parse_num checks endptr == strval + len). Replaced
parse_strtoul_threshold with dl_strtoul_consumes_all that replicates
_dl_strtoul's parsing and checks full consumption. Now accepts -1
(wraps to SIZE_MAX), 0, 0x20000, 010 (octal), and values above
i64::MAX.
The duplicate-= validation for GLIBC_TUNABLES (from round 1) is kept —
glibc's parse_tunables_string returns -1 if any entry's value contains
a duplicate =, rejecting the entire string.
Added dl_strtoul_consumes_all tests covering decimal, hex, octal,
negative, zero, empty, whitespace, trailing garbage, and sign-only
inputs. Updated usable_glibc_tunable_threshold tests to accept
negative, zero, and empty values. Verified: clippy clean and tests
pass on macOS (non-glibc); glibc-gated code type-checks and clippy
clean for x86_64-unknown-linux-gnu.
* Address round-4 review: add overflow detection, fix sign-only test assertions
Two Greptile P1 findings:
1. Overflowing tunables bypass threshold pinning: dl_strtoul_consumes_all
consumed every digit and returned true for values like
18446744073709551616 (u64::MAX + 1), but glibc's _dl_strtoul stops at
the overflowing digit (sets endptr there, returns UINT64_MAX), so
tunable_parse_num rejects the value (endptr != strval + len). Added
overflow detection matching glibc's cutoff/cutlim logic — on overflow,
the parser stops and returns false.
2. Sign-only parser assertions fail: the test asserted
!dl_strtoul_consumes_all("-") and !dl_strtoul_consumes_all("+"), but
_dl_strtoul skips the sign, finds no digit, sets endptr to the position
after the sign (== end of string), and returns 0. tunable_parse_num
sees endptr == strval + len → true. So glibc accepts sign-only strings
as value 0. Fixed the test assertions to expect true.
Also fixed "0x" with no hex digits: _dl_strtoul parses "0" as octal, then
stops at "x" (not an octal digit), so endptr != end of string → rejected.
The base-detection now requires a hex digit after "0x" before switching
to hex; otherwise "0" is parsed as octal and "x" stops the parser.
Added overflow regression tests: 18446744073709551616 (u64::MAX + 1),
99999999999999999999 (20 nines), 0x10000000000000000 (2^64). Verified:
clippy clean and tests pass on macOS (non-glibc); glibc-gated code
type-checks and clippy clean for x86_64-unknown-linux-gnu.
* Address round-5 review: accept bare 0x prefix, remove unused helper
Two review-bot findings (Devin BUG + CodeRabbit Major) on the same issue:
the round-4 fix required a hex digit after "0x" before switching to hex
base, but glibc's _dl_strtoul unconditionally advances past "0x"/"0X"
when the first char is '0' and the next is 'x'/'X' — even if no hex digit
follows. In that case the digit loop breaks immediately, endptr reaches
the end, and the value is 0. tunable_parse_num accepts it.
Removed the is_digit_in_base lookahead from the base-detection condition
and the now-unused is_digit_in_base helper. Updated the test assertions
for "0x" and "0X" to expect true (accepted as value 0).
The Greptile P1 overflow comment is invalid: glibc's _dl_strtoul rejects
18446744073709551616 (u64::MAX + 1) — on overflow it sets endptr to the
overflowing digit (not end of string) and returns UINT64_MAX, so
tunable_parse_num sees endptr != strval + len and rejects. My
implementation correctly returns false for this value, matching glibc.
Verified: clippy clean and tests pass on macOS (non-glibc); glibc-gated
code type-checks and clippy clean for x86_64-unknown-linux-gnu.
* Address round-6 review: rewrite tunable parser to match glibc exactly
Two Greptile P1 comments (3975151906, 3975151911) both invalid, but
investigation revealed a real bug in the split(':')-based parser:
Bug: usable_glibc_tunable_threshold used split(':') which loses the
distinction between an entry terminated by ':' (glibc skips it) and one
terminated by '\0' with no '=' (glibc rejects the entire string). Examples:
- "glibc.malloc.mmap_threshold=262144:glibc.cpu.x" (no '=' at end):
glibc rejects entire string, old code accepted it.
- "glibc.malloc.mmap_threshold=262144:" (trailing ':'):
glibc rejects entire string, old code accepted it.
Fix: replaced split(':') with a character-by-character parser matching
glibc's parse_tunables_string exactly. The parser tracks position in the
original string and correctly handles all three terminators ('=', ':', '\0')
for both name and value scanning.
Comment 3975151906 (near-maximum values): Invalid. Verified against
_dl_strtoul: for 18446744073709551615 (u64::MAX), cutoff = u64::MAX/10,
cutlim = u64::MAX%10 = 5. After 19 digits result == cutoff. 20th digit 5:
overflow check (digval > cutlim) is 5 > 5 = false → no overflow. glibc
accepts u64::MAX. Added regression test asserting it's accepted.
Comment 3975151911 (later malformed entry): Invalid. Verified against
parse_tunables (elf/dl-tunables.c): when parse_tunables_string returns -1,
parse_tunables prints a warning and returns immediately without applying
ANY tunable — including ones already parsed into the array. Added
regression test for "threshold=262144:check=2=2" (threshold before
malformed sibling) asserting it's rejected.
Added regression tests: u64::MAX accepted, threshold-before-malformed
rejected, no-'=' at end rejected, trailing ':' rejected, leading ':'
accepted. Verified: clippy clean and tests pass on macOS; glibc-gated
code type-checks and clippy clean for x86_64-unknown-linux-gnu.
* Fix CI: correct hex trailing-garbage test assertion
The test asserted !dl_strtoul_consumes_all("0x20000abc"), but in hex
mode a-f are valid digits — "0x20000abc" is a valid hex number
(0x20000abc = 536874044), not trailing garbage. _dl_strtoul consumes
the entire string and tunable_parse_num accepts it. The assertion
failed on Linux CI where the glibc-gated test actually runs.
Replaced with "0x20000g" — 'g' is not a hex digit, so _dl_strtoul
stops at 'g' and tunable_parse_num rejects the value.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
This commit is contained in:
Eliah Rusin
co-authored by
Claude Opus 5
Chris Lu
Claude Opus 5
Chris Lu
parent
e919bec9d1
commit
4f9bbd51cb
@@ -1,5 +1,6 @@
|
||||
pub mod config;
|
||||
pub mod images;
|
||||
pub mod malloc_tuning;
|
||||
pub mod metrics;
|
||||
pub mod remote_storage;
|
||||
pub mod security;
|
||||
|
||||
@@ -39,6 +39,11 @@ const GRPC_MAX_HEADER_LIST_SIZE: u32 = 8 * 1024 * 1024;
|
||||
const GRPC_MAX_CONCURRENT_STREAMS: u32 = 1000;
|
||||
|
||||
fn main() {
|
||||
// Before anything allocates: stop glibc from training its mmap threshold
|
||||
// upward on our large EC buffers and turning them into heap it never
|
||||
// returns. See seaweed_volume::malloc_tuning for the measurements.
|
||||
let malloc_tuning = seaweed_volume::malloc_tuning::pin_mmap_threshold();
|
||||
|
||||
install_default_crypto_provider();
|
||||
|
||||
// Initialize tracing
|
||||
@@ -65,6 +70,19 @@ fn main() {
|
||||
"SeaweedFS Volume Server (Rust) v{}",
|
||||
seaweed_volume::version::full_version()
|
||||
);
|
||||
match malloc_tuning {
|
||||
seaweed_volume::malloc_tuning::MallocTuning::Pinned(bytes) => {
|
||||
info!("pinned glibc M_MMAP_THRESHOLD to {} bytes", bytes)
|
||||
}
|
||||
seaweed_volume::malloc_tuning::MallocTuning::DeferredToEnv => info!(
|
||||
"an allocator mmap-threshold override ({}) is set; leaving glibc's mmap threshold to the environment",
|
||||
seaweed_volume::malloc_tuning::MMAP_THRESHOLD_ENV
|
||||
),
|
||||
seaweed_volume::malloc_tuning::MallocTuning::Failed => {
|
||||
warn!("mallopt(M_MMAP_THRESHOLD) failed; large freed buffers may stay resident")
|
||||
}
|
||||
seaweed_volume::malloc_tuning::MallocTuning::NotApplicable => {}
|
||||
}
|
||||
|
||||
// Register Prometheus metrics
|
||||
metrics::register_metrics();
|
||||
|
||||
@@ -0,0 +1,467 @@
|
||||
//! Keep glibc from silently converting large short-lived buffers into heap the
|
||||
//! process never gives back.
|
||||
//!
|
||||
//! glibc serves an allocation with `mmap` when it is at least
|
||||
//! `M_MMAP_THRESHOLD` (128 KiB by default), and `munmap`s it on free, so the
|
||||
//! pages go straight back to the OS. That threshold is **adaptive**: whenever a
|
||||
//! block that came from `mmap` is freed, glibc raises the threshold to that
|
||||
//! block's size — up to 32 MiB — on the theory that a workload repeatedly
|
||||
//! allocating buffers of that size is better served from the heap.
|
||||
//!
|
||||
//! For a volume server that theory is wrong in a specific, expensive way. EC
|
||||
//! reconstruction and needle reassembly allocate large, short-lived buffers.
|
||||
//! The first few are mmap'd and freed, which trains the threshold upward; every
|
||||
//! later buffer of that size is then carved out of the heap instead. Heap
|
||||
//! memory is only returned to the OS from the top of the arena, so those pages
|
||||
//! stay resident as anonymous memory for the life of the process. They are
|
||||
//! still *reusable* — this is not a leak, and a repeat workload does not grow
|
||||
//! the footprint further — but under a hard cgroup `MemoryMax` they are
|
||||
//! indistinguishable from a leak, because anonymous pages cannot be reclaimed
|
||||
//! under pressure the way page cache can. The retained footprint eats exactly
|
||||
//! the headroom that a burst of maintenance work needs, and the process is
|
||||
//! OOM-killed while most of its resident memory is free-but-unreturned.
|
||||
//!
|
||||
//! Measured on a 17-node cluster (EC 10+4, `--index=redb`), one node, two
|
||||
//! identical `ec.scrub -mode full` rounds over 10912 EC files each, comparing
|
||||
//! the same unit restarted with and without a pinned threshold:
|
||||
//!
|
||||
//! | | baseline | round 1 | round 2 | 60s idle |
|
||||
//! |---|---|---|---|---|
|
||||
//! | default (adaptive) | 10 MB | 84 MB | 88 MB | **88 MB** |
|
||||
//! | pinned threshold | 10 MB | 13 MB | 14 MB | **14 MB** |
|
||||
//!
|
||||
//! 78 MB retained versus 4 MB for identical work. On that cluster's heavier
|
||||
//! mixed scrub workloads the same effect reached ~600 MB of retained anonymous
|
||||
//! memory per volume server, against a 3 GiB cap.
|
||||
//!
|
||||
//! Calling `mallopt(M_MMAP_THRESHOLD, ...)` sets the threshold *and* disables
|
||||
//! the dynamic adjustment, which is the documented behaviour of setting it
|
||||
//! explicitly. We pin it to glibc's own default rather than inventing a value:
|
||||
//! the goal is to stop the adaptation, not to second-guess the default.
|
||||
|
||||
/// glibc's own default `M_MMAP_THRESHOLD`. Pinning to this value changes
|
||||
/// nothing about which allocations use `mmap` on a freshly started process; it
|
||||
/// only prevents the threshold from drifting upward later.
|
||||
#[cfg(all(target_os = "linux", target_env = "gnu"))]
|
||||
const DEFAULT_MMAP_THRESHOLD: libc::c_int = 128 * 1024;
|
||||
|
||||
/// Legacy environment variable glibc reads for the same setting. If an operator
|
||||
/// has set it, honour their value and do not override it.
|
||||
pub const MMAP_THRESHOLD_ENV: &str = "MALLOC_MMAP_THRESHOLD_";
|
||||
|
||||
/// Modern glibc tunables environment variable. Operators may set the threshold
|
||||
/// via `GLIBC_TUNABLES=glibc.malloc.mmap_threshold=...` instead of the legacy
|
||||
/// variable; that override is honoured too.
|
||||
pub const GLIBC_TUNABLES_ENV: &str = "GLIBC_TUNABLES";
|
||||
|
||||
/// The tunable name within `GLIBC_TUNABLES` that maps to `M_MMAP_THRESHOLD`.
|
||||
#[cfg(all(target_os = "linux", target_env = "gnu"))]
|
||||
const MMAP_THRESHOLD_TUNABLE: &str = "glibc.malloc.mmap_threshold";
|
||||
|
||||
/// Outcome of the tuning attempt, so the caller can log it and tests can assert
|
||||
/// on it without inspecting global allocator state.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum MallocTuning {
|
||||
/// Threshold pinned to `DEFAULT_MMAP_THRESHOLD`; dynamic adjustment is off.
|
||||
Pinned(i32),
|
||||
/// An allocator override (`MALLOC_MMAP_THRESHOLD_` or
|
||||
/// `GLIBC_TUNABLES=glibc.malloc.mmap_threshold=...`) was set, so the
|
||||
/// operator's value wins.
|
||||
DeferredToEnv,
|
||||
/// `mallopt` reported failure. Not fatal — the server runs, it just keeps
|
||||
/// glibc's adaptive behaviour.
|
||||
Failed,
|
||||
/// Not glibc, so there is no adaptive threshold to pin.
|
||||
NotApplicable,
|
||||
}
|
||||
|
||||
/// Pin glibc's mmap threshold unless the operator has set an allocator override.
|
||||
/// Safe to call more than once; call it before serving traffic, since the point
|
||||
/// is to prevent the threshold from being trained upward by early allocations.
|
||||
pub fn pin_mmap_threshold() -> MallocTuning {
|
||||
pin_mmap_threshold_inner()
|
||||
}
|
||||
|
||||
#[cfg(all(target_os = "linux", target_env = "gnu"))]
|
||||
fn pin_mmap_threshold_inner() -> MallocTuning {
|
||||
if operator_mmap_threshold_override_active() {
|
||||
return MallocTuning::DeferredToEnv;
|
||||
}
|
||||
// SAFETY: `mallopt` is a libc entry point that takes two ints and mutates
|
||||
// only allocator-internal tunables. It has no preconditions and no effect
|
||||
// on memory this process already owns.
|
||||
let rc = unsafe { libc::mallopt(libc::M_MMAP_THRESHOLD, DEFAULT_MMAP_THRESHOLD) };
|
||||
if rc == 1 {
|
||||
MallocTuning::Pinned(DEFAULT_MMAP_THRESHOLD)
|
||||
} else {
|
||||
MallocTuning::Failed
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(not(all(target_os = "linux", target_env = "gnu")))]
|
||||
fn pin_mmap_threshold_inner() -> MallocTuning {
|
||||
MallocTuning::NotApplicable
|
||||
}
|
||||
|
||||
#[cfg(all(target_os = "linux", target_env = "gnu"))]
|
||||
fn operator_mmap_threshold_override_active() -> bool {
|
||||
// MALLOC_MMAP_THRESHOLD_: glibc calls atoi(value) then mallopt, which
|
||||
// always sets the threshold and disables dynamic adjustment — even for
|
||||
// empty, negative, or non-numeric values (atoi returns 0). So any presence
|
||||
// of the variable means the operator's override is in effect.
|
||||
std::env::var_os(MMAP_THRESHOLD_ENV).is_some()
|
||||
|| usable_glibc_tunable_threshold(std::env::var_os(GLIBC_TUNABLES_ENV))
|
||||
}
|
||||
|
||||
/// Look for `glibc.malloc.mmap_threshold=<value>` among the colon-separated
|
||||
/// tunables in `GLIBC_TUNABLES`. glibc's `parse_tunables_string` (elf/dl-tunables.c)
|
||||
/// rejects the **entire** string (returns -1) if it reaches `\0` before finding
|
||||
/// `=` in a name (last entry has no `=`), or if any entry's value contains a
|
||||
/// duplicate `=`. When `parse_tunables_string` returns -1, `parse_tunables`
|
||||
/// prints a warning and returns immediately without applying ANY tunable —
|
||||
/// including ones already parsed into the tunables array. We match that by
|
||||
/// returning `false` for the entire string on any of those conditions.
|
||||
///
|
||||
/// glibc parses tunable values with `_dl_strtoul`, which accepts decimal,
|
||||
/// `0x` hex, `0` octal, an optional sign (negatives wrap to `unsigned long`),
|
||||
/// and requires the entire value to be consumed; we match that with
|
||||
/// `dl_strtoul_consumes_all`.
|
||||
#[cfg(all(target_os = "linux", target_env = "gnu"))]
|
||||
fn usable_glibc_tunable_threshold(tunables: Option<std::ffi::OsString>) -> bool {
|
||||
let s = match tunables.and_then(|v| v.into_string().ok()) {
|
||||
Some(s) => s,
|
||||
None => return false,
|
||||
};
|
||||
if s.is_empty() {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Parse the string character-by-character, matching glibc's
|
||||
// parse_tunables_string logic exactly. Using split(':') would lose the
|
||||
// distinction between an entry terminated by ':' (skip) and one terminated
|
||||
// by '\0' with no '=' (reject entire string).
|
||||
let bytes = s.as_bytes();
|
||||
let mut pos = 0;
|
||||
let mut found_threshold = false;
|
||||
|
||||
loop {
|
||||
// Find where the name ends ('=', ':', or end of string).
|
||||
let name_start = pos;
|
||||
while pos < bytes.len() && bytes[pos] != b'=' && bytes[pos] != b':' {
|
||||
pos += 1;
|
||||
}
|
||||
|
||||
// End of string before '=' → glibc returns -1 (reject entire string).
|
||||
if pos >= bytes.len() {
|
||||
return false;
|
||||
}
|
||||
|
||||
// ':' before '=' → glibc skips this entry and continues.
|
||||
if bytes[pos] == b':' {
|
||||
pos += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Skip the '='.
|
||||
let name_end = pos;
|
||||
pos += 1;
|
||||
|
||||
// Find where the value ends ('=', ':', or end of string).
|
||||
let val_start = pos;
|
||||
while pos < bytes.len() && bytes[pos] != b'=' && bytes[pos] != b':' {
|
||||
pos += 1;
|
||||
}
|
||||
|
||||
// '=' in value → glibc returns -1 (reject entire string).
|
||||
if pos < bytes.len() && bytes[pos] == b'=' {
|
||||
return false;
|
||||
}
|
||||
|
||||
let key = &s[name_start..name_end];
|
||||
let val = &s[val_start..pos];
|
||||
|
||||
if key == MMAP_THRESHOLD_TUNABLE && dl_strtoul_consumes_all(val) {
|
||||
found_threshold = true;
|
||||
}
|
||||
|
||||
// End of string → done.
|
||||
if pos >= bytes.len() {
|
||||
break;
|
||||
}
|
||||
|
||||
// Skip the ':'.
|
||||
pos += 1;
|
||||
}
|
||||
|
||||
found_threshold
|
||||
}
|
||||
|
||||
/// Replicate glibc's `_dl_strtoul` (elf/dl-misc.c) just enough to determine
|
||||
/// whether it would consume the entire string — which is what
|
||||
/// `tunable_parse_num` checks (`endptr == strval + len`). Returns `true` if
|
||||
/// glibc would accept the value and apply it.
|
||||
///
|
||||
/// `_dl_strtoul` skips leading spaces/tabs, accepts an optional `+`/`-` sign,
|
||||
/// and parses `0x`-prefixed hex, `0`-prefixed octal, or plain decimal. A
|
||||
/// negative result wraps to `unsigned long` (`-1` → `SIZE_MAX`). If no digit is
|
||||
/// found after the sign, the end pointer stays at the current position — which
|
||||
/// still counts as "consumed" when the string is empty or whitespace-only
|
||||
/// (value 0). On overflow, `_dl_strtoul` stops at the overflowing digit (endptr
|
||||
/// does not reach the end), so `tunable_parse_num` rejects the value.
|
||||
#[cfg(all(target_os = "linux", target_env = "gnu"))]
|
||||
fn dl_strtoul_consumes_all(s: &str) -> bool {
|
||||
let bytes = s.as_bytes();
|
||||
let mut pos = 0;
|
||||
|
||||
// Skip leading whitespace (spaces and tabs, matching _dl_strtoul).
|
||||
while pos < bytes.len() && (bytes[pos] == b' ' || bytes[pos] == b'\t') {
|
||||
pos += 1;
|
||||
}
|
||||
|
||||
// Optional sign.
|
||||
if pos < bytes.len() && (bytes[pos] == b'-' || bytes[pos] == b'+') {
|
||||
pos += 1;
|
||||
}
|
||||
|
||||
// Must have at least one digit (0-9) to start parsing, unless we're already
|
||||
// at the end (empty / whitespace-only / sign-only → value 0, consumed).
|
||||
if pos >= bytes.len() {
|
||||
return true;
|
||||
}
|
||||
if bytes[pos] < b'0' || bytes[pos] > b'9' {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Determine base: 0x → hex, 0 → octal, else decimal. _dl_strtoul unconditionally
|
||||
// advances past "0x"/"0X" when the first char is '0' and the next is 'x'/'X',
|
||||
// even if no hex digit follows — in that case the digit loop breaks immediately,
|
||||
// endptr reaches the end, and the value is 0.
|
||||
let base: u32 = if bytes[pos] == b'0'
|
||||
&& pos + 1 < bytes.len()
|
||||
&& (bytes[pos + 1] == b'x' || bytes[pos + 1] == b'X')
|
||||
{
|
||||
pos += 2; // skip "0x"
|
||||
16
|
||||
} else if bytes[pos] == b'0' {
|
||||
8
|
||||
} else {
|
||||
10
|
||||
};
|
||||
|
||||
// Parse digits with overflow detection, matching _dl_strtoul's cutoff/cutlim
|
||||
// logic. On overflow, _dl_strtoul sets endptr to the overflowing digit and
|
||||
// returns UINT64_MAX — so the value is NOT fully consumed and
|
||||
// tunable_parse_num rejects it.
|
||||
let mut result: u64 = 0;
|
||||
let cutoff = u64::MAX / base as u64;
|
||||
let cutlim = u64::MAX % base as u64;
|
||||
|
||||
while pos < bytes.len() {
|
||||
let b = bytes[pos];
|
||||
let digval: u32 = match digit_value(b, base) {
|
||||
Some(v) => v,
|
||||
None => break,
|
||||
};
|
||||
if result > cutoff || (result == cutoff && digval as u64 > cutlim) {
|
||||
// Overflow: _dl_strtoul stops here, endptr points at this digit.
|
||||
return false;
|
||||
}
|
||||
result *= base as u64;
|
||||
result += digval as u64;
|
||||
pos += 1;
|
||||
}
|
||||
|
||||
// The entire string must be consumed (matching tunable_parse_num's check).
|
||||
pos == bytes.len()
|
||||
}
|
||||
|
||||
/// Returns the numeric value of a digit byte in the given base, or `None` if
|
||||
/// the byte is not a valid digit in that base.
|
||||
#[cfg(all(target_os = "linux", target_env = "gnu"))]
|
||||
fn digit_value(b: u8, base: u32) -> Option<u32> {
|
||||
if (b'0'..=b'0' + (base - 1).min(9) as u8).contains(&b) {
|
||||
return Some((b - b'0') as u32);
|
||||
}
|
||||
if base == 16 {
|
||||
if (b'a'..=b'f').contains(&b) {
|
||||
return Some((b - b'a' + 10) as u32);
|
||||
}
|
||||
if (b'A'..=b'F').contains(&b) {
|
||||
return Some((b - b'A' + 10) as u32);
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn env_override_constants_match_glibc_names() {
|
||||
// Verified against the real accessor rather than a copy of the name, so
|
||||
// renaming the constant cannot silently break the override contract.
|
||||
assert_eq!(MMAP_THRESHOLD_ENV, "MALLOC_MMAP_THRESHOLD_");
|
||||
assert_eq!(GLIBC_TUNABLES_ENV, "GLIBC_TUNABLES");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn calling_twice_is_stable() {
|
||||
// Startup paths get re-entered in tests and in `weed mini`; the second
|
||||
// call must not report a different outcome from the first.
|
||||
let first = pin_mmap_threshold();
|
||||
let second = pin_mmap_threshold();
|
||||
assert_eq!(first, second);
|
||||
}
|
||||
|
||||
#[cfg(all(target_os = "linux", target_env = "gnu"))]
|
||||
#[test]
|
||||
fn pins_threshold_on_glibc_when_no_override_is_set() {
|
||||
// The env override is not set in the test process, so this exercises the
|
||||
// mallopt path. If an override happens to be present, defer to it.
|
||||
if operator_mmap_threshold_override_active() {
|
||||
assert_eq!(pin_mmap_threshold(), MallocTuning::DeferredToEnv);
|
||||
return;
|
||||
}
|
||||
assert_eq!(
|
||||
pin_mmap_threshold(),
|
||||
MallocTuning::Pinned(DEFAULT_MMAP_THRESHOLD),
|
||||
"mallopt(M_MMAP_THRESHOLD) should succeed on glibc"
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(not(all(target_os = "linux", target_env = "gnu")))]
|
||||
#[test]
|
||||
fn is_a_noop_off_glibc() {
|
||||
// No glibc adaptive threshold exists off glibc, so there is nothing to
|
||||
// pin regardless of any environment variables that happen to be set.
|
||||
assert_eq!(pin_mmap_threshold(), MallocTuning::NotApplicable);
|
||||
}
|
||||
|
||||
#[cfg(all(target_os = "linux", target_env = "gnu"))]
|
||||
#[test]
|
||||
fn dl_strtoul_consumes_all_matches_glibc_parser() {
|
||||
// Decimal — any non-empty decimal integer is accepted, including
|
||||
// negative (wraps to unsigned) and zero.
|
||||
assert!(dl_strtoul_consumes_all("131072"));
|
||||
assert!(dl_strtoul_consumes_all("0"));
|
||||
assert!(dl_strtoul_consumes_all("-1"));
|
||||
assert!(dl_strtoul_consumes_all("-131072"));
|
||||
// Values above i64::MAX are valid for glibc's unsigned parser.
|
||||
assert!(dl_strtoul_consumes_all("9223372036854775808"));
|
||||
// Hex with 0x prefix.
|
||||
assert!(dl_strtoul_consumes_all("0x20000"));
|
||||
assert!(dl_strtoul_consumes_all("0X20000"));
|
||||
assert!(dl_strtoul_consumes_all("0x0"));
|
||||
// Octal with leading 0.
|
||||
assert!(dl_strtoul_consumes_all("010"));
|
||||
// Leading whitespace (spaces and tabs) is skipped.
|
||||
assert!(dl_strtoul_consumes_all(" 131072"));
|
||||
assert!(dl_strtoul_consumes_all("\t0x20000"));
|
||||
// Empty and whitespace-only strings are accepted (value 0).
|
||||
assert!(dl_strtoul_consumes_all(""));
|
||||
assert!(dl_strtoul_consumes_all(" "));
|
||||
assert!(dl_strtoul_consumes_all("\t"));
|
||||
// Sign-only strings are accepted: _dl_strtoul skips the sign, finds no
|
||||
// digit, sets endptr to the position after the sign (== end of string),
|
||||
// and returns 0. tunable_parse_num sees endptr == strval + len → true.
|
||||
assert!(dl_strtoul_consumes_all("-"));
|
||||
assert!(dl_strtoul_consumes_all("+"));
|
||||
|
||||
// Trailing garbage is rejected — _dl_strtoul stops at the first
|
||||
// non-digit and tunable_parse_num requires the entire string consumed.
|
||||
assert!(!dl_strtoul_consumes_all("131072abc"));
|
||||
// In hex mode, a-f are digits, so "0x20000abc" is a valid hex number.
|
||||
// Use a non-hex character like 'g' to test trailing garbage in hex.
|
||||
assert!(!dl_strtoul_consumes_all("0x20000g"));
|
||||
assert!(!dl_strtoul_consumes_all("128K"));
|
||||
// Non-numeric strings are rejected.
|
||||
assert!(!dl_strtoul_consumes_all("abc"));
|
||||
// "0x" with no hex digits: _dl_strtoul advances past "0x", the digit loop
|
||||
// breaks immediately (no hex digit), endptr reaches the end, value is 0.
|
||||
// tunable_parse_num accepts it.
|
||||
assert!(dl_strtoul_consumes_all("0x"));
|
||||
assert!(dl_strtoul_consumes_all("0X"));
|
||||
|
||||
// Overflow: _dl_strtoul stops at the overflowing digit (endptr points
|
||||
// there, not at the end), so tunable_parse_num rejects the value.
|
||||
assert!(!dl_strtoul_consumes_all("18446744073709551616")); // u64::MAX + 1
|
||||
assert!(!dl_strtoul_consumes_all("99999999999999999999")); // 20 nines
|
||||
assert!(!dl_strtoul_consumes_all("0x10000000000000000")); // 2^64
|
||||
// u64::MAX itself is accepted: the last digit (5) equals cutlim (=5),
|
||||
// so the overflow check (digval > cutlim) is false.
|
||||
assert!(dl_strtoul_consumes_all("18446744073709551615")); // u64::MAX
|
||||
}
|
||||
|
||||
#[cfg(all(target_os = "linux", target_env = "gnu"))]
|
||||
#[test]
|
||||
fn usable_glibc_tunable_threshold_detects_mmap_threshold() {
|
||||
// Decimal, hex, octal, negative, and zero values are all accepted by
|
||||
// glibc's _dl_strtoul and cause the threshold to be pinned.
|
||||
assert!(usable_glibc_tunable_threshold(Some(
|
||||
"glibc.malloc.mmap_threshold=131072".into()
|
||||
)));
|
||||
assert!(usable_glibc_tunable_threshold(Some(
|
||||
"glibc.malloc.mmap_threshold=0x20000".into()
|
||||
)));
|
||||
assert!(usable_glibc_tunable_threshold(Some(
|
||||
"glibc.malloc.mmap_threshold=0".into()
|
||||
)));
|
||||
assert!(usable_glibc_tunable_threshold(Some(
|
||||
"glibc.malloc.mmap_threshold=-1".into()
|
||||
)));
|
||||
assert!(usable_glibc_tunable_threshold(Some(
|
||||
"glibc.malloc.mmap_threshold=9223372036854775808".into()
|
||||
)));
|
||||
// u64::MAX is accepted by _dl_strtoul (last digit == cutlim, no overflow).
|
||||
assert!(usable_glibc_tunable_threshold(Some(
|
||||
"glibc.malloc.mmap_threshold=18446744073709551615".into()
|
||||
)));
|
||||
// Appears alongside other tunables.
|
||||
assert!(usable_glibc_tunable_threshold(Some(
|
||||
"glibc.cpu.x=1:glibc.malloc.mmap_threshold=131072".into()
|
||||
)));
|
||||
// Leading ':' is accepted — glibc skips the empty entry and continues.
|
||||
assert!(usable_glibc_tunable_threshold(Some(
|
||||
":glibc.malloc.mmap_threshold=131072".into()
|
||||
)));
|
||||
// Empty value is accepted by _dl_strtoul (value 0).
|
||||
assert!(usable_glibc_tunable_threshold(Some(
|
||||
"glibc.malloc.mmap_threshold=".into()
|
||||
)));
|
||||
|
||||
// Non-numeric values are rejected by _dl_strtoul.
|
||||
assert!(!usable_glibc_tunable_threshold(Some(
|
||||
"glibc.malloc.mmap_threshold=abc".into()
|
||||
)));
|
||||
assert!(!usable_glibc_tunable_threshold(Some(
|
||||
"glibc.malloc.mmap_threshold=128K".into()
|
||||
)));
|
||||
// A malformed sibling entry (duplicate '=') makes glibc reject the
|
||||
// entire string, so we must not accept the threshold entry either.
|
||||
// This applies regardless of whether the threshold is before or after
|
||||
// the malformed entry — parse_tunables_string returns -1, and
|
||||
// parse_tunables discards all tunables without applying any.
|
||||
assert!(!usable_glibc_tunable_threshold(Some(
|
||||
"glibc.malloc.check=2=2:glibc.malloc.mmap_threshold=131072".into()
|
||||
)));
|
||||
assert!(!usable_glibc_tunable_threshold(Some(
|
||||
"glibc.malloc.mmap_threshold=262144:glibc.malloc.check=2=2".into()
|
||||
)));
|
||||
// A trailing entry with no '=' makes glibc reject the entire string
|
||||
// (parse_tunables_string hits '\0' before '=' and returns -1).
|
||||
assert!(!usable_glibc_tunable_threshold(Some(
|
||||
"glibc.malloc.mmap_threshold=262144:glibc.cpu.x".into()
|
||||
)));
|
||||
// A trailing ':' makes glibc reject the entire string (the empty entry
|
||||
// after ':' hits '\0' before '=' and returns -1).
|
||||
assert!(!usable_glibc_tunable_threshold(Some(
|
||||
"glibc.malloc.mmap_threshold=262144:".into()
|
||||
)));
|
||||
// Unrelated tunables do not count.
|
||||
assert!(!usable_glibc_tunable_threshold(Some(
|
||||
"glibc.cpu.x=1".into()
|
||||
)));
|
||||
assert!(!usable_glibc_tunable_threshold(None));
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user