Sources are now metrics endpoints rather than component names, so
cluster-wide aggregates match on any source and let the metric name
identify the component. Detail pages iterate the scrape endpoints and
only list one when it actually serves that component's series, which
keeps a combined "weed server" from appearing as four separate nodes.
The empty-state notice now tells operators to start servers with
-metricsPort, since that is what creates the listener.
Under-replicated, writable and crowded volume counts are cluster-wide
values that only the leader maintains. Summing them across masters
double-counted them, and a demoted master keeps serving its last values,
so a leadership change could inflate the health alerts.
Series were paired positionally, so a failed scrape, a missing metric, a
counter reset or a histogram interval with no observations shifted a
series and plotted unrelated samples at the same instant. Samples now
carry their scrape time, charts lay every series out on a shared time
axis, and gaps break the line rather than being drawn through.
SeaweedFS latency quantiles are frequently sub-millisecond, so a fixed
millisecond format rendered every axis tick as '0 ms'. Use µs below 1 ms
and add decimals below 10 ms.
The request counters label "type" with the HTTP method, so filtering on
type=read/write matched nothing and left the throughput chart empty. A
live cluster reports type=GET/POST.
Links the six monitoring pages, with the active link highlighted. The
generated diff is large because inserting a nav section renumbers templ's
string literals.
Six server-rendered pages (overview plus volume server, filer, S3,
master and worker detail) built from shared chartCard and statCard
components, following the existing Bootstrap card and table conventions.
Charts are inline SVG, so no chart library is pulled in. Pages show a
'no metrics yet' notice until the first scrapes land.
Templates live in the app package, so the chart types and renderer must
be exported. Also anchors the y-axis at zero, left-pads shorter series so
multi-series charts stay aligned, escapes text nodes, renders a
'collecting data' placeholder before enough samples exist, and adds a
seconds unit for latency quantiles.
Aggregates the scraped series into per-page structs: overview health,
throughput, latency, errors and maintenance, plus per-server detail for
volume servers, filers, S3, masters and workers. Rates and counts are
summed across servers; latency quantiles take the worst server, since
quantiles cannot be summed. Disk usage is derived from the volume server
resource gauge's used/all mounts.
Series now carry their source, metric name, and labels so callers can
filter by label (error codes, slot state, disk resource type) instead of
string-matching composite keys. Drops the unused single-series accessors.
Replaces scraping each node's service port with the dedicated Prometheus
listener each node now advertises. Nodes started without -metricsPort
advertise 0 and are skipped, so nothing is ever fetched from a
client-facing port.
Endpoints are deduplicated by address because a combined "weed server"
advertises one listener for all of its components; series are attributed
by metric name, which already identifies the component.
Raw counters and histograms are process-lifetime cumulative, so
charting them directly is meaningless. Counters now become per-second
rates and histograms become p50/p95/p99, both computed from the delta
against the previous scrape, with counter resets skipped. Quantiles use
linear interpolation within the matching bucket, as Prometheus does.
The admin's maintenance and worker metrics live in the local
stats.Gather registry, so record them directly under the admin/local
source instead of scraping over HTTP. Adds metricsStore.match for
prefix/metric lookups.
Extends the existing sparklineSVG approach into a full chart helper
with axes, gridlines, multi-series lines/areas, legends, threshold
lines, and unit formatters (bytes, bps, ms, pct). No JS chart library;
safe to inline in templ pages.
Adds a 15s scrape loop that fetches /metrics from every discovered
master, volume, filer, and S3 server (addresses come from the
existing topology + ListClusterNodes helpers) and records each
series into the in-memory store. Parses Prometheus text exposition
via prometheus/common/expfmt.
Bounded ring buffer (240 samples) keyed by source/metric[/labels],
reusing the dashSample ring pattern. No persistence; powers the
upcoming monitoring charts.
weed mini builds MasterOptions directly and never set metricsHttpPort, so
reading it in toMasterOption dereferenced nil and crashed startup. Point
mini's master, volume and filer at its single -metricsPort listener, the
same way weed server does, and treat an unset port as disabled so a
partially initialised MasterOptions cannot panic again.
Reproduced with 'weed mini -dir=... -s3.port=...', which is what the S3
filer-group and delete-regression suites start.
Filer, S3 and broker report it on the KeepConnected registration, volume
servers in their heartbeat, and masters return their own in
GetMasterConfiguration. MasterClient gains SetMetricsPort so the eight
callers that have no metrics listener are untouched, and the volume
server takes it as a constructor argument because its heartbeat goroutine
starts there.
In combined "weed server" one metrics listener serves the whole shared
registry, so every component advertises the same port.
Each server already has a -metricsPort Prometheus listener, but nothing
advertises it, so a central scraper cannot find it. Adds metrics_port to
KeepConnectedRequest and ListClusterNodes (filer, S3, broker), Heartbeat
and DataNodeInfo (volume servers), and GetMasterConfigurationResponse
(masters).
Note the existing metrics_address fields are unrelated: they carry the
Prometheus push gateway that servers push to, not a scrape target.
* s3: preserve versions for POST policy uploads
Route POST policy uploads through the existing version-aware write helpers
and validate promoted Object Lock headers before writing.
Return the generated version ID when versioning is enabled, return
x-amz-version-id: null when versioning is suspended, and omit the header
when versioning has never been enabled.
* s3: reuse versioning helpers in POST policy handler
Route the POST policy handler through the existing getVersioningState
and isObjectLockEnabled helpers instead of open-coding the object-lock
forces-versioning-enabled rule, matching the PUT path.
Drop the x-amz-version-id: null response header for suspended
versioning; the PUT handler omits it and the S3 PutObject sample
response for suspended buckets does not include it. Trim the moved
fileSize comment.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* build(deps): bump github.com/redis/go-redis/v9 from 9.21.0 to 9.22.0
Bumps [github.com/redis/go-redis/v9](https://github.com/redis/go-redis) from 9.21.0 to 9.22.0.
- [Release notes](https://github.com/redis/go-redis/releases)
- [Changelog](https://github.com/redis/go-redis/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/redis/go-redis/compare/v9.21.0...v9.22.0)
---
updated-dependencies:
- dependency-name: github.com/redis/go-redis/v9
dependency-version: 9.22.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* test(redis_conf): track go-redis 9.22.0 default read timeout of 5s
go-redis 9.22.0 raised the default ReadTimeout from 3s to 5s (part of the
cross-SDK configuration alignment). Update TestUnsetKeepsGoRedisDefaults to
expect the new default so the bump in #11306 stops failing CI.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* helm: grant List to the generated read-only S3 identity
The chart's anvReadOnly identity only carried the Read action, so its
credentials could GetObject and HeadObject but every ListObjects request
was denied: List is a separate action and the identity check is an exact
match. Add List so the read-only credentials can list buckets and objects.
Writes stay denied. Update the README example to match.
Bump the chart to 4.47.1. The chart label is part of the s3 and
all-in-one pod templates, so the upgrade rolls the gateways and they
reload the identity config, which is only read at startup.
Fixes#11317
* helm: roll standalone S3 and all-in-one on s3 config changes
Mirror the filer checksum/s3config pod annotation in the standalone S3
and all-in-one deployments so a changed generated S3 secret triggers a
rollout during a normal helm upgrade without relying on a chart version
bump.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: populate Initiated timestamp in ListMultipartUploads
ListMultipartUploads returned each upload with only Key and UploadId,
omitting the Initiated timestamp. Clients such as GeeseFS rely on this
field to expire stale uploads and crash on its absence. Set Initiated
from the upload directory entry creation time so repeated listings
preserve the original initiation time.
* test/s3: verify Initiated timestamp in ListMultipartUploads
Add an integration test that initiates a multipart upload, lists it,
and asserts the Initiated field is populated and preserved across
repeated listings rather than reflecting the listing time.
* admin: extract isFlagExplicitlySet helper from applyViperFallback
No behavior change; the inline flag-visit check becomes a reusable
helper so the upcoming bind-address default can share it.
* admin: default to 0.0.0.0 for authenticated HTTP, keep worker gRPC on loopback
PR #11185 made the admin HTTP server default to 127.0.0.1 to stop
exposing the unauthenticated admin API on the network by accident.
That also locked out operators who already authenticated with
-adminPassword: their UI became unreachable from the network after
upgrade unless they added -ip=0.0.0.0 (see #11303).
An authenticated deployment is safe to expose, so auto-upgrade the -ip
default to 0.0.0.0 when -adminPassword or [https.admin] mTLS is
configured. The loopback default stays for the unauthenticated case, so
the unauthenticated API is never exposed on the network. An explicit
-ip is always honored.
The worker gRPC control plane has no password auth (only mTLS), so it
must not follow the HTTP upgrade. Give it a separate bind address that
stays on loopback unless -ip is explicit, so adminPassword no longer
re-exposes the unauthenticated worker stream.
* admin: hint loopback-only bind in startup banner
When the admin server binds to loopback (the default for the
unauthenticated case), print a one-line hint that it is not reachable
from other hosts and how to expose it. This helps operators who, after
the #11185 loopback default, can no longer reach the UI from another
machine quickly see the cause and the fix without reading the docs.
* admin: keep worker gRPC on loopback, decouple from https.admin mTLS
The worker gRPC auto-upgrade to 0.0.0.0 was gated on hasMTLS, which
reads the https.admin (HTTP) mTLS config. The worker gRPC mTLS comes
from grpc.admin + grpc.ca, a separate config, so:
- https.admin mTLS without grpc.admin mTLS widened the worker gRPC to
0.0.0.0 unauthenticated (re-exposing the control plane), and
- grpc.admin mTLS without https.admin mTLS left the worker gRPC on
loopback, blocking authenticated remote workers.
Drop the worker gRPC auto-upgrade entirely. The worker gRPC keeps the
raw -ip value (loopback by default), matching the pre-existing
behavior; an operator who wants remote workers sets -ip explicitly.
Only the HTTP admin listener auto-upgrades to 0.0.0.0 when
authenticated.
Addresses review feedback on #11314 from Devin and Greptile.
* s3api: add TrustedProxies allowlist helper for aws:SourceIp extraction
Introduces a policy_engine.TrustedProxies type that parses a
comma-separated list of bare IPs and CIDRs (mirroring Guard.UpdateWhiteList)
and extracts the client IP for aws:SourceIp condition evaluation.
When the direct TCP peer is in the allowlist, X-Forwarded-For is walked
right-to-left skipping trusted hops (then X-Real-Ip); otherwise the direct
peer address is returned. This is the building block for restoring
configurable forwarded-header trust removed in b88156f (#11231), as
proposed in #11302.
* s3api: honor trusted-proxy allowlist in bucket/IAM policy engine
Make ExtractConditionValuesFromRequest a method on *PolicyEngine so it
can use the engine TrustedProxies when resolving aws:SourceIp. With no
allowlist configured the behavior is unchanged from b88156f: the direct
TCP peer is used and forwarded headers are ignored. When an allowlist is
configured via SetTrustedProxies, requests from a trusted peer honor
X-Forwarded-For (right-to-left) then X-Real-Ip.
Update the two call sites (auth_credentials.go, s3api_bucket_policy_engine.go)
and the engine tests to the method form, and add a regression test for the
trusted-proxy path.
* s3api: honor trusted-proxy allowlist in IAM role/session policies
Make extractRequestContext and extractSourceIP methods on
*S3IAMIntegration so they can use the integration TrustedProxies when
resolving aws:SourceIp. With no allowlist configured the behavior is
unchanged from b88156f: the direct TCP peer is used and forwarded
headers are ignored. When an allowlist is configured via
SetTrustedProxies, requests from a trusted peer honor X-Forwarded-For
(right-to-left) then X-Real-Ip.
Update the call site in isActionExplicitlyDeniedByIAM to type-assert
the integration and use the method, and add a regression test for the
trusted-proxy path.
* s3api: load [s3.trusted_proxies] from security.toml and wire to engines
Read s3.trusted_proxies.white_list (comma-separated IPs/CIDRs) from
security.toml and propagate the allowlist to the bucket policy engine,
the IAM policy engine (persisted across rebuilds via
IdentityAccessManagement.SetTrustedProxies), and the IAM integration.
Reloaded on SIGHUP alongside the JWT signing keys. Document the new
section in the scaffold security.toml.
Closes#11302.
* s3api: harden TrustedProxies parsing and X-Forwarded-For traversal
Canonicalize bare IP entries (via net.ParseIP + String) so non-canonical
IPv6 allowlist entries such as 2001:0db8::1 match peers rendered as
2001:db8::1, and log+skip unparseable bare entries instead of storing
them inertly.
When walking X-Forwarded-For right-to-left, stop at the first malformed
(non-empty, unparseable) entry instead of skipping it, and only fall
back to the leftmost valid IP when the chain was well-formed. This
prevents a malformed hop from masking a forged IP to its left.
Addresses review feedback on #11315.
* s3api: make TrustedProxies reload race-free via atomic.Pointer
Store the trusted-proxy allowlist behind sync/atomic.Pointer in
PolicyEngine and S3IAMIntegration so SIGHUP reloads (which swap the
allowlist) cannot race with concurrent request handlers reading it.
This mirrors the existing Guard guardState pattern. The
IdentityAccessManagement copy is already protected by iam.m.
Addresses review feedback on #11315.
* shell: warn when fs.mergeVolumes source holds only orphan needles
fs.mergeVolumes traverses filer entries, so a source volume whose
needles are all orphans — filer entries lost to a crashed write or a
wiped filer store — produces only the plan header and exits 0: no move,
no skip, no error. Operators read that as a successful merge while the
real cleanup (volume.fsck) never runs, and dat>idx volumes keep coming
back read-only after restarts.
Count the source-volume needles seen during traversal and, when a plan
source was never seen but its index still reports needles, print a
warning pointing at volume.fsck. Dry-run warns too.
* shell: make needle counting concurrency-safe and count manifest sub-chunks
TraverseBfs runs its callbacks from five workers, so the plain
needlesSeen map raced between source-heavy merges (fatal concurrent
map writes). All increments now funnel through a mutex-guarded
recordSeen closure.
Manifest sub-chunks that live on planned source volumes are now
recorded too — rewriteManifestChunk visits them (including dry-run
and capacity-skipped ones) but previously never marked their source,
which produced false 'orphan needles' warnings for sources whose
chunks were all reached through manifests.
* shell: extract sourceNeedleCounter so the concurrency test covers the production path
The orphan-warning recording was a closure local to Do, so
TestWarnUnreferencedSources_ConcurrentRecording could only exercise a
test-local copy of it — a regression in the production mutex would pass
the test. Lift the map and mutex into a sourceNeedleCounter type with
record/count methods and use it from Do and the test, so the -race test
now drives the actual recording path. Trim the verbose comments added
with the warning while here.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* rust: apply clippy --fix to both crates
The mechanical part of a clippy sweep: `cargo clippy --all-targets --fix`
on seaweed-volume and the seaweed-worker workspace, hand-reviewed. Both
manifests declare their MSRV (1.91.1 and 1.94.1), so every suggestion
clippy applied is within it: the collapsible_if sites become let chains
(1.88, edition 2024), `% n == 0` becomes is_multiple_of (1.87),
chunks_exact with a constant becomes as_chunks (1.88), repeat().take()
becomes repeat_n (1.82), and io::Error::new(Other, ..) becomes
io::Error::other (1.74). The rest is redundant clones, borrows, casts,
closures and field names.
Nothing here changes behaviour. The three let_and_return sites in
needle_map.rs and store_ec.rs deserve a note: the `let result = ..;
result` shape was a deliberate edition-2021 workaround to drop a redb
guard before the table it borrows. Edition 2024 drops tail-expression
temporaries before locals, which is why clippy now flags it, and the
two comments that described the workaround say so instead.
Manual edits on top of the tool output: the blocks clippy rewrote are
re-indented the way rustfmt lays them out (only those blocks — the
crate is not rustfmt-clean and a whole-crate fmt would bury this diff),
the blank lines let_and_return left behind are removed, and the CRC
legacy_value test compares against a literal worked out from the
original shift formula rather than restating rotate_right.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust: clear the clippy warnings --fix cannot apply, and say why the rest stay
Hand fixes for the lints clippy only reports. Behaviour is unchanged
throughout; each rewrite is the one clippy names.
- needless_range_loop (7): index loops over shard vectors become
iterator loops. Where the old code indexed `v[..n]` the new loop
iterates `v[..n]` so an undersized vector still panics the same way.
- field_reassign_with_default (6): struct literals with `..Default`.
- redundant_pattern_matching (3): `if let Err(_) = guard.check()` becomes
`.is_err()`, which also releases the read guard at the end of the
condition instead of at the end of the block.
- manual_strip (2), manual_checked_ops, format_in_format_args,
redundant_locals, wrong_self_convention (to_vif takes self by value,
so it is into_vif; CompactEntry is Copy, so to_needle_value takes self).
- type_complexity (2): `OrphanShardLoad` and `RawNeedleEntry` name two
tuples that were spelled out inline.
- new_without_default: CompactNeedleMap gets a Default that calls new().
- suspicious_open_options: a test helper spells out `.truncate(false)`,
which is what `.create(true).write(true)` already did.
What stays, and the attribute that says so:
- too_many_arguments (10): `#[expect]` on each function. Folding 8–15
parameters into a struct is a design change, not a lint fix.
- await_holding_lock / readonly_write_lock: one test holds the store
write guard across a sleep on purpose, as a barrier that parks the
copy task at the mount block. `#[expect(.., reason = ..)]` records it.
- module_inception: needle/needle.rs mirrors the Go package layout.
Two lints become crate-wide policy in `[lints.clippy]`, with the reason
next to each: result_large_err, because every RPC path returns
tonic::Status (176 bytes) and boxing it would change every handler
signature; and needless_update, because `..Default::default()` on a
protobuf message literal is what lets a proto gain a field without
touching every constructor (all 11 sites are pb messages). The worker
workspace gets the same table and its members opt in with
`lints.workspace = true`; its generated plugin.rs also allows
large_enum_variant on prost's oneof enums.
Both crates are now clean under `cargo clippy --all-targets -- -D warnings`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust volume: use the std APIs the 1.91 MSRV already pays for
The crate declares rust-version 1.91.1, so a few things the code still
worked around are plain std now. All of them come from the 1.85–1.91
release notes; nothing here needs a newer toolchain than the manifest
already requires.
- std::sync::LazyLock (1.80) replaces the lazy_static! block in
metrics.rs, and the lazy_static dependency goes. Every use site reads
the same through Deref, so no caller changes.
- Duration::from_mins / from_hours (1.91) replace `from_secs(v * 60)`
and `from_secs(v * 3600)` in the option parser and the shard-location
refresh TTLs. One difference for the parser: an absurd count that
overflows u64 seconds now panics in release builds too, where the
multiplication used to wrap.
- Result::flatten (1.89) replaces `.and_then(|r| r)` on the replication
join handle.
- OsStr::display (1.87) replaces `to_string_lossy()` where the name was
only being formatted; the output is byte-identical.
- `#[allow]` becomes `#[expect]` (1.81) on the suppressions that are
meant to be permanent, so a suppression that stops being needed
becomes a warning rather than lingering. Doing that found four that
already had: dead_code on ChunkManifest, base_name and last_io_error,
and too_many_arguments on read_from_data_shards, which is down to
seven parameters. Those attributes are deleted. The three allows that
depend on cfg (a unix-only mutation, a linux-only field set, a
profiling-only parameter) stay as allow, because expect would be
unfulfilled on the other platforms.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* ci: add a commented-out clippy step to both Rust workflows
Both crates are warning-free under `cargo clippy --all-targets
-D warnings` now. Whether that becomes a gate is a policy call, so the
step is present but commented out; uncommenting it is the whole change.
The comment points at the `[lints.clippy]` table where crate-wide
exceptions are recorded, so the gate does not become a reason to
sprinkle allows.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust volume: guard parse_duration against overflow panics
Duration::from_mins/from_hours panic when the count overflows u64
seconds. Use checked_mul so an oversized CLI value falls back to the
parser default instead of crashing volume startup.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* admin: bind worker gRPC listener to -ip instead of wildcard
The worker/plugin gRPC control plane called net.Listen("tcp", ":port")
directly, so it wildcard-bound every interface and ignored the -ip setting.
A cluster bound to loopback still exposed the unauthenticated
WorkerService/PluginControlService streams on 0.0.0.0. Bind through
util.JoinHostPort(bindIp, port) so the listener honors -ip like the
master, filer, and volume gRPC listeners.
* admin: warn when worker gRPC is exposed off loopback without mTLS
The worker gRPC stream has no password auth, so grpc.admin mTLS is the
only effective control once the listener leaves loopback. An operator who
sets -adminPassword and binds -ip=0.0.0.0 authenticates the HTTP API but
still exposes the unauthenticated worker control plane. Log a startup
warning naming the port and the mTLS knobs so the exposure is not silent.
* admin: address review on worker gRPC bind fix
- mini: reserve the admin gRPC port with util.JoinHostPort so an IPv6
bindIp (e.g. ::1) does not form an invalid unbracketed address and
lose the reservation.
- worker gRPC: track whether grpc.admin mTLS credentials actually loaded
rather than only whether they were configured, and gate the
non-loopback exposure warning on that. A cert/key that fails to load
now still warns instead of silently suppressing.
seedAndSpread() calls `volume.grow -dataNode X` without pinning the rack.
The master's grow picks the rack by weighted-random when -rack is unset,
and only one of the three racks holds the requested data node, so an
unpinned grow lands on the wrong rack two times out of three. The
VolumeGrow RPC swallows the "No matching data node" failure for
non-cache collections, so those grows count as success without creating
a volume. The per-server cap (maxGrowsPerServer=4) is then exhausted by
silent no-ops before the volumes ever spread, and seedAndSpread times
out with "volumes never spread across >=2 disks on all 3 nodes".
Pin -dataCenter dc1 and -rack rack{i} alongside -dataNode so every grow
reaches the target node. This removes the timing-sensitive assumption
that made TestECVacuumDuplicateShardClaimAcrossDisks flaky.
* rust-volume: fix Windows build of find_needle_from_ecx
The .ecx binary-search fallback path used on non-Unix targets
(Seek + Read, both &mut self receivers) requires the ecx_file
binding to be mutable. On Unix the read_exact_at path takes &self,
so the mut would be unused there — gate that warning with
#[cfg_attr(unix, allow(unused_mut))].
Without this the build-rust-volume-windows CI job fails with
E0596 at ec_volume.rs:1033, breaking the weed-volume_windows_amd64
release asset.
* rust-volume: use positional seek_read for .ecx lookups on Windows
The previous fix (making ecx_file mut) compiled but left the Windows
fallback using Seek + Read on the shared .ecx file cursor. Concurrent
find_needle_from_ecx calls could interleave seek/read and read the
wrong index entry, corrupting the binary search (raised by Devin and
Greptile review on the PR).
Switch the Windows path to std::os::windows::fs::FileExt::seek_read,
which is positional (offset passed via OVERLAPPED, cursor untouched)
and takes &self — so the binding no longer needs mut, and concurrent
callers on the cached handle can't interfere. Mirrors the existing
read_exact_at helper in storage::volume.
Add a compile_error fallback for non-unix/non-windows targets to
match the convention in storage::volume.
filer: clean up manifest resolve error propagation and add webdav test (#78)
Drop GitHub issue references from comments and trim verbose comments.
Replace the viewFromChunksOrErr helper with the existing
NonOverlappingVisibleIntervals + ViewFromVisibleIntervals at the stream
call sites, and add a WebDavFile.Read regression test for the manifest
resolution failure path.
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* filer: add guarded remote-storage client builder hook for lazy fetch
The lazy-remote fetch path (maybeLazyFetchFromRemote) resolved its
remote-storage client through the unguarded shared cache, bypassing the
SSRF chokepoint (BuildGuardedRemoteStorageClient) that the CVE-2026-73080
remediation wired into the volume, filer stream and s3 stream dial paths.
Add a RemoteStorageClientBuilder hook on Filer plus conf-only lookups on
FilerRemoteStorage, and route the lazy fetch through the builder when set
(endpoint deny-list + DNS-rebinding-safe dialer), falling back to the
shared cache otherwise. The filer server wires the builder in a follow-up.
* filer: route lazy directory listing through the guarded remote client
maybeLazyListFromRemote shared the unguarded client resolution of the
fetch path, so a caller-supplied remote endpoint was dialed without the
SSRF deny-list or rebinding-safe dialer. Resolve the conf and build the
client through buildRemoteStorageClient so the same guard covers listing.
* filer: route lazy remote delete through the guarded remote client
maybeDeleteFromRemote issued outbound DELETE/RemoveDirectory requests
through the unguarded client, giving a write-side SSRF to a caller-chosen
endpoint. Resolve the conf and build the client through
buildRemoteStorageClient so the endpoint deny-list and rebinding-safe
dialer apply to the delete path as well.
* filer server: wire the guarded remote client builder into the filer
Set Filer.BuildGuardedRemoteClient to BuildGuardedRemoteStorageClient and
forward AllowUntrustedRemoteEndpoints so the lazy-remote fetch, list and
delete paths apply the same SSRF endpoint checks as the volume and
streaming read paths.
* filer: test lazy fetch honors the guarded remote client builder
Add a regression test that sets BuildGuardedRemoteClient to a rejecting
builder and asserts maybeLazyFetchFromRemote returns no entry without
reaching the remote, covering the SSRF guard wired in the prior commits.
* filer: skip remote client for local-only lazy deletes
maybeDeleteFromRemote resolved and validated the mount's remote client
before checking entry.Remote, so a local-only file (no Remote entry) under
a mount whose endpoint the guard rejects failed to delete: the guard
error aborted the metadata deletion, leaving a file that needs no remote
operation undeletable. Move the local-only check ahead of client
construction so only remote-backed files and directories pay the guard.
* filer: build the guarded remote client inside the lazy singleflight
The lazy fetch and list paths built the guarded client before their
singleflight blocks, so concurrent requests for the same key each
allocated a fresh SDK client and HTTP transport even though only one
remote operation ran. Move client construction inside the singleflight
so the deduplicated operation builds it once, matching the per-request
guard semantics of the sibling streaming paths without the duplicate
transport churn.
* filer: test guarded rejection for the lazy list and delete paths
Add regression tests that set BuildGuardedRemoteClient to a rejecting
builder and assert the lazy list does not reach the remote, a
remote-backed file delete is blocked, and a local-only file under a
rejected mount still deletes (covering the local-only fix).
* filer: decouple lazy guarded-client build from the first caller's context
Building the guarded client inside the singleflight made concurrent
fetches share the first caller's context. If that caller canceled while
endpoint DNS validation was running, the builder returned an error and
published a not-found result to other callers whose contexts were still
valid. Build with context.WithoutCancel so the guard's DNS validation
is not tied to any single caller's cancellation, matching the list
path's existing decoupling for the remote operation itself.
* filer: reject remote-storage confs that dial blocked endpoints at load
The filer's lazy-fetch / lazy-list / remote-delete paths resolve remote
storage clients by name from FilerRemoteStorage.storageNameToConf and
dial them via remote_storage.GetRemoteStorage, which bypasses the SSRF
deny-list the volume server (BuildGuardedRemoteStorageClient) and the
filer's own direct-read path apply. A RemoteConf planted under
/etc/remote with a loopback / private / IMDS S3 endpoint is reloaded into
storageNameToConf on the next metadata-change event and then dialed on
the next cache miss — server-side request forgery from the filer.
Apply the volume server's SSRF deny-list at conf load time, the single
chokepoint that populates storageNameToConf:
- Add RemoteStorageConfValidator, injected into FilerRemoteStorage by
the filer server (the filer package cannot import the server package).
A conf that fails validation is dropped from storageNameToConf, so the
name-based client resolution on the lazy paths returns "not found"
instead of dialing the blocked endpoint.
- Add ValidateRemoteConfForLoad in weed_server, which mirrors
BuildGuardedRemoteStorageClient's gcs credential + endpoint checks
(validateRemoteEndpoint via guardedRemoteClient) without building a
client. allowUntrusted skips the check, mirroring the volume server
opt-out (-filer.allowUntrustedRemoteEndpoints).
- The filer server injects the validator at construction.
A conf whose type dials a fixed provider host (no caller-supplied
endpoint) passes; only caller-influenced endpoints are denied.
* filer: skip DNS resolution in the load-time SSRF validator
ValidateRemoteConfForLoad resolved hostnames during /etc/remote reload,
so a transient DNS failure (2s timeout) dropped the conf from the fresh
map that replaces the live map, disabling a working mount until the next
metadata event. The build-time guard (BuildGuardedRemoteStorageClient)
already re-resolves and re-validates the endpoint at dial time with the
rebinding-safe dialer, so DNS at load is redundant for security.
Split the static checks (scheme, IMDS hostnames, IP-literal blocked
addresses, gcs credentials) into validateRemoteEndpointForLoad, which
does no DNS. Hostname endpoints pass at load and are caught at dial if
they resolve to a blocked address. This preserves fail-fast for
statically-blocked confs (loopback IPs, IMDS hostnames) without letting
transient DNS failures disable mounts.
* filer: accept empty S3 endpoints in the guarded remote client builder
guardedRemoteClient returned ok=true with an empty endpoint for a
standard AWS S3 config (no custom S3Endpoint), so
BuildGuardedRemoteStorageClient and ValidateRemoteConfForLoad rejected
it with "remote endpoint is empty" — breaking standard AWS S3 mounts on
the lazy paths and the sibling streaming read paths that already use the
guarded builder.
An empty endpoint is not caller-supplied: the AWS SDK derives the
regional endpoint from the region, so there is nothing for the SSRF
guard to validate. Return ok=false for empty S3-compatible endpoints so
the builder falls through to the shared unguarded cache, matching the
historical behavior for standard AWS S3.
* filer: accept gcs credentials file paths in the guarded remote client builder
checkGcsCredentials rejected all filesystem paths, so a gcs mount
configured with remote.configure -gcs.appCredentialsFile (which stores
a path in GcsGoogleApplicationCredentials) was rejected by
BuildGuardedRemoteStorageClient with "gcs credentials must be inline
JSON". This broke existing gcs mounts on the volume, filer, and s3
remote-mount read paths that use the guarded builder.
Read and validate the file content instead of rejecting the path,
mirroring what the gcs client itself does in MakeWithHTTPClient. A path
that does not exist or does not contain valid gcs credentials is still
rejected before any client is built. guardedRemoteClient now reads the
file to extract the token exchange URL for the SSRF deny-list, so the
rebinding-safe dialer still guards the token endpoint.
* filer: resolve gcs credential paths and avoid leaking file existence
loadGcsCredentialsContent passed the raw credentials string to os.ReadFile,
so a documented ~/path (as written by remote.configure
-gcs.appCredentialsFile=~/...) was rejected because os.ReadFile does not
expand ~. It also wrapped the os.ReadFile error, which includes the
file path, exposing file existence to a caller who planted a conf with
an arbitrary path.
Resolve the path with util.ResolvePath, matching the gcs client's own
behavior in MakeWithHTTPClient. Return a generic sentinel error on read
failure so the path is not reflected in the error message. The credential
type validation still runs on the file content, so a path that does not
contain valid gcs credentials is rejected before any client is built.
* test/ec: fix EC interruption matrix slot exhaustion
The EC integration test cluster (test/erasure_coding/chaos_lifecycle_test.go)
configured each disk with -max 4 and the seedAndSpread spread loop fired
volume.grow -count 4 every 2 s with no per-server cap. Because the master
topology lags the volume.grow writes, the loop re-fired before the prior
grow was visible, over-filling disks to capacity. A full disk leaves zero
free EC shard slots (failing the cluster-wide capacity check with "no
free ec shard slots") and drops the source disk below the encode's
FreeVolumeCount >= 2 health check (failing with "no healthy replicas"),
which aborted ec.encode before any phase marker printed and made every
encode scenario in TestECInterruptionMatrix fail.
Three changes to the test cluster:
1. Raise -max from 4 to 8 per disk so the source disk always retains
FreeVolumeCount >= 2 for ec.encode's 14-shard generation (2 volume-slot
equivalents) even after the spread loop and multiple encodes.
2. Switch the spread loop from -count 4 to -count 1 so each grow lands
exactly one volume on the volume server's least-loaded disk, giving
deterministic cross-disk spreading instead of relying on a single
multi-volume grow to fan out.
3. Cap grows per server at 4 so heartbeat lag cannot run away and
over-fill disks before the master registers the prior grow.
4. Pass -minFreeSpace 0 so the test is not falsely gated by the physical
disk's free-space percentage on the host running CI (the EC shard slot
calculation separately enforces a 90 % disk-usage cap via
balancer.DiskTooFullAfter, which already guards against an over-set
maxVolumeCount on a physically full disk).
Verified locally by running TestECInterruptionMatrix twice (all encode,
decode, and balance scenarios pass, including the previously failing
encode@Deletingoriginalvolumes).
* test/ec: only count successful grows toward the spread cap
A failed volume.grow (e.g. a transient collectTopologyInfo or VolumeGrow
RPC error) would otherwise consume one of the four permitted attempts
without creating any volume, exhausting the retry budget and leaving the
loop to only poll until the Eventually timeout. Increment the per-server
counter only when commandGrow.Do returns nil.
* iceberg: confine commit/transaction/view-update write paths to authorized bucket
The create, register, and createView handlers already confine the client-
supplied metadata location to the caller table bucket and reject ".."
segments. The commit, create-on-commit, transaction, and view-update paths
read the stored metadataLocation back from the catalog and skipped the same
guard, so a location poisoned via the raw S3Tables UpdateTable API (which
persists metadataLocation verbatim) could escape the caller bucket through
a ".." segment that path.Join collapses in saveMetadataBlob.
Add confineMetadataLocation and apply it after parseS3Location on every
commit/update/transaction/view write path, mirroring the create/register/
createView check. Reject with 400 so a poisoned stored location fails the
commit instead of writing into another tenant bucket tree.
* s3tables: validate metadataLocation at the store layer
The raw S3Tables API (CreateTable, RegisterTable, UpdateTable, CreateView,
UpdateView) persisted the client-supplied metadataLocation verbatim with no
bucket-confinement or traversal check, so a caller could store a location
pointing outside its own bucket. The Iceberg REST gateway commit paths then
read that stored value back and wrote through it.
Add ValidateMetadataLocation and call it in every s3tables store handler
that accepts a metadataLocation, rejecting locations whose bucket differs
from the caller table bucket or whose path contains traversal segments. This
prevents a poisoned location from ever being persisted, complementing the
per-write-path guard added to the Iceberg commit handlers.
* iceberg/s3tables: validate location before repair and after idempotency check
Address review feedback:
- Move the commit-path confinement check ahead of repairManifests so a
poisoned stored location cannot reach manifest repair I/O before the
commit is rejected.
- Move ValidateMetadataLocation in CreateTable/CreateView to after the
existing-resource check so idempotent retries that do not consume the
requested location are not rejected for an unused bad location.
- Assert HTTP 400 in the cross-tenant reproduction tests so an unrelated
failure cannot satisfy them.
* iceberg: confine staged metadata location before load in create-on-commit
The create-on-commit path parsed the staged metadata location from the
stage-create marker and called loadMetadataFile before validating that the
staged bucket/path stay within the authorized bucket. Add the same
confineMetadataLocation guard before the read so a tampered marker cannot
direct a cross-tenant metadata read.
* iceberg/s3tables: reject bucket-only metadata locations
ValidateMetadataLocation and confineMetadataLocation accepted s3://bucket
with an empty table path. metadataDirPath then maps every such table to the
shared <TablesPath>/<bucket>/metadata directory, so tables could overwrite
or read each other's metadata files. Require a non-empty table path in both
validators; the empty-location case (where the catalog derives one) is
unaffected.
* iceberg/s3tables: reject slash-only table paths in location validation
s3://bkt/// parses to tablePath="/" which passed the empty-string check
but path.Join cleans it away, mapping to the bucket-level metadata
directory shared across tables. Update isValidTablePath to require at
least one non-empty segment and mirror the same check in
ValidateMetadataLocation, closing the gap in all callers.
* s3/lifecycle: attach admin Bearer token on internal LifecycleDelete clients
Export credential.WithS3InternalAdminAuth (renamed from withIamCacheAdminAuth)
and use it in the worker and shell lifecycle RPC adapters so lifecycle calls
carry the same admin token the IAM-cache propagation already attaches. No-op
when jwt.filer_signing.key is unset, matching the server-side checkAdminAuth.
Prepares the internal clients for the server-side auth gate that follows.
* s3/lifecycle: gate LifecycleDelete behind admin Bearer auth
Add checkAdminAuth to LifecycleDelete, matching the SeaweedS3IamCache handlers on the same internal gRPC listener (PR #11190). No-op when jwt.filer_signing.key is unset; rejects unauthenticated callers when it is. The internal worker/shell clients already attach the token in the previous commit.
* master: name the unlabeled disk layout plainly in assign errors
When no volume server serves the layout an assign targets, the error
named the empty disk type as "hdd" (HardDriveType is the empty string),
sending operators looking for servers labeled hdd when the actual
mismatch is labeled (e.g. -disk=ssd) servers versus unlabeled clients.
- describe the layout as "default (unlabeled)" when the disk type is
empty, keep %q naming for labeled types
- log the unserved-layout condition once per option instead of letting
every failing write repeat an unactionable line
Observed in production: volume servers started with -disk=ssd while CSI
mounts assign with the unlabeled layout; the per-write error stream
pointed at a nonexistent hdd fleet.
* master: bound and expire the unserved-layout warning dedupe
The dedupe map retained every distinct option key permanently. Option
keys embed request-derived fields (collection, disk type), so repeated
assignments with distinct options would grow master memory without
bound, and a retained key suppressed the warning if the same option
went unserved again after the topology recovered.
Remember last-warned timestamps instead, expiring after an hour, with a
hard cap that resets the set when a client-driven key flood fills it.
* master: silence per-retry unserved-layout log and name explicit hdd
Addresses Devin Review comments on #11290.
- The unserved-layout branch already rate-limits its warning via
assignUnservedLayoutWarning.Do, but the common epilogue still logged
lastErr at V(0) on every retry, so the flood the dedup was meant to
stop continued. Skip the epilogue log when the unserved-layout branch
owns the logging; the error is still returned to the client.
- describeDiskLayout took the canonicalized option.DiskType, but
ToDiskType folds both "" and "hdd" into HardDriveType, so an explicit
disk=hdd request was mislabeled "default (unlabeled)". Pass the
original request disk type instead: only an empty request is the
unlabeled default; an explicit hdd is named "hdd".
Adds TestAssignFailsFastNamesExplicitHdd covering the explicit-hdd
wording.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
makeupDiff called dstDatBackend.Sync() (fsync) per needle in the loop
over incrementedHasUpdatedIndexEntry. With 20000 entries in
TestLDBIndexCompaction this resulted in up to 20000 fsync calls, which
on slow CI disks exceeded the 10-minute test timeout.
Batch the sync: write all needles/tombstones first, then fsync the dat
file once in the defer alongside the existing idx fsync. The durability
guarantee is unchanged — both files are still synced before CommitCompact
writes the .cpc commit marker and swaps the files.
* cache resolved chunk manifests for Mount
* Address PR review: per-mount cache, singleflight, reuse ResolveOneChunkManifest
- Own the manifest cache per WFS mount instead of a process-global
variable, so manifests from one filer backend are never served to
another (Devin/CodeRabbit major bug).
- Coalesce concurrent cold misses via singleflight so only one fetch
runs during a cold burst (Greptile P2).
- Copy cached data after releasing the mutex so a large copy does not
block concurrent hits, inserts, and evictions (CodeRabbit nitpick).
- Reuse the existing ResolveOneChunkManifest function name instead of
introducing a new resolveOneChunkManifest wrapper.
- Validate (unmarshal) manifest bytes before caching so malformed
manifests do not poison the cache.
- Add TestChunkGroupManifestResolutionCoalescesColdMisses covering
the singleflight cold-miss path.
* Address round 2 review: coalesced-miss cancellation, test overlap
- Use singleflight.DoChan in fetchOrLoad and select on ctx.Done() so a
caller whose context is canceled while waiting for an in-flight fetch
returns ctx.Err() promptly instead of blocking for the leader's
result (Devin BUG).
- Add TestResolveOneChunkManifestCanceledWaiterReturnsDuringCoalescedMiss
covering the canceled-waiter path.
- Delay the cold-miss fixture response so the leader's fetch is still
in flight when concurrent opens join the singleflight, making the
one-fetch assertions reliable (CodeRabbit Minor).
* Address review: keep ResolveOneChunkManifest four-argument
Restore the exported ResolveOneChunkManifest to its original
four-argument signature so external callers keep compiling. Move the
cache-aware resolution into an unexported resolveOneChunkManifest
helper that accepts the per-mount ChunkManifestCache. The exported
function delegates to the helper with a nil cache, preserving the
historical uncached behavior for every non-Mount caller. The Mount
path (ChunkGroup.SetChunks) now calls the unexported helper with the
mount-owned cache. Tests and benchmarks that exercise the cache path
call the unexported helper directly.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* mount: fail reads with error when chunk manifest resolution fails
When SetChunks fails to resolve a chunk manifest (e.g. the volume is on a
remote tier with reads disabled), the sections map stays empty and
readDataAtSequential/readDataAtParallel zero-fill every missing section as
if it were a sparse hole. Reads then return all-zero data with no error,
so a plain cp of a large manifest-based file silently produces a
completely zero-filled file.
Remember the resolve error in ChunkGroup (guarded by sectionsLock) and
return it from ReadDataAt. A later successful SetChunks clears it.
Fixes the mount path of #11286.
* filer: propagate manifest resolve errors in streaming read paths
ViewFromChunks discards the chunk manifest resolve error returned by
NonOverlappingVisibleIntervals. On failure the chunk views come back
empty, and the streaming paths zero-fill the entire requested range,
serving HTTP 200 / WebDAV 200 responses whose body is all zeros.
Propagate the error in PrepareStreamContentWithThrottler,
PrepareStreamContentWithPrefetch and the WebDAV read path so these
requests fail with 500 instead.
Fixes the filer HTTP and WebDAV paths of #11286.
* mount: fail lseek with EIO when chunk manifest resolution fails
SearchChunks still consulted the stale section map after SetChunks
recorded a manifest resolution failure, so SEEK_DATA/SEEK_HOLE would
describe the unresolved regions as sparse holes or return ENXIO.
Return the recorded error from SearchChunks and map it to EIO in
Lseek.
Also add regression tests for the stream preparation error paths.
Addresses review feedback on #11287.
Chris Lu