* volume: keep reads over the download limit local when readMode is local
With -readMode=local the server never sends a read for a volume it does
not hold elsewhere, but a read over -concurrentDownloadLimitMB on a
replicated volume was still redirected to another replica. Skip the
replica fallback in local mode, so the read waits for a download slot
and answers 429 when the wait times out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): keep reads over the download limit local when readMode is local
Mirror the Go volume server: with readMode local, a read over the
download limit waits for a slot instead of being redirected to a
replica. proxy_or_redirect_to_target no longer needs the local-mode
redirect switch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: check download capacity under the wait lock
waitForDownloadSlot read the in-flight size outside the cond lock and
the read handler broadcast without it, so a download finishing between
the check and the wait was missed and the read waited out the timeout
for a 429. Check capacity with util.WaitUntil under the lock and
broadcast under it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): register for the download notification before checking capacity
The wait loop created its Notified future only after the capacity
check, so a body dropped in between had its notify_waiters() lost and
the read waited out the deadline for a 429. Create the future first,
then check, as Go now checks under the cond lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* security: reload JWT signing keys on SIGHUP
Signing keys were read once in the server constructors and never
refreshed. After a key rotation (Secret update, divergent reads) the
in-memory key stayed stale and every request kept failing "wrong jwt"
until the affected process was restarted.
Add Guard.UpdateSigningKeys and call it from the master, volume and
filer reload paths and the s3 reload hook, next to the existing
whitelist refresh. Make the global chunk-read JWT cache reloadable via
an atomic swap, and register the master's Reload with grace.OnReload --
it was never wired, so the master ignored SIGHUP entirely.
Mirror the same refresh in the Rust volume server's SIGHUP handler.
* security: swap signing keys behind an atomic pointer
Addresses review feedback on the in-place key swap: SigningKey is a
[]byte, so reassigning the Guard fields while a request handler reads
them is a data race that can tear the multi-word slice header and read
out of bounds.
Hold the four signing-key fields in an immutable signingConfig snapshot
behind atomic.Pointer; UpdateSigningKeys swaps the whole pointer, so a
reader sees either the old keys or the new ones. Reads go through new
SigningKey/ExpiresAfterSec/ReadSigningKey/ReadExpiresAfterSec accessors.
The Rust guard is already safe: every read and the SIGHUP write go
through the shared RwLock<Guard>.
* security: fold whitelist + auth state into the atomic snapshot
Review follow-up. UpdateSigningKeys still wrote isWriteActive while the
request path read it (and the whitelist maps) unsynchronized, so a SIGHUP
under load could expose an inconsistent mix of activation bits and
whitelist contents.
Move all hot-reloadable Guard state -- keys, expirations, whitelist, and
the activation flags -- into a single immutable guardState swapped behind
one atomic.Pointer. The Update* methods take a small mutex to serialize
the read-modify-write; readers stay lock-free. The concurrency test now
also rotates the whitelist and probes IsWhiteListed under -race.
Also read each signing key once per branch in the volume/filer JWT auth
checks, so a reload landing mid-check can't take the allow-fast-path
after auth was enabled or verify against a different key than the branch
saw.
fix: add debug logging for JWT validation failures (#7788)
When JWT file ID validation fails during replication, add a log message
showing both the expected and actual file IDs to help diagnose issues.
Ref #7788