Github-Mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-20 13:30:46 +02:00
Code Issues Packages Projects Releases Wiki Activity
13,714 Commits 181 Branches 360 Tags
b9bf45cb2e99adaa6c46406b1ff938ea85173f2d
Commit Graph
8846 Commits
Author SHA1 Message Date
Chris Lu b9bf45cb2e fix(shell): scope volume.fsck filer walk when -volumeId selects one bucketed collection (#9347) 
* fix(shell): scope volume.fsck filer walk to the bucket when -volumeId selects one bucketed collection

Closes #9345.

-volumeId only filtered which volume .idx files were pulled; the filer-side
BFS still walked from "/", printing every directory under -v and making it
look like the flag was ignored. When all requested volumes share a single
non-empty collection that maps to an existing <bucketsPath>/<collection>
directory, restrict the BFS root to that bucket. Empty-collection volumes
or multi-collection selections fall back to the full walk, since chunks
for those can live anywhere.

* trim comments

* address review: collapse getCollectFilerFilePath; unshadow receiver in loop
 2026-05-07 10:04:01 -07:00
Chris Lu 4e10669221 docs(s3/lifecycle): event-driven redesign (#9346) 
* docs(s3/lifecycle): event-driven redesign

Replaces the synchronous PUT-handler walk with an event-driven worker
model: meta-log reader subscribed to one filer, client-side heap merge
with per-filer-shard MessagePosition cursors, bucket-level bootstrap
with inline delete, blocked-cursor handling for fatal events, durable
retry budget for sustained-transient promotion, retention mode gate
that downgrades reader-driven rules to scan_only when log retention
falls below the rule's event-log horizon.

* docs(s3/lifecycle): record Phase 0 verified assumptions

ReadPersistedLogBuffer payload carries Extended (event marshaled via
SubscribeMetadataResponse → ToProtoEntry). Meta-log files at
topics/.system/log/<date>/<HH-MM>.<filerId> are written without TtlSec
in filer_notify_append.go; retention is unbounded by default and only
shrinks if an operator sets a filer.conf rule with a TTL on the
SystemLogDir prefix. .versions/ filenames are v_<16h-ts><16h-rand>
with old/new (inverted) format distinguished by threshold
0x4000000000000000; getVersionTimestamp / compareVersionIds give
format-agnostic ordering for successor-version discovery.

* feat(s3/lifecycle): rule evaluator and dueAt helper

Evaluate(rule, info, now) returns the EvalResult by object shape:
IsMPUInit -> AbortMultipartUpload, IsDeleteMarker -> ExpireDeleteMarker
when sole survivor, IsLatest -> DeleteObject (Days or Date), non-current
-> DeleteVersion (Days fallback to ModTime when SuccessorModTime is
zero; NewerNoncurrent retention enforced when both are set).

ComputeDueAt mirrors Evaluate's shape and returns the earliest eligible
wall-clock time for the same (rule, info), used by the reader/bootstrap
to decide pending vs inline-delete.

Adds StatusEnabled/Disabled and SmallDelay consts and an IsMPUInit
flag on ObjectInfo so .uploads/<id>/ entries route to AbortMPU without
overloading IsLatest.

No callers; package compiles standalone.

* feat(s3/lifecycle): MinTriggerAge for safety-scan cadence

Returns the smallest non-zero day threshold across the rule's actions.
Used as max(MinTriggerAge, kindFloor) for the per-kind cadence; date,
count-only, and delete-marker-only rules return 0 so callers fall
through to their kind-specific floor.

* feat(s3/lifecycle): EventLogHorizon for retention mode gate

Returns the maximum event age the reader needs for a rule. Days-based
kinds return their day threshold; pure NewerNoncurrent (count) and
ExpiredObjectDeleteMarker return SmallDelay. Date rules return 0 (the
gate skips them). Multi-action rules take the max — strictest horizon
wins.

Drives the Phase 2 mode gate: metaLogRetention < EventLogHorizon(rule)
+ bootstrapLookbackMin -> scan_only with RETENTION_BELOW_HORIZON.

* feat(s3/lifecycle): RuleHash for per-rule state CAS

sha256 over canonicalized form, first 8 bytes. Stable across tag-key
reorder, prefix trailing-slash variation, ID renames, and Status flips
(state continuity is preserved when an operator toggles
Enabled/Disabled). Different action shapes — different days, filter,
or action type — hash differently.

Used by the per-rule state directory layout
/etc/s3/lifecycle/<bucket>/<rule_hash_hex>/ and by the bootstrap
detector's reconcile-on-PUT.

* feat(s3/lifecycle): add s3_lifecycle.proto storage schema

Defines the durable types backing the lifecycle worker:
LifecycleState (per-rule mode + bootstrap_complete + degraded_reason
incl. RETENTION_BELOW_HORIZON / LOST_LOG), PendingItem, EntryIdentity,
BootstrapState, ReaderState (per-filer-shard cursors plus
tail_drained_streams marker), BlockerRecord (rule_hash optional for
pre-evaluation failures), RetryBudgetEntry with the four-shape
StreamKey oneof, and RetryTarget (no action / no expected_identity —
retry replays handler against current state).

No callers; schema only. Wired into the pb Makefile.

* feat(s3/lifecycle): add S3LifecycleParams to TaskParams

Wires lifecycle subroutines into the existing worker dispatch.
Subtype is READ (cluster-singleton meta-log reader), BOOTSTRAP
(per-bucket walker), or DRAIN (per-rule pending). bucket / rule_hash
populated for the latter two; ContinuationHint is an advisory resume
point for kill-resumable BOOTSTRAP / READ.

oneof tag = 14, after the existing ec_balance_params at 13.

* fix(s3/lifecycle): noncurrent delete markers honor NoncurrentDays

A non-current delete marker is just another version per AWS S3 spec
and is eligible under NoncurrentVersionExpirationDays. The
IsDeleteMarker special case is meant only for the *current* delete
marker (sole-survivor ExpiredObjectDeleteMarker action), so guard
that switch arm with IsLatest. Without IsLatest, the bootstrap walker
silently skipped pre-existing non-current delete markers under a
NoncurrentDays rule because ComputeDueAt returned zero.

Mirrors the same fix in evaluate.go so the runtime decision matches.

* fix(s3/lifecycle): preserve prefix trailing slash in RuleHash

"logs" and "logs/" match different object sets under literal
strings.HasPrefix semantics: "logs" matches "logsmore/x", "logs/" does
not. Collapsing them in the hash would let an XML edit silently bind
the new rule to the previous rule's durable state directory, causing
stale bootstrap_complete and stale pending entries against a rule
that now matches a different set.

* fix(s3/lifecycle): add UNSPECIFIED sentinels to enum zero values

Proto3 best practice: enum 0 should be an _UNSPECIFIED sentinel, not
an active value. Persisted state schemas care most: a partially
populated payload (or one written by an older binary that didn't set
the field) would otherwise silently default to a semantically active
value.

- LifecycleState.RuleKind: shifts EXPIRATION_DAYS..EXPIRED_DELETE_MARKER
  from 0..5 to 1..6, with RULE_KIND_UNSPECIFIED at 0.
- LifecycleState.RuleMode: shifts EVENT_DRIVEN..PENDING_BOOTSTRAP from
  0..4 to 1..5.
- LifecycleState.DegradedReason: replaces NONE=0 with
  DEGRADED_REASON_UNSPECIFIED=0 (operators treat both as healthy).
- StreamKind: shifts ORIGINAL..PENDING from 0..3 to 1..4.
- S3LifecycleParams.Subtype: shifts READ..DRAIN from 0..2 to 1..3, so
  an unset subtype no longer routes into the cluster-singleton READ task.

No on-disk state has been written yet; renumbering is safe.

* docs(s3/lifecycle): per-action state, not per-rule

A single AWS lifecycle XML <Rule> can declare multiple actions in
parallel (e.g. ExpirationDays=90 + AbortMPU=7 + NoncurrentDays=30).
Each must drive its own delay/horizon/mode/pending stream
independently. Modeling the rule as one compiled entry with one kind
collapses these — picking the smallest delay (7d MPU) means the 90d
expiration cursor advances past objects that aren't yet due, and the
90d action never re-fires.

Restructure storage to per-action: every XML rule expands into N
compiled actions; state lives at <bucket>/<rule_hash>/<action_kind>/.
The intermediate rule_hash directory keeps a rule's actions grouped
for operator listing. Each action has its own state file with its own
mode + bootstrap_complete + degraded_reason; sibling actions of the
same rule can degrade independently.

* fix(s3/lifecycle): per-action proto schema + safety-scan counters

Realigns the durable schema with the per-action storage model and the
safety-scan contract in the design doc.

- Promote LifecycleState.RuleKind to a top-level ActionKind enum and
  rename rule_kind -> action_kind. The same enum now also keys
  BootstrapKey, PendingKey, and BlockerRecord so per-action streams
  under one rule never collapse.
- LifecycleState now keyed by (rule_hash, action_kind). Added
  last_safety_scan_ts_ns / next_safety_scan_ts_ns and the four
  observability counters the design specifies (evaluated_total,
  expired_total, metadata_only_total, error_total). Dropped
  last_evaluated_ns, deleted_total, skipped_object_lock_total, and
  pending_size — subsumed or out-of-band.
- BlockerRecord.action_kind is OPTIONAL (UNSPECIFIED for
  pre-evaluation failures, just like rule_hash).

No on-disk state has been written yet; the renumbering / rename is
free.

* fix(s3/lifecycle): per-action MinTriggerAge / EventLogHorizon helpers

The retention mode gate and safety-scan cadence run on each compiled
action independently. Taking a "min/max across all actions in the
rule" — as the previous helpers did — was wrong: a 90d ExpirationDays
sibling alongside a 7d AbortMPU action would cause the 90d cursor to
advance at 7d (because MinTriggerAge picked the smallest), and the
ExpirationDays action would never re-fire on objects that aged past
the 7d sweep window before reaching 90d.

Both helpers now take an ActionKind and return the threshold for that
specific action only. Returns 0 if the rule does not declare the
requested kind, which makes the gate a no-op and the cadence fall
through to the kind floor.

Also adds RuleActionKinds(rule), the canonical expansion that the
engine uses at compile time to turn one XML rule into N compiled
actions. NewerNoncurrentVersions paired with NoncurrentDays is
subsumed into a single NONCURRENT_DAYS action (AWS-paired
conditions); only when NewerNoncurrent stands alone does it become
NEWER_NONCURRENT.

* fix(s3/lifecycle): length-prefix RuleHash to remove delimiter ambiguity

The previous encoding used "tag=K=V\n" lines, which is ambiguous: a
tag (a=b, c) serializes identically to (a, b=c). A prefix containing
"\nexp_days=99" could likewise forge an action field. Either could
silently bind two semantically different rules to the same per-rule
state directory.

Switch to a length-prefixed canonical form: each scalar is written as
<field-tag-byte> <uvarint-length> <bytes>. Field-tag bytes namespace
each scalar so ("a=b" tag-key) and ("a" tag-key with "=b" leakage)
can't collide. Three regression tests pin the resistance.

* docs(s3/lifecycle): ActionKey is the engine identity, not rule_hash

Finishes the per-action restructure across the engine pseudocode,
bootstrap completion, drain locks, detector paths, policy CAS, and
Phase 2 plan. Every per-action data structure — engine indexes,
target modes, newly-completed sets, bootstrap completion bits, drain
keys, locks, metrics, status — is now keyed by
ActionKey{rule_hash, action_kind}, not by rule_hash alone.

Without this, sibling actions under one XML rule still shared
scheduling state in the engine: originalDelayGroups holding
[]ruleHash means a rule's 7d AbortMPU and 90d ExpirationDays
collapse into one entry, the smaller delay wins for cursor advance,
and the larger sibling never re-fires.

Helper API tightened: EvaluateAction(rule, kind, info, now) and
ComputeDueAt(rule, kind, info) replace the aggregate-rule signatures
so a caller asks one specific action's eligibility against one
entry, never "any action of this rule." Drain task lock includes
action_kind so siblings have independent re-arm timers. Policy CAS
moves from rule_hash to ActionKey membership.

* fix(s3/lifecycle): kind-aware EvaluateAction / ComputeDueAt

Replaces the aggregate-rule signatures with per-action ones:
EvaluateAction(rule, kind, info, now) and ComputeDueAt(rule, kind,
info). The old Evaluate(rule, info, now) could return the verdict of
ANY action declared on the rule, which sat wrong with the per-action
engine indexing (each ActionKey has its own delay group, mode, and
pending stream).

Each helper now decides exactly one (rule, kind) compiled action's
fate against one entry. Asking for a kind the rule doesn't declare,
or asking against the wrong object shape for that kind, returns
ActionNone / zero — never silently routes to a sibling.

Multi-action regression test: a rule with ExpirationDays=90 and
AbortMPU=7 evaluates each kind independently for the same entry; the
7d window has no influence on the 90d eligibility decision.
 2026-05-07 10:02:32 -07:00
Minsoo Kim a1e5eb9dad Fix UI prefix url encoding (#9344) 
* Fix filer UI navigation for URL-sensitive object prefixes

* Fix filer UI navigation for URL-sensitive object prefixes

* Clarify filer UI path escaping test name

Rename the legacy filer UI
  path test to describe the actual behavior being checked.

  The printpath helper preserves timestamp characters that are valid in URL path
  components, while the PR fix is focused on query-string escaping for path and cursor
  parameters.
 2026-05-06 19:14:36 -07:00
Chris Lu 487b93eb49 fix(volume): don't panic on read when needle map is nil (#9342) 
* fix(volume): don't panic on read when needle map is nil

A failed CommitCompact reload (and #9335's new error path for a
remote-tiered volume with a stray .vif but no .idx) leaves v.nm == nil
on a volume that's still in the store. readNeedle / readNeedleDataInto
dereferenced v.nm with no guard, so the next GET segfaulted the
http handler instead of returning an error the client could retry on
another replica.

Add the same v.nm == nil check the other Volume accessors already use,
including the slow-read inner loop where the lock is released between
iterations and a failed reload can race in.

Fixes #9339.

* match rust nm-nil read behavior; trim comments

seaweed-volume's read_needle_with_option / re_lookup_needle_data_offset
already lift Option<NeedleMap> through ok_or(NotFound). Use ErrorNotFound
on the Go side too instead of a generic 500-mapped error so both volume
servers respond identically when v.nm is nil.

* log once when reads hit nil needle map

ErrorNotFound alone hides the real cause: a half-loaded volume just
returns 404s and the operator has nothing to grep for. Add a once-per-
volume Errorf on the nil path, reset on successful load. Mirror the
same in seaweed-volume via nm_or_not_found().

* trim comments

* drop once-flag, log inline on every nil-nm read
 2026-05-06 18:23:06 -07:00
Chris Lu e96190d128 fix(mount): skip pressure-eviction of gappy page chunks (#9330) (#9334) 
* fix(mount): skip pressure-eviction of gappy page chunks (#9330)

A page chunk whose written-interval list has an internal hole was being
sealed under buffer-pressure eviction, then SaveContent would emit one
volume chunk per maximal adjacent run with no chunk covering the hole;
reads then silently zero-fill the gap (filer/stream.go:177-186). On a
sequential cp through FUSE, that bakes in-flight 4 KiB writes into split
volume chunks and leaves chunk-sized blocks of zeros on the destination.

Filter the pressure-driven sealers (SaveDataAt's over-limit path,
EvictOneWritableChunk, ProactiveFlush) to only seal chunks whose written
intervals form one unbroken run. The flush-on-close path (FlushAll) is
unchanged: at close every gap is by definition a sparse-file write that
the app legitimately never made.

* fix(mount): also gate IsContiguouslyWritten on leading zero-offset

Tighten IsContiguouslyWritten to also reject empty lists and lists whose
first interval does not start at offset 0. The internal-gap and
leading-gap cases are symmetric for pressure-driven sealing: both put
in-flight FUSE writeback for the missing range at risk of being baked
into split volume chunks. The flush-on-close path is still unfiltered
(sparse writes are sealed legitimately at FlushAll).

Also align EvictOneWritableChunk's bestBytes initialization with
SaveDataAt (start at 0) so an empty chunk is never picked, matching
the new semantic.

Addresses gemini-code-assist review on PR #9334.

* fix(mount): preserve cap-pressure liveness in EvictOneWritableChunk

The previous version of this fix had EvictOneWritableChunk return false
whenever every dirty chunk was gappy. That broke the accountant's
Reserve loop: cond.Wait only wakes on Release, Release only fires on
upload completion, and refusing to seal anything means no upload starts
— the writer hangs at the -writeBufferSizeMB cap forever.

Two-pass selection: prefer the fullest gap-free chunk (issue #9330: this
is what protects sequential cp from racing FUSE writeback), fall back to
the oldest non-empty writer when nothing is gap-free. Oldest-first
maximizes the chance that FUSE writeback for the gap range has already
settled. The actual sealing path is unchanged — SaveContent still emits
one volume chunk per maximal adjacent run; pages that arrive after the
seal land in a fresh MemChunk for the same logicChunkIndex and are
sealed in turn, so coverage is reconstructed at read time by
readResolvedChunks.

Sequential cp at default settings always hits the strict pass (writes
arrive contiguous-from-0 within their logicChunkIndex), so the bug-fix
behavior is preserved; the fallback only runs under genuinely sparse
workloads or under FUSE writeback so backed up that no chunk has
settled, where forced progress is preferable to a hung mount.

* test(mount): pin ProactiveFlush gap-skip behavior (#9330)

Sibling regression test for the ProactiveFlush guard added in this
series: same 3-chunk setup as TestEvictOneWritableChunk_SkipsGappyChunks
(internal gap, leading gap, contiguous). Verifies ProactiveFlush picks
the contiguous chunk when staleness criteria are otherwise satisfied,
returns false when only gappy chunks remain (no liveness fallback like
EvictOneWritableChunk has — failing here is just a missed optimization),
and that filling the holes lets the chunks auto-seal via maybeMoveToSealed.

* style(mount): trim verbose comments on #9330 fix
 2026-05-06 15:26:56 -07:00
Chris Lu 7b0b64db65 fix(admin/view): wrap plugin history URL with basePath (#9341) 
Plugin tabs/sub-tabs use history.pushState/replaceState to keep the
URL bar in sync with the active view, but updateURL fed it the raw
output of buildPluginURL ("/plugin/lanes/<lane>/..."). Under a
urlPrefix deployment that strips the prefix, so reloading the page
hit /plugin/... directly and 404'd at the proxy.

Wrap with basePath() so the rewritten URL keeps the deployment
prefix.

Reported at #9240.
 2026-05-06 15:25:06 -07:00
Chris Luandgemini-code-assist[bot] 1c0e24f06a fix(balance): don't move remote-tiered volumes; don't fatal on missing .idx (#9335) 
* fix(volume): don't fatal on missing .idx for remote-tiered volume

A .vif left behind without its .idx (orphaned by a crashed move, partial
copy, or hand-edit) would trip glog.Fatalf in checkIdxFile and take the
whole volume server down on boot, killing every healthy volume on it
too. For remote-tiered volumes treat it as a per-volume load error so
the server can come up and the operator can clean up the stray .vif.

Refs #9331.

* fix(balance): skip remote-tiered volumes in admin balance detection

The admin/worker balance detector had no equivalent of the shell-side
guard ("does not move volume in remote storage" in
command_volume_balance.go), so it scheduled moves on remote-tiered
volumes. The "move" copies .idx/.vif to the destination and then calls
Volume.Destroy on the source, which calls backendStorage.DeleteFile —
deleting the remote object the destination's new .vif now points at.

Populate HasRemoteCopy on the metrics emitted by both the admin
maintenance scanner and the worker's master poll, then drop those
volumes at the top of Detection.

Fixes #9331.

* Apply suggestion from @gemini-code-assist[bot]

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* fix(volume): keep remote data on volume-move-driven delete

The on-source delete after a volume move (admin/worker balance and
shell volume.move) ran Volume.Destroy with no way to opt out of the
remote-object cleanup. Volume.Destroy unconditionally calls
backendStorage.DeleteFile for remote-tiered volumes, so a successful
move would copy .idx/.vif to the destination and then nuke the cloud
object the destination's new .vif was already pointing at.

Add VolumeDeleteRequest.keep_remote_data and plumb it through
Store.DeleteVolume / DiskLocation.DeleteVolume / Volume.Destroy. The
balance task and shell volume.move set it to true; the post-tier-upload
cleanup of other replicas and the over-replication trim in
volume.fix.replication also set it to true since the remote object is
still referenced. Other real-delete callers keep the default. The
delete-before-receive path in VolumeCopy also sets it: the inbound copy
carries a .vif that may reference the same cloud object as the
existing volume.

Refs #9331.

* test(storage): in-process remote-tier integration tests

Cover the four operations the user is most likely to run against a
cloud-tiered volume — balance/move, vacuum, EC encode, EC decode — by
registering a local-disk-backed BackendStorage as the "remote" tier and
exercising the real Volume / DiskLocation / EC encoder code paths.

Locks in:
- Destroy(keepRemoteData=true) preserves the remote object (move case)
- Destroy(keepRemoteData=false) deletes it (real-delete case)
- Vacuum/compact on a remote-tier volume never deletes the remote object
- EC encode requires the local .dat (callers must download first)
- EC encode + rebuild round-trips after a tier-down

Tests run in-process and finish in under a second total — no cluster,
binary, or external storage required.

* fix(rust-volume): keep remote data on volume-move-driven delete

Mirror the Go fix in seaweed-volume: plumb keep_remote_data through
grpc volume_delete → Store.delete_volume → DiskLocation.delete_volume
→ Volume.destroy, and skip the s3-tier delete_file call when the flag
is set. The pre-receive cleanup in volume_copy passes true for the
same reason as the Go side: the inbound copy carries a .vif that may
reference the same cloud object as the existing volume.

The Rust loader already warns rather than fataling on a stray .vif
without an .idx (volume.rs load_index_inmemory / load_index_redb), so
no counterpart to the Go fatal-on-missing-idx fix is needed.

Refs #9331.

* fix(volume): preserve remote tier on IO-error eviction; fix EC test target

Two review nits:

- Store.MaybeAddVolumes' periodic cleanup pass deleted IO-errored
  volumes with keepRemoteData=false, so a transient local fault on a
  remote-tiered volume would also nuke the cloud object. Track the
  delete reason via a parallel slice and pass keepRemoteData=v.HasRemoteFile()
  for IO-error evictions; TTL-expired evictions still pass false.

- TestRemoteTier_ECEncodeDecode_AfterDownload deleted shards 0..3 but
  called them "parity" — by the klauspost/reedsolomon convention shards
  0..DataShardsCount-1 are data and DataShardsCount..TotalShardsCount-1
  are parity. Switch the loop to delete the parity range so the
  intent matches the indices.

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-06 15:19:43 -07:00
Chris Lu 12f283357f fix(iam): four phase-3 follow-ups (provider scoping, public path wrapper, static mirror, claim-mode RoleArn) (#9333) 
* fix(iam): scope IAM-managed OIDC provider lookup by role account

Two account-scoped OIDC records sharing an issuer were collapsed into a
single map slot keyed only by the URL. The last-write-wins entry then
served every AssumeRoleWithWebIdentity, so a token destined for account
B's role could be validated by account A's record (its clientIDs and
thumbprints), defeating the per-account isolation the records exist for.

The role-account check in enforceProviderAccountScope still rejected
the cross-account assumption, but only after the wrong record's
audience and TLS pin had already accepted the token.

Refresh now keys IAM-managed records as (issuer, account), and
validation parses the requested role's account up front and matches
the record under that issuer in this order: exact account, global
(account-less), static-config fallback. An unknown account hint
deliberately skips account-scoped entries — picking one arbitrarily is
the bug this commit fixes — and falls through to global or static.

* fix(iam): route public AssumeRoleWithWebIdentity through IAMManager

handleAssumeRoleWithWebIdentity called stsService.AssumeRoleWithWebIdentity
directly, bypassing the IAMManager wrapper. The wrapper is where
enforceProviderAccountScope rejects cross-account assumption attempts
and capDurationByRole clamps to the role's MaxSessionDuration; both
silently became no-ops for any AWS-SDK caller hitting the public
endpoint.

Dispatch through the IAMManager (via the existing IAMManagerProvider
interface that other handlers in this file already use) when one is
wired. Embedded test setups without an IAM integration fall back to
the bare STS service unchanged.

* fix(iam): mirror thumbprints, principal-tag keys, and policy claim from static OIDC config

initOIDCProviderStore mirrored only URL and ClientIDs. Once
RefreshOIDCProvidersFromStore ran (on any IAM-managed mutation, or on
boot once the metadata-subscribe loop kicked in),
buildOIDCProviderFromRecord rebuilt the runtime provider from this
truncated record. Because IAM-managed entries take precedence over the
static-config map, the rebuild silently shadowed the bootstrap with a
weaker provider:

- Thumbprints: dropped, so TLS-pinned issuers fell back to the system
  trust store.
- AllowedPrincipalTagKeys: dropped, so principal-tag claims stopped
  reaching the session.
- PolicyClaim: dropped, so claim-based policy mode stopped triggering.

Pull all three from the provider's static Config map at mirror time so
the stored record round-trips to a runtime provider equivalent to the
one the static config produced directly.

* fix(iam): allow empty RoleArn in AssumeRoleWithWebIdentity HTTP handler

Phase 3b advertises that RoleArn MAY be omitted in claim-based policy
mode — the STS service then derives the assumed-role ARN from the
configured policy claim. The HTTP handler still rejected empty RoleArn
up front with MissingParameter, so SDK callers using the documented
omitted-role flow never reached the STS layer.

Drop the pre-check; STS still validates that claim-based mode is
configured and that the IDP emits policies, returning a precise error
when either is missing. The existing error mapping below this point
surfaces those as InvalidParameterValue, matching what an AWS SDK
expects.

* test(iam): update missing-RoleArn STS integration test for the new contract

The previous commit drops the HTTP-layer RoleArn pre-check so claim-based
mode can derive the ARN from a JWT claim. The integration test still
asserted MissingParameter for the missing-RoleArn case, which now
reaches the STS layer and surfaces a JWT-parse error instead. Update
the assertion to match: missing RoleArn alone must no longer surface
as MissingParameter, but a bogus JWT must still be rejected.
 2026-05-05 19:14:44 -07:00
Chris Lu 9af1b212d3 feat(iam): OIDC provider audit trail (Phase 3e) (#9325) 
* feat(iam): claim-based policy mode for AssumeRoleWithWebIdentity

When the caller passes the sentinel RoleArn arn:aws:iam:::role/sts-claim-based
(or omits it entirely) and the matched OIDC provider has policyClaim set,
mint a session whose effective policies come from that JWT claim instead
of from a server-side role mapping. Accepts string, comma-separated
string, or array shapes — MinIO-compatible behaviour for IDPs that
already attach policies to the user.

Trust-policy validation is skipped in claim-mode: the IDP is the sole
authority for both authentication and authorization, mirroring the
contract MinIO documents for its DummyRoleARN flow. Concrete-role mode
is unchanged and still requires the role definition + trust policy.

* fix(iam): trim policy-claim array elements + clean up stale comments

Three medium-priority cleanups gemini flagged on the claim-based path:

- extractClaimPolicies's array branch was leaving whitespace on each
  element while the string/comma-separated branch trimmed via
  splitPolicyClaimString. An IDP that emits ["readonly", " billing "]
  would create a "billing" policy lookup that didn't match the stored
  name. Trim every array element, drop empties.
- The "synthetic ARN keyed on the session name" comment was wrong —
  effectiveRoleArn here is the literal sentinel; it's the assumed-role
  ARN generated downstream that's session-keyed. Reword.
- The empty if/else block at the start of
  validateAssumeRoleWithWebIdentityRequest existed only to host a
  comment about deferred validation; the comment now lives in the
  function godoc and the empty branch is gone.

Addresses three gemini medium reviews on PR #9322.

* feat(iam): account-scoped OIDC providers

Add OIDCProviderRecord.AccountID enforcement: when a role lives in
account A, the OIDC provider validating the assume-role token must be
either global (AccountID="") or also live in account A. Cross-account
use is rejected at the IAM-manager layer before reaching the trust
policy validator.

OIDCProviderStore gains GetProviderByIssuerAndAccount; both the in-
memory and filer-backed stores implement it. Static-config-only
deployments are unaffected since they don't populate the store.

* fix(iam): account-scoped lookup for cross-account check

enforceProviderAccountScope was calling GetProviderByIssuer, which
returns the first match arbitrarily when multiple providers share an
issuer (one global + one per tenant is the canonical setup). On a
two-record collision the wrong record could come back first and
falsely reject a valid same-account or global-provider request.

Use GetProviderByIssuerAndAccount as the primary lookup so the
filter happens in the store. On miss, fall back to
GetProviderByIssuer purely to distinguish "issuer entirely unknown"
(let the STS layer reject) from "issuer registered in a different
account" (surface a precise cross-account error).

Addresses gemini high-priority review on PR #9323.

* feat(iam): opt-in session revocation via JTI blocklist

Add SessionRevocationStore (memory + filer implementations) and wire
it into the IAMManager.IsActionAllowed path so a revoked session is
rejected on the next signed request. Session JWTs now embed the
session id as the JTI claim, giving the blocklist a stable key
without requiring a second secret.

Operators who don't configure a store keep the existing fully-stateless
behaviour: every session stays valid until natural expiry. Operators
who do configure one accept one filer lookup per signed request in
exchange for being able to invalidate compromised tokens before
expiry. Revocation entries carry the original session expiry so the
blocklist self-trims via PurgeRevokedSessions.

* fix(iam): hash JTI filenames + paginate Purge with proper EOF handling

Three reviewer-flagged issues on the filer-backed revocation store:

1. Path traversal (security-medium): RevokeSession is exported and takes
   an arbitrary string. Using the JTI verbatim as a filename meant a
   caller could pass "../../etc/passwd" to write outside the basePath.
   SHA-1 hash the JTI to a fixed-width hex name; lookups still find the
   entry because Revoke and IsRevoked share the same hash function.

2. Purge swallowed errors. The inner `err` from stream.Recv() shadowed
   the outer err and the loop just broke on any failure, so a mid-stream
   gRPC error returned (count, nil) and the caller had no idea the
   purge was incomplete. Switch to errors.Is(io.EOF) for end-of-stream
   and propagate everything else.

3. Purge had a hardcoded 10000-entry cap. Stream-paginate via
   StartFromFileName so the operator-cron can clean a backlog larger
   than that without losing rows.

* feat(iam): OIDC provider audit trail

Emit one structured event per IAM-managed OIDC provider lifecycle
mutation (Create, Delete, Add/Remove ClientID, UpdateThumbprints,
Tag, Untag). Three sinks ship in-tree:

- GlogAuditSink — default; events become structured log lines.
- MemoryAuditSink — in-process buffer for tests / inspection.
- FilerAuditSink — durable, one file per event under
  /etc/iam/audit/oidc-providers (operator-overridable basePath).

Audit emission is best-effort: a failing sink never blocks an IAM
mutation that has already succeeded. Use-events (per token validation)
are intentionally not emitted yet — too hot for unconditional sinks.

* fix(iam): collision-free audit filenames + correct file mtime

Two issues gemini flagged on FilerAuditSink.Emit:

1. Filename was %d-%s.json (UnixNano + Type). Two mutations at the
   same nano against different ARNs (think batch script touching
   several providers) collided on the same path and the second
   CreateEntry failed silently. Append a short ARN hash to make the
   name unique per-event without leaking the ARN.

2. File Mtime/Crtime were set to time.Now() at write time. For audit
   integrity those should reflect the event's occurrence time so
   filer-level "ls -lt" output matches the contents.

Addresses three medium-priority gemini reviews on PR #9325.
 2026-05-05 13:37:24 -07:00
Chris Lu 9d6a699b94 feat(iam): opt-in session revocation via JTI blocklist (Phase 3d) (#9324) 
* feat(iam): claim-based policy mode for AssumeRoleWithWebIdentity

When the caller passes the sentinel RoleArn arn:aws:iam:::role/sts-claim-based
(or omits it entirely) and the matched OIDC provider has policyClaim set,
mint a session whose effective policies come from that JWT claim instead
of from a server-side role mapping. Accepts string, comma-separated
string, or array shapes — MinIO-compatible behaviour for IDPs that
already attach policies to the user.

Trust-policy validation is skipped in claim-mode: the IDP is the sole
authority for both authentication and authorization, mirroring the
contract MinIO documents for its DummyRoleARN flow. Concrete-role mode
is unchanged and still requires the role definition + trust policy.

* fix(iam): trim policy-claim array elements + clean up stale comments

Three medium-priority cleanups gemini flagged on the claim-based path:

- extractClaimPolicies's array branch was leaving whitespace on each
  element while the string/comma-separated branch trimmed via
  splitPolicyClaimString. An IDP that emits ["readonly", " billing "]
  would create a "billing" policy lookup that didn't match the stored
  name. Trim every array element, drop empties.
- The "synthetic ARN keyed on the session name" comment was wrong —
  effectiveRoleArn here is the literal sentinel; it's the assumed-role
  ARN generated downstream that's session-keyed. Reword.
- The empty if/else block at the start of
  validateAssumeRoleWithWebIdentityRequest existed only to host a
  comment about deferred validation; the comment now lives in the
  function godoc and the empty branch is gone.

Addresses three gemini medium reviews on PR #9322.

* feat(iam): account-scoped OIDC providers

Add OIDCProviderRecord.AccountID enforcement: when a role lives in
account A, the OIDC provider validating the assume-role token must be
either global (AccountID="") or also live in account A. Cross-account
use is rejected at the IAM-manager layer before reaching the trust
policy validator.

OIDCProviderStore gains GetProviderByIssuerAndAccount; both the in-
memory and filer-backed stores implement it. Static-config-only
deployments are unaffected since they don't populate the store.

* fix(iam): account-scoped lookup for cross-account check

enforceProviderAccountScope was calling GetProviderByIssuer, which
returns the first match arbitrarily when multiple providers share an
issuer (one global + one per tenant is the canonical setup). On a
two-record collision the wrong record could come back first and
falsely reject a valid same-account or global-provider request.

Use GetProviderByIssuerAndAccount as the primary lookup so the
filter happens in the store. On miss, fall back to
GetProviderByIssuer purely to distinguish "issuer entirely unknown"
(let the STS layer reject) from "issuer registered in a different
account" (surface a precise cross-account error).

Addresses gemini high-priority review on PR #9323.

* feat(iam): opt-in session revocation via JTI blocklist

Add SessionRevocationStore (memory + filer implementations) and wire
it into the IAMManager.IsActionAllowed path so a revoked session is
rejected on the next signed request. Session JWTs now embed the
session id as the JTI claim, giving the blocklist a stable key
without requiring a second secret.

Operators who don't configure a store keep the existing fully-stateless
behaviour: every session stays valid until natural expiry. Operators
who do configure one accept one filer lookup per signed request in
exchange for being able to invalidate compromised tokens before
expiry. Revocation entries carry the original session expiry so the
blocklist self-trims via PurgeRevokedSessions.

* fix(iam): hash JTI filenames + paginate Purge with proper EOF handling

Three reviewer-flagged issues on the filer-backed revocation store:

1. Path traversal (security-medium): RevokeSession is exported and takes
   an arbitrary string. Using the JTI verbatim as a filename meant a
   caller could pass "../../etc/passwd" to write outside the basePath.
   SHA-1 hash the JTI to a fixed-width hex name; lookups still find the
   entry because Revoke and IsRevoked share the same hash function.

2. Purge swallowed errors. The inner `err` from stream.Recv() shadowed
   the outer err and the loop just broke on any failure, so a mid-stream
   gRPC error returned (count, nil) and the caller had no idea the
   purge was incomplete. Switch to errors.Is(io.EOF) for end-of-stream
   and propagate everything else.

3. Purge had a hardcoded 10000-entry cap. Stream-paginate via
   StartFromFileName so the operator-cron can clean a backlog larger
   than that without losing rows.
 2026-05-05 13:25:22 -07:00
Chris Lu 6483583491 feat(iam): account-scoped OIDC providers (Phase 3c) (#9323) 
* feat(iam): claim-based policy mode for AssumeRoleWithWebIdentity

When the caller passes the sentinel RoleArn arn:aws:iam:::role/sts-claim-based
(or omits it entirely) and the matched OIDC provider has policyClaim set,
mint a session whose effective policies come from that JWT claim instead
of from a server-side role mapping. Accepts string, comma-separated
string, or array shapes — MinIO-compatible behaviour for IDPs that
already attach policies to the user.

Trust-policy validation is skipped in claim-mode: the IDP is the sole
authority for both authentication and authorization, mirroring the
contract MinIO documents for its DummyRoleARN flow. Concrete-role mode
is unchanged and still requires the role definition + trust policy.

* fix(iam): trim policy-claim array elements + clean up stale comments

Three medium-priority cleanups gemini flagged on the claim-based path:

- extractClaimPolicies's array branch was leaving whitespace on each
  element while the string/comma-separated branch trimmed via
  splitPolicyClaimString. An IDP that emits ["readonly", " billing "]
  would create a "billing" policy lookup that didn't match the stored
  name. Trim every array element, drop empties.
- The "synthetic ARN keyed on the session name" comment was wrong —
  effectiveRoleArn here is the literal sentinel; it's the assumed-role
  ARN generated downstream that's session-keyed. Reword.
- The empty if/else block at the start of
  validateAssumeRoleWithWebIdentityRequest existed only to host a
  comment about deferred validation; the comment now lives in the
  function godoc and the empty branch is gone.

Addresses three gemini medium reviews on PR #9322.

* feat(iam): account-scoped OIDC providers

Add OIDCProviderRecord.AccountID enforcement: when a role lives in
account A, the OIDC provider validating the assume-role token must be
either global (AccountID="") or also live in account A. Cross-account
use is rejected at the IAM-manager layer before reaching the trust
policy validator.

OIDCProviderStore gains GetProviderByIssuerAndAccount; both the in-
memory and filer-backed stores implement it. Static-config-only
deployments are unaffected since they don't populate the store.

* fix(iam): account-scoped lookup for cross-account check

enforceProviderAccountScope was calling GetProviderByIssuer, which
returns the first match arbitrarily when multiple providers share an
issuer (one global + one per tenant is the canonical setup). On a
two-record collision the wrong record could come back first and
falsely reject a valid same-account or global-provider request.

Use GetProviderByIssuerAndAccount as the primary lookup so the
filter happens in the store. On miss, fall back to
GetProviderByIssuer purely to distinguish "issuer entirely unknown"
(let the STS layer reject) from "issuer registered in a different
account" (surface a precise cross-account error).

Addresses gemini high-priority review on PR #9323.
 2026-05-05 13:06:53 -07:00
Chris Lu bc1d458fe6 fix(iam): reject empty issuer in ComputeParentUser (#9326) 
Without iss, the same `sub` from two different IDPs would collapse to
the same parent_user hash. Short-circuit to empty when either input
is missing so callers see "no identity" instead of a colliding hash.

Addresses coderabbit review on PR #9318.
 2026-05-05 13:01:33 -07:00
Chris Lu 1d3454ca5c feat(iam): claim-based policy mode for AssumeRoleWithWebIdentity (Phase 3b) (#9322) 
* feat(iam): claim-based policy mode for AssumeRoleWithWebIdentity

When the caller passes the sentinel RoleArn arn:aws:iam:::role/sts-claim-based
(or omits it entirely) and the matched OIDC provider has policyClaim set,
mint a session whose effective policies come from that JWT claim instead
of from a server-side role mapping. Accepts string, comma-separated
string, or array shapes — MinIO-compatible behaviour for IDPs that
already attach policies to the user.

Trust-policy validation is skipped in claim-mode: the IDP is the sole
authority for both authentication and authorization, mirroring the
contract MinIO documents for its DummyRoleARN flow. Concrete-role mode
is unchanged and still requires the role definition + trust policy.

* fix(iam): trim policy-claim array elements + clean up stale comments

Three medium-priority cleanups gemini flagged on the claim-based path:

- extractClaimPolicies's array branch was leaving whitespace on each
  element while the string/comma-separated branch trimmed via
  splitPolicyClaimString. An IDP that emits ["readonly", " billing "]
  would create a "billing" policy lookup that didn't match the stored
  name. Trim every array element, drop empties.
- The "synthetic ARN keyed on the session name" comment was wrong —
  effectiveRoleArn here is the literal sentinel; it's the assumed-role
  ARN generated downstream that's session-keyed. Reword.
- The empty if/else block at the start of
  validateAssumeRoleWithWebIdentityRequest existed only to host a
  comment about deferred validation; the comment now lives in the
  function godoc and the empty branch is gone.

Addresses three gemini medium reviews on PR #9322.
 2026-05-05 12:21:55 -07:00
Chris Lu 6554ab7928 feat(iam): principal session tags from OIDC tokens (Phase 3a) (#9321) 
* feat(iam): principal session tags from OIDC tokens

Extract the AWS principal-tags namespace claim
(`https://aws.amazon.com/tags/principal_tags`) from validated OIDC
tokens, filter through a per-provider AllowedPrincipalTagKeys allowlist,
and surface as `aws:PrincipalTag/<key>` in the STS session request
context. Empty allowlist means "no tags surfaced" — operators must opt
keys in explicitly so a misconfigured IDP can't pollute policy
evaluation.

Policy engine now accepts `aws:PrincipalTag/...` and
`aws:RequestTag/...` as substitutable variables so resource-level
ABAC policies can reference them.

* fix(iam): case-insensitive principal-tag allowlist + sharper comment

filterPrincipalTags compared keys case-sensitively, but AWS IAM
session tag keys are case-insensitive (the docs are explicit). An
IDP whose claim casing drifts from the operator-configured allowlist
string would silently filter the value out — surprising failure mode.
Lowercase both sides during the lookup; the original key casing is
preserved on the output so policy variables still match what the
caller sees.

Also reword the "anything the IDP signs is acceptable" comment in
sts_service.go: it predates the per-provider allowlist that's
already filtering before this point. The reality is now "everything
reaching here is on the operator's opt-in list, dropped entirely if
the allowlist is empty."

Addresses two gemini medium reviews on PR #9321.
 2026-05-05 11:43:05 -07:00
Chris Lu f8973b3ed6 feat(iam): OIDC provider mutations + multi-client + TLS thumbprints (Phase 2b) (#9320) 
* feat(iam): OIDC provider mutations + multi-client + TLS thumbprints

- Mutating IAM actions: CreateOpenIDConnectProvider,
  DeleteOpenIDConnectProvider, AddClientIDToOpenIDConnectProvider,
  RemoveClientIDFromOpenIDConnectProvider,
  UpdateOpenIDConnectProviderThumbprint, TagOpenIDConnectProvider,
  UntagOpenIDConnectProvider. Each enforces AWS-shape input bounds and
  the read-only mode rejects all mutations.
- Multiple client_ids per provider in OIDCConfig (clientIds list, plural)
  with full backward compatibility — singular clientId still works and is
  merged into the audience allowlist. Provider factory accepts both.
- AWS-compatible TLS thumbprint pinning: when OIDCConfig.Thumbprints is
  non-empty, JWKS fetches enforce that the negotiated TLS chain contains
  a certificate whose SHA-1 hex matches the allowlist. Empty list keeps
  the existing system-trust path.

* fix(iam): factor Tags.member.N parser into a helper

CreateOpenIDConnectProvider and TagOpenIDConnectProvider were both
walking the AWS Tags.member.N.Key / Tags.member.N.Value query-string
convention with copy-pasted loops. Factor into extractTags so the
parsing rules and the "no tags present" semantics live in one place.

Addresses gemini medium review on PR #9320.

* fix(iam): sentinel errors for OIDC provider not-found / already-exists

The s3api dispatcher was using strings.Contains(err.Error(), "not found")
and "already exists" to map IAM-manager errors back to AWS error codes.
Substring matching on a formatted message couples the API error code
to the exact wording of the upstream message — touching the message
silently changes the IAM API contract.

Define ErrOIDCProviderNotFound and ErrOIDCProviderAlreadyExists in the
integration package, fmt.Errorf("%w: ...") them at the four return
sites in iam_manager.go and oidc_provider_store.go, and use errors.Is
at the s3api call sites. Same control flow, no string-match fragility.

Addresses gemini medium review on PR #9320.

* fix(iam): surface non-NotFound errors from CreateOIDCProvider lookup

Previously CreateOIDCProvider only treated GetProviderByARN's success path
as "exists" and silently fell through on any error, including transient
backend failures. That hid real problems and still attempted a write.
Distinguish ErrOIDCProviderNotFound (the only "safe to create" case) from
other errors so we don't mask filer outages or partition issues.

* fix(iam): enforce 100-client-ID cap on AddClientIDToOIDCProvider

CreateOIDCProvider and the implicit update path through validateOIDC-
ProviderRecord both reject lists with more than 100 client IDs, but
AddClientIDToOIDCProvider could grow the list past that bound one
ID at a time. Refuse the add when the list is already at the cap so
the invariant holds across every mutation entry point.

* feat(iam): IAM-managed OIDC provider live view in STS service

Add a separate, mutex-guarded map of admin-managed OIDC providers on
the STS service. The map can be atomically replaced via
SetIAMManagedOIDCProvidersByIssuer; AssumeRoleWithWebIdentity lookups
consult it first and fall back to the existing static-config map, so
records persisted through the IAM API can shadow bootstrap entries
without a restart.

This is the runtime hook the IAM API and the metadata-subscribe path
will both call when the OIDCProviderStore changes (next two commits).

* feat(iam): refresh STS service runtime view after OIDC mutations

Add IAMManager.RefreshOIDCProvidersFromStore: lists every persisted
OIDCProviderRecord, builds a runtime OIDCProvider for each, and atomically
publishes the issuer-keyed map into the STS service. Each mutating IAM API
call (Create / Delete / AddClientID / RemoveClientID / UpdateThumbprints)
now triggers this refresh inline so the local instance picks up the change
without waiting for a metadata-subscribe round trip. Tag mutations skip
the refresh because tags do not affect token validation.

Refresh failures only log; the persisted write has already succeeded by
that point, so a transient list error must not surface to the API caller.
The peer-instance update path (filer metadata subscription) is added in a
follow-up commit.

* feat(iam): subscribe to OIDC provider changes on the filer

Watch /etc/iam/oidc-providers under the existing s3 metadata-subscribe
loop and call RefreshOIDCProvidersFromStore on any create / update /
delete / rename. This is the cross-instance update path: S3 server A
writes via the IAM API, the filer fans out the metadata change, and S3
servers B..N pick up the new runtime view without a restart.

Mirrors the existing onIamConfigChange / onCircuitBreakerConfigChange
pattern. The handler short-circuits when the path is unrelated, and
when no IAMManager is wired in (static-only configurations).
 2026-05-05 11:26:08 -07:00
Chris Lu 6141222ab0 fix(test/s3/policy): allocate fresh admin port per subtest (#9332) 
* fix(test/s3/policy): allocate fresh admin port per subtest

startMiniCluster ran weed mini in-process and explicitly assigned
master/volume/filer/s3 ports allocated by MustAllocatePorts, but it
left -admin.port and -admin.port.grpc unset, so each subtest reused
the hardcoded defaults 23646 / 33646.

The package's subtests run sequentially within the same go test
process. The previous subtest's admin goroutine is still bound to
23646 by the time the next subtest spins up its own mini, so the
new admin can never bind, mini.go's waitForAdminServerReady hits
its 240-attempt cap, and glog.Fatalf kills the test binary. This
has been the dominant cause of "admin server did not become ready"
flakes across recent IAM PRs.

Allocate two extra ports for admin and pass them through. The other
subprocess-based tests (s3tables/*) are not affected because each
launches weed mini in a fresh OS process.

* fix(mini): make admin readiness wait context-aware

waitForAdminServerReady polled for 240 attempts × 500ms regardless of
whether the surrounding mini context was cancelled. When mini is run
in-process from a test harness (test/s3/policy/...) and the test calls
its cancel func, the leftover wait keeps spinning for the full two
minutes and then glog.Fatalf's, terminating the entire test binary —
including any sibling subtest that has since started its own mini.

Thread the existing miniClientsCtx through the wait so a Stop / cancel
returns context.Canceled immediately. The caller (startMiniAdminWithWorker)
treats a context-cancelled outcome as a graceful shutdown signal and
logs+returns instead of fataling.
 2026-05-05 11:24:43 -07:00
Chris Lu 95560076e6 fix(mini): raise admin readiness timeout to 2 minutes (#9329) 
The 30-second ceiling on waitForAdminServerReady was too tight on busy
CI runners. master + filer + volume + admin all start in parallel on a
shared worker, and S3 Policy Shell Integration Tests has been flaking
across multiple PRs with "admin server did not become ready... after
60 attempts" even though the server still comes up within a minute or
two. Two minutes (240 attempts at 500ms) leaves headroom for runner
contention without being absurd in a local-dev run.
 2026-05-05 07:59:25 -07:00
Chris Lu 4ded97a321 feat(iam): OIDC provider store + read-only IAM API (Phase 2a) (#9319) 
* feat(iam): STS web-identity AWS-fidelity polish

- OIDC discovery via .well-known/openid-configuration; falls back to
  /.well-known/jwks.json when discovery is absent. Reject discovery docs
  whose issuer claim does not match the configured issuer to defend
  against issuer-substitution.
- ComputeParentUser derives a stable per-identity hash from (sub, iss).
  Surface as aws:userid in the request context and as a parent_user
  claim in the session JWT so per-user state survives token rotation.
- Per-role MaxSessionDuration (3600..43200) clamps requested
  DurationSeconds before the STS service applies its own caps.
- Tighten RoleSessionName to the AWS contract: 2..64 chars from
  [\w+=,.@-].
- Populate PackedPolicySize in AssumeRole / AssumeRoleWithWebIdentity /
  AssumeRoleWithLDAPIdentity responses as a percentage of the 2048-byte
  inline session policy budget.

* fix(iam): leave omitted DurationSeconds nil so STS default applies

capDurationByRole was substituting the role's MaxSessionDuration
when the caller omitted DurationSeconds entirely. AWS returns the
configured default (typically 1 hour) in that case, not the role's
upper bound — a 12h MaxSessionDuration shouldn't silently make every
no-duration assume-role mint a 12h session.

Return nil when requested is nil; let the downstream
calculateSessionDuration in the STS service apply its TokenDuration
default. The role-max upper bound still clamps when the request
arrives with a concrete value above the cap.

Addresses gemini high-priority review on PR #9318.

* fix(iam): synchronize OIDCProvider JWKS cache fields

jwksCache, jwksFetchedAt, resolvedJWKSUri, and discoveryFailed are
mutated lazily on the first token-validate call and refreshed
afterwards on TTL expiry. Multiple S3 requests can land here in
parallel, so the writes were racing against subsequent reads on
every other goroutine. resolvedJWKSUri/discoveryFailed inherited
the same un-protected pattern when discovery shipped.

Add sync.RWMutex; getPublicKey takes the read lock for the
common cache-hit path and promotes to the write lock for misses
+ refreshes. fetchJWKSLocked / resolveJWKSUriLocked assume the
write lock is held by the caller; fetchJWKS keeps the
test-friendly entry point that acquires the lock itself.

Addresses gemini high-priority review on PR #9318.

* fix(iam): trim trailing slash + retry discovery after transient failure

Two OIDC discovery edge cases reviewers flagged:

1. Issuer comparison was sensitive to trailing slashes. resolveJWKSUri
   trims them when building the discovery URL, but the doc.Issuer ↔
   p.config.Issuer check did not, so an IDP whose issuer claim drops or
   adds the slash relative to the configured value would be falsely
   rejected. Trim a single trailing slash on each side before comparing.

2. discoveryFailed flipped to true on any error and stayed there for the
   process lifetime. A transient 5xx at startup permanently locked the
   provider into the /.well-known/jwks.json fallback. Reset the flag at
   the top of fetchJWKSLocked when no URI has been cached yet, so each
   JWKS refresh (typically once per TTL = 1h) reattempts discovery.
   Successful discovery remains cached via resolvedJWKSUri so we don't
   pay the discovery RTT on every refresh.

Addresses gemini security-medium + medium reviews on PR #9318.

* fix(iam): require non-empty issuer in OIDC discovery doc

The previous "doc.Issuer != "" && ..." guard let a discovery document
that omitted the issuer field bypass the issuer-mismatch check
entirely, letting the doc steer fetchJWKS at any URL it provided.
OIDC Discovery 1.0 §3 mandates the issuer field; treat missing as a
hard failure same as mismatched. Trailing-slash equivalence still
applies.

Adds TestDiscoveryRejectsMissingIssuer alongside the existing
TestDiscoveryRejectsIssuerMismatch via a new omitDiscoveryIssuer
toggle on fakeIDP.

* feat(iam): OIDC provider store + read-only IAM API

Add OIDCProviderRecord — the persisted, IAM-managed view of an OIDC
identity provider — and an OIDCProviderStore interface with memory and
filer implementations mirroring the existing role-store pattern.

The store is hydrated at boot from the static STS.Providers list so the
new IAM API surfaces the same set the STS service already validates
against. Two read-only actions land now:

- ListOpenIDConnectProviders -> ARN-only list, AWS-shape XML.
- GetOpenIDConnectProvider   -> URL, ClientIDList, ThumbprintList,
                                Tags, CreateDate.

Mutations (Create/Delete/Add-Remove ClientID/Update Thumbprint), multiple
client_ids per provider, and TLS thumbprint pinning come in Phase 2b.

* fix(iam): preserve CreatedAt across boots + paginate ListProviders

Two medium-priority issues gemini flagged on the read-only IAM API:

1. The static-config bootstrap was setting CreatedAt = time.Now() on
   every server start, so the IAM GetOpenIDConnectProvider response's
   CreateDate shifted on each restart even when backed by a persistent
   store. Look up the existing record via GetProviderByARN first and
   preserve its CreatedAt; only the UpdatedAt advances.

2. FilerOIDCProviderStore.ListProviders had a hardcoded Limit: 1000
   that silently truncated above that. Stream-paginate via
   StartFromFileName, returning io.EOF naturally and surfacing all
   other errors instead of swallowing them.

Addresses two gemini medium reviews on PR #9319.
 2026-05-04 22:15:03 -07:00
Chris Lu d951a8df5a feat(iam): STS web-identity AWS-fidelity polish (Phase 1) (#9318) 
* feat(iam): STS web-identity AWS-fidelity polish

- OIDC discovery via .well-known/openid-configuration; falls back to
  /.well-known/jwks.json when discovery is absent. Reject discovery docs
  whose issuer claim does not match the configured issuer to defend
  against issuer-substitution.
- ComputeParentUser derives a stable per-identity hash from (sub, iss).
  Surface as aws:userid in the request context and as a parent_user
  claim in the session JWT so per-user state survives token rotation.
- Per-role MaxSessionDuration (3600..43200) clamps requested
  DurationSeconds before the STS service applies its own caps.
- Tighten RoleSessionName to the AWS contract: 2..64 chars from
  [\w+=,.@-].
- Populate PackedPolicySize in AssumeRole / AssumeRoleWithWebIdentity /
  AssumeRoleWithLDAPIdentity responses as a percentage of the 2048-byte
  inline session policy budget.

* fix(iam): leave omitted DurationSeconds nil so STS default applies

capDurationByRole was substituting the role's MaxSessionDuration
when the caller omitted DurationSeconds entirely. AWS returns the
configured default (typically 1 hour) in that case, not the role's
upper bound — a 12h MaxSessionDuration shouldn't silently make every
no-duration assume-role mint a 12h session.

Return nil when requested is nil; let the downstream
calculateSessionDuration in the STS service apply its TokenDuration
default. The role-max upper bound still clamps when the request
arrives with a concrete value above the cap.

Addresses gemini high-priority review on PR #9318.

* fix(iam): synchronize OIDCProvider JWKS cache fields

jwksCache, jwksFetchedAt, resolvedJWKSUri, and discoveryFailed are
mutated lazily on the first token-validate call and refreshed
afterwards on TTL expiry. Multiple S3 requests can land here in
parallel, so the writes were racing against subsequent reads on
every other goroutine. resolvedJWKSUri/discoveryFailed inherited
the same un-protected pattern when discovery shipped.

Add sync.RWMutex; getPublicKey takes the read lock for the
common cache-hit path and promotes to the write lock for misses
+ refreshes. fetchJWKSLocked / resolveJWKSUriLocked assume the
write lock is held by the caller; fetchJWKS keeps the
test-friendly entry point that acquires the lock itself.

Addresses gemini high-priority review on PR #9318.

* fix(iam): trim trailing slash + retry discovery after transient failure

Two OIDC discovery edge cases reviewers flagged:

1. Issuer comparison was sensitive to trailing slashes. resolveJWKSUri
   trims them when building the discovery URL, but the doc.Issuer ↔
   p.config.Issuer check did not, so an IDP whose issuer claim drops or
   adds the slash relative to the configured value would be falsely
   rejected. Trim a single trailing slash on each side before comparing.

2. discoveryFailed flipped to true on any error and stayed there for the
   process lifetime. A transient 5xx at startup permanently locked the
   provider into the /.well-known/jwks.json fallback. Reset the flag at
   the top of fetchJWKSLocked when no URI has been cached yet, so each
   JWKS refresh (typically once per TTL = 1h) reattempts discovery.
   Successful discovery remains cached via resolvedJWKSUri so we don't
   pay the discovery RTT on every refresh.

Addresses gemini security-medium + medium reviews on PR #9318.

* fix(iam): require non-empty issuer in OIDC discovery doc

The previous "doc.Issuer != "" && ..." guard let a discovery document
that omitted the issuer field bypass the issuer-mismatch check
entirely, letting the doc steer fetchJWKS at any URL it provided.
OIDC Discovery 1.0 §3 mandates the issuer field; treat missing as a
hard failure same as mismatched. Trailing-slash equivalence still
applies.

Adds TestDiscoveryRejectsMissingIssuer alongside the existing
TestDiscoveryRejectsIssuerMismatch via a new omitDiscoveryIssuer
toggle on fakeIDP.
 2026-05-04 22:10:49 -07:00
Chris Lu 2417ba0354 fix(volume): add authentication to destructive gRPC admin endpoints (#8876) 
* fix(volume): add authentication to destructive gRPC admin endpoints

Three destructive VolumeServer gRPC endpoints (DeleteCollection,
VolumeDelete, VolumeServerLeave) had no authentication checks, unlike
their HTTP counterparts which are protected by the Guard whitelist.

Add IsWhiteListed(host) to security.Guard and a checkGrpcAdminAuth
helper on VolumeServer that extracts the peer IP from gRPC context and
validates it against the guard whitelist. Gate all three endpoints
behind this check.

* fix(volume): tolerate unparseable gRPC peer address in admin auth check

S3 Filer Group integration tests were failing with
PermissionDenied "bad peer address: address @: missing port in address"
when DeleteCollection ran across the in-process gRPC connection
between filer and volume server — the peer addr surfaces as "@" there
and net.SplitHostPort can't parse it. The check rejected before
IsWhiteListed could exercise its allow-all path for empty-whitelist
deployments.

Hand the raw peer string to IsWhiteListed when SplitHostPort fails.
With no whitelist configured (the test environment's mode) it accepts;
with a whitelist configured the unparseable host won't match anything
and the call still gets denied as it should.

Adds three regression tests for IsWhiteListed pinning the empty-config
allow-all, populated-list reject-unknown, and signing-key-only allow-
all branches that the gRPC admin helper relies on.

* refactor(security): dedup checkWhiteList through IsWhiteListed

The HTTP-side checkWhiteList and the gRPC-side IsWhiteListed had the
same lookup logic in two places; future drift was just a matter of
time. Have checkWhiteList delegate so the membership semantics live
in exactly one function.

Behaviour is unchanged: the new path still returns nil for
isEmptyWhiteList (signing-key-only mode) and still rejects unknown
hosts when a whitelist is configured.

Addresses gemini medium review on PR #8876.

* fix(volume): protect remaining state-altering gRPC admin endpoints

DeleteCollection, VolumeDelete, and VolumeServerLeave were the
truly-destructive endpoints, but AllocateVolume, VolumeMount,
VolumeUnmount, VolumeConfigure, VolumeMarkReadonly, and
VolumeMarkWritable also modify server state and should sit behind
the same whitelist gate. Read-only endpoints (VolumeStatus,
VolumeServerStatus, VolumeNeedleStatus, Ping) stay open.

The check is a no-op when no whitelist is configured (the default),
so existing deployments keep working; operators who lock down their
volume servers via guard.white_list now get consistent coverage.

Addresses gemini security-high review on PR #8876.

* fix(volume): typed peer addr + audit log for gRPC admin auth

Prefer a typed *net.TCPAddr when extracting the peer IP — string
parsing was already a fallback for the in-process case but using the
typed form first is cleaner and skips an unnecessary parse on the
common path. Log failed authorization attempts at V(0) so an operator
running with a whitelist sees the host that was rejected (and the
raw remote address in case the IP lookup itself was the failure
mode), matching what the HTTP Guard already does.

Addresses gemini medium review on PR #8876.

* fix(volume): protect vacuum + scrub + EC-shards-delete admin endpoints

Five more master/admin-driven destructive operations live outside
volume_grpc_admin.go and were missing the same whitelist gate:

- VacuumVolumeCompact, VacuumVolumeCommit, VacuumVolumeCleanup
- ScrubVolume
- VolumeEcShardsDelete

VacuumVolumeCheck stays open (read-only). BatchDelete also stays
open: it's the data-plane multi-object delete called from the S3 API
and filer, not an admin operation; gating it would break ordinary S3
DeleteObjects calls.

Addresses gemini security-high review on PR #8876.

* fix(volume): simplify no-peer-info branch in gRPC admin auth

The IsWhiteListed("") fallback was defending against a scenario
that doesn't actually arise — real gRPC connections always populate
peer info. Drop the branch and just deny when peer info is missing,
which is the safer default and matches "if we don't know who the
caller is, refuse".

* fix(volume-rust): mirror gRPC admin auth on the rust volume server

The rust volume server has the same set of destructive admin
endpoints as the Go side and the same Guard infrastructure, but
nothing was wired together — every endpoint accepted unauthenticated
calls regardless of guard configuration. Same vulnerability class
the Go fix on this PR closes; this commit closes it on the rust
side too so the two stacks stay aligned.

Adds VolumeGrpcService::check_grpc_admin_auth that pulls the peer
SocketAddr off the tonic Request and runs Guard::check_whitelist on
its IP, then applies the helper to the same set the Go side covers:
DeleteCollection, AllocateVolume, VolumeMount, VolumeUnmount,
VolumeDelete, VolumeMarkReadonly, VolumeMarkWritable,
VolumeConfigure, VacuumVolumeCompact, VacuumVolumeCommit,
VacuumVolumeCleanup, VolumeServerLeave, ScrubVolume,
VolumeEcShardsDelete. Read-only endpoints stay open; BatchDelete
stays open as a data-plane multi-object delete.
 2026-05-04 21:14:55 -07:00
Chris Lu 0a91b57f16 fix(s3): encrypt SSE-S3 KEK at rest with AES-GCM wrapping (#8880) 
* fix(s3): encrypt SSE-S3 KEK at rest using passphrase-derived wrapping key

* fix(s3): surface KEK migration failures instead of silently dropping them

The legacy-plaintext -> encrypted-at-rest path used to swallow both
wrapKEK and updateKEKContent errors. An operator who configured a
passphrase but had a filer permission issue (or a wrap failure) saw
nothing in the logs and the KEK stayed on disk in plaintext, with the
migration retried on every restart and silently failing every time.

Log each failure path explicitly so the unmigrated state is visible.
The server still starts with the in-memory key loaded — refusing to
boot here would be worse than the warning.

Addresses gemini and coderabbit reviews on PR #8880.

* fix(s3): use a per-installation random salt for KEK wrapping HKDF

The original implementation hardcoded
"seaweedfs-sse-s3-kek-wrapping-v1" as the HKDF salt for the KEK
wrapping key. Two SeaweedFS installations using the same passphrase
therefore produced byte-identical wrapping keys, opening a
precomputation/rainbow-table angle against weaker passphrases.

Generate a random 32-byte salt every time wrapKEK runs and embed it
in the on-disk payload alongside the AES-GCM ciphertext. The new
format is base64(magic("SWv2") || salt || nonce || ciphertext+tag);
unwrapKEK detects the magic and reads the salt out of the payload.
KEKs wrapped under the legacy fixed-salt format still unwrap cleanly
and are opportunistically re-wrapped into v2 on the next load so
operators get the stronger format without manual migration.

Addresses gemini review on PR #8880.

* fix(s3): plumb KEK passphrase from env into the global key manager

InitializeGlobalSSES3KeyManager used to ignore the kekPassphrase
field entirely — the global manager was always constructed with the
empty constructor, so the encrypted-at-rest code path never engaged
in production. Read the passphrase from WEED_S3_SSE_KEK_PASSPHRASE
and apply it before InitializeWithFiler so the load path picks up the
encrypted format. Log a warning when the env var is unset to make the
plaintext fallback visible to operators upgrading from earlier builds.

Adds SetKEKPassphrase as the public seam used by the global init and
by tests, plus regression tests for the wrap/unwrap round-trip,
random-salt independence across managers sharing a passphrase, and
the no-passphrase fallback that preserves the legacy hex-decode path.

Addresses coderabbit review on PR #8880.

* fix(s3): drop redundant base64 decode in KEK migration check

unwrapKEK already does the base64 decode and the magic-prefix check;
isV2WrappedKEK was repeating both passes purely so the migration
branch in loadSuperKeyFromFiler could ask "was this v1 or v2". Have
unwrapKEK return the version flag directly and delete the redundant
helper. Single decode pass per load.

Addresses gemini review on PR #8880.

* fix(s3): updateKEKContent must overwrite, not create

filer_pb.MkFile maps to CreateEntry, which is O_EXCL: it fails with
ErrEntryAlreadyExists when the file already exists. Both KEK migration
paths (legacy v1→v2 rewrap and plaintext→encrypted) call
updateKEKContent against the entry they just read, so MkFile errored
out every time and the migrations only ran in memory while the on-disk
KEK stayed in its old format. The previous commit logged the failure
loudly but the result was the same: a pre-existing deployment never
got migrated.

Switch updateKEKContent to LookupDirectoryEntry + UpdateEntry so the
overwrite actually persists. Surface the lookup/update errors so
the caller's existing "migration failed; KEK still on disk in old
form" warnings fire on the right cases.

Addresses coderabbit critical review on PR #8880.

* chore(s3): drop unused generateAndSaveSuperKeyToFiler

Master removed this as dead code in #8913 and I reintroduced it during
the merge resolution thinking the migration paths still needed it.
On second look it has no callers in this branch — every KEK-creation
path on PR #8880 goes through the existing reader code that handles
"file not present, generate" inline. Drop the duplicate.

Addresses gemini medium review on PR #8880.

* fix(s3): updateKEKContent honours km.kekPath instead of hardcoded path

The migration write path was always pointing at /etc/s3/sse_kek even
when the manager was configured with an operator-overridden kekPath.
Split km.kekPath at the last "/" so the lookup + UpdateEntry land on
the same file the read path used. Defaults match defaultKEKPath when
kekPath is unset.

Addresses gemini medium review on PR #8880.

* fix(s3): KEK passphrase via Viper key, with env-var fallback

The KEK passphrase was read straight from os.Getenv, but every other
SSE-S3 secret (s3.sse.kek, s3.sse.key) goes through Viper so an
operator can set them in security.toml or via WEED_ env vars
interchangeably. Add s3.sse.kek.passphrase to the same path; the
existing SSES3KEKPassphraseEnv lookup stays as a fallback so
deployments wired before this commit keep working.

Addresses gemini medium review on PR #8880.
 2026-05-04 19:21:41 -07:00
Chris Lu e1d5e3899f fix(s3): add HMAC-SHA256 key commitment to SSE-S3 and SSE-KMS (#8879) 
* fix(s3): add HMAC-SHA256 key commitment to SSE encryption modes

* fix(s3): validate SSE-S3 IV length before cipher.NewCTR

CreateSSES3DecryptedReader took the IV directly from object metadata
and passed it to cipher.NewCTR. The standard library panics when the
IV length differs from the block size, so a tampered metadata field
turned what should be a decryption error into a crash. Validate via
the existing ValidateIV helper first.

Addresses coderabbit review on PR #8879.

* fix(s3): centralize SSE-KMS key commitment in createSSEKMSKey

KeyCommitment used to be set explicitly in only one of the four
encryption entry points; the multipart-aware variants and the bucket-
bound CreateSSEKMSEncryptedReaderForBucket left it empty, so writes
through those paths landed with metadata that VerifyKeyCommitment
treats as legacy and accepts unconditionally — exactly the downgrade
gap the gemini review flagged.

Move the HMAC computation into createSSEKMSKey so every helper-driven
path picks it up automatically, and add the same call to the bucket-
scoped path that builds its own struct literal.

Addresses gemini review on PR #8879.

* fix(s3): store base IV (not derived IV) for offset-aware SSE-KMS chunks

CreateSSEKMSEncryptedReaderWithBaseIVAndOffset used to store the
already-offset-derived IV plus the chunk offset in metadata. The
decrypt path then applied calculateIVWithOffset a second time to
that stored IV, producing the wrong CTR keystream and a decryption
mismatch. Centralizing the key commitment made the bug visible as a
commitment failure, but the underlying issue predates that change.

Pass the base IV through to createSSEKMSKey so sseKey.IV is the
unmodified base on disk; the decrypt path's offset application then
recovers the correct chunk-level IV. The HMAC commitment binds the
base IV — same value the verify call at decrypt time hashes — so the
new commitment path stays consistent.

Addresses gemini security-high review on PR #8879.

* fix(s3): opt-in strict-commitment mode to close the downgrade vector

WEED_S3_REQUIRE_KEY_COMMITMENT=true flips VerifyKeyCommitment from
accept-when-missing (the AWS-compatible default needed for objects
written before commitments shipped) to reject-when-missing. With
the env var set, an attacker who strips the commitment field from
object metadata can no longer bypass integrity verification — every
object must carry a commitment that hashes to the right value.

Default stays false so existing legacy objects keep decrypting; the
warning the gemini review raised about the silent-downgrade vector
is closed for operators who explicitly opt in once their bucket is
fully migrated. SetRequireKeyCommitment exposes a runtime seam for
tests and future config-reload paths.

Addresses the security-medium review on PR #8879.

* test(s3): fix mislabelled IV literal in strict-commitment test

The "strict-mode-iv-16" literal was actually 17 bytes — the trailing
"16" was meant as a comment but counted as content. The IV is fed
into HMAC, not AES, so the length didn't matter for behaviour, but
the discrepancy was confusing. Tighten to a real 16-byte literal and
explain the choice in a comment.

Addresses coderabbit minor review on PR #8879.

* fix(s3): store KMS-resolved KeyID in SSE-KMS metadata, not the request

CreateSSEKMSEncryptedReaderForBucket built its SSEKMSKey with the
caller-supplied keyID, but CreateSSEKMSDecryptedReader later compares
decryptResp.KeyID against sseKey.KeyID. A request that used an alias
would resolve to a different ARN in the response; storing the request
form would then trip the mismatch check at decrypt time and surface
as a "KMS key ID mismatch" error against the operator's own object.
The helper-driven encryption paths already do the right thing via
createSSEKMSKey; this is the bucket-bound path catching up.

Addresses coderabbit review on PR #8879.

* test(s3): cover key commitment rejection paths under tampering

Adds the negative-path tests coderabbit flagged as missing: a
tampered key, IV, algorithm, or commitment field must fail
VerifyKeyCommitment, otherwise a regression in the rejection logic
could land silently. The HMAC binds all three inputs plus the
commitment itself, so any single mutation is enough.

Addresses coderabbit nitpick on PR #8879.
 2026-05-04 19:14:41 -07:00
Chris Lu 66d9b89cd2 fix(iam): deny IAM users with no policies instead of granting full access (#9317) 
* fix(iam): deny IAM users with zero policies instead of falling through to DefaultEffect=Allow

A user created via the S3 IAM API with no policies attached was inheriting
full S3 access. With `weed s3 -iam` and no explicit IAM config, the policy
engine's DefaultEffect defaults to Allow for the in-memory zero-config path.
The "no matching statement" guard in IsActionAllowed only triggered when the
user already had at least one policy, so a fresh user with PolicyNames=[]
slipped through and got allow-all.

Track hasManagedSubject whenever the principal resolves to a registered user
or role (or PolicyNames are supplied directly) and deny on no-match. The
DefaultEffect=Allow fallback now only applies to truly unmanaged callers.

* test(iam): cover non-matching attached-policy case for managed-subject deny

* test(iam): cover role-with-empty-AttachedPolicies deny path

Sibling case to TestIsActionAllowed_RegisteredUserWithoutPoliciesIsDenied:
a managed role that resolves but has zero AttachedPolicies must also
fall through to deny under DefaultEffect=Allow, not inherit full access.
The fix already handles this branch via the hasManagedSubject flag;
this test pins the regression class so we don't lose coverage on
either side of the user/role split.

Addresses coderabbit nitpick on PR #9317.
 2026-05-04 17:56:10 -07:00
Chris Lu 57bb7b2f39 quiet noisy 'shard X not found' log when EC shard lives on another server (#9316) 
quiet "shard X not found" log when EC shard lives on another server

When EC shards are spread across multiple volume servers, every read
that targets a shard not present locally was logging at V(0) with the
same wording as a real read failure. Under rclone-style traffic this
floods the logs and makes a healthy EC cluster look broken (issue #9310).

Distinguish the two cases with an errShardNotLocal sentinel: the
expected fall-through-to-remote path now logs at V(4); genuine local
read failures still log at V(0).
 2026-05-04 15:30:22 -07:00
Chris Lu 73fc9e3833 4.23 2026-05-03 23:15:34 -07:00
Chris Lu d605feb403 refactor(command): expand "~" in all path-style CLI flags (#9306) 
* refactor(command): expand "~" in all path-style CLI flags

Many of weed's path-bearing flags (-s3.config, -s3.iam.config,
-admin.dataDir, -webdav.cacheDir, -volume.dir.idx, TLS cert/key
files, profile output paths, mount cache dirs, sftp key files, ...)
were never run through util.ResolvePath, so a value like "~/iam.json"
was used literally. Tilde only worked when the shell expanded it,
which silently fails for the common -flag=~/path form (bash leaves
the tilde literal in --opt=~/path).

- Extend util.ResolvePath to also handle "~user" / "~user/rest",
  matching shell tilde expansion. Add unit tests.
- Apply util.ResolvePath at the top of each shared start* function
  (s3, webdav, sftp) so mini/server/filer/standalone callers all
  inherit it; resolve at the few one-off use sites (mount cache
  dirs, volume idx folder, mini admin.dataDir, profile paths).
- Drop the duplicate expandHomeDir helper from admin.go in favor of
  the now-equivalent util.ResolvePath.

* fixup: handle comma-separated -dir flags for tilde expansion

`weed mini -dir`, `weed server -dir`, and `weed volume -dir` accept
comma-separated paths (`dir[,dir]...`). Calling util.ResolvePath on
the whole string mishandled multi-folder values with tilde, e.g.
"~/d1,~/d2" would resolve as if "d1,~/d2" were a single subpath.

- Add util.ResolveCommaSeparatedPaths: split on ",", run each entry
  through ResolvePath, rejoin. Short-circuits when no "~" present.
- Use it for *miniDataFolders (mini.go), *volumeDataFolders (server.go),
  and resolve each entry of v.folders in-place (volume.go) so all
  downstream consumers see resolved paths.
- Add 7-case TestResolveCommaSeparatedPaths covering empty, single,
  multiple, and mixed inputs.

* address PR review: metaFolder + Windows backslash

- master.go: resolve *m.metaFolder at the top of runMaster so
  util.FullPath(*m.metaFolder) on the next line sees an expanded
  path. Drop the now-redundant ResolvePath in TestFolderWritable.
- server.go: same treatment for *masterOptions.metaFolder, paired
  with the existing cpu/mem profile resolves. Drop the redundant
  inner ResolvePath at TestFolderWritable.
- file_util.go: ResolvePath now accepts filepath.Separator as a
  separator after the tilde, so "~\\data" works on Windows. Other
  platforms keep current behaviour (backslash stays literal because
  it is a valid filename character in usernames and paths).
- file_util_test.go: add two cases using filepath.Separator that
  exercise the new code path on Windows and remain a no-op on Unix.

* address PR review: resolve "~" in remaining command path flags

Comprehensive sweep of path-bearing flags across every weed
subcommand, applying util.ResolvePath in-place at the top of each
run* function so all downstream consumers see expanded paths.

- webdav.go: resolve *wo.cacheDir at the top of startWebDav so
  mini/server/filer/standalone callers all inherit it.
- mount_std.go: cpu/mem profile paths.
- filer_sync.go: cpu/mem profile paths.
- mq_broker.go: cpu/mem profile paths.
- benchmark.go: cpuprofile output path.
- backup.go: -dir resolved once at runBackup; drop the duplicated
  inline ResolvePath in NewVolume calls.
- compact.go: -dir resolved at runCompact; drop inline ResolvePath.
- export.go: -dir and -o resolved at runExport; drop inline
  ResolvePath in LoadFromIdx and ScanVolumeFile.
- download.go: -dir resolved at runDownload; drop inline.
- update.go: -dir resolved at runUpdate so filepath.Join uses the
  expanded path; drop inline ResolvePath in TestFolderWritable.
- scaffold.go: -output expanded before filepath.Join.
- worker.go: -workingDir expanded before being passed to runtime.

* address PR review: resolve option-struct paths at run* entry points

server.go:381 propagates s3Options.config to filerOptions.s3ConfigFile
*before* startS3Server runs, which meant the filer-side code saw the
unresolved tilde-prefixed pointer. Same pattern for webdavOptions and
sftpOptions (and equivalent in mini.go / filer.go).

The fix: hoist resolution from the shared start* functions up to the
run* entry points, where every shared pointer is set up before any
propagation happens.

- s3.go, webdav.go, sftp.go: extract a resolvePaths() method on each
  Options struct that runs every path field through util.ResolvePath
  in-place. Idempotent.
- runS3, runWebDav, runSftp: call the standalone struct's resolvePaths
  before starting metrics / loading security config.
- runServer, runMini, runFiler: call resolvePaths on every embedded
  options struct, plus resolve loose flags (serverIamConfig,
  miniS3Config, miniIamConfig, miniMasterOptions.metaFolder, and
  filer's defaultLevelDbDirectory) so they're expanded before any
  pointer copy or use.
- Drop the now-redundant inline ResolvePath at filer's
  defaultLevelDbDirectory composition.

* address PR review: re-resolve mini -dir post-config, cover misc paths

- mini.go: applyConfigFileOptions can overwrite -dir with a literal
  ~/data from mini.options. Re-resolve *miniDataFolders after the
  config-file apply, alongside the other path resolves, so the mini
  filer no longer ends up with a literal ~/data/filerldb2.
- benchmark.go: resolve *b.idListFile (-list).
- filer_sync.go: resolve *syncOptions.aSecurity / .bSecurity
  (-a.security / -b.security) before LoadClientTLSFromFile.
- filer_cat.go: resolve *filerCat.output (-o) before os.OpenFile.
- admin.go: drop trailing blank line at EOF (git diff --check).

* address PR review: resolve -a.security/-b.security/-config before use

Three follow-up fixes:

- filer_sync.go: the -a.security / -b.security resolves were placed
  *after* LoadClientTLSFromFile / LoadHTTPClientFromFile were called,
  so weed filer.sync -a.security=~/a.toml still passed the literal
  tilde path. Hoist the resolves above the security-loading block so
  TLS clients see expanded paths.
- filer_sync_verify.go: same flag pair was never resolved at all in
  the verify command; resolve at the top of runFilerSyncVerify.
- filer_meta_backup.go: -config (the backup_filer.toml path) was
  passed directly to viper. Resolve at the top of runFilerMetaBackup.
- mini.go: master.dir defaulted to the entire comma-joined
  miniDataFolders. With weed mini -dir=~/d1,~/d2 (or any multi-dir
  setup), TestFolderWritable then stat'd the joined string instead
  of a single directory. Default to the first entry via StringSplit
  to mirror the disk-space calculation a few lines below, and drop
  the now-redundant ResolvePath in TestFolderWritable.
 2026-05-03 21:46:21 -07:00
Chris Lu 6844ec067c fix(s3): cache remote-only source before CopyObject (#9304) (#9305) 
* fix(s3): cache remote-only source before CopyObject (#9304)

CopyObject from a remote.mount source whose object lives only upstream
created a destination entry with FileSize > 0 but no chunks/content,
because the resolved source entry has no local chunks and the copy path
fell into the "inline/empty chunks" branch with empty entry.Content.
A subsequent GET returned 500 with "data integrity error: size N
reported but no content available". CopyObjectPart had the same shape
via copyChunksForRange iterating an empty chunk list.

Detect entry.IsInRemoteOnly() right after resolving the source in both
CopyObjectHandler and CopyObjectPartHandler and cache the object to the
local cluster first via a new cacheRemoteObjectForCopy helper (a
copy-time analogue of cacheRemoteObjectForStreaming with a bounded 30s
timeout and version-aware path resolution). If caching fails or
produces no chunks, return 503 with Retry-After: 5 instead of writing
a metadata-only destination, mirroring the GetObject behavior added in
the #7817 cold-cache fix.

Adds TestCopyObjectRemoteOnlySourceDetection pinning the four entry
shapes the fix branches on plus the pre-fix broken-output shape.

* address PR review on remote-only copy fix

- Use the resolved entry's version id when srcVersionId is empty so a
  CopyObject reading the latest object in a versioning-enabled bucket
  caches the correct .versions/v_<id> path instead of getting stuck in
  a 503 retry loop. New helper resolvedSourceVersionId handles the
  fallback for both CopyObject and CopyObjectPart.
- Drop the redundant cachedEntry.IsInRemoteOnly() recheck in both
  handlers; the cache helper now reports success based on local data
  presence, and IsInRemoteOnly does not look at inline Content so
  keeping the check would 503 on small inline-cached objects.
- Treat inline Content as a successful cache result in both
  cacheRemoteObjectForStreaming and cacheRemoteObjectForCopy via a
  shared cachedEntryHasLocalData predicate. The CopyObject inline
  branch already handles entries that have Content but no chunks.
- Extract buildVersionedRemoteObjectPath so the streaming and copy
  cache helpers share path construction.

Adds TestResolvedSourceVersionId and TestCachedEntryHasLocalData to
pin the new helpers' contracts.

* narrow streaming cache contract back to chunks-only

CodeRabbit flagged that cacheRemoteObjectForStreaming's caller in
streamFromVolumeServers (lines 997-1002) still required non-empty
chunks, so content-only cache hits would fall through to a 503 retry
loop instead of being honored.

Resolve by keeping the helper's contract chunks-only: the filer's
caching code only ever writes chunks, the streaming downstream isn't
wired to read inline Content from a cached entry, and a partial range-
aware inline writer here would be overkill for a path that doesn't
actually occur in practice. cacheRemoteObjectForCopy keeps the relaxed
contract since the copy path's inline branch genuinely handles both
chunked and content-only entries.

Document the asymmetry on cachedEntryHasLocalData and on
cacheRemoteObjectForStreaming so a future reader can see why the two
helpers diverge.

* extend version-id resolution to streaming cache path

CodeRabbit flagged that GetObjectHandler still passed the raw query
versionId to cacheRemoteObjectForStreaming. For latest-version reads
in versioning-enabled buckets that stays empty even though the
resolved entry lives at .versions/v_<id>, so remote-only GETs would
keep caching the wrong path and 503-ing forever. Reuse the new
resolvedSourceVersionId helper at the streaming call site too.

Also document on cachedEntryHasLocalData that the zero-byte case
flagged in the same review is handled upstream (IsInRemoteOnly
requires RemoteSize > 0, so the cache helper is never invoked for
empty remote objects -- CopyObject's pre-existing inline branch
writes a correct empty destination directly). Pin this with a new
test case.

* trim verbose comments

Drop tutorial-style and review-history comments. Keep only the WHY
that isn't obvious from identifiers: the #9304 reference on the new
branches in CopyObject / CopyObjectPart, the latest-version-fallback
rationale on resolvedSourceVersionId, and the streaming/copy contract
asymmetry on cachedEntryHasLocalData.

* drop issue references from comments

Issue numbers belong in PR descriptions and commit messages, not in
source comments where they rot. Replace with the underlying invariant
the code is preserving.

* test: drive remote-object cache helpers through real gRPC

Existing tests only re-enacted helper-function branching in test
space, so they could not have caught a handler that consumed a
remote-only entry without going through the cache. Stand up an
in-process filer gRPC server (UnimplementedSeaweedFilerServer +
configurable CacheRemoteObjectToLocalCluster response) and exercise
the two cache helpers end-to-end.

What's pinned:
- cacheRemoteObjectForCopy returns nil when the cache makes no
  progress (response is still remote-only), lets gRPC errors
  through as nil, accepts both chunked and inline-content cache
  hits, and surfaces deadline-exceeded as nil so callers can 503
  instead of holding the request open.
- Versioned source paths route to .versions/v_<id>; non-versioned
  and "null" stay at the bucket-relative path. Captured by reading
  the request the stub server received.
- cacheRemoteObjectForStreaming holds the stricter chunks-only
  contract: a content-only cache hit is not propagated, since
  streamFromVolumeServers' downstream isn't wired to read from
  inline Content there.

Any current or future handler that calls these helpers exercises
the same gRPC path under test, so the bug class is closed for
helper-routed cache calls.

* move remote-only copy test into the integration suite

The previous gRPC-stub test in weed/s3api/ was integration-flavored
but stubbed; relocate the coverage to the existing two-server suite
under test/s3/remote_cache/, which already exercises the real
remote.mount + remote.uncache flow against a primary SeaweedFS plus
a secondary acting as remote storage.

The new test/s3/remote_cache/remote_cache_copy_test.go drives:
- TestRemoteCacheCopyObject: upload to primary, uncache (entry now
  remote-only), CopyObject to a new key, GET the destination. Pre-
  fix the GET returned 500 'data integrity error: size N reported
  but no content'; this pins the fixed behavior over real HTTP
  through the actual handler stack.
- TestRemoteCacheCopyObjectPart: same shape via multipart
  UploadPartCopy on a 6 MiB object split into two parts, exercising
  CopyObjectPartHandler's range-copy path.

Drop weed/s3api/s3api_remote_storage_grpc_test.go: the helper-level
classification tests in s3api_remote_storage_test.go still cover
the contract pieces (cachedEntryHasLocalData, resolvedSourceVersionId,
the remote-only entry shape), and the integration suite covers the
end-to-end behavior that those classifications enable.
 2026-05-03 18:52:45 -07:00
Chris Lu f16353de0b feat(mini): add -bucket flag to pre-create an S3 bucket on startup (#9302) 
* feat(mini): add -bucket flag to pre-create an S3 bucket on startup

Lets users hand a pre-provisioned object store to clients/CI without a
post-start `weed shell s3.bucket.create` step. The flag is a no-op when
empty (default) and idempotent on subsequent starts.

* mini: bound bucket-creation RPCs with a timeout off miniClientsCtx

Address PR review feedback: derive the lookup/mkdir context from
miniClientsCtx() so Ctrl+C cancels the bucket RPCs, and cap with a 5s
timeout so a stalled filer cannot block the welcome message
indefinitely. Also wrap the DoMkdir error for parity with the lookup
path.

* mini: fall back to S3_BUCKET env var for -bucket

Mirrors the existing -s3.externalUrl / S3_EXTERNAL_URL pattern so
container/Kubernetes deployments can pre-create the bucket via env
without overriding the entrypoint command.

* docs(readme): lead weed mini quick start with credentials + bucket

Promote the one-line setup (env vars + bucket) so users get a
ready-to-use S3 endpoint without hopping between sections to find
credential and bucket setup.

* mini: accept comma-separated -bucket list

Lets a single startup pre-create multiple S3 buckets, e.g.
-bucket=bucket1,bucket2 (or S3_BUCKET=bucket1,bucket2). Names are
trimmed and deduped; per-bucket errors are logged and the loop continues
so one bad name does not block the rest.

* mini: add -tableBucket flag for pre-creating S3 Tables buckets

Mirrors -bucket but creates S3 Tables (Iceberg) buckets via
s3tables.Manager so users can hand the all-in-one binary a ready-to-use
table catalog without a follow-up weed shell call. Comma-separated, env
fallback to S3_TABLE_BUCKET, idempotent on restart, owned by the
DefaultAccountID placeholder.

* mini: use errors.Is for ErrNotFound check in bucket lookup

Matches the rest of the codebase (~20 call sites in weed/s3api). The
direct equality works today because LookupEntry returns ErrNotFound
unwrapped, but errors.Is future-proofs against any future wrapping.
 2026-05-02 21:02:21 -07:00
Chris Lu 1f6f473995 refactor(worker): co-locate plugin handlers with their task packages (#9301) 
* refactor(worker): co-locate plugin handlers with their task packages

Move every per-task plugin handler from weed/plugin/worker/ into the
matching weed/worker/tasks/<name>/ package, so each task owns its
detection, scheduling, execution, and plugin handler in one place.

Step 0 (within pluginworker, no behavior change): extract shared helpers
that previously lived inside individual handler files into dedicated
files and export the ones now consumed across packages.

  - activity.go: BuildExecutorActivity, BuildDetectorActivity
  - config.go: ReadStringConfig/Double/Int64/Bytes/StringList, MapTaskPriority
  - interval.go: ShouldSkipDetectionByInterval
  - volume_state.go: VolumeState + consts, FilterMetricsByVolumeState/Location
  - collection_filter.go: CollectionFilterMode + consts
  - volume_metrics.go: export CollectVolumeMetricsFromMasters,
    MasterAddressCandidates, FetchVolumeList
  - testing_senders_test.go: shared test stubs

Phase 1: move the per-task plugin handlers (and the iceberg subpackage)
into their task packages.

  weed/plugin/worker/vacuum_handler.go         -> weed/worker/tasks/vacuum/plugin_handler.go
  weed/plugin/worker/ec_balance_handler.go     -> weed/worker/tasks/ec_balance/plugin_handler.go
  weed/plugin/worker/erasure_coding_handler.go -> weed/worker/tasks/erasure_coding/plugin_handler.go
  weed/plugin/worker/volume_balance_handler.go -> weed/worker/tasks/balance/plugin_handler.go
  weed/plugin/worker/iceberg/                   -> weed/worker/tasks/iceberg/

  weed/plugin/worker/handlers/handlers.go now blank-imports all five
  task subpackages so their init() registrations fire.

  weed/command/mini.go and the worker tests construct the handler with
  vacuum.DefaultMaxExecutionConcurrency (the constant moved with the
  vacuum handler).

admin_script remains in weed/plugin/worker/ because there is no
underlying weed/worker/tasks/admin_script/ package to merge with.

* refactor(worker): update test/plugin_workers imports for moved handlers

Three handler constructors moved out of pluginworker into their task
packages — update the integration test files in test/plugin_workers/
to import from the new locations:

  pluginworker.NewVacuumHandler        -> vacuum.NewVacuumHandler
  pluginworker.NewVolumeBalanceHandler -> balance.NewVolumeBalanceHandler
  pluginworker.NewErasureCodingHandler -> erasure_coding.NewErasureCodingHandler

The pluginworker import is kept where the file still uses
pluginworker.WorkerOptions / pluginworker.JobHandler.

* refactor(worker): update test/s3tables iceberg import path

The iceberg subpackage moved from weed/plugin/worker/iceberg/ to
weed/worker/tasks/iceberg/. test/s3tables/maintenance/maintenance_integration_test.go
still imported the old path, breaking S3 Tables / RisingWave / Trino /
Spark / Iceberg-catalog / STS integration test builds.

Mirrors the OSS-side fix needed by every job in the run that
transitively imports test/s3tables/maintenance.

* chore: gofmt PR-touched files

The S3 Tables Format Check job runs `gofmt -l` over weed/s3api/s3tables
and test/s3tables, then fails if anything is unformatted. Files this
PR moved or modified had import-grouping and trailing-spacing issues
introduced by perl-based renames; reformat them with gofmt -w.

Touched files:
  test/plugin_workers/erasure_coding/{detection,execution}_test.go
  test/s3tables/maintenance/maintenance_integration_test.go
  weed/plugin/worker/handlers/handlers.go
  weed/worker/tasks/{balance,ec_balance,erasure_coding,vacuum}/plugin_handler*.go

* refactor(worker): bounds-checked int conversions for plugin config values

CodeQL flagged 18 go/incorrect-integer-conversion warnings on the moved
plugin handler files: results of pluginworker.ReadInt64Config (which
ultimately calls strconv.ParseInt with bit size 64) were being narrowed
to int32/uint32/int without an upper-bound check, so a malicious or
malformed admin/worker config value could overflow the target type.

Add three helpers in weed/plugin/worker/config.go that wrap
ReadInt64Config and clamp out-of-range values back to the caller's
fallback:

  ReadInt32Config (math.MinInt32 .. math.MaxInt32)
  ReadUint32Config (0 .. math.MaxUint32)
  ReadIntConfig    (math.MinInt32 .. math.MaxInt32, platform-portable)

Update each flagged call site in the four moved task packages to use
the bounds-checked helper. For protobuf uint32 fields (volume IDs)
the variable type also becomes uint32, removing the trailing
uint32(volumeID) casts and changing the "missing volume_id" check
from `<= 0` to `== 0`.

Touched files:
  weed/plugin/worker/config.go
  weed/worker/tasks/balance/plugin_handler.go
  weed/worker/tasks/erasure_coding/plugin_handler.go
  weed/worker/tasks/vacuum/plugin_handler.go

* refactor(worker): use ReadIntConfig for clamped derive-worker-config helpers

CodeQL still flagged three call sites where ReadInt64Config was being
narrowed to int after a value-range clamp (max_concurrent_moves <= 50,
batch_size <= 100, min_server_count >= 2). The clamp is correct but
CodeQL's flow analysis didn't recognize the bound, so it flagged them
as unbounded narrowing.

Switch to ReadIntConfig (already int32-bounded by the helper) for
those three sites, drop the now-redundant int64 intermediate variables.

Also drops the now-unused `> math.MaxInt32` clamp in
ec_balance.deriveECBalanceWorkerConfig (the helper covers it).
 2026-05-02 18:03:13 -07:00
Chris Lu 31e5e0dee2 fix(mount): keep async flush when LockOwner has no POSIX locks (#9300) 
FlushIn.LockOwner is populated by the kernel for any fd that may have
participated in locking, not only when locks were actually taken. The
previous Flush logic treated any non-zero LockOwner as a closing lock
holder and forced a synchronous flush, which silently disabled the
writebackCache async-flush path (introduced in #8727) for most
ordinary close() calls.

Consult the POSIX lock table before forcing sync: only owners that
currently hold a non-flock byte-range lock need the synchronous path
to coordinate with blocked SetLkw waiters. Other closes go async as
intended.
 2026-05-01 19:51:27 -07:00
Chris Lu f2c3bd7b77 fix(admin/view): define basePath in plugin IIFE scopes (#9298) 
The plugin.templ and plugin_lane.templ components use basePath() in their IIFE
(Immediately Invoked Function Expression) scopes to handle subdirectory
deployments. However, basePath was not defined locally, causing "basePath is
not defined" errors when accessing plugin pages.

Added local basePath function definitions in both files, matching the pattern
from admin.js. This function checks window.__BASE_PATH__ (set by the layout
during page initialization) and prepends it to API paths.
 2026-05-01 18:22:39 -07:00
Lars Lehtonen 913f98db10 refactor(weed/storage): log volume file removal failures (#9297) 2026-05-01 11:52:40 -07:00
Yoann KatchourineandChris Lu 1127672d10 fix(s3api,iamapi): avoid full SaveConfiguration when creating a single IAM user (#9261) 
* fix(s3api,iamapi): avoid full SaveConfiguration when creating a single IAM user

When CreateUser was called, both the embedded IAM path (s3api_embedded_iam.go)
and the standalone IAM path (iamapi_management_handlers.go) would:
  1. Load the full config (all N users)
  2. Append the new user
  3. Call SaveConfiguration, which re-writes ALL N user files in the filer_etc
     store, triggering N file-change events and N reload cycles.

The fix replaces the full-save path with credentialManager.CreateUser, which
writes only the single new user's file. Set changed=false to skip the redundant
SaveConfiguration call, and add "CreateUser" to the reload-after-targeted-write
block so the in-memory config is refreshed.

Also adds a nil guard around iama.iam.GetCredentialManager() in DoActions to
avoid a nil-pointer panic in legacy test fixtures that leave iam unset.

Add SetCredentialManagerForTest to IdentityAccessManagement so tests can inject
an in-memory credential store without touching production code paths.

* test(s3api,iamapi): regression tests for CreateUser targeted-write fix

Add TestCreateUserDoesNotSaveAllUsers (iamapi) and
TestEmbeddedIamCreateUserDoesNotSaveAllUsers (s3api) to guard against the
regression where CreateUser would call SaveConfiguration and re-write all
N existing user files.

Both tests:
  - Pre-populate 3 existing users
  - Invoke CreateUser via the HTTP API
  - Assert PutS3ApiConfiguration (full-config save) was NOT called
  - Assert the new user is visible in the credential store
  - Assert all pre-existing users are still intact

Also update TestEmbeddedIamExecuteAction to verify persistence via the
credential manager directly (mockConfig is no longer updated on CreateUser
since we skip the SaveConfiguration path).

* refactor(s3api,iamapi): share credential-error to IAM-code mapping

Move the credentialErrToIamErrCode helper from weed/iamapi to
weed/s3api as exported CredentialErrToIamErrCode and call it from
both the standalone IAM handler (iamapi) and the embedded IAM
handler (s3api). Previously the standalone path used the helper
while the embedded path duplicated the same switch inline; the two
sites could drift out of sync.

Also extend the mapping to cover ErrUserNotFound and
ErrAccessKeyNotFound (404 NoSuchEntity) so non-CreateUser callers
that opt into the helper get the right HTTP status.

* test(s3api): seed credential store explicitly in CreateUser regression

Previously the test relied on getS3ApiConfigurationFunc's syncOnce
side effect to populate the credential store with mockConfig
identities. If that fixture ever stopped seeding, the post-test
"pre-existing users still exist" assertion would silently start
passing for the wrong reason (the users were never created).

Seed cm.SaveConfiguration directly and assert the seed precondition
before the API call so any seeding regression fails loudly.

* fix(s3api): honor skipPersist in embedded CreateUser targeted path

ExecuteAction documents that skipPersist=true means "the changed
configuration is not saved to the persistent store", but the
targeted credentialManager.CreateUser added for the avoid-bulk-save
fix ran unconditionally. Gate it on !skipPersist so no-persist
callers don't silently leak a write to the credential store. Leave
changed=true in the skipPersist branch so the tail's existing
`if changed { if !skipPersist { ... } }` block keeps suppressing
persistence the same way it does for every other action.

Add TestEmbeddedIamCreateUserSkipPersist to pin the contract: a
CreateUser invocation with skipPersist=true must not call
PutS3ApiConfiguration and must not leave the user in the
credential store.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
 2026-05-01 01:14:15 -07:00
Chris Lu 6572b472c3 fix(s3): honor X-Forwarded-For in audit log remote_ip (#9295) 
* fix(s3): honor X-Forwarded-For in audit log remote_ip

When SeaweedFS S3 sits behind a reverse proxy (e.g., Caddy), the audit
log's `remote_ip` was reporting the proxy's address because only
`X-Real-IP` and `r.RemoteAddr` were consulted. Caddy and most other
proxies set `X-Forwarded-For` by default but not `X-Real-IP`, so the
real client IP was lost.

Check `X-Forwarded-For` first (using the left-most non-empty entry as
the originating client), then fall back to `X-Real-IP`, then
`r.RemoteAddr`.

Fixes #9293

* fix(s3): strip port from RemoteAddr fallback in audit log

Address PR review: the X-Forwarded-For and X-Real-IP paths return
host-only values, while the RemoteAddr fallback was returning
"host:port", making the remote_ip field inconsistent with both the
other code paths and the "192.0.2.3" example in the AccessLog struct.
Use net.SplitHostPort to strip the port, falling back to the raw
RemoteAddr for non-IP markers (e.g., "@" for unix sockets).
 2026-04-30 15:19:04 -07:00
Chris Lu d265274e13 fix(nfs): accept dirpath any-where under the export, mirroring rclone (#9291) 
* fix(nfs): accept any MOUNT3 dirpath, mirroring rclone's permissive policy

weed nfs has exactly one export per process, so the MOUNT3 dirpath
argument has no second export to disambiguate against. Strict
comparison only translated PV-path typos into the inconsistent
"mount succeeds but empty" / "mount fails completely" split that
operators see.

Match rclone's serve nfs Handler.Mount: ignore the dirpath, log an INFO
line when it differs from the configured export, and always serve the
export root. Apply the same change to the UDP MOUNT3 path so kernel
clients defaulting to mountproto=udp see identical behaviour. Access
control still goes through -allowedClients / -ip.bind, and file-handle
scoping in FromHandle is unchanged so handles still cannot escape the
export.

Replace the prior single-path reject tests with table tests covering
the shapes operators commonly hit: root, parent, sibling, deeper child,
unrelated, empty, relative form, exact match, and trailing slash, at
the Handler.Mount, UDP MOUNT3, and full RPC layers.

* feat(nfs): mount at subdirectory when MOUNT3 dirpath is under the export

Make the dirpath argument meaningful when the client asks for a subtree
of the configured export. With -filer.path=/buckets, a client mounting
<server>:/buckets/data lands directly inside /buckets/data instead of
at the export root.

  - dirpath equals the export root: serve the export root.
  - dirpath strictly under the export, directory entry: serve that
    subdirectory; the returned filehandle encodes its inode.
  - dirpath strictly under the export, missing or non-directory: reject
    with NoEnt or NotDir.
  - dirpath outside the export: keep the rclone-style fallback to the
    export root.

TCP returns a sub-rooted seaweedFileSystem and lets go-nfs's onMount
call ToHandle to encode the FH; UDP encodes the FH itself. FromHandle
is unchanged: handles are content-addressed by inode and resolve via
the inode index, so they remain stable across mounts and across
process restarts.

The trimmed permissive tests keep their outside-export shapes; new
subexport tests cover under-export directories, missing entries, and
non-directory entries on Handler.Mount, the UDP MOUNT3 wire, and
through the full RPC stack.

* nfs: propagate request context through MOUNT3 resolution

Mount now accepts the gonfs context and threads it through
resolveMountFilesystem and lstatExportStatus so a slow filer call
during MOUNT cannot outlive a cancelled or timed-out request.

lstatExportStatus uses fileInfoForVirtualPath(ctx, "/") directly
instead of billy.Filesystem.Lstat, which would otherwise drop the
context on the floor by calling fileInfoForVirtualPathWithOptions
with context.Background().

Lower the successful subexport-mount log from V(0) to V(1). The
fallback log stays at V(0) so operator typos still surface; the
success line is per-mount churn that adds up on NFS-CSI deployments.

* nfs: mirror TCP defensive checks on the UDP MOUNT3 path

Two transport-parity bugs the rabbit caught:

(1) The exact-export-root and outside-export branches were returning
MNT3_OK unconditionally, while the TCP handler runs lstatExportStatus
on those same branches. If the configured -filer.path has been
removed from the filer, TCP returns NoEnt/ServerFault but UDP would
still hand out a synthetic root handle pointing at nothing. Add
rootMountStatus as the UDP analogue and call it on both branches.

(2) resolveSubexportFileHandle did filer I/O on the single UDP serve
loop with context.Background(). One slow filer round-trip would
block every later MOUNT packet. Wrap each MOUNT call's filer work in
context.WithTimeout(mountUDPLookupTimeout) and thread that ctx
through both rootMountStatus and resolveSubexportFileHandle.

Lower the successful subexport log to V(1) to match the TCP side.

* nfs: assert TCP/UDP MOUNT3 produce byte-identical filehandles

The existing UDP subexport assertions only checked the decoded inode
and kind. A regression that drifted the generation, exportID, or
encoding format on one transport but not the other would have slipped
through. Build the TCP Handler from the same Server, drive its Mount
with the same dirpath, and require ToHandle to match the raw UDP FH
bytes for every OK case.

* nfs: take MOUNT3 dirpath as string in resolveMountFilesystem

Convert req.Dirpath to string once at the call site instead of
sprinkling string(...) casts through every log line and conversion
inside the function. Behavior unchanged.

* nfs: share rootFS lifecycle between TCP and UDP MOUNT handlers

Server.rootFilesystem() lazily constructs the seaweedFileSystem rooted
at the configured export the first time anything asks for it, then
hands the same instance to every subsequent caller. newHandler() and
mountUDPServer.rootMountStatus() now both go through it, so:

  - Both transports observe the same chunk reader cache and chunk
    invalidator without depending on call order during startup.
  - The UDP defensive Lstat doesn't allocate a fresh wrapper per
    MOUNT request anymore; one struct lives for the life of the
    Server.

The sub-rooted seaweedFileSystem the subexport branch builds in
resolveSubexportFileHandle is still per-request because actualRoot
varies with the requested dirpath.

* nfs: drive rootFilesystem before reading sharedReaderCache on UDP

The UDP listener is started before serve() calls newHandler(), so an
under-export MOUNT3 request can reach resolveSubexportFileHandle before
Server.sharedReaderCache has been assigned. Reading it directly would
hand newSeaweedFileSystem a nil cache and the sub-fs would build a
throwaway ReaderCache that never gets shared with the TCP path.

Take rootFS off Server.rootFilesystem() (which drives the sync.Once
that initializes the shared cache) and read readerCache off that
instead, so subexport sub-fs instances always share the same reader
cache as rootFS regardless of which transport sees the first MOUNT.

* nfs: collapse exact-match and outside-export MOUNT branches

The two branches return the same filesystem (export root) and the
same status; only the log line differs. Combine the conditions and
guard the fallback log inline. Behavior unchanged.
 2026-04-30 10:06:44 -07:00
Chris Lu 14cd426cf9 templ 2026-04-29 16:08:38 -07:00
Lars Lehtonen 89b07f3e12 chore(weed/mq/kafka/protocol): prune no-op test (#9287) 2026-04-29 14:11:51 -07:00
Lars Lehtonen 02da928e25 chore(weed/mq/kafka/protocol): prune dead code (#9288) 2026-04-29 14:11:37 -07:00
Jaehoon KimandChris Lu be451d22b5 feat(filer.sync): add -verifySync mode to filer.sync for cross-cluster file comparison (#9284) 
* Add -verifySync flag to filer.sync for cross-cluster file comparison

Add a verification mode to filer.sync that compares entries between two
filers without performing actual synchronization. Uses directory-level
sorted merge of ListEntries to detect missing files, size mismatches,
and ETag mismatches. Supports -isActivePassive for unidirectional check
and -modifyTimeAgo to skip recently modified files during sync lag.

* Add mtime annotation and JSON output to filer.sync -verifySync

Add automatic mtime relation analysis for SIZE_MISMATCH and
ETAG_MISMATCH diffs, and an NDJSON output mode for external tooling.

mtime classification:
- B_NEWER => "late_updates_skip_likely" hint. Surfaces the case
  where target has a stub entry whose mtime is ahead of source's
  real file, causing UpdateEntry's mtime guard in filersink to
  permanently skip the update.
- A_NEWER => "sync_lag_or_event_miss" hint.
- EQUAL   => no hint (chunk-level issue suspected).

Text output example:
  [SIZE_MISMATCH] /path (a=996, b=0, B newer +274d [late-updates skip likely])

Add -verifyJsonOutput flag. When set, emits one JSON object per
line (NDJSON) for diffs and a final SUMMARY object, suitable for
piping into external diagnostic pipelines.

Concurrent writes from the directory worker pool are now serialized
via outputMu to keep both text lines and JSON records atomic.

* fix(filer.sync): use shared global semaphore in verifySync to bound goroutine explosion

Replace the per-call local semaphore in compareDirectory with a single
shared semaphore created in runVerifySync. The old per-level semaphore
applied a limit of verifySyncConcurrency only within each directory level,
allowing effective concurrency to grow as verifySyncConcurrency^depth on
deep trees.

The shared semaphore is held only for each directory's I/O phase
(listEntries + merge) and released before recursing into subdirectories,
so a parent never blocks waiting for children to acquire slots — which
would deadlock once tree depth exceeds the semaphore capacity.

Extract the capacity into a named constant (verifySyncConcurrency = 5)
with a comment explaining the memory vs. performance trade-off.

Add unit tests:
- correctness: missing file, only-in-B, size mismatch, active-passive mode
- concurrency bound: peak concurrent listings ≤ verifySyncConcurrency
- no-deadlock: binary tree of depth 10 completes within timeout

* fix(filer.sync): stream directory entries to prevent OOM on large directories

Replace the listEntries helper (which accumulated all entries into a
single []filer_pb.Entry slice) with an entryStream type that pages
through the directory in the background and forwards entries one at a
time through a buffered channel. Memory per directory comparison is now
O(channel buffer size = 64) regardless of how many entries the directory
contains.

Key design points:
- entryStream wraps a goroutine + buffered channel with a one-entry
  lookahead (peek/advance) so the two-pointer sorted merge in
  compareDirectory can work without buffering any full listing.
- A child context (mergeCtx) is passed to both stream goroutines so
  they are cancelled promptly if compareDirectory returns early (e.g.
  on error); the ctx.Done() select arm in the callback prevents
  goroutine leaks when the consumer stops reading.
- stream.err is written by the goroutine before close(ch), so it is
  safe to read after the channel is exhausted (Go memory model:
  channel close happens-before the zero-value receive).
- countMissingRecursive is rewritten to use ReadDirAllEntries with a
  direct callback, eliminating its own slice allocation.
- listEntries is removed; it is no longer called anywhere.

* fix(filer.sync): address verifySync review findings

Four real bugs found and fixed; one finding already resolved (shared
semaphore was introduced in a prior commit).

path.Join for child paths (filer_sync_verify.go)
  fmt.Sprintf("%s/%s", dir, name) produced "//name" when dir was "/".
  Replace all child-path concatenations with path.Join so root-level
  walks emit clean paths.

cutoffTime check for ONLY_IN_B entries (filer_sync_verify.go)
  The B-only branch ignored -modifyTimeAgo, so files recently written
  to B were reported as ONLY_IN_B instead of being skipped. Mirror the
  A-side mtime guard: skip and increment skippedRecent when the entry
  is newer than cutoffTime.

Summary emitted before error check (filer_sync_verify.go)
  A filer I/O error mid-walk still caused a SUMMARY record (or text
  summary) to be printed, making partial runs appear complete. Move the
  error check to before summary emission; on error, return immediately
  without printing any summary.

Return false on verification failure (filer_sync.go)
  runVerifySync returned true (exit 0) even when diffs were found or the
  walk failed. Return false so the main binary sets exit status 1,
  consistent with how all other commands signal failure.

* test(filer.sync): add missing verifySync test coverage

Four new tests covering gaps identified during review:

TestVerifySyncETagMismatch
  Verifies that two files with identical size but different Md5 checksums
  are counted as etagMismatch (not sizeMismatch). Exercises the second
  branch of compareEntries that was previously untested.

TestVerifySyncCutoffTime (4 subtests)
  A-only recent  — recent file skipped (skippedRecent++), not MISSING
  A-only old     — old file reported as MISSING
  B-only recent  — recent file skipped (skippedRecent++), not ONLY_IN_B
  B-only old     — old file reported as ONLY_IN_B
  The B-only subtests specifically cover the cutoffTime fix added in the
  previous commit.

TestVerifySyncRootPath
  Regression for the path.Join fix: walks from "/" and verifies that the
  child directory is reached and compared correctly (the old Sprintf
  produced "//data" which would silently produce wrong results).
  Asserts dirCount=2 and fileCount=1 to confirm the full tree is walked.

* fix(filer.sync): use os.Exit(2) instead of return false on verify failure

return false triggered weed.go's error handler which printed the full
command usage — appropriate for invalid arguments, not for a completed
verification that found differences. Use os.Exit(2) consistent with
the existing pattern in filer_sync.go (lines 251, 293).

* refactor(filer.sync.verify): split verify into its own command

The verify mode is a one-shot batch operation with a fundamentally
different lifecycle from the long-running sync subscriber, and most of
filer.sync's flags (replication, metrics port, debug pprof, concurrency,
etc.) do not apply to it. Extract it into a sibling command alongside
filer.copy/filer.backup/filer.export rather than a flag mode on
filer.sync.

Also rename modifyTimeAgo to modifiedTimeAgo (grammatical) and drop the
verifyJsonOutput prefix to plain jsonOutput now that the verify context
is implicit in the command name.

* fix(filer.sync.verify): address review comments

- Bounded worker pool: cap subdirectory goroutines per level via a
  jobs channel and min(verifySyncConcurrency, len(subDirs)) workers
  instead of spawning one goroutine per child. Wide directories no
  longer park ~2KB per queued goroutine.

- Don't gate recursion on a directory's mtime: a fresh child write
  bumps the parent mtime, but older files inside should still be
  reported as missing. Always recurse for missing-in-B directories
  and apply the cutoff per-file inside countMissingRecursive.

- Apply -modifiedTimeAgo symmetrically: matched-name files now skip
  the comparison when EITHER side is recently modified, not just A.
  This restores lag tolerance when B was just rewritten.

Adds tests for both new behaviors and a shared isTooRecent helper.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
 2026-04-29 12:33:53 -07:00
chenshiandchenshi5012 eebffd9df6 fix(storage): fix verifyDeletedNeedleIntegrity using wrong offset (#9273) 
* fix(storage): fix verifyDeletedNeedleIntegrity using wrong offset

verifyDeletedNeedleIntegrity was ignoring the offset recorded in the
index file and instead always reading from fileSize-DiskSize(0), which
is only correct when the deleted needle happens to be the very last
entry in the .dat file.

When a deletion tombstone is written in the middle of the volume (e.g.
after subsequent writes), the function reads the wrong bytes, causing
the key-mismatch check to either silently pass (if the bytes happen to
form a valid needle with the same ID) or produce a spurious integrity
error.

Fix: accept the actual offset from the index entry and use
types.TombstoneFileSize as the size, mirroring the approach used by
verifyNeedleIntegrity for live needles.

* fix(storage): fix error message in doCheckAndFixVolumeData

The error message in the deleted-needle branch referenced
verifyNeedleIntegrity, but the function being called is
verifyDeletedNeedleIntegrity. This makes log output misleading when
a deleted-needle integrity check fails.

Suggested by gemini-code-assist review on PR #9273.

---------

Co-authored-by: chenshi5012 <chenshi5012@github.com>
 2026-04-29 11:22:29 -07:00
baracudazCopilotbaracudaz
 		db34e8b6fd feat(admin): prefer stored S3 Content-Type metadata over key-extension MIME inference (#9286) 
* feat(admin): prefer stored filer mime in file browser and properties

* feat(mime): enhance MIME type registration and improve fallback logic

Co-authored-by: Copilot <copilot@github.com>

---------

Co-authored-by: baracudaz <baracudaz@users.noreply.github.com>
Co-authored-by: Copilot <copilot@github.com>
 2026-04-29 10:21:03 -07:00
qzhello 60c76120fc fix(shell): use exact match for volume.balance -racks/-nodes filter (#9279) 
* fix(shell): correct volume.list -writable filter unit and comparison

* fix(shell): correct volume.list -writable filter unit and comparison

* chore(shell): fix typo in EC shard helper param names

* fix(shell): use exact match for volume.balance -racks/-nodes filter

The old strings.Contains-based filter quietly included any id that was a
  substring of the user-supplied flag value (e.g. -racks=rack10 also matched
  rack1). Replace it with an exact-match set parsed from the comma-separated
  flag value, and add regression tests for both -racks and -nodes paths.

  Also fix a small typo in the "remote storage" error returned by
  maybeMoveOneVolume.

* fix(shell): use exact match for volume.balance -racks/-nodes filter

The old strings.Contains-based filter quietly included any id that was a
  substring of the user-supplied flag value (e.g. -racks=rack10 also matched
  rack1). Replace it with an exact-match set parsed from the comma-separated
  flag value, and add regression tests for both -racks and -nodes paths.

  Also fix a small typo in the "remote storage" error returned by
  maybeMoveOneVolume.

* refactor(shell): drop nil sentinel in splitCSVSet, use len() in callers
 2026-04-29 10:19:16 -07:00
Chris Lu 1f515f9d02 fix(s3api): cross-SSE copy operations and bring them back into CI (#9281) (#9282) 
* fix(s3api): cross-SSE copy operations and bring them back into CI (#9281)

Four cross-SSE copy tests were broken on master and excluded from CI with
the comment "pre-existing SSE-C issues":

  - TestSSECObjectCopyIntegration/Copy_SSE-C_to_SSE-C_with_different_key
  - TestSSEKMSObjectCopyIntegration/Copy_SSE-KMS_with_different_key
  - TestCrossSSECopy/SSE-S3_to_SSE-C
  - TestSSEMultipartCopy/Copy_SSE-KMS_Multipart_Object

Each surfaced as a different symptom — 500 InternalError, CRC32 mismatch,
"unexpected EOF", MD5 mismatch — but they were all instances of the same
root pattern that #8908 hit on UploadPartCopy: copy paths writing
destination chunks tagged inconsistently with the bytes on disk, so
detectPrimarySSEType / IsSSE*Encrypted disagreed about what the read
path should do.

Five fixes in this PR, each with its own targeted test:

  1. SSE-C IV format: putToFiler stored entry.Extended[SeaweedFSSSEIV]
     as raw bytes (with a comment saying so), but StoreSSECIVInMetadata
     stored it base64-encoded. The two readers (the GET handler reading
     it raw, and GetSSECIVFromMetadata reading it base64-decoded) each
     matched one writer but not the other. Standardise on raw bytes
     everywhere; GetSSECIVFromMetadata accepts the legacy base64 form
     for backward compat.

  2. SSE-C single-part copy chunk tagging: copyChunkWithReencryption
     re-encrypted the bytes for the destination but never set the
     destination chunk's SseType / SseMetadata. With chunks left
     SseType=NONE, detectPrimarySSEType returned "None" and the GET
     served still-encrypted volume bytes raw without decryption.
     Tag the chunk after re-encryption.

  3. SSE-KMS single-part copy chunk tagging: same shape as (2). Also,
     the function discarded the destSSEKey returned from
     CreateSSEKMSEncryptedReaderWithBucketKey (with `_`) — that key
     carries the freshly-minted EncryptedDataKey + IV the read path
     needs, so it must be captured and serialized into the destination
     chunk's per-chunk metadata (and bubbled up to the entry-level
     SeaweedFSSSEKMSKey for single-chunk objects whose read path falls
     back to the entry-level key).

  4. SSE-KMS multipart source decryption: copyChunkWithSSEKMSReencryption
     decrypted every source chunk with the entry-level sourceSSEKey.
     For multipart SSE-KMS objects each chunk has its own EDK + IV in
     per-chunk metadata, so the entry-level key is wrong. Decrypt with
     per-chunk metadata when present.

  5. Same-key copy fast path chunk tagging: copySingleChunk uses
     createDestinationChunk which dropped SseType / SseMetadata. For
     same-key copies (e.g. SSE-KMS source → SSE-KMS dest with the same
     KMS key) the fast path reuses the source ciphertext as-is, so the
     destination chunks must keep the source's SSE tagging. Add a
     createDestinationChunkPreservingSSE helper for the fast path; the
     re-encryption paths still call createDestinationChunk and then
     overwrite the SSE fields after re-encrypting.

CI: extend the comprehensive-test TEST_PATTERN to include the four test
families that were previously excluded (`.*ObjectCopyIntegration`,
`TestCrossSSECopy`, `TestSSEMultipartCopy`) so this category of
regression is caught going forward. The exclusion comment is removed.

Tests:
  - All four originally-failing tests pass.
  - The full pre-existing TestSSE* / TestCrossSSE / TestGitHub7562 /
    TestCopyToBucketDefaultEncryptedRegression / TestSSEMultipart suite
    still passes.
  - go test -race ./weed/s3api/ passes.

Refs #8908, #9280.

* fix(s3api): SSE-KMS copy ChunkOffset must stay 0 (review feedback on #9282)

CreateSSEKMSEncryptedReaderWithBucketKey initialises a fresh CTR stream
at counter 0 with a per-chunk random IV — there is no base-IV-plus-offset
relationship. The previous commit on this branch wrote
`destSSEKey.ChunkOffset = chunk.Offset` onto the per-chunk metadata,
which the read-side CreateSSEKMSDecryptedReader applies as
calculateIVWithOffset(IV, ChunkOffset) — i.e. it advances the
decryption IV by chunk.Offset/16 blocks beyond where the encryption
actually wrote.

The bug only manifests for SSE-KMS-to-SSE-KMS-with-different-key copies
of multipart sources (where source chunks live at non-zero offsets),
which is why the existing TestSSEKMSObjectCopyIntegration (single-chunk
source) and TestSSEMultipartCopy/Copy_SSE-KMS_Multipart_Object (same-key
copy that takes the fast preserving path, not the re-encrypt path)
both happened to pass. Set ChunkOffset to 0 to match the actual
encryption position. Existing tests still pass; the dangerous case is
only reachable with a multipart SSE-KMS source and a different
destination key, which is not currently exercised in CI.

Found by gemini-code-assist review on PR #9282.

* fix(s3api): use first dst chunk's full key for entry-level SSE-KMS metadata in remaining copy paths (review feedback on #9282)

Earlier this branch fixed copyChunksWithSSEKMSReencryption to populate
the entry-level SeaweedFSSSEKMSKey from the first destination chunk's
fully-formed metadata (with EDK + IV) instead of a stub key with only
KeyID + EncryptionContext + BucketKeyEnabled. The same fix needs to
apply to the other two paths that build entry-level SSE-KMS metadata:

  - copyMultipartCrossEncryption() — cross-encryption to SSE-KMS dest.
    Per-chunk metadata comes from copyCrossEncryptionChunk's
    CreateSSEKMSEncryptedReaderWithBucketKey call, so chunks[0] has
    a real EDK + IV. Use it.

  - copyChunksWithSSEKMS() direct (same-key) branch. After
    createDestinationChunkPreservingSSE in copySingleChunk, dst chunks
    carry the source's per-chunk SSE-KMS metadata. Use chunks[0] for
    the entry-level key so single-chunk same-key copies don't fall
    back to a stub key on the read path.

Without this, single-chunk SSE-KMS reads through these two paths
failed at GET with "Invalid ciphertext format" — KMS unwrap was
called on an empty EDK.

Found by coderabbitai review on PR #9282.

* fix(s3api): add 0-byte fallback to SSE-KMS reencryption entry-level metadata (review feedback on #9282)

copyChunksWithSSEKMSReencryption was missing the fallback for 0-byte
objects (where dstChunks is empty), inconsistent with the fallback in
copyChunksWithSSEKMS direct branch and copyMultipartCrossEncryption.
Without it, a 0-byte SSE-KMS copy would land with no entry-level
SeaweedFSSSEKMSKey, so the read path's IsSSEKMSEncryptedInternal check
would not recognise the empty object as SSE-KMS.

Mirror the existing fallback: build a stub SSEKMSKey with KeyID,
context and bucket-key state; serialize it as the entry-level key.

Found by gemini-code-assist review on PR #9282.

* fix(s3api): SSE-KMS direct copy must check encryption context + bucket-key, not just key ID (review feedback on #9282)

DetermineSSEKMSCopyStrategy / CanDirectCopySSEKMS only compares the
source and destination KMS key IDs, but the destination request can
also change the encryption context or the BucketKey flag. Both are
embedded in the source ciphertext's wrapped EDK; preserving the source
metadata verbatim does not satisfy a destination request that asks
for different settings, so the destination object would silently
report the source's context/flag instead of what was requested.

Add srcSSEKMSStateMatchesDest: deserialize the source's stored
SSEKMSKey and compare its EncryptionContext + BucketKeyEnabled to the
destination request. If either differs, force the slow re-encrypt
path (SSEKMSCopyStrategyDecryptEncrypt) so the destination gets a
freshly-wrapped EDK bound to the requested context/flag.

A malformed source key is treated as non-matching (conservative).
nil and empty encryption-context maps are treated as equal to avoid
spurious divergence when the request omits the context header.

Found by coderabbitai review on PR #9282.

* fix(s3api): copyMultipartSSEKMSChunk falls back to entry-level key + entry-level metadata uses first chunk's full key (review feedback on #9282)

Two related issues in copyMultipartSSEKMSChunks / copyMultipartSSEKMSChunk:

1. copyMultipartSSEKMSChunks built the destination's entry-level
   SeaweedFSSSEKMSKey from a stub (KeyID + context + bucket-key only),
   missing the EDK + IV. Single-chunk reads through this path fall
   back to entry-level keyData and would fail at GET because KMS would
   be asked to unwrap an empty EDK. Mirrors the fix in
   copyChunksWithSSEKMS / copyMultipartCrossEncryption /
   copyChunksWithSSEKMSReencryption: prefer the first dst chunk's
   full per-chunk metadata, fall back to the stub only for 0-byte
   objects.

2. copyMultipartSSEKMSChunk hard-failed when chunk.GetSseMetadata()
   was empty. Newer multipart SSE-KMS uploads populate per-chunk
   metadata, but legacy objects may have only entry-level metadata
   and would now be impossible to copy. Add a sourceEntrySSEKey
   fallback parameter (deserialized once by the caller from
   entry.Extended[SeaweedFSSSEKMSKey]); use it when per-chunk
   metadata is absent.

Found by coderabbitai review on PR #9282.

* refactor(s3api): extract entry-level SSE-KMS deserialization and per-chunk fallback into helpers (review feedback on #9282)

Three medium-priority maintainability comments from gemini-code-assist:

  - The same "deserialize entry.Extended[SeaweedFSSSEKMSKey]" pattern
    appeared in srcSSEKMSStateMatchesDest, copyMultipartSSEKMSChunks
    and copyChunksWithSSEKMSReencryption.

  - The "prefer per-chunk metadata, fall back to entry-level key"
    selection logic appeared inline in copyMultipartSSEKMSChunk and
    copyChunkWithSSEKMSReencryption with subtly different shapes.

  - encryptionContextEqual hand-rolled a map comparison.

Pull both patterns out into named helpers:

  - deserializeEntrySSEKMSKey: returns the entry-level SSEKMSKey or nil
    on missing/malformed data, with a single V(2) log line.
  - resolveChunkSSEKMSKey: centralises the chunk-vs-entry-level
    selection so all sites use the same decryption-side selection
    logic (which must mirror the encryption side).

Replace encryptionContextEqual's manual loop with reflect.DeepEqual,
keeping the empty-vs-nil shortcut at the top because DeepEqual treats
those as different.

No behaviour change; existing copy tests still pass.
 2026-04-29 10:06:51 -07:00
Chris Lu 82cf60a44f fix(s3api): re-encrypt UploadPartCopy bytes for the destination's SSE config (#8908) (#9280) 
* fix(s3api): re-encrypt UploadPartCopy bytes for the destination's SSE config (#8908)

The remaining failure mode in #8908 was that Docker Registry's blob
finalization (server-side Move via UploadPartCopy) silently corrupts
SSE-S3 multipart objects. Reproduces with `aws s3api upload-part-copy`
under bucket-default SSE-S3: the GET on the completed object returns
deterministic wrong bytes (correct length, same wrong SHA-256 across
runs). The metadata is mathematically self-consistent — every chunk's
stored IV equals `calculateIVWithOffset(baseIV_dst, partLocalOffset)` —
but the bytes on disk were encrypted with the SOURCE upload's key+baseIV.

Root cause:

  - `copyChunksForRange` (and `createDestinationChunk`) constructs new
    chunks for UploadPartCopy without copying `SseType` / `SseMetadata`,
    so destination chunks are written with `SseType=NONE`.

  - At completion, `completedMultipartChunk` (PR #9224's NONE→SSE_S3
    backfill, intended to recover from a different missing-metadata
    bug) sees those NONE chunks under an SSE-S3 multipart upload and
    backfills SSE-S3 metadata derived from the destination upload's
    baseIV. The chunk metadata is now internally consistent and the
    GET path applies decryption — but the bytes on disk are encrypted
    with the source upload's key, not the destination's. Decryption
    produces deterministic garbage. Docker Registry pulls then fail
    with "Digest did not match".

Fix: when either the source object or the destination multipart upload
has any SSE configured, take a slow-path UploadPartCopy that
(1) opens a plaintext reader of the source range — decrypting the
source's per-chunk SSE-S3 metadata if needed via a reused
`buildMultipartSSES3Reader`, and
(2) feeds that plaintext through `putToFiler`'s existing encryption
pipeline by staging the destination upload entry's SSE-S3/SSE-KMS
headers on a cloned request. Encryption then matches PutObjectPart's
contract: every part starts a fresh CTR stream from counter 0 with
`baseIV_dst`, and each internal chunk's metadata records
`calculateIVWithOffset(baseIV_dst, chunk.partLocalOffset)`.

The `non-SSE → non-SSE` case still takes the existing fast raw-byte
copy path — bytes on disk are plaintext on both sides, so chunk-level
metadata is irrelevant.

Cross-encryption from SSE-KMS / SSE-C sources is left as TODO — the
new path returns an explicit error rather than the previous silent
corruption. SSE-S3 (the user-reported case) round-trips correctly.

Tests:

  - test/s3/sse/s3_sse_uploadpartcopy_integration_test.go pins three
    UploadPartCopy shapes against bucket-default SSE-S3:
    * Docker-Registry-shape 32MB+tail (the user's exact 5-chunk /
      2-part metadata layout)
    * single full-object UploadPartCopy
    * many small range copies
    Each round-trips SHA-256.

  - test/s3/sse/s3_sse_concurrent_repro_test.go covers the parallel
    multipart-upload shape from the user report (5 blobs in parallel,
    full GET and chunked range GET both hash-checked) — pre-existing
    coverage; added here as a regression sentinel.

* test(s3-sse): rename UploadPartCopy regression test so CI matches it

The CI workflow .github/workflows/s3-sse-tests.yml dispatches on the
TEST_PATTERN ".*Multipart.*Integration" — i.e. the test name must contain
both "Multipart" and "Integration" for CI to run it. The previous name
TestSSES3UploadPartCopyIntegration had only "Integration"; "UploadPart"
isn't "Multipart". Rename to TestSSES3MultipartUploadPartCopyIntegration
so the regression test actually runs in CI rather than only locally.

* fix(s3api): map unsupported UploadPartCopy SSE source to 501, not 500 (review feedback on #9280)

openSourcePlaintextReader explicitly rejects SSE-KMS and SSE-C sources
(SSE-S3 is the only one wired up in this slow path so far). Earlier
the caller blanket-mapped that to ErrInternalError, which collapses
"this shape isn't implemented yet" into the same 500 response a
real server failure would produce. Clients can no longer tell whether
they hit a feature gap or a bug.

Introduce a sentinel errCopySourceSSEUnsupported and have
copyObjectPartViaReencryption errors.Is-check it; on match, return
ErrNotImplemented (501) instead of ErrInternalError (500). Other
failures still map to 500.

Found by coderabbitai review on PR #9280.

* fix(s3api): UploadPartCopy must fail with NoSuchUpload when upload entry is missing (review feedback on #9280)

CopyObjectPartHandler's earlier checkUploadId call only verifies that
the uploadID's hash prefix matches dstObject; it does not prove the
upload directory exists in the filer. The previous logic silently
swallowed filer_pb.ErrNotFound from getEntry(uploadDir) and fell
through with uploadEntry=nil, which then skipped the destination
SSE check and could route a plain-source copy through the raw-byte
fast path even though the destination's encryption state is unknown.

Treat ErrNotFound as ErrNoSuchUpload so the client sees the right
status, matching the AWS S3 contract for UploadPartCopy on a
non-existent upload.

Found by coderabbitai review on PR #9280.

* feat(s3api): set SSE response headers on UploadPartCopy slow path (review feedback on #9280)

PutObjectPartHandler writes x-amz-server-side-encryption (and the KMS
key-id header for SSE-KMS) on every successful part response so clients
can confirm the destination's encryption state. The new UploadPartCopy
slow path was missing this — it returned only the ETag in the response
body and no SSE response headers.

Plumb putToFiler's SSEResponseMetadata back through
copyObjectPartViaReencryption to the handler, then call
setSSEResponseHeaders before writing the XML response, matching the
PutObjectPart contract.

Found by gemini-code-assist review on PR #9280.

* fix(s3api): map transient filer errors on UploadPartCopy upload-entry fetch to 503 (review feedback on #9280)

Earlier non-ErrNotFound errors from getEntry(uploadDir, uploadID) all
returned 500 InternalError, which most SDKs treat as fatal — even
though a transient filer outage (gRPC Unavailable, leader election in
flight, deadline exceeded) is exactly the kind of failure SDK retry
logic is supposed to recover from.

Add an isTransientFilerError helper that recognises:
  - context.DeadlineExceeded / context.Canceled
  - gRPC codes.Unavailable, DeadlineExceeded, ResourceExhausted, Aborted

When the upload-entry fetch fails for one of those reasons, return
503 ServiceUnavailable so the client retries; everything else still
maps to 500. Log line now also carries dstObject (in addition to
dstBucket and uploadID) to make incident triage easier.

Found by gemini-code-assist review on PR #9280.
 2026-04-29 09:46:44 -07:00
Chris Lu 7a461ffc2f fix(mount): copy xattr value bytes to avoid FUSE buffer aliasing (#9278) 
fix(mount): copy xattr value bytes to avoid FUSE buffer aliasing (#9275)

SetXAttr stored the caller-supplied `data` slice directly into
entry.Extended. That slice aliases go-fuse's per-request input buffer,
which is returned to a pool the moment the handler returns. When a file
is open during setxattr (the open-fh path defers persistence to flush),
the next FUSE request recycles the buffer and silently overwrites the
stored xattr bytes; flushMetadataToFiler then ships the corrupted bytes
to the filer. `cp -a` reproduces this because it issues a setxattr while
holding an open fh, then continues to issue follow-up FUSE ops that
reuse the same buffer.

The path-based setxattr (e.g. setfattr without an open fh) saves
synchronously inside the same handler, so the bytes were marshalled
before the buffer could be reused — that is why the source file in the
report looked fine and only the cp -a destination was garbage.

Defensively copy the bytes when storing them, and add a unit test that
mutates the caller buffer after SetXAttr returns to lock in the
invariant.
 2026-04-28 23:54:35 -07:00
qzhello 108e42fb8b chore(shell): fix typo in EC shard helper param names (#9277) 
* fix(shell): correct volume.list -writable filter unit and comparison

* fix(shell): correct volume.list -writable filter unit and comparison

* chore(shell): fix typo in EC shard helper param names
 2026-04-28 23:09:26 -07:00
Chris Lu d9b86fb495 fix(s3api): clear stale latest-version pointer when .versions dir cleanup is blocked (#9269) 
* fix(s3api): clear stale latest-version pointer when post-deletion cleanup is blocked

In versioned buckets, when the only remaining children of a .versions
directory are orphan entries (v_<id> files that lack the version-id
extended attribute - e.g. left behind by older code paths or interrupted
writes), updateLatestVersionAfterDeletion's selectLatestVersion finds
zero promotable versions and falls through to s3a.rm. The non-recursive
rm fails because the orphan blocks the directory deletion. Previously
the code logged the failure and returned, leaving the LatestVersionId
pointer pointing at the version we just deleted.

For Veeam-style workloads that GET-PUT-GET-DELETE a small lock object on
every checkpoint, the stale pointer poisons every subsequent run: the
next GET re-enters getLatestObjectVersion's 13x retry loop on the
missing pointer file plus the self-heal rescan, all to return the same
404. The cycle is self-perpetuating until the orphan is removed by hand.

When rm fails, additionally clear the LatestVersionId / LatestVersionFile
pointer fields on the .versions directory entry. The orphan files stay
in place (an operator can audit and remove them); from the S3 API
perspective the object is now correctly absent and subsequent reads
short-circuit to ErrNotFound on the fast path instead of replaying the
heal cycle.

* fix(s3api): clear stale latest-version pointer on read-side self-heal failure

healStaleLatestVersionPointer is invoked by getLatestObjectVersion when the
pointed-at version file is missing. The rescan path can find no remaining
version-id-tagged entries (e.g. when only orphan v_<id> files lacking the
version-id extended attribute remain). Prior code returned ErrNotFound but
left the stale pointer in place, so every subsequent read replayed the same
13x retry loop on the missing file and re-entered self-heal, all to return
the same 404.

Reuse the same pointer-clear logic introduced for updateLatestVersionAfterDeletion.
The two call sites are now identical, so factor the body out into
clearStaleLatestVersionPointer. The caller parameter carries the source
function name so the log lines operators were already grepping
(updateLatestVersionAfterDeletion: cleared stale ... and
healStaleLatestVersionPointer: cleared stale ...) keep the same prefix.

* fix(s3api): re-validate before clearing latest-version pointer (CAS)

Reviewer feedback (gemini-code-assist, coderabbitai) on PR #9269 flagged
a TOCTOU in clearStaleLatestVersionPointer: between the caller loading
versionsEntry and this function persisting the cleared map, a concurrent
PUT could promote a fresh version. Persisting the caller's snapshot then
silently overwrites the live pointer and re-introduces the missing-pointer
state on a now-existing object.

Make the persist CAS-style:

  1. Re-scan .versions for any version-id-tagged entry. If one now exists,
     abort - the concurrent writer has populated the directory and either
     already updated the pointer or the next read's self-heal will pick
     up the new entry.
  2. Re-fetch the live .versions directory entry and only proceed if its
     latest-pointer fields still match the stale id the caller observed.
     A concurrent pointer update by another writer is detected here and
     the clear is skipped.
  3. Persist with mkFile using the live Extended map (minus the two
     pointer fields and the cached metadata) so any other Extended fields
     written concurrently between (2) and the persist are preserved.

A note on the literal suggestion of mutating updatedEntry.Extended in
the mkFile callback: that does not work because mkFile constructs a
fresh *filer_pb.Entry rather than reading the live entry first
(weed/pb/filer_pb/filer_client.go:247). The callback's updatedEntry is
nil at invocation, so a delete on it would be a no-op and we would lose
all Extended fields, not just the two we want to clear. The correct
shape - re-fetching the live entry before mkFile and carrying its
Extended map into the persist - is what this change implements.

True atomic CAS would require filer-level conditional update support;
this change narrows the race window from the full caller scope to the
~ms gap between (2) and (3), which is the best we can do without that.
 2026-04-28 21:02:52 -07:00
chenshi c93018d987 fix(s3api): fix uint16 overflow in doListFilerEntries Limit calculation (#9271) 2026-04-28 20:50:05 -07:00
Chris Lu 35fe3c801b feat(nfs): UDP MOUNT v3 responder + real-Linux e2e mount harness (#9267) 
* feat(nfs): add UDP MOUNT v3 responder

The upstream willscott/go-nfs library only serves the MOUNT protocol
over TCP. Linux's mount.nfs and the in-kernel NFS client default
mountproto to UDP in many configurations, so against a stock weed nfs
deployment the kernel queries portmap for "MOUNT v3 UDP", gets port=0
("not registered"), and either falls back inconsistently or surfaces
EPROTONOSUPPORT — surfacing as the user-visible "requested NFS version
or transport protocol is not supported" reported in #9263. The user has
to add `mountproto=tcp` or `mountport=2049` to mount options to coerce
TCP just for the MOUNT phase.

Add a small UDP responder that speaks just enough of MOUNT v3 to handle
the procedures the kernel actually invokes during mount setup and
teardown: NULL, MNT, and UMNT. The wire layout for MNT mirrors
handler.go's TCP path so both transports produce the same root
filehandle and the same auth flavor list for the same export. Other
v3 procedures (DUMP, EXPORT, UMNTALL) cleanly return PROC_UNAVAIL.

This commit only adds the responder; portmap-advertise and Server.Start
wire-up follow in subsequent commits so each step stays independently
reviewable.

References: RFC 1813 §5 (NFSv3/MOUNTv3), RFC 5531 (RPC). Existing
constants and parseRPCCall / encodeAcceptedReply helpers from
portmap.go are reused so behaviour stays consistent across both UDP
listening goroutines.

* feat(nfs): advertise UDP MOUNT v3 in the portmap responder

The portmap responder advertised TCP-only entries because go-nfs only
serves TCP, but with the new UDP MOUNT responder in place we can now
honestly advertise MOUNT v3 over UDP as well. Linux clients whose
default mountproto is UDP query portmap during mount setup; if the
answer is "not registered" some kernels translate the result to
EPROTONOSUPPORT instead of falling back to TCP, which is exactly the
failure pattern reported in #9263.

Add the entry, refresh the doc comment, and extend the existing
GETPORT and DUMP unit tests so a regression that drops the entry shows
up at unit-test granularity rather than only in an end-to-end mount.

* feat(nfs): start UDP MOUNT v3 responder alongside the TCP NFS listener

Plug the new mountUDPServer into Server.Start so it comes up on the
same bind/port as the TCP NFS listener. Started before portmap so a
portmap query that races a fast client never returns a UDP MOUNT entry
the responder isn't actually answering, and shut down via the same
defer chain so a portmap-or-listener startup failure doesn't leave the
UDP responder dangling.

The portmap startup log now reflects all three advertised entries
(NFS v3 tcp, MOUNT v3 tcp, MOUNT v3 udp) so operators can confirm at a
glance that the UDP MOUNT path is up.

Verified end-to-end: built a Linux/arm64 binary, ran weed nfs in a
container with -portmap.bind, and mounted from another container using
both the user-reported failing setup from #9263 (vers=3 + tcp without
mountport) and an explicit mountproto=udp to force the new code path.
The trace `mount.nfs: trying ... prog 100005 vers 3 prot UDP port 2049`
now leads to a successful mount instead of EPROTONOSUPPORT.

* docs(nfs): note that the plain mount form works on UDP-default clients

With UDP MOUNT v3 now served alongside TCP, the only path that ever
required mountproto=tcp / mountport=2049 — clients whose default
mountproto is UDP — works against the plain mount example. Update the
startup mount hint and the `weed nfs` long help so users don't go
hunting for a mount-option workaround that no longer applies.

The "without -portmap.bind" branch is unchanged: that path still has
to bypass portmap entirely because there is no portmap responder for
the kernel to query.

* test(nfs): add kernel-mount e2e tests under test/nfs

The existing test/nfs/ harness boots a real master + volume + filer +
weed nfs subprocess stack and drives it via go-nfs-client. That covers
protocol behaviour from a Go client's perspective, but anything
mis-coded once a real Linux kernel parses the wire bytes is invisible:
both ends of the test use the same RPC library, so identical bugs
round-trip cleanly. The two NFS issues hit recently were exactly that
shape — NFSv4 mis-routed to v3 SETATTR (#9262) and missing UDP MOUNT v3
— and only surfaced in a real client.

Add three end-to-end tests that mount the harness's running NFS server
through the in-tree Linux client:

  - TestKernelMountV3TCP: NFSv3 + MOUNT v3 over TCP (baseline).
  - TestKernelMountV3MountProtoUDP: NFSv3 over TCP, MOUNT v3 over UDP
    only — regression test for the new UDP MOUNT v3 responder.
  - TestKernelMountV4RejectsCleanly: vers=4 against the v3-only server,
    asserting the kernel surfaces a protocol/version-level error rather
    than a generic "mount system call failed" — regression test for the
    PROG_MISMATCH path from #9262.

The tests pass explicit port=/mountport= mount options so the kernel
never queries portmap, which means the harness doesn't need to bind
the privileged port 111 and won't collide with a system rpcbind on a
shared CI runner. They t.Skip cleanly when the host isn't Linux, when
mount.nfs isn't installed, or when the test process isn't running as
root.

Run locally with:

	cd test/nfs
	sudo go test -v -run TestKernelMount ./...

CI wiring follows in the next commit.

* ci(nfs): run kernel-mount e2e tests in nfs-tests workflow

Wire the new TestKernelMount* tests from test/nfs into the existing
NFS workflow:

  - Existing protocol-layer step now skips '^TestKernelMount' so a
    "skipped because not root" line doesn't appear on every run.
  - New "Install kernel NFS client" step pulls nfs-common (mount.nfs +
    helpers) and netbase (/etc/protocols, which mount.nfs's protocol-
    name lookups need to resolve `tcp`/`udp`).
  - New privileged step runs only the kernel-mount tests under sudo,
    preserving PATH and pointing GOMODCACHE/GOCACHE at the user's
    caches so the second `go test` invocation reuses already-built
    test binaries instead of redownloading modules under root.

The summary block now lists the three kernel-mount cases explicitly
so a regression on either of #9262 or this PR's UDP MOUNT change is
traceable from the workflow run page.
 2026-04-28 14:06:35 -07:00
Chris Lu 735e94f6ba mount: expose -fuse.maxBackground and -fuse.congestionThreshold flags (closes #9258) (#9268) 
* mount: expose `-fuse.maxBackground` flag (closes #9258)

The Linux FUSE driver caps in-flight async requests via
`/sys/fs/fuse/connections/<id>/max_background` (and a derived
`congestion_threshold = 3/4 * max_background`). Heavy upload workloads
need this raised, but the cap currently lives only in `/sys`, so it
resets on reboot/remount.

`weed mount` was hardcoding `MaxBackground: 128`. Promote it to a flag,
default unchanged. Setting `-fuse.maxBackground=2048` reproduces the
manual `echo 2048 > .../max_background` (and gives 1536 for
congestion_threshold automatically) persistently across remounts.

`congestion_threshold` is not exposed as a separate flag because
go-fuse derives it as 3/4 of MaxBackground in InitOut and offers no
hook to override; users wanting a different ratio can still write
/sys/fs/fuse/connections/<id>/congestion_threshold post-mount.

* mount: add `-fuse.congestionThreshold` flag, bump go-fuse to v2.9.3

go-fuse v2.9.3 exposes CongestionThreshold as a separate MountOption,
so we can now let users override the kernel's default 3/4-of-max_background
ratio at mount time instead of having to write
/sys/fs/fuse/connections/<id>/congestion_threshold post-mount on every
remount/reboot.

Default 0 preserves existing behavior (kernel derives it as
3/4 * max_background). Non-zero is sent to the kernel verbatim; the
kernel clamps it to max_background if higher.
 2026-04-28 13:42:58 -07:00