mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 15:41:15 +02:00
8f2a3d92bb51bbac5783f8e33a4bb65e0bb51007
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Lu
|
8f2a3d92bb |
docker: upgrade libcrypto3/libssl3 to clear Trivy HIGH (CVE-2026-28390) (#9059)
* docker: upgrade libcrypto3/libssl3 to clear Trivy HIGH Trivy gate on ghcr.io/seaweedfs/seaweedfs:latest-amd64 flagged CVE-2026-28390 in libcrypto3 3.5.5-r0 (fixed in 3.5.6-r0) on the alpine 3.23.3 base. Add libcrypto3/libssl3 to the existing apk upgrade so rebuilt images pick up the patched openssl without waiting for a new alpine base tag. * docker: apk add libcrypto3/libssl3 so they install at patched version Per review, apk upgrade <pkg> is a no-op when the package isn't already installed. libcrypto3/libssl3 come in transitively via curl, so list them in apk add to guarantee installation at the latest (patched) version from the alpine repo. |
||
|
Chris Lu
|
ef77df6141 |
admin: include EC volumes in bucket size reporting (#9058)
* admin: include EC volumes in bucket size reporting The Object Store buckets page computed per-collection size by iterating only regular volumes, so once a bucket's data was EC-encoded it silently disappeared from the reported size — breaking usage-based billing. Walk EcShardInfos alongside VolumeInfos in collectCollectionStats: add raw shard bytes to PhysicalSize, and the parity-stripped value (shardBytes * DataShardsCount / TotalShardsCount) to LogicalSize, matching the normalization used by `weed shell` cluster.status. * admin: derive EC logical size from shard bitmap, not constants Use ShardsInfoFromVolumeEcShardInformationMessage + MinusParityShards to sum actual data-shard bytes instead of scaling raw bytes by the DataShardsCount/TotalShardsCount ratio. Keeps the data/parity split encapsulated in the erasure_coding package and is exact when shard sizes differ (e.g. last shard). * admin: regression test for EC shard size aggregation Cover the uneven-tail-shard case (data shard 9 < 1000 bytes) and the empty-collection-name path to pin PhysicalSize/LogicalSize behavior for collectCollectionStats against future changes. |
||
|
Chris Lu
|
7aaa431bb4 |
s3api: prune bucket-scoped IAM actions on DeleteBucket (#9054)
* s3api: prune bucket-scoped IAM actions on DeleteBucket DeleteBucket removed the bucket directory and collection but left behind any identity actions configured via s3.configure that were scoped to that bucket (e.g. Read:bucket, Write:bucket/prefix), leaving stale auth metadata that users expected to be cleaned up along with the bucket. After a successful delete, strip actions whose resource is exactly the bucket or a prefix under it, save via the credential manager, and let the existing filer metadata subscription fan the reload out to every S3 server. Wildcarded resources and global actions are preserved since they may cover other buckets; static identities are left untouched. Fixes #5310 * s3api: address review feedback on bucket IAM prune - Apply per-identity updates via credentialManager.UpdateUser instead of a full LoadConfiguration/SaveConfiguration round-trip, so the prune no longer clobbers concurrent IAM edits made by s3.configure or the IAM API during a DeleteBucket. - Use a 30s bounded background context for the post-delete cleanup so it survives client disconnect — the bucket is already gone by then and this is best-effort bookkeeping. - Skip static identities via IsStaticIdentity, since the credential store never persists them and UpdateUser would return NotFound. |
||
|
Chris Lu
|
ae08e77979 |
fix(scheduler): give worker tasks a real per-attempt execution deadline (#9041)
* fix(scheduler): give worker tasks a real per-attempt execution deadline The plugin scheduler derived the per-attempt execution deadline as DetectionTimeoutSeconds * 2, which capped every worker task at twice the cluster-scan budget regardless of actual work. For volume_balance batches this was 240s — far too short for 20 large volume copies, so every attempt died at "context deadline exceeded" and all in-flight sub-RPCs surfaced as "context canceled". Retries restarted from move 1 and hit the same wall. Add an explicit ExecutionTimeoutSeconds field to the plugin proto and make each handler declare its own baseline (1800s for vacuum, balance, EC; 3600s for iceberg). Size-aware handlers also emit an estimated_runtime_seconds parameter on each proposal so the scheduler extends the per-attempt deadline based on actual workload: - volume_balance batch: max(largest single move, total / concurrency) at 5 min/GB, so a skewed batch with one big volume isn't averaged away. - volume_balance single, vacuum (already), erasure_coding (10 min/GB), ec_balance (5 min/GB): per-volume budgets. admin_script and iceberg keep the configurable handler default since their workloads are opaque to the detector. * fix(scheduler): apply descriptor defaults to existing persisted configs The previous commit added execution_timeout_seconds to the proto and each handler's descriptor defaults, but two paths still left existing deployments broken: 1. deriveSchedulerAdminRuntime returned stored AdminRuntime configs as-is. Persisted configs from older versions have no execution_timeout_seconds, so the scheduler fell back to the 90s default — worse than the prior 240s behavior. Overlay descriptor defaults for any zero numeric fields when loading. 2. The admin form did not round-trip execution_timeout_seconds, so a normal save would clear it back to zero. Add the input field, the fillAdminSettings/collectAdminSettings hooks, and as defense in depth reapply descriptor defaults in UpdatePluginJobTypeConfigAPI before persisting so a stale form can never silently clobber a baseline. * fix(volume_balance): account for partial scheduling rounds in batch estimate With N moves and C slots, the busiest slot processes ceil(N/C) moves, not N/C. Dividing total seconds by C underestimates wall-clock time whenever N is not a multiple of C — e.g. 6 moves at concurrency 5 needs 2 rounds, not 1.2. Use avg * ceil(N/C) so partial rounds are counted as full ones. * fix(volume_balance): scale minBudget per wave instead of per move Orchestration overhead (setup/teardown for the parallel move runner) happens once per wave, not once per move. Use numRounds*60 as the floor instead of len(moves)*60 so the minimum doesn't inflate linearly with batch size when individual moves are tiny. |
||
|
Chris Lu
|
28d1ef24ec |
fix(admin): allow control chars in file paths when browsing filer (#9043)
* fix(admin): allow control chars in file paths when browsing filer The admin UI rejected any path containing \x00, \r, or \n as "path contains invalid characters". These bytes are legal in S3 object keys, so objects created through the S3 API (or replicated via filer.sync) could exist on the filer but be unreachable from the admin UI — browse, download, and upload all failed with "Invalid file path". Drop the control-character rejection and instead URL-escape the path when constructing filer request URLs, so that such bytes cannot inject into the HTTP request target. Path traversal protection via path.Clean is unchanged. * test(admin): strengthen file path tests with byte-preserving checks Assert full expected output for validateAndCleanFilePath so silent stripping of control characters would fail the test, and cover \r and \x00 escaping in filerFileURL in addition to \n and space. |
||
|
Chris Lu
|
edf7d2a074 |
fix(filer): eliminate redundant disk reads causing memory/CPU regression (#9039)
* fix(filer): eliminate redundant disk reads causing memory/CPU regression (#9035) Since 4.18, LocalMetaLogBuffer's ReadFromDiskFn was set to readPersistedLogBufferPosition, causing LoopProcessLogData to call ReadPersistedLogBuffer on every 250ms health-check tick when a subscriber encounters ResumeFromDiskError. Each call creates an OrderedLogVisitor (ListDirectoryEntries on the filer store), spawns a readahead goroutine with a 1024-element channel, finds no data, and returns — 4 times per second even on an idle filer. This is redundant because SubscribeLocalMetadata already manages disk reads explicitly with its own shouldReadFromDisk / lastCheckedFlushTsNs tracking in the outer loop. Set ReadFromDiskFn back to nil for LocalMetaLogBuffer. When LoopProcessLogData encounters ResumeFromDiskError with nil ReadFromDiskFn, the HasData() guard returns ResumeFromDiskError to the caller (SubscribeLocalMetadata), which blocks efficiently on listenersCond.Wait() instead of polling. * fix(filer): add gap detection for slow consumers after disk-read stall When a slow consumer falls behind and LoopProcessLogData returns ResumeFromDiskError with no flush or read-position progress, there may be a gap between persisted data and in-memory data (e.g. writes stopped while consumer was still catching up). Without this, the consumer would block on listenersCond.Wait() forever. Skip forward to the earliest in-memory time to resume progress, matching the gap-handling pattern already used in the shouldReadFromDisk path. * fix(filer): clear stale ResumeFromDiskError after gap-skip to avoid stall The gap-detection block added in the previous commit skips lastReadTime forward to GetEarliestTime() and continues the outer loop. On the next iteration, shouldReadFromDisk becomes true (currentReadTsNs > lastDiskReadTsNs), the disk read returns processedTsNs == 0, and the existing gap handler at the top of the loop runs its own gap check. That check uses readInMemoryLogErr == ResumeFromDiskError as the entry condition — but readInMemoryLogErr is still the stale error from two iterations ago. GetEarliestTime() now equals lastReadTime.Time (we already advanced to it), so earliestTime.After(lastReadTime.Time) is false and the handler falls into listenersCond.Wait() — stuck. Clear readInMemoryLogErr at the gap-skip point, matching the existing pattern at the earlier gap handler that already clears it for the same reason. * fix(log_buffer): GetEarliestTime must include sealed prev buffers GetEarliestTime previously returned only logBuffer.startTime (the active buffer's first timestamp). That is narrower than ReadFromBuffer's tsMemory, which is the min across active + prev buffers. Callers using GetEarliestTime for gap detection after ResumeFromDiskError (the SubscribeLocalMetadata outer loop's disk-read path, the new gap-skip in the in-memory ResumeFromDiskError handler, and MQ HasData) saw a time that was *newer* than the real earliest in-memory data. Impact in SubscribeLocalMetadata's slow-consumer path: - tsMemory = earliest prev buffer time (T_prev) - GetEarliestTime() = active startTime (T_active, later than T_prev) - Consumer position = T1, with T_prev < T1 < T_active - ReadFromBuffer returns ResumeFromDiskError (T1 < tsMemory) - Gap detect: GetEarliestTime().After(T1) = T_active.After(T1) = true - Skip forward to T_active -- silently drops the prev-buffer data - And when T_active happens to equal the stuck position, gap detect evaluates false, and the subscriber stalls on listenersCond.Wait() This reproduces the TestMetadataSubscribeSlowConsumerKeepsProgressing failure in CI where the consumer stalled at 10220/20000 after writing stopped -- the buffer still had data in prev[0..3], but gap detection was comparing against the active buffer's startTime. Fix: scan all sealed prev buffers under RLock, return the true minimum startTime. Matches the min-of-buffers logic in ReadFromBuffer. * test(log_buffer): make DiskReadRetry test deterministic The previous test added the message via AddToBuffer + ForceFlush and relied on a race: the second disk read had to happen before the data was delivered through the in-memory path. Under the race detector or on a slow CI runner, the reader is woken by AddToBuffer's notification, finds the data in the active buffer or its prev slot, and returns after exactly one disk read — failing the >= 2 disk reads assertion even though the loop behaved correctly. Reproduced on master with race detector (2/5 failures). Rewrite the test to deliver the data exclusively through the disk-read path: no AddToBuffer, no ForceFlush. The test waits until the reader has issued at least one no-op disk read, then atomically flips a "dataReady" flag. The reader's next iteration through readFromDiskFn returns the entry. This deterministically exercises the retry-loop behavior the test was originally written to protect, and removes the in-memory delivery race entirely. |
||
|
Chris Lu
|
10e7f0f2bc |
fix(shell): s3.user.provision handles existing users by attaching policy (#9040)
* fix(shell): s3.user.provision handles existing users by attaching policy Instead of erroring when the user already exists, the command now creates the policy and attaches it to the existing user via UpdateUser. Credentials are only generated and displayed for newly created users. * fix(shell): skip duplicate policy attachment in s3.user.provision Check if the policy is already attached before appending and calling UpdateUser, making repeated runs idempotent. * fix(shell): generate service account ID in s3.serviceaccount.create The command built a ServiceAccount proto without setting Id, which was rejected by credential.ValidateServiceAccountId on any real store. Now generates sa:<parent>:<uuid> matching the format used by the admin UI. * test(s3): integration tests for s3.* shell commands Adds TestShell* integration tests covering ~40 previously untested shell commands: user, accesskey, group, serviceaccount, anonymous, bucket, policy.attach/detach, config.show, and iam.export/import. Switches the test cluster's credential store from memory to filer_etc because the memory store silently drops groups and service accounts in LoadConfiguration/SaveConfiguration. * fix(shell): rollback policy on key generation failure in s3.user.provision If iam.GenerateRandomString or iam.GenerateSecretAccessKey fails after the policy was persisted, the policy would be left orphaned. Extracts the rollback logic into a local closure and invokes it on all failure paths after policy creation for consistency. * address PR review feedback for s3 shell tests and serviceaccount - s3.serviceaccount.create: use 16 bytes of randomness (hex-encoded) for the service account UUID instead of 4 bytes to eliminate collision risk - s3.serviceaccount.create: print the actual ID and drop the outdated "server-assigned" note (the ID is now client-generated) - tests: guard createdAK in accesskey rotate/delete subtests so sibling failures don't run invalid CLI calls - tests: requireContains/requireNotContains use t.Fatalf to fail fast - tests: Provision subtest asserts the "Attached policy" message on the second provision call for an existing user - tests: update extractServiceAccountID comment example to match the sa:<parent>:<uuid> format - tests: drop redundant saID empty-check (extractServiceAccountID fatals) * test(s3): use t.Fatalf for precondition check in serviceaccount test |
||
|
Chris Lu
|
e8a8449553 |
feat(mount): pre-allocate file IDs in pool for writeback cache mode (#9038)
* feat(mount): pre-allocate file IDs in pool for writeback cache mode When writeback caching is enabled, chunk uploads no longer block on a per-chunk AssignVolume RPC. Instead, a FileIdPool pre-allocates file IDs in batches using a single AssignVolume(Count=N, ExpectedDataSize=ChunkSize) call and hands them out instantly to upload workers. Pool size is 2x ConcurrentWriters, refilled in background when it drops below ConcurrentWriters. Entries expire after 25s to respect JWT TTL. Sequential needle keys are generated from the base file ID returned by the master, so one Assign RPC produces N usable IDs. This cuts per-chunk upload latency from 2 RTTs (assign + upload) to 1 RTT (upload only), with the assign cost amortized across the batch. * test: add benchmarks for file ID pool vs direct assign Benchmarks measure: - Pool Get vs Direct AssignVolume at various simulated latencies - Batch assign scaling (Count=1 through Count=32) - Concurrent pool access with 1-64 workers Results on Apple M4: - Pool Get: constant ~3ns regardless of assign latency - Batch=16: 15.7x more IDs/sec than individual assigns - 64 concurrent workers: 19M IDs/sec throughput * fix(mount): address review feedback on file ID pool 1. Fix race condition in Get(): use sync.Cond so callers wait for an in-flight refill instead of returning an error when the pool is empty. 2. Match default pool size to async flush worker count (128, not 16) when ConcurrentWriters is unset. 3. Add logging to UploadWithAssignFunc for consistency with UploadWithRetry. 4. Document that pooled assigns omit the Path field, bypassing path-based storage rules (filer.conf). This is an intentional tradeoff for writeback cache performance. 5. Fix flaky expiry test: widen time margin from 50ms to 1s. 6. Add TestFileIdPoolGetWaitsForRefill to verify concurrent waiters. * fix(mount): use individual Count=1 assigns to get per-fid JWTs The master generates one JWT per AssignResponse, bound to the base file ID (master_grpc_server_assign.go:158). The volume server validates that the JWT's Fid matches the upload exactly (volume_server_handlers.go:367). Using Count=N and deriving sequential IDs would fail this check. Switch to individual Count=1 RPCs over a single gRPC connection. This still amortizes connection overhead while getting a correct per-fid JWT for each entry. Partial batches are accepted if some requests fail. Remove unused needle import now that sequential ID generation is gone. * fix(mount): separate pprof from FUSE protocol debug logging The -debug flag was enabling both the pprof HTTP server and the noisy go-fuse protocol logging (rx/tx lines for every FUSE operation). This makes profiling impractical as the log output dominates. Split into two flags: - -debug: enables pprof HTTP server only (for profiling) - -debug.fuse: enables raw FUSE protocol request/response logging * perf(mount): replace LevelDB read+write with in-memory overlay for dir mtime Profile showed TouchDirMtimeCtime at 0.22s — every create/rename/unlink in a directory did a LevelDB FindEntry (read) + UpdateEntry (write) just to bump the parent dir's mtime/ctime. Replace with an in-memory map (same pattern as existing atime overlay): - touchDirMtimeCtimeLocal now stores inode→timestamp in dirMtimeMap - applyInMemoryDirMtime overlays onto GetAttr/Lookup output - No LevelDB I/O on the mutation hot path The overlay only advances timestamps forward (max of stored vs overlay), so stale entries are harmless. Map is bounded at 8192 entries. * perf(mount): skip self-originated metadata subscription events in writeback mode With writeback caching, this mount is the single writer. All local mutations are already applied to the local meta cache (via applyLocalMetadataEvent or direct InsertEntry). The filer subscription then delivers the same event back, causing redundant work: proto.Clone, enqueue to apply loop, dedup ring check, and sometimes redundant LevelDB writes when the dedup ring misses (deferred creates). Check EventNotification.Signatures against selfSignature and skip events that originated from this mount. This eliminates the redundant processing for every self-originated mutation. * perf(mount): increase kernel FUSE cache TTL in writeback cache mode With writeback caching, this mount is the single writer — the local meta cache is authoritative. Increase EntryValid and AttrValid from 1s to 10s so the kernel doesn't re-issue Lookup/GetAttr for every path component and stat call. This reduces FUSE /dev/fuse round-trips which dominate the profile at 38% of CPU (syscall.rawsyscalln). Each saved round-trip eliminates a kernel→userspace→kernel transition. Normal (non-writeback) mode retains the 1s TTL for multi-mount consistency. |
||
|
Chris Lu
|
b37bbf541a |
feat(master): drain pending size before marking volume readonly (#9036)
* feat(master): drain pending size before marking volume readonly When vacuum, volume move, or EC encoding marks a volume readonly, in-flight assigned bytes may still be pending. This adds a drain step: immediately remove from writable list (stop new assigns), then wait for pending to decay below 4MB or 30s timeout. - Add volumeSizeTracking struct consolidating effectiveSize, reportedSize, and compactRevision into a single map - Add GetPendingSize, waitForPendingDrain, DrainAndRemoveFromWritable, DrainAndSetVolumeReadOnly to VolumeLayout - UpdateVolumeSize detects compaction via compactRevision change and resets effectiveSize instead of decaying - Wire drain into vacuum (topology_vacuum.go) and volume mark readonly (master_grpc_server_volume.go) * fix: use 2MB pending size drain threshold * fix: check crowded state on initial UpdateVolumeSize registration * fix: respect context cancellation in drain, relax test timing - DrainAndSetVolumeReadOnly now accepts context.Context and returns early on cancellation (for gRPC handler timeout/cancel) - waitForPendingDrain uses select on ctx.Done instead of time.Sleep - Increase concurrent heartbeat test timeout from 10s to 15s for CI * fix: use time-based dedup so decay runs even when reported size is unchanged The value-based dedup (same reportedSize + compactRevision = skip) prevented decay from running when pending bytes existed but no writes had landed on disk yet. The reported size stayed the same across heartbeats, so the excess never decayed. Fix: dedup replicas within the same heartbeat cycle using a 2-second time window instead of comparing values. This allows decay to run once per heartbeat cycle even when the reported size is unchanged. Also confirmed finding 1 (draining re-add race) is a false positive: - Vacuum: ensureCorrectWritables only runs for ReadOnly-changed volumes - Move/EC: readonlyVolumes flag prevents re-adding during drain * fix: make VolumeMarkReadonly non-blocking to fix EC integration test timeout The DrainAndSetVolumeReadOnly call in VolumeMarkReadonly gRPC blocked up to 30s waiting for pending bytes to decay. In integration tests (and real clusters during EC encoding), this caused timeouts because multiple volumes are marked readonly sequentially and heartbeats may not arrive fast enough to decay pending within the drain window. Fix: VolumeMarkReadonly now calls SetVolumeReadOnly immediately (stops new assigns) and only logs a warning if pending bytes remain. The drain wait is kept only for vacuum (DrainAndRemoveFromWritable) which runs inside the master's own goroutine pool. Remove DrainAndSetVolumeReadOnly as it's no longer used. * fix: relax test timing, rename test, add post-condition assert * test: add vacuum integration tests with CI workflow Full-cluster integration test for vacuum, modeled on the EC integration tests. Starts a real master + 2 volume servers, uploads data, deletes entries to create garbage, runs volume.vacuum via shell command, and verifies garbage cleanup and data integrity. Test flow: 1. Start cluster (master + 2 volume servers) 2. Upload 10 files to create volume with data 3. Delete 5 files to create ~50% garbage 4. Verify garbage ratio > 10% 5. Run volume.vacuum command 6. Verify garbage cleaned up 7. Verify remaining 5 files are still accessible CI workflow runs on push/PR to master with 15-minute timeout. Log collection on failure via artifact upload. * fix: use 500KB files and delete 75% to exceed vacuum garbage threshold * fix: add shell lock before vacuum command, fix compilation error * fix: strengthen vacuum integration test assertions - waitForServer: use net.DialTimeout instead of grpc.NewClient for real TCP readiness check - verify_garbage_before_vacuum: t.Fatal instead of warning when no garbage detected - verify_cleanup_after_vacuum: t.Fatal if no server reported the volume or cleanup wasn't verified - verify_remaining_data: read actual file contents via HTTP and compare byte-for-byte against original uploaded payloads * fix: use http.Client with timeout and close body before retry |
||
|
Chris Lu
|
10b0bdce02 |
feat: pass expected_data_size from clients for size-aware assignment (#9032)
* feat: pass expected_data_size from clients for size-aware assignment Add expected_data_size field to AssignRequest (master proto) and AssignVolumeRequest (filer proto) so clients can hint how large the data will be. The master uses this instead of the 1MB default when tracking pending volume sizes for weighted assignment. - Add expected_data_size to master.proto AssignRequest - Add expected_data_size to filer.proto AssignVolumeRequest - Wire through filer AssignVolume handler - Wire through HTTP submit handler (uses actual upload size) - Add ExpectedDataSize to VolumeAssignRequest in operation package - Topology.PickForWrite accepts optional expectedDataSize parameter * fix: guard integer conversions in expected_data_size path - common.go: clamp OriginalDataSize to non-negative before uint64 cast - topology.go: cap expectedDataSize at math.MaxInt64 before int64 cast * fix: parse dataSize hint in HTTP /dir/assign and test non-zero expectedDataSize - HTTP /dir/assign now parses optional "dataSize" query parameter and passes it to PickForWrite instead of hardcoded 0 - Add test assertion for PickForWrite with non-zero expectedDataSize |
||
|
Chris LuandCopilot Autofix powered by AI
|
e2c79af6ec |
feat(master): size-aware volume assignment with weighted selection (#9031)
* feat(master): size-aware volume assignment with weighted selection PickForWrite now selects volumes proportional to remaining capacity instead of uniform random, so emptier volumes receive more writes. - Add vid2size map to VolumeLayout tracking effective volume sizes - Weighted pick via random sampling (k=3) for O(1) cost - RecordAssign tracks estimated pending bytes between heartbeats - Exponential decay on heartbeat: halve excess each cycle - Proactive crowded detection using effective size - Zero extra heap allocations on the unconstrained hot path Benchmark (20 writable volumes, unconstrained): Before: 36 ns/op, 32 B/op, 2 allocs/op After: 85 ns/op, 32 B/op, 2 allocs/op * fix: address review feedback on size-aware assignment - RecordAssign: use write lock (Lock) instead of read lock (RLock) since it mutates vid2size map and crowded set - RegisterVolume: clear crowded flag when heartbeat decay drops effective size below the threshold - pickWeightedByRemaining: fix misleading Fisher-Yates comment, simplify to plain random sampling (duplicates are harmless) - ShouldGrowVolumesByDcAndRack: read vid2size under RLock * fix: decay once per heartbeat cycle, not per replica RegisterVolume is called once per replica of a volume. For replicated volumes, the pending size decay was running multiple times per heartbeat cycle, reducing the excess by 75% instead of 50% (for 2 replicas). Fix: track vid2reportedSize and only run decay when the heartbeat- reported size actually changes. A second replica reporting the same size in the same cycle is a no-op. Also fix CodeQL alert: cap count*EstimatedNeedleSizeBytes to avoid uint64→int64 overflow in RecordAssign call. * Potential fix for pull request finding 'CodeQL / Incorrect conversion between integer types' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * fix: fail fast in test setup on JSON errors - setupWithLimit now takes testing.TB and calls t.Fatalf on unmarshal errors or type assertion failures instead of printing and continuing - benchSetup removed; benchmarks reuse setupWithLimit directly * fix: run size decay on every heartbeat, not just new volumes RegisterVolume is only called for newly discovered volumes, not on every heartbeat. The pending size decay was never running in production. - Extract decay logic into UpdateVolumeSize(), called from SyncDataNodeRegistration for every reported volume on every heartbeat - RegisterVolume only initializes vid2size for brand-new volumes - Constrained PickForWrite: scan from random offset, collect up to pickSampleSize matches in a stack array (no append allocation) - Tests now exercise UpdateVolumeSize directly instead of RegisterVolume to match the production heartbeat path * fix: compute pending bytes in uint64 to satisfy CodeQL --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
388cc018ab |
fix(mount): reduce unnecessary filer RPCs across all mutation operations (#9030)
* fix(mount): reduce filer RPCs for mkdir/rmdir operations 1. Mark newly created directories as cached immediately. A just-created directory is guaranteed to be empty, so the first Lookup or ReadDir inside it no longer triggers a needless EnsureVisited filer round-trip. 2. Use touchDirMtimeCtimeLocal instead of touchDirMtimeCtime for both Mkdir and Rmdir. The filer already processed the mutation, so updating the parent's mtime/ctime locally avoids an extra UpdateEntry RPC. Net effect: mkdir goes from 3 filer RPCs to 1. * fix(mount): eliminate extra filer RPCs for parent dir mtime updates Every mutation (create, unlink, symlink, link, rename) was calling touchDirMtimeCtime after the filer already processed the mutation. That function does maybeLoadEntry + saveEntry (UpdateEntry RPC) just to bump the parent directory's mtime/ctime — an unnecessary round-trip. Switch all call sites to touchDirMtimeCtimeLocal which updates the local meta cache directly. Remove the now-unused touchDirMtimeCtime. Affected operations: Create (Mknod path), Unlink, Symlink, Link, Rename. Each saves one filer RPC per call. * fix(mount): defer RemoveXAttr for open files, skip redundant existence check 1. RemoveXAttr now defers the filer RPC when the file has an open handle, consistent with SetXAttr which already does this. The xattr change is flushed with the file metadata on close. 2. Create() already checks whether the file exists before calling createRegularFile(). Skip the duplicate maybeLoadEntry() inside createRegularFile when called from Create, avoiding a redundant filer GetEntry RPC when the parent directory is not cached. * fix(mount): skip distributed lock when writeback caching is enabled Writeback caching implies single-writer semantics — the user accepts that only one mount writes to each file. The DLM lock (NewBlockingLongLivedLock) is a blocking gRPC call to the filer's lock manager on every file open-for-write, Create, and Rename. This is unnecessary overhead when writeback caching is on. Skip lockClient initialization when WritebackCache is true. All DLM call sites already guard on `wfs.lockClient != nil`, so they are automatically skipped. * fix(mount): async filer create for Mknod with writeback caching With writeback caching, Mknod now inserts the entry into the local meta cache immediately and fires the filer CreateEntry RPC in a background goroutine, similar to how Create defers its filer RPC. The node is visible locally right away (stat, readdir, open all work from the local cache), while the filer persistence happens asynchronously. This removes the synchronous filer RPC from the Mknod hot path. * fix(mount): address review feedback on async create and DLM logging 1. Log when DLM is skipped due to writeback caching so operators understand why distributed locking is not active at startup. 2. Add retry with backoff for async Mknod create RPC (reuses existing retryMetadataFlush helper). On final failure, remove the orphaned local cache entry and invalidate the parent directory cache so the phantom file does not persist. * fix(mount): restore filer RPC for parent dir mtime when not using writeback cache The local-only touchDirMtimeCtimeLocal updates LevelDB but lookupEntry only reads from LevelDB when the parent directory is cached. For uncached parents, GetAttr goes to the filer which has stale timestamps, causing pjdfstest failures (mkdir/00.t, rmdir/00.t, unlink/00.t, etc.). Introduce touchDirMtimeCtimeBest which: - WritebackCache mode: local meta cache only (no filer RPC) - Normal mode: filer UpdateEntry RPC for POSIX correctness The deferred file create path keeps touchDirMtimeCtimeLocal since no filer entry exists yet. * fix(mount): use touchDirMtimeCtimeBest for deferred file create path The deferred create path (Create with deferFilerCreate=true) was using touchDirMtimeCtimeLocal unconditionally, but this only updates the local LevelDB cache. Without writeback caching, the parent directory's mtime/ctime must be updated on the filer for POSIX correctness (pjdfstest open/00.t). * test: add link/00.t and unlink/00.t to pjdfstest known failures These tests fail nlink assertions (e.g. expected nlink=2, got nlink=3) after hard link creation/removal. The failures are deterministic and surfaced by caching changes that affect the order in which entries are loaded into the local meta cache. The root cause is a filer-side hard link counter issue, not mount mtime/ctime handling. |
||
|
Chris Lu
|
c390448906 |
fix(s3): preserve exact policy document in embedded IAM put/get-user-policy (#9025)
* fix(s3): preserve exact policy document in embedded IAM PutUserPolicy/GetUserPolicy (#9008) The embedded IAM implementation (used when IAM requests go through the S3 gateway) discarded the original policy document on PutUserPolicy, storing only the lossy ident.Actions representation. GetUserPolicy then reconstructed the document from these coarse-grained actions, producing wildcard-expanded actions (s3:GetObject → s3:Get*), duplicates, and collapsed resources (array → single string). PR #9009 fixed this in the standalone IAM server (weed/iamapi/) but the embedded IAM (weed/s3api/) — which is the code path most users hit — had the same bugs. Changes: - Add InlinePolicyStore optional interface to credential store, with implementations for FilerEtcStore (uses existing PoliciesCollection), MemoryStore, and PropagatingCredentialStore. - Embedded IAM PutUserPolicy now persists the original policy document via CredentialManager.PutUserInlinePolicy for lossless round-trips. - Embedded IAM GetUserPolicy first tries the stored inline policy; only falls back to lossy reconstruction from ident.Actions when no stored document exists (e.g. policies created before this fix). - Fix the fallback reconstruction: add action deduplication and preserve resource paths verbatim (no more spurious /* appending). - Update DeleteUserPolicy/ListUserPolicies to use stored inline policies. * fix(s3): address PR review feedback for embedded IAM inline policies - Validate PolicyName is non-empty in PutUserPolicy and DeleteUserPolicy - Add recomputeActions() to aggregate ident.Actions from ALL stored inline policies on put/delete, fixing the issue where a second PutUserPolicy would overwrite the first policy's enforcement - Log errors from GetUserInlinePolicy in the GetUserPolicy fallback instead of silently ignoring them - Add initialization guards to MemoryStore GetUserInlinePolicy and ListUserInlinePolicies for consistency with other read methods * fix(s3): make inline policy persistence fatal and propagate recompute errors Address second round of review feedback: - recomputeActions() now returns ([]string, error) so callers can distinguish store failures from "no stored policies" and abort the mutation on transient errors instead of silently falling back. - PutUserInlinePolicy and DeleteUserInlinePolicy failures are now fatal: the API call returns ServiceFailure instead of logging and continuing, keeping ident.Actions and stored policy state in sync. * chore: gofmt weed/s3api/iceberg/handlers_oauth.go Pre-existing formatting issue from #9017; fixes S3 Tables Format Check CI. |
||
|
Chris Lu
|
066f7c3a0d |
fix(mount): track directory subdirectory count for correct nlink (#9028)
Track subdirectory count per-inode in memory via InodeEntry.subdirCount. Increment on mkdir, decrement on rmdir, adjust on cross-directory rename. applyDirNlink uses this count instead of listing metacache entries, so nlink is correct immediately after mkdir without needing a prior readdir. Remove tests/rename/24.t from known_failures.txt (all 13 subtests now pass). |
||
|
Chris Lu
|
ae724ac9d5 |
test: remove unlink/14.t from pjdfstest known failures (#9029)
fix(mount): skip metadata flush for unlinked-while-open files When a file is unlinked while still open (open-unlink-close pattern), the synchronous doFlush path recreated the entry on the filer during close. Check fh.isDeleted before flushing metadata, matching the existing check in the async flush path. Remove tests/unlink/14.t from known_failures.txt (all 7 subtests now pass). Full suite: 235 files, 8803 tests, Result: PASS. |
||
|
Chris Lu
|
2e64c0fe2a |
fix(mount): skip metadata flush for unlinked-while-open files (#9027)
When a file is unlinked while still open (open-unlink-close pattern), the synchronous doFlush path would recreate the entry on the filer during close. Check fh.isDeleted before flushing metadata, matching the async flush path which already had this check. |
||
|
Chris Lu
|
ef30d91b7d |
test: switch to sanwan/pjdfstest fork for NAME_MAX-aware tests (#9024)
The upstream pjd/pjdfstest uses hardcoded ~768-byte filenames which exceed the Linux FUSE kernel NAME_MAX=255 limit. The sanwan fork (used by JuiceFS) uses pathconf(_PC_NAME_MAX) to dynamically determine the filesystem's actual NAME_MAX and generates test names accordingly. This removes all 26 NAME_MAX-related entries from known_failures.txt, reducing the skip list from 31 to 5 entries. |
||
|
Chris Lu
|
8aa5809824 |
fix(mount): gate directory nlink counting behind -posix.dirNLink option (#9026)
The directory nlink counting (2 + subdirectory count) requires listing cached directory entries on every stat, which has a performance cost. Gate it behind the -posix.dirNLink flag (default: off). When disabled, directories report nlink=2 (POSIX baseline). When enabled, directories report nlink=2 + number of subdirectories from cached entries. |
||
|
Chris Lu
|
39e76b8e94 |
fix(mount): report correct nlink for directories (#9023)
fix(mount): report correct nlink for directories (2 + subdirectory count) POSIX requires directory nlink = 2 (for . and ..) + number of subdirectories. Previously SeaweedFS reported nlink=1 for all dirs. - Set nlink baseline to 2 for directories in setAttrByPbEntry, setAttrByFilerEntry, and setRootAttr - Add applyDirNlink() that counts subdirectories from the local metacache and sets nlink = 2 + count - Call it from GetAttr and Lookup for directory entries When the metacache has no entries (before readdir), nlink=2 is used as a safe POSIX-compliant default. |
||
|
Chris Lu
|
2a7ec8d033 |
fix(filer): do not abort entry deletion when hard link cleanup fails (#9022)
When unlinking a hard-linked file, DeleteOneEntry and DeleteEntry both called DeleteHardLink before removing the directory entry from the store. If DeleteHardLink returned an error (e.g. KV storage issue, decode failure), the function returned early without deleting the directory entry itself. This left a stale entry in the filer store, causing subsequent rmdir to fail with ENOTEMPTY. Change both functions to log the hard link cleanup error and continue to delete the directory entry regardless. This ensures the parent directory can always be removed after all its children are unlinked. Remove tests/unlink/14.t from the pjdfstest known failures list since this fix addresses the root cause. |
||
|
Chris Lu
|
07cd741380 |
fix(filer): update hard link nlink/ctime when rename replaces a hard-linked target (#9020)
fix(filer): fix hard link nlink/ctime when rename replaces a hard-linked target The CreateEntry → UpdateEntry → handleUpdateToHardLinks path already calls DeleteHardLink() when the existing target has a different HardLinkId. Combined with the ctime update added to DeleteHardLink() in a prior commit, remaining hard links now see correct nlink and updated ctime after a rename replaces the target. Remove tests/rename/23.t and tests/rename/24.t from known_failures.txt. |
||
|
Chris Lu
|
2264941a17 |
fix(mount): update parent directory mtime/ctime on deferred file create (#9021)
* fix(mount): update parent directory mtime/ctime on deferred file create * style: run go fmt on mount package |
||
|
Chris Lu
|
de5b6f2120 |
fix(filer,mount): add nanosecond timestamp precision (#9019)
* fix(filer,mount): add nanosecond timestamp precision Add mtime_ns and ctime_ns fields to the FuseAttributes protobuf message to store the nanosecond component of timestamps (0-999999999). Previously timestamps were truncated to whole seconds. - Update EntryAttributeToPb/PbToEntryAttribute to encode/decode ns - Update setAttrByPbEntry/setAttrByFilerEntry to set Mtimensec/Ctimensec - Update in-memory atime map to store time.Time (preserves nanoseconds) - Remove tests/utimensat/08.t from known_failures.txt (all 9 subtests pass) * fix: sync nanosecond fields on all mtime/ctime write paths Ensure MtimeNs/CtimeNs are updated alongside Mtime/Ctime in all code paths: truncate, flush, link, copy_range, metadata flush, and directory touch. * fix: set ctime/ctime_ns in copy_range and metadata flush paths |
||
|
Chris Lu
|
3f36846642 |
fix(filer): update hard link ctime when nlink changes on unlink (#9018)
* fix(filer): update hard link ctime when nlink changes on unlink When a hard link is unlinked, POSIX requires that the remaining links' ctime is updated because the inode's nlink count changed. The filer's DeleteHardLink() decremented the counter in the KV store but did not update the ctime field. Set ctime to time.Now() on the KV entry before writing it back when the hard link counter is decremented but still > 0. Remove tests/unlink/00.t from known_failures.txt (all 112 subtests now pass). * style: use time.Now().UTC() for ctime in DeleteHardLink |
||
|
Chris Lu
|
2b8c16160f |
feat(iceberg): add OAuth2 token endpoint for DuckDB compatibility (#9017)
* feat(iceberg): add OAuth2 token endpoint for DuckDB compatibility (#9015) DuckDB's Iceberg connector uses OAuth2 client_credentials flow, hitting POST /v1/oauth/tokens which was not implemented, returning 404. Add the OAuth2 token endpoint that accepts S3 access key / secret key as client_id / client_secret, validates them against IAM, and returns a signed JWT bearer token. The Auth middleware now accepts Bearer tokens in addition to S3 signature auth. * fix(test): use weed shell for table bucket creation with IAM enabled The S3 Tables REST API requires SigV4 auth when IAM is configured. Use weed shell (which bypasses S3 auth) to create table buckets, matching the pattern used by the Trino integration tests. * address review feedback: access key in JWT, full identity in Bearer auth - Include AccessKey in JWT claims so token verification uses the exact credential that signed the token (no ambiguity with multi-key identities) - Return full Identity object from Bearer auth so downstream IAM/policy code sees an authenticated request, not anonymous - Replace GetSecretKeyForIdentity with GetCredentialByAccessKey for unambiguous credential lookup - DuckDB test now tries the full SQL script first (CREATE SECRET + catalog access), falling back to simple CREATE SECRET if needed - Tighten bearer auth test assertion to only accept 200/500 Addresses review comments from coderabbitai and gemini-code-assist. * security: use PostFormValue, bind signing key to access key, fix port conflict - Use r.PostFormValue instead of r.FormValue to prevent credentials from leaking via query string into logs and caches - Reject client_secret in URL query parameters explicitly - Include access key in HMAC signing key derivation to prevent cross-credential token forgery when secrets happen to match - Allocate dedicated webdav port in OAuth test env to avoid port collision with the shared TestMain cluster |
||
|
Chris Lu
|
bf31f404bc |
test: add pjdfstest POSIX compliance suite (#9013)
* test: add pjdfstest POSIX compliance suite Adds a script and CI workflow that runs the upstream pjdfstest POSIX compliance test suite against a SeaweedFS FUSE mount. The script starts a self-contained `weed mini` server, mounts the filesystem with `weed mount`, builds pjdfstest from source, and runs it under prove(1). * fix: address review feedback on pjdfstest setup - Use github.ref instead of github.head_ref in concurrency group so push events get a stable group key - Add explicit timeout check after filer readiness polling loop - Refresh pjdfstest checkout when PJDFSTEST_REPO or PJDFSTEST_REF are overridden instead of silently reusing stale sources * test: add Docker-based pjdfstest for faster iteration Adds a docker-compose setup that reuses the existing e2e image pattern: - master, volume, filer services from chrislusf/seaweedfs:e2e - mount service extended with pjdfstest baked in (Dockerfile extends e2e) - Tests run via `docker compose exec mount /run.sh` - CI workflow gains a parallel `pjdfstest (docker)` job This avoids building Go from scratch on each iteration — just rebuild the e2e image once and iterate on the compose stack. * fix: address second round of review feedback - Use mktemp for WORK_DIR so each run starts with a clean filer state - Pin PJDFSTEST_REF to immutable commit (03eb257) instead of master - Use cp -r instead of cp -a to avoid preserving ownership during setup * fix: address CI failure and third round of review feedback - Fix docker job: fall back to plain docker build when buildx cache export is not supported (default docker driver in some CI runners) - Use /healthz endpoint for filer healthcheck in docker-compose - Copy logs to a fixed path (/tmp/seaweedfs-pjdfstest-logs/) for reliable CI artifact upload when WORK_DIR is a mktemp path * fix(mount): improve POSIX compliance for FUSE mount Address several POSIX compliance gaps surfaced by the pjdfstest suite: 1. Filename length limit: reduce from 4096 to 255 bytes (NAME_MAX), returning ENAMETOOLONG for longer names. 2. SUID/SGID clearing on write: clear setuid/setgid bits when a non-root user writes to a file (POSIX requirement). 3. SUID/SGID clearing on chown: clear setuid/setgid bits when file ownership changes by a non-root user. 4. Sticky bit enforcement: add checkStickyBit helper and enforce it in Unlink, Rmdir, and Rename — only file owner, directory owner, or root may delete entries in sticky directories. 5. ctime (inode change time) tracking: add ctime field to the FuseAttributes protobuf message and filer.Attr struct. Update ctime on all metadata-modifying operations (SetAttr, Write/flush, Link, Create, Mkdir, Mknod, Symlink, Truncate). Fall back to mtime for backward compatibility when ctime is 0. * fix: add -T flag to docker compose exec for CI Disable TTY allocation in the pjdfstest docker job since GitHub Actions runners have no interactive TTY. * fix(mount): update parent directory mtime/ctime on entry changes POSIX requires that a directory's st_mtime and st_ctime be updated whenever entries are created or removed within it. Add touchDirMtimeCtime() helper and call it after: - mkdir, rmdir - create (including deferred creates), mknod, unlink - symlink, link - rename (both source and destination directories) This fixes pjdfstest failures in mkdir/00, mkfifo/00, mknod/00, mknod/11, open/00, symlink/00, link/00, and rmdir/00. * fix(mount): enforce sticky bit on destination directory during rename POSIX requires sticky-bit enforcement on both source and destination directories during rename. When the destination directory has the sticky bit set and a target entry already exists, only the file owner, directory owner, or root may replace it. * fix(mount): add in-memory atime tracking for POSIX compliance Track atime separately from mtime using a bounded in-memory map (capped at 8192 entries with random eviction). atime is not persisted to the filer — it's only kept in mount memory to satisfy POSIX stat requirements for utimensat and related syscalls. This fixes utimensat/00, utimensat/02, utimensat/04, utimensat/05, and utimensat/09 pjdfstest failures where atime was incorrectly aliased to mtime. * fix(mount): restore long filename support, fix permission checks - Restore 4096-byte filename limit (was incorrectly reduced to 255). SeaweedFS stores names as protobuf strings with no ext4-style constraint — the 255 limit is not applicable. - Fix AcquireHandle permission check to map filer uid/gid to local space before calling hasAccess, matching the pattern used in Access(). - Fix hasAccess fallback when supplementary group lookup fails: fall through to "other" permissions instead of requiring both group AND other to match, which was overly restrictive for non-existent UIDs. * fix(mount): fix permission checks and enforce NAME_MAX=255 - Fix AcquireHandle to map uid/gid from filer-space to local-space before calling hasAccess, consistent with the Access handler. - Fix hasAccess fallback when supplementary group lookup fails: use "other" permissions only instead of requiring both group AND other. - Enforce NAME_MAX=255 with a comment explaining the Linux FUSE kernel module's VFS-layer limit. Files >255 bytes can be created via direct FUSE protocol calls but can't be stat'd/chmod'd via normal syscalls. - Don't call touchDirMtimeCtime for deferred creates to avoid invalidating the just-cached entry via filer metadata events. * ci: mark pjdfstest steps as continue-on-error The pjdfstest suite has known failures (Linux FUSE NAME_MAX=255 limitation, hard link nlink/ctime tracking, nanosecond precision) that cannot be fixed in the mount layer. Mark the test steps as continue-on-error so the CI job reports results without blocking. * ci: increase pjdfstest bare metal timeout to 90 minutes * fix: use full commit hash for PJDFSTEST_REF in run.sh Short hashes cannot be resolved by git fetch --depth 1 on shallow clones. Use the full 40-char SHA. * test: add pjdfstest known failures skip list Add known_failures.txt listing 33 test files that cannot pass due to: - Linux FUSE kernel NAME_MAX=255 (26 files) - Hard link nlink/ctime tracking requiring filer changes (3 files) - Parent dir mtime on deferred create (1 file) - Directory rename permission edge case (1 file) - rmdir after hard link unlink (1 file) - Nanosecond timestamp precision (1 file) Both run.sh and run_inside_container.sh now skip these tests when running the full suite. Any failure in a non-skipped test will cause CI to fail, catching regressions immediately. Remove continue-on-error from CI steps since the skip list handles known failures. Result: 204 test files, 8380 tests, all passing. * ci: remove bare metal pjdfstest job, keep Docker only The bare metal job consistently gets stuck past its timeout due to weed processes not exiting cleanly. The Docker job covers the same tests reliably and runs faster. |
||
|
Chris Lu
|
eb5624233d |
[filer] fix log buffer idle polling (#9012)
* fix log buffer idle polling * log_buffer: document notificationHealthCheckInterval tradeoffs Explain that notifyChan is the primary wakeup path and this interval only bounds the fallback / state-recheck cadence, so future maintainers don't tune it without understanding the implications for client-disconnect detection latency. * log_buffer: rename waitForNotification to awaitNotificationOrTimeout The helper returns after either a notification or the health-check timeout; the old name read like it blocked indefinitely. No behavior change. * log_buffer: wake blocked subscribers on shutdown awaitNotificationOrTimeout previously only returned on notifyChan or the health-check timeout, so ShutdownLogBuffer on an idle buffer (where copyToFlush returns nil and loopFlush never fires the post-flush notification) would leave subscribers parked for up to 250ms before they noticed IsStopping. Add an internal shutdownCh closed by ShutdownLogBuffer and select on it from awaitNotificationOrTimeout, which is now a method on *LogBuffer. Subscribers wake immediately, re-check IsStopping, and exit. No change to LoopProcessLogData signatures or any caller (filer metadata subscribers, MQ broker, local partition subscribe). * log_buffer: regression tests for flush-notify wake-up TestLoopFlush_NotifiesSubscribersAfterFlush directly verifies that loopFlush calls notifySubscribers after processing a flush, so a reader parked on notifyChan wakes promptly when a flush lands. Verified to fail if that notification is removed. TestLoopProcessLogDataWithOffset_WakesOnDataArrival is the end-to-end counterpart: a real LoopProcessLogDataWithOffset reader parks on notifyChan via the ResumeFromDiskError branch, then wakes and processes the entry well under the 250ms fallback once data arrives. * log_buffer: keep notification-timeout logs at V(4) Revert the V(4)->V(5) demotion. Now that the shutdown wake-up path exists and (with the follow-up fix) idle-polling CPU churn is bounded by the 250ms health check, these timeout logs no longer flood at V=4 the way they did on the 10ms fallback, so the previous verbosity is appropriate again. * log_buffer: exit reader loops cleanly on shutdown awaitNotificationOrTimeout returns true on both data notifications and shutdown (shutdownCh closed). Without an explicit IsStopping() guard, the ResumeFromDiskError, offset-based no-data, empty-buffer, and timestamp-wait paths would either tight-spin against a closed shutdownCh or, in the offset-based case, return ResumeFromDiskError to the caller instead of exiting. Add an IsStopping() check after each awaitNotificationOrTimeout call that previously continued or returned ResumeFromDiskError, so subscribers exit promptly with isDone=true and err=nil when ShutdownLogBuffer is called. * log_buffer: regression test for shutdown wake-up Park a real LoopProcessLogDataWithOffset reader on notifyChan via the ResumeFromDiskError branch, call ShutdownLogBuffer, and assert the reader exits with isDone=true and err=nil well under the 250ms fallback. Verified to fail (timeout) if the IsStopping() guards added in the prior commit are removed. * log_buffer: bump reader-park sleep to 50ms with rationale Both wake-path tests use a sleep to give the goroutine time to reach awaitNotificationOrTimeout before the test triggers the wake-up. Bump from 20ms to 50ms and document the timing assumption to reduce flakiness on slow CI. Both paths are race-free either way (a buffered notification or a closed shutdownCh stays valid until consumed), so this is purely about exercising the park-then-wake path rather than the already-pending fast path. |
||
|
Chris Lu
|
546f255b46 |
fix(filer/postgres): use pgx v5 API for PgBouncer simple protocol (#9010)
* fix(filer/postgres): use pgx v5 API for PgBouncer simple protocol
In pgx/v5 the `prefer_simple_protocol` DSN parameter was removed, so
appending it to the connection string caused PgBouncer/PostgreSQL to
reject it as an unknown startup parameter:
FATAL: unsupported startup parameter: prefer_simple_protocol (SQLSTATE 08P01)
Parse the DSN with pgx.ParseConfig and, when pgbouncer_compatible is
set, configure DefaultQueryExecMode = QueryExecModeSimpleProtocol and
disable the statement/description caches. Register the config via
stdlib.RegisterConnConfig before sql.Open.
Fixes #9005
* refactor(filer/postgres): extract shared OpenPGXDB helper with cleanup
Extract the pgx v5 ParseConfig/RegisterConnConfig/sql.Open/Ping logic
into a shared postgres.OpenPGXDB helper used by both postgres and
postgres2 filer stores, eliminating ~60 lines of duplication.
The helper also unregisters the conn config via stdlib.UnregisterConnConfig
on every failure path (sql.Open error, Ping error) so we do not leak
entries in stdlib's global connection config map when initialization
fails.
* refactor(filer/postgres): use stdlib.OpenDB to avoid conn config leak
Switch OpenPGXDB from RegisterConnConfig + sql.Open("pgx", connStr) to
stdlib.OpenDB(*connConfig). The former leaks an entry in stdlib's global
conn config map on every successful initialization; stdlib.OpenDB takes
the config directly and keeps no global registration.
Addresses CodeRabbit review feedback on #9010.
|
||
|
Chris Lu
|
e4bcfb96d8 |
fix(iam): preserve actions/resources in GetUserPolicy fallback (#9009)
* fix(iam): preserve actions/resources in GetUserPolicy fallback (#9008) When GetUserPolicy cannot find a stored inline policy document and falls back to reconstructing one from the aggregated ident.Actions, it produced mangled output: bare-bucket paths like "b-le*/*" got another "/*" appended (becoming "b-le*/*/*"), and distinct s3 actions that map to the same coarse verb (e.g. s3:GetObject and s3:GetBucketLocation -> s3:Get*) were emitted multiple times in the same statement. - Use SplitN so paths containing ':' are not shredded. - Only append "/*" to bare bucket patterns; paths already containing '/' are used as-is. - Dedupe reconstructed actions per resource. Adds a regression test using the exact reproducer from the issue. * fix(iam): preserve bucket-level ARNs in fallback reconstruction Addresses CodeRabbit review feedback on #9009: - Use stored path verbatim in the GetUserPolicy fallback so bucket-level resources (e.g. arn:aws:s3:::b-le*) are not rewritten to object-level ARNs (arn:aws:s3:::b-le*/*). Previously bare bucket patterns had "/*" appended, conflating bucket and object resources. - Extend TestPutGetUserPolicyIssue9008 to also exercise the fallback reconstruction path by clearing the persisted inline policy between the two GetUserPolicy calls, validating that bucket and object resources stay distinct. * chore: revert accidental scheduled_tasks.lock change |
||
|
Chris Lu
|
dd203769b1 |
chore(helm): document worker job categories and use 'all' as default (#9002)
chore(helm): document worker job categories and use "all" as default Update the worker jobType comment to document the category system (all, default, heavy) with all available job types, and change the default value to "all" to match the CLI default. |
||
|
Chris Lu
|
6f036c7015 |
fix(master): skip redundant DoJoinCommand on resumeState to prevent deadlock (#8998)
* fix(master): skip redundant DoJoinCommand on resumeState to prevent deadlock When fastResume is active (single-master + resumeState + non-empty log), the raft server becomes leader within ~1ms. DoJoinCommand then enters the leaderLoop's processCommand path, which calls setCommitIndex to commit all pending entries. The goraft setCommitIndex implementation returns early when it encounters a JoinCommand entry (to recalculate quorum), which can prevent the new entry's event channel from being notified — leaving DoJoinCommand blocked forever. Each restart appends a new raft:join entry to the log, while the conf file's commitIndex (only persisted on AddPeer) lags behind. After 3-4 restarts the uncommitted range contains old JoinCommand entries that trigger the early return before the new entry is reached. Fix: skip DoJoinCommand when the raft log already has entries (the server was already joined in a previous run). The fastResume mechanism handles leader election independently. * fix(master): handle Hashicorp Raft in HasExistingState Add Hashicorp Raft support to HasExistingState by checking AppliedIndex, consistent with how other RaftServer methods handle both raft implementations. * fix(master): use LastIndex() instead of AppliedIndex() for Hashicorp Raft AppliedIndex() reflects in-memory FSM state which starts at 0 before log replay completes. LastIndex() reads from persisted stable storage, correctly mirroring the non-Hashicorp IsLogEmpty() check. |
||
|
Chris Lu
|
8b16507059 |
fix(master): stop endless volume growth in DCs with more racks than replica count (#8996)
fix(master): stop endless volume growth in DCs with more racks than replica count (#8986) ShouldGrowVolumesByDcAndRack checked every DC+rack for a writable volume replica. With "010" replication (different-rack), volumes only span 2 racks. In a DC with 3+ racks, at least one rack always lacked a replica, causing the periodic growth loop to create new volumes endlessly. When DiffRackCount > 0, check at the DC level instead: if any rack in the DC has a non-crowded writable volume, skip growth for uncovered racks. |
||
|
Chris Lu
|
ba90ae5c94 |
fix(s3): don't count ErrNotFound as filer health failure in failover (#8995)
* fix(s3): don't count ErrNotFound as filer health failure in failover The S3 gateway's filer client failover was recording ErrNotFound (entry doesn't exist) as a filer health failure. In multi-filer setups where filers have separate metadata stores, normal object lookups that return "not found" accumulated in the circuit breaker, eventually marking healthy filers as unhealthy after just 3 lookups. This caused the distributed lock integration test to fail with 500 InternalError: once a filer was circuit-broken, subsequent lookups could no longer fall back, turning a would-be 412 PreconditionFailed into an unrecoverable internal error. Only record actual transport/server failures in the health tracker. The failover still tries other filers for data locality, but no longer penalizes filers for correctly reporting missing entries. * style: inline isNotFound variable for consistency The variable was only used once; inlining it matches the pattern already used in the failover loop a few lines below. |
||
|
Chris Lu
|
e21d7602c3 |
feat(iam): implement group inline policy actions (#8992)
* feat(iam): implement group inline policy actions Add PutGroupPolicy, GetGroupPolicy, DeleteGroupPolicy, and ListGroupPolicies to both embedded and standalone IAM servers. The standalone IAM stores group inline policies in a new GroupInlinePolicies field in the Policies JSON, mirroring the existing user inline policy pattern. DeleteGroup now also checks for inline policies before allowing deletion. * fix: address review feedback for group inline policies - Embedded IAM: return NotImplemented for group inline policies instead of silently succeeding as no-ops (Gemini + CodeRabbit) - Standalone IAM: recompute member actions after PutGroupPolicy and DeleteGroupPolicy (Gemini) - Add parameter validation for GroupName/PolicyName/PolicyDocument on PutGroupPolicy, DeleteGroupPolicy, ListGroupPolicies (Gemini) - Add UserName validation for ListUserPolicies in standalone IAM - Call cleanupGroupInlinePolicies from DeleteGroup (Gemini) - Migrate GroupInlinePolicies on group rename in UpdateGroup (CodeRabbit) - Fix integration test cleanup order (CodeRabbit) * fix: persist recomputed actions and improve error handling - Set changed=true for PutGroupPolicy/DeleteGroupPolicy in standalone IAM DoActions so recomputed member actions are persisted (Gemini critical) - Make cleanupGroupInlinePolicies accept policies parameter to avoid redundant I/O, return error (Gemini) - Make migrateGroupInlinePolicies return error, handle in caller (Gemini) * fix: include group policies in action recomputation Extend computeAllActionsForUser to also aggregate group inline policies and group managed policies when s3cfg is provided. Previously, group inline policies were stored but never reflected in member Identity.Actions. (CodeRabbit critical) * perf: use identity index in recomputeActionsForGroupMembers for O(N+M) * fix: skip group inline policy integration test on embedded IAM The embedded IAM returns NotImplemented for group inline policies. Skip TestIAMGroupInlinePolicy when running against embedded mode to avoid CI failures in the group integration test matrix. |
||
|
Chris Lu
|
3af571a5f3 |
feat(mount): add -dlm flag for distributed lock cross-mount write coordination (#8989)
* feat(cluster): add NewBlockingLongLivedLock to LockClient Add a hybrid lock acquisition method that blocks until the lock is acquired (like NewShortLivedLock) and then starts a background renewal goroutine (like StartLongLivedLock). This is needed for weed mount DLM integration where Open() must block until the lock is held, but the lock must be renewed for the entire write session until close. * feat(mount): add -dlm flag and DLM plumbing for cross-mount write coordination Add EnableDistributedLock option, LockClient field to WFS, and dlmLock field to FileHandle. The -dlm flag is opt-in and off by default. When enabled, a LockClient is created at mount startup using the filer's gRPC connection. * feat(mount): acquire DLM lock on write-open, release on close When -dlm is enabled, opening a file for writing acquires a distributed lock (blocking until held) with automatic renewal. The lock is released when the file handle is closed, after any pending flush completes. This ensures only one mount can have a file open for writing at a time, preventing cross-mount data loss from concurrent writers. * docs(mount): document DLM lock coverage in flush paths Add comments to flushMetadataToFiler and flushFileMetadata explaining that when -dlm is enabled, the distributed lock is already held by the FileHandle for the entire write session, so no additional DLM acquisition is needed in these functions. * test(fuse_dlm): add integration tests for DLM cross-mount write coordination Add test/fuse_dlm/ with a full cluster framework (1 master, 1 volume, 2 filers, 2 FUSE mounts with -dlm) and four test cases: - TestDLMConcurrentWritersSameFile: two mounts write simultaneously, verify no data corruption - TestDLMRepeatedOpenWriteClose: repeated write cycles from both mounts, verify consistency - TestDLMStressConcurrentWrites: 16 goroutines across 2 mounts writing to 5 shared files - TestDLMWriteBlocksSecondWriter: verify one mount's write-open blocks while another mount holds the file open * ci: add GitHub workflow for FUSE DLM integration tests Add .github/workflows/fuse-dlm-integration.yml that runs the DLM cross-mount write coordination tests on ubuntu-22.04. Triggered on changes to weed/mount/**, weed/cluster/**, or test/fuse_dlm/**. Follows the same pattern as fuse-integration.yml and s3-mutation-regression-tests.yml. * fix(test): use pb.NewServerAddress format for master/filer addresses SeaweedFS components derive gRPC port as httpPort+10000 unless the address encodes an explicit gRPC port in the "host:port.grpcPort" format. Use pb.NewServerAddress to produce this format for -master and -filer flags, fixing volume/filer/mount startup failures in CI where randomly allocated gRPC ports differ from httpPort+10000. * fix(mount): address review feedback on DLM locking - Use time.Ticker instead of time.Sleep in renewal goroutine for interruptible cancellation on Stop() - Set isLocked=0 on renewal failure so IsLocked() reflects actual state - Use inode number as DLM lock key instead of file path to avoid race conditions during renames where the path changes while lock is held * fix(test): address CodeRabbit review feedback - Add weed/command/mount*.go to CI workflow path triggers - Register t.Cleanup(c.Stop) inside startDLMTestCluster to prevent process leaks if a require fails during startup - Use stopCmd (bounded wait with SIGKILL fallback) for mount shutdown instead of raw Signal+Wait which can hang on wedged FUSE processes - Verify actual FUSE mount by comparing device IDs of mount point vs parent directory, instead of just checking os.ReadDir succeeds - Track and assert zero write errors in stress test instead of silently logging failures * fix(test): address remaining CodeRabbit nitpicks - Add timeout to gRPC context in lock convergence check to avoid hanging on unresponsive filers - Check os.MkdirAll errors in all start functions instead of ignoring * fix(mount): acquire DLM lock in Create path and fix test issues - Add DLM lock acquisition in Create() for new files. The Create path bypasses AcquireHandle and calls fhMap.AcquireFileHandle directly, so the DLM lock was never acquired for newly created files. - Revert inode-based lock key back to file path — inode numbers are per-mount (derived from hash(path)+crtime) and differ across mounts, making inode-based keys useless for cross-mount coordination. - Both mounts connect to same filer for metadata consistency (leveldb stores are per-filer, not shared). - Simplify test assertions to verify write integrity (no corruption, all writes succeed) rather than cross-mount read convergence which depends on FUSE kernel cache invalidation timing. - Reduce stress test concurrency to avoid excessive DLM contention in CI environments. * feat(mount): add DLM locking for rename operations Acquire DLM locks on both old and new paths during rename to prevent another mount from opening either path for writing during the rename. Locks are acquired in sorted order to prevent deadlocks when two mounts rename in opposite directions (A→B vs B→A). After a successful rename, the file handle's DLM lock is migrated from the old path to the new path so the lock key matches the current file location. Add integration tests: - TestDLMRenameWhileWriteOpen: verify rename blocks while another mount holds the file open for writing - TestDLMConcurrentRenames: verify concurrent renames from different mounts are serialized without metadata corruption * fix(test): tolerate transient FUSE errors in DLM stress test Under heavy DLM contention with 8 goroutines per mount, a small number of transient FUSE flush errors (EIO on close) can occur. These are infrastructure-level errors, not DLM correctness issues. Allow up to 10% error rate in the stress test while still verifying file integrity. * fix(test): reduce DLM stress test concurrency to avoid timeouts With 8 goroutines per mount contending on 5 files, each DLM-serialized write takes ~1-2s, leading to 80+ seconds of serialized writes that exceed the test timeout. Reduce to 2 goroutines, 3 files, 3 cycles (12 writes total) for reliable completion. * fix(test): increase stress test FUSE error tolerance to 20% Transient FUSE EIO errors on close under DLM contention are infrastructure-level, not DLM correctness issues. With 12 writes and a 10% threshold (max 1 error), 2 errors caused flaky failures. Increase to ~20% tolerance for reliable CI. * fix(mount): synchronize DLM lock migration with ReleaseHandle Address review feedback: - Hold fhLockTable during DLM lock migration in handleRenameResponse to prevent racing with ReleaseHandle's dlmLock.Stop() - Replace channel-consuming probes with atomic.Bool flags in blocking tests to avoid draining the result channel prematurely - Make early completion a hard test failure (require.False) instead of a warning, since DLM should always block - Add TestDLMRenameWhileWriteOpenSameMount to verify DLM lock migration on same-mount renames * fix(mount): fix DLM rename deadlock and test improvements - Skip DLM lock on old path during rename if this mount already holds it via an open file handle, preventing self-deadlock - Synchronize DLM lock migration with fhLockTable to prevent racing with concurrent ReleaseHandle - Remove same-mount rename test (macOS FUSE kernel serializes rename and close on the same inode, causing unavoidable kernel deadlock) - Cross-mount rename test validates the DLM coordination correctly * fix(test): remove DLM stress test that times out in CI DLM serializes all writes, so multiple goroutines contending on shared files just becomes a very slow sequential test. With DLM lock acquisition + write + flush + release taking several seconds per operation, the stress test exceeds CI timeouts. The remaining 5 tests already validate DLM correctness: concurrent writes, repeated writes, write blocking, rename blocking, and concurrent renames. * fix(test): prevent port collisions between DLM test runs - Hold all port listeners open until the full batch is allocated, then close together (prevents OS from reassigning within a batch) - Add 2-second sleep after cluster Stop to allow ports to exit TIME_WAIT before the next test allocates new ports |
||
|
Chris Lu
|
b1265de78f |
feat(shell): add group management commands (#8993)
* feat(shell): add group management commands Add weed shell commands for IAM group management: - s3.group.create -name <group> - s3.group.delete -name <group> - s3.group.list - s3.group.show -name <group> - s3.group.add-user -group <group> -user <user> - s3.group.remove-user -group <group> -user <user> All commands use GetConfiguration/PutConfiguration gRPC pattern, consistent with existing shell commands like s3.user.list. * fix: add nil check for Configuration in group shell commands Guard against nil Configuration response from GetConfiguration gRPC call to prevent potential panics. (Gemini review) |
||
|
Chris Lu
|
7f3908297c |
fix(weed/shell): suppress prompt when piped (#8990)
* fix(weed/shell): suppress prompt when stdin or stdout is not a TTY When piping weed shell output (e.g. `echo "s3.user.list" | weed shell | jq`), the "> " prompt was written to stdout, breaking JSON parsers. `liner.TerminalSupported()` only checks platform support, not whether stdin/stdout are actual TTYs. Add explicit checks using `term.IsTerminal()` so the shell falls back to the non-interactive scanner path when piped. Fixes #8962 * fix(weed/shell): suppress informational logs unless -verbose is set Suppress glog info messages and connection status logs on stderr by default. Add -verbose flag to opt in to the previous noisy behavior. This keeps piped output clean (e.g. `echo "s3.user.list" | weed shell | jq`). * fix(weed/shell): defer liner init until after TTY check Move liner.NewLiner() and related setup (history, completion, interrupt handler) inside the interactive block so the terminal is not put into raw mode when stdout is redirected. Previously, liner would set raw mode unconditionally at startup, leaving the terminal broken when falling back to the scanner path. Addresses review feedback from gemini-code-assist. * refactor(weed/shell): consolidate verbose logging into single block Group all verbose stderr output within one conditional block instead of scattering three separate if-verbose checks around the filer logic. Addresses review feedback from gemini-code-assist. * fix(weed/shell): clean up global liner state and suppress logtostderr - Set line=nil after Close() to prevent stale state if RunShell is called again (e.g. in tests) - Add nil check in OnInterrupt handler for non-interactive sessions - Also set logtostderr=false when not verbose, in case it was enabled Addresses review feedback from gemini-code-assist. * refactor(weed/shell): make liner state local to eliminate data race Replace the package-level `line` variable with a local variable in RunShell, passing it explicitly to setCompletionHandler, loadHistory, and saveHistory. This eliminates a data race between the OnInterrupt goroutine and the defer that previously set the global to nil. Addresses review feedback from gemini-code-assist. * rename(weed/shell): rename -verbose flag to -debug Avoid conflict with -verbose flags already used by individual shell commands (e.g. ec.encode, volume.fix.replication, volume.check.disk). |
||
|
Chris Lu
|
45ee2ab4b9 |
feat(iam): implement ListUserPolicies API action (#8991)
* feat(iam): implement ListUserPolicies API action (#8987) Add ListUserPolicies support to both embedded and standalone IAM servers, resolving the NotImplemented error when calling `aws iam list-user-policies`. * fix: address review feedback for ListUserPolicies - Add handleImplicitUsername for ListUserPolicies in both IAM servers so omitting UserName defaults to the calling user (Gemini review) - Assert synthetic policy name in unit test (CodeRabbit) - Use require.True for error type assertion in integration test (CodeRabbit) |
||
|
Chris Lu
|
fbe758efa8 |
test: consolidate port allocation into shared test/testutil package (#8982)
* test: consolidate port allocation into shared test/testutil package Move duplicated port allocation logic from 15+ test files into a single shared package at test/testutil/. This fixes a port collision bug where independently allocated ports could overlap via the gRPC offset (port+10000), causing weed mini to reject the configuration. The shared package provides: - AllocatePorts: atomic allocation of N unique ports - AllocateMiniPorts/MustFreeMiniPorts: gRPC-offset-aware allocation that prevents port A+10000 == port B collisions - WaitForPort, WaitForService, FindBindIP, WriteIAMConfig, HasDocker * test: address review feedback and fix FUSE build - Revert fuse_integration change: it has its own go.mod and cannot import the shared testutil package - AllocateMiniPorts: hold all listeners open until the entire batch is allocated, preventing race conditions where other processes steal ports - HasDocker: add 5s context timeout to avoid hanging on stalled Docker - WaitForService: only treat 2xx HTTP status codes as ready * test: use global rand in AllocateMiniPorts for better seeding Go 1.20+ auto-seeds the global rand generator. Using it avoids identical sequences when multiple tests call at the same nanosecond. * test: revert WaitForService status code check S3 endpoints return non-2xx (e.g. 403) on bare GET requests, so requiring 2xx caused the S3 integration test to time out. Any HTTP response is sufficient proof that the service is running. * test: fix gofmt formatting in s3tables test files |
||
|
Chris Lu
|
75dcb97187 |
filer: bootstrap pre-existing metadata when a new filer joins (#8979)
* filer: bootstrap pre-existing metadata when a new filer joins a cluster When a filer connects to a peer for the first time (no stored sync offset), it now does a full BFS traversal of the peer's metadata via TraverseBfsMetadata before starting the incremental change stream. This ensures filer2 sees all data that existed before it started, fixing the issue where only post-startup changes were synced. Closes #8961 * filer: upsert during bootstrap and persist offset immediately - Use upsert (insert, then update on conflict) during metadata traversal so the bootstrap doesn't fail on the root directory or after a partial previous attempt. - Persist the sync offset right after a successful traversal so a retry doesn't redo the full BFS. * filer: address review feedback on metadata bootstrap - Use peer-side max Mtime as the streaming cursor instead of local time.Now() to avoid missing events due to clock skew between filers. traversePeerMetadata now returns the high-water Mtime (nanoseconds) observed during BFS traversal. - Compare Mtime before overwriting during bootstrap: if a local entry is newer than the peer's version, skip the update instead of clobbering it. - Only trigger full BFS traversal on ErrKvNotFound (key genuinely missing). Transient KvGet errors (connection issues, etc.) are now propagated instead of silently falling through to a full re-sync. Changed readOffset to use %w so errors.Is works through the chain. * filer: address review findings on bootstrap sync - Use wall-clock time with safety margin for stream cursor instead of entry Mtime. Mtime is file modification time (can be arbitrary), while the metadata stream uses TsNs (event log time). Using time.Now() minus 1 minute before traversal ensures no events are missed even with clock skew, matching the proven filer.meta.backup pattern. - Pass ExcludedPrefixes=[SystemLogDir] to TraverseBfsMetadata so the server prunes internal log entries server-side instead of transferring them over the network only to be filtered client-side. - Fail fast if updateOffset fails after bootstrap. If we can't persist the offset, bail out rather than proceeding and potentially losing the expensive BFS work on the next retry. |
||
|
Chris Lu
|
940eed0bd3 |
fix(ec): generate .ecx before EC shards to prevent data inconsistency (#8972)
* fix(ec): generate .ecx before EC shards to prevent data inconsistency In VolumeEcShardsGenerate, the .ecx index was generated from .idx AFTER the EC shards were generated from .dat. If any write occurred between these two steps (e.g. WriteNeedleBlob during replica sync, which bypasses the read-only check), the .ecx would contain entries pointing to data that doesn't exist in the EC shards, causing "shard too short" and "size mismatch" errors on subsequent reads and scrubs. Fix by generating .ecx FIRST, then snapshotting datFileSize, then encoding EC shards. If a write sneaks in after .ecx generation, the EC shards contain more data than .ecx references — which is harmless (the extra data is simply not indexed). Also snapshot datFileSize before EC encoding to ensure the .vif reflects the same .dat state that .ecx was generated from. Add TestEcConsistency_WritesBetweenEncodeAndEcx that reproduces the race condition by appending data between EC encoding and .ecx generation. * fix: pass actual offset to ReadBytes, improve test quality - Pass offset.ToActualOffset() to ReadBytes instead of 0 to preserve correct error metrics and error messages within ReadBytes - Handle Stat() error in assembleFromIntervalsAllowError - Rename TestEcConsistency_DatFileGrowsDuringEncoding to TestEcConsistency_ExactLargeRowEncoding (test verifies fixed-size encoding, not concurrent growth) - Update test comment to clarify it reproduces the old buggy sequence - Fix verification loop to advance by readSize for full data coverage * fix(ec): add dat/idx consistency check in worker EC encoding The erasure_coding worker copies .dat and .idx as separate network transfers. If a write lands on the source between these copies, the .idx may have entries pointing past the end of .dat, leading to EC volumes with .ecx entries that reference non-existent shard data. Add verifyDatIdxConsistency() that walks the .idx and verifies no entry's offset+size exceeds the .dat file size. This fails the EC task early with a clear error instead of silently producing corrupt EC volumes. * test(ec): add integration test verifying .ecx/.ecd consistency TestEcIndexConsistencyAfterEncode uploads multiple needles of varying sizes (14B to 256KB), EC-encodes the volume, mounts data shards, then reads every needle back via the EC read path and verifies payload correctness. This catches any inconsistency between .ecx index entries and EC shard data. * fix(test): account for needle overhead in test volume fixture WriteTestVolumeFiles created a .dat of exactly datSize bytes but the .idx entry claimed a needle of that same size. GetActualSize adds header + checksum + timestamp overhead, so the consistency check correctly rejects this as the needle extends past the .dat file. Fix by sizing the .dat to GetActualSize(datSize) so the .idx entry is consistent with the .dat contents. * fix(test): remove flaky shard ID assertion in EC scrub test When shard 0 is truncated on disk after mount, the volume server may detect corruption via parity mismatches (shards 10-13) rather than a direct read failure on shard 0, depending on OS caching/mmap behavior. Replace the brittle shard-0-specific check with a volume ID validation. * fix(test): close upload response bodies and tighten file count assertion Wrap UploadBytes calls with ReadAllAndClose to prevent connection/fd leaks during test execution. Also tighten TotalFiles check from >= 1 to == 1 since ecSetup uploads exactly one file. |
||
|
Chris Lu
|
6098ef4bd3 |
fix(test): remove flaky shard ID assertion in EC scrub test (#8978)
* test: add integration tests for volume and EC volume scrubbing Add scrub integration tests covering normal volumes (full data scrub, corrupt .dat detection, mixed healthy/broken batches, missing volume error) and EC volumes (INDEX/LOCAL modes on healthy volumes, corrupt shard detection with broken shard info reporting, corrupt .ecx index, auto-select, unsupported mode error). Also adds framework helpers: CorruptDatFile, CorruptEcxFile, CorruptEcShardFile for fault injection in scrub tests. * fix: correct dat/ecx corruption helpers and ecx test setup - CorruptDatFile: truncate .dat to superblock size instead of overwriting bytes (ensures scrub detects data file size mismatch) - TestScrubEcVolumeIndexCorruptEcx: corrupt .ecx before mount so the corrupted size is loaded into memory (EC volumes cache ecx size at mount) * fix(test): remove flaky shard ID assertion in EC scrub test When shard 0 is truncated on disk after mount, the volume server may detect corruption via parity mismatches (shards 10-13) rather than a direct read failure on shard 0, depending on OS caching/mmap behavior. Replace the brittle shard-0-specific check with a volume ID validation. * fix(test): close upload response bodies and tighten file count assertion Wrap UploadBytes calls with ReadAllAndClose to prevent connection/fd leaks during test execution. Also tighten TotalFiles check from >= 1 to == 1 since ecSetup uploads exactly one file. |
||
|
Chris Lu
|
4bf6d195e4 |
test: add integration tests for volume and EC scrubbing (#8977)
* test: add integration tests for volume and EC volume scrubbing Add scrub integration tests covering normal volumes (full data scrub, corrupt .dat detection, mixed healthy/broken batches, missing volume error) and EC volumes (INDEX/LOCAL modes on healthy volumes, corrupt shard detection with broken shard info reporting, corrupt .ecx index, auto-select, unsupported mode error). Also adds framework helpers: CorruptDatFile, CorruptEcxFile, CorruptEcShardFile for fault injection in scrub tests. * fix: correct dat/ecx corruption helpers and ecx test setup - CorruptDatFile: truncate .dat to superblock size instead of overwriting bytes (ensures scrub detects data file size mismatch) - TestScrubEcVolumeIndexCorruptEcx: corrupt .ecx before mount so the corrupted size is loaded into memory (EC volumes cache ecx size at mount) |
||
|
Chris Lu
|
74905c4b5d |
shell: s3.* commands always output JSON, connection messages to stderr (#8976)
* shell: s3.* commands output JSON, connection messages to stderr
All s3.user.* and s3.policy.attach|detach commands now output structured
JSON to stdout instead of human-readable text:
- s3.user.create: {"name","access_key"} (secret key to stderr only)
- s3.user.list: [{name,status,policies,keys}]
- s3.user.show: {name,status,source,account,policies,credentials,...}
- s3.user.delete: {"name"}
- s3.user.enable/disable: {"name","status"}
- s3.policy.attach/detach: {"policy","user"}
Connection startup messages (master/filer) moved to stderr so they
don't pollute structured output when piping.
Closes #8962 (partial — covers merged s3.user/policy commands).
* shell: fix secret leak, duplicate JSON output, and non-interactive prompt
- s3.user.create: only echo secret key to stderr when auto-generated,
never echo caller-supplied secrets
- s3.user.enable/disable: fix duplicate JSON output — remove inner
write in early-return path, keep single write site after gRPC call
- shell_liner: use bufio.Scanner when stdin is not a terminal instead
of liner.Prompt, suppressing the "> " prompt in piped mode
* shell: check scanner error, idempotent enable output, history errors to stderr
- Check scanner.Err() after non-interactive input loop to surface read errors
- s3.user.enable: always emit JSON regardless of current state (idempotent)
- saveHistory: write error messages to stderr instead of stdout
|
||
|
Chris Lu
|
b0e79ad207 |
fix(admin): respect urlPrefix for root redirect and JS API calls (#8975)
* fix(admin): respect urlPrefix for root redirect and JS API calls (#8967) Two issues when running admin UI behind a reverse proxy with -urlPrefix: 1. Visiting the prefix path without trailing slash (e.g. /s3-admin) caused a redirect to / instead of /s3-admin/ because http.StripPrefix produced an empty path that the router redirected to root. 2. Several JavaScript API calls in admin.js used hardcoded paths instead of basePath(), causing file upload, download, and preview to fail. * fix(admin): preserve query params in prefix redirect and use 302 Use http.StatusFound instead of 301 to avoid aggressive browser caching of a configuration-dependent redirect, and preserve query parameters. |
||
|
Chris Lu
|
2919bb27e5 |
fix(sync): use per-cluster TLS for HTTP volume connections in filer.sync (#8974)
* fix(sync): use per-cluster TLS for HTTP volume connections in filer.sync (#8965) When filer.sync runs with -a.security and -b.security flags, only gRPC connections received per-cluster TLS configuration. HTTP clients for volume server reads and uploads used a global singleton with the default security.toml, causing TLS verification failures when clusters use different self-signed certificates. Load per-cluster HTTPS client config from the security files and pass dedicated HTTP clients to FilerSource (for downloads) and FilerSink (for uploads) so each direction uses the correct cluster's certificates. * fix(sync): address review feedback for per-cluster HTTP TLS - Add insecure_skip_verify support to NewHttpClientWithTLS and read it from per-cluster security config via https.client.insecure_skip_verify - Error on partial mTLS config (cert without key or vice versa) - Add nil-check for client parameter in DownloadFileWithClient - Document SetUploader as init-only (same pattern as SetChunkConcurrency) |
||
|
Chris Lu
|
d50889002b |
shell: add s3.iam.*, s3.config.show, s3.user.provision; hide legacy commands (#8956)
* shell: add s3.iam.*, s3.config.show, s3.user.provision; hide legacy commands Add import/export, configuration summary, and a convenience provisioning command: - s3.iam.export: dump full IAM state as JSON (stdout or file) - s3.iam.import: replace IAM state from a JSON file - s3.config.show: human-readable summary (users, policies, service accounts, groups with status and counts) - s3.user.provision: one-step user+policy+credentials creation for common readonly/readwrite/admin roles Hide legacy commands from help listing: - s3.configure: still works but hidden from help output - s3.bucket.access: still works but hidden from help output Both hidden commands remain fully functional for existing scripts. Also adds a Hidden command tag and filters it from printGenericHelp. * shell: address review feedback for s3.iam.*, s3.config.show, s3.user.provision - Simplify joinMax using strings.Join - Fix rolePolicies: remove s3:ListBucket from object-level actions (already covered by bucket-level statement) - Fix admin role: grant s3:* on bucket resource too - Return flag parse errors instead of swallowing them * shell: address missed review feedback for PR 3 - s3.iam.import: require -force flag for destructive IAM overwrite - s3.config.show: add nil guard for resp.Configuration - s3.user.provision: check if user exists before creating policy - s3.user.provision: reject wildcard bucket names (* ?) * shell: distinguish NotFound from transient errors in provision, use %w wrapping - s3.user.provision: check gRPC status code on GetUser error — only proceed on NotFound, abort on transient/network errors - s3.iam.import: use %w for error wrapping to preserve error chains, wrap PutConfiguration error with context * shell: remove duplicate joinMax after PR 8954 merge command_s3_helpers.go defined joinMax which is already in command_s3_user_list.go from the merged PR 8954. * shell: restrict export file permissions, rollback policy on user create failure - s3.iam.export: use os.OpenFile with mode 0600 instead of os.Create to protect exported credentials from other users - s3.user.provision: rollback the created policy if CreateUser fails, with a warning if the rollback itself fails |
||
|
Chris Lu
|
efc7f3936f |
fix(s3): handle empty URL path in forwarded prefix signature verification (#8973)
fix(s3): handle empty URL path in forwarded prefix signature verification (#8966) When S3 is behind a reverse proxy with a forwarded prefix (e.g. /s3), requests with an empty URL path (like ListBuckets) would incorrectly get a trailing slash appended (e.g. /s3/), causing signature verification to fail because the client signs /s3 without the slash. |
||
|
Chris Lu
|
79a48256f5 |
fix(s3): populate s3:prefix from query param for ListObjects policy conditions (#8971)
* fix(s3): populate s3:prefix from query param for ListObjects policy conditions (#8969) ListObjectsV2/V1 requests with prefix-restricted STS session policies were denied because: 1. s3:prefix was derived from objectKey, which the auth middleware set to the prefix value, but the resource ARN then included the prefix (e.g. arn:aws:s3:::bucket/prefix) instead of staying at bucket level (arn:aws:s3:::bucket) as AWS requires for ListBucket. 2. When objectKey was empty (no middleware propagation), s3:prefix was never populated from the query parameter at all. Now AuthorizeAction extracts the prefix query parameter directly, sets it as s3:prefix in the request context, and uses a bucket-level resource ARN when the objectKey matches the propagated prefix. * fix(s3): use AWS-style wildcard matching for StringLike policy conditions filepath.Match treats * as not matching /, which breaks IAM StringLike conditions on paths (e.g. arn:aws:s3:::bucket/* won't match nested keys). Replace with a case-sensitive variant of AwsWildcardMatch that correctly treats * as matching any character including /. * refactor(s3): replace regex wildcard matching with string-based matcher Use the existing wildcard.MatchesWildcard utility instead of compiling and caching regexes for IAM wildcard matching. Removes the regexCache, its mutex, and the sync import. * refactor(s3): inline and remove AwsWildcardMatch wrapper functions Replace all call sites with direct wildcard.MatchesWildcard calls. * fix(s3): scope s3:prefix condition key to list operations only The s3:prefix logic was running for all actions, so a GetObject on "foo/bar" would wrongly populate s3:prefix. Restrict it to action "List" and always reset resourceObjectKey to "" so the resource ARN stays at bucket level. Also set s3:prefix to "" when no prefix is provided, so policies with StringEquals {"s3:prefix": ""} evaluate correctly. |
||
|
Chris Lu
|
a4753b6a3b |
S3: delay empty folder cleanup to prevent Spark write failures (#8970)
* S3: delay empty folder cleanup to prevent Spark write failures (#8963) Empty folders were being cleaned up within seconds, causing Apache Spark (s3a) writes to fail when temporary directories like _temporary/0/task_xxx/ were briefly empty. - Increase default cleanup delay from 5s to 2 minutes - Only process queue items that have individually aged past the delay (previously the entire queue was drained once any item triggered) - Make the delay configurable via filer.toml: [filer.options] s3.empty_folder_cleanup_delay = "2m" * test: increase cleanup wait timeout to match 2m delay The empty folder cleanup delay was increased to 2 minutes, so the Spark integration test needs to wait longer for temporary directories to disappear. * fix: eagerly clean parent directories after empty folder deletion After deleting an empty folder, immediately try to clean its parent rather than relying on cascading metadata events that each re-enter the 2-minute delay queue. This prevents multi-minute waits when cleaning nested temporary directory trees (e.g. Spark's _temporary hierarchy with 3+ levels would take 6m+ vs near-instant). Fixes the CI failure where lingering _temporary parent directories were not cleaned within the test's 3-minute timeout. |
||
|
Chris Lu
|
761ec7da00 |
fix(iceberg): use dot separator for namespace paths instead of unit separator (#8960)
* fix(iceberg): use dot separator for namespace paths instead of unit separator The Iceberg REST Catalog handler was using \x1F (unit separator) to join multi-level namespaces when constructing S3 location and filer paths. The S3 Tables storage layer uses "." (dot) as the namespace separator, causing tables created via the Iceberg REST API to point to different paths than where S3 Tables actually stores them. Fixes #8959 * fix(iceberg): use dot separator in log messages for readable namespace output * fix(iceberg): use path.Join for S3 location path segments Use path.Join to construct the namespace/table path segments in fallback S3 locations for robustness and consistency with handleCreateTable. * test(iceberg): add multi-level namespace integration tests for Spark and Trino Add regression tests for #8959 that create a two-level namespace (e.g. "analytics.daily"), create a table under it, insert data, and query it back. This exercises the dot-separated namespace path construction and verifies that Spark/Trino can actually read the data at the S3 location returned by the Iceberg REST API. * fix(test): enable nested namespace in Trino Iceberg catalog config Trino requires `iceberg.rest-catalog.nested-namespace-enabled=true` to support multi-level namespaces. Without this, CREATE SCHEMA with a dotted name fails with "Nested namespace is not enabled for this catalog". * fix(test): parse Trino COUNT(*) output as integer instead of substring match Avoids false matches from strings.Contains(output, "3") by parsing the actual numeric result with strconv.Atoi and asserting equality. * fix(test): use separate Trino config for nested namespace test The nested-namespace-enabled=true setting in Trino changes how SHOW SCHEMAS works, causing "Internal error" for all tests sharing that catalog config. Move the flag to a dedicated config used only by TestTrinoMultiLevelNamespace. * fix(iceberg): support parent query parameter in ListNamespaces for nested namespaces Add handling for the Iceberg REST spec's `parent` query parameter in handleListNamespaces. When Trino has nested-namespace-enabled=true, it sends `GET /v1/namespaces?parent=<ns>` to list child namespaces. The parent value is decoded from the Iceberg unit separator format and converted to a dot-separated prefix for the S3 Tables layer. Also simplify TestTrinoMultiLevelNamespace to focus on namespace operations (create, list, show tables) rather than data operations, since Trino's REST catalog has a non-empty location check that conflicts with server-side metadata creation. * fix(test): expand Trino multi-level namespace test and merge config helpers - Expand TestTrinoMultiLevelNamespace to create a table with explicit location, insert rows, query them back, and verify the S3 file path contains the dot-separated namespace (not \x1F). This ensures the original #8959 bug would be caught by the Trino integration test. - Merge writeTrinoConfig and writeTrinoNestedNamespaceConfig into a single parameterized function using functional options. |
||
|
Chris Lu
|
d4548376a1 |
fix(ec): off-by-one in nLargeBlockRows causes EC read corruption (#8957)
* fix(ec): off-by-one in nLargeBlockRows causes EC read corruption (#8947) The nLargeBlockRows formula in locateOffset used (shardDatSize-1)/largeBlockLength, which produces an off-by-one error when shardDatSize is an exact multiple of largeBlockLength (e.g. a 30GB volume with 10 data shards = 3GB per shard). This causes needles in the last large block row to be mislocated as small blocks, reading from completely wrong shard positions and returning garbage data. Fix: remove the -1 from locateOffset and only apply it in the ecdFileSize fallback path (old volumes without datFileSize in .vif), where it's needed to handle the ambiguous case conservatively. Also fix ReadEcShardNeedle to pass offset=0 to ReadBytes, consistent with the scrub path, since the bytes buffer already starts at position 0. * fix: add volume context to EC read errors, remove contextless glog The glog.Errorf in ReadBytes logged "entry not found" without any volume ID, making it impossible to identify which volume was affected. Remove this contextless log and instead add volume ID, needle ID, offset, and size to the error returned from the EC read path. The EC scrub callers already wrap errors with volume context. |
||
|
Chris Lu
|
45bf3ad058 |
shell: add s3.user.* and s3.policy.attach|detach commands (#8954)
* shell: add s3.user.* and s3.policy.attach|detach commands Add focused IAM shell commands following a noun-verb model: - s3.user.create: create user with auto-generated or explicit credentials - s3.user.list: tabular listing with status, policies, key count - s3.user.show: detailed user view (status, source, policies, credentials) - s3.user.delete: delete a user - s3.user.enable: enable a disabled user - s3.user.disable: disable a user (preserves credentials and policies) - s3.policy.attach: attach a named policy to a user - s3.policy.detach: detach a policy from a user These commands are thin wrappers over the existing IAM gRPC service, producing human-readable output instead of raw protobuf text. This is part of a larger effort to replace the monolithic s3.configure command with a composable set of single-purpose commands. * shell: address review feedback for s3.user.* and s3.policy.attach|detach - Return flag parse errors instead of swallowing them (all commands) - Use GetConfiguration instead of N+1 GetUser calls in s3.user.list - Add nil check for resp.Identity in s3.user.show - Fix GetPolicy error masking in s3.policy.attach (wrap original error) - Simplify joinMax using strings.Join * shell: add nil identity guards and wrap gRPC errors - Add nil check for resp.Identity in policy_attach, policy_detach, user_enable, user_disable - Wrap GetUser errors with user context for better diagnostics |
||
|
Chris Lu
|
d123a2768b |
shell: add s3.accesskey.*, s3.anonymous.*, s3.serviceaccount.* commands (#8955)
* shell: add s3.accesskey.*, s3.anonymous.*, s3.serviceaccount.* commands Add credential, anonymous access, and service account management commands: Access key commands: - s3.accesskey.create: add credentials to an existing user - s3.accesskey.list: list access keys for a user (key ID + status) - s3.accesskey.delete: remove a specific access key - s3.accesskey.rotate: atomic create-new + delete-old key rotation Anonymous access commands: - s3.anonymous.set: set/remove public access on a bucket - s3.anonymous.get: show anonymous access for a bucket - s3.anonymous.list: list all buckets with anonymous access Service account commands: - s3.serviceaccount.create: create with optional action subset and expiry - s3.serviceaccount.list: tabular listing, optionally filtered by parent - s3.serviceaccount.show: detailed view of a service account - s3.serviceaccount.delete: remove a service account These replace the credential and anonymous portions of the monolithic s3.configure and s3.bucket.access commands. * shell: address review feedback for s3.accesskey.*, s3.anonymous.*, s3.serviceaccount.* - Return flag parse errors instead of swallowing them (all commands) - Add action validation in s3.anonymous.set (Read, Write, List, Tagging, Admin) - Fix s3.serviceaccount.create output: note to use list for server-assigned ID since CreateServiceAccountResponse does not return the ID * shell: fix bucket matching and action validation in s3.anonymous.* - Use SplitN instead of HasSuffix for bucket name matching to avoid false positives when one bucket name is a suffix of another - Make action validation case-insensitive with canonical normalization * shell: fix nil panics, dedup actions, validate service account actions - Fix nil-pointer panic in getOrCreateAnonymousUser when GetUser returns err==nil with nil Identity (status.FromError(nil) returns nil status) - Add nil Identity guards in s3.anonymous.get and s3.anonymous.list - Deduplicate action values in s3.anonymous.set (e.g. -access Read,Read) - Add action validation in s3.serviceaccount.create with case normalization * shell: dedup actions and reject negative expiry in s3.serviceaccount.create - Deduplicate -actions values (e.g. Read,read,Read produces one entry) - Reject negative -expiry values instead of silently treating as no expiration |
||
|
Chris Lu
|
733517df30 |
fix(s3): s3:PutObject bucket policy now implicitly allows multipart uploads (#8968)
* fix(s3): s3:PutObject bucket policy now implicitly allows multipart uploads The PolicyEngine.evaluateStatement() method used raw regex matching for actions, bypassing the multipart-inherits-PutObject logic that only existed in the unused CompiledStatement.MatchesAction() code path. When a bucket policy granted only s3:PutObject, multipart upload operations (CreateMultipartUpload, UploadPart, CompleteMultipartUpload, etc.) were denied, forcing users to explicitly list every multipart action. Fixes https://github.com/seaweedfs/seaweedfs/discussions/8751 * fix(s3): add s3:UploadPartCopy to multipartActionSet and improve test coverage Add missing S3_ACTION_UPLOAD_PART_COPY constant and include it in multipartActionSet so UploadPartCopy is implicitly allowed by s3:PutObject. Also add a bucket-ARN sub-test for ListBucketMultipartUploads to verify that an object-only resource pattern does not match bucket-level requests. |
||
|
Chris Lu
|
0fed72d95a |
volume.tier.move: fulfill target replication before deleting old replicas (#8950)
* volume.tier.move: fulfill target replication before deleting old replicas When -toReplication is specified, volume.tier.move now creates all required replicas on the destination tier before deleting old replicas. This closes the data-loss window where only one copy existed on the target tier while awaiting volume.fix.replication. If replication fulfillment fails, old replicas are preserved and marked writable so the volume remains accessible. Also extracts replicateVolumeToServer and configureVolumeReplication helpers to reduce duplication across volume.tier.move and volume.fix.replication. Fixes #8937 * volume.tier.move: always fulfill replication before deleting old replicas When -toReplication is specified, use that replication setting. Otherwise, read the volume's existing replication from the super block. In both cases, all required replicas are created on the destination tier before old replicas are deleted. If replication fulfillment fails (e.g. not enough destination nodes), old replicas are preserved and marked writable so no data is lost. * volume.tier.move: address review feedback on ensureReplicationFulfilled - Add 5s delay before re-collecting topology to allow master heartbeat propagation after the move - Add nil guard for targetTierReplicas to prevent panic if the moved replica is not yet visible in the topology - Treat configureVolumeReplication failure as a hard error instead of a warning, so the rollback logic preserves old replicas * volume.tier.move: harden replication config error handling - Make configureVolumeReplication failure on the primary moved replica a hard error that aborts the move, instead of logging and continuing - Configure replication metadata on all existing target-tier replicas (not just newly created ones) when -toReplication is specified - Deletion of old replicas cannot affect new replicas since the locations list only contains pre-move servers (verified, no change) * volume.tier.move: fix cleanup deleting fulfilled replicas and broken recovery Fix 1: The cleanup loop now preserves pre-existing target-tier replicas that ensureReplicationFulfilled counted toward the replication target. Previously, a mixed-tier volume with an existing replica on the target tier could have that replica deleted right after being counted as fulfilled, leaving the volume under-replicated. ensureReplicationFulfilled now returns a preserveServers set that the deletion loop checks before removing any old replica. Fix 2: Failure paths after LiveMoveVolume (which deletes the source replica) now use restoreSurvivingReplicasWritable instead of markVolumeReplicasWritable. The old helper stopped on first error, so attempting to mark the already-deleted source writable would prevent all surviving replicas from being restored. The new helper skips the deleted source and continues through all remaining locations, logging per-replica errors instead of aborting. * volume.tier.move: mark preserved replicas writable, skip nodes with existing volume Fix 1: Preserved pre-existing target-tier replicas were left read-only after the move completed. They were marked read-only at the start (along with all other replicas) but never restored since the old code deleted them. Now they are explicitly marked writable before cleanup. Fix 2: The fulfillment loop could pick a candidate node that already hosts this volume on a different disk type, causing a VolumeCopy conflict. Added a guard that skips any node already hosting the volume (on any disk) before attempting replication. |
||
|
Chris Lu
|
69218c88fe |
fix(stats): fix build on openbsd, solaris, and windows (#8951)
fix(stats): replace undefined calculateDiskRemaining with inline calculation disk_openbsd.go, disk_solaris.go, and disk_windows.go all call calculateDiskRemaining() which is never defined, causing build failures on those platforms. Replace with the same inline calculation used in disk_supported.go. |
||
|
Chris Lu
|
b0a4647d87 |
fix: prevent stack overflow in ECBalanceTask.reportProgress (#8949)
* fix: prevent stack overflow in ECBalanceTask.reportProgress Add re-entry guard to reportProgress() to prevent infinite recursion. The progressCallback invoked by ReportProgressWithStage can re-enter reportProgress, causing a stack overflow that crashes the worker process (goroutine stack exceeds 1GB limit after ~22M frames). * fix: use atomics for progress and re-entry guard to avoid data races Address review feedback: GetProgress() can be called from a different goroutine while reportProgress is updating the value. Use atomic operations for both the progress field (via Float64bits/Float64frombits) and the reporting re-entry guard (via CompareAndSwap). |
||
|
Chris Lu
|
7ab6306e15 |
fix(kafka): resolve consumer group resumption timeout in e2e tests (#8935)
* fix(kafka): resolve consumer group resumption timeout in e2e tests Three issues caused ConsumerGroupResumption to time out when the second consumer tried to resume from committed offsets: 1. ForceCompleteRebalance deadlock: performCleanup() held group.Mu.Lock then called ForceCompleteRebalance() which tried to acquire the same lock — a guaranteed deadlock on Go's non-reentrant sync.Mutex. Fixed by requiring callers to hold the lock (matching actual call sites). 2. Unbounded fallback fetch: when the multi-batch fetch timed out, the fallback GetStoredRecords call used the connection context (no deadline). A slow broker gRPC call could block the data-plane goroutine indefinitely, causing head-of-line blocking for all responses on that connection. Fixed with a 10-second timeout. 3. HWM lookup failure caused empty responses: after a consumer leaves and the partition is deactivated, GetLatestOffset can fail. The fetch handler treated this as "no data" and entered the long-poll loop (up to 10s × 4 retries = 40s timeout). Fixed by assuming data may exist when HWM lookup fails, so the actual fetch determines availability. * fix(kafka): address review feedback on HWM sentinel and fallback timeout - Don't expose synthetic HWM (requestedOffset+1) to clients; keep result.highWaterMark at 0 when the real HWM lookup fails. - Tie fallback timeout to client's MaxWaitTime instead of a fixed 10s, so one slow partition doesn't hold the reader beyond the request budget. * fix(kafka): use large HWM sentinel and clamp fallback timeout - Use requestedOffset+10000 as sentinel HWM instead of +1, so FetchMultipleBatches doesn't artificially limit to 1 record. - Add 2s floor to fallback timeout so disk reads via gRPC have a reasonable chance even when maxWaitMs is small or zero. * fix(kafka): use MaxInt64 sentinel and derive HWM from fetch result - Use math.MaxInt64 as HWM sentinel to avoid integer overflow risk (previously requestedOffset+10000 could wrap on large offsets). - After the fetch, derive a meaningful HWM from newOffset so the client never sees MaxInt64 or 0 in the response. * fix(kafka): use remaining time budget for fallback fetch The fallback was restarting the full maxWaitMs budget even though the multi-batch fetch already consumed part of it. Now compute remaining time from either the parent context deadline or maxWaitMs minus elapsed, skip the fallback if budget is exhausted, and clamp to [2s, 10s] bounds. |
||
|
Chris Lu
|
72eb93919c |
fix(gcssink): prevent empty object finalization on write failure (#8933)
* fix(gcssink): prevent empty object finalization on write failure The GCS writer was created unconditionally with defer wc.Close(), which finalizes the upload even when content decryption or copy fails. This silently overwrites valid objects with empty data. Remove the unconditional defer, explicitly close on success to propagate errors, and delete the object on write failure. * fix(gcssink): use context cancellation instead of obj.Delete on failure obj.Delete() after a failed write would delete the existing object at that key, causing data loss on updates. Use a cancelable context instead — cancelling before Close() aborts the GCS upload without touching any pre-existing object. |
||
|
Chris Lu
|
4fd974b16b |
fix(azuresink): delete freshly created blob on write failure (#8934)
* fix(azuresink): delete freshly created blob on write failure appendBlobClient.Create() runs before content decryption and copy. If MaybeDecryptContent or CopyFromChunkViews fails, an empty blob is left behind, silently replacing any previous valid data. Add cleanup that deletes the blob on content write errors when we were the ones who created it. * fix(azuresink): track recreated blobs for cleanup on write failure handleExistingBlob deletes and recreates the blob when overwrite is needed, but freshlyCreated was only set on the initial Create success path. Set freshlyCreated = needsWrite after handleExistingBlob so recreated blobs are also cleaned up on content write failure. |
||
|
Chris Lu
|
b8fc99a9cd |
fix(s3): apply PutObject multipart expansion to STS session policies (#8932)
* fix(s3): apply PutObject multipart expansion to STS session policy evaluation (#8929) PR #8445 added logic to implicitly grant multipart upload actions when s3:PutObject is authorized, but only in the S3 API policy engine's CompiledStatement.MatchesAction(). STS session policies are evaluated through the IAM policy engine's matchesActions() -> awsIAMMatch() path, which did plain pattern matching without the multipart expansion. Add the same multipart expansion logic to the IAM policy engine's matchesActions() so that session policies containing s3:PutObject correctly allow multipart upload operations. * fix: make multipart action set lookup case-insensitive and optimize Address PR review feedback: - Lowercase multipartActionSet keys and use strings.ToLower for lookup, since AWS IAM actions are case-insensitive - Only check for s3:PutObject permission when the requested action is actually a multipart action, avoiding unnecessary awsIAMMatch calls - Add test case for case-insensitive multipart action matching |
||
|
Chris Lu
|
076d504044 |
fix(admin): reduce memory usage and verbose logging for large clusters (#8927)
* fix(admin): reduce memory usage and verbose logging for large clusters (#8919) The admin server used excessive memory and produced thousands of log lines on clusters with many volumes (e.g., 33k volumes). Three root causes: 1. Scanner duplicated all volume metrics: getVolumeHealthMetrics() created VolumeHealthMetrics objects, then convertToTaskMetrics() copied them all into identical types.VolumeHealthMetrics. Now uses the task-system type directly, eliminating the duplicate allocation and removing convertToTaskMetrics. 2. All previous task states loaded at startup: LoadTasksFromPersistence read and deserialized every .pb file from disk, logging each one. With thousands of balance tasks persisted, this caused massive startup I/O, memory usage, and log noise (including unguarded DEBUG glog.Infof per task). Now starts with an empty queue — the scanner re-detects current needs from live cluster state. Terminal tasks are purged from memory and disk when new scan results arrive. 3. Verbose per-volume/per-node logging: V(2) and V(3) logs produced thousands of lines per scan. Per-volume logs bumped to V(4), per-node/rack/disk logs bumped to V(3). Topology summary now logs counts instead of full node ID arrays. Also removes lastTopologyInfo field from MaintenanceScanner — the raw protobuf topology is returned as a local value and not retained between 30-minute scans. * fix(admin): delete stale task files at startup, add DeleteAllTaskStates Old task .pb files from previous runs were left on disk. The periodic CleanupCompletedTasks still loads all files to find completed ones — the same expensive 4GB path from the pprof profile. Now at startup, DeleteAllTaskStates removes all .pb files by scanning the directory without reading or deserializing them. The scanner will re-detect any tasks still needed from live cluster state. * fix(admin): don't persist terminal tasks to disk CompleteTask was saving failed/completed tasks to disk where they'd accumulate. The periodic cleanup only triggered for completed tasks, not failed ones. Now terminal tasks are deleted from disk immediately and only kept in memory for the current session's UI. * fix(admin): cap in-memory tasks to 100 per job type Without a limit, the task map grows unbounded — balance could create thousands of pending tasks for a cluster with many imbalanced volumes. Now AddTask rejects new tasks when a job type already has 100 in the queue. The scanner will re-detect skipped volumes on the next scan. * fix(admin): address PR review - memory-only purge, active-only capacity - purgeTerminalTasks now only cleans in-memory map (terminal tasks are already deleted from disk by CompleteTask) - Per-type capacity limit counts only active tasks (pending/assigned/ in_progress), not terminal ones - When at capacity, purge terminal tasks first before rejecting * fix(admin): fix orphaned comment, add TaskStatusCancelled to terminal switch - Move hasQueuedOrActiveTaskForVolume comment to its function definition - Add TaskStatusCancelled to the terminal state switch in CompleteTask so cancelled task files are deleted from disk |
||
|
Chris Lu
|
4efe0acaf5 |
fix(master): fast resume state and default resumeState to true (#8925)
* fix(master): fast resume state and default resumeState to true When resumeState is enabled in single-master mode, the raft server had existing log entries so the self-join path couldn't promote to leader. The server waited the full election timeout (10-20s) before self-electing. Fix by temporarily setting election timeout to 1ms before Start() when in single-master + resumeState mode with existing log, then restoring the original timeout after leader election. This makes resume near-instant. Also change the default for resumeState from false to true across all CLI commands (master, mini, server) so state is preserved by default. * fix(master): prevent fastResume goroutine from hanging forever Use defer to guarantee election timeout is always restored, and bound the polling loop with a timeout so it cannot spin indefinitely if leader election never succeeds. * fix(master): use ticker instead of time.After in fastResume polling loop |
||
|
Chris Lu
|
896114d330 |
fix(admin): fix master leader link showing incorrect port in Admin UI (#8924)
fix(admin): use gRPC address for current server in RaftListClusterServers The old Raft implementation was returning the HTTP address (ms.option.Master) for the current server, while peers used gRPC addresses (peer.ConnectionString). The Admin UI's GetClusterMasters() converts all addresses from gRPC to HTTP via GrpcAddressToServerAddress (port - 10000), which produced a negative port (-667) for the current server since its address was already in HTTP format (port 9333). Use ToGrpcAddress() for consistency with both HashicorpRaft (which stores gRPC addresses) and old Raft peers. Fixes #8921 |
||
|
Chris Lu
|
f6df7126b6 |
feat(admin): add profiling options for debugging high memory/CPU usage (#8923)
* feat(admin): add profiling options for debugging high memory/CPU usage Add -debug, -debug.port, -cpuprofile, and -memprofile flags to the admin command, matching the profiling support already available in master, volume, and other server commands. This enables investigation of resource usage issues like #8919. * refactor(admin): move profiling flags into AdminOptions struct Move cpuprofile and memprofile flags from global variables into the AdminOptions struct and init() function for consistency with other flags. * fix(debug): bind pprof server to localhost only and document profiling flags StartDebugServer was binding to all interfaces (0.0.0.0), exposing runtime profiling data to the network. Restrict to 127.0.0.1 since this is a development/debugging tool. Also add a "Debugging and Profiling" section to the admin command's help text documenting the new flags. |
||
|
Chris Lu
|
9add18e169 |
fix(volume-rust): fix volume balance between Go and Rust servers (#8915)
Two bugs prevented reliable volume balancing when a Rust volume server is the copy target: 1. find_last_append_at_ns returned None for delete tombstones (Size==0 in dat header), falling back to file mtime truncated to seconds. This caused the tail step to re-send needles from the last sub-second window. Fix: change `needle_size <= 0` to `< 0` since Size==0 delete needles still have a valid timestamp in their tail. 2. VolumeTailReceiver called read_body_v2 on delete needles, which have no DataSize/Data/flags — only checksum+timestamp+padding after the header. Fix: skip read_body_v2 when size == 0, reject negative sizes. Also: - Unify gRPC server bind: use TcpListener::bind before spawn for both TLS and non-TLS paths, propagating bind errors at startup. - Add mixed Go+Rust cluster test harness and integration tests covering VolumeCopy in both directions, copy with deletes, and full balance move with tail tombstone propagation and source deletion. - Make FindOrBuildRustBinary configurable for default vs no-default features (4-byte vs 5-byte offsets). |
||
|
Chris Lu
|
d1823d3784 |
fix(s3): include static identities in listing operations (#8903)
* fix(s3): include static identities in listing operations Static identities loaded from -s3.config file were only stored in the S3 API server's in-memory state. Listing operations (s3.configure shell command, aws iam list-users) queried the credential manager which only returned dynamic identities from the backend store. Register static identities with the credential manager after loading so they are included in LoadConfiguration and ListUsers results, and filtered out before SaveConfiguration to avoid persisting them to the dynamic store. Fixes https://github.com/seaweedfs/seaweedfs/discussions/8896 * fix: avoid mutating caller's config and defensive copies - SaveConfiguration: use shallow struct copy instead of mutating the caller's config.Identities field - SetStaticIdentities: skip nil entries to avoid panics - GetStaticIdentities: defensively copy PolicyNames slice to avoid aliasing the original * fix: filter nil static identities and sync on config reload - SetStaticIdentities: filter nil entries from the stored slice (not just from staticNames) to prevent panics in LoadConfiguration/ListUsers - Extract updateCredentialManagerStaticIdentities helper and call it from both startup and the grace.OnReload handler so the credential manager's static snapshot stays current after config file reloads * fix: add mutex for static identity fields and fix ListUsers for store callers - Add sync.RWMutex to protect staticIdentities/staticNames against concurrent reads during config reload - Revert CredentialManager.ListUsers to return only store users, since internal callers (e.g. DeletePolicy) look up each user in the store and fail on non-existent static entries - Merge static usernames in the filer gRPC ListUsers handler instead, via the new GetStaticUsernames method - Fix CI: TestIAMPolicyManagement/managed_policy_crud_lifecycle was failing because DeletePolicy iterated static users that don't exist in the store * fix: show static identities in admin UI and weed shell The admin UI and weed shell s3.configure command query the filer's credential manager via gRPC, which is a separate instance from the S3 server's credential manager. Static identities were only registered on the S3 server's credential manager, so they never appeared in the filer's responses. - Add CredentialManager.LoadS3ConfigFile to parse a static S3 config file and register its identities - Add FilerOptions.s3ConfigFile so the filer can load the same static config that the S3 server uses - Wire s3ConfigFile through in weed mini and weed server modes - Merge static usernames in filer gRPC ListUsers handler - Add CredentialManager.GetStaticUsernames helper - Add sync.RWMutex to protect concurrent access to static identity fields - Avoid importing weed/filer from weed/credential (which pulled in filer store init() registrations and broke test isolation) - Add docker/compose/s3_static_users_example.json * fix(admin): make static users read-only in admin UI Static users loaded from the -s3.config file should not be editable or deletable through the admin UI since they are managed via the config file. - Add IsStatic field to ObjectStoreUser, set from credential manager - Hide edit, delete, and access key buttons for static users in the users table template - Show a "static" badge next to static user names - Return 403 Forbidden from UpdateUser and DeleteUser API handlers when the target user is a static identity * fix(admin): show details for static users GetObjectStoreUserDetails called credentialManager.GetUser which only queries the dynamic store. For static users this returned ErrUserNotFound. Fall back to GetStaticIdentity when the store lookup fails. * fix(admin): load static S3 identities in admin server The admin server has its own credential manager (gRPC store) which is a separate instance from the S3 server's and filer's. It had no static identity data, so IsStaticIdentity returned false (edit/delete buttons shown) and GetStaticIdentity returned nil (details page failed). Pass the -s3.config file path through to the admin server and call LoadS3ConfigFile on its credential manager, matching the approach used for the filer. * fix: use protobuf is_static field instead of passing config file path The previous approach passed -s3.config file path to every component (filer, admin). This is wrong because the admin server should not need to know about S3 config files. Instead, add an is_static field to the Identity protobuf message. The field is set when static identities are serialized (in GetStaticIdentities and LoadS3ConfigFile). Any gRPC client that loads configuration via GetConfiguration automatically sees which identities are static, without needing the config file. - Add is_static field (tag 8) to iam_pb.Identity proto message - Set IsStatic=true in GetStaticIdentities and LoadS3ConfigFile - Admin GetObjectStoreUsers reads identity.IsStatic from proto - Admin IsStaticUser helper loads config via gRPC to check the flag - Filer GetUser gRPC handler falls back to GetStaticIdentity - Remove s3ConfigFile from AdminOptions and NewAdminServer signature |
||
|
Chris Lu
|
0798b274dd |
feat(s3): add concurrent chunk prefetch for large file downloads (#8917)
* feat(s3): add concurrent chunk prefetch for large file downloads Add a pipe-based prefetch pipeline that overlaps chunk fetching with response writing during S3 GetObject, SSE downloads, and filer proxy. While chunk N streams to the HTTP response, fetch goroutines for the next K chunks establish HTTP connections to volume servers ahead of time, eliminating the RTT gap between sequential chunk fetches. Uses io.Pipe for minimal memory overhead (~1MB per download regardless of chunk size, vs buffering entire chunks). Also increases the streaming read buffer from 64KB to 256KB to reduce syscall overhead. Benchmark results (64KB chunks, prefetch=4): - 0ms latency: 1058 → 2362 MB/s (2.2× faster) - 5ms latency: 11.0 → 41.7 MB/s (3.8× faster) - 10ms latency: 5.9 → 23.3 MB/s (4.0× faster) - 20ms latency: 3.1 → 12.1 MB/s (3.9× faster) * fix: address review feedback for prefetch pipeline - Fix data race: use *chunkPipeResult (pointer) on channel to avoid copying struct while fetch goroutines write to it. Confirmed clean with -race detector. - Remove concurrent map write: retryWithCacheInvalidation no longer updates fileId2Url map. Producer only reads it; consumer never writes. - Use mem.Allocate/mem.Free for copy buffer to reduce GC pressure. - Add local cancellable context so consumer errors (client disconnect) immediately stop the producer and all in-flight fetch goroutines. * fix(test): remove dead code and add Range header support in test server - Remove unused allData variable in makeChunksAndServer - Add Range header handling to createTestServer for partial chunk read coverage (206 Partial Content, 416 Range Not Satisfiable) * fix: correct retry condition and goroutine leak in prefetch pipeline - Fix retry condition: use result.fetchErr/result.written instead of copied to decide cache-invalidation retry. The old condition wrongly triggered retry when the fetch succeeded but the response writer failed on the first write (copied==0 despite fetcher having data). Now matches the sequential path (stream.go:197) which checks whether the fetcher itself wrote zero bytes. - Fix goroutine leak: when the producer's send to the results channel is interrupted by context cancellation, the fetch goroutine was already launched but the result was never sent to the channel. The drain loop couldn't handle it. Now waits on result.done before returning so every fetch goroutine is properly awaited. |
||
|
Chris Lu
|
3efe88c718 |
feat(s3): store and return checksum headers for additional checksum algorithms (#8914)
* feat(s3): store and return checksum headers for additional checksum algorithms When clients upload with --checksum-algorithm (SHA256, CRC32, etc.), SeaweedFS validated the checksum but discarded it. The checksum was never stored in metadata or returned in PUT/HEAD/GET responses. Now the checksum is computed alongside MD5 during upload, stored in entry extended attributes, and returned as the appropriate x-amz-checksum-* header in all responses. Fixes #8911 * fix(s3): address review feedback and CI failures for checksum support - Gate GET/HEAD checksum response headers on x-amz-checksum-mode: ENABLED per AWS S3 spec, fixing FlexibleChecksumError on ranged GETs and multipart copies - Verify computed checksum against client-provided header value for non-chunked uploads, returning BadDigest on mismatch - Add nil check for getCheckSumWriter to prevent panic - Handle comma-separated values in X-Amz-Trailer header - Use ordered slice instead of map for deterministic checksum header selection; extract shared mappings into package-level vars * fix(s3): skip checksum header for ranged GET responses The stored checksum covers the full object. Returning it for ranged (partial) responses causes SDK checksum validation failures because the SDK validates the header value against the partial content received. Skip emitting x-amz-checksum-* headers when a Range request header is present, fixing PyArrow large file read failures. * fix(s3): reject unsupported checksum algorithm with 400 detectRequestedChecksumAlgorithm now returns an error code when x-amz-sdk-checksum-algorithm or x-amz-checksum-algorithm contains an unsupported value, instead of silently ignoring it. * feat(s3): compute composite checksum for multipart uploads Store the checksum algorithm during CreateMultipartUpload, then during CompleteMultipartUpload compute a composite checksum from per-part checksums following the AWS S3 spec: concatenate raw per-part checksums, hash with the same algorithm, format as "base64-N" where N is part count. The composite checksum is persisted on the final object entry and returned in HEAD/GET responses (gated on x-amz-checksum-mode: ENABLED). Reuses existing per-part checksum storage from putToFiler and the getCheckSumWriter/checksumHeaders infrastructure. * fix(s3): validate checksum algorithm in CreateMultipartUpload, error on missing part checksums - Move detectRequestedChecksumAlgorithm call before mkdir callback so an unsupported algorithm returns 400 before the upload is created - Change computeCompositeChecksum to return an error when a part is missing its checksum (the upload was initiated with a checksum algorithm, so all parts must have checksums) - Propagate the error as ErrInvalidPart in CompleteMultipartUpload * fix(s3): return checksum header in CompleteMultipartUpload response, validate per-part algorithm - Add ChecksumHeaderName/ChecksumValue fields to CompleteMultipartUploadResult and set the x-amz-checksum-* HTTP response header in the handler, matching the AWS S3 CompleteMultipartUpload response spec - Validate that each part's stored checksum algorithm matches the upload's expected algorithm before assembling the composite checksum; return an error if a part was uploaded with a different algorithm |
||
|
Chris Lu
|
36f37b9b6a |
fix(filer): remove cancellation guard from RollbackTransaction and clean up #8909 (#8916)
* fix(filer): remove cancellation guard from RollbackTransaction and clean up #8909 RollbackTransaction is a cleanup operation that must succeed even when the context is cancelled — guarding it causes the exact orphaned state that #8909 was trying to prevent. Also: - Use single-evaluation `if err := ctx.Err(); err != nil` pattern instead of double-calling ctx.Err() - Remove spurious blank lines before guards - Add context.DeadlineExceeded test coverage - Simplify tests from ~230 lines to ~130 lines * fix(filer): call cancel() in expiredCtx and test rollback with expired context - Call cancel() instead of suppressing it to avoid leaking timer resources - Test RollbackTransaction with both cancelled and expired contexts |
||
|
Chris Lu
|
995dfc4d5d |
chore: remove ~50k lines of unreachable dead code (#8913)
* chore: remove unreachable dead code across the codebase Remove ~50,000 lines of unreachable code identified by static analysis. Major removals: - weed/filer/redis_lua: entire unused Redis Lua filer store implementation - weed/wdclient/net2, resource_pool: unused connection/resource pool packages - weed/plugin/worker/lifecycle: unused lifecycle plugin worker - weed/s3api: unused S3 policy templates, presigned URL IAM, streaming copy, multipart IAM, key rotation, and various SSE helper functions - weed/mq/kafka: unused partition mapping, compression, schema, and protocol functions - weed/mq/offset: unused SQL storage and migration code - weed/worker: unused registry, task, and monitoring functions - weed/query: unused SQL engine, parquet scanner, and type functions - weed/shell: unused EC proportional rebalance functions - weed/storage/erasure_coding/distribution: unused distribution analysis functions - Individual unreachable functions removed from 150+ files across admin, credential, filer, iam, kms, mount, mq, operation, pb, s3api, server, shell, storage, topology, and util packages * fix(s3): reset shared memory store in IAM test to prevent flaky failure TestLoadIAMManagerFromConfig_EmptyConfigWithFallbackKey was flaky because the MemoryStore credential backend is a singleton registered via init(). Earlier tests that create anonymous identities pollute the shared store, causing LookupAnonymous() to unexpectedly return true. Fix by calling Reset() on the memory store before the test runs. * style: run gofmt on changed files * fix: restore KMS functions used by integration tests * fix(plugin): prevent panic on send to closed worker session channel The Plugin.sendToWorker method could panic with "send on closed channel" when a worker disconnected while a message was being sent. The race was between streamSession.close() closing the outgoing channel and sendToWorker writing to it concurrently. Add a done channel to streamSession that is closed before the outgoing channel, and check it in sendToWorker's select to safely detect closed sessions without panicking. |
||
|
Chris Lu
|
8fad85aed7 |
feat(s3): support WEED_S3_SSE_KEY env var for SSE-S3 KEK (#8904)
* feat(s3): support WEED_S3_SSE_KEY env var for SSE-S3 KEK Add support for providing the SSE-S3 Key Encryption Key (KEK) via the WEED_S3_SSE_KEY environment variable (hex-encoded 256-bit key). This avoids storing the master key in plaintext on the filer at /etc/s3/sse_kek. Key source priority: 1. WEED_S3_SSE_KEY environment variable (recommended) 2. Existing filer KEK at /etc/s3/sse_kek (backward compatible) 3. Auto-generate and save to filer (deprecated for new deployments) Existing deployments with a filer-stored KEK continue to work unchanged. A deprecation warning is logged when auto-generating a new filer KEK. * refactor(s3): derive KEK from any string via HKDF instead of requiring hex Accept any secret string in WEED_S3_SSE_KEY and derive a 256-bit key using HKDF-SHA256 instead of requiring a hex-encoded key. This is simpler for users — no need to generate hex, just set a passphrase. * feat(s3): add WEED_S3_SSE_KEK and WEED_S3_SSE_KEY env vars for KEK Two env vars for providing the SSE-S3 Key Encryption Key: - WEED_S3_SSE_KEK: hex-encoded, same format as /etc/s3/sse_kek. If the filer file also exists, they must match. - WEED_S3_SSE_KEY: any string, 256-bit key derived via HKDF-SHA256. Refuses to start if /etc/s3/sse_kek exists (must delete first). Only one may be set. Existing filer-stored KEKs continue to work. Auto-generating and storing new KEKs on filer is deprecated. * fix(s3): stop auto-generating KEK, fail only when SSE-S3 is used Instead of auto-generating a KEK and storing it on the filer when no key source is configured, simply leave SSE-S3 disabled. Encrypt and decrypt operations return a clear error directing the user to set WEED_S3_SSE_KEK or WEED_S3_SSE_KEY. * refactor(s3): move SSE-S3 KEK config to security.toml Move KEK configuration from standalone env vars to security.toml's new [sse_s3] section, following the same pattern as JWT keys and TLS certs. [sse_s3] kek = "" # hex-encoded 256-bit key (same format as /etc/s3/sse_kek) key = "" # any string, HKDF-derived Viper's WEED_ prefix auto-mapping provides env var support: WEED_SSE_S3_KEK and WEED_SSE_S3_KEY. All existing behavior is preserved: filer KEK fallback, mismatch detection, and HKDF derivation. * refactor(s3): rename SSE-S3 config keys to s3.sse.kek / s3.sse.key Use [s3.sse] section in security.toml, matching the existing naming convention (e.g. [s3.*]). Env vars: WEED_S3_SSE_KEK, WEED_S3_SSE_KEY. * fix(s3): address code review findings for SSE-S3 KEK - Don't hold mutex during filer retry loop (up to 20s of sleep). Lock only to write filerClient and superKey. - Remove dead generateAndSaveSuperKeyToFiler and unused constants. - Return error from deriveKeyFromSecret instead of ignoring it. - Fix outdated doc comment on InitializeWithFiler. - Use t.Setenv in tests instead of manual os.Setenv/Unsetenv. * fix(s3): don't block startup on filer errors when KEK is configured - When s3.sse.kek is set, a temporarily unreachable filer no longer prevents startup. The filer consistency check becomes best-effort with a warning. - Same treatment for s3.sse.key: filer unreachable logs a warning instead of failing. - Rewrite error messages to suggest migration instead of file deletion, avoiding the risk of orphaning encrypted data. Finding 3 (restore auto-generation) intentionally skipped — auto-gen was removed by design to avoid storing plaintext KEK on filer. * fix(test): set WEED_S3_SSE_KEY in SSE integration test server startup SSE-S3 no longer auto-generates a KEK, so integration tests must provide one. Set WEED_S3_SSE_KEY=test-sse-s3-key in all weed mini invocations in the test Makefile. |
||
|
Chris Lu
|
2e98902f29 |
fix(s3): use URL-safe secret keys for dashboard users and service accounts (#8902)
* fix(s3): use URL-safe secret keys for admin dashboard users and service accounts The dashboard's generateSecretKey() used base64.StdEncoding which produces +, /, and = characters that break S3 signature authentication. Reuse the IAM package's GenerateSecretAccessKey() which was already fixed in #7990. Fixes #8898 * fix: handle error from GenerateSecretAccessKey instead of ignoring it |
||
|
Chris Lu
|
059bee683f |
feat(s3): add STS GetFederationToken support (#8891)
* feat(s3): add STS GetFederationToken support Implement the AWS STS GetFederationToken API, which allows long-term IAM users to obtain temporary credentials scoped down by an optional inline session policy. This is useful for server-side applications that mint per-user temporary credentials. Key behaviors: - Requires SigV4 authentication from a long-term IAM user - Rejects calls from temporary credentials (session tokens) - Name parameter (2-64 chars) identifies the federated user - DurationSeconds supports 900-129600 (15 min to 36 hours, default 12h) - Optional inline session policy for permission scoping - Caller's attached policies are embedded in the JWT token - Returns federated user ARN: arn:aws:sts::<account>:federated-user/<Name> No performance impact on the S3 hot path — credential vending is a separate control-plane operation, and all policy data is embedded in the stateless JWT token. * fix(s3): address GetFederationToken PR review feedback - Fix Name validation: max 32 chars (not 64) per AWS spec, add regex validation for [\w+=,.@-]+ character whitelist - Refactor parseDurationSeconds into parseDurationSecondsWithBounds to eliminate duplicated duration parsing logic - Add sts:GetFederationToken permission check via VerifyActionPermission mirroring the AssumeRole authorization pattern - Change GetPoliciesForUser to return ([]string, error) so callers fail closed on policy-resolution failures instead of silently returning nil - Move temporary-credentials rejection before SigV4 verification for early rejection and proper test coverage - Update tests: verify specific error message for temp cred rejection, add regex validation test cases (spaces, slashes rejected) * refactor(s3): use sts.Action* constants instead of hard-coded strings Replace hard-coded "sts:AssumeRole" and "sts:GetFederationToken" strings in VerifyActionPermission calls with sts.ActionAssumeRole and sts.ActionGetFederationToken package constants. * fix(s3): pass through sts: prefix in action resolver and merge policies Two fixes: 1. mapBaseActionToS3Format now passes through "sts:" prefix alongside "s3:" and "iam:", preventing sts:GetFederationToken from being rewritten to s3:sts:GetFederationToken in VerifyActionPermission. This also fixes the existing sts:AssumeRole permission checks. 2. GetFederationToken policy embedding now merges identity.PolicyNames (from SigV4 identity) with policies from the IAM manager (which may include group-attached policies), deduplicated via a map. Previously the IAM manager lookup was skipped when identity.PolicyNames was non-empty, causing group policies to be omitted from the token. * test(s3): add integration tests for sts: action passthrough and policy merge Action resolver tests: - TestMapBaseActionToS3Format_ServicePrefixPassthrough: verifies s3:, iam:, and sts: prefixed actions pass through unchanged while coarse actions (Read, Write) are mapped to S3 format - TestResolveS3Action_STSActionsPassthrough: verifies sts:AssumeRole, sts:GetFederationToken, sts:GetCallerIdentity pass through ResolveS3Action unchanged with both nil and real HTTP requests Policy merge tests: - TestGetFederationToken_GetPoliciesForUser: tests IAMManager.GetPoliciesForUser with no user store (error), missing user, user with policies, user without - TestGetFederationToken_PolicyMergeAndDedup: tests that identity.PolicyNames and IAM-manager-resolved policies are merged and deduplicated (SharedPolicy appears in both sources, result has 3 unique policies) - TestGetFederationToken_PolicyMergeNoManager: tests that when IAM manager is unavailable, identity.PolicyNames alone are embedded * test(s3): add end-to-end integration tests for GetFederationToken Add integration tests that call GetFederationToken using real AWS SigV4 signed HTTP requests against a running SeaweedFS instance, following the existing pattern in test/s3/iam/s3_sts_assume_role_test.go. Tests: - TestSTSGetFederationTokenValidation: missing name, name too short/long, invalid characters, duration too short/long, malformed policy, anonymous rejection (7 subtests) - TestSTSGetFederationTokenRejectTemporaryCredentials: obtains temp creds via AssumeRole then verifies GetFederationToken rejects them - TestSTSGetFederationTokenSuccess: basic success, custom 1h duration, 36h max duration with expiration time verification - TestSTSGetFederationTokenWithSessionPolicy: creates a bucket, obtains federated creds with GetObject-only session policy, verifies GetObject succeeds and PutObject is denied using the AWS SDK S3 client |
||
|
Chris Lu
|
a4b896a224 |
fix(s3): skip directories before marker in ListObjectVersions pagination (#8890)
* fix(s3): skip directories before marker in ListObjectVersions pagination ListObjectVersions was re-traversing the entire directory tree from the beginning on every paginated request, only skipping entries at the leaf level. For buckets with millions of objects in deep hierarchies, this caused exponentially slower responses as pagination progressed. Two optimizations: 1. Use keyMarker to compute a startFrom position at each directory level, skipping directly to the relevant entry instead of scanning from the beginning (mirroring how ListObjects uses marker descent). 2. Skip recursing into subdirectories whose keys are entirely before the keyMarker. Changes per-page cost from O(entries_before_marker) to O(tree_depth). * test(s3): add integration test for deep-hierarchy version listing pagination Adds TestVersioningPaginationDeepDirectoryHierarchy which creates objects across 20 subdirectories at depth 6 (mimicking Veeam 365 backup layout) and paginates through them with small maxKeys. Verifies correctness (no duplicates, sorted order, all objects found) and checks that later pages don't take dramatically longer than earlier ones — the symptom of the pre-fix re-traversal bug. Also tests delimiter+pagination interaction across subdirectories. * test(s3): strengthen deep-hierarchy pagination assertions - Replace timing warning (t.Logf) with a failing assertion (t.Errorf) so pagination regressions actually fail the test. - Replace generic count/uniqueness/sort checks on CommonPrefixes with exact equality against the expected prefix slice, catching wrong-but- sorted results. * test(s3): use allKeys for exact assertion in deep-hierarchy pagination test Wire the allKeys slice (previously unused dead code) into the version listing assertion, replacing generic count/uniqueness/sort checks with an exact equality comparison against the keys that were created. |
||
|
Chris Lu
|
7c59b639c9 |
STS: add GetCallerIdentity support (#8893)
* STS: add GetCallerIdentity support Implement the AWS STS GetCallerIdentity action, which returns the ARN, account ID, and user ID of the caller based on SigV4 authentication. This is commonly used by AWS SDKs and CLI tools (e.g. `aws sts get-caller-identity`) to verify credentials and determine the authenticated identity. * test: remove trivial GetCallerIdentity tests Remove the XML unmarshal test (we don't consume this response as input) and the routing constant test (just asserts a literal equals itself). * fix: route GetCallerIdentity through STS in UnifiedPostHandler and use stable UserId - UnifiedPostHandler only dispatched actions starting with "AssumeRole" to STS, so GetCallerIdentity in a POST body would fall through to the IAM path and get AccessDenied for non-admin users. Add explicit check for GetCallerIdentity. - Use identity.Name as UserId instead of credential.AccessKey, which is a transient value and incorrect for STS assumed-role callers. |
||
|
Chris Lu
|
98714b9f70 |
fix(test): address flaky S3 distributed lock integration test (#8888)
* fix(test): address flaky S3 distributed lock integration test Two root causes: 1. Lock ring convergence race: After waitForFilerCount(2) confirms the master sees both filers, there's a window where filer0's lock ring still only contains itself (master's LockRingUpdate broadcast is delayed by the 1s stabilization timer). During this window filer0 considers itself primary for ALL keys, so both filers can independently grant the same lock. Fix: Add waitForLockRingConverged() that acquires the same lock through both filers and verifies mutual exclusion before proceeding. 2. Hash function mismatch: ownerForObjectLock used util.HashStringToLong (MD5 + modulo) to predict lock owners, but the production DLM uses CRC32 consistent hashing via HashRing. This meant the test could pick keys that route to the same filer, not exercising the cross-filer coordination it intended to test. Fix: Use lock_manager.NewHashRing + GetPrimary() to match production routing exactly. * fix(test): verify lock denial reason in convergence check Ensure the convergence check only returns true when the second lock attempt is denied specifically because the lock is already owned, avoiding false positives from transient errors. * fix(test): check one key per primary filer in convergence wait A single arbitrary key can false-pass: if its real primary is the filer with the stale ring, mutual exclusion holds trivially because that filer IS the correct primary. Generate one test key per distinct primary using the same consistent-hash ring as production, so a stale ring on any filer is caught deterministically. |
||
|
Chris Lu
|
9552e80b58 |
filer.sync: show active chunk transfers when sync progress stalls (#8889)
* filer.sync: show active chunk transfers when sync progress stalls When the sync watermark is not advancing, print each in-progress chunk transfer with its file path, bytes received so far, and current status (downloading, uploading, or waiting with backoff duration). This helps diagnose which files are blocking progress during replication. Closes #8542 * filer.sync: include last error in stall diagnostics * filer.sync: fix data races in ChunkTransferStatus Add sync.RWMutex to ChunkTransferStatus and lock around all field mutations in fetchAndWrite. ActiveTransfers now returns value copies under RLock so callers get immutable snapshots. |
||
|
Chris Lu
|
ab4e52ae2f |
fix(s3): use recursive delete for .versions directory cleanup (#8887)
* fix(s3): use recursive delete for .versions directory cleanup When only delete markers remain in a .versions directory, updateLatestVersionAfterDeletion tried to delete it non-recursively, which failed with "fail to delete non-empty folder" because the delete marker entries were still present. Use recursive deletion so the directory and its remaining delete marker entries are cleaned up together. * fix(s3): guard .versions directory deletion against truncated listings When the version listing is truncated (>1000 entries), content versions may exist beyond the first page. Skip the recursive directory deletion in this case to prevent data loss. * fix(s3): preserve delete markers in .versions directory Delete markers must be preserved per S3 semantics — they are only removed by an explicit DELETE with versionId. The previous fix would recursively delete the entire .versions directory (including delete markers) when no content versions were found. Now the logic distinguishes three cases: 1. Content versions exist → update latest version metadata 2. Only delete markers remain (or listing truncated) → keep directory 3. Truly empty → safe to delete directory (non-recursive) |
||
|
Chris Lu
|
888c32cbde |
fix(admin): respect urlPrefix in S3 bucket and S3Tables navigation links (#8885)
* fix(admin): respect urlPrefix in S3 bucket and S3Tables navigation links (#8884) Several admin UI templates used hardcoded URLs (templ.SafeURL) instead of dash.PUrl(ctx, ...) for navigation links, causing 404 errors when the admin is deployed with --urlPrefix. Fixed in: s3_buckets.templ, s3tables_buckets.templ, s3tables_tables.templ * fix(admin): URL-escape bucketName in S3Tables navigation links Add url.PathEscape(bucketName) for consistency and correctness in s3tables_tables.templ (back-to-namespaces link) and s3tables_buckets.templ (namespace link), matching the escaping already used in the table details link. |
||
|
Chris Lu
|
efbed39e25 |
S3: map canned ACL to file permissions and add configurable default file mode (#8886)
* S3: map canned ACL to file permissions and add configurable default file mode S3 uploads were hardcoded to 0660 regardless of ACL headers. Now the X-Amz-Acl header maps to Unix file permissions per-object: - public-read, authenticated-read, bucket-owner-read → 0644 - public-read-write → 0666 - private, bucket-owner-full-control → 0660 Also adds -defaultFileMode / -s3.defaultFileMode flag to set a server-wide default when no ACL header is present. Closes #8874 * Address review feedback for S3 file mode feature - Extract hardcoded 0660 to defaultFileMode constant - Change parseDefaultFileMode to return error instead of calling Fatalf - Add -s3.defaultFileMode flag to filer.go and mini.go (was missing) - Add doc comment to S3Options about updating all four flag sites - Add TestResolveFileMode with 10 test cases covering ACL mapping, server default, and priority ordering |
||
|
Chris Luandgemini-code-assist[bot]
|
b3e50bb12f |
fix(s3): remove customer encryption key from SSE-C debug log (#8875)
* fix(s3): remove customer encryption key from SSE-C debug log The debug log in validateAndParseSSECHeaders was logging the raw customer-provided encryption key bytes in hex format (keyBytes=%x), leaking sensitive key material to log output. Remove the key bytes from the log statement while keeping the MD5 hash comparison info. * Apply suggestion from @gemini-code-assist[bot] Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
4287b7b12a |
Add manual trigger to Rust volume server release build workflow (#8873)
* Add manual trigger to Rust volume server release build workflow When triggered manually via workflow_dispatch, binaries are uploaded as downloadable workflow artifacts instead of release assets. On tag push the existing release upload behavior is unchanged. * Vendor OpenSSL for cross-compilation of Rust volume server The aarch64-unknown-linux-gnu build fails because openssl-sys cannot find OpenSSL via pkg-config when cross-compiling. Adding openssl with the vendored feature builds OpenSSL from source, fixing the issue. * Fix aarch64 cross-compilation: install libssl-dev:arm64 instead of vendoring OpenSSL The vendored OpenSSL feature breaks the S3 tier unit test by altering the TLS stack behavior. Instead, install the aarch64 OpenSSL dev libraries and point the build at them via OPENSSL_DIR/LIB_DIR/INCLUDE_DIR. |
||
|
Chris Lu
|
8572aae403 |
filer.sync: support per-cluster mTLS with -a.security and -b.security (#8872)
* filer.sync: support per-cluster mTLS with -a.security and -b.security flags When syncing between two clusters that use different certificate authorities, a single security.toml cannot authenticate to both. Add -a.security and -b.security flags so each filer can use its own security.toml for TLS. Closes #8481 * security: fatal on failure to read explicitly provided security config When -a.security or -b.security is specified, falling back to insecure credentials on read error would silently bypass mTLS. Fatal instead. * fix(filer.sync): use source filer's fromTsMs flag in initOffsetFromTsMs A→B was using bFromTsMs and B→A was using aFromTsMs — these were swapped. Each path should seed the target's offset with the source filer's starting timestamp. * security: return error from LoadClientTLSFromFile, resolve relative PEM paths Change LoadClientTLSFromFile to return (grpc.DialOption, error) so callers can handle failures explicitly instead of a silent insecure fallback. Resolve relative PEM paths (grpc.ca, grpc.client.cert, grpc.client.key) against the config file's directory. |
||
|
Chris Lu
|
44d5cb8f90 |
Fix Admin UI master list showing gRPC port instead of HTTP port (#8869)
* Fix Admin UI master list showing gRPC port instead of HTTP port for followers (#8867) Raft stores server addresses as gRPC addresses. The Admin UI was using these addresses directly via ToHttpAddress(), which cannot extract the HTTP port from a plain gRPC address. Use GrpcAddressToServerAddress() to properly convert gRPC addresses back to HTTP addresses. * Use httpAddress consistently as masterMap key Address review feedback: masterInfo.Address (HTTP form) was already computed but the raw address was used as the map key, causing potential key mismatches between topology and raft data. |
||
|
Chris LuandCopilot
|
af68449a26 |
Process .ecj deletions during EC decode and vacuum decoded volume (#8863)
* Process .ecj deletions during EC decode and vacuum decoded volume (#8798) When decoding EC volumes back to normal volumes, deletions recorded in the .ecj journal were not being applied before computing the dat file size or checking for live needles. This caused the decoded volume to include data for deleted files and could produce false positives in the all-deleted check. - Call RebuildEcxFile before HasLiveNeedles/FindDatFileSize in VolumeEcShardsToVolume so .ecj deletions are merged into .ecx first - Vacuum the decoded volume after mounting in ec.decode to compact out deleted needle data from the .dat file - Add integration tests for decoding with non-empty .ecj files * storage: add offline volume compaction helper Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ec: compact decoded volumes before deleting shards Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ec: address PR review comments - Fall back to data directory for .ecx when idx directory lacks it - Make compaction failure non-fatal during EC decode - Remove misleading "buffer: 10%" from space check error message * ec: collect .ecj from all shard locations during decode Each server's .ecj only contains deletions for needles whose data resides in shards held by that server. Previously, sources with no new data shards to contribute were skipped entirely, losing their .ecj deletion entries. Now .ecj is always appended from every shard location so RebuildEcxFile sees the full set of deletions. * ec: add integration tests for .ecj collection during decode TestEcDecodePreservesDeletedNeedles: verifies that needles deleted via VolumeEcBlobDelete are excluded from the decoded volume. TestEcDecodeCollectsEcjFromPeer: regression test for the fix in collectEcShards. Deletes a needle only on a peer server that holds no new data shards, then verifies the deletion survives decode via .ecj collection. * ec: address review nits in decode and tests - Remove double error wrapping in mountDecodedVolume - Check VolumeUnmount error in peer ecj test - Assert 404 specifically for deleted needles, fail on 5xx --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Chris LuandCopilot
|
75a6a34528 |
dlm: resilient distributed locks via consistent hashing + backup replication (#8860)
* dlm: replace modulo hashing with consistent hash ring Introduce HashRing with virtual nodes (CRC32-based consistent hashing) to replace the modulo-based hashKeyToServer. When a filer node is removed, only keys that hashed to that node are remapped to the next server on the ring, leaving all other mappings stable. This is the foundation for backup replication — the successor on the ring is always the natural takeover node. * dlm: add Generation and IsBackup fields to Lock Lock now carries IsBackup (whether this node holds the lock as a backup replica) and Generation (a monotonic fencing token that increments on each fresh acquisition, stays the same on renewal). Add helper methods: AllLocks, PromoteLock, DemoteLock, InsertBackupLock, RemoveLock, GetLock. * dlm: add ReplicateLock RPC and generation/is_backup proto fields Add generation field to LockResponse for fencing tokens. Add generation and is_backup fields to Lock message. Add ReplicateLock RPC for primary-to-backup lock replication. Add ReplicateLockRequest/ReplicateLockResponse messages. * dlm: add async backup replication to DistributedLockManager Route lock/unlock via consistent hash ring's GetPrimaryAndBackup(). After a successful lock or unlock on the primary, asynchronously replicate the operation to the backup server via ReplicateFunc callback. Single-server deployments skip replication. * dlm: add ReplicateLock handler and backup-aware topology changes Add ReplicateLock gRPC handler for primary-to-backup replication. Revise OnDlmChangeSnapshot to handle three cases on topology change: - Promote backup locks when this node becomes primary - Demote primary locks when this node becomes backup - Transfer locks when this node is neither primary nor backup Wire up SetupDlmReplication during filer server initialization. * dlm: expose generation fencing token in lock client LiveLock now captures the generation from LockResponse and exposes it via Generation() method. Consumers can use this as a fencing token to detect stale lock holders. * dlm: update empty folder cleaner to use consistent hash ring Replace local modulo-based hashKeyToServer with LockRing.GetPrimary() which uses the shared consistent hash ring for folder ownership. * dlm: add unit tests for consistent hash ring Test basic operations, consistency on server removal (only keys from removed server move), backup-is-successor property (backup becomes new primary when primary is removed), and key distribution balance. * dlm: add integration tests for lock replication failure scenarios Test cases: - Primary crash with backup promotion (backup has valid token) - Backup crash with primary continuing - Both primary and backup crash (lock lost, re-acquirable) - Rolling restart across all nodes - Generation fencing token increments on new acquisition - Replication failure (primary still works independently) - Unlock replicates deletion to backup - Lock survives server addition (topology change) - Consistent hashing minimal disruption (only removed server's keys move) * dlm: address PR review findings 1. Causal replication ordering: Add per-lock sequence number (Seq) that increments on every mutation. Backup rejects incoming mutations with seq <= current seq, preventing stale async replications from overwriting newer state. Unlock replication also carries seq and is rejected if stale. 2. Demote-after-handoff: OnDlmChangeSnapshot now transfers the lock to the new primary first and only demotes to backup after a successful TransferLocks RPC. If the transfer fails, the lock stays as primary on this node. 3. SetSnapshot candidateServers leak: Replace the candidateServers map entirely instead of appending, so removed servers don't linger. 4. TransferLocks preserves Generation and Seq: InsertLock now accepts generation and seq parameters. After accepting a transferred lock, the receiving node re-replicates to its backup. 5. Rolling restart test: Add re-replication step after promotion and assert survivedCount > 0. Add TestDLM_StaleReplicationRejected. 6. Mixed-version upgrade note: Add comment on HashRing documenting that all filer nodes must be upgraded together. * dlm: serve renewals locally during transfer window on node join When a new node joins and steals hash ranges from surviving nodes, there's a window between ring update and lock transfer where the client gets redirected to a node that doesn't have the lock yet. Fix: if the ring says primary != self but we still hold the lock locally (non-backup, matching token), serve the renewal/unlock here rather than redirecting. The lock will be transferred by OnDlmChangeSnapshot, and subsequent requests will go to the new primary once the transfer completes. Add tests: - TestDLM_NodeDropAndJoin_OwnershipDisruption: measures disruption when a node drops and a new one joins (14/100 surviving-node locks disrupted, all handled by transfer logic) - TestDLM_RenewalDuringTransferWindow: verifies renewal succeeds on old primary during the transfer window * dlm: master-managed lock ring with stabilization batching The master now owns the lock ring membership. Instead of filers independently reacting to individual ClusterNodeUpdate add/remove events, the master: 1. Tracks filer membership in LockRingManager 2. Batches rapid changes with a 1-second stabilization timer (e.g., a node drop + join within 1 second → single ring update) 3. Broadcasts the complete ring snapshot atomically via the new LockRingUpdate message in KeepConnectedResponse Filers receive the ring as a complete snapshot and apply it via SetSnapshot, ensuring all filers converge to the same ring state without intermediate churn. This eliminates the double-churn problem where a rapid drop+join would fire two separate ring mutations, each triggering lock transfers and disrupting ownership on surviving nodes. * dlm: track ring version, reject stale updates, remove dead code SetSnapshot now takes a version parameter from the master. Stale updates (version < current) are rejected, preventing reordered messages from overwriting a newer ring state. Version 0 is always accepted for bootstrap. Remove AddServer/RemoveServer from LockRing — the ring is now exclusively managed by the master via SetSnapshot. Remove the candidateServers map that was only used by those methods. * dlm: fix SelectLocks data race, advance generation on backup insert - SelectLocks: change RLock to Lock since the function deletes map entries, which is a write operation and causes a data race under RLock. - InsertBackupLock: advance nextGeneration to at least the incoming generation so that after failover promotion, new lock acquisitions get a generation strictly greater than any replicated lock. - Bump replication failure log from V(1) to Warningf for production visibility. * dlm: fix SetSnapshot race, test reliability, timer edge cases - SetSnapshot: hold LockRing lock through both version update and Ring.SetServers() so they're atomic. Prevents a concurrent caller from seeing the new version but applying stale servers. - Transfer window test: search for a key that actually moves primary when filer4 joins, instead of relying on a fixed key that may not. - renewLock redirect: pass the existing token to the new primary instead of empty string, so redirected renewals work correctly. - scheduleBroadcast: check timer.Stop() return value. If the timer already fired, the callback picks up latest state. - FlushPending: only broadcast if timer.Stop() returns true (timer was still pending). If false, the callback is already running. - Fix test comment: "idempotent" → "accepted, state-changing". * dlm: use wall-clock nanoseconds for lock ring version The lock ring version was an in-memory counter that reset to 0 on master restart. A filer that had seen version 5 would reject version 1 from the restarted master. Fix: use time.Now().UnixNano() as the version. This survives master restarts without persistence — the restarted master produces a version greater than any pre-restart value. * dlm: treat expired lock owners as missing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * dlm: reject stale lock transfers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * dlm: order replication by generation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * dlm: bootstrap lock ring on reconnect Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Chris Lu
|
9205140bd5 |
Use Unix sockets for gRPC in weed server mode (#8858)
* Use Unix sockets for gRPC between co-located services in weed server Extends the Unix socket gRPC optimization (added for mini mode in #8856) to `weed server`. Registers Unix socket paths for each service's gRPC port before startup, so co-located services (master, volume, filer, S3) communicate via Unix sockets instead of TCP loopback. Only services actually started in this process get registered. The gRPC port is resolved early (port + 10000 if unset) so the socket path is known before any service dials another. * Refactor gRPC Unix socket registration into a data-driven loop |
||
|
Chris Lu
|
4705d8b82b |
Fix stale admin lock metric when lock expires and is reacquired (#8859)
* Fix stale admin lock metric when lock expires and is reacquired (#8857) When a lock expired without an explicit unlock and a different client acquired it, the old client's metric was never cleared, causing multiple clients to appear as simultaneously holding the lock. * Use DeleteLabelValues instead of Set(0) to remove stale metric series Avoids cardinality explosion from accumulated stale series when client names are dynamic. |
||
|
Chris Lu
|
ced2236cc6 |
Adjust rename events metadata format (#8854)
* rename metadata events * fix subscription filter to use NewEntry.Name for rename path matching The server-side subscription filter constructed the new path using OldEntry.Name instead of NewEntry.Name when checking if a rename event's destination matches the subscriber's path prefix. This could cause events to be incorrectly filtered when a rename changes the file name. * fix bucket events to handle rename of bucket directories onBucketEvents only checked IsCreate and IsDelete. A bucket directory rename via AtomicRenameEntry now emits a single rename event (both OldEntry and NewEntry non-nil), which matched neither check. Handle IsRename by deleting the old bucket and creating the new one. * fix replicator to handle rename events across directory boundaries Two issues fixed: 1. The replicator filtered events by checking if the key (old path) was under the source directory. Rename events now use the old path as key, so renames from outside into the watched directory were silently dropped. Now both old and new paths are checked, and cross-boundary renames are converted to create or delete. 2. NewParentPath was passed to the sink without remapping to the sink's target directory structure, causing the sink to write entries at the wrong location. Now NewParentPath is remapped alongside the key. * fix filer sync to handle rename events crossing directory boundaries The early directory-prefix filter only checked resp.Directory (old parent). Rename events now carry the old parent as Directory, so renames from outside the source path into it were dropped before reaching the existing cross-boundary handling logic. Check both old and new directories against sourcePath and excludePaths so the downstream old-key/new-key logic can properly convert these to create or delete operations. * fix metadata event path matching * fix metadata event consumers for rename targets * Fix replication rename target keys Logical rename events now reach replication sinks with distinct source and target paths.\n\nHandle non-filer sinks as delete-plus-create on the translated target key, and make the rename fallback path create at the translated target key too.\n\nAdd focused tests covering non-filer renames, filer rename updates, and the fallback path.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix filer sync rename path scoping Use directory-boundary matching instead of raw prefix checks when classifying source and target paths during filer sync.\n\nAlso apply excludePaths per side so renames across excluded boundaries downgrade cleanly to create/delete instead of being misclassified as in-scope updates.\n\nAdd focused tests for boundary matching and rename classification.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix replicator directory boundary checks Use directory-boundary matching instead of raw prefix checks when deciding whether a source or target path is inside the watched tree or an excluded subtree.\n\nThis prevents sibling paths such as /foo and /foobar from being misclassified during rename handling, and preserves the earlier rename-target-key fix.\n\nAdd focused tests for boundary matching and rename classification across sibling/excluded directories.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix etc-remote rename-out handling Use boundary-safe source/target directory membership when classifying metadata events under DirectoryEtcRemote.\n\nThis prevents rename-out events from being processed as config updates, while still treating them as removals where appropriate for the remote sync and remote gateway command paths.\n\nAdd focused tests for update/removal classification and sibling-prefix handling.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Defer rename events until commit Queue logical rename metadata events during atomic and streaming renames and publish them only after the transaction commits successfully.\n\nThis prevents subscribers from seeing delete or logical rename events for operations that later fail during delete or commit.\n\nAlso serialize notification.Queue swaps in rename tests and add failure-path coverage.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Skip descendant rename target lookups Avoid redundant target lookups during recursive directory renames once the destination subtree is known absent.\n\nThe recursive move path now inserts known-absent descendants directly, and the test harness exercises prefixed directory listing so the optimization is covered by a directory rename regression test.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tighten rename review tests Return filer_pb.ErrNotFound from the bucket tracking store test stub so it follows the FilerStore contract, and add a webhook filter case for same-name renames across parent directories.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix HardLinkId format verb in InsertEntryKnownAbsent error HardLinkId is a byte slice. %d prints each byte as a decimal number which is not useful for an identifier. Use %x to match the log line two lines above. * only skip descendant target lookup when source and dest use same store moveFolderSubEntries unconditionally passed skipTargetLookup=true for every descendant. This is safe when all paths resolve to the same underlying store, but with path-specific store configuration a child's destination may map to a different backend that already holds an entry at that path. Use FilerStoreWrapper.SameActualStore to check per-child and fall back to the full CreateEntry path when stores differ. * add nil and create edge-case tests for metadata event scope helpers * extract pathIsEqualOrUnder into util.IsEqualOrUnder Identical implementations existed in both replication/replicator.go and command/filer_sync.go. Move to util.IsEqualOrUnder (alongside the existing FullPath.IsUnder) and remove the duplicates. * use MetadataEventTargetDirectory for new-side directory in filer sync The new-side directory checks and sourceNewKey computation used message.NewParentPath directly. If NewParentPath were empty (legacy events, older filer versions during rolling upgrades), sourceNewKey would be wrong (/filename instead of /dir/filename) and the UpdateEntry parent path rewrite would panic on slice bounds. Derive targetDir once from MetadataEventTargetDirectory, which falls back to resp.Directory when NewParentPath is empty, and use it consistently for all new-side checks and the sink parent path. |
||
|
Chris Lu
|
2eaf98a7a2 |
Use Unix sockets for gRPC in mini mode (#8856)
* Use Unix sockets for gRPC between co-located services in mini mode In `weed mini`, all services run in one process. Previously, inter-service gRPC traffic (volume↔master, filer↔master, S3↔filer, worker↔admin, etc.) went through TCP loopback. This adds a gRPC Unix socket registry in the pb package: mini mode registers a socket path per gRPC port at startup, each gRPC server additionally listens on its socket, and GrpcDial transparently routes to the socket via WithContextDialer when a match is found. Standalone commands (weed master, weed filer, etc.) are unaffected since no sockets are registered. TCP listeners are kept for external clients. * Handle Serve error and clean up socket file in ServeGrpcOnLocalSocket Log non-expected errors from grpcServer.Serve (ignoring grpc.ErrServerStopped) and always remove the Unix socket file when Serve returns, ensuring cleanup on Stop/GracefulStop. |
||
|
Chris Lu
|
7d426d2a56 |
Retry uploader on volume full (#8853)
* retry uploader on volume full * drop unused upload retry helper |
||
|
Chris Lu
|
955a011f39 |
fix large_disk container rust build (#8839)
* fix docker rust build inputs for large disk image * simplify rust builder weed copy |
||
|
Chris Lu
|
d074830016 |
fix(worker): pass compaction revision and file sizes in EC volume copy (#8835)
* fix(worker): pass compaction revision and file sizes in EC volume copy The worker EC task was sending CopyFile requests without the current compaction revision (defaulting to 0) and with StopOffset set to math.MaxInt64. After a vacuum compaction this caused the volume server to reject the copy or return stale data. Read the volume file status first and forward the compaction revision and actual file sizes so the copy is consistent with the compacted volume. * propagate erasure coding task context * fix(worker): validate volume file status and detect short copies Reject zero dat file size from ReadVolumeFileStatus — a zero-sized snapshot would produce 0-byte copies and broken EC shards. After streaming, verify totalBytes matches the expected stopOffset and return an error on short copies instead of logging success. * fix(worker): reject zero idx file size in volume status validation A non-empty dat with zero idx indicates an empty or corrupt volume. Without this guard, copyFileFromSource gets stopOffset=0, produces a 0-byte .idx, passes the short-copy check, and generateEcShardsLocally runs against a volume with no index. * fix fake plugin volume file status * fix plugin volume balance test fixtures |
||
|
Chris Lu
|
e3359badfc |
fix unsupported platform container builds (#8838)
* ci: reintroduce Trivy report and gate workflow * ci: add dry-run mode to container release workflow * docker: fix unsupported platform builds |
||
|
Chris Lu
|
44fea49816 |
docker: use alpine packages for Rust builder to fix linux/386 builds (#8837)
The upstream rust:alpine manifest list no longer includes linux/386, breaking multi-platform builds. Switch the Rust volume server builder stage to alpine:3.23 and install Rust toolchain via apk instead. Also adds openssl-dev which is needed for the build. |
||
|
Chris Lu
|
e5ad5e8d4a |
fix(filer): apply default disk type after location-prefix resolution in gRPC AssignVolume (#8836)
* fix(filer): apply default disk type after location-prefix resolution in gRPC AssignVolume The gRPC AssignVolume path was applying the filer's default DiskType to the request before calling detectStorageOption. This caused the default to shadow any disk type configured via a filer location-prefix rule, diverging from the HTTP write path which applies the default only when no rule matches. Extract resolveAssignStorageOption to apply the filer default disk type after detectStorageOption, so location-prefix rules take precedence. * fix(filer): apply default disk type after location-prefix resolution in TUS upload path Same class of bug as the gRPC AssignVolume fix: the TUS tusWriteData handler called detectStorageOption0 but never applied the filer's default DiskType when no location-prefix rule matched. This made TUS uploads ignore the -disk flag entirely. |
||
|
Chris Lu
|
0761be58d3 |
fix(s3): preserve explicit directory markers during empty folder cleanup (#8831)
* fix(s3): preserve explicit directory markers during empty folder cleanup PR #8292 switched empty-folder cleanup from per-folder implicit checks to bucket-level policy, inadvertently dropping the check that preserved explicitly created directories (e.g., PUT /bucket/folder/). This caused user-created folders to be deleted when their last file was removed. Add IsDirectoryKeyObject check in executeCleanup to skip folders that have a MIME type set, matching the canonical pattern used throughout the S3 listing and delete handlers. * fix: handle ErrNotFound in IsDirectoryKeyObject for race safety Entry may be deleted between the emptiness check and the directory marker lookup. Treat not-found as false rather than propagating the error, avoiding unnecessary error logging in the cleanup path. * refactor: consolidate directory marker tests and tidy error handling - Combine two separate test functions into a table-driven test - Nest ErrNotFound check inside the err != nil block |
||
|
Chris Lu
|
937a168d34 |
notification.kafka: add SASL authentication and TLS support (#8832)
* notification.kafka: add SASL authentication and TLS support (#8827) Wire sarama SASL (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512) and TLS configuration into the Kafka notification producer and consumer, enabling connections to secured Kafka clusters. * notification.kafka: validate mTLS config * kafka notification: validate partial mTLS config, replace panics with errors - Reject when only one of tls_client_cert/tls_client_key is provided - Replace three panic() calls in KafkaInput.initialize with returned errors * kafka notification: enforce minimum TLS 1.2 for Kafka connections |