Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-10-24 15:00:43 +02:00
Code Issues Packages Projects Releases Wiki Activity
15,701 Commits 223 Branches 362 Tags
8541f959a93da49380e19237fd09ad2dbfd122fd
Commit Graph
15701 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Chris LuandDevin 8541f959a9 filer.conf: merge fsync on presence like worm (#11953) 
* filer.conf: merge fsync on presence like worm

fsync is a pure preference, so OR-merging it meant a nested rule could
never turn a parent's fsync off. Make PathConf.fsync optional and merge
it most-specific-wins, matching worm: an unset rule inherits, an explicit
value overrides in either direction. readOnly and disableChunkDeletion
stay OR'ed — they are fail-closed locks quota enforcement depends on.

Pre-version-2 configs were written with EmitUnpopulated, so every rule
carries an explicit fsync=false that meant nothing; it is dropped to
unset on load so it does not read as an opt-out.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.conf: test fsync override survives a save/reload round trip

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 20:45:46 -07:00
Chris LuandDevin 673d1a02e0 operation: euthanize pooled upload conns that stall far below a fast host (#12003) 
* operation: euthanize pooled upload conns that stall far below a fast host

A pooled keep-alive conn can wedge and drain the request body at a trickle
while fresh conns to the same volume run at full rate; every later upload
routed onto it stalls for tens of seconds, indefinitely. Track per-host
healthy-rate uploads (>=1MiB/s on a >=256KiB body) and, on a host that was
fast within the last minute, cancel an upload still in flight past
max(10s, size/256KiB/s): the transport discards the wedged conn and the
existing retry dials a fresh one. A watchdog kill also clears the fast mark
so the retried attempt runs unbounded, and hosts that never prove fast keep
today's unbounded behavior - a chronically slow link is never penalized.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* operation: keep the stall watchdog armed through the response read

uploadDone was set as soon as Do() returned, so a conn that accepted the
request body then wedged on the reply escaped the watchdog and could
park the upload for hours — the same failure this watchdog exists to
kill. Deferring the done-store keeps the timer live until the function
returns, so a stalled io.ReadAll is cancelled and retried like a
stalled write. The redundant post_err recheck goes with it.

* operation: pin watchdog coverage of a stalled response read

* operation: disarm the stall watchdog for an upload's remaining attempts

Once the watchdog has cancelled an attempt, a concurrent healthy upload
to the same host can restore the fast mark during the retry backoff.
The re-armed watchdog would then cancel the retry too, and a large
upload that merely exceeds the floor could burn every attempt. Track
the watchdog kill on uploadStatusError and carry a
stallWatchdogDisarmed flag through retriedUploadData so the rest of
that upload's attempts run unbounded; other uploads to the host keep
their watchdog.

* operation: pin watchdog disarm on retry, fresh-dial, and slow-upload cases

* operation: drop the send-stall watchdog once the request body drains

A wedged pooled conn is only blameworthy while it is still trickling the
request body; once the body is on the wire the wait is server-side commit,
which the transport's response header timeout already bounds. Cancelling
then would fail a healthy write whose replication or fsync is simply slow.
Track body EOF through a wrapping reader, gate the send-side timer on it,
and re-arm the watchdog only around the response-body read, which the
header timeout does not cover.

* operation: prove a host fast only on a verified upload

A quick rejection moves bytes no faster than the wire did, so mark the
host from the body-send duration and only after the response checks out.

* operation: count an abandoned request body as sent

The transport closes a request body it never finished draining when the
server answers early, so EOF alone under-reports the send boundary and
the stall watchdog could still blame the conn during the reply wait.

* operation: scope the body-sent mark to the current attempt

A lagging Close or EOF from the first attempt's body could stamp the
shared mark after the inner retry reset it, disarming the watchdog for a
send that never happened; each attempt now points the watchdog at its
own sent timestamp.

* operation: drop only the fast mark the stall timer observed

A concurrent upload can re-prove the host while a stalled attempt is in
flight; deleting the key outright would erase that newer mark, so the
watchdog now removes only the value it armed against.

* operation: keep the stall disarm on a per-call upload option

retriedUploadData marked the caller's UploadOption, so a caller reusing it
across UploadData calls would start the next upload with the watchdog off
and lose the cancellation this adds.

* test: stall watchdog retry must fail if the conn gets cancelled

The fake retry slept through any cancellation, so a re-armed watchdog
firing mid-retry still let the upload succeed — the guard was untested.
Park on the request context like a real wedged conn so a re-armed
watchdog cancels the attempt and fails the test.

* operation: prove the fast-host mark with elapsed-to-response

Body-drain time only measures when the transport consumed the reader:
kernel socket buffers can swallow a small body instantly on a slow link,
inflating the rate and marking the host fast. The next upload then gets
its conn cancelled mid-send despite making progress. Elapsed-to-response
bounds remote speed from below — the server cannot answer before the
body lands — so the mark can only under-mark, never over-mark.

* operation: scale the stall budget off the host's measured rate

Fixed bps limits fit no link well: 256KiB/s never fires on a slow link
and waits far too long on a fast one. Record the wire rate of each
sizeable upload and let a later upload run for slowFactor times the
body's span at that rate — a wedged conn shows as an order-of-magnitude
drop on any link speed. The recorded rate uses the wire body size so a
pre-compressed payload no longer inflates the mark.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 20:34:31 -07:00
Chris LuandDevin 6e8a2cdb14 filer.backup: process metadata events through the concurrent processor (#11951) 
* filer.backup: process metadata events through the concurrent processor

Each event was handled inline on the subscription loop, so one
fetch+write at a time capped the stream at a few entries per second and
a busy source kept building lag. Route events through the same
MetadataProcessor filer.sync uses: it fans jobs out behind -concurrency
while serializing same-path and subtree-barrier conflicts, and the
checkpoint now advances on the processor's contiguous watermark via
GetResumeTsNs, with Resubscribe replaying anything a failure pins.
-disableErrorRetry keeps its log-and-skip meaning by returning nil after
logging, since the processor would otherwise pin the checkpoint forever.

* filer.backup: drain in-flight jobs before the stream retry loop reconnects

FollowMetadata returns as soon as the stream ends while admitted jobs keep writing; the next doFilerBackup builds a fresh processor that cannot order its replayed writes against them, so an older upload could finish last and clobber a newer file.

* filer.backup: checkpoint the completed watermark on a ticker and at stream end

The offset callback only fires while events arrive, so work draining on a quiet stream stayed uncheckpointed until the next event and a restart recopied it. Persist the processed watermark every 3s when it advances, and once more after the post-stream drain.

* filer.backup: bound the checkpoint wait before reconnecting

setOffset has no context, so a stalled KvPut would hold persistDone and block the retry loop forever. The persist goroutine now performs the final write itself on stopPersist, and the caller waits at most 30s.

* filer.backup: carry the processed position across retry attempts

A follow attempt that ended with a pinned failure recomputed startFrom on retry, so a -timeAgo window could move past the failed event and skip it. doFilerBackup now returns the failure pin (or processed watermark) and the retry loop clamps startFrom to it.

* filer.backup: resume retries at the previous attempt's pinned position

The retry loop threaded resumeTsNs through but never consulted it, so a
-timeAgo run recomputed startFrom as now-minus-the-window on every attempt
— a failed event older than the window was skipped on reconnect. Use the
pinned position when one exists; -timeAgo and the persisted checkpoint
only choose the first starting point.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: resume below the failure pin so the failed event replays

The metadata reader treats the resume position as exclusive — events at
or below it are skipped — so resuming at the failed event's own timestamp
would skip it. Back one tick off the pin.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: label the progress line as the received-event rate

The counter increments when a job is admitted, not when its copy
finishes, so the old wording showed backup throughput during a burst of
slow copies while the processed watermark had not moved.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: bound the offset checkpoint RPC

setOffset called KvPut on the background context, so a stalled write
could pin a caller's drain or persistence loop indefinitely.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: keep the resume pin when filer-source setup fails

A retried attempt carries resumeTsNs so the failed event replays from the
same position, but filer-source and SSE initialization failures returned 0,
so the next attempt recomputed -timeAgo and could move the window past the
pinned event.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: serialize checkpoint writes per key

A checkpoint write can outlive its caller's wait and land beside the next
subscription attempt's writes, moving saved progress backward. Serialize
writes per key inside setOffset and drop any value older than one already
written from this process.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: gofmt

* replication: serialize checkpoint writes per filer and key

A single process-wide lock made a stalled checkpoint RPC block every
stream's checkpoint, including the opposite direction of a bidirectional
filer.sync. Keeping the last-written watermark per persisted key also let
one stream's offset shadow another stream checkpointing the same key to
a different target filer. Lock and track progress per (filer, key).

* replication: keep the resume position behind a cleared failure pin

A redelivery that clears failedTs lets the processed watermark advance
past the earliest failure, but events dropped while the processor was
stopped — possibly sharing the failure's timestamp — have to replay.
Latched earliestFailedTsNs floors the resume position one tick below
that failure so reconnects still cover them.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: persist floored checkpoints through the write ordering

setOffset drops a value older than the last written, which also swallows
a deliberate regression: the resume floor, an operator-seeded offset, or
a snapshot high-water mark. allowRegress marks those callers so the
checkpoint can move backward when a failure forces replay.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: order checkpoint writes by issuance, not value

The value-based stale guard could never fire — every caller passed
allowRegress because floors and seeds are deliberate regressions — and
ordering by value cannot tell a stale high watermark apart from real
progress: a write that outlives its attempt could still land after a
floored resume and skip events owed a replay. Order by issuance instead:
a write issued before the last one to land under the key is dropped, so
late writes from a superseded attempt can no longer move the checkpoint
in either direction while deliberate regressions still apply.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: keep the resume position one tick below the frontier

* remote.sync: resume and persist through the floored watermark

* test: concurrent metadata jobs fan out up to the limit

Distinct-path events must all be in flight before any finishes — a
serial loop could only ever have one running — while the limit still
bounds the pool. Pins the property behind the backup-throughput fix.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 20:02:55 -07:00
github-actions[bot] cd562b8514 docs: regenerate star history chart 2026-10-24 02:56:42 +00:00
Chris LuandDevin 617599859b docker: build RocksDB v11.1.2 for grocksdb v1.11.1 (#12002) 
The rocksdb release image fails to compile: grocksdb v1.11.1 calls
C.rocksdb_block_based_options_set_block_align,
C.rocksdb_block_based_options_set_index_block_search_type, and
C.rocksdb_block_based_options_set_separate_key_value_in_data_block, all
of which only exist in RocksDB v11.x headers. grocksdb v1.11.1's own
build script pins 11.1.2, so move every default from v10.10.1 to
v11.1.2 to match.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 18:50:27 -07:00
Chris Lu ce4c1fe31e ci: install fuse3 when runners lack fusermount3 (#12013) 
Self-hosted runners do not ship fuse3, so the setuid repair step
failed with 'no fusermount3 in PATH' before the FUSE jobs could run.
Install the package instead of failing outright.
 2026-10-23 18:49:04 -07:00
Chris Lu 462e4445e2 test: skip catalog client images that cannot execute on the host CPU (#12014) 
New upstream releases build for x86-64-v3 (AVX2) - ClickHouse since
26.5, plus the DuckDB iceberg extension and pylance wheels - so on a
runner whose CPU lacks AVX2 the container dies on SIGILL before the
test reaches any SeaweedFS code. Probe each third-party image first
and skip on the illegal-instruction exit instead of failing; other
errors still surface normally.
 2026-10-23 18:46:26 -07:00
Chris Lu 0d4e11a6e5 filer: bootstrap a timestamp-less peer entry onto a missing local path (#11975) 
A zero local commit time conflated 'no local state' with 'state at the
epoch', so a missing entry was skipped whenever the sender reported no
commit time and the entry mtime was at or before the epoch. Only hold
the copy off when a local entry actually exists.
 2026-10-23 18:45:09 -07:00
Chris Lu 85f2f9d2d4 ci: make docker registry and compose steps idempotent on self-hosted runners (#12012) 
The registry container is started with --restart=always but never
removed during teardown, so once a machine runs the job, every later
run on that machine fails with a container name conflict. Reuse the
running container when it exists and recreate it only when missing or
broken.

Some self-hosted runners (e.g. seaweedfs-server03) also lack the
Docker Compose plugin, so install it via docker/setup-compose-action,
which is a no-op where compose is already available.
 2026-10-23 18:37:36 -07:00
Chris LuandDevin 679835e4fc volume.fsck: run all filer-entry purges through the shared worker pool (#11960) 
* volume.fsck: queue per-volume orphan deletes through the shared purge pool

The worker pool added for -reallyDeleteFilerEntries only covered entries
whose volume was missing entirely; the per-volume check that finds filer
entries pointing at absent needles still issued one synchronous HTTP
DELETE per orphan, which is the bulk of the work the report measured.
One pool now serves both purge phases and is drained before the empty-
directory sweep.

* volume.fsck: keep hard link ids in the filer fid file

Orphans queued by the per-volume check must hash into the pool by their
hard link id just like the missing-volume path, so two paths sharing an
id still serialize on one worker. The fid file now stores the id beside
each path and readFilerFileIdFile hands it to the caller.

* volume.fsck: cover orphan purge queuing and hard link id round-trip

* volume.fsck: fail on a truncated hard link id in the fid file

A .fid record cut short inside the hard link id length was emitted with a
nil id, so its purge could hash to a different worker than the entries it
shares the link with. Read the length strictly like the header and path.

* volume.fsck: drain the save consumer before returning a root genFn error

The early return left the save goroutine blocked on outputChan while the
caller ran finishPurge, so a late queuePurge could send on a closed
channel and panic.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* storage: skip EC shard-size encoding tests when disk space is low

TestCalculateExpectedShardSizeWithRealEncoding writes a .dat file plus
14 EC shards per subtest (~1.3GB peak for the 512MB case) into the temp
dir. Shared CI runners can run low on space while other test packages
write to /tmp in parallel, and the encode then dies with ENOSPC
mid-write, surfacing as a test failure. Check free space up front and
skip the subtest instead.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 18:33:51 -07:00
Mohd Quamar TyagiDevinchrislusf
 		3810fb1123 s3: url-encode ListMultipartUploads results for encoding-type=url (#11998) 
* s3: url-encode ListMultipartUploads results for encoding-type=url

The response echoed EncodingType=url but returned raw upload keys, and
XML 1.0 cannot carry control characters, so the encoder replaced them
with U+FFFD. The listed key then matched no upload: AbortMultipartUpload
with it answered 404 NoSuchUpload, leaving such uploads uncleanable.

Encode the key-named elements AWS encodes for this API (Key, Prefix,
Delimiter, KeyMarker, NextKeyMarker) the way ListObjects encodes its
keys when encoding-type=url is requested. Upload ids stay opaque, and
responses without encoding-type=url are unchanged.

Fixes #11991

* s3: echo EncodingType only when encoding-type=url is honored

A request with an unrecognized encoding-type echoed the value back in
<EncodingType> while the keys stayed raw, matching neither AWS (which
omits the element unless url encoding applies) nor the ListObjects
handlers, which gate the echo on the parsed url flag.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 18:21:26 -07:00
Chris LuandDevin 12d0537215 test: gofmt s3tables test files (#12011) 
Unformatted test files leave the S3 Tables Format Check red on every
branch until they are re-formatted.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 18:18:32 -07:00
Chris LuandDevin b2817e853d test: raise mini-server startup wait from 30s to 90s in Makefiles (#12005) 
* test: raise mini-server startup wait from 30s to 90s in Makefiles

Self-hosted/persistent runners under load regularly need >30s for
weed mini (master+volume+filer+s3+iceberg+lance+webdav+worker) to
become healthy; hosted runners start in <10s. Siblings already wait
90s (s3/checksum, s3/cors). Fixes the S3 Filer Group first-run flake
on busy self-hosted runners.

* test: align Makefile startup guards and progress totals with 90s bound

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 18:15:01 -07:00
Chris Lu 1e0f571e75 ci: set up Go for CodeQL autobuild (#12007) 
The workflow relied on a Go toolchain preinstalled on the runner
image; self-hosted runners without setup-go fail at autobuild with
'go: executable file not found'.
 2026-10-23 17:52:50 -07:00
Chris Lu baf0dd680e ec.encode: require shard placement to meet -shardReplicaPlacement (#11947) 
* ec: measure shard placement against replica-placement caps

ecShardPlacementViolations reports where a volume's newest-generation
shard placement exceeds the per-node and per-rack caps a
-shardReplicaPlacement requests, using the same digit mapping the
balance planner enforces (SameRackCount = shards/node, DiffRackCount =
shards/rack, dc-qualified rack keys).

* ec.encode: -strictPlacement keeps originals when placement breaches caps

-shardReplicaPlacement tells the balancer the requested per-node and
per-rack shard limits, but nothing verifies the outcome. With
-strictPlacement the pre-delete shard verification also measures the
final placement against those caps: a volume still over the node or
rack limit aborts before its source is deleted, so the encode only
completes when the requested failure-domain spread is actually met.

* ec.encode: cover the strict-placement delete gate with a test

The gate in verifyEcShardsBeforeDelete now runs against a stubbed
topology: a compliant spread passes while an over-cap placement is
refused before source deletion. The retry knobs become package vars so
the test does not wait out the heartbeat-propagation backoff.

* ec.encode: enforce -shardReplicaPlacement without a flag

Drop the -strictPlacement option: the requested per-node and per-rack
caps are always requirements at the pre-delete verification, so the
encode only completes when the placement actually honors them.
 2026-10-23 17:29:36 -07:00
Jay HemnanichrislusfDevin
 		f5e319ae8e iam: GetUserPolicy returns NoSuchEntity for a policy name the user does not have (#11944) 
* iam: GetUserPolicy returns NoSuchEntity for a policy name the user does not have

When the user has stored inline policies, an unknown name fell through to the
policy rebuilt from the user's actions, so a deleted policy still read back.
The standalone IAM server already returns NoSuchEntity here.

* iam: return service failure when listing inline policies fails

A ListUserInlinePolicies error in GetUserPolicy fell through to
action-based reconstruction, masking a store failure as a policy
document. Propagate it as ServiceFailureException instead, matching
the standalone iam server and the other callers in this file.

* iam: return service failure when the inline policy store read fails

---------

Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 17:08:49 -07:00
Chris Lu f8579bf728 s3: gate POST Object acl fields behind PutObjectAcl (#11939) 
* s3: gate POST Object acl fields behind PutObjectAcl

The POST Object handler promotes acl and x-amz-grant-* form fields onto
the upload's ACL headers, and resolveFileMode honors the promoted canned
ACL. The PUT path only applies such headers after an s3:PutObjectAcl
grant; mirror that on POST by running the shared per-key authorization
with the WriteAcp action whenever the request carries ACL headers.

* s3: cover the POST Object acl gate

Signed POST requests carrying an acl form field are denied for a writer
without the WriteAcp action and for a writer a bucket policy denies
s3:PutObjectAcl on the target key; the authorization boundary is
exercised directly as well.

* s3: cover x-amz-grant-* POST fields and the no-ACL case

* s3: encode POST policy conditions as JSON in the acl-gate test
 2026-10-23 17:05:44 -07:00
Chris LuandDevin 168789d5f3 master: cap writable volumes per collection via -maxWritableVolumes (#11923) 
* master: cap writable volumes per collection via -maxWritableVolumes

Some deployments serialize disk writes (e.g. an Unraid array where
concurrent writes to multiple disks collapse throughput). With more
than one writable volume in a collection, the master's weighted pick
spreads assignments over all of them, so several disks get written at
once.

-maxWritableVolumes=N narrows the pick candidates to the N fullest
writable volumes, so a collection's writes concentrate on one (or few)
volumes at a time. 0 keeps the current spread-everywhere behavior.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: give master.follower a maxWritableVolumes default

* master: cap writable volumes inside the requested location

* master: pick the capped volumes without sorting the writable list

* master: assert the capped candidate set instead of random pick outcomes

* master: describe -maxWritableVolumes as a per-layout cap

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 17:04:36 -07:00
Chris Lu 8b26da8c53 ci: gate go/java/python tool caches to github-hosted runners (#12001) 
Follow-up to #11985/#11986: setup-go/setup-java/setup-python steps
added or restructured since (e.g. by #11981's shared-artifact layout)
were missed. On self-hosted runners setup-go restores/saves a ~5GB
module cache archive per job — pure waste when the module cache is
already on local disk — and the restore blocks the job for minutes.
 2026-10-23 16:25:23 -07:00
Chris Lu cd97c93deb command: keep filer, s3, and sftp up when the local unix socket cannot bind (#11999) 
* pb: share the stale-socket tolerant unix listener

ServeGrpcOnLocalSocket already treats the local socket as optional; pull the remove-and-listen block out so the filer, s3, and sftp http sockets can use it too.

* command: keep filer, s3, and sftp up when the local socket cannot bind

A stale socket file owned by another uid (e.g. left in /tmp by a containerized job) made os.Remove fail with EPERM and the Fatalf took the whole server down. The gRPC local socket already treats the unix listener as optional; do the same for the http/sftp sockets.

* filer: clear shared localSocket flag when the unix socket cannot bind

s3.localFilerSocket and (in weed server) the sftp local socket share the
filer's -localSocket flag pointer. If the filer cannot bind the socket it
now keeps running without it, so consumers still holding the configured
path would dial a dead or foreign-owned socket.

* command: keep sftp on its own localSocket flag under weed server

sftpOptions.localSocket was overwritten with the filer's -localSocket
flag, so -sftp.localSocket was ignored and an explicit filer socket path
made the sftp server try to bind the same socket the filer already held.

* s3: fall back to TCP when the filer local socket cannot be dialed

The embedded S3 server copies the filer's -localSocket flag before the filer tries to bind it, so a socket that never comes up left every filer HTTP request dialing a dead path. Try the unix socket first and fall back to the request's TCP address on dial failure.

* filer: keep the shared localSocket flag read-only after startup

Clearing *fo.localSocket raced with the embedded s3 goroutine reading the same flag pointer and could not help a reader that had already copied it. The s3 filer client's TCP fallback covers the unbound-socket case without mutating shared flag state.
 2026-10-23 16:07:32 -07:00
Chris Lu 6c87fb70f9 server: skip local unix sockets that cannot be bound (#12000) 
server: skip default local unix sockets that cannot be bound

weed server fatally exited during S3, filer, or SFTP startup when the default /tmp/seaweedfs-<svc>-<port>.sock path could not be removed or bound. On shared hosts (e.g. CI runners where another user's weed left a socket in the sticky /tmp directory) a stale socket file turns into 'operation not permitted' and kills the whole server, even though the TCP listeners are unaffected. Warn and skip the unix socket listener instead, matching how the gRPC local socket already degrades. An explicitly configured -localSocket path still fails fast, since a co-located S3 server may be configured to dial the filer through it.
 2026-10-23 16:02:33 -07:00
Chris Lu 29b81a827d volume: also listen on localhost like other components (#11995) 
* volume: also listen on localhost like other components

Master, filer and s3 bind both the detected host ip and localhost via
NewIpAndLocalListeners, but the volume server used a plain NewListener on
bindIp, so a volume bound to the host ip never answers on loopback. Any
tooling that polls localhost (e.g. the java test workflow's readiness
curl on localhost:8080/status) then waits out its full timeout even
though the volume is up and registered with the master.

Use NewIpAndLocalListeners for the volume grpc, public and cluster http
listeners, and add a small multiHttpdownServer composite so graceful
shutdown still covers both listeners.

* volume: stop and wait dual listeners concurrently

A serial multiHttpdownServer.Stop let the localhost listener keep
accepting requests while the bound-ip listener drained, and a serial
Wait hid a localhost serving failure until the bound-ip listener ended.
Run both waits and both stops in parallel: Wait reports the first error
promptly, Stop closes every listener when shutdown begins.
 2026-10-23 15:44:18 -07:00
Chris Lu aeb5db1b13 tests: pin weed mini admin ports and disable unused aux services (#11997) 
* tests: pin weed mini admin ports and disable unused aux services

Every weed mini that does not configure them still binds the fixed
default ports for the admin server (12535 + derived 22535 gRPC), its
legacy compat listener (23646 + 33646), WebDAV (7333), Lance (9101) and
the Iceberg REST catalog (8181). Sequential or concurrent minis on one
host then collide on them: the port check warns and shifts, and the
admin's late bind can land on a stolen ephemeral port, adding noise and
flakiness to test logs (e.g. run 40017181958, job 120659267479).

Add testutil.MiniAdminArgs to pin the admin server to an allocated
mini-style port (HTTP + port+10000 gRPC, both below the Linux ephemeral
port floor since admin binds late) and disable the legacy compat
listener. Every test that spawns weed mini now also disables the aux
services it does not use: WebDAV, Lance and the Iceberg REST endpoint
get port=0 unless the test actually exercises them, in which case their
ports stay pinned as before.

* tests: take mini admin ports from the below-ephemeral range

* tests: reuse MiniAdminArgs for the remote-s3 fetch mini

* tests: fully disable webdav and co-allocate mini admin ports

- -webdav.port=0 only makes the service bind a random port; add
  -webdav=false so minis that do not test WebDAV skip it entirely.
- Spark, RisingWave and s3 Spark launchers allocated the admin port in
  a second MustFreeMiniPort call, which could pick a port already
  handed out by the first batch. Fold 'Admin' into the same
  MustFreeMiniPorts call.
- test/fuse_integration is a separate Go module and cannot import
  testutil; allocate the admin ports with its local freePort helper
  instead.

* test/fuse: derive the mini admin gRPC port from its HTTP port

Two independent freePort calls could return the same port, making admin.port equal admin.port.grpc (or collide with the filer port). Allocate once and derive the gRPC port with the +10000 convention.

* test/fuse: pin the mini admin port in the fuse_test framework copy

The fuse_test package framework spawns weed mini too; without a pinned admin port it still binds the fixed defaults 12535/23646 and collides with other minis.

* tests: drop the redundant -webdav.port=0 flag

-webdav=false alone disables the service; the port flag is dead config.
 2026-10-23 15:38:09 -07:00
Chris Lu c925869dcc shell: retry filer discovery until a filer registers (#11994) 
* shell: retry filer discovery until a filer registers

When -filer is not given, the shell discovers a filer with a single
ListClusterNodes call right after connecting to the master. A filer can
take several seconds to register after its port reports ready, so the
list routinely comes back empty and the shell proceeds with an empty
FilerAddress. Commands then fail with "dial tcp: missing address", which
is how the s3tables Dremio integration test flaked in CI.

Treat an empty list as retryable and keep polling ListClusterNodes until
a filer appears or -connectTimeout expires. On timeout, print a clear
"no filer registered" notice instead of failing later inside a command;
master-only shell commands keep working either way.

* shell: report the last filer discovery attempt failure

A successful empty ListClusterNodes followed by a master outage left the
diagnostic claiming no filer registered, hiding the real RPC failure.
Track the outcome of the latest attempt instead: if the final list call
failed, report that error; only when the last response was genuinely
empty report that no filer registered within the timeout.

* shell: test filer discovery retries against a fake master

Extract the discovery loop into listFilers and cover the retry behavior
with a fake master_pb.SeaweedServer: an empty cluster turning populated,
an empty cluster exhausting the window without error, the last attempt's
RPC failure being reported, and master-only commands still running after
the discovery timeout.
 2026-10-23 15:21:41 -07:00
Chris LuDevinchrislusf
 		34f06ee92f test/fuse: fix cluster startup in the fuse test framework (#11977) 
* test/fuse: fix cluster startup in the fuse test framework

weed volume Fatalfs when -dir does not exist, so every cluster setup
failed its volume wait after 30s. The mount also exited on the unknown
-cacheSizeMB flag (it is -cacheCapacityMB now).

* test/fuse: manage cluster processes and allocate free service ports

Each service now runs through a managed process that captures its log and
reports an early exit from the port wait, instead of a bare 30s timeout.
Ports are allocated free pairs (http + grpc sibling) rather than fixed
19333/18080/18888 so the framework no longer collides with weed mini's
fixed ports or another framework instance.

* test/fuse: bail out of the mount wait when the mount process exits early

The port waits already report an early exit with the captured log tail; the
mount wait still polled the device id for the full timeout, so a mount that
died at startup (bad flag, missing filer) surfaced as a bare 30s timeout
with no diagnosis. Check the managed process the same way and include its
log tail.

* test/fuse: deduplicate service ports handed out by freeServicePort

A probed port is released before the service binds it, so the three
sequential allocations for master, volume and filer could all return the
same still-unbound port and leave later services dying at startup. Track
handed-out ports so each allocation is unique.

* test/fuse: allocate service ports in Setup after the external-mount branch

* test/fuse: verify service readiness through its own /healthz endpoint

* test/fuse: bound the readiness probe so a silent listener cannot hang setup

* test/fuse: only accept a ready response while the service process still runs

* test/fuse: include the service log tail on readiness timeout

A startup that stalls instead of exiting still failed with a bare
"not ready within timeout" — the same opaque message that made the
original flake hard to diagnose. The process-exit path already tails
the captured service log; the timeout path now does too.

* test/fuse: report pjdfstest category results without failing the suite

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: allow allow_other mounts in posix-compliance

The job mounts with -allowOthers but never sets user_allow_other in
/etc/fuse.conf, so fusermount rejects the option and the mount exits
before the suite runs. The other FUSE workflows set it; do the same.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: keep the standalone-framework service logs on failure

Cleanup removed the temp dir holding master/volume/filer/mount logs, so
a CI failure left nothing to diagnose. Dump the tails on failure and
copy them under /tmp/seaweedfs-fuse-logs/standalone/ for the artifact
upload, next to the mini framework's logs.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: gate setup on a real write through the mount

Mount readiness only proves the daemon is serving; a fresh cluster still
cannot assign a file id until a volume heartbeats in and a grow round
finishes, so the first ~10-30s of writes surface as EIO on close. Probe
with a real write/read/remove before handing the mount to the suite.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: gate the mini framework setup on a real write too

Same cold-start window as the standalone framework: mini's volume has to
register and grow before a write can be assigned. Keep the two
frameworks consistent so later suites do not flake on the same stall.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: save each framework's CI logs under its test name

Fixed-name mini.log/mount.log in /tmp/seaweedfs-fuse-logs let a later
framework instance overwrite an earlier test's logs, so an uploaded
artifact could describe a passing run instead of the failure under
investigation. Apply to both framework copies since they share the
directory.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: bound each waitForWritable probe attempt

A wedged mount can hold a file op open past the 90s deadline, so the
loop could not check the deadline or a mount exit and Setup hung until
the overall test timeout. Each attempt now runs off the loop and is
abandoned when the remaining time runs out. Same fix in both framework
copies.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: keep one write probe in flight until it finishes or the deadline

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@seaweedfs.com>
 2026-10-23 08:10:31 -07:00
Chris Lu 2c8079625c test: stop EC integration tests from pinning default master port 9333 (#11988) 
The EC integration tests failed in CI with "raft.Server: Not current
leader" on every assign for tens of seconds — longer than any goraft
election takes. A stale weed process on the self-hosted runner was
still holding 9333, so each test's fresh master died at bind time and
the tests kept talking to a raft non-leader. Only the 9333 cluster
(startSeaweedFSCluster) failed; tests on 9334-9338 all passed.

Allocate mini-safe ports (http + grpc+10000 both verified free) for the
master and volume servers instead of hardcoding 9333/8080-8085, wait
for the master to actually report raft leadership via /cluster/status
before issuing raft-gated RPCs, and make waitForServer a real TCP dial —
grpc.NewClient never connects, so the old check always passed
instantly.
 2026-10-23 08:08:49 -07:00
bati-valensasandChris Lu c4002fafc7 helm: fix bucket hook treating an existing bucket as missing (#11982) 
* helm: fix bucket hook treating an existing bucket as missing

The skip check pipes awk into `grep -q`, which exits on the first match and
can leave awk writing into a closed pipe. awk then dies of SIGPIPE and
`set -o pipefail` makes the pipeline report that, so a bucket that exists
reads as missing: the hook runs `s3.bucket.create`, it errors with "already
exists", and `sh -ec` aborts the hook - failing the release.

Match in a single awk instead, which reads to EOF and cannot be killed by a
closing reader.

* helm: force string compare in bucket hook existence check

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
 2026-10-23 06:43:35 -07:00
Chris LuandDevin 384523761b ci: skip actions/cache on self-hosted runners (#11986) 
cargo registry/target and docker-image caches are already local on a
persistent self-hosted runner; the save/restore pass is pure overhead
(the setup-go cache-save stall on awscli-tests was the same pattern).
Gate every actions/cache step on runner.environment == 'github-hosted'.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 06:19:56 -07:00
Chris LuandDevin 277c9809c8 ci: skip setup-go cache on self-hosted runners (#11985) 
* ci: skip setup-go cache on self-hosted runners

The module cache is already local on a persistent self-hosted runner;
archiving ~/go/pkg/mod into cache.tzst at every job's end is pure cost
and can stall (a ~4-minute zstdmt hang was observed on awscli-tests).
runner.environment gates the cache to GitHub-hosted runners only.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: gate the remaining setup-go caches too

The first pass only covered go-version-file steps; go-version steps in
nine workflows (including awscli-tests' own file) kept uploading the
cache archive on self-hosted runners.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 06:05:33 -07:00
Chris LuandDevin 728fc69976 ci: cache docker build layers for container_dev (#11980) 
* ci: cache docker build layers for container_dev

The Go-build image rebuilds its layers on every run; type=gha backend
caching reuses them across runs and platforms.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: pin container build to the pushed sha under layer caching

Dockerfile.go_build clones master inside the image; a cache-from layer
would otherwise keep serving the previous commit's clone and publish
stale code under the dev tag. Passing BRANCH=github.sha makes the clone
layer's key commit-specific while toolchain layers still hit.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: pass BRANCH via build-args so the per-commit cache key reaches buildx

docker/build-push-action has no `args` input — the BRANCH override was
silently dropped, leaving the build arg at its `master` default and the
clone/checkout layers serving a stale commit under the gha cache.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 06:02:08 -07:00
Chris LuandDevin 9b59bc6b0d ci: build test binaries once, share them across shard jobs (#11981) 
* ci: build test binaries once, share them across shard jobs

Each of the 6 rust-volume-go-tests shards, the rust-integration-tests
job, and the 6 volume-server-integration-tests shards rebuilt the same
weed and weed-volume binaries — 13 duplicate builds per run. A
build-test-binaries / build-weed job now compiles them once under the
existing cargo cache and publishes them as artifacts; the consumers
download and chmod them instead of building.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: preserve binary paths through the artifact round-trip

upload-artifact stores single files flat, so weed/weed would come back
at the workspace root and the consumers' expected paths break. Pack the
binaries in a tar before upload and extract on download.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: extract the shared test binaries in the rust go-test shards

The download step leaves rust-test-binaries.tar at the workspace root;
without untarring it the chmod fails on missing files and every shard
fails before running a test.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 04:50:29 -07:00
Chris LuandDevin f7af816ab4 ci: cancel superseded workflow runs on PR pushes (#11979) 
* ci: cancel superseded workflow runs on PR pushes

PR-triggered workflows without a concurrency group keep running (and
holding runner slots) after a force-push or quick follow-up commit
supersedes them. Add the same group/cancel-in-progress block the go
workflow already uses, keyed on head_ref, to the remaining PR workflows.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: key concurrency groups on PR number or ref, not head_ref

github.head_ref is empty on tag pushes and manual runs (every run then
shares one group and cancels the last) and collides across forks when
two PRs share a branch name. pull_request.number is unique per PR;
github.ref covers push/tag/dispatch. Applied to all workflows using the
pattern, including pre-existing ones.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: restrict cancel-in-progress to PR and branch-push runs

* ci: serialize container_latest manual runs by destination tag

Two workflow_dispatch runs sharing an image_tag could otherwise build
concurrently: their per-arch images publish to the same platform tags,
so one run's create-manifest job can combine platforms from different
source refs. Group dispatch runs by the destination tag (with
cancel-in-progress already disabled for non-PR/non-branch events) so
same-tag publishes queue instead of interleaving.

* ci: serialize manual publish runs by destination tag in remaining container workflows

* ci: group foundationdb release dispatch runs by release tag

A manual run now shares the tag-push concurrency group for the same
release, so a re-publish of a tag queues behind (or ahead of) the push
run instead of racing it on the same image tag.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: serialize foundationdb_version dispatch runs by destination tag

Two manual runs resolving to the same docker tag now share a concurrency
group (explicit image_tag, else the computed
foundationdb_<fdb>_seaweedfs_<ref> default), so they queue instead of
racing pushes to the same tag.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: drop unsupported toLower() from foundationdb_version concurrency group

* ci: key rocksdb_version dispatch group on its published tag

* ci: queue publish runs via turnstyle instead of cancel-free shared groups

* Revert "ci: queue publish runs via turnstyle instead of cancel-free shared groups"

This reverts commit 039148f8d5.

* ci: queue same-tag publish runs instead of replacing the pending one

Dispatch/tag-only workflows can never cancel in progress, so opt their
concurrency groups into queue: max — up to 100 same-tag publishes wait
in line rather than each new dispatch cancelling the pending one.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 04:49:52 -07:00
Chris Lu 7c478fd89a fuse integration: run posix suites on weed mini and allow allow_other mounts (#11984) 
The separate master/volume/filer processes in framework.go bound fixed
ports, discarded child output, and never noticed an early process exit,
so a slow or dead volume server burned the whole 30s timeout with no
logs. Switch to the same weed-mini approach the fuse_test package
framework already uses: dynamic filer ports, child output captured to
log files, and readiness waits that bail out the moment the process
dies. The external-mount path (TEST_MOUNT_POINT) is unchanged.

The posix-compliance workflow mounts with -allowOthers, but fusermount
refuses allow_other unless user_allow_other is set in /etc/fuse.conf,
mirroring the fuse-integration workflow FUSE setup.
 2026-10-23 04:33:14 -07:00
Javier de la TorreJavier de la TorreChris LuDevin
 		ad89f003a5 s3: end a catalog-vended session with the access key it was vended for (#11945) 
* s3: end a catalog-vended session with the access key it was vended for

A table credential the Iceberg catalog vends is a self-contained STS session:
it stays valid until it expires (an hour by default) even after the access key
that loaded the table is deleted or deactivated, or its identity is removed
or disabled. An operator revoking a leaked key cannot cut off the table
credentials that key already obtained.

Sessions are stateless JWTs, so there is no revocation list to consult. This
change follows the same pattern as role sessions, which are already refused
once their role's ID no longer matches (checkSessionRoleBinding): a vended
session carries the access key it was vended for (src_ak, recorded by the
previous change), and validateSTSSessionToken checks it against the identity
table held in memory. A session of provider iceberg-catalog that carries a
source key is refused (InvalidAccessKeyId, as for an unknown key) once its
identity is gone or disabled, or the key is no longer one of that identity's
active, unexpired credentials.

Scope:
- Only catalog-vended sessions are checked; web-identity, AssumeRole and
  every other STS session are validated as before.
- The vendor binds a session to the caller's key only when the caller is a
  stored identity holding that key. A caller that signed with another STS
  session's key, an OIDC user, or a request that recorded no key gets an
  unbound session, as today; binding it would refuse every request made with
  it. A session minted before this change carries no key and is unbound.
- The check is one map lookup under the read lock and a scan of one
  identity's keys (about 11 ns, 0 allocations, with 1,000 identities).

Tests (s3api_iceberg_vended_session_test.go) mint real sessions through the
catalog vendor and validate them the way a data-path request is validated:
deactivating, expiring or deleting key A ends A's session while the session
vended through key B on the same identity keeps working; disabling or
deleting the identity ends it; callers that are not a stored identity holding
the key get unbound sessions that keep working; other STS sessions are not
checked. The first two tests and the third fail without this change.

* s3: keep the vended-session key binding when the key dies before vend

vendedSessionKeyLive gated the binding, so a key revoked between the
request's authentication and the vend skipped it and minted an unbound
session that stayed usable until it expired. Every catalog auth path
pairs the identity name in context with the key that authenticated, so
a stored-identity principal always means the request signed with that
identity's own key. Bind whenever the principal is a stored identity: a
session bound to a now-dead key is refused on its first request, which
is what the binding is for.

* s3: gate the vended-session key binding on the authenticated caller type

lookupByIdentityName(principal) named only that some identity exists, not
that the request signed with a stored key: an STS session or OIDC principal
that collides with an identity name got a session bound to its temporary
key, which vendedSessionKeyLive then refused on every request. Verifiers
now record the verified key as a stored credential, and the vendor binds
only when that marker is present — keeping the binding even when the key
was revoked between authentication and the vend.

* s3: enforce the vended key binding under a dedicated claim

Sessions minted before the binding existed carry the caller's key in the
audit-only SourceAccessKeyId — including temporary STS keys that can never
resolve in the identity table. Enforcing on that field would end live table
credentials at upgrade; the binding now travels under BoundAccessKeyId,
leaving legacy sessions unbound as before.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: bind vended sessions to a credential fingerprint

The key id alone cannot identify a credential: an operator can delete a
key and re-create it under the same id with a different secret, reviving
sessions minted for the original. Verifiers now record a fingerprint of
the secret at authentication; the session dies unless it still matches.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: keep recording the caller's key on every catalog session

SourceAccessKeyId was unconditional audit metadata before the binding
existed; gating it on the stored-credential marker dropped the record
for callers that authenticated another way. With the binding under its
own claim there is no ambiguity left, so the field records the caller's
key again regardless of how it authenticated.

* s3: key the vended-session credential fingerprint with the signing key

The bound-credential fingerprint travels inside the vended session
token. As a plain SHA-256 of the secret it gave any token holder an
offline guessing oracle for weak secrets, and CodeQL flags it twice.

Store the verified secret in the request marker instead and HMAC it
with the STS signing key at vend and at validation. The claim can no
longer be mined, and the per-request auth path stops hashing secrets.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 03:58:06 -07:00
Chris LuandDevin bf3081e23a s3 lifecycle: route version-file writes to current-version expiration (#11956) 
* s3lifecycle: schedule ExpirationDays from version-file writes

On a versioned bucket a PUT lands in the meta-log as the version file
plus the .versions/ directory update — there is no bare-key event, so
the version-folder skip dropped every version-file event for all kinds
but ExpiredDeleteMarker. ExpirationDays therefore only fired via the
recovery walker's cold-start / rule-change pass: once a shard had a
saved cursor, no daily run ever scheduled another delete marker.

A version-file write is the latest for its logical key at write time,
so route it through the same event-driven evaluation the regular path
runs. The match carries the file's identity with an empty VersionID —
the dispatcher resolves the current latest at due time, and a fresher
PUT meanwhile resolves to STALE_IDENTITY there.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* lifecycle test: versioned Expiration fires on the second daily run

Establishes the shard cursor with a first pass, then PUTs and backdates
a new versioned object and runs again. The delete marker must appear
without a rule change forcing a recovery walk.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: run TestLifecycleVersionedBucketExpiresOnSecondRun in the lifecycle matrix

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 03:19:41 -07:00
Chris LuandDevin 2e7e691bed volume: drop stale index entries that resolve to a different needle (#11948) 
* volume: drop index entries that resolve to a different needle

An index row whose offset lands on a needle with a different id and a
different size cannot have been produced by any write path: every appender
indexes a key at the offset and size of the record it wrote, and
WriteNeedleBlob keeps its indexed size consistent with the blob header.
Such rows are corruption (for example an offset replayed with stale high
bytes) and until now made the volume reject every write for the key with
"mismatching cookie" while reads resolved to the wrong needle.

When the duplicate check finds the indexed record belongs to a different
key and size, drop the stale entry and let the append re-index the key.
Same-key conflicts still fail the cookie check, and entries staying
consistent with a re-keyed blob keep their existing semantics.

* volume: verify the indexed needle header before streaming reads

The buffered read path re-parses the record and rejects a needle whose
stored size disagrees with the index, but the streamed path serves data
bytes without ever looking at the header, so a stale index row would
stream another needle's payload. Probe the header once up front and
report a miss when the record is neither the requested key nor the
indexed size, the same staleness test the write path applies.

* rust volume: drop index entries that resolve to a different needle

Mirror of the Go volume change: when the duplicate check finds the
indexed record belongs to a different key and size, delete the stale
entry and let the append re-index the key.

* volume: overwrite a stale index entry on commit, not up front

Deleting the stale row during the duplicate check published a tombstone
before the replacement append and, for durable writes, before its fsync.
A failed append or sync then rolled back the data but left the key
deleted. The publish step already overwrites any older offset, so leave
the entry in place and let it be replaced once the record commits.

* volume: report streaming header-probe failures to the io error tracker

A failed header read returned before the tracked body-read loop, so
repeated EIO on the probe never reached checkReadWriteError. Skip io.EOF
the way the loop does: it is a truncated record, not a disk fault.

* rust volume: overwrite a stale index entry on commit, not up front

Mirror of the Go volume change: the staged delete published a tombstone
before the replacement append and its .dat flush, so a failed append or
sync left the key deleted. publish_write already overwrites any older
offset once the record commits.

* rust volume: report needle meta read failures to the io error tracker

read_meta is the header probe ahead of a streamed body: an EIO there
escaped check_read_write_error entirely, so a failing disk could not
trip the tolerance. Report the outcome the way read_full already does.

* volume: retry the wrapped offset before judging an index entry stale

A 4-byte index value can index past the 32 GiB wrap: the low offset may
hold another needle while the real record sits at +MaxPossibleVolumeSize,
which is where reads already retry on a size mismatch. Skipping the
cookie check on the low-offset record alone would accept a foreign-cookie
write over a live key. Retry the wrapped header read first, the same
condition the read path uses; a wrapped read that fails for reasons other
than EOF stops the write like the low-offset read does.

* volume: probe the streamed needle header under the access lock

With HasSlowRead the access lock is dropped after the index lookup, so a
compaction commit could swap the backend between the lookup and the
header probe and fail a valid download on the old offset. Take the read
lock for the probe and resolve the compaction revision first, the way
the streaming loop re-resolves it on every page.

* rust volume: retry the wrapped offset before judging an index entry stale

Mirror of the Go volume change: probe offset + MAX_POSSIBLE_VOLUME_SIZE
under the 4-byte-offset build before treating a mismatched record as
stale, so a live wrapped entry still gets its cookie checked.

* rust volume: do not clear the io error streak on a successful meta read

read_meta runs before the payload read in read_needle_for_get, so recording success there reset the streak before every failing download and the volume could never reach quarantine. Report failures only; read_full still records success after a complete read.

* volume: record a meta-only read's success against the io error streak

A meta read no longer clears the streak because a payload read still
follows on GETs, but requests the meta read ends outright — HEAD answers
and cookie mismatches — never reached the body path, so their success
left earlier failures counted. Note the success on those terminal paths;
mirrored in the Rust volume server.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: report streamed reads to the io error streak

Streamed GETs never touched the tracker: a failing chunk read went
uncounted and a completed download never cleared the streak, unlike
readNeedleDataInto which does both.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: reset the io error streak only at terminal meta reads

A streamed GET's metadata read runs before its payload read, so clearing
the streak there erased the count before every failing download and the
volume could never reach quarantine. Record the meta-read success only
where the request actually ends: must-meta-only reads, HEAD responses,
and cookie mismatches — matching the Rust volume server's placement.

* rust volume: record the streamed read's success at the final frame

Hyper can complete a Content-Length response without polling the body
again, so a success recorded only when the stream is next polled could
be skipped, leaving an earlier io error streak in place after a
successful download.

* volume: record a meta-only read's success on conditional 304 replies

* rust volume: record a meta-only read's success on conditional 304 replies

* volume: reset the io error streak only after the checksum passes

A streamed read that ends in a CRC mismatch still cleared the EIO
streak, so corruption could mask progress toward quarantine. The Rust
side records success only after the final frame's checksum verifies;
move the reset below the CRC check to match.

* volume: satisfy codespell in the meta-read comment

* rust volume: report range reads to the io error streak

* rust volume: cover the range read's io error reporting

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 02:30:53 -07:00
Chris LuandDevin 27fa285d48 filer: do not stall startup waiting out the leader-settle window (#11972) 
* filer: do not stall startup waiting out the leader-settle window

ListExistingPeerUpdates re-polled the leader for up to 10s when it
reported zero filers. A fresh single-filer cluster can never gain peers,
so every filer start paid the full settle timeout before listening —
past the 6s health window in weed mini and short probe budgets.

Take the leader's first answer instead; the fallback fullest-answer
scan below still covers a leader that is mid-convergence.

* filer: bound leader-settle re-poll below the startup health window

* filer: update ListExistingPeerUpdates doc for empty-leader fallback

* filer: re-check empty membership at the announce gate before registering

A fresh filer that scans empty membership during a master restart or
re-election skipped bootstrap and announced immediately, serving missing
files until a peer happened to reconnect — and a late peer only got an
incremental subscription, so pre-existing metadata was never backfilled.

The announce gate now re-polls ListExistingPeerUpdates while the scan can
still be disproved (10s), backfills from any late peers before their
streams start, and then waits for their catch-up like startup peers. A
genuinely empty cluster pays a delayed master listing rather than a
stalled startup, since this runs after the ports already serve.

* filer: overlap the late-peer copy cutoff and stream start

A peer event stamped just behind this filer's clock could land after
the traversal read but carry a timestamp below the stream start —
the startup scan margins both cutoffs by LogFlushInterval for exactly
this skew; mirror it for peers discovered at the announce gate.

* filer: bound the late-peer bootstrap copy by the shared announce deadline

MaybeBootstrapFromOnePeer now takes the caller's context so a stalled
peer metadata copy cannot out-wait the announce cap, and the bootstrap
re-check shares one announceCatchUpCap deadline with the catch-up wait
instead of each spending a full minute.

* filer: merge bootstrap entries by commit time instead of blind insert

The late-peer copy runs while gRPC already serves, so a blind InsertEntry
could replace a write the local filer just accepted. Routing each entry
through applyTraversedEntry orders it against the local commit time under
the replay locks — a fresher local write survives the backfill.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 02:04:27 -07:00
Chris LuandDevin b65832bdc9 iam/oidc: reject reused jti claims with a bounded replay cache (#11936) 
* iam/oidc: reject reused jti claims with a bounded replay cache

* iam/sts: pass jti replay settings through convertToOIDCConfig

* iam/oidc: keep consumed jtis and stop the cleanup worker across re-init

Re-initializing a provider reset the JTI store, so an already-used token
could be presented again after a config refresh; the store is now
allocated once and survives Initialize. The cleanup worker is stopped on
every re-init (previously only when protection stayed enabled), is
started only after the rest of Initialize has succeeded, and Shutdown no
longer consumes a permanent sync.Once, so a re-initialized worker can be
stopped again.

* iam/oidc: enforce the jti replay cache bound for every token

Tokens without a usable jti claim skipped the cache entirely and stayed
reusable; they are now keyed by the token's own fingerprint so a repeat
presentation is still a replay. Tokens without exp previously failed as
expired on first use; their entry now gets a bounded retention. The
replay test-and-set now runs before any capacity work so replayed tokens
cannot each trigger a full-cache scan, the entry count is incremented as
part of the insert so concurrent validations cannot overshoot the bound,
and a guaranteed serialized sweep runs before a capacity rejection while
opportunistic sweeps are rate limited and confined to one at a time. Raw
token identifiers are no longer written to logs or errors, a stable
fingerprint is logged instead.

* iam/sts: shut down providers dropped by the managed-provider swap

Each managed OIDC provider runs a background JTI cleanup worker, but
SetIAMManagedOIDCProviders replaced the provider map without stopping
the old providers, so every refresh leaked goroutines. Providers absent
from the new map are now shut down; providers that stay keep running.

* iam/integration: reuse providers across refreshes for unchanged records

Every refresh rebuilt providers from their records, so a refreshed
provider forgot the JTIs it had already consumed and an already-used
token could be presented again. Records are now fingerprinted and an
unchanged record reuses its provider, keeping the consumed-token cache;
changed records are rebuilt, removed records are dropped, and the swap
in STS releases the workers of whatever is no longer referenced.

* iam/oidc: consume replay entries only on the single-use exchange path

Bearer authentication validates the same token on every request, so
consuming it in shared ValidateToken/Authenticate made every token
single-use and rejected all but the first request. The replay check now
runs only for the STS exchange via AuthenticateSingleUse; bearer paths
validate without consuming. A token without exp also no longer gets a
bounded replay entry that ages out while the token stays valid: its
entry is kept until the capacity bound evicts it, so it can never become
usable again.

* iam/oidc: keep the mock provider on the mock path for single-use calls

* iam/oidc: require exp on single-use tokens

A cache slot frees only by expiry, so an expiry-less entry would hold it
forever; a burst of distinct tokens could exhaust the bound permanently
and block every later exchange.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: give each web-identity test token a unique jti

Subtests minted tokens with identical claims inside one second, so the
exchange saw the same token fingerprint twice and the second call read
as a replay. Each exchange now carries its own jti.

* iam/oidc: carry the consumed-token replay store across provider rebuilds

A record change rebuilds the managed provider with a fresh JTI store, so a token consumed before the rebuild could be presented again until it expired. The new provider adopts the replaced provider's replay store before it is published, and the replaced provider is shut down so only one cleanup worker sweeps the shared store.

* iam/oidc: stop the cleanup worker of providers dropped from the store

A refresh removed the provider object from the managed cache but left its JTI cleanup goroutine running for the life of the process.

* iam/oidc: merge a provider's consumed replay entries into another's store

MergeReplayStateFrom folds one provider's consumed-token entries into
another's, skipping already-expired ones. Unlike AdoptReplayStateFrom it
swaps no pointers, so it is safe on a provider still serving validators.

* iam/integration: share one provider and replay store across identical issuer records

Records for the same issuer URL with identical validation settings now
reuse a single provider, so a token consumed through one account's
record is rejected through the other's — previously each record owned a
separate jti store and a spent token could be replayed cross-account.

A superseded provider's consumed entries merge into the shared store so
no rejection is lost, and a removed or rebuilt provider is only shut
down once no remaining record references it.

* iam/oidc: hand the superseded provider's live replay store to the shared one

* iam/oidc: carry merged replay entries without charging the capacity bound

* iam/oidc: key the jti-less replay fingerprint on the signed input

Hashing the whole token lets ECDSA signature malleability (s -> N-s)
mint a second cache key for identical claims, so a consumed token could
be replayed for another exchange. Key the fallback on header.payload
instead.

* iam/oidc: publish the jti cache bound before the enabled flag

Initialize stored jtiEnabled ahead of jtiMaxSize, so a provider flipped
from disabled to enabled exposed enabled=true with a zero bound for a
window; a single-use validation landing there would compute
newCount > maxSize and reject with "JTI cache capacity exceeded". Store
the bound first so the flag is the last thing published.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
4.49 		2026-10-23 00:46:14 -07:00
Chris LuandDevin 186bc5cf7f s3: bound ListMultipartUploads memory to the page size (#11971) 
* s3: bound ListMultipartUploads memory to the page size

listMultipartUploads loaded every open upload in the bucket into a slice
and selectMultipartUploads filtered and sorted a second full copy, so one
request cost O(total open uploads) heap regardless of prefix, key-marker,
or max-uploads; concurrent calls multiplied it and could OOM the gateway.

The response is ordered by the object key kept in extended attributes, not
the upload id the filer sorts by, so every entry still has to be seen.
Streaming the listing into a bounded top-K collector keeps at most
maxUploads (capped at maxUploadsList) candidates, bounding heap by the
page instead of the folder. A caller-supplied max-uploads above the
server cap now also paginates rather than accumulating unboundedly.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: cover bounded ListMultipartUploads accumulation

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: report the effective max-uploads cap in ListMultipartUploads

* s3: cover ListMultipartUploads retry through the collector

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-23 00:46:10 -07:00
Chris LuandDevin 7154732b84 worker/ec: retry the pinned volume copy after a mid-copy compaction (#11929) 
* worker/ec: retry the pinned volume copy after a mid-copy compaction

The EC task pins .idx/.dat copies to the CompactionRevision read from
ReadVolumeFileStatus so the pair comes from one volume generation. When
vacuum compaction lands between that read and either CopyFile, the
server rejects the copy with "volume N is compacted" and the whole task
fails — on clusters with queued maintenance this makes most EC tasks
die to a race they cannot avoid. On a compaction mismatch the copy now
restarts under a freshly read revision, up to a bounded number of
attempts; other errors still fail immediately.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* worker/ec: read the retried copy's expiry from its own file status

A mid-copy compaction makes the source reload the volume, and the fresh
ReadVolumeFileStatus can carry a different expireAtSec than the
pre-compaction read. Keeping the attempt-0 expiry could let the EC
volume outlive its source TTL.

* worker/ec: cover the compaction retry with a stub volume server

* worker/ec: also observe revised .idx bytes in the compaction retry test

* worker/ec: read the fake source under its mutex in the retry test

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-22 21:32:57 -07:00
Chris LuandDevin f2d5c27f65 operation: renew the write JWT when a fan-out rollback delete is rejected (#11970) 
* operation: renew the cleanup JWT when a fan-out rollback delete is rejected

A stalled holder can out-wait the assignment JWT's 10s expiry, so the
type=replicate delete that rolls back landed copies comes back 401 and the
needle leaks. Retry the delete once with a fresh token minted through the new
LookupJwtFunc hook when a holder answers unauthorized.

* s3api: wire a cleanup-JWT renewer into chunked uploads

Keep the master client on the server and use it to mint fresh volume JWTs for
rollback deletes that run past the assignment token's expiry.

* admin: wire a cleanup-JWT renewer into file-browser uploads

* operation: bound the cleanup-JWT renewal with a per-attempt deadline

A rollback delete that hits 401 renewed the token through a background
context, so a master outage could park the wait for a connection and the
lookup RPC inside the failed upload's cleanup path. Thread a context
through LookupJwtFunc and LookupJwt, and bound each renewal with a 10s
deadline covering both the master wait and the lookup call.

* operation: let every holder's 401 retry with a renewed cleanup JWT

A single renewed flag meant only the first rejecting holder refreshed the
token; a holder that stalled past the renewed JWT's lifetime left later
copies stranded with a 401 they could not resolve. The retry limit is now
per holder — each delete is attempted once with the current token and once
more after a bounded renewal — so a slow middle holder cannot strand the
rest.

* operation: log the volume server's error body on a failed cleanup delete

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* operation: count test upload attempts atomically

The chunk upload loop fans AssignFunc out to parallel goroutines, so the
plain int counter in the partial-results test raced under -race.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-22 21:32:53 -07:00
github-actions[bot] b97ed0f101 4.49 2026-10-23 03:49:00 +00:00
github-actions[bot] 13cbe78537 docs: regenerate star history chart 2026-10-23 02:28:25 +00:00
Mohd Quamar Tyagi 94dd96a322 weed filer.remote.sync: ignore .system directory under buckets (#11966) (#11969) 2026-10-22 18:40:30 -07:00
Chris Lu 00e64441c6 fix: clone protobuf snapshots without copying internal locks (#11968) 
* fix: clone protobuf snapshots without copying internal locks

* fix: reuse cloned snapshot fields when filtering
 2026-10-22 18:38:33 -07:00
Chris Lu f08f075d23 deps: patch two open dependabot alerts (klauspost/compress, aws-smithy-json) (#11959) 
* test/kafka: bump klauspost/compress to v1.20.3 in simple-publisher

Patch an s2 integer-overflow / out-of-bounds write affecting
klauspost/compress <= 1.18.6; matches the version the main module
already uses. The go directive rises to 1.25, still under the ^1.26
toolchain the kafka test workflows run with.

* seaweed-volume: bump aws-smithy-json to 0.62.7

Patch an uncontrolled-recursion denial of service in the unknown-key
skip path of aws-smithy-json <= 0.62.6 (reachable over the S3 SDK's
response parsing). The smithy runtime family moves together:
aws-smithy-runtime-api 1.12.3 and aws-smithy-types 1.5.0, all selected
within the workspace's Rust 1.91.1 compatibility bound.
 2026-10-22 14:38:16 -07:00
Chris LuandDevin 7c980774e7 FUSE mount: add posix compliance reports (#7183) 
* fuse integration: add POSIX compliance test suite

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: add posix-compliance workflow

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: skip non-executable weed path candidates

* fuse integration: size readv buffer to the written segment

* fuse integration: reference posix_Makefile explicitly in recursive make calls

GNU make does not propagate -f through MAKEFLAGS, so bare $(MAKE) re-read the default Makefile in this directory.

* fuse integration: fail posix test targets when go test fails

Every suite pipes go test through tee under /bin/sh, so the recipe status was tee's and test failures stayed green. Run recipes under bash -o pipefail and emit the color escapes with printf since bash echo does not expand them.

* fuse integration: run only benchmarks in benchmark-posix

go test -bench=. without -run also executes the whole suite, mixing compliance output into the benchmark log.

* fuse integration: emit single-number test counts in reports

grep -c prints 0 and exits nonzero when nothing matches, so '|| echo 0' produced two lines that broke the expr arithmetic and JSON fields. awk prints exactly one count either way.

* fuse integration: skip the sendfile test on macOS

macOS sendfile(2) only accepts a socket destination; the test copies into a regular file and always gets ENOTSOCK.

* fuse integration: honor TestConfig and own only self-started resources

NewFuseTestFramework dropped its config argument and Cleanup read a fresh
DefaultTestConfig, so SkipCleanup never did anything. Cleanup also skipped
teardown whenever TEST_SKIP_CLUSTER_SETUP was set, even when TEST_MOUNT_POINT
was empty and Setup had started the cluster itself. The framework now keeps
the config it was built with, runs external-mount tests inside a per-run work
dir under the mount, and only skips teardown when an external mount was
actually used.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: pass the debug flag before the subcommand

-v is a weed global flag; appended after master/volume/filer/mount it lands
on the subcommand flagset, which rejects it and the process dies at flag
parsing whenever EnableDebug is on.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: require a real mount before running tests

Setup creates the mount point before weed mount starts, so statting and
listing it succeeded on the plain directory and tests ran against local
disk whenever the mount was slow or died. waitForMount now waits until the
mount point's device id differs from its parent's.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: resolve the weed binary to an absolute path

The started processes run with cmd.Dir at the framework's temp dir, so a
relative candidate like ../weed no longer resolves. Prefer PATH, then probe
the usual build outputs and return absolute paths only.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: build the external test work dir with MkdirTemp

The suite staged cloned tools and generated scripts under a predictable
/tmp path another user could pre-create and swap mid-run.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: use the pjdfstest tree setup-pjdfstest installs

The test gated on a pjdfstest binary in PATH but then proved a suite it
cloned itself, so a tree installed by make setup-pjdfstest was never found.
Look for the built suite under external_tools first, then clone into the
work dir as before.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: require a real mount before posix tests

The wait loop fell back to listing the mount point, which passes on the
directory created before weed mount runs — a dead or slow mount meant the
suite exercised local disk. Only mountpoint counts now, and a dead mount
process fails fast instead of timing out.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: collect failure logs before cleanup deletes them

The always-on cleanup step removed /tmp/seaweedfs before the failure-only
debug step could copy the service logs, so failed runs uploaded nothing.
Run cleanup after log collection.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: point MOUNT_POINT at TEST_MOUNT_POINT

Targets that work against MOUNT_POINT (pjdfstest, nfstest, fio configs)
ignored TEST_MOUNT_POINT and ran against the local-disk default while the
go tests used the external mount.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: skip the permission-denied check as root

Root bypasses mode-bit checks, so writing the read-only file succeeded and
the assertion failed in root-run environments like containers.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: align the posix docs with the suite

The doc listed a docker-test-posix target and make variables that do not
exist, and ran go test against single files that cannot compile alone. The
roadmap also still claimed the platform-specific tests this suite now
implements were all skipped.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: honor SkipCleanup in the mini-based framework too

Same gap as the fuse-package framework: Cleanup read a fresh
DefaultTestConfig, so a test's SkipCleanup setting was ignored.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: normalize config in Setup and clean the work dir too

Setup(nil) panicked on config.EnableDebug and external-mount Cleanup left
the temp dir behind; use the stored config for the start* calls and remove
tempDir alongside the external work dir.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: point external posix suites at the workflow FUSE mount

* fuse integration: create the fio target directory before running

* fuse integration: use unix.O_DIRECT instead of a hard-coded x86 value

0x4000 is O_DIRECT only on x86 and amd64; on arm64 it is O_DIRECTORY.
The test module already depends on golang.org/x/sys.

* fuse integration: run autoreconf and configure before building pjdfstest

A fresh pjdfstest clone is autotools-based and has no Makefile, so the
Go-side setup fallback always failed and the suite silently skipped.

* ci: require external posix tools only when setup installed them

setup-external-tools always exits 0 because each sub-target tolerates
failure, so POSIX_REQUIRE_EXTERNAL_TOOLS was set even with no tools
installed and the suite then hard-failed on the missing suites.

* ci: fail the posix workflow when the mount point is not a FUSE mount

The file write/read check passes on the plain directory created before
weed mount, so a dead mount let the whole suite grade the local disk.

* fuse integration: skip when the host has no FUSE device node

Setup failed the test outright on machines without /dev/fuse instead of
skipping; cover every suite entry point by skipping inside Setup.

* fuse integration: drop the chmod 666 /dev/fuse troubleshooting tip

World-writable device access is a security risk; the fuse group
membership on the line below is the safe fix.

* fuse integration: require go 1.21+ in check-go, not 1.20+

* fuse integration: refuse external suites on an unmounted path

* fuse integration: verify TEST_MOUNT_POINT is a real mount

* fuse integration: clean TEST_MOUNT_POINT before the parent-device check

filepath.Dir of a trailing-slash path returns the mount itself, so a mounted TEST_MOUNT_POINT ending in / was refused.

* fuse integration: detect WinFsp mounts on Windows in mountedAt

A directory mount is a reparse point and a drive mount is a volume root; a plain directory is now refused instead of accepted.

* fuse integration: strip a trailing slash off MOUNT_POINT in require-mounted

The mount-table grep fallback on hosts without mountpoint(8) would not match a slash-suffixed entry.

* fuse integration: require an external mount to be a FUSE filesystem

A device-ID boundary alone does not prove FUSE: a tmpfs mount would pass mountedAt and report its results as FUSE compliance. Check the filesystem type (statfs magic on linux, fstypename on the BSDs); platforms without a check keep the mount and write verification.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-22 14:37:03 -07:00
Chris Lu 8f6121694a test: fix HEAD bucket assertions and FUSE failover startup timing (#11967) 
test: fix HEAD bucket assertions and warm up failover clusters
 2026-10-22 14:34:52 -07:00
Chris Lu 137d29633b s3: fix prefix filtering and delimiter pagination (#11962) 2026-10-22 09:31:20 -07:00
github-actions[bot] ca0314ea4f docs: regenerate star history chart 2026-10-22 12:49:43 +00:00