A device-ID boundary alone does not prove FUSE: a tmpfs mount would pass mountedAt and report its results as FUSE compliance. Check the filesystem type (statfs magic on linux, fstypename on the BSDs); platforms without a check keep the mount and write verification.
Setup(nil) panicked on config.EnableDebug and external-mount Cleanup left
the temp dir behind; use the stored config for the start* calls and remove
tempDir alongside the external work dir.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Same gap as the fuse-package framework: Cleanup read a fresh
DefaultTestConfig, so a test's SkipCleanup setting was ignored.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The doc listed a docker-test-posix target and make variables that do not
exist, and ran go test against single files that cannot compile alone. The
roadmap also still claimed the platform-specific tests this suite now
implements were all skipped.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Root bypasses mode-bit checks, so writing the read-only file succeeded and
the assertion failed in root-run environments like containers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Targets that work against MOUNT_POINT (pjdfstest, nfstest, fio configs)
ignored TEST_MOUNT_POINT and ran against the local-disk default while the
go tests used the external mount.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The test gated on a pjdfstest binary in PATH but then proved a suite it
cloned itself, so a tree installed by make setup-pjdfstest was never found.
Look for the built suite under external_tools first, then clone into the
work dir as before.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The suite staged cloned tools and generated scripts under a predictable
/tmp path another user could pre-create and swap mid-run.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The started processes run with cmd.Dir at the framework's temp dir, so a
relative candidate like ../weed no longer resolves. Prefer PATH, then probe
the usual build outputs and return absolute paths only.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Setup creates the mount point before weed mount starts, so statting and
listing it succeeded on the plain directory and tests ran against local
disk whenever the mount was slow or died. waitForMount now waits until the
mount point's device id differs from its parent's.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
-v is a weed global flag; appended after master/volume/filer/mount it lands
on the subcommand flagset, which rejects it and the process dies at flag
parsing whenever EnableDebug is on.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
NewFuseTestFramework dropped its config argument and Cleanup read a fresh
DefaultTestConfig, so SkipCleanup never did anything. Cleanup also skipped
teardown whenever TEST_SKIP_CLUSTER_SETUP was set, even when TEST_MOUNT_POINT
was empty and Setup had started the cluster itself. The framework now keeps
the config it was built with, runs external-mount tests inside a per-run work
dir under the mount, and only skips teardown when an external mount was
actually used.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
grep -c prints 0 and exits nonzero when nothing matches, so '|| echo 0' produced two lines that broke the expr arithmetic and JSON fields. awk prints exactly one count either way.
Every suite pipes go test through tee under /bin/sh, so the recipe status was tee's and test failures stayed green. Run recipes under bash -o pipefail and emit the color escapes with printf since bash echo does not expand them.
The POSIX suites skirt this: pjdfstest's unlink/14.t covers only fstat and
pread on an unlinked descriptor — its driver has no fchmod at all and never
opens a directory. Pin the full rule in the FUSE integration suite:
ftruncate, fchmod, futimes, fstat, and the f*xattr calls keep working
between the removal of the last name and the final close, for a file after
unlink and a directory after rmdir, with nlink 0 and the changes visible to
a following fstat.
Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK
* ci: move the fusermount3 repair into a composite action
Three copies of the same block were already drifting apart, and the
target comes from PATH: only ever add setuid root to a root-owned,
non-symlink binary under the system bin paths, and say why otherwise.
* test: say that the process exited in the wait errors
"process exit status 1 before ... accepted connections" is missing its
verb. Also mark the SIGTERM return discarded - it fails with
os.ErrProcessDone exactly when the select below already handles it.
* ci: prefer the distro fusermount3 over escalating a shadow copy
The shadowing /usr/local/bin/fusermount3 is not root-owned either, so
setting its setuid bit would have handed root to a binary the runner
user owns - the repair now symlinks the distro one earlier in PATH and
touches nothing, keeping the in-place chmod for a root-owned binary with
no distro alternative. A setuid bit only grants root when root owns the
file, so accept an existing one only then.
* ci: run the FUSE workflows when the shared action changes
Their paths filters listed each workflow file but not the composite
action all three now call.
* ci: restore the setuid bit on a shadowed fusermount3
Newer ubuntu-22.04 runner images carry a source-built fusermount3 in
/usr/local/bin that shadows the distro one in PATH and is not setuid
root. go-fuse looks the helper up through PATH, so every unprivileged
mount fails with "mount failed: Operation not permitted".
* test: fail a fuse test as soon as its mount process dies
A mount that cannot mount at all exits within a second, but the harness
still waited out the 30s readiness timeout and then reported "mount
point not ready within timeout", leaving the real cause buried in the
log tail. Watch the child processes and report their exit instead.
* mount: report a failed mount without a goroutine dump
A mount failure is an environment problem - no /dev/fuse, fusermount not
setuid, stale mount point - and the all-goroutine stack dump Fatalf adds
buries the one line that says so.
The FUSE harness picks the filer HTTP port and lets "weed mount" derive the
filer gRPC port as HTTP+10000. freePort kept the HTTP port under the Linux
ephemeral floor (32768) but not the derived gRPC port, which ranged up to
42000. When the gRPC port landed above the floor an outbound connection could
transiently hold it, so mini relocated its filer gRPC port while mount kept
dialing HTTP+10000, timing the mount out.
Cap the HTTP port at 22000 so the gRPC sibling stays at or below 32000, and
verify both ports are bindable before returning.
glog flags (-v, -logtostderr) are registered on weed's global flagset,
so passing them after the subcommand name kills the process at flag
parsing: flag provided but not defined: -logtostderr. The old stat-based
mount readiness probe masked this — TestWriteBufferCap silently ran
against the bare local directory and passed. The device-ID readiness
check now surfaces the dead mount as a not-ready timeout.
Move glog flags into MountGlobalOptions, emitted before the subcommand,
and do the same for the EnableDebug verbosity flag on mini and mount.
waitForMount probed the mount point with stat+ReadDir, which a bare local
directory also passes, so Setup could return before the weed mount process
finished mounting. Tests then wrote to the local disk underneath the mount
point; when the mount activated it shadowed those files, producing the
intermittent TestConcurrentFileOperations/ConcurrentReadWrite ENOENT with
an empty ReadDir. Require the mount point's device ID to differ from its
parent's before reporting ready.
* filer: POSIX advisory lock set primitive (phase 1)
Pure per-inode conflict/coalesce/range-split logic for fcntl byte-range
and flock whole-file locks, extracted from the mount's PosixLockTable
without its wait queue or inode-map concurrency. Owner identity is
(Sid, Owner) so the same FUSE owner on different mounts never aliases,
and ReleaseSession reaps a dead mount's locks. The owner filer will hold
one Set per inode under the per-path lock; no concurrency control here.
* test: tolerate transient FUSE invisibility in ConcurrentReadWrite
A concurrent truncating overwrite leaves a short-lived dentry/cache window
where the file is momentarily ENOENT to another opener. Retry the reads and
writes a few times before failing, as ConcurrentDirectoryOperations does.
The 45m suite alarm fires on TestWritebackCacheStressSmallFiles with no
output from the test, since t.Logf is buffered until the test completes
and the alarm panic skips that flush. Add streaming stderr progress, an
explicit verify-phase budget that t.Fatalf's with a goroutine dump on
overrun, and per-retry/per-failure logging so the next hang shows which
file(s) the mount could not read back.
PR #9230 attempt 1 hit an intermittent
TestConcurrentFileOperations/ConcurrentReadWrite failure where stat
returned ENOENT for a path all writers had just succeeded against, and
the captured mount.log carried no signal about which layer dropped the
entry because the relevant lookup logged at V(4).
Two diagnostic-only changes (no behavior change on the happy path):
- weed/mount/weedfs.go: in lookupEntry, when filer GetEntry returns
ErrNotFound for a path whose inode is still tracked locally with no
in-flight create or flush, log Warningf with inode + dirtyHandle +
pendingFlush + localCache + dirCached. This surfaces layer-by-layer
state at the moment of the suspicious ENOENT.
- test/fuse_integration/framework_test.go: on AssertFileExists failure,
dump five 100ms-spaced stat retries, a parent ReadDir, and a direct
O_RDONLY open before failing. Triangulates kernel dentry caching vs
mount lookup vs filer state.
TestPosixFileLocking/ConcurrentLockContention failed in CI (run
24857323067) with ENOENT when re-opening the file after all 8 workers
had successfully written and closed. The 20s openWithRetry budget was
exhausted, pointing at a real but unproven metaCache/parent-cache
coherence issue in the mount under bursts of concurrent Release.
Test: hold the initial fd open for the whole subtest; use it for the
post-workers Sync() and the verification read. Workers still exercise
the concurrent-flock invariant and per-record write correctness; the
re-open path is no longer load-bearing. On Eventually failure, dump
ReadDir of the parent, Stat, and a fresh O_RDONLY open so a future
recurrence has state to debug from. Drop the darwin-only ENOENT
t.Skip branches that hid this same flake.
Mount: in weedfs.lookupEntry, when returning ENOENT from the
"parent cached but child missing" branch, log at Warningf instead of
V(4) when the kernel is still tracking this path's inode. That
combination is the smoking-gun signal for cache drift and is rare
enough in normal use not to spam the log.
TestPosixFileLocking/FcntlReleaseOnClose was flaky because the
subprocess spawned by startLockHolder occasionally saw ENOENT when
opening a file the parent had just created on the FUSE mount. Retry on
ENOENT (matching the existing openWithRetry pattern used in
testConcurrentLockContention) so the subprocess waits for the mount's
dentry state to propagate before reporting the lock acquire as failed.
freePort allocated in [20000, 55535], which overlaps the Linux ephemeral
range (32768-60999). The kernel could reuse the chosen port for an
outbound connection between the test closing its listener and weed mini
re-checking availability, causing:
Port allocation failed: port N for Filer (specified by flag
filer.port) is not available on 0.0.0.0 and cannot be used
Narrow the range to [20000, 32000] to stay below the ephemeral floor,
and pass -ip.bind=127.0.0.1 so mini's pre-check runs on the same IP the
test actually reserved the port on.
* feat(mount): cap write buffer with -writeBufferSizeMB
Without a bound on the per-mount write pipeline, sustained upload
failures (e.g. volume server returning "Volume Size Exceeded" while
the master hasn't yet rotated assignments) let sealed chunks pile up
across open file handles until the swap directory — by default
os.TempDir() — fills the disk. Reported on 4.19 filling /tmp to 1.8 TB
during a large rclone sync.
Add a global WriteBufferAccountant shared across every UploadPipeline
in a mount. Creating a new page chunk (memory or swap) first reserves
ChunkSize bytes; when the cap is reached the writer blocks until an
uploader finishes and releases, turning swap overflow into natural
FUSE-level backpressure instead of unbounded disk growth.
The new -writeBufferSizeMB flag (also accepted via fuse.conf) defaults
to 0 = unlimited, preserving current behavior. Reserve drops
chunksLock while blocking so uploader goroutines — which take
chunksLock on completion before calling Release — cannot deadlock,
and an oversized reservation on an empty accountant succeeds to avoid
single-handle starvation.
* fix(mount): plug write-budget leaks in pipeline Shutdown
Review on #9066 caught two accounting bugs on the Destroy() path:
1. Writable-chunk leak (high). SaveDataAt() reserves ChunkSize before
inserting into writableChunks, but Shutdown() only iterated
sealedChunks. Truncate / metadata-invalidation flows call Destroy()
(via ResetDirtyPages) without flushing first, so any dirty but
unsealed chunks would permanently shrink the global write budget.
Shutdown now frees and releases writable chunks too.
2. Double release with racing uploader (medium). Shutdown called
accountant.Release directly after FreeReference, while the async
uploader goroutine did the same on normal completion — under a
Destroy-before-flush race this could underflow the accountant and
let later writes exceed the configured cap. Move accounting into
SealedChunk.FreeReference itself: the refcount-zero transition is
exactly-once by construction, so any number of FreeReference calls
release the slot precisely once.
Add regression tests for the writable-leak and the FreeReference
idempotency guarantee.
* test(mount): remove sleep-based race in accountant blocking test
Address review nits on #9066:
- Replace time.Sleep(50ms) proxy for "goroutine entered Reserve" with
a started channel the goroutine closes immediately before calling
Reserve. Reserve cannot make progress until Release is called, so
landed is guaranteed false after the handshake — no arbitrary wait.
- Short-circuit WriteBufferAccountant.Used() in unlimited mode for
consistency with Reserve/Release, avoiding a mutex round-trip.
* test(mount): add end-to-end write-buffer cap integration test
Exercises the full write-budget plumbing with a small cap (4 chunks of
64 KiB = 256 KiB) shared across three UploadPipelines fed by six
concurrent writers. A gated saveFn models the "volume server rejecting
uploads" condition from the original report: no sealed chunk can drain
until the test opens the gate. A background sampler records the peak
value of accountant.Used() throughout the run.
The test asserts:
- writers fill the budget and then block on Reserve (Used() stays at
the cap while stalled)
- Used() never exceeds the configured cap even under concurrent
pressure from multiple pipelines
- after the gate opens, writers drain to zero
- peak observed Used() matches the cap (262144 bytes in this run)
While wiring this up, the race detector surfaced a pre-existing data
race on UploadPipeline.uploaderCount: the two glog.V(4) lines around
the atomic Add sites read the field non-atomically. Capture the new
value from AddInt32 and log that instead — one-liner each, no
behavioral change.
* test(fuse): end-to-end integration test for -writeBufferSizeMB
Exercise the new write-buffer cap against a real weed mount so CI
(fuse-integration.yml) covers the FUSE→upload-pipeline→filer path, not
just the in-package unit tests. Uses a 4 MiB cap with 2 MiB chunks so
every subtest's total write demand is multiples of the budget and
Reserve/Release must drive forward progress for writes to complete.
Subtests:
- ConcurrentLargeWrites: six parallel 6 MiB files (36 MiB total, ~18
chunk allocations) through the same mount, verifies every byte
round-trips.
- SingleFileExceedingCap: one 20 MiB file (10 chunks) through a single
handle, catching any self-deadlock when the pipeline's own earlier
chunks already fill the global budget.
- DoesNotDeadlockAfterPressure: final small write with a 30s timeout,
catching budget-slot leaks that would otherwise hang subsequent
writes on a still-full accountant.
Ran locally on Darwin with macfuse against a real weed mini + mount:
=== RUN TestWriteBufferCap
--- PASS: TestWriteBufferCap (1.82s)
* test(fuse): loosen write-buffer cap e2e test + fail-fast on hang
On Linux CI the previous configuration (-writeBufferSizeMB=4,
-concurrentWriters=4 against a 20 MiB single-handle write)
deterministically hung the "Run FUSE Integration Tests" step to the
45-minute workflow timeout, while on macOS / macfuse the same test
completes in ~2 seconds (see run 24386197483). The Linux hang shows
up after TestWriteBufferCap/ConcurrentLargeWrites completes cleanly,
then TestWriteBufferCap/SingleFileExceedingCap starts and never
emits its PASS line.
Change:
- Loosen the cap to 16 MiB (8 × 2 MiB chunk slots) and drop the
custom -concurrentWriters override. The subtests still drive demand
well above the cap (32 MiB concurrent, 12 MiB single-handle), so
Reserve/Release is still on every chunk-allocation path; the cap
just gives the pipeline enough headroom that interactions with the
per-file writableChunkLimit and the go-fuse MaxWrite batching don't
wedge a single-handle writer on a slow runner.
- Wrap every os.WriteFile in a writeWithTimeout helper that dumps every
live goroutine on timeout. If this ever re-regresses, CI surfaces
the actual stuck goroutines instead of a 45-minute walltime.
- Also guard the concurrent-writer goroutines with the same timeout +
stack dump.
The in-package unit test TestWriteBufferCap_SharedAcrossPipelines
remains the deterministic, controlled verification of the blocking
Reserve/Release path — this e2e test is now a smoke test for
correctness and absence of deadlocks through a real FUSE mount, which
is all it should be.
* fix: address PR #9066 review — idempotent FreeReference, subtest watchdog, larger single-handle test
FreeReference on SealedChunk now early-returns when referenceCounter is
already <= 0. The existing == 0 body guard already made side effects
idempotent, but the counter itself would still decrement into the
negatives on a double-call — ugly and a latent landmine for any future
caller that does math on the counter. Make double-call a strict no-op.
test(fuse): per-subtest watchdog + larger single-handle test
- Add runSubtestWithWatchdog and wrap every TestWriteBufferCap subtest
with a 3-minute deadline. Individual writes were already
timeout-wrapped but the readback loops and surrounding bookkeeping
were not, leaving a gap where a subtest body could still hang. On
watchdog fire, every live goroutine is dumped so CI surfaces the
wedge instead of a 45-minute walltime.
- Bump testLargeFileUnderCap from 12 MiB → 20 MiB (10 chunks) to
exceed the 16 MiB cap (8 slots) again and actually exercise
Reserve/Release backpressure on a single file handle. The earlier
e2e hang was under much tighter params (-writeBufferSizeMB=4,
-concurrentWriters=4, writable limit 4); with the current loosened
config the pressure is gentle and the goroutine-dump-on-timeout
safety net is in place if it ever regresses.
Declined: adding an observable peak-Used() assertion to the e2e test.
The mount runs as a subprocess so its in-process WriteBufferAccountant
state isn't reachable from the test without adding a metrics/RPC
surface. The deterministic peak-vs-cap verification already lives in
the in-package unit test TestWriteBufferCap_SharedAcrossPipelines.
Recorded this rationale inline in TestWriteBufferCap's doc comment.
* test(fuse): capture mount pprof goroutine dump on write-timeout
The previous run (24388549058) hung on LargeFileUnderCap and the
test-side dumpAllGoroutines only showed the test process — the test's
syscall.Write is blocked in the kernel waiting for FUSE to respond,
which tells us nothing about where the MOUNT is stuck. The mount runs
as a subprocess so its in-process stacks aren't reachable from the
test.
Enable the mount's pprof endpoint via -debug=true -debug.port=<free>,
allocate the port from the test, and on write-timeout fetch
/debug/pprof/goroutine?debug=2 from the mount process and log it. This
gives CI the only view that can actually diagnose a write-buffer
backpressure deadlock (writer goroutines blocked on Reserve, uploader
goroutines stalled on something, etc).
Kept fileSize at 20 MiB so the Linux CI run will still hit the hang
(if it's genuinely there) and produce an actionable mount-side dump;
the alternative — silently shrinking the test below the cap — would
lose the regression signal entirely.
* review: constructor-inject accountant + subtest watchdog body on main
Two PR-#9066 review fixes:
1. NewUploadPipeline now takes the WriteBufferAccountant as a
constructor parameter; SetWriteBufferAccountant is removed. In
practice the previous setter was only called once during
newMemoryChunkPages, before any goroutine could touch the
pipeline, so there was no actual race — but constructor injection
makes the "accountant is fixed at construction time" invariant
explicit and eliminates the possibility of a future caller
mutating it mid-flight. All three call sites (real + two tests)
updated; the legacy TestUploadPipeline passes a nil accountant,
preserving backward-compatible unlimited-mode behavior.
2. runSubtestWithWatchdog now runs body on the subtest main goroutine
and starts a watcher goroutine that only calls goroutine-safe t
methods (t.Log, t.Logf, t.Errorf). The previous version ran body
on a spawned goroutine, which meant any require.* or writeWithTimeout
t.Fatalf inside body was being called from a non-test goroutine —
explicitly disallowed by Go's testing docs. The watcher no longer
interrupts body (it can't), so body must return on its own —
which it does via writeWithTimeout's internal 90s timeout firing
t.Fatalf on (now) the main goroutine. The watchdog still provides
the critical diagnostic: on timeout it dumps both test-side and
mount-side (via pprof) goroutine stacks and marks the test failed
via t.Errorf.
* fix(mount): IsComplete must detect coverage across adjacent intervals
Linux FUSE caps per-op writes at FUSE_MAX_PAGES_PER_REQ (typically
1 MiB on x86_64) regardless of go-fuse's requested MaxWrite, so a
2 MiB chunk filled by a sequential writer arrives as two adjacent
1 MiB write ops. addInterval in ChunkWrittenIntervalList does not
merge adjacent intervals, so the resulting list has two elements
{[0,1M], [1M,2M]} — fully covered, but list.size()==2.
IsComplete previously returned `list.size() == 1 &&
list.head.next.isComplete(chunkSize)`, which required a single
interval covering [0, chunkSize). Under that rule, chunks filled by
adjacent writes never reach IsComplete==true, so maybeMoveToSealed
never fires, and the chunks sit in writableChunks until
FlushAll/close. SaveContent handles the adjacency correctly via its
inline merge loop, so uploads work once they're triggered — but
IsComplete is the gate that triggers them.
This was a latent bug: without the write-buffer cap, the overflow
path kicks in at writableChunkLimit (default 128) and force-seals
chunks, hiding the leak. #9066's -writeBufferSizeMB adds a tighter
global cap, and with 8 slots / 20 MiB test, the budget trips long
before overflow. The writer blocks in Reserve, waiting for a slot
that never frees because no uploader ever ran — observed in the CI
run 24390596623 mount pprof dump: goroutine 1 stuck in
WriteBufferAccountant.Reserve → cond.Wait, zero uploader goroutines
anywhere in the 89-goroutine dump.
Walk the (sorted) interval list tracking the furthest covered
offset; return true if coverage reaches chunkSize with no gaps. This
correctly handles adjacent intervals, overlapping intervals, and
out-of-order inserts. Added TestIsComplete_AdjacentIntervals
covering single-write, two adjacent halves (both orderings), eight
adjacent eighths, gaps, missing edges, and overlaps.
* test(fuse): route mount glog to stderr + dump mount on any write error
Run 24392087737 (with the IsComplete fix) no longer hangs on Linux —
huge progress. Now TestWriteBufferCap/LargeFileUnderCap fails with
'close(...write_buffer_cap_large.bin): input/output error', meaning
a chunk upload failed and pages.lastErr propagated via FlushData to
close(). But the mount log in the CI artifact is empty because weed
mount's glog defaults to /tmp/weed.* files, which the CI upload step
never sees, so we can't tell WHICH upload failed or WHY.
Add -logtostderr=true -v=2 to MountOptions so glog output goes to
the mount process's stderr, which the framework's startProcess
redirects into f.logDir/mount.log, which the framework's DumpLogs
then prints to the test output on failure. The -v=2 floor enables
saveDataAsChunk upload errors (currently logged at V(0)) plus the
medium-level write_pipeline/upload traces without drowning the log
in V(4) noise.
Also dump MOUNT goroutines on any writeWithTimeout error (not just
timeout). The IsComplete fix means we now get explicit errors
instead of silent hangs, and the goroutine dump at the error moment
shows in-flight upload state (pending sealed chunks, retry loops,
etc) that a post-failure log alone can't capture.
* test: consolidate port allocation into shared test/testutil package
Move duplicated port allocation logic from 15+ test files into a single
shared package at test/testutil/. This fixes a port collision bug where
independently allocated ports could overlap via the gRPC offset
(port+10000), causing weed mini to reject the configuration.
The shared package provides:
- AllocatePorts: atomic allocation of N unique ports
- AllocateMiniPorts/MustFreeMiniPorts: gRPC-offset-aware allocation
that prevents port A+10000 == port B collisions
- WaitForPort, WaitForService, FindBindIP, WriteIAMConfig, HasDocker
* test: address review feedback and fix FUSE build
- Revert fuse_integration change: it has its own go.mod and cannot
import the shared testutil package
- AllocateMiniPorts: hold all listeners open until the entire batch is
allocated, preventing race conditions where other processes steal ports
- HasDocker: add 5s context timeout to avoid hanging on stalled Docker
- WaitForService: only treat 2xx HTTP status codes as ready
* test: use global rand in AllocateMiniPorts for better seeding
Go 1.20+ auto-seeds the global rand generator. Using it avoids
identical sequences when multiple tests call at the same nanosecond.
* test: revert WaitForService status code check
S3 endpoints return non-2xx (e.g. 403) on bare GET requests, so
requiring 2xx caused the S3 integration test to time out. Any HTTP
response is sufficient proof that the service is running.
* test: fix gofmt formatting in s3tables test files
- concurrent_operations_test: Add retry loop for transient I/O errors
on file close during ConcurrentDirectoryOperations
- git_operations_test: Wait for pushed objects to become visible through
FUSE mount before cloning in Phase 3
After git reset --hard on a FUSE mount, the kernel dcache can
transiently show the directory then drop it moments later. Add a
1-second stabilisation delay and re-verification in
resetToCommitWithRecovery and tryPullFromCommit so that recovery
retries if the entry vanishes in that window.
Chris Lu