test(fuse): two mounts staging the same files without -dlm (#12704)

* test(fuse): two mounts staging the same files without -dlm

Two jobs that extract the same archive into one shared directory, each
through its own mount and without -dlm, both create many of the same
files: a file that exists is skipped, otherwise it is created, written
and closed, then chmod'ed and given its mtime. Before #12593, #12594 and
#12596 the attribute update and the content flush sent the mount's whole
cached entry. Sent after the other mount had rewritten the file, the
update put back chunks that the rewrite had dropped and the filer had
deleted, so the entry pointed at deleted needles and the file failed to
read once its volume was vacuumed. Nothing told the writers.

TestStageSameFilesWithoutDLM runs that on the fuse_dlm cluster with the
mounts started without -dlm (startTestCluster takes a dlm flag;
startDLMTestCluster is unchanged) and checks every staged file: each
chunk is still on the volume server (HEAD on the fid, without
readDeleted) and the filer serves the bytes written. What the losing
writer is told depends on the version and is only logged.

With a 4.53 weed binary the test fails: 66, 71 and 70 of the 400 files
were broken in three runs. With master it passes in about 48 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fuse): bound the staging test's filer and volume checks

The checks used the default HTTP client and a background context, so a
server that accepts a request and stops answering held the test until the
package timeout instead of failing the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fuse): fail the staging test unless both mounts wrote the same files

A mount whose MkdirAll failed returned early, and a mount that started
late could skip every file, so the test passed with one writer. Start both
loops together, fail on setup errors, and require files both mounts wrote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fuse): bound the staging test's checks with one deadline

A per-call timeout still let 400 hung lookups and reads run for hours.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
This commit is contained in:
authored and GitHub committed 2026-11-17 17:08:39 -08:00
1 parent 0d3f5a220c
commit c76fb36416
2 files changed
+196 -8

No files matched your search

+19 -8
View File
@@ -28,11 +28,13 @@ import (
const filerGroup = "fuse-dlm-test"
// dlmTestCluster manages a full SeaweedFS cluster with 2 filers and 2 FUSE
// mounts for testing DLM-based cross-mount write coordination.
// mounts for testing DLM-based cross-mount write coordination, and what two
// mounts do without it.
type dlmTestCluster struct {
t testing.TB
baseDir string
weedBinary string
dlm bool // the mounts run with -dlm
masterPort int
masterGrpcPort int
@@ -52,6 +54,11 @@ type dlmTestCluster struct {
}
func startDLMTestCluster(t testing.TB) *dlmTestCluster {
return startTestCluster(t, true)
}
// startTestCluster starts the cluster; dlm says whether the mounts run with -dlm.
func startTestCluster(t testing.TB, dlm bool) *dlmTestCluster {
binary := findWeedBinary()
if binary == "" {
t.Skip("weed binary not found; set WEED_BINARY or ensure it is on PATH")
@@ -64,6 +71,7 @@ func startDLMTestCluster(t testing.TB) *dlmTestCluster {
t: t,
baseDir: baseDir,
weedBinary: binary,
dlm: dlm,
}
// Register cleanup early so processes are stopped even if a require fails below.
t.Cleanup(c.Stop)
@@ -224,17 +232,20 @@ func (c *dlmTestCluster) startMount(idx int, configDir string) error {
if err := os.MkdirAll(cacheDir, 0755); err != nil {
return fmt.Errorf("create cache dir: %w", err)
}
c.mountCmds[idx] = exec.Command(c.weedBinary,
"-logdir="+filepath.Join(c.baseDir, "logs"),
args := []string{
"-logdir=" + filepath.Join(c.baseDir, "logs"),
"mount",
"-filer="+c.filerAddress(0), // both mounts use filer0 for shared metadata
"-dir="+c.mountPoints[idx],
"-filer=" + c.filerAddress(0), // both mounts use filer0 for shared metadata
"-dir=" + c.mountPoints[idx],
"-filer.path=/",
"-dirAutoCreate",
"-allowOthers=false",
"-cacheDir="+cacheDir,
"-dlm",
)
"-cacheDir=" + cacheDir,
}
if c.dlm {
args = append(args, "-dlm")
}
c.mountCmds[idx] = exec.Command(c.weedBinary, args...)
return c.startCmd(c.mountCmds[idx], fmt.Sprintf("mount%d", idx))
}