mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-19 02:15:02 +01:00
* mount: move an open file's read view to the chunks a merge wrote When a flush finds that a file's chunks take more than twice its size, maybeMergeChunks reads the file back and writes it again as new chunks. The commit that publishes them makes the filer delete the chunks they replaced. The flush put the new chunks into the handle's entry but left its read view on the old ones, so every descriptor that kept the file open went on reading the replaced chunks. Those read back until their volume is vacuumed. After that a read first retries the fetch of the missing needle for about 13 s; then refreshEntryForRead installs the filer's entry and the read is retried. The refresh skips a handle with writes it has not flushed, and such a read returns EIO instead. Files that are kept open and rewritten in place, such as PostgreSQL data files and SQLite databases, are the ones whose chunks bloat, so they are the ones that hit this. installRewrittenChunks puts the merge's chunks into the entry and the read view together, under the entry lock. Chunks attached while the merge ran were not part of it and stay. Flushes without a merge are unchanged. TestMergeMovesReadViewOffReplacedChunks covers the handle. TestReadOpenHandleAfterChunkMerge runs it on a real mount, with and without an unflushed write; for it, the fuse_integration framework starts mini on a free master port and exposes the address. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: drain uploads before a merge replaces the chunks Uploads attach their chunks without the handle lock, first to the entry and then to the read view, and writes that landed after doFlush drained the pages may still be uploading when the flush takes its lock. An upload that attached during the merge could keep a chunk older than data the merge already holds, or put a chunk the merge replaced back into the read view. Either outranks the merge's zero-timestamp chunk. When the chunks are bloated enough to merge, the flush now drains the pages again while its lock holds writes off, and takes the chunks again. FlushAll returns only after every upload has attached to both the entry and the read view, so nothing attaches during the merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: leave a pre-merge upload failure to the next flush The drain before a merge returned an upload failure through the metadata flush. doFlush then retried it as a filer error and answered EIO, where the page flush answers a full disk with ENOSPC. The drain is there only to wait for the uploads: a failure stays with the pages, and the next flush reports it as a write error, as it would without a merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep chunks attached while a flush installs its compacted list A write that lands after doFlush drained the pages can still be uploading when flushMetadataToFiler takes its chunk snapshot. Its upload attaches the chunk without the handle lock, and the plain flush then replaced the entry's chunks with the list compacted from the snapshot, dropping it from the handle and from every later commit. Keep chunks attached after the snapshot, as the merge path already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: fail a merge's flush on a failed drain A failed upload found by the drain before a merge was only logged, and the flush went on to publish a size covering bytes that no chunk stores. It now fails the flush as a pageUploadError before anything is published; doFlush maps it with writeErrorToFuseStatus, as it does its own FlushData failure, and does not retry it as a filer error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: clone a flush's request entry under the entry lock The request snapshot cloned the live entry after GetEntry had released its read lock. Uploads attach chunks without the handle lock, so an AppendChunks that landed during the clone raced it on the Chunks field. Clone inside UpdateEntry, as content flush recovery does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: leave a merge to a later flush while uploads are pending Before a merge the flush drained the pages while it held the handle and publication locks. A writeback upload retries a transient storage error until it succeeds, so that drain could hold both locks for as long as storage stayed down, stalling reads, writes and stat on the handle. The flush now checks for pending uploads and, if any, publishes the compacted chunks and leaves the merge to a later flush. With none pending the lock still holds writes off, so nothing attaches during the merge, and a failed upload still fails the flush as a write error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep room for pending uploads when a flush merges onto a moved entry A flush that leaves the chunk merge to a later flush publishes while uploads are still pending, and writes that landed after the drain can be uploading on any flush. Merging a peer's inline content or a shrink into such a publication lets the upload attach its chunk next to the inline bytes, where reads never reach it. Tell createContentEntry when uploads are pending so the merge keeps room for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test chunk merges through the filer's chunk proxy, also under load The tests serve the filer's chunk proxy, and like the filer the fake deletes the chunks each commit replaces, so a read view still naming them fails: - TestFlushMergeThroughChunkProxy/Quiet: the flush merges, and reads through the open handle return the file from the merged chunk alone. - TestFlushMergeThroughChunkProxy/UploadInFlight: an older page is still uploading while a newer page of the same bytes has attached. The flush leaves the merge, reads return the newer page, and the next flush merges. - TestFlushMergeRacesWithWritesAndReads: four writers, three flushers and a reader on one handle, with uploads delayed at random. The writers pause now and then, so 14 to 18 flushes a run merge and the others leave the merge to a later one. Every read must succeed within 2 s, and at the end the handle and the filer must both hold exactly what was written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: let a held upload go when UploadInFlight fails The subtest's proxy holds the older page's upload until the test releases it, and only the passing path did. A failure in between left the handler blocked, so the proxy's cleanup waited on it and the failure surfaced as a suite timeout. Release it from a cleanup as well, once, and bound the waits for the upload and the drains. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
660 lines
20 KiB
Go
660 lines
20 KiB
Go
package fuse_test
|
|
|
|
import (
|
|
"fmt"
|
|
"io/fs"
|
|
"net"
|
|
"os"
|
|
"os/exec"
|
|
"path/filepath"
|
|
"strconv"
|
|
"strings"
|
|
"syscall"
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/stretchr/testify/require"
|
|
)
|
|
|
|
// FuseTestFramework provides utilities for FUSE integration testing.
|
|
// It starts a single "weed mini" process (master+volume+filer in one)
|
|
// and a separate "weed mount" process for the FUSE filesystem.
|
|
type FuseTestFramework struct {
|
|
t *testing.T
|
|
tempDir string
|
|
mountPoint string
|
|
dataDir string
|
|
logDir string
|
|
miniProcess *managedProcess
|
|
mountProcess *managedProcess
|
|
filerAddr string
|
|
filerPort int
|
|
masterPort int
|
|
weedBinary string
|
|
isSetup bool
|
|
config *TestConfig
|
|
}
|
|
|
|
// TestConfig holds configuration for FUSE tests
|
|
type TestConfig struct {
|
|
Collection string
|
|
Replication string
|
|
ChunkSizeMB int
|
|
CacheSizeMB int
|
|
NumVolumes int
|
|
EnableDebug bool
|
|
// MountGlobalOptions are glog flags (-v, -logtostderr, ...) registered on
|
|
// weed's global flagset; they must precede the subcommand name or the
|
|
// mount process dies at flag parsing.
|
|
MountGlobalOptions []string
|
|
MountOptions []string
|
|
SkipCleanup bool // for debugging failed tests
|
|
}
|
|
|
|
// DefaultTestConfig returns a default configuration for FUSE tests
|
|
func DefaultTestConfig() *TestConfig {
|
|
return &TestConfig{
|
|
Collection: "",
|
|
Replication: "000",
|
|
ChunkSizeMB: 4,
|
|
CacheSizeMB: 100,
|
|
NumVolumes: 3,
|
|
EnableDebug: false,
|
|
MountOptions: []string{},
|
|
SkipCleanup: false,
|
|
}
|
|
}
|
|
|
|
// NewFuseTestFramework creates a new FUSE testing framework.
|
|
func NewFuseTestFramework(t *testing.T, config *TestConfig) *FuseTestFramework {
|
|
if config == nil {
|
|
config = DefaultTestConfig()
|
|
}
|
|
|
|
tempDir, err := os.MkdirTemp("", "seaweedfs_fuse_test_")
|
|
require.NoError(t, err)
|
|
|
|
filerPort := freePort(t)
|
|
masterPort := freePort(t)
|
|
for masterPort == filerPort {
|
|
masterPort = freePort(t)
|
|
}
|
|
|
|
return &FuseTestFramework{
|
|
t: t,
|
|
tempDir: tempDir,
|
|
mountPoint: filepath.Join(tempDir, "mount"),
|
|
dataDir: filepath.Join(tempDir, "data"),
|
|
logDir: filepath.Join(tempDir, "logs"),
|
|
filerPort: filerPort,
|
|
filerAddr: fmt.Sprintf("127.0.0.1:%d", filerPort),
|
|
masterPort: masterPort,
|
|
weedBinary: findWeedBinary(),
|
|
isSetup: false,
|
|
config: config,
|
|
}
|
|
}
|
|
|
|
// grpcPortOffset mirrors weed's HTTP->gRPC port convention (gRPC = HTTP + offset);
|
|
// "weed mount" derives the filer gRPC port from the filer HTTP address the same way.
|
|
const grpcPortOffset = 10000
|
|
|
|
// freePort returns a free filer HTTP port whose gRPC sibling (port+grpcPortOffset)
|
|
// is also free. Both must stay below the Linux ephemeral floor (32768): a gRPC port
|
|
// above it can be transiently grabbed by an outbound connection, forcing mini to
|
|
// relocate its filer gRPC port while "weed mount" keeps dialing HTTP+10000 — the
|
|
// mount then never connects and the test times out. Capping HTTP at 22000 keeps the
|
|
// gRPC port at or below 32000.
|
|
func freePort(t *testing.T) int {
|
|
t.Helper()
|
|
const (
|
|
minServicePort = 20000
|
|
maxServicePort = 22000
|
|
)
|
|
|
|
portCount := maxServicePort - minServicePort + 1
|
|
start := minServicePort + int(time.Now().UnixNano()%int64(portCount))
|
|
|
|
for attempt := 0; attempt < portCount; attempt++ {
|
|
port := minServicePort + (start-minServicePort+attempt)%portCount
|
|
if portIsFree(port) && portIsFree(port+grpcPortOffset) {
|
|
return port
|
|
}
|
|
}
|
|
|
|
t.Fatalf("failed to allocate a free HTTP/gRPC port pair in [%d,%d]", minServicePort, maxServicePort)
|
|
return 0
|
|
}
|
|
|
|
// portIsFree reports whether a TCP port can currently be bound on 127.0.0.1.
|
|
func portIsFree(port int) bool {
|
|
l, err := net.Listen("tcp", net.JoinHostPort("127.0.0.1", strconv.Itoa(port)))
|
|
if err != nil {
|
|
return false
|
|
}
|
|
l.Close()
|
|
return true
|
|
}
|
|
|
|
// Setup starts "weed mini" and mounts the FUSE filesystem.
|
|
func (f *FuseTestFramework) Setup(config *TestConfig) error {
|
|
if f.isSetup {
|
|
return fmt.Errorf("framework already setup")
|
|
}
|
|
if config != nil {
|
|
f.config = config
|
|
}
|
|
config = f.config
|
|
|
|
dirs := []string{f.mountPoint, f.logDir, f.dataDir}
|
|
for _, dir := range dirs {
|
|
if err := os.MkdirAll(dir, 0755); err != nil {
|
|
return fmt.Errorf("failed to create directory %s: %v", dir, err)
|
|
}
|
|
}
|
|
|
|
// Start weed mini (master + volume + filer in one process)
|
|
if err := f.startMini(config); err != nil {
|
|
return fmt.Errorf("failed to start weed mini: %v", err)
|
|
}
|
|
|
|
// Wait for filer to be ready (mini starts all services on filerPort)
|
|
if err := f.waitForService(f.miniProcess, f.filerAddr, 90*time.Second); err != nil {
|
|
f.dumpLog("mini")
|
|
return fmt.Errorf("weed mini not ready: %v", err)
|
|
}
|
|
|
|
// Mount FUSE filesystem
|
|
if err := f.mountFuse(config); err != nil {
|
|
return fmt.Errorf("failed to mount FUSE: %v", err)
|
|
}
|
|
|
|
// Wait for mount to be ready
|
|
if err := f.waitForMount(90 * time.Second); err != nil {
|
|
f.dumpLog("mount")
|
|
return fmt.Errorf("FUSE mount not ready: %v", err)
|
|
}
|
|
|
|
// Gate on a real write: a fresh mini still cannot assign a file id until
|
|
// its volume registers and a grow round finishes.
|
|
if err := f.waitForWritable(90 * time.Second); err != nil {
|
|
f.dumpLog("mount")
|
|
return fmt.Errorf("FUSE mount not writable: %v", err)
|
|
}
|
|
|
|
f.isSetup = true
|
|
return nil
|
|
}
|
|
|
|
// Cleanup stops all processes and removes temporary files.
|
|
// If the test failed, it dumps logs automatically.
|
|
func (f *FuseTestFramework) Cleanup() {
|
|
if f.t.Failed() {
|
|
f.DumpLogs()
|
|
}
|
|
|
|
// Stop processes in reverse order
|
|
f.unmountFuse()
|
|
if f.miniProcess != nil {
|
|
f.miniProcess.stop()
|
|
f.miniProcess = nil
|
|
}
|
|
|
|
f.copyLogsForCI()
|
|
|
|
if !f.config.SkipCleanup {
|
|
os.RemoveAll(f.tempDir)
|
|
}
|
|
}
|
|
|
|
// DumpLogs prints the tail of all SeaweedFS process logs to test output.
|
|
func (f *FuseTestFramework) DumpLogs() {
|
|
for _, name := range []string{"mini", "mount"} {
|
|
f.dumpLog(name)
|
|
}
|
|
}
|
|
|
|
// GetMountPoint returns the FUSE mount point path
|
|
func (f *FuseTestFramework) GetMountPoint() string {
|
|
return f.mountPoint
|
|
}
|
|
|
|
// GetFilerAddr returns the filer address
|
|
func (f *FuseTestFramework) GetFilerAddr() string {
|
|
return f.filerAddr
|
|
}
|
|
|
|
// GetMasterAddr returns the master HTTP address of the mini cluster.
|
|
func (f *FuseTestFramework) GetMasterAddr() string {
|
|
return fmt.Sprintf("127.0.0.1:%d", f.masterPort)
|
|
}
|
|
|
|
// managedProcess is a started weed sub-command whose exit is watched, so a wait
|
|
// for it to come up ends the moment it dies instead of burning its full timeout.
|
|
type managedProcess struct {
|
|
cmd *exec.Cmd
|
|
done chan struct{}
|
|
err error // exit error, read only after done is closed
|
|
}
|
|
|
|
// exited returns the exit error once the process is gone, nil while it runs.
|
|
func (p *managedProcess) exited() error {
|
|
select {
|
|
case <-p.done:
|
|
if p.err != nil {
|
|
return p.err
|
|
}
|
|
return fmt.Errorf("exit status 0")
|
|
default:
|
|
return nil
|
|
}
|
|
}
|
|
|
|
// stop asks the process to terminate and waits for it to go away.
|
|
func (p *managedProcess) stop() {
|
|
// Signal fails with os.ErrProcessDone when the child is already gone, which
|
|
// is exactly the case the select below handles.
|
|
_ = p.cmd.Process.Signal(syscall.SIGTERM)
|
|
select {
|
|
case <-p.done:
|
|
case <-time.After(10 * time.Second):
|
|
p.cmd.Process.Kill()
|
|
<-p.done
|
|
}
|
|
}
|
|
|
|
// startProcess is a helper that starts a weed sub-command with output captured
|
|
// to a log file in f.logDir.
|
|
func (f *FuseTestFramework) startProcess(name string, args []string) (*managedProcess, error) {
|
|
logFile, err := os.Create(filepath.Join(f.logDir, name+".log"))
|
|
if err != nil {
|
|
return nil, fmt.Errorf("create log file: %v", err)
|
|
}
|
|
cmd := exec.Command(f.weedBinary, args...)
|
|
cmd.Dir = f.tempDir
|
|
cmd.Stdout = logFile
|
|
cmd.Stderr = logFile
|
|
if err := cmd.Start(); err != nil {
|
|
logFile.Close()
|
|
return nil, err
|
|
}
|
|
// Close the file handle — the child process inherited it.
|
|
logFile.Close()
|
|
|
|
p := &managedProcess{cmd: cmd, done: make(chan struct{})}
|
|
go func() {
|
|
p.err = cmd.Wait()
|
|
close(p.done)
|
|
}()
|
|
return p, nil
|
|
}
|
|
|
|
// dumpLog prints the last lines of a process log file to the test output
|
|
// for debugging when a service fails to start or a test fails.
|
|
func (f *FuseTestFramework) dumpLog(name string) {
|
|
data, err := os.ReadFile(filepath.Join(f.logDir, name+".log"))
|
|
if err != nil {
|
|
f.t.Logf("[%s log] (not available: %v)", name, err)
|
|
return
|
|
}
|
|
// Show last 16KB on failure for meaningful context.
|
|
const maxTail = 16 * 1024
|
|
if len(data) > maxTail {
|
|
data = data[len(data)-maxTail:]
|
|
}
|
|
f.t.Logf("[%s log tail (%d bytes)]\n%s", name, len(data), string(data))
|
|
}
|
|
|
|
// copyLogsForCI copies SeaweedFS process logs under a per-test directory in
|
|
// /tmp/seaweedfs-fuse-logs/ so the CI workflow can upload them as artifacts.
|
|
// Each framework instance gets its own directory: every test in the suite
|
|
// builds one, and fixed filenames would let a later passing test overwrite
|
|
// the logs of an earlier failing one.
|
|
func (f *FuseTestFramework) copyLogsForCI() {
|
|
ciLogDir := filepath.Join("/tmp/seaweedfs-fuse-logs", strings.NewReplacer("/", "_", " ", "_").Replace(f.t.Name()))
|
|
if err := os.MkdirAll(ciLogDir, 0755); err != nil {
|
|
return
|
|
}
|
|
for _, name := range []string{"mini", "mount"} {
|
|
src := filepath.Join(f.logDir, name+".log")
|
|
data, err := os.ReadFile(src)
|
|
if err != nil {
|
|
continue
|
|
}
|
|
os.WriteFile(filepath.Join(ciLogDir, name+".log"), data, 0644)
|
|
}
|
|
}
|
|
|
|
// startMini starts "weed mini" which runs master+volume+filer in one process.
|
|
func (f *FuseTestFramework) startMini(config *TestConfig) error {
|
|
var args []string
|
|
if config.EnableDebug {
|
|
args = append(args, "-v=4")
|
|
}
|
|
adminPort := freePort(f.t)
|
|
for adminPort == f.filerPort || adminPort == f.masterPort {
|
|
adminPort = freePort(f.t)
|
|
}
|
|
args = append(args,
|
|
"mini",
|
|
"-dir="+f.dataDir,
|
|
"-ip=127.0.0.1",
|
|
"-ip.bind=127.0.0.1",
|
|
"-filer.port="+strconv.Itoa(f.filerPort),
|
|
"-master.port="+strconv.Itoa(f.masterPort),
|
|
"-master.port.grpc="+strconv.Itoa(f.masterPort+grpcPortOffset),
|
|
"-s3=false",
|
|
"-webdav=false",
|
|
"-admin.ui=false",
|
|
"-admin.port="+strconv.Itoa(adminPort),
|
|
"-admin.port.grpc="+strconv.Itoa(adminPort+grpcPortOffset),
|
|
"-admin.port.legacy=0",
|
|
)
|
|
|
|
proc, err := f.startProcess("mini", args)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
f.miniProcess = proc
|
|
return nil
|
|
}
|
|
|
|
// mountFuse mounts the SeaweedFS FUSE filesystem
|
|
func (f *FuseTestFramework) mountFuse(config *TestConfig) error {
|
|
args := append([]string{}, config.MountGlobalOptions...)
|
|
if config.EnableDebug {
|
|
args = append(args, "-v=4")
|
|
}
|
|
args = append(args,
|
|
"mount",
|
|
"-filer=127.0.0.1:"+strconv.Itoa(f.filerPort),
|
|
"-dir="+f.mountPoint,
|
|
"-filer.path=/",
|
|
"-dirAutoCreate",
|
|
"-allowOthers=false",
|
|
)
|
|
|
|
if config.Collection != "" {
|
|
args = append(args, "-collection="+config.Collection)
|
|
}
|
|
if config.Replication != "" {
|
|
args = append(args, "-replication="+config.Replication)
|
|
}
|
|
if config.ChunkSizeMB > 0 {
|
|
args = append(args, fmt.Sprintf("-chunkSizeLimitMB=%d", config.ChunkSizeMB))
|
|
}
|
|
if config.CacheSizeMB > 0 {
|
|
args = append(args, fmt.Sprintf("-cacheCapacityMB=%d", config.CacheSizeMB))
|
|
}
|
|
|
|
args = append(args, config.MountOptions...)
|
|
|
|
proc, err := f.startProcess("mount", args)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
f.mountProcess = proc
|
|
return nil
|
|
}
|
|
|
|
// unmountFuse unmounts the FUSE filesystem
|
|
func (f *FuseTestFramework) unmountFuse() error {
|
|
if f.mountProcess != nil {
|
|
f.mountProcess.stop()
|
|
f.mountProcess = nil
|
|
}
|
|
|
|
// Also try system unmount as backup
|
|
exec.Command("fusermount3", "-u", f.mountPoint).Run()
|
|
exec.Command("fusermount", "-u", f.mountPoint).Run()
|
|
return nil
|
|
}
|
|
|
|
// waitForService waits for a service to be available
|
|
func (f *FuseTestFramework) waitForService(proc *managedProcess, addr string, timeout time.Duration) error {
|
|
deadline := time.Now().Add(timeout)
|
|
for time.Now().Before(deadline) {
|
|
conn, err := net.DialTimeout("tcp", addr, 1*time.Second)
|
|
if err == nil {
|
|
conn.Close()
|
|
return nil
|
|
}
|
|
if exitErr := proc.exited(); exitErr != nil {
|
|
return fmt.Errorf("process exited (%v) before %s accepted connections", exitErr, addr)
|
|
}
|
|
time.Sleep(100 * time.Millisecond)
|
|
}
|
|
return fmt.Errorf("service at %s not ready within timeout", addr)
|
|
}
|
|
|
|
// waitForMount waits for the FUSE mount to be ready. A stat/ReadDir probe
|
|
// alone is not enough: the bare mount point directory passes both before the
|
|
// mount process finishes starting, letting tests race ahead and write to the
|
|
// local disk underneath the mount. The mount point's device ID differing from
|
|
// its parent's confirms a filesystem is actually mounted there.
|
|
func (f *FuseTestFramework) waitForMount(timeout time.Duration) error {
|
|
parentDev, err := deviceID(filepath.Dir(f.mountPoint))
|
|
if err != nil {
|
|
return fmt.Errorf("stat mount point parent: %v", err)
|
|
}
|
|
deadline := time.Now().Add(timeout)
|
|
for time.Now().Before(deadline) {
|
|
if dev, err := deviceID(f.mountPoint); err == nil && dev != parentDev {
|
|
if _, err := os.ReadDir(f.mountPoint); err == nil {
|
|
return nil
|
|
}
|
|
}
|
|
// A mount that cannot mount at all (no /dev/fuse, fusermount not setuid)
|
|
// dies within a second; reporting that beats waiting out the timeout.
|
|
if exitErr := f.mountProcess.exited(); exitErr != nil {
|
|
return fmt.Errorf("mount process exited (%v)", exitErr)
|
|
}
|
|
time.Sleep(100 * time.Millisecond)
|
|
}
|
|
return fmt.Errorf("mount point not ready within timeout")
|
|
}
|
|
|
|
// waitForWritable probes the mount with a real write+read+remove until one
|
|
// succeeds, covering the window where a fresh cluster cannot yet assign a
|
|
// file id (volume not registered, grow pending, leader settle). Each attempt
|
|
// runs off the loop because a wedged mount can hold a file op open, and one
|
|
// attempt stays in flight until it finishes or the overall deadline passes.
|
|
func (f *FuseTestFramework) waitForWritable(timeout time.Duration) error {
|
|
probe := filepath.Join(f.mountPoint, ".framework_write_probe")
|
|
payload := []byte("probe")
|
|
deadline := time.Now().Add(timeout)
|
|
var lastErr error
|
|
for time.Now().Before(deadline) {
|
|
if exitErr := f.mountProcess.exited(); exitErr != nil {
|
|
return fmt.Errorf("mount process exited (%v)", exitErr)
|
|
}
|
|
result := make(chan error, 1)
|
|
go func() {
|
|
if err := os.WriteFile(probe, payload, 0644); err != nil {
|
|
result <- err
|
|
return
|
|
}
|
|
data, err := os.ReadFile(probe)
|
|
if err != nil {
|
|
result <- err
|
|
return
|
|
}
|
|
if len(data) != len(payload) {
|
|
result <- fmt.Errorf("read back %d bytes, want %d", len(data), len(payload))
|
|
return
|
|
}
|
|
result <- os.Remove(probe)
|
|
}()
|
|
tick := time.NewTicker(500 * time.Millisecond)
|
|
attemptOpen := true
|
|
for attemptOpen {
|
|
select {
|
|
case err := <-result:
|
|
attemptOpen = false
|
|
if err == nil {
|
|
tick.Stop()
|
|
return nil
|
|
}
|
|
lastErr = err
|
|
case <-tick.C:
|
|
if exitErr := f.mountProcess.exited(); exitErr != nil {
|
|
tick.Stop()
|
|
return fmt.Errorf("mount process exited (%v)", exitErr)
|
|
}
|
|
if !time.Now().Before(deadline) {
|
|
tick.Stop()
|
|
if lastErr == nil {
|
|
return fmt.Errorf("write probe did not complete within timeout")
|
|
}
|
|
return fmt.Errorf("write probe failed within timeout: %v", lastErr)
|
|
}
|
|
}
|
|
}
|
|
tick.Stop()
|
|
time.Sleep(500 * time.Millisecond)
|
|
}
|
|
return fmt.Errorf("write probe failed within timeout: %v", lastErr)
|
|
}
|
|
|
|
// deviceID returns the device ID of the filesystem containing path.
|
|
func deviceID(path string) (uint64, error) {
|
|
var st syscall.Stat_t
|
|
if err := syscall.Stat(path, &st); err != nil {
|
|
return 0, err
|
|
}
|
|
return uint64(st.Dev), nil
|
|
}
|
|
|
|
// findWeedBinary locates the weed binary.
|
|
func findWeedBinary() string {
|
|
if p, err := exec.LookPath("weed"); err == nil {
|
|
return p
|
|
}
|
|
|
|
candidates := []string{
|
|
"../../weed/weed",
|
|
"./weed",
|
|
"../weed",
|
|
}
|
|
for _, candidate := range candidates {
|
|
if info, err := os.Stat(candidate); err == nil && !info.IsDir() {
|
|
abs, _ := filepath.Abs(candidate)
|
|
return abs
|
|
}
|
|
}
|
|
|
|
return "weed"
|
|
}
|
|
|
|
// Helper functions for test assertions
|
|
|
|
// AssertFileExists checks if a file exists in the mount point. On failure,
|
|
// it gathers diagnostic state (retry stat, parent listing, direct open) so
|
|
// transient FUSE flakes leave enough information to identify the layer that
|
|
// dropped the entry. The diagnostic block runs only on the failure path.
|
|
func (f *FuseTestFramework) AssertFileExists(relativePath string) {
|
|
fullPath := filepath.Join(f.mountPoint, relativePath)
|
|
_, err := os.Stat(fullPath)
|
|
if err == nil {
|
|
return
|
|
}
|
|
f.dumpExistenceDiagnostics(relativePath, fullPath, err)
|
|
require.NoError(f.t, err, "file should exist: %s (see diagnostic logs above)", relativePath)
|
|
}
|
|
|
|
// dumpExistenceDiagnostics is invoked when AssertFileExists fails. It probes
|
|
// whether the missing entry is transient (retries appear), missing from the
|
|
// parent directory listing, or missing via a direct open syscall — three
|
|
// signals that triangulate where the entry was lost (kernel dentry cache vs
|
|
// mount lookup vs filer).
|
|
func (f *FuseTestFramework) dumpExistenceDiagnostics(relativePath, fullPath string, initialErr error) {
|
|
f.t.Logf("AssertFileExists diagnostic dump for %s", relativePath)
|
|
f.t.Logf(" initial stat: err=%v isNotExist=%v", initialErr, os.IsNotExist(initialErr))
|
|
|
|
// Retry stat to detect transient invisibility — if it becomes visible
|
|
// the issue is a short-lived dentry/cache window; if it stays missing
|
|
// the entry is genuinely gone from the filer or the local cache.
|
|
for attempt := 1; attempt <= 5; attempt++ {
|
|
time.Sleep(100 * time.Millisecond)
|
|
_, err := os.Stat(fullPath)
|
|
f.t.Logf(" retry #%d after %dms: err=%v", attempt, attempt*100, err)
|
|
if err == nil {
|
|
break
|
|
}
|
|
}
|
|
|
|
// List the parent directory: if the file appears here but stat fails,
|
|
// the failure is in the kernel's per-name dentry cache; if not, the
|
|
// mount-side LOOKUP itself is dropping the entry.
|
|
parentDir := filepath.Dir(fullPath)
|
|
target := filepath.Base(fullPath)
|
|
if entries, listErr := os.ReadDir(parentDir); listErr != nil {
|
|
f.t.Logf(" ReadDir(%s) failed: %v", parentDir, listErr)
|
|
} else {
|
|
found := false
|
|
names := make([]string, 0, len(entries))
|
|
for _, e := range entries {
|
|
names = append(names, e.Name())
|
|
if e.Name() == target {
|
|
found = true
|
|
}
|
|
}
|
|
f.t.Logf(" ReadDir(%s): %d entries, target %q present=%v", parentDir, len(entries), target, found)
|
|
if len(names) <= 32 {
|
|
f.t.Logf(" ReadDir entries: %v", names)
|
|
}
|
|
}
|
|
|
|
// Direct O_RDONLY open — exercises the same FUSE Lookup+Open path
|
|
// userspace would, but separately from stat, in case stat-only caching
|
|
// is interfering.
|
|
if fd, openErr := os.OpenFile(fullPath, os.O_RDONLY, 0); openErr != nil {
|
|
f.t.Logf(" Open(%s, O_RDONLY): %v", fullPath, openErr)
|
|
} else {
|
|
st, statErr := fd.Stat()
|
|
fd.Close()
|
|
size := int64(-1)
|
|
if statErr == nil && st != nil {
|
|
size = st.Size()
|
|
}
|
|
f.t.Logf(" Open(%s, O_RDONLY): success size=%d statErr=%v", fullPath, size, statErr)
|
|
}
|
|
}
|
|
|
|
// AssertFileNotExists checks if a file does not exist in the mount point
|
|
func (f *FuseTestFramework) AssertFileNotExists(relativePath string) {
|
|
fullPath := filepath.Join(f.mountPoint, relativePath)
|
|
_, err := os.Stat(fullPath)
|
|
require.True(f.t, os.IsNotExist(err), "file should not exist: %s", relativePath)
|
|
}
|
|
|
|
// AssertFileContent checks if a file has expected content
|
|
func (f *FuseTestFramework) AssertFileContent(relativePath string, expectedContent []byte) {
|
|
fullPath := filepath.Join(f.mountPoint, relativePath)
|
|
actualContent, err := os.ReadFile(fullPath)
|
|
require.NoError(f.t, err, "failed to read file: %s", relativePath)
|
|
require.Equal(f.t, expectedContent, actualContent, "file content mismatch: %s", relativePath)
|
|
}
|
|
|
|
// AssertFileMode checks if a file has expected permissions
|
|
func (f *FuseTestFramework) AssertFileMode(relativePath string, expectedMode fs.FileMode) {
|
|
fullPath := filepath.Join(f.mountPoint, relativePath)
|
|
info, err := os.Stat(fullPath)
|
|
require.NoError(f.t, err, "failed to stat file: %s", relativePath)
|
|
require.Equal(f.t, expectedMode, info.Mode(), "file mode mismatch: %s", relativePath)
|
|
}
|
|
|
|
// CreateTestFile creates a test file with specified content
|
|
func (f *FuseTestFramework) CreateTestFile(relativePath string, content []byte) {
|
|
fullPath := filepath.Join(f.mountPoint, relativePath)
|
|
dir := filepath.Dir(fullPath)
|
|
require.NoError(f.t, os.MkdirAll(dir, 0755), "failed to create directory: %s", dir)
|
|
require.NoError(f.t, os.WriteFile(fullPath, content, 0644), "failed to create file: %s", relativePath)
|
|
}
|
|
|
|
// CreateTestDir creates a test directory
|
|
func (f *FuseTestFramework) CreateTestDir(relativePath string) {
|
|
fullPath := filepath.Join(f.mountPoint, relativePath)
|
|
require.NoError(f.t, os.MkdirAll(fullPath, 0755), "failed to create directory: %s", relativePath)
|
|
}
|