Files
seaweedfs/test/fuse_integration
11af0e0c12 mount: move an open file's read view to the chunks a merge wrote (#12722)
* mount: move an open file's read view to the chunks a merge wrote

When a flush finds that a file's chunks take more than twice its size,
maybeMergeChunks reads the file back and writes it again as new chunks.
The commit that publishes them makes the filer delete the chunks they
replaced. The flush put the new chunks into the handle's entry but left
its read view on the old ones, so every descriptor that kept the file
open went on reading the replaced chunks. Those read back until their
volume is vacuumed. After that a read first retries the fetch of the
missing needle for about 13 s; then refreshEntryForRead installs the
filer's entry and the read is retried. The refresh skips a handle with
writes it has not flushed, and such a read returns EIO instead.

Files that are kept open and rewritten in place, such as PostgreSQL
data files and SQLite databases, are the ones whose chunks bloat, so
they are the ones that hit this.

installRewrittenChunks puts the merge's chunks into the entry and the
read view together, under the entry lock. Chunks attached while the
merge ran were not part of it and stay. Flushes without a merge are
unchanged.

TestMergeMovesReadViewOffReplacedChunks covers the handle.
TestReadOpenHandleAfterChunkMerge runs it on a real mount, with and
without an unflushed write; for it, the fuse_integration framework
starts mini on a free master port and exposes the address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drain uploads before a merge replaces the chunks

Uploads attach their chunks without the handle lock, first to the entry
and then to the read view, and writes that landed after doFlush drained
the pages may still be uploading when the flush takes its lock. An
upload that attached during the merge could keep a chunk older than
data the merge already holds, or put a chunk the merge replaced back
into the read view. Either outranks the merge's zero-timestamp chunk.

When the chunks are bloated enough to merge, the flush now drains the
pages again while its lock holds writes off, and takes the chunks again.
FlushAll returns only after every upload has attached to both the entry
and the read view, so nothing attaches during the merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave a pre-merge upload failure to the next flush

The drain before a merge returned an upload failure through the
metadata flush. doFlush then retried it as a filer error and answered
EIO, where the page flush answers a full disk with ENOSPC. The drain is
there only to wait for the uploads: a failure stays with the pages, and
the next flush reports it as a write error, as it would without a merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep chunks attached while a flush installs its compacted list

A write that lands after doFlush drained the pages can still be uploading
when flushMetadataToFiler takes its chunk snapshot. Its upload attaches the
chunk without the handle lock, and the plain flush then replaced the
entry's chunks with the list compacted from the snapshot, dropping it from
the handle and from every later commit. Keep chunks attached after the
snapshot, as the merge path already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fail a merge's flush on a failed drain

A failed upload found by the drain before a merge was only logged, and
the flush went on to publish a size covering bytes that no chunk stores.
It now fails the flush as a pageUploadError before anything is
published; doFlush maps it with writeErrorToFuseStatus, as it does its
own FlushData failure, and does not retry it as a filer error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: clone a flush's request entry under the entry lock

The request snapshot cloned the live entry after GetEntry had released
its read lock. Uploads attach chunks without the handle lock, so an
AppendChunks that landed during the clone raced it on the Chunks field.
Clone inside UpdateEntry, as content flush recovery does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave a merge to a later flush while uploads are pending

Before a merge the flush drained the pages while it held the handle and
publication locks. A writeback upload retries a transient storage error
until it succeeds, so that drain could hold both locks for as long as
storage stayed down, stalling reads, writes and stat on the handle.
The flush now checks for pending uploads and, if any, publishes the
compacted chunks and leaves the merge to a later flush. With none
pending the lock still holds writes off, so nothing attaches during the
merge, and a failed upload still fails the flush as a write error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep room for pending uploads when a flush merges onto a moved entry

A flush that leaves the chunk merge to a later flush publishes while
uploads are still pending, and writes that landed after the drain can be
uploading on any flush. Merging a peer's inline content or a shrink into
such a publication lets the upload attach its chunk next to the inline
bytes, where reads never reach it. Tell createContentEntry when uploads
are pending so the merge keeps room for them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test chunk merges through the filer's chunk proxy, also under load

The tests serve the filer's chunk proxy, and like the filer the fake
deletes the chunks each commit replaces, so a read view still naming
them fails:
- TestFlushMergeThroughChunkProxy/Quiet: the flush merges, and reads
  through the open handle return the file from the merged chunk alone.
- TestFlushMergeThroughChunkProxy/UploadInFlight: an older page is still
  uploading while a newer page of the same bytes has attached. The flush
  leaves the merge, reads return the newer page, and the next flush
  merges.
- TestFlushMergeRacesWithWritesAndReads: four writers, three flushers
  and a reader on one handle, with uploads delayed at random. The
  writers pause now and then, so 14 to 18 flushes a run merge and the
  others leave the merge to a later one. Every read must succeed within
  2 s, and at the end the handle and the filer must both hold exactly
  what was written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: let a held upload go when UploadInFlight fails

The subtest's proxy holds the older page's upload until the test
releases it, and only the passing path did. A failure in between left
the handler blocked, so the proxy's cleanup waited on it and the failure
surfaced as a suite timeout. Release it from a cleanup as well, once,
and bound the waits for the upload and the drains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 10:51:31 -08:00
..
2026-08-17 15:39:20 -07:00
2026-08-17 15:39:20 -07:00
2025-07-16 12:43:08 -07:00
2025-10-13 18:05:17 -07:00

SeaweedFS FUSE Integration Testing Framework

Overview

This directory contains a comprehensive integration testing framework for SeaweedFS FUSE operations. The current SeaweedFS FUSE tests are primarily performance-focused (using FIO) but lack comprehensive functional testing. This framework addresses those gaps.

⚠️ Current Status

Note: Due to Go module conflicts between this test framework and the parent SeaweedFS module, the full test suite currently requires manual setup. The framework files are provided as a foundation for comprehensive FUSE testing once the module structure is resolved.

Working Components

  • ✅ Framework design and architecture (framework.go)
  • ✅ Individual test file structure and compilation
  • ✅ Test methodology and comprehensive coverage
  • ✅ Documentation and usage examples
  • ⚠️ Full test suite execution (requires Go module isolation)

Verified Working Test

cd test/fuse_integration
go test -v simple_test.go

Current Testing Gaps Addressed

1. Limited Functional Coverage

  • Current: Only basic FIO performance tests
  • New: Comprehensive testing of all FUSE operations (create, read, write, delete, mkdir, rmdir, permissions, etc.)

2. No Concurrency Testing

  • Current: Single-threaded performance tests
  • New: Extensive concurrent operation tests, race condition detection, thread safety validation

3. Insufficient Error Handling

  • Current: Basic error scenarios
  • New: Comprehensive error condition testing, edge cases, failure recovery

4. Missing Edge Cases

  • Current: Simple file operations
  • New: Large files, sparse files, deep directory nesting, many small files, permission variations

Framework Architecture

Core Components

  1. framework.go - Test infrastructure and utilities

    • FuseTestFramework - Main test management struct
    • Automated SeaweedFS cluster setup/teardown
    • FUSE mount/unmount management
    • Helper functions for file operations and assertions
  2. basic_operations_test.go - Fundamental FUSE operations

    • File create, read, write, delete
    • File attributes and permissions
    • Large file handling
    • Sparse file operations
  3. directory_operations_test.go - Directory-specific tests

    • Directory creation, deletion, listing
    • Nested directory structures
    • Directory permissions and rename operations
    • Complex directory scenarios
  4. concurrent_operations_test.go - Concurrency and stress testing

    • Concurrent file and directory operations
    • Race condition detection
    • High-frequency operations
    • Stress testing scenarios

Key Features

Automated Test Environment

framework := NewFuseTestFramework(t, DefaultTestConfig())
defer framework.Cleanup()
require.NoError(t, framework.Setup(DefaultTestConfig()))
  • Automatic cluster setup: Master, Volume, Filer servers
  • FUSE mounting: Proper mount point management
  • Cleanup: Automatic teardown of all resources

Configurable Test Parameters

config := &TestConfig{
    Collection:   "test",
    Replication:  "001", 
    ChunkSizeMB:  8,
    CacheSizeMB:  200,
    NumVolumes:   5,
    EnableDebug:  true,
    MountOptions: []string{"-allowOthers"},
}

Rich Assertion Helpers

framework.AssertFileExists("path/to/file")
framework.AssertFileContent("file.txt", expectedContent)
framework.AssertFileMode("script.sh", 0755)
framework.CreateTestFile("test.txt", []byte("content"))

Test Categories

1. Basic File Operations

  • Create/Read/Write/Delete: Fundamental file operations
  • File Attributes: Size, timestamps, permissions
  • Append Operations: File appending behavior
  • Large Files: Files exceeding chunk size limits
  • Sparse Files: Non-contiguous file data

2. Directory Operations

  • Directory Lifecycle: Create, list, remove directories
  • Nested Structures: Deep directory hierarchies
  • Directory Permissions: Access control testing
  • Directory Rename: Move operations
  • Complex Scenarios: Many files, deep nesting

3. Concurrent Operations

  • Multi-threaded Access: Simultaneous file operations
  • Race Condition Detection: Concurrent read/write scenarios
  • Directory Concurrency: Parallel directory operations
  • Stress Testing: High-frequency operations

4. Error Handling & Edge Cases

  • Permission Denied: Access control violations
  • Disk Full: Storage limit scenarios
  • Network Issues: Filer/Volume server failures
  • Invalid Operations: Malformed requests
  • Recovery Testing: Error recovery scenarios

Usage Examples

Basic Test Run

# Build SeaweedFS binary
make

# Run all FUSE tests
cd test/fuse_integration
go test -v

# Run specific test category
go test -v -run TestBasicFileOperations
go test -v -run TestConcurrentFileOperations

Custom Configuration

func TestCustomFUSE(t *testing.T) {
    config := &TestConfig{
        ChunkSizeMB:  16,           // Larger chunks
        CacheSizeMB:  500,          // More cache
        EnableDebug:  true,         // Debug output
        SkipCleanup:  true,         // Keep files for inspection
    }
    
    framework := NewFuseTestFramework(t, config)
    defer framework.Cleanup()
    require.NoError(t, framework.Setup(config))
    
    // Your tests here...
}

Debugging Failed Tests

config := &TestConfig{
    EnableDebug:  true,     // Enable verbose logging
    SkipCleanup:  true,     // Keep temp files for inspection
}

Advanced Features

Performance Benchmarking

func BenchmarkLargeFileWrite(b *testing.B) {
    framework := NewFuseTestFramework(t, DefaultTestConfig())
    defer framework.Cleanup()
    require.NoError(t, framework.Setup(DefaultTestConfig()))
    
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        // Benchmark file operations
    }
}

Custom Test Scenarios

func TestCustomWorkload(t *testing.T) {
    framework := NewFuseTestFramework(t, DefaultTestConfig())
    defer framework.Cleanup()
    require.NoError(t, framework.Setup(DefaultTestConfig()))
    
    // Simulate specific application workload
    simulateWebServerWorkload(t, framework)
    simulateDatabaseWorkload(t, framework)
    simulateBackupWorkload(t, framework)
}

Integration with CI/CD

GitHub Actions Example

name: FUSE Integration Tests
on: [push, pull_request]

jobs:
  fuse-tests:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-go@v3
      with:
        go-version: '1.26'
    
    - name: Install FUSE
      run: sudo apt-get install -y fuse
    
    - name: Build SeaweedFS
      run: make
    
    - name: Run FUSE Tests
      run: |
        cd test/fuse_integration
        go test -v -timeout 30m

Docker Testing

FROM golang:1.26
RUN apt-get update && apt-get install -y fuse
COPY . /seaweedfs
WORKDIR /seaweedfs
RUN make
CMD ["go", "test", "-v", "./test/fuse_integration/..."]

Comparison with Current Testing

Aspect Current Tests New Framework
Operations Covered Basic FIO read/write All FUSE operations
Concurrency Single-threaded Multi-threaded stress tests
Error Scenarios Limited Comprehensive error handling
File Types Regular files only Large, sparse, many small files
Directory Testing None Complete directory operations
Setup Complexity Manual Docker setup Automated cluster management
Test Isolation Shared environment Isolated per-test environments
Debugging Limited Rich debugging and inspection

Benefits

1. Comprehensive Coverage

  • Tests all FUSE operations supported by SeaweedFS
  • Covers edge cases and error conditions
  • Validates behavior under concurrent access

2. Reliable Testing

  • Isolated test environments prevent test interference
  • Automatic cleanup ensures consistent state
  • Deterministic test execution

3. Easy Maintenance

  • Clear test organization and naming
  • Rich helper functions reduce code duplication
  • Configurable test parameters for different scenarios

4. Real-world Validation

  • Tests actual FUSE filesystem behavior
  • Validates integration between all SeaweedFS components
  • Catches issues that unit tests might miss

Future Enhancements

1. Extended FUSE Features

  • Extended attributes (xattr) testing
  • Symbolic link operations
  • Hard link behavior
  • File locking mechanisms

2. Performance Profiling

  • Built-in performance measurement
  • Memory usage tracking
  • Latency distribution analysis
  • Throughput benchmarking

3. Fault Injection

  • Network partition simulation
  • Server failure scenarios
  • Disk full conditions
  • Memory pressure testing

4. Integration Testing

  • Multi-filer configurations
  • Cross-datacenter replication
  • S3 API compatibility while mounted
  • Backup/restore operations

Getting Started

  1. Prerequisites

    # Install FUSE
    sudo apt-get install fuse  # Ubuntu/Debian
    brew install macfuse       # macOS
    
    # Build SeaweedFS
    make
    
  2. Run Tests

    cd test/fuse_integration
    go test -v
    
  3. View Results

    • Test output shows detailed operation results
    • Failed tests include specific error information
    • Debug mode provides verbose logging

This framework represents a significant improvement in SeaweedFS FUSE testing capabilities, providing comprehensive coverage, real-world validation, and reliable automation that will help ensure the robustness and reliability of the FUSE implementation.