mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 23:50:43 +02:00
Chris Luanddevin-ai-integration[bot]
811b8b5734
* add a per-mount cache_wait_ms to the remote storage mount mapping A read of an uncached remote-only object waits on a hardcoded size tier before it can fall back to the origin, so every ranged read of a large remote-only object pays that wait. Carry the wait in the mount mapping so it can be tuned, or set to zero, per mount. * resolve the cache wait of an uncached remote-only read from its mount The wait came only from the object size, so an operator could not trade cache hits for time to first byte. Both read paths now resolve the mount covering the object and let its cache_wait_ms replace the size tiers. * read straight from the remote when a mount waits zero for its cache A mount used as a streaming source pays the cache wait on every ranged read of an object too large to finish caching, and the caching itself is wasted work. A zero wait now skips the cache call, so both read paths go to the origin immediately. * let remote.mount set the cache wait of a mount remote.mount -cacheWait=0 turns a mount into a streaming source, and any other duration trades cache hits against time to first byte. * keep the size based wait for a version-specific read A read pinned to a version cannot fall back to the origin, since the mounted remote only holds the current key, so a mount that opts out of caching would leave it on the 503 retry loop forever. * let the operator allow a remote-only read to dial an internal endpoint The remote-mount read paths in the filer and the S3 gateway always refused an endpoint resolving to a loopback or private host, so a mount backed by an internal S3 could never be read from its origin, only through the local cache. Both now take the allowance the volume server already has, still off by default. * skip the background cache of a mount that waits zero for its cache GetObjectHandler kicks off caching for every remote-only read, so a mount serving as a streaming source kept downloading whole objects even though no read ever waited for them. * cover a zero cache wait end to end The read has to reach a real origin, so the harness also opts the filer and the S3 gateway into dialing the loopback remote it already allows for the volume server. * resolve the S3 cache wait once so the background cache follows it too The background cache that GetObjectHandler starts read the mount on its own, so it skipped a version-specific read that the foreground path still waits for. Both now ask the same resolver. * answer 404 when the origin of a zero-wait read is gone Metadata can outlive the object it points at, and with no cache to fill the read would sit on the 503 retry path forever. The remote backends already report a missing object as ErrRemoteObjectNotFound. * open the origin at write time for a multipart range Every part of a multipart Range is prepared before any is written, so opening eagerly would hold one origin connection per part and leak the ones already opened when a later part fails to open. * reject a cache wait shorter than a millisecond The mapping stores milliseconds, so -cacheWait=500us truncated to zero and silently turned caching off instead of waiting. * restore the doc comment of cacheRemoteObjectForStreamingWithShortTimeout Extracting the wait resolver left its comment on the new function. * stat the origin before committing a multipart range Opening at write time keeps no connection through the preparation, but it also moved a failure past the point where the multipart body picks the response status, so a gone origin truncated a 206 instead of answering 404. One stat up front puts the status back. * stat the origin once per request Every part of a multipart Range is prepared on its own, so the preflight ran once per range instead of once per read. * map Azure and GCS stream not-found to ErrRemoteObjectNotFound ReadFileAsStream on Azure and GCS returned provider-specific not-found errors instead of ErrRemoteObjectNotFound, so a zero-wait read of a deleted object was misclassified as a transient cache failure and retried indefinitely. Map BlobNotFound and ErrObjectNotExist the same way StatFile already does. * Update weed/remote_storage/gcs/gcs_storage_client.go Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Remote Object Cache Integration Tests
This directory contains integration tests for the remote object caching feature with singleflight deduplication.
Test Flow
Each test follows this pattern:
- Write to local - Upload data to primary SeaweedFS (local storage)
- Uncache - Push data to remote storage and remove local chunks
- Read - Read data (triggers caching from remote back to local)
This tests the full remote caching workflow including singleflight deduplication.
Architecture
┌─────────────────────────────────────────────────────────────────┐
│ Test Client │
│ │
│ 1. PUT data to primary SeaweedFS │
│ 2. remote.cache.uncache (push to remote, purge local) │
│ 3. GET data (triggers caching from remote) │
│ 4. Verify singleflight deduplication │
└──────────────────────────────────┬──────────────────────────────┘
│
┌─────────────────┴─────────────────┐
▼ ▼
┌────────────────────────────────────┐ ┌────────────────────────────────┐
│ Primary SeaweedFS │ │ Remote SeaweedFS │
│ (port 8333) │ │ (port 8334) │
│ │ │ │
│ - Being tested │ │ - Acts as "remote" S3 │
│ - Has remote storage mounted │──▶│ - Receives uncached data │
│ - Caches remote objects │ │ - Serves data for caching │
│ - Singleflight deduplication │ │ │
└────────────────────────────────────┘ └────────────────────────────────┘
What's Being Tested
Test Files and Coverage
| Test File | Commands Tested | Test Count | Description |
|---|---|---|---|
remote_cache_test.go |
Basic caching | 6 tests | Original caching workflow and singleflight tests, plus a mount with -cacheWait=0 reading straight from the remote |
remote_cache_copy_test.go |
S3 CopyObject / UploadPartCopy from a remote-only source | 2 tests | Source object lives only in remote storage; CopyObject and UploadPartCopy must cache it locally before persisting the destination so the result is readable |
command_remote_configure_test.go |
remote.configure |
6 tests | Configuration management |
command_remote_mount_test.go |
remote.mount, remote.unmount, remote.mount.buckets |
10 tests | Mount operations |
command_remote_cache_test.go |
remote.cache, remote.uncache |
13 tests | Cache/uncache with filters |
command_remote_copy_local_test.go |
remote.copy.local |
12 tests | NEW in PR #8033 - Local to remote copy |
command_remote_meta_sync_test.go |
remote.meta.sync |
8 tests | Metadata synchronization |
command_edge_cases_test.go |
All commands | 11 tests | Edge cases and stress tests |
Total: 68 test cases covering 8 weed shell commands and the S3 copy paths for remote-only sources
Commands Tested
remote.configure- Configure remote storage backendsremote.mount- Mount remote storage to local directoryremote.unmount- Unmount remote storageremote.mount.buckets- Mount all buckets from remoteremote.cache- Cache remote files locallyremote.uncache- Remove local cache, keep metadataremote.copy.local- Copy local files to remote (NEW in PR #8033)remote.meta.sync- Sync metadata from remote
Test Coverage
Basic Operations:
- Basic caching workflow (Write → Uncache → Read)
- Singleflight deduplication (concurrent reads trigger ONE cache operation)
- Large object caching (5MB-100MB files)
- Range requests (partial reads)
- Not found handling
File Filtering:
- Include patterns (
*.pdf,*.txt, etc.) - Exclude patterns
- Size filters (
-minSize,-maxSize) - Age filters (
-minAge,-maxAge) - Combined filters
Command Options:
- Dry run mode (
-dryRun=true) - Concurrency settings (
-concurrent=N) - Force update (
-forceUpdate=true) - Non-empty directory mounting (
-nonempty=true)
Edge Cases:
- Empty directories
- Nested directory hierarchies
- Special characters in filenames
- Very large files (100MB+)
- Many small files (100+)
- Rapid cache/uncache cycles
- Concurrent command execution
- Invalid paths
- Zero-byte files
Running Tests
Run All Tests
# Full automated workflow
make test-with-server
# Or manually
go test -v ./...
Run Specific Test Files
# Test remote.configure command
go test -v -run TestRemoteConfigure
# Test remote.mount/unmount commands
go test -v -run TestRemoteMount
go test -v -run TestRemoteUnmount
# Test remote.cache/uncache commands
go test -v -run TestRemoteCache
go test -v -run TestRemoteUncache
# Test remote.copy.local command (PR #8033)
go test -v -run TestRemoteCopyLocal
# Test remote.meta.sync command
go test -v -run TestRemoteMetaSync
# Test edge cases
go test -v -run TestEdgeCase
Quick Start
Run Full Test Suite (Recommended)
# Build SeaweedFS, start both servers, run tests, stop servers
make test-with-server
Manual Steps
# 1. Build SeaweedFS binary
make build-weed
# 2. Start remote SeaweedFS (acts as "remote" storage)
make start-remote
# 3. Start primary SeaweedFS (the one being tested)
make start-primary
# 4. Configure remote storage mount
make setup-remote
# 5. Run tests
make test
# 6. Clean up
make clean
Configuration
Primary SeaweedFS (Being Tested)
| Service | Port |
|---|---|
| S3 API | 8333 |
| Filer | 8888 |
| Master | 9333 |
| Volume | 8080 |
Remote SeaweedFS (Remote Storage)
| Service | Port |
|---|---|
| S3 API | 8334 |
| Filer | 8889 |
| Master | 9334 |
| Volume | 8081 |
Makefile Targets
make help # Show all available targets
make build-weed # Build SeaweedFS binary
make start-remote # Start remote SeaweedFS
make start-primary # Start primary SeaweedFS
make setup-remote # Configure remote storage mount
make test # Run tests
make test-with-server # Full automated test workflow
make logs # Show server logs
make health # Check server status
make clean # Stop servers and clean up
Test Details
TestRemoteCacheBasic
Basic workflow test:
- Write object to primary (local)
- Uncache (push to remote, remove local chunks)
- Read (triggers caching from remote)
- Read again (from local cache - should be faster)
TestRemoteCacheConcurrent
Singleflight deduplication test:
- Write 1MB object
- Uncache to remote
- Launch 10 concurrent reads
- All should succeed with correct data
- Only ONE caching operation should run (singleflight)
TestRemoteCacheLargeObject
Large file test (5MB) to verify chunked transfer works correctly.
TestRemoteCacheRangeRequest
Tests HTTP range requests work correctly after caching.
TestRemoteCacheNotFound
Tests proper error handling for non-existent objects.
TestRemoteCacheWaitZero
Remounts with remote.mount -cacheWait=0 and checks that a read of a remote-only
object is served from the remote and leaves the object uncached, while the default
size-based wait caches it.
Troubleshooting
View logs
make logs # Show recent logs from both servers
make logs-primary # Follow primary logs in real-time
make logs-remote # Follow remote logs in real-time
Check server health
make health
Clean up and retry
make clean
make test-with-server