make the remote-mount cache wait configurable per mount (#11168)

* add a per-mount cache_wait_ms to the remote storage mount mapping

A read of an uncached remote-only object waits on a hardcoded size tier
before it can fall back to the origin, so every ranged read of a large
remote-only object pays that wait. Carry the wait in the mount mapping so
it can be tuned, or set to zero, per mount.

* resolve the cache wait of an uncached remote-only read from its mount

The wait came only from the object size, so an operator could not trade
cache hits for time to first byte. Both read paths now resolve the mount
covering the object and let its cache_wait_ms replace the size tiers.

* read straight from the remote when a mount waits zero for its cache

A mount used as a streaming source pays the cache wait on every ranged
read of an object too large to finish caching, and the caching itself is
wasted work. A zero wait now skips the cache call, so both read paths go
to the origin immediately.

* let remote.mount set the cache wait of a mount

remote.mount -cacheWait=0 turns a mount into a streaming source, and any
other duration trades cache hits against time to first byte.

* keep the size based wait for a version-specific read

A read pinned to a version cannot fall back to the origin, since the
mounted remote only holds the current key, so a mount that opts out of
caching would leave it on the 503 retry loop forever.

* let the operator allow a remote-only read to dial an internal endpoint

The remote-mount read paths in the filer and the S3 gateway always refused
an endpoint resolving to a loopback or private host, so a mount backed by
an internal S3 could never be read from its origin, only through the local
cache. Both now take the allowance the volume server already has, still
off by default.

* skip the background cache of a mount that waits zero for its cache

GetObjectHandler kicks off caching for every remote-only read, so a mount
serving as a streaming source kept downloading whole objects even though no
read ever waited for them.

* cover a zero cache wait end to end

The read has to reach a real origin, so the harness also opts the filer and
the S3 gateway into dialing the loopback remote it already allows for the
volume server.

* resolve the S3 cache wait once so the background cache follows it too

The background cache that GetObjectHandler starts read the mount on its
own, so it skipped a version-specific read that the foreground path still
waits for. Both now ask the same resolver.

* answer 404 when the origin of a zero-wait read is gone

Metadata can outlive the object it points at, and with no cache to fill
the read would sit on the 503 retry path forever. The remote backends
already report a missing object as ErrRemoteObjectNotFound.

* open the origin at write time for a multipart range

Every part of a multipart Range is prepared before any is written, so
opening eagerly would hold one origin connection per part and leak the
ones already opened when a later part fails to open.

* reject a cache wait shorter than a millisecond

The mapping stores milliseconds, so -cacheWait=500us truncated to zero
and silently turned caching off instead of waiting.

* restore the doc comment of cacheRemoteObjectForStreamingWithShortTimeout

Extracting the wait resolver left its comment on the new function.

* stat the origin before committing a multipart range

Opening at write time keeps no connection through the preparation, but it
also moved a failure past the point where the multipart body picks the
response status, so a gone origin truncated a 206 instead of answering
404. One stat up front puts the status back.

* stat the origin once per request

Every part of a multipart Range is prepared on its own, so the preflight
ran once per range instead of once per read.

* map Azure and GCS stream not-found to ErrRemoteObjectNotFound

ReadFileAsStream on Azure and GCS returned provider-specific not-found
errors instead of ErrRemoteObjectNotFound, so a zero-wait read of a
deleted object was misclassified as a transient cache failure and
retried indefinitely. Map BlobNotFound and ErrObjectNotExist the same
way StatFile already does.

* Update weed/remote_storage/gcs/gcs_storage_client.go

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>

test/s3/remote_cache/remote_cache_test.go

@@ -40,6 +40,9 @@ const (
 	// Bucket name - mounted on primary as remote storage
 	testBucket = "remotemounted"
 
+	// Bucket name on the remote that testBucket is mounted on
+	remoteBucket = "remotesourcebucket"
+
 	// Path to weed binary
 	weedBinary = "../../../weed/weed_binary"
 )
@@ -388,6 +391,66 @@ func TestRemoteCacheRangeRequest(t *testing.T) {
 	t.Log("Range request test passed")
 }
 
+// mountRemote remounts the test bucket, keeping the metadata already pulled,
+// with the given extra remote.mount flags.
+func mountRemote(t *testing.T, extraFlags string) {
+	cmd := fmt.Sprintf("remote.mount -dir=/buckets/%s -remote=seaweedremote/%s -nonempty -metadataStrategy=lazy %s", testBucket, remoteBucket, extraFlags)
+	output, err := runWeedShell(t, cmd)
+	require.NoErrorf(t, err, "remount failed: %s", output)
+	time.Sleep(time.Second)
+}
+
+// copyLocalToRemote pushes matching local objects to the remote, which is what
+// lets remote.uncache drop their local chunks afterwards.
+func copyLocalToRemote(t *testing.T, pattern string) {
+	output, err := runWeedShell(t, fmt.Sprintf("remote.copy.local -dir=/buckets/%s -include=%s", testBucket, pattern))
+	require.NoErrorf(t, err, "remote.copy.local failed: %s", output)
+}
+
+// localChunkCount reports how many local chunks an entry has, so a test can
+// tell a cached read from one served straight out of the remote.
+func localChunkCount(t *testing.T, key string) string {
+	meta, err := runWeedShell(t, fmt.Sprintf("fs.meta.cat /buckets/%s/%s", testBucket, key))
+	require.NoError(t, err)
+	// fs.meta.cat closes with "chunks N meta size: ..."
+	summary := strings.LastIndex(meta, "chunks ")
+	require.GreaterOrEqualf(t, summary, 0, "no chunk count in %s", meta)
+	return strings.Fields(meta[summary+len("chunks "):])[0]
+}
+
+// TestRemoteCacheWaitZero tests that a mount with -cacheWait=0 serves a read of
+// an uncached object from the remote without caching it locally.
+func TestRemoteCacheWaitZero(t *testing.T) {
+	checkServersRunning(t)
+
+	testKey := fmt.Sprintf("test-cachewait-%d.bin", time.Now().UnixNano())
+	testData := make([]byte, 1024*1024)
+	for i := range testData {
+		testData[i] = byte(i % 256)
+	}
+
+	t.Log("Step 1: Writing 1MB object, pushing it to the remote and dropping the local chunks...")
+	uploadToPrimary(t, testKey, testData)
+	copyLocalToRemote(t, testKey)
+	uncacheLocal(t, testKey)
+	require.Equal(t, "0", localChunkCount(t, testKey), "the object must start out remote-only")
+
+	t.Log("Step 2: Remounting with -cacheWait=0...")
+	mountRemote(t, "-cacheWait=0")
+	defer mountRemote(t, "")
+
+	t.Log("Step 3: Reading the object (should stream from remote)...")
+	assert.Equal(t, testData, getFromPrimary(t, testKey), "data mismatch reading from remote")
+	assert.Equal(t, "0", localChunkCount(t, testKey), "the read must not cache chunks locally")
+
+	t.Log("Step 4: Reading with the size based wait (should cache locally)...")
+	mountRemote(t, "")
+	assert.Equal(t, testData, getFromPrimary(t, testKey), "data mismatch reading through the cache")
+	assert.NotEqual(t, "0", localChunkCount(t, testKey), "the read must cache chunks locally")
+
+	t.Log("Zero cache wait test passed")
+}
+
 // TestRemoteCacheNotFound tests that non-existent objects return proper errors
 func TestRemoteCacheNotFound(t *testing.T) {
 	checkServersRunning(t)