iceberg maintenance: cache a remote-only file before reading it (#12191)

* iceberg maintenance: an unreadable parquet file fails the job, not the process

parquet.NewReader panics on any file it cannot open (wrong magic, a
truncated footer, zero bytes), and parquet-go's row decoder panics on a
corrupt page under a valid footer. Maintenance runs in the worker's
goroutines with no recover, so one bad delete or data file in any table
ended the whole process.

- Every parquet open in compaction goes through newParquetReader, which
  opens with parquet.OpenFile and returns the error.
- A panic while decoding a delete file or a data file's rows is recovered
  into an error naming the file (recoverRead/guardRead).
- As a backstop, detection recovers per table and Execute per operation:
  the table is skipped or the operation fails, and the worker keeps
  running.

A failed delete file still fails the table's compaction (no commit), as a
read error did before; a failed data file still skips only its bin.

* iceberg maintenance: cache a remote-only file before reading it

On a bucket mounted with remote.mount, a file that is not cached locally
has no chunks and no inline content, only its RemoteEntry. Maintenance
read it as an empty file: a parquet reader failed on zero bytes (before
the previous change, it panicked and ended the process) and a manifest
failed to parse, so a table whose files had been uncached could not be
maintained.

lookupFileByIcebergPath, the lookup every read goes through, now calls
CacheRemoteObjectToLocalCluster for a remote-only entry, as the S3
gateway does on a read, and reads the entry it returns; the call is tried
twice, and an entry still remote-only after caching is an error. The
lookups that only resolve a path (orphan removal) do not cache.
readEntryContent reports a file with no content and no chunks as empty
instead of returning zero bytes.

The fake volume server in the tests now serves chunk reads, so the
remote-only tests read the cached file through the chunk reader as a
real one is read.

* iceberg maintenance: bound the metadata location read before caching a remote file

* iceberg maintenance: bound the metadata read on the entry the cache returns

* iceberg maintenance: enforce the metadata size bound inside the remote cache call

The worker checks the resolved entry before asking the filer to cache a
remote-only object, but the filer resolves the entry again at cache time
and downloads whatever its remote size then says. A replacement landing in
between could push an oversized object through the download even though
the post-cache check would refuse to read it.

CacheRemoteObjectToLocalClusterRequest gains an optional max_size; the
filer rejects an over-limit remote object before the download starts.
loadMetadata passes metadataFileMaxBytes; data-file reads leave it unset.

* iceberg maintenance: key the remote cache singleflight by the size limit

Callers sharing one flight share its outcome, including a bounded
caller's over-limit rejection: an unrestricted reader joining the same
path would fail even though its own bound allows the download. The
limit joins the key so only callers with the same bound share a flight.

* iceberg: gofmt compact_order_test.go

* filer: bound remote cache callers before the shared download

Each caller's max_size is now enforced on the entry the filer resolves
before that caller joins the singleflight, which is keyed by path again.
A bounded caller's rejection is never shared with other callers, and
callers with different bounds share one download instead of pulling the
same remote object twice.

* filer: bind the remote cache size bound to the downloaded entry

The pre-flight check and the shared download resolve the entry
separately, so a remote metadata refresh landing in between lets a
larger object through: the download ran to the refreshed size and only
the reader's post-check refused it. The bound is now re-checked inside
the flight on the entry the download streams.

The rejection is typed so a co-tenant sharing the flight does not
inherit it: callers whose own bound permits the reported size retry on
a fresh flight instead of failing for another caller's limit.

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>

weed/worker/tasks/iceberg/compact_order_test.go

@@ -363,6 +363,7 @@ func TestResolveCompactionOrderSkipsUncoveredSortField(t *testing.T) {
 		t.Fatalf("uncovered sort field: got %+v, want nil", order)
 	}
 }
+
 // A full-bin repack that produces nothing must not discard the ordered runs
 // it was meant to replace — they are still valid compaction work.
 func TestCompactDataFilesKeepsRunsWhenRepackFails(t *testing.T) {

weed/worker/tasks/iceberg/filer_io.go

@@ -181,14 +181,24 @@ func (s *tableState) loadMetadata(ctx context.Context, client filer_pb.SeaweedFi
 	if s.Metadata != nil {
 		return nil
 	}
-	entry, err := lookupFileByIcebergPath(ctx, client, bucketName, s.DataPath, s.MetadataLocation)
+	entry, fullPath, err := lookupResolvedEntry(ctx, client, bucketName, s.DataPath, s.MetadataLocation)
 	if err != nil {
 		return fmt.Errorf("load metadata file %s: %w", s.MetadataLocation, err)
 	}
-	if len(entry.Content) == 0 && len(entry.Chunks) > 0 {
-		if size := int64(filer.FileSize(entry)); size > metadataFileMaxBytes {
-			return fmt.Errorf("metadata file %s too large: %d bytes", s.MetadataLocation, size)
-		}
+	// Bound what the location would pull before it is pulled: a remote
+	// entry's object is downloaded whole by the cache call, a chunked
+	// file's read is bounded by FileSize.
+	if size := metadataEntrySize(entry); size > metadataFileMaxBytes {
+		return fmt.Errorf("metadata file %s too large: %d bytes", s.MetadataLocation, size)
+	}
+	entry, err = cacheRemoteOnlyEntry(ctx, client, fullPath, entry, metadataFileMaxBytes)
+	if err != nil {
+		return fmt.Errorf("load metadata file %s: %w", s.MetadataLocation, err)
+	}
+	// The cache call returns the entry the filer then held; a replacement
+	// written during the download keeps the name but not the checked size.
+	if size := metadataEntrySize(entry); size > metadataFileMaxBytes {
+		return fmt.Errorf("metadata file %s too large: %d bytes", s.MetadataLocation, size)
 	}
 	content, err := readEntryContent(ctx, client, entry, s.MetadataLocation)
 	if err != nil {
@@ -202,6 +212,23 @@ func (s *tableState) loadMetadata(ctx context.Context, client filer_pb.SeaweedFi
 	return nil
 }
 
+// metadataEntrySize is the bytes an entry's content would pull into memory:
+// a remote object's size or a chunked file's FileSize. Inline content is
+// exempt — the REST load path accepts it unbounded and the lookup already
+// holds it.
+func metadataEntrySize(entry *filer_pb.Entry) int64 {
+	if len(entry.Content) > 0 {
+		return 0
+	}
+	if isRemoteOnly(entry) {
+		return entry.GetRemoteEntry().GetRemoteSize()
+	}
+	if len(entry.Chunks) > 0 {
+		return int64(filer.FileSize(entry))
+	}
+	return 0
+}
+
 func isFilerNotFound(err error) bool {
 	return errors.Is(err, filer_pb.ErrNotFound) || status.Code(err) == codes.NotFound
 }
@@ -258,10 +285,57 @@ func addFilerPathSpellings(set map[string]struct{}, bucketName, dataPath, iceber
 }
 
 // lookupFileByIcebergPath returns the filer entry holding an Iceberg-style
-// path. dataPath is the bucket-relative directory the table's files live in.
+// path, ready to read. dataPath is the bucket-relative directory the table's
+// files live in. A file of a remote-mounted bucket that is not cached locally
+// is cached first, as the S3 gateway does on a read.
 func lookupFileByIcebergPath(ctx context.Context, client filer_pb.SeaweedFilerClient, bucketName, dataPath, icebergPath string) (*filer_pb.Entry, error) {
-	entry, _, err := lookupResolvedEntry(ctx, client, bucketName, dataPath, icebergPath)
-	return entry, err
+	entry, fullPath, err := lookupResolvedEntry(ctx, client, bucketName, dataPath, icebergPath)
+	if err != nil {
+		return nil, err
+	}
+	return cacheRemoteOnlyEntry(ctx, client, fullPath, entry, 0)
+}
+
+// cacheRemoteOnlyEntry downloads a remote-only entry's object to the local
+// cluster and returns the cached entry. maxSize bounds what the filer will
+// download: the entry it resolves at cache time can differ from the one
+// checked here, and the download would run to its full remote size. One
+// retry covers a cache call racing an uncache of the same entry; an entry
+// still remote-only after caching is an error.
+func cacheRemoteOnlyEntry(ctx context.Context, client filer_pb.SeaweedFilerClient, fullPath string, entry *filer_pb.Entry, maxSize int64) (*filer_pb.Entry, error) {
+	if !isRemoteOnly(entry) {
+		return entry, nil
+	}
+	dir, name := path.Dir(fullPath), path.Base(fullPath)
+	var err error
+	for attempt := 0; attempt < 2 && isRemoteOnly(entry); attempt++ {
+		var resp *filer_pb.CacheRemoteObjectToLocalClusterResponse
+		resp, err = client.CacheRemoteObjectToLocalCluster(ctx, &filer_pb.CacheRemoteObjectToLocalClusterRequest{
+			Directory: dir,
+			Name:      name,
+			MaxSize:   maxSize,
+		})
+		switch {
+		case err != nil:
+		case resp == nil || resp.Entry == nil:
+			err = errors.New("no entry returned")
+		default:
+			entry = resp.Entry
+		}
+	}
+	if err != nil {
+		return nil, fmt.Errorf("cache remote file %s: %w", fullPath, err)
+	}
+	if isRemoteOnly(entry) {
+		return nil, fmt.Errorf("cache remote file %s: still remote-only after caching", fullPath)
+	}
+	return entry, nil
+}
+
+// isRemoteOnly reports whether an entry of a remote-mounted bucket holds no
+// bytes locally: no chunks and no inline content, only its RemoteEntry.
+func isRemoteOnly(entry *filer_pb.Entry) bool {
+	return entry.IsInRemoteOnly() && len(entry.Content) == 0
 }
 
 // loadFileByIcebergPath loads a file from the filer given an Iceberg-style path.
@@ -276,9 +350,14 @@ func loadFileByIcebergPath(ctx context.Context, client filer_pb.SeaweedFilerClie
 // readEntryContent returns an entry's bytes: its inline content, or a chunked
 // file read through the volume servers.
 func readEntryContent(ctx context.Context, client filer_pb.SeaweedFilerClient, entry *filer_pb.Entry, name string) ([]byte, error) {
-	if len(entry.Content) > 0 || len(entry.Chunks) == 0 {
+	if len(entry.Content) > 0 {
 		return entry.Content, nil
 	}
+	if len(entry.Chunks) == 0 {
+		// Everything read here is Iceberg metadata, a manifest or a data
+		// file, none of which can be empty.
+		return nil, fmt.Errorf("file %s is empty (%d bytes in its attributes, no content or chunks)", name, entry.GetAttributes().GetFileSize())
+	}
 
 	initGlobalHTTPClientOnce.Do(util_http.InitGlobalHttpClient)
 	reader := filer.NewFileReader(singleFilerClient{client: client}, entry)

weed/worker/tasks/iceberg/filer_io_save_test.go

@@ -1,6 +1,7 @@
 package iceberg
 
 import (
+	"bytes"
 	"context"
 	"encoding/json"
 	"io"
@@ -10,12 +11,14 @@ import (
 	"strings"
 	"sync"
 	"testing"
+	"time"
 
 	util_http "github.com/seaweedfs/seaweedfs/weed/util/http"
 )
 
 // fakeVolumeServer accepts chunk uploads and records what it received, so a
-// test can check the bytes actually left the filer entry.
+// test can check the bytes actually left the filer entry. It also serves
+// chunk reads of what it holds.
 type fakeVolumeServer struct {
 	mu     sync.Mutex
 	server *httptest.Server
@@ -29,6 +32,18 @@ func startFakeVolumeServer(t *testing.T) *fakeVolumeServer {
 	v := &fakeVolumeServer{parts: make(map[string][]byte)}
 	v.server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		fid := strings.TrimPrefix(r.URL.Path, "/")
+		if r.Method == http.MethodGet || r.Method == http.MethodHead {
+			// A chunk read, answered as a volume server does (ranges included).
+			v.mu.Lock()
+			data, ok := v.parts[fid]
+			v.mu.Unlock()
+			if !ok {
+				http.NotFound(w, r)
+				return
+			}
+			http.ServeContent(w, r, "", time.Time{}, bytes.NewReader(data))
+			return
+		}
 		// The uploader sends the payload as a single form-data part named
 		// "file" with an empty filename, which FormFile will not match.
 		reader, err := r.MultipartReader()
@@ -58,6 +73,12 @@ func startFakeVolumeServer(t *testing.T) *fakeVolumeServer {
 	return v
 }
 
+func (v *fakeVolumeServer) put(fid string, data []byte) {
+	v.mu.Lock()
+	defer v.mu.Unlock()
+	v.parts[fid] = data
+}
+
 func (v *fakeVolumeServer) hostPort(t *testing.T) string {
 	t.Helper()
 	u, err := url.Parse(v.server.URL)

weed/worker/tasks/iceberg/remote_only_test.go

@@ -0,0 +1,304 @@
+package iceberg
+
+import (
+	"bytes"
+	"context"
+	"fmt"
+	"net"
+	"path"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/credentials/insecure"
+)
+
+// remoteCachingFiler is the fake filer plus what a remote mount needs, shaped
+// as the real filer answers: a file of a remote-mounted bucket that is not
+// cached locally has only its RemoteEntry; caching it writes its bytes to a
+// volume and returns the entry with chunks and no inline content, so a read
+// goes through the chunk reader, the volume lookup and the volume server.
+type remoteCachingFiler struct {
+	*fakeFilerServer
+	volume     *fakeVolumeServer
+	volumeHost string
+
+	mu         sync.Mutex
+	remote     map[string][]byte // dir/name -> the remote object's bytes
+	cacheCalls int
+	nextFid    int
+	stayRemote bool // answer without caching, as on a racing uncache
+	failFirst  int  // fail the first N cache calls
+	// cacheEntry swaps the stored entry at cache time, as a replacement
+	// written during the download would.
+	cacheEntry func(dir, name string, entry *filer_pb.Entry) *filer_pb.Entry
+}
+
+func (f *remoteCachingFiler) CacheRemoteObjectToLocalCluster(_ context.Context, req *filer_pb.CacheRemoteObjectToLocalClusterRequest) (*filer_pb.CacheRemoteObjectToLocalClusterResponse, error) {
+	f.mu.Lock()
+	body, ok := f.remote[req.Directory+"/"+req.Name]
+	f.cacheCalls++
+	f.nextFid++
+	fid := fmt.Sprintf("7,%08x", f.nextFid)
+	stay := f.stayRemote
+	fail := f.failFirst > 0
+	if fail {
+		f.failFirst--
+	}
+	f.mu.Unlock()
+	if fail {
+		return nil, fmt.Errorf("remote read timed out")
+	}
+	entry := f.getEntry(req.Directory, req.Name)
+	if !ok || entry == nil {
+		return nil, fmt.Errorf("no remote object %s/%s", req.Directory, req.Name)
+	}
+	if req.MaxSize > 0 && entry.GetRemoteEntry().GetRemoteSize() > req.MaxSize {
+		return nil, fmt.Errorf("remote object %s/%s is %d bytes, over the %d byte limit",
+			req.Directory, req.Name, entry.GetRemoteEntry().GetRemoteSize(), req.MaxSize)
+	}
+	if stay {
+		return &filer_pb.CacheRemoteObjectToLocalClusterResponse{Entry: entry}, nil
+	}
+	if f.cacheEntry != nil {
+		entry = f.cacheEntry(req.Directory, req.Name, entry)
+	}
+	f.volume.put(fid, body)
+	cached := &filer_pb.Entry{Name: entry.Name, Attributes: entry.Attributes, RemoteEntry: entry.RemoteEntry,
+		Chunks: []*filer_pb.FileChunk{{FileId: fid, Offset: 0, Size: uint64(len(body)), ModifiedTsNs: time.Now().UnixNano()}}}
+	f.putEntry(req.Directory, req.Name, cached)
+	return &filer_pb.CacheRemoteObjectToLocalClusterResponse{Entry: cached}, nil
+}
+
+func (f *remoteCachingFiler) LookupVolume(_ context.Context, req *filer_pb.LookupVolumeRequest) (*filer_pb.LookupVolumeResponse, error) {
+	out := &filer_pb.LookupVolumeResponse{LocationsMap: map[string]*filer_pb.Locations{}}
+	for _, vid := range req.VolumeIds {
+		out.LocationsMap[vid] = &filer_pb.Locations{Locations: []*filer_pb.Location{{Url: f.volumeHost, PublicUrl: f.volumeHost}}}
+	}
+	return out, nil
+}
+
+func startRemoteCachingFiler(t *testing.T) (*remoteCachingFiler, filer_pb.SeaweedFilerClient) {
+	t.Helper()
+	vol := startFakeVolumeServer(t)
+	srv := &remoteCachingFiler{fakeFilerServer: newFakeFilerServer(), volume: vol, volumeHost: vol.hostPort(t), remote: map[string][]byte{}}
+	listener, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatalf("listen: %v", err)
+	}
+	server := grpc.NewServer()
+	filer_pb.RegisterSeaweedFilerServer(server, srv)
+	go func() { _ = server.Serve(listener) }()
+	t.Cleanup(server.GracefulStop)
+	conn, err := grpc.NewClient(listener.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
+	if err != nil {
+		t.Fatalf("dial: %v", err)
+	}
+	t.Cleanup(func() { conn.Close() })
+	return srv, filer_pb.NewSeaweedFilerClient(conn)
+}
+
+// putRemoteOnly records body as the remote object behind an Iceberg path and
+// puts its remote-only entry: no chunks, no content, only the RemoteEntry.
+func putRemoteOnly(t *testing.T, srv *remoteCachingFiler, bucket, dataPath, file string, body []byte) {
+	t.Helper()
+	full, err := icebergFilerPath(bucket, dataPath, file)
+	if err != nil {
+		t.Fatalf("path: %v", err)
+	}
+	dir, name := path.Dir(full), path.Base(full)
+	srv.remote[dir+"/"+name] = body
+	srv.putEntry(dir, name, &filer_pb.Entry{Name: name,
+		Attributes:  &filer_pb.FuseAttributes{FileSize: uint64(len(body)), Mtime: time.Now().Unix()},
+		RemoteEntry: &filer_pb.RemoteEntry{StorageName: "r", RemoteSize: int64(len(body)), RemoteMtime: time.Now().Unix()}})
+}
+
+func TestRemoteOnlyFileIsCachedThenReadThroughItsChunks(t *testing.T) {
+	srv, client := startRemoteCachingFiler(t)
+	const bucket, dataPath, file = "b", "ns/t", "data/d1-deletes.parquet"
+	putRemoteOnly(t, srv, bucket, dataPath, file, positionDeleteParquet(t, []positionDeleteTestRow{
+		{"s3://b/ns/t/data/f1.parquet", 3}, {"s3://b/ns/t/data/f1.parquet", 9},
+	}))
+
+	var got map[string][]int64
+	var err error
+	noPanic(t, func() {
+		got, err = readPositionDeleteFile(context.Background(), client, bucket, dataPath, file)
+	})
+	if err != nil {
+		t.Fatalf("read: %v", err)
+	}
+	if srv.cacheCalls != 1 {
+		t.Fatalf("CacheRemoteObjectToLocalCluster called %d times, want 1", srv.cacheCalls)
+	}
+	if n := len(got["s3://b/ns/t/data/f1.parquet"]); n != 2 {
+		t.Fatalf("got %v, want the two positions of the remote object", got)
+	}
+}
+
+func TestRemoteOnlyFileSchemaIsReadThroughItsChunks(t *testing.T) {
+	srv, client := startRemoteCachingFiler(t)
+	const bucket, dataPath, file = "b", "ns/t", "data/d2-deletes.parquet"
+	putRemoteOnly(t, srv, bucket, dataPath, file, positionDeleteParquet(t, []positionDeleteTestRow{{"s3://b/ns/t/data/f1.parquet", 1}}))
+	schema, err := parquetSchemaByIcebergPath(context.Background(), client, bucket, dataPath, file)
+	if err != nil {
+		t.Fatalf("schema: %v", err)
+	}
+	if _, ok := schema.Lookup("pos"); !ok {
+		t.Fatalf("schema %v has no pos column", schema)
+	}
+}
+
+func TestACachedFileIsNotCachedAgain(t *testing.T) {
+	srv, client := startRemoteCachingFiler(t)
+	const bucket, dataPath, file = "b", "ns/t", "data/d3-deletes.parquet"
+	full, _ := icebergFilerPath(bucket, dataPath, file)
+	body := positionDeleteParquet(t, []positionDeleteTestRow{{"s3://b/ns/t/data/f1.parquet", 1}})
+	srv.volume.put("7,000000ff", body)
+	srv.putEntry(path.Dir(full), path.Base(full), &filer_pb.Entry{Name: path.Base(full),
+		Attributes:  &filer_pb.FuseAttributes{FileSize: uint64(len(body))},
+		RemoteEntry: &filer_pb.RemoteEntry{StorageName: "r", RemoteSize: int64(len(body))},
+		Chunks:      []*filer_pb.FileChunk{{FileId: "7,000000ff", Offset: 0, Size: uint64(len(body)), ModifiedTsNs: 1}}})
+	if _, err := readPositionDeleteFile(context.Background(), client, bucket, dataPath, file); err != nil {
+		t.Fatalf("read: %v", err)
+	}
+	if srv.cacheCalls != 0 {
+		t.Fatalf("CacheRemoteObjectToLocalCluster called %d times for a cached file", srv.cacheCalls)
+	}
+}
+
+func TestAFileStillRemoteOnlyAfterCachingIsACacheError(t *testing.T) {
+	srv, client := startRemoteCachingFiler(t)
+	srv.stayRemote = true
+	const bucket, dataPath, file = "b", "ns/t", "data/d4-deletes.parquet"
+	putRemoteOnly(t, srv, bucket, dataPath, file, positionDeleteParquet(t, []positionDeleteTestRow{{"s3://b/ns/t/data/f1.parquet", 1}}))
+	var err error
+	noPanic(t, func() {
+		_, err = readPositionDeleteFile(context.Background(), client, bucket, dataPath, file)
+	})
+	if err == nil || !strings.Contains(err.Error(), "still remote-only") {
+		t.Fatalf("err = %v, want a cache error", err)
+	}
+}
+
+func TestAFailedCacheCallIsRetriedOnce(t *testing.T) {
+	const bucket, dataPath, file = "b", "ns/t", "data/d5-deletes.parquet"
+	body := positionDeleteParquet(t, []positionDeleteTestRow{{"s3://b/ns/t/data/f1.parquet", 1}})
+
+	srv, client := startRemoteCachingFiler(t)
+	putRemoteOnly(t, srv, bucket, dataPath, file, body)
+	srv.failFirst = 1
+	if _, err := readPositionDeleteFile(context.Background(), client, bucket, dataPath, file); err != nil {
+		t.Fatalf("one failed cache call, then a good one: %v", err)
+	}
+	if srv.cacheCalls != 2 {
+		t.Fatalf("%d cache calls, want 2", srv.cacheCalls)
+	}
+
+	srv, client = startRemoteCachingFiler(t)
+	putRemoteOnly(t, srv, bucket, dataPath, file, body)
+	srv.failFirst = 2
+	if _, err := readPositionDeleteFile(context.Background(), client, bucket, dataPath, file); err == nil || !strings.Contains(err.Error(), "remote read timed out") {
+		t.Fatalf("two failed cache calls: err = %v, want the cache error", err)
+	}
+}
+
+// A location-only record pointing at a remote object over the metadata bound
+// is rejected on the remote entry's size, without downloading the object: the
+// bound exists because the catalog pointer may name any file in the bucket.
+func TestARemoteOnlyMetadataFileOverTheBoundIsNotCached(t *testing.T) {
+	srv, client := startRemoteCachingFiler(t)
+	original := metadataFileMaxBytes
+	metadataFileMaxBytes = 8
+	defer func() { metadataFileMaxBytes = original }()
+
+	setup := tableSetup{BucketName: "b", Namespace: "ns", TableName: "t"}
+	putRemoteOnly(t, srv, setup.BucketName, setup.tablePath(), "metadata/big.metadata.json",
+		bytes.Repeat([]byte{1}, int(metadataFileMaxBytes)+1))
+	state := &tableState{DataPath: setup.tablePath(), MetadataLocation: fmt.Sprintf(
+		"s3://%s/%s/metadata/big.metadata.json", setup.BucketName, setup.tablePath())}
+	if err := state.loadMetadata(context.Background(), client, setup.BucketName); err == nil || !strings.Contains(err.Error(), "too large") {
+		t.Fatalf("err = %v, want a too-large rejection", err)
+	}
+	if srv.cacheCalls != 0 {
+		t.Fatalf("%d cache calls, want 0", srv.cacheCalls)
+	}
+}
+
+// A remote object that grew past the bound between the worker's check and the
+// filer's cache resolve is rejected before its bytes are downloaded: the
+// bound rides the cache request for the entry the filer resolves, not only
+// the one the worker saw.
+func TestAMetadataFileThatGrewPastTheBoundIsNotDownloaded(t *testing.T) {
+	srv, client := startRemoteCachingFiler(t)
+	original := metadataFileMaxBytes
+	metadataFileMaxBytes = 8
+	defer func() { metadataFileMaxBytes = original }()
+
+	setup := tableSetup{BucketName: "b", Namespace: "ns", TableName: "t"}
+	putRemoteOnly(t, srv, setup.BucketName, setup.tablePath(), "metadata/grew.metadata.json", []byte{1})
+	full, err := icebergFilerPath(setup.BucketName, setup.tablePath(), "metadata/grew.metadata.json")
+	if err != nil {
+		t.Fatalf("path: %v", err)
+	}
+	dir, name := path.Dir(full), path.Base(full)
+	client = swapEntryOnCacheClient{SeaweedFilerClient: client, swap: func() {
+		srv.putEntry(dir, name, &filer_pb.Entry{Name: name,
+			Attributes:  &filer_pb.FuseAttributes{FileSize: uint64(metadataFileMaxBytes) + 1, Mtime: time.Now().Unix()},
+			RemoteEntry: &filer_pb.RemoteEntry{StorageName: "r", RemoteSize: metadataFileMaxBytes + 1, RemoteMtime: time.Now().Unix()}})
+	}}
+	state := &tableState{DataPath: setup.tablePath(), MetadataLocation: fmt.Sprintf(
+		"s3://%s/%s/metadata/grew.metadata.json", setup.BucketName, setup.tablePath())}
+	if err := state.loadMetadata(context.Background(), client, setup.BucketName); err == nil || !strings.Contains(err.Error(), "over the") {
+		t.Fatalf("err = %v, want a bound rejection", err)
+	}
+}
+
+// swapEntryOnCacheClient runs swap inside the cache call, as a replacement
+// write landing between the worker's lookup and the filer's resolve would.
+type swapEntryOnCacheClient struct {
+	filer_pb.SeaweedFilerClient
+	swap func()
+}
+
+func (c swapEntryOnCacheClient) CacheRemoteObjectToLocalCluster(ctx context.Context, req *filer_pb.CacheRemoteObjectToLocalClusterRequest, opts ...grpc.CallOption) (*filer_pb.CacheRemoteObjectToLocalClusterResponse, error) {
+	c.swap()
+	return c.SeaweedFilerClient.CacheRemoteObjectToLocalCluster(ctx, req, opts...)
+}
+
+// The bound also applies to the entry the cache call returns: a replacement
+// written during the download can be larger than the remote entry that was
+// checked.
+func TestAReplacedMetadataFileOverTheBoundIsNotRead(t *testing.T) {
+	srv, client := startRemoteCachingFiler(t)
+	original := metadataFileMaxBytes
+	metadataFileMaxBytes = 8
+	defer func() { metadataFileMaxBytes = original }()
+
+	setup := tableSetup{BucketName: "b", Namespace: "ns", TableName: "t"}
+	putRemoteOnly(t, srv, setup.BucketName, setup.tablePath(), "metadata/swapped.metadata.json",
+		bytes.Repeat([]byte{1}, 4))
+	srv.cacheEntry = func(_, _ string, entry *filer_pb.Entry) *filer_pb.Entry {
+		return &filer_pb.Entry{Name: entry.Name, RemoteEntry: entry.RemoteEntry,
+			Attributes: &filer_pb.FuseAttributes{FileSize: uint64(metadataFileMaxBytes) + 1, Mtime: time.Now().Unix()}}
+	}
+	state := &tableState{DataPath: setup.tablePath(), MetadataLocation: fmt.Sprintf(
+		"s3://%s/%s/metadata/swapped.metadata.json", setup.BucketName, setup.tablePath())}
+	if err := state.loadMetadata(context.Background(), client, setup.BucketName); err == nil || !strings.Contains(err.Error(), "too large") {
+		t.Fatalf("err = %v, want a too-large rejection", err)
+	}
+}
+
+func TestAFileWithNoContentOrChunksIsAnEmptyFileError(t *testing.T) {
+	srv, client := startRemoteCachingFiler(t)
+	const bucket, dataPath, file = "b", "ns/t", "snap-1.avro"
+	full, _ := icebergFilerPath(bucket, dataPath, file)
+	srv.putEntry(path.Dir(full), path.Base(full), &filer_pb.Entry{Name: path.Base(full),
+		Attributes: &filer_pb.FuseAttributes{FileSize: 2057}}) // its size says bytes, it holds none
+	if _, err := loadFileByIcebergPath(context.Background(), client, bucket, dataPath, file); err == nil || !strings.Contains(err.Error(), "empty") {
+		t.Fatalf("err = %v, want an empty-file error", err)
+	}
+}