filer: stop full-chunk reads at the chunk size after a mount truncate (#12632)

* util/http: stop a full-chunk read at the chunk's recorded size

weed mount shrinks a chunk's Size in place on ftruncate but leaves its
needle untouched. A view covering the whole shortened chunk is fetched
without a Range header, so the reader returned every stored byte. The
filer GET then wrote past the declared Content-Length and sent an empty
body, and the stream reader served the stale tail.

ReadUrlAsStream and the cipher path now deliver at most size bytes;
size 0 still reads the whole needle for manifest and log chunks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: limit a plain chunk read to the requested size

A full-chunk read sends no Range header, so a chunk shortened by a mount
truncate streamed its whole needle into the S3 response.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: still report a failure past the chunk size

The volume server aborts a full-needle response when the needle fails
its checksum, and only at the end. Stopping at the chunk size missed
that, so a damaged copy read as healthy. Read the rest of the body and
return its error so the caller tries another copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chris LuandClaude Opus 5.5 authored and GitHub committed 2026-11-13 20:17:46 -08:00
1 parent 63c99183d1
commit 6eb2fb4842
4 files changed
+125 -3

No files matched your search

+1 -1
View File
@@ -2325,7 +2325,7 @@ func (s3a *S3ApiServer) readChunkRange(ctx context.Context, fileId string, offse
return nil, err
}
return resp.Body, nil
return &rc{Reader: io.LimitReader(resp.Body, size), Closer: resp.Body}, nil
}
// getEncryptedStreamFromVolumes gets a raw encrypted data stream from volume
@@ -1072,6 +1072,36 @@ func TestFetchChunkDataDecryptsVolumeCipher(t *testing.T) {
}
}
// A truncated chunk keeps its longer needle; full-chunk reads stop at the
// chunk's recorded size.
func TestFetchChunkDataStopsAtChunkSize(t *testing.T) {
volume := startFakeVolumeServer(t)
filerImpl := &ambiguousPutFiler{volume: volume, entries: map[string]*filer_pb.Entry{}}
s3a := newPutTestServer(t, startFakeFiler(t, filerImpl))
stored := bytes.Repeat([]byte("t"), 300)
fid := "3,01637037d6"
volume.mu.Lock()
volume.stored[fid] = stored
volume.mu.Unlock()
view := &filer.ChunkView{FileId: fid, ViewSize: 100, ChunkSize: 100}
for name, fetch := range map[string]func(context.Context, *filer.ChunkView) (io.ReadCloser, error){
"full": s3a.fetchFullChunk,
"view": s3a.fetchChunkViewData,
} {
r, err := fetch(context.Background(), view)
if err != nil {
t.Fatal(err)
}
got, _ := io.ReadAll(r)
r.Close()
if !bytes.Equal(got, stored[:100]) {
t.Fatalf("%s chunk read returned %d bytes, want 100", name, len(got))
}
}
}
// A suspended-versioning overwrite resolves the conditional check against the
// entry the read path serves: the null slot when present, else the latest
// version under .versions/. The routed transaction carries that version as the