2 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	12627d376d	mount: fix four readdir pagination bugs (#10624) ... * mount: size the direct listing slice from the batch, not the offset The limit passed here is skipCount+batchSize when a client resumes a fresh handle partway through a directory, so preallocating for it turns the client's cookie into an allocation: a readdir at offset 3,000,000 reserves 24MB before the first entry arrives, and an offset near the uint32 ceiling asks makeslice for ~4.29e9 elements. * mount: stop replaying a directory that shrank past the resume offset A client that opens a fresh handle and resumes at a cookie from an earlier, larger listing gets a preload that cannot reach the entry before that offset. The resume name was then left empty and the follow-up batch listed from the directory's first child again, so the client was handed every name a second time. The stream always runs from the first child, so failing to reach that entry means the directory is simply shorter than the offset. That is the end of it. * mount: page a directory from where the store reached The batch loader treated a short batch as the end of the directory, but the meta cache drops an expired child after the store has already spent it against the limit, so a batch that filled up could still deliver fewer entries than asked for. A directory with a handful of expired children would stop listing early and hide every child behind them; a whole batch of expired ones truncated the listing to nothing. ListDirectoryEntries now reports the name the store itself reached. That is both the sound end-of-directory signal -- the store returning nothing -- and the right cursor, since resuming from the last visible name would re-read the dropped children every round and never get past a batch that was entirely expired. * mount: drop the entries a directory walk has already passed entryStreamOffset was only ever written by reset, so the stream a handle holds grew for the life of the walk and a directory was retained whole even though nothing could read the entries behind the client's position again. A 10M-entry walk parked millions of entries per handle, and NFS-Ganesha opens several on the same directory. Offsets index into the stream from entryStreamOffset, so advancing the two together keeps them lined up. One entry is held back because the next batch resumes from the name immediately before the offset. Seeking back behind what is still held now restarts the directory, which is what the offset scheme can honestly support -- it previously returned nothing.	2026-08-07 12:11:27 -07:00
Chris Lu	b46946ece5	filer: list directories without decoding chunk lists (#10616) ... * filer: decode a listed entry without building its chunk list A readdir reads attributes and never looks at chunks, but decoding an entry builds the whole chunk list first: four allocations per chunk, all of it thrown away. On a directory of ordinary 4MB-chunked files that is most of what listing costs. DecodeAttributesOnly walks the wire format and hands everything except the chunks to the generated unmarshaller, so new fields in filer.proto need no attention here. The chunks are still measured, because the S3 copy and multipart paths deliberately store a zero FileSize and let the chunk extents define the size, but nothing is allocated to do it. The blob is only re-encoded once a chunk is actually seen, so an entry without any -- every directory, for one -- is unmarshalled where it lies and pays nothing for the walk. Listings opt in through the context, the way the lazy remote paths already do; a store that ignores it stays correct. chunks full attrs-only allocs 0 312.8n 310.1n ~ 1 -> 1 1 686.1n 411.1n -40.07% 7 -> 1 4 1.742u 667.4n -61.69% 24 -> 1 16 5.770u 1.544u -73.25% 86 -> 1 64 25.23u 6.004u -76.20% 328 -> 1 * mount: list directories with chunk lists omitted The two meta cache listings behind a readdir are the only callers, and neither reads a chunk. On 200k single-chunk files one enumeration goes from 364ms to 277ms and drops a million allocations. The read-through listing still fetches whole entries from the filer, which would need the request to say it wants attributes only. * mount: give the readdir benchmark's entries a chunk Chunkless entries made the decode look far cheaper than it is, which is the part of a listing worth measuring. * filer: let a listing ask for entries without their chunk lists The read-through readdir fetches whole entries over gRPC, and for a wide directory the chunk lists are most of what crosses the wire and most of what the client then unmarshals. A 4MB-chunked file is 113 bytes of entry against 46 without its chunk. ListEntriesRequest gains omit_chunks. The size a client needs is already in the attributes, where the store decode folded the chunk extents in, so dropping the list costs the client nothing. The filer still reads the entries whole. A listing is where a TTL-expired entry gets collected and deleted, and deleting one needs its chunks to find the data, so omitting them there would leak. Only the response is trimmed. The hint moves to filer_pb so one context flag serves both transports: the gRPC request sets omit_chunks, and a listing served from the local store skips building the chunks. Cache population is unaffected either way, since EnsureVisited starts from its own context. * filer: reject a chunk the full decoder would reject The walk skipped a chunk's bytes without looking inside them, so a FileChunk carrying a corrupt nested fid, or a string that is not valid UTF-8, sailed past the listing decoder while every other read of the same entry still failed. The file listed with a plausible size and then gave EIO on open, and corruption that used to fail the listing loudly was hidden instead. The chunk bytes are the one part of the blob the generated unmarshaller never sees, so the two checks it would have made are made here: a submessage has to parse, and a proto3 string has to be valid UTF-8. FileChunk's only submessages are FileIds of scalars, so walking them is a complete check. A descriptor-driven test fails if FileChunk ever gains a field of either kind that the walk does not know to check, which is the part that keeps this honest as filer.proto grows. Taking the scratch buffer lazily, only once a chunk is actually dropped, also takes the pool out of the path for entries that have none. Those were measurably slower than the full decoder before; they are now level with it. Each chunk's length prefix is parsed once rather than twice. chunks full attrs-only vs base 0 171.4n 176.9n ~ (p=0.670) 1 366.6n 259.4n -29.24% 4 1.034u 500.2n -51.60% 16 3.905u 1.464u -62.52% 64 13.48u 5.195u -61.46% * filer: carry the size before dropping chunks over the wire Dropping the chunk list assumed every store folds the chunk extents into FileSize when it decodes. A store that keeps entries as JSON rather than as an encoded Entry never re-derives it, so an object written with a zero FileSize kept its real size only in the chunks, and stripping them left the client reading the file as empty. Stamp the size into the attributes first, which costs nothing and does not depend on how the store loaded the entry. * mount: test that the readdir context reaches the store decode Everything else exercises the decoder directly, so a refactor that stopped threading the context would have reverted the whole thing with every test still passing. The benchmark's chunks also carried a constant legacy FileId, which BeforeEntrySerialization reparses over Fid on the way in, so all 200k entries stored one byte-identical chunk rather than the varying fixture it looked like.	2026-08-07 12:03:18 -07:00