3 Commits

Include renames

Author	SHA1	Message	Date
Javier de la Torreanduser.email	4be28fef2d	iceberg maintenance: cache a remote-only file before reading it (#12191) ... * iceberg maintenance: an unreadable parquet file fails the job, not the process parquet.NewReader panics on any file it cannot open (wrong magic, a truncated footer, zero bytes), and parquet-go's row decoder panics on a corrupt page under a valid footer. Maintenance runs in the worker's goroutines with no recover, so one bad delete or data file in any table ended the whole process. - Every parquet open in compaction goes through newParquetReader, which opens with parquet.OpenFile and returns the error. - A panic while decoding a delete file or a data file's rows is recovered into an error naming the file (recoverRead/guardRead). - As a backstop, detection recovers per table and Execute per operation: the table is skipped or the operation fails, and the worker keeps running. A failed delete file still fails the table's compaction (no commit), as a read error did before; a failed data file still skips only its bin. * iceberg maintenance: cache a remote-only file before reading it On a bucket mounted with remote.mount, a file that is not cached locally has no chunks and no inline content, only its RemoteEntry. Maintenance read it as an empty file: a parquet reader failed on zero bytes (before the previous change, it panicked and ended the process) and a manifest failed to parse, so a table whose files had been uncached could not be maintained. lookupFileByIcebergPath, the lookup every read goes through, now calls CacheRemoteObjectToLocalCluster for a remote-only entry, as the S3 gateway does on a read, and reads the entry it returns; the call is tried twice, and an entry still remote-only after caching is an error. The lookups that only resolve a path (orphan removal) do not cache. readEntryContent reports a file with no content and no chunks as empty instead of returning zero bytes. The fake volume server in the tests now serves chunk reads, so the remote-only tests read the cached file through the chunk reader as a real one is read. * iceberg maintenance: bound the metadata location read before caching a remote file * iceberg maintenance: bound the metadata read on the entry the cache returns * iceberg maintenance: enforce the metadata size bound inside the remote cache call The worker checks the resolved entry before asking the filer to cache a remote-only object, but the filer resolves the entry again at cache time and downloads whatever its remote size then says. A replacement landing in between could push an oversized object through the download even though the post-cache check would refuse to read it. CacheRemoteObjectToLocalClusterRequest gains an optional max_size; the filer rejects an over-limit remote object before the download starts. loadMetadata passes metadataFileMaxBytes; data-file reads leave it unset. * iceberg maintenance: key the remote cache singleflight by the size limit Callers sharing one flight share its outcome, including a bounded caller's over-limit rejection: an unrestricted reader joining the same path would fail even though its own bound allows the download. The limit joins the key so only callers with the same bound share a flight. * iceberg: gofmt compact_order_test.go * filer: bound remote cache callers before the shared download Each caller's max_size is now enforced on the entry the filer resolves before that caller joins the singleflight, which is keyed by path again. A bounded caller's rejection is never shared with other callers, and callers with different bounds share one download instead of pulling the same remote object twice. * filer: bind the remote cache size bound to the downloaded entry The pre-flight check and the shared download resolve the entry separately, so a remote metadata refresh landing in between lets a larger object through: the download ran to the refreshed size and only the reader's post-check refused it. The bound is now re-checked inside the flight on the entry the download streams. The rejection is typed so a co-tenant sharing the flight does not inherit it: callers whose own bound permits the reported size retry on a fresh flight instead of failing for another caller's limit. --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com>	2026-10-29 11:13:23 -07:00
3 people	a6be3c89c2	filer: support nanosecond timestamps for conditional delete (#11339) (#12071) ... * filer: support nanosecond timestamps for conditional delete (#11339) * shell: fs.verify prune carries the entry mtime in nanoseconds IfNotModifiedAfter now resolves to sub-second precision on the filer, so pass the looked-up mtime_ns along with mtime; the re-lookup check catches same-second rewrites that share the second, and the test mock mirrors the filer's seconds/nanoseconds cutoff. * shell: volume.fsck guards empty-directory deletes with mtime nanoseconds The looked-up mtime_ns joins the seconds in IfNotModifiedAfter so a directory rewritten within the same second is not dropped; the stub filer mirrors the filer's scale heuristic and the race fixtures use a realistic mtime so the nanosecond branch engages. * filer: cover the seconds guard and equality boundary Test second-resolution cutoffs on both sides of the entry's second and the exact-equality nanosecond guard that still deletes. * filer: compare the nanosecond guard as a time, not UnixNano UnixNano overflows for mtimes outside 1678-2262, letting a far-future entry slip past the guard; time.After compares exactly at every epoch. The seconds branch keeps Unix() so a same-second cutoff still deletes. * filer_pb: add explicit if_not_modified_after_ns to DeleteEntryRequest * filer: prefer explicit nanoseconds guard, keep if_not_modified_after as seconds * shell: send seconds and nanoseconds delete cutoffs, skip unexpressible mtimes * filer: merge conditional-delete guards into one nanosecond cutoff DeleteEntryMetaAndData carried both the seconds and nanoseconds guards even though the precedence between them is fixed, so every caller passed a dead parameter. The gRPC handler now folds if_not_modified_after into nanoseconds at the boundary, where (sec+1)*1e9-1 keeps the whole-second cutoff exact, and the filer takes a single ifNotModifiedAfterNs. * filer: hold the conditional-delete cutoff as a time, not int64 nanoseconds A seconds-only guard at or past year 2262 saturated to MaxInt64 nanoseconds (~2262), so a delete whose cutoff matched a far-future mtime was skipped while the request still reported success, and any entry between 2262 and the real cutoff survived pruning. The merged cutoff is now a time.Time, where a whole-second guard maps to the last nanosecond of its second for any int64 second. --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-10-26 07:51:33 -07:00
Peter Dodd	5f1a742726	fix: return NoSuchKey and drop stale entries when a remote-mounted object is gone from the remote (#11680) ... A remote-only entry whose object was deleted from the remote storage outside the filer answered GET with 500 and stayed in the filer. The remote's not-found was lost on the way: the backends' ReadFile returned it as an untyped error, so FetchAndWriteNeedle failed with codes.Unknown and nothing downstream could tell it from any other failure. - remote_storage: GCS, S3 (NoSuchKey) and Azure (BlobNotFound) reads return ErrRemoteObjectNotFound. GCS reports a missing bucket the same way as a missing object, so it confirms the bucket with a listing. - volume server and filer: the not-found crosses gRPC as codes.NotFound carrying the sentinel's text, and the filer's cache RPC returns codes.NotFound, which the S3 gateway already maps to NoSuchKey. - s3api: the origin fallback answers NoSuchKey on a confirmed not-found. - filer: a confirmed not-found removes the stale entry, so the lazy remote-metadata cache converges on the remote. Only remote-only files outside .versions and without an active object lock are removed, only if unchanged since the fetch (checked on the object's write owner, under the lock S3 object writes take), and with a metadata-only delete: the filer skips its inline remote delete, and the delete events' entries carry a marker that makes filer.remote.sync and filer.remote.gateway skip their remote delete, while filer.sync still replicates it. The replicated DeleteEntryRequest carries keep_remote_object, so the destination's delete events are marked too. The store drops the marker from every write, so clients cannot plant it.	2026-10-10 11:00:04 +08:00