7 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandDevin	9620e7ebfe	filer: auto-enable per-path commit records when a peer has its own store (#12062) ... * filer: enable per-path commit records only when a peer has its own store Since seaweedfs#11698 every metadata change writes a per-path commit record into the filer store (an MpTs key, plus an MpDl key on deletes), and the records stay after the path is deleted. They order the events a filer replays from peer filers that keep their own store, since replay only runs when peerSignature != f.Signature. Filers that all share one store (one Redis, MySQL, TiKV, ...) never replay each other's events, so the records are never read, yet each change still costs one or two extra store writes and the key count grows with every path ever changed — unbounded growth for in-memory stores. This turns the records off by default and latches them on the first time a peer reports a different store signature — no option needed. The latch is persisted in the store (MpRep) so it survives restarts and converges filers sharing a store even before they discover the peer themselves. While off, a filer keeps an MpOff marker so enabling later bumps the records' generation (MpGen, folded into each record key) past the unrecorded window: a record older than a change made while off cannot let an older replayed peer event win. Ordering for unrecorded changes falls back to the entry's mtime, the same as entries written before the records existed. Supersedes the manual option approach in seaweedfs#12060. * filer: keep delete tombstones while records off, harden enable path Review on the first commit: - A delete committed while the records were off left no entry and no usable tombstone: after enabling, an older replayed create could resurrect the path. Deletes and rename sources are now always recorded under the generation-0 MpDl key — they are never fenced by a generation bump, and they cannot go stale because every delete is recorded. Creates/updates still write nothing while off. - bumpPathCommitGen could regress MpGen under concurrent bumpers on a shared store. It now loops: each pass writes only a generation higher than any it has seen, so concurrent bumps converge upward. - doSubscribeToOneFiler proceeded into replay when enablePathCommitRecords failed, running the whole session without ordering. It now aborts the subscribe so the reconnect retries with the records on. * filer: bind pending commit-record repairs to their generation A failed MpTs write kept in metaTsPending could land after a shared-store generation bump, writing a pre-bump timestamp as a current-generation record and overriding newer unrecorded state. Each pending change record now carries the generation it failed under: repairs write it only while that generation is current, and ordering reads ignore it otherwise. Pending delete records need no binding — MpDl keys are never fenced. * filer: never move the local records' generation backward The converge loop in bumpPathCommitGen seeded from zero and both it and syncPathCommitGen stored unconditionally: a stale writer could leave the stored generation below what this filer already adopted, and the next sync or bump then walked the local value back — orphaning records it wrote under the higher generation. The bump now seeds from the local generation and every store only moves it forward. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: name a records generation after the off marker it consumes Two filers consuming the same off marker each counted one past the stored generation, so the second could store N+2 after the first stored N+1 and write records under a key the other could not read until the next check. The bump now takes the marker's own timestamp — filers racing on one marker land on one value — a marker older than the stored generation is cleaned as a leftover, and a stored value that already passed the bump is adopted instead of counted past. A newly enabling filer whose marker was already consumed names its own off window with the current time, so its unrecorded period is still fenced. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: tell a fresh off marker from one the generation already consumed A marker whose timestamp reads at or below the stored generation was dropped as leftover, but a new off period can also read that old — the off filer's clock lagging, or a host clock moving back — and dropping it kept records from before that period live, so stale replay could win. The stored generation now names the marker it was bumped for: a marker the store still shows is consumed unless it is that one, however old it reads, and only the consumed marker itself is cleared without a bump. The generation trails in the value, so a whole-slice read still yields it on stores written before the pair was stored. Enabling fences the filer's own off window with the current time instead of a marker a sharer may already have consumed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: cover racing off-marker consumers settling on one generation * filer: adopt the stored generation, not the consumed marker it names A filer syncing with no off marker present read the value's first field — the consumed marker's timestamp — so a restart after a lagging-clock off period under-counted the generation and kept records the bump dropped. * filer: re-check the off marker before consuming it A sync whose marker read ran behind a later off period's consume cycle still bumped for its stale read: the stored generation already covered a different marker, so the bump overwrote the consumed name, counted the generation past records a newer consumption meant to keep, and cleared whatever marker was live by then. Each pass now re-reads the marker and only bumps or clears while it is still the one the pass saw. * filer: keep a failed generation read from resetting the records' generation The final generation read ignored errors and then stored whatever gen held — possibly zero — so a transient store failure could hide records the current generation wrote. Unexpected read errors now return before the local generation changes. Also cover a sync whose marker read runs behind another consumer's delete: it must adopt the stored generation, not bump for the stale read. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: fail a bump that cannot verify the stored generation A generation read that failed mid-bump was treated as absent: the write could land below a generation a racing bumper had stored, or re-store a marker that bumper had already consumed. Read failures now abort the pass like every other store error on the path. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: cover generation reads that fail instead of returning a value Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: carry the commit timestamp on the entry itself The per-path MpTs record only exists because a commit timestamp had nowhere else to live. Stamping commit_ts_ns on the entry before its store write - and reusing that stamp as the emitted event's TsNs - makes the entry its own commit record: replay and bootstrap traversal order on it with zero extra writes for creates and updates. Deletes still need a durable key since a removed entry cannot carry a timestamp, so MpDl tombstones stay (one write per delete, growth bounded by delete history). Entries predating the field fall back to mtime, matching what peers and old records already assume. This replaces the MpTs/MpOff/MpGen/MpRep machinery entirely: shared- store filers write nothing extra, and dedicated-store replication keeps last-writer-wins ordering plus tombstone coverage for deletes and ancestor drops. * filer: keep a directory update replay from tombstoning its subtree A same-path update wrote a delete record before the delete-and-reinsert, so a directory metadata change read as a subtree delete: a later peer event for a child older than the update was then skipped even though nobody deleted the children, leaving files missing on this filer. Only file updates take the tombstone now — the re-inserted directory entry's own commit stamp orders the path itself. * filer: order unstamped entries by their recorded commit before mtime Entries written before stamping existed have no CommitTsNs, so the ordering check fell back to the client-settable mtime and dropped the MpTs record those versions wrote: an entry committed at event time 300 with mtime 100 lost to a delayed peer event at 200 after upgrade. Read the legacy record for unstamped entries in localCommitTsNs, CommitTsNs, and the traverse stamp (which also stopped falling back to mtime, regressing entry_ts_ns to 0). * filer: release queued rename event stamps when the rename fails Each moved entry carried an in-flight stamp its queued event would adopt at emit time. When the move succeeded but a later child move or the commit failed, the handler rolled back without emitting the queued events or releasing their stamps, pinning the delivered and flushed watermarks below the aborted rename forever. Release every queued stamp on both failure exits in both rename handlers. * filer: apply a directory update replay with an in-place upsert Deleting the stored entry before inserting its replacement left a window where the directory had no entry and no ordering record — a tombstone cannot stand in for it without reading as a subtree delete that suppresses the children's own records. A stale peer event accepted in that gap could replay an older update or delete and sweep the directory's children. Upsert the replacement instead: the entry keeps its recorded commit order across the whole update, and a directory has no chunks or hard links for a delete to release. * filer: update an existing directory entry on replay instead of upserting InsertEntry is not an upsert on SQL stores configured without upsert support — PostgreSQL with enableUpsert=false issues a plain INSERT and its duplicate-key error does not match the MySQL-only fallback, so a replay updating a present directory failed on every retry. Pick UpdateEntry when the entry exists and InsertEntry when it is absent; UpdateEntry touches no row that is not there. * filer: detect a shared-store delete racing a replayed directory update UpdateEntry reports success even when the row was deleted between the find and the update — the UPDATE changes zero rows and the replayed change is dropped while the subscription offset advances. Re-read the entry after the update and retry the write, re-checking the recorded commit order each pass so a newer peer change wins instead of being resurrected over. * filer: declare commit_ts_ns in field-number order Field 6 sits between extended and hard_link_id; the declaration now reads in the same order. Wire format unchanged. Also syncs the Java client's checked-in copy of filer.proto, which the pb Makefile copies on gen. * filer: option to skip delete tombstones entirely filer.options.deleteTombstones=false stops writing MpDl records on deletes - a single filer or shared-store cluster never replays peer events, so the tombstones are dead weight. The tradeoff is explicit: if a dedicated-store peer does replicate, a stale event can resurrect a deleted path. * filer: drop delete tombstones entirely Deletes no longer write MpDl records, and the pending-repair machinery they needed is gone with them: per-mutation metadata overhead is now zero for every store topology. Ordering information lives only on the entry itself via commit_ts_ns. The accepted tradeoff: a dedicated-store peer replaying an event older than a delete can resurrect the path, and a kept directory no longer fences stale child creates. Recorded tombstones and MpTs records left by older builds are still read, so upgrades keep the protections they already have. * filer: stop reading legacy per-path records too Drops the MpTs/MpDl read path along with the writes: meta_path_ts.go keeps only the entry-field readers (CommitTsNs, EntryCommitTsNs, localCommitTsNs). Entries written before stamping existed fall back to mtime, and tombstones or commit records older builds left behind no longer fence replayed events. * filer: trim commit timestamp comments --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-25 11:37:06 -07:00
3 people	103328e70f	filer: create parent directories for lazily fetched remote entries (#11727) ... * filer: create parent directories for lazily fetched remote entries A remote-mounted object first read through the filer was persisted without its parent directories, so remote.uncache and the remote cache evictor, which walk down from the mount directory, never reached it: its cached chunks could not be evicted. Persist the missing parents with the entry. With those directories in place, the empty-folder cleaner leaves folders under a remote mount alone: deleting one removes its whole prefix on the remote, which can still hold objects that were never read locally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: keep remote objects at parent keys and remote prefixes of empty folders Look up a lazily fetched entry's missing parents through lazy fetch before creating them, so a remote object stored at a parent's key is materialized and promoted to a prefix object rather than hidden behind a new directory. Deleting a folder that is empty in the filer is metadata-only under a remote mount, both for a non-recursive delete and for the expiry cleanup of empty parents: the mount is populated lazily, so the folder can still hold remote objects never read, and removing its remote directory deletes every object under that prefix. This replaces the empty-folder cleaner's own guard, which covered only one of those paths. A metadata-only delete records no remote-deletion tombstone, locally or from its metadata event: nothing is owed to the remote, so lazy reads stay open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: stop the lazy parent lookup at a store error Only a parent the store reports as not found is missing; a parent it cannot read ends the walk, so the lookup does not stat ancestors above it on the remote. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: stat each missing remote parent once A lazy fetch that finds k missing parents asked FindEntry for each, and every nested fetch re-walked its own still-missing ancestors, re-stating remote keys already proven absent: 1+k(k+1)/2 remote stats for one first read. The flagged parent lookups now skip their own walk; their entries are created by the same CreateEntry that already persists the chain. * filer: materialize parents for lazily listed remote entries maybeLazyListFromRemote persisted children and the listing directory with skipCreateParentDir, so a first access that goes through the list path still left entries unreachable to a walk from the mount directory. Materialize the missing ancestors the same way the lazy fetch does, and let the synced_at directory create its parent chain. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-10-15 16:23:33 -07:00
Chris LuandDevin	2f6c237238	filer: keep lazy remote reads from resurrecting deleted paths (#11452) ... * filer: keep lazy remote reads from resurrecting deleted paths Under a remote mount with filer.remote.sync as write-back, a path that was deleted or renamed away could come back as a chunkless remote-only entry: between the local delete and the daemon's remote delete, a store miss made maybeLazyFetchFromRemote trust a bucket that was behind the filer. The ghost then outlived the remote object -- HEAD answered 200, GET failed, and nothing cleaned it up. The filer now tombstones paths it deletes under a remote mount, learned both synchronously from its own delete path and from peer metadata events. The lazy fetch and the lazy listing skip a tombstoned path until the path is written again, until the mount's persisted write-back sync offset has passed the delete event (the remote delete has landed), or until a generous TTL covers a mount without a daemon. Fixes #11440 Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: cover recursive remote deletes with an ancestor tombstone A recursive delete now records the directory tombstone before walking children, so a partial traversal or a store that drops the subtree without listing it still leaves every descendant covered. Directory tombstones also subsume older descendant entries on add, descendant adds covered by a standing ancestor are skipped, and an existing tombstone can be refreshed even at capacity. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: scope remote tombstones to the deleted object's generation A remote object whose own mtime postdates the local delete is a new generation, not the one the tombstone hides, so a recreated directory can surface remote writes made after its delete while old-generation objects stay hidden. Lazy fetch now stats the remote object before deciding, listings pass each child's remote mtime, and a sync offset releases a tombstone once it reaches the delete's own timestamp. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: rebuild remote deletion tombstones after restart In-memory tombstones are lost on restart while remote write-back offsets persist, so a filer boot replays the persisted metadata log from the oldest mount offset and folds deletes back into the tombstone set through the same event handler. Lazy remote reads hold off while the replay runs so a pending delete cannot resurrect in the gap. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: release remote tombstones only after their delete event lands The write-back offset orders against event timestamps, but the synchronous delete path recorded tombstones with the local clock before its event was emitted — a later unrelated event could already have pushed the mount's watermark past that guess, releasing the tombstone before the daemon applied the delete. Tombstones recorded ahead of their event are now marked pending and can only be lifted by the event confirming them or by TTL; event-stamped tombstones release through the offset as before. The remote-mtime generation bypass is dropped: remote and filer clocks are independent, and a pending remote delete removes whatever object sits at the path, so a "newer" remote object would only resurrect as a phantom. Tombstoned lookups now skip the remote stat entirely. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: drop dir tombstone when recursive delete fails before listing The ancestor tombstone is recorded before the child listing; if that listing fails nothing was deleted, and the leftover tombstone would hide still-existing remote children for the whole TTL. Tombstones for children already deleted stay, since their remote deletes are still owed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: block lazy remote reads on startup tombstone rebuild The rebuild gate is now a done-channel set synchronously before the replay goroutine starts, so no lazy read can slip through in between. Reads wait on it with context cancellation instead of returning an empty miss that makes remote-only objects look deleted. The replay start is floored at now-TTL: mounts without a recorded write-back offset previously replayed the whole persisted history, and events older than the TTL would only build already-expired tombstones. The gate check now runs after the mount lookup so replaying the meta log's own directory listings does not deadlock on the gate, and the replay retries with backoff until it succeeds instead of failing open. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: mark restamped tombstone pending until its delete event lands When a local delete raises an existing tombstone's timestamp, the new value is only a local clock guess ahead of that delete's event. Leaving the tombstone un-pending lets a write-back offset release it before the event is actually consumed, reopening the resurrection window. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: bound tombstone replay to the tombstone TTL Persisted-log replay retried forever, keeping lazy remote reads gated indefinitely when the log cannot be read. Cap retries at the tombstone TTL measured from replay start: past that point every tombstone would have expired anyway, so opening the gate loses no protection. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: re-check deletion tombstone before persisting lazy fetch A delete landing while StatFile is in flight passed the earlier tombstone check but still persisted the fetched entry, resurrecting a path whose remote delete is pending. Re-check right before CreateEntry. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: retract a lazily persisted entry when a delete raced the insert The pre-insert tombstone check still leaves a window between the check and the store insert. Since deletes always record the tombstone before removing the entry, a tombstone visible right after a successful insert means the delete already ran: delete the entry back out so the tombstoned path stays deleted. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: note why the replay deadline can safely open the gate Deletes made after startup are captured by the live delete and event paths, so a stalled replay can only be missing pre-restart deletes, all of which are past the tombstone TTL by the deadline. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: retract only the entry a lazy remote read materialized Deleting by path after a raced delete could remove a legitimate rewrite that replaced the fetched entry. Verify the stored entry still matches the remote object (or the just-created directory shape) before deleting, and apply the same post-insert check to lazy listing children. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: require full-entry equality before retracting a lazy entry Remote-only matching still removed a write that had updated the fetched entry, e.g. appended chunks. Compare the persisted entry against what this read materialized; any change means a real update owns the path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-26 08:00:39 +08:00
Chris Lu	38c14d3c13	filer: apply SSRF guard to the lazy-remote fetch/list/delete paths (#11294) ... * filer: add guarded remote-storage client builder hook for lazy fetch The lazy-remote fetch path (maybeLazyFetchFromRemote) resolved its remote-storage client through the unguarded shared cache, bypassing the SSRF chokepoint (BuildGuardedRemoteStorageClient) that the CVE-2026-73080 remediation wired into the volume, filer stream and s3 stream dial paths. Add a RemoteStorageClientBuilder hook on Filer plus conf-only lookups on FilerRemoteStorage, and route the lazy fetch through the builder when set (endpoint deny-list + DNS-rebinding-safe dialer), falling back to the shared cache otherwise. The filer server wires the builder in a follow-up. * filer: route lazy directory listing through the guarded remote client maybeLazyListFromRemote shared the unguarded client resolution of the fetch path, so a caller-supplied remote endpoint was dialed without the SSRF deny-list or rebinding-safe dialer. Resolve the conf and build the client through buildRemoteStorageClient so the same guard covers listing. * filer: route lazy remote delete through the guarded remote client maybeDeleteFromRemote issued outbound DELETE/RemoveDirectory requests through the unguarded client, giving a write-side SSRF to a caller-chosen endpoint. Resolve the conf and build the client through buildRemoteStorageClient so the endpoint deny-list and rebinding-safe dialer apply to the delete path as well. * filer server: wire the guarded remote client builder into the filer Set Filer.BuildGuardedRemoteClient to BuildGuardedRemoteStorageClient and forward AllowUntrustedRemoteEndpoints so the lazy-remote fetch, list and delete paths apply the same SSRF endpoint checks as the volume and streaming read paths. * filer: test lazy fetch honors the guarded remote client builder Add a regression test that sets BuildGuardedRemoteClient to a rejecting builder and asserts maybeLazyFetchFromRemote returns no entry without reaching the remote, covering the SSRF guard wired in the prior commits. * filer: skip remote client for local-only lazy deletes maybeDeleteFromRemote resolved and validated the mount's remote client before checking entry.Remote, so a local-only file (no Remote entry) under a mount whose endpoint the guard rejects failed to delete: the guard error aborted the metadata deletion, leaving a file that needs no remote operation undeletable. Move the local-only check ahead of client construction so only remote-backed files and directories pay the guard. * filer: build the guarded remote client inside the lazy singleflight The lazy fetch and list paths built the guarded client before their singleflight blocks, so concurrent requests for the same key each allocated a fresh SDK client and HTTP transport even though only one remote operation ran. Move client construction inside the singleflight so the deduplicated operation builds it once, matching the per-request guard semantics of the sibling streaming paths without the duplicate transport churn. * filer: test guarded rejection for the lazy list and delete paths Add regression tests that set BuildGuardedRemoteClient to a rejecting builder and assert the lazy list does not reach the remote, a remote-backed file delete is blocked, and a local-only file under a rejected mount still deletes (covering the local-only fix). * filer: decouple lazy guarded-client build from the first caller's context Building the guarded client inside the singleflight made concurrent fetches share the first caller's context. If that caller canceled while endpoint DNS validation was running, the builder returned an error and published a not-found result to other callers whose contexts were still valid. Build with context.WithoutCancel so the guard's DNS validation is not tied to any single caller's cancellation, matching the list path's existing decoupling for the remote operation itself. * filer: reject remote-storage confs that dial blocked endpoints at load The filer's lazy-fetch / lazy-list / remote-delete paths resolve remote storage clients by name from FilerRemoteStorage.storageNameToConf and dial them via remote_storage.GetRemoteStorage, which bypasses the SSRF deny-list the volume server (BuildGuardedRemoteStorageClient) and the filer's own direct-read path apply. A RemoteConf planted under /etc/remote with a loopback / private / IMDS S3 endpoint is reloaded into storageNameToConf on the next metadata-change event and then dialed on the next cache miss — server-side request forgery from the filer. Apply the volume server's SSRF deny-list at conf load time, the single chokepoint that populates storageNameToConf: - Add RemoteStorageConfValidator, injected into FilerRemoteStorage by the filer server (the filer package cannot import the server package). A conf that fails validation is dropped from storageNameToConf, so the name-based client resolution on the lazy paths returns "not found" instead of dialing the blocked endpoint. - Add ValidateRemoteConfForLoad in weed_server, which mirrors BuildGuardedRemoteStorageClient's gcs credential + endpoint checks (validateRemoteEndpoint via guardedRemoteClient) without building a client. allowUntrusted skips the check, mirroring the volume server opt-out (-filer.allowUntrustedRemoteEndpoints). - The filer server injects the validator at construction. A conf whose type dials a fixed provider host (no caller-supplied endpoint) passes; only caller-influenced endpoints are denied. * filer: skip DNS resolution in the load-time SSRF validator ValidateRemoteConfForLoad resolved hostnames during /etc/remote reload, so a transient DNS failure (2s timeout) dropped the conf from the fresh map that replaces the live map, disabling a working mount until the next metadata event. The build-time guard (BuildGuardedRemoteStorageClient) already re-resolves and re-validates the endpoint at dial time with the rebinding-safe dialer, so DNS at load is redundant for security. Split the static checks (scheme, IMDS hostnames, IP-literal blocked addresses, gcs credentials) into validateRemoteEndpointForLoad, which does no DNS. Hostname endpoints pass at load and are caught at dial if they resolve to a blocked address. This preserves fail-fast for statically-blocked confs (loopback IPs, IMDS hostnames) without letting transient DNS failures disable mounts. * filer: accept empty S3 endpoints in the guarded remote client builder guardedRemoteClient returned ok=true with an empty endpoint for a standard AWS S3 config (no custom S3Endpoint), so BuildGuardedRemoteStorageClient and ValidateRemoteConfForLoad rejected it with "remote endpoint is empty" — breaking standard AWS S3 mounts on the lazy paths and the sibling streaming read paths that already use the guarded builder. An empty endpoint is not caller-supplied: the AWS SDK derives the regional endpoint from the region, so there is nothing for the SSRF guard to validate. Return ok=false for empty S3-compatible endpoints so the builder falls through to the shared unguarded cache, matching the historical behavior for standard AWS S3.	2026-09-13 14:43:55 -07:00
Chris Lu	25ab4c3cac	preserve Content-Encoding for remote-mounted objects (#10340) ... * remote storage: carry Content-Encoding into mounted entries A RemoteEntry now records the remote object's Content-Encoding, and every path that materializes a local entry from remote metadata (lazy fetch, lazy listing, remote.mount, remote.meta.sync, remote.cache) stamps it into the entry extended attributes, so HTTP and S3 HeadObject/GetObject return the header. GCS and Azure populate it on listing and stat; S3 only exposes it via HeadObject, so listings leave it empty. * remote storage: set Content-Encoding when uploading to the remote An entry carrying Content-Encoding in its extended attributes (a native S3 upload, or a value pulled from the remote) now keeps it when filer.remote.sync or remote.copy.local writes the object to GCS, S3, or Azure, instead of silently dropping it. * gcs: read remote objects without decompressive transcoding GCS transparently decompresses gzip-encoded objects on download, which ignores range requests and returns byte counts that disagree with the tracked RemoteSize. Request the stored bytes instead; chunked reads of gzip-encoded objects then behave like any other object. * remote storage: track Content-Encoding presence so removals propagate A listing that does not report encodings (S3) leaves the field unset and the local header untouched, while an authoritative report of no encoding (GCS, Azure, any stat) now clears a previously stamped header instead of leaving it stale. remote.cache also schedules a metadata update when only the reported encoding changes. * remote storage: propagate Content-Encoding on metadata-only updates filer.remote.sync routes same-content changes through UpdateFileMetadata, which only touched custom metadata (GCS, Azure) or tags (S3), so a Content-Encoding change in the extended attributes never reached the remote object's real header. GCS now patches contentEncoding alongside the metadata, and Azure reissues the blob's HTTP headers with the new value, carrying the others over since the call replaces the full set. S3 stays tags-only: changing the header there means rewriting the object, which the sync already does whenever content changes. * remote.meta.sync: optional per-file stat for listing-omitted metadata S3 listings carry no Content-Encoding, so entries synced from them never learn it and the lazy-stat path never runs once an entry exists. With -statFiles, each new or changed file whose listing left the encoding unreported is stat-ed before reconciling, and the stat-derived value is persisted so the next run only stats files that changed. Off by default: it costs one remote request per file, and GCS and Azure listings already carry the encoding. * s3: apply metadata-only Content-Encoding changes with an in-place copy Content-Encoding is S3 system metadata, so the tags-only metadata update silently left the object's real header untouched. When the encoding differs, reissue the object as a self-copy with replaced metadata, carrying the content type and configured storage class like a fresh write does. CopyObject caps at 5 GiB; beyond that the change is logged and applies on the next content write. * azure: skip the metadata call when user metadata is unchanged An encoding-only change reissues the blob's HTTP headers; sending the unchanged user metadata alongside it wastes a round trip and bumps the blob's ETag once more than needed. * s3: carry existing object metadata through the encoding copy The replace directive drops everything not resent, and a mounted entry usually has no local mime or user metadata, so the in-place copy wiped the object's Content-Type, Cache-Control, user metadata, encryption settings, and storage class. Read them back with a HeadObject first and carry them over, overriding only what SeaweedFS manages: the encoding, a locally set mime, and the configured storage class. S3 reports Expires as a string while the copy input wants a time, so it is parsed and skipped when malformed.	2026-07-15 19:20:00 -07:00
Chris Lu	83195fc111	filer: reuse the caller's fetched entry in CreateEntry (#9645) ... CreateEntry starts with a FindEntry to load the current entry. A conditional CreateEntry already fetched that entry to evaluate the precondition under the per-path lock, so the create repeated the lookup. Add an existing *Entry parameter: when non-nil it is used as the current entry and the internal lookup is skipped; nil keeps the lookup. The gRPC CreateEntry handler passes the entry it fetched for the precondition, removing the redundant read while the lock is held. All other callers pass nil.	2026-05-23 21:40:41 -07:00
Chris Lu	f3c5ba3cd6	feat(filer): add lazy directory listing for remote mounts (#8615) ... * feat(filer): add lazy directory listing for remote mounts Directory listings on remote mounts previously only queried the local filer store. With lazy mounts the listing was empty; with eager mounts it went stale over time. Add on-demand directory listing that fetches from remote and caches results with a 5-minute TTL: - Add `ListDirectory` to `RemoteStorageClient` interface (delimiter-based, single-level listing, separate from recursive `Traverse`) - Implement in S3, GCS, and Azure backends using each platform's hierarchical listing API - Add `maybeLazyListFromRemote` to filer: before each directory listing, check if the directory is under a remote mount with an expired cache, fetch from remote, persist entries to the local store, then let existing listing logic run on the populated store - Use singleflight to deduplicate concurrent requests for the same directory - Skip local-only entries (no RemoteEntry) to avoid overwriting unsynced uploads - Errors are logged and swallowed (availability over consistency) * refactor: extract xattr key to constant xattrRemoteListingSyncedAt * feat: make listing cache TTL configurable per mount via listing_cache_ttl_seconds Add listing_cache_ttl_seconds field to RemoteStorageLocation protobuf. When 0 (default), lazy directory listing is disabled for that mount. When >0, enables on-demand directory listing with the specified TTL. Expose as -listingCacheTTL flag on remote.mount command. * refactor: address review feedback for lazy directory listing - Add context.Context to ListDirectory interface and all implementations - Capture startTime before remote call for accurate TTL tracking - Simplify S3 ListDirectory using ListObjectsV2PagesWithContext - Make maybeLazyListFromRemote return void (errors always swallowed) - Remove redundant trailing-slash path manipulation in caller - Update tests to match new signatures * When an existing entry has Remote != nil, we should merge remote metadata into it rather than replacing it. * fix(gcs): wrap ListDirectory iterator error with context The raw iterator error was returned without bucket/path context, making it harder to debug. Wrap it consistently with the S3 pattern. * fix(s3): guard against nil pointer dereference in Traverse and ListDirectory Some S3-compatible backends may return nil for LastModified, Size, or ETag fields. Check for nil before dereferencing to prevent panics. * fix(filer): remove blanket 2-minute timeout from lazy listing context Individual SDK operations (S3, GCS, Azure) already have per-request timeouts and retry policies. The blanket timeout could cut off large directory listings mid-operation even though individual pages were succeeding. * fix(filer): preserve trace context in lazy listing with WithoutCancel Use context.WithoutCancel(ctx) instead of context.Background() so trace/span values from the incoming request are retained for distributed tracing, while still decoupling cancellation. * fix(filer): use Store.FindEntry for internal lookups, add Uid/Gid to files, fix updateDirectoryListingSyncedAt - Use f.Store.FindEntry instead of f.FindEntry for staleness check and child lookups to avoid unnecessary lazy-fetch overhead - Set OS_UID/OS_GID on new file entries for consistency with directories - In updateDirectoryListingSyncedAt, use Store.UpdateEntry for existing directories instead of CreateEntry to avoid deleteChunksIfNotNew and NotifyUpdateEvent side effects * fix(filer): distinguish not-found from store errors in lazy listing Previously, any error from Store.FindEntry was treated as "not found," which could cause entry recreation/overwrite on transient DB failures. Now check for filer_pb.ErrNotFound explicitly and skip entries or bail out on real store errors. * refactor(filer): use errors.Is for ErrNotFound comparisons	2026-03-13 09:36:54 -07:00