* filer: copy a remote-only entry on the remote when renaming it
Renaming a file under a remote.mount whose content exists only on the
remote (never cached, or remote.uncache'd) lost it: the new entry kept
the old RemoteEntry, deleting the old entry deleted the old object, and
with no chunks nothing ever wrote the new key. filer.remote.sync could
not finish the rename either, since the old object was already gone.
moveSelfEntry now copies such an entry's object to the new key before
it touches anything, and the new entry gets the RemoteEntry of the
copy. The copy is a new optional RemoteStorageObjectCopier interface,
implemented for S3 with CopyObject up to 5 GB and UploadPartCopy above
that; the multipart copy carries the source's metadata and, where the
remote supports tagging, its tags. When an existing target is
overwritten, its delete keeps the remote object (the copy replaced it)
and its event is marked metadata-only so the sync leaves it alone too.
A remote-only rename that leaves its mount, or whose remote cannot
copy, is refused instead of deleting the only copy. Empty entries and
entries with local data rename as before.
UpdateFileMetadata builds its copy source with the same helper, which
escapes each key segment and keeps the slashes.
* filer: delete a copied rename's old object only after the commit
Review follow-ups:
- The old key of a copied remote-only entry, and the old directory of a
rename that copied one, are now deleted once the rename has committed.
A rename that fails after the copy leaves the source entry and its
object in place, plus an extra object at the new key.
- Every part of a multipart copy is conditioned on the ETag the copy
started from, so an object overwritten mid-copy fails the copy
instead of mixing versions.
- Both copy paths keep the source's server-side encryption settings.
- filer.remote.sync no longer deletes a key that a remote-only entry
holds by the time the delete event is processed: a file deleted and
then replaced by a remote-only rename has its content only there.
* filer: do not sweep the old prefix of a directory rename that copied
Removing the old directory on the remote deletes everything under its
prefix. Run after the commit, that could also take an object another
rename copied in under the same path meanwhile. When a child was copied,
leave the prefix alone and delete only the copied children's old keys.
* filer: keep an old key a remote-only entry has retaken by cleanup time
A directory rename does not lock its children's paths, so between the
commit and the deferred delete of a copied child's old key another
rename can put a remote-only entry there, whose content is then only in
that object. Check the store before deleting.
* filer: keep a renamed entry's old object when the path lookup fails
The deferred delete must only run once the lookup shows no remote-only
entry at the old path. A failed lookup shows nothing, and an extra
object loses nothing.
* filer: copy over a remote-only rename target only after the commit
Copying the source object onto the target's key before the transaction
committed destroyed the target's only content, which a rolled-back
commit cannot restore. When the target is remote-only the copy now runs
after the commit: the new entry keeps the old key's stamp until the copy
lands and re-stamps it, the target's object is kept meanwhile, and a
failed copy keeps the old object so remote.sync can finish it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote.sync: match the delete event's etag before deleting the object
A rename can copy another entry's content onto a key between the filer's
check and the remote delete, and the delete then removes the copy. The
delete event carries the object's etag; delete only when the live object
still matches it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: finish a rename's remote work outside the committed transaction
The SQL stores keep the transaction in the context and refuse every
call through it once it has committed. The work that runs after
CommitTransaction used that context, so on postgres2 the deferred
old-key delete could not look up the old path and kept the object, and
finishRemoteCopies could not read or stamp the entry. Both failed with
"sql: transaction has already been committed or rolled back" on every
copied rename; remote.sync cleaned up after them.
Keep the context from before the transaction for the work after the
commit. The rename test store gets a strict mode that fails calls
through a committed transaction the way database/sql does.
* filer: carry a deferred copy's stamp in the rename event; delete orphaned uploads
- The deferred copy stamps the entry with a plain UpdateEntry, which
publishes no event, so the queued rename event still carried the old
key's stamp to peer filers and subscribers. Give it the copy's
RemoteEntry before it is sent. The two notify loops share the code now.
- remote.sync kept any object whose ETag changed since the delete. With
no entry at the path that is an upload which finished after the
delete, and keeping it leaves it on the remote for good. Look up the
path first and keep a changed object only when an entry holds it.
* remote.sync: keep a changed object even with no entry at its path
Deleting a changed object when no entry holds the path also deletes a
copy a directory rename has written but not yet made visible, while a
delete of that path waits in the sync; the rename then removes the old
key and the content is gone. Keep any object whose ETag changed since
the delete, as before. An upload that finishes after the delete leaks
its object instead, which is the cheaper failure.
* filer: read a pending remote rename copy from the source key
A remote-only rename over a remote-only target commits before its copy
lands, so reads on the new path served the target's old object until the
copy stamped the entry — and indefinitely when the copy failed without
remote.sync running. The committed entry now records the renamed-from
path, and remote-only reads resolve that object's key while the stamp
is pending; the copy's stamp and the sync's completion stamp both clear
the marker.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: stamp the rename event with the deferred copy's result
A remote-only rename over a remote-only target queued the new entry's
event with the source's stamp before the deferred copy ran. Peers
replaying it saved a stamp the copy had already superseded, and the
sync serving such a peer rejected the destination as a match and tried
to copy the deleted source. The event now carries the copy's result
and drops the pending-source marker once it lands.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote.sync: delete only the object the event's stamp describes
An object at a deleted key is not always the one the delete meant: a
rename copy can land before its filer entry is visible, and a rewrite
can upload over it. When the event recorded no remote stamp, or the
live stat cannot confirm the recorded etag, the delete now keeps the
object instead of treating it as a late upload.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: tighten the pending rename source marker
Follow-ups to reading a pending remote rename copy from its source key:
- Only the filer sets the marker. The HTTP upload and tagging handlers
store Seaweed-* request headers as extended attributes, so they skip
this one, and a marker that leads out of the entry's own mount is
ignored, so it cannot point reads at another remote.
- A pending entry renamed again copies from the marker's source, not
from its own key, which still holds the old target's bytes. A further
deferred rename keeps naming the original source; an immediate copy
clears the marker.
- When remote.sync finds the destination already holding the content,
it stamps the entry (clearing the marker) before deleting the source,
and deletes the source only once the entry no longer reads it.
- A read that resolved the source just before the copy finished and
the source went away retries with the entry as it is now.
* filer, remote.sync: follow a pending rename's source across renames
When a remote-only entry is renamed over another remote-only file and
the deferred copy fails, it reads the source key through its marker.
Renamed again the same way, the marker keeps naming that source, but:
- remote.sync completed each rename from the event's old key. For the
second one that is the first target's old object, which it copied
over the destination before clearing the marker. It now copies from
the source the entry's marker names.
- For the first rename it found no entry at the new path and deleted
the old key, which the later entry still reads. When the event's new
entry carried a marker and the path is gone, it now leaves the key to
the entry that moved on.
- When the filer copies a pending entry immediately on a later rename,
it now also releases the source the copy read, after the commit and
with the same checks as the old key.
* remote.sync: release a pending rename's source, the filer leaves it
The filer deleted a pending entry's source after copying it on a later
rename, but that rename locks only its own paths: another rename can
copy onto the source key before its entry is visible, and the cleanup
then deleted that copy. And deleting a pending entry never released its
source at all.
The filer now leaves marked sources alone. remote.sync releases one
when the pending entry is deleted, or when a later rename was copied
right away, and only while the live object is still the one the
entry's stamp describes and no remote-only entry holds the path: a
copy that landed there since has a different ETag and is kept.
* remote.sync: release a pending source when the renamed entry is cached
A rename copied right away left its pending source to remote.sync,
which released it only while the destination was still remote-only.
Cached before the sync got there, the destination took the other path
and the source stayed on the remote for good. Release it there too,
with the same checks.
* remote.sync: release a pending source only after the destination is safe
e6d6438 released the source a renamed entry's old version read before
the destination was uploaded. A cached destination that still carried
the pending marker then had nothing behind it: had its upload failed,
remote.uncache could drop its chunks and leave it reading a deleted
source. Release the source only after the upload is stamped, and only
when the entry no longer reads through a marker.
* remote.sync: delete a rename's old key only if it is still the renamed content
completeRemoteOnlyRename deleted the old key once the destination held
the content. Another rename onto the vacated path can copy its object
there before the sync reaches the first rename, and that copy was then
deleted with it. Delete the old key only while the live object is still
the one the renamed entry's stamp describes.
Also keep the pending-source marker out of DELETE ?tagging, which
clears every Seaweed-* attribute: the marker is not a tag, and losing it
points reads back at the old target's bytes.
* remote.sync: compare a rename's old key against the source's own stamp
A retry can find the renamed entry already stamped with the copy, and a
copy of a multipart source has another ETag, so the unchanged source
was kept and stayed on the remote. Carry the stamp the event recorded
for the source (or the pending entry's, for a marked source) and compare
against that.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A remote-only entry whose object was deleted from the remote storage
outside the filer answered GET with 500 and stayed in the filer. The
remote's not-found was lost on the way: the backends' ReadFile returned
it as an untyped error, so FetchAndWriteNeedle failed with codes.Unknown
and nothing downstream could tell it from any other failure.
- remote_storage: GCS, S3 (NoSuchKey) and Azure (BlobNotFound) reads
return ErrRemoteObjectNotFound. GCS reports a missing bucket the same
way as a missing object, so it confirms the bucket with a listing.
- volume server and filer: the not-found crosses gRPC as codes.NotFound
carrying the sentinel's text, and the filer's cache RPC returns
codes.NotFound, which the S3 gateway already maps to NoSuchKey.
- s3api: the origin fallback answers NoSuchKey on a confirmed not-found.
- filer: a confirmed not-found removes the stale entry, so the lazy
remote-metadata cache converges on the remote. Only remote-only files
outside .versions and without an active object lock are removed, only
if unchanged since the fetch (checked on the object's write owner,
under the lock S3 object writes take), and with a metadata-only
delete: the filer skips its inline remote delete, and the delete
events' entries carry a marker that makes filer.remote.sync and
filer.remote.gateway skip their remote delete, while filer.sync still
replicates it. The replicated DeleteEntryRequest carries
keep_remote_object, so the destination's delete events are marked too.
The store drops the marker from every write, so clients cannot plant
it.
* pb: add multipart concurrency fields to RemoteConf and tier move requests
RemoteConf gains upload_concurrency/download_concurrency (0 = client
default); VolumeTierMoveDatToRemote/FromRemote requests gain a
concurrency field (0 = backend default).
* remote storage: honor RemoteConf upload/download concurrency in s3 and azure clients
s3 client: ReadFile passes conf download_concurrency to the downloader,
WriteFile uses upload_concurrency for the uploader; previously
hard-coded 1 upload / 5 download parts. 0 keeps defaults. Same for
azure client.
* storage: plumb concurrency through backend interface and tier upload/download
BackendStorage.CopyFile/DownloadFile take a concurrency hint (<=0 =
backend configured default); s3 backend reads
upload_concurrency/download_concurrency from scaffold config with
parseConcurrency fallback, rclone updated to the new signature. Tier
move gRPC handlers forward the request concurrency to the backend.
* shell: -upload_concurrency/-download_concurrency for remote.configure, -concurrent for volume.tier
remote.configure exposes upload/download concurrency persisted into
RemoteConf; volume.tier move/evict commands forward -concurrent to the
tier move requests. Documented in master-cloud.toml scaffold.
* test: cover concurrency propagation in remote tier integration test
* remote.configure: merge existing config on partial update
Load the stored RemoteConf before saving so a partial update (e.g. only
-upload_concurrency) preserves credentials, endpoints, and type instead
of replacing them with new-config defaults. Only treat a confirmed
ErrNotFound as a new configuration; propagate all other load errors so a
transient filer failure does not overwrite stored settings.
On a type transition, reset backend-specific fields to the destination
type's new-config defaults rather than inheriting the old backend's
empty values. Bound configured concurrency to a sane maximum.
* remote storage: honor configured download concurrency in S3 and Azure
ReadFileWithConcurrency now resolves a zero request override against the
client's configured download_concurrency (new downloadConcurrency()
helpers), so the remote-mount/cache read path honors
RemoteConf.DownloadConcurrency instead of the hard-coded default.
Azure also clamps the resolved value to math.MaxUint16 regardless of
whether the fallback was used, preventing uint16 wraparound when a
configured value exceeds 65535.
* shell: rename -concurrent to -concurrency and validate tier transfer bounds
Rename the -concurrent flag to -concurrency across volume.tier.upload,
volume.tier.download, and volume.tier.compact to match the proto field and
RemoteConf field names. Add validateTierConcurrency to reject values that
would wrap int32 or exceed a 1024 cap before constructing the request.
* server: clamp tier move concurrency in gRPC handlers
Add clampTierConcurrency to both VolumeTierMoveDatToRemote and
VolumeTierMoveDatFromRemote handlers so a direct gRPC caller cannot spawn
an unbounded number of network workers.
* trim verbose comments added with concurrency feature
Remove redundant doc comments on the backend interface, rclone backend,
s3_backend parseConcurrency, and test helpers that restated the obvious.
* remote.configure: apply type defaults before re-parse so explicit flags win
applyTypeDefaults ran after the second flag parse, overwriting explicit
destination flags (e.g. -s3.region=eu-west-1) with new-config defaults.
Move the type-transition default reset before the re-parse so user-supplied
flags override the destination defaults.
* remote.configure: only treat explicit -type as a type transition
The first parse defaults -type to s3, so a concurrency-only update on an
existing non-S3 config captured requestedType=s3 and wrongly triggered a
type transition, resetting the stored backend to S3. Use fs.Visit to
detect whether -type was explicitly supplied; an omitted -type keeps the
stored backend.
---------
Co-authored-by: Jack Meredith <9480542+jackusm@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* remote_storage: build S3-compatible clients through one constructor
The eight non-s3 S3-SDK providers each duplicated the AWS session setup
and only the s3 maker could take a custom *http.Client. Route every
S3-compatible type (s3, wasabi, b2, aliyun, tencent, baidu, filebase,
storj, contabo) through MakeWithHTTPClient with a single options table,
and add S3CompatibleEndpoint so callers can resolve the endpoint a given
type dials. No behavior change.
* volume: apply the remote-endpoint check to all S3-compatible providers
FetchAndWriteNeedle validated the endpoint and used the pinned dialer only
for type "s3". Every S3-SDK backend (wasabi, b2, aliyun, tencent, baidu,
filebase, storj, contabo) dials a caller-supplied endpoint through the same
client, so gate on S3CompatibleEndpoint to apply the same check uniformly.
-volume.allowUntrustedRemoteEndpoints still opts out.
* volume: don't route the guarded remote-endpoint client through a proxy
The guarded client exists to dial the validated endpoint directly and
re-check the resolved IP at connect time. With http.ProxyFromEnvironment
set, the dialer only validates the proxy's address while the proxy
re-resolves the endpoint host, which reopens the rebinding window. Drop
the proxy on this path; operators that need one can opt out with
-volume.allowUntrustedRemoteEndpoints.
* s3 remote: honor s3.support_tagging in UpdateFileMetadata
The write path already skips tagging when the remote is configured
without tagging support, but the metadata-update path sent
PutObjectTagging or DeleteObjectTagging unconditionally. On remotes
that reject tagging requests, any metadata update failed -- including
updates with no tags at all, which land on the DeleteObjectTagging
branch.
* s3 remote: drop the never-read supportTagging field
Every maker set it, nothing read it: the tagging decision is made from
conf.S3SupportTagging. Keeping a field that looks like the switch but
is not invites exactly the inconsistency the previous commit fixed.
* remote storage: actually delete objects when a directory is removed
On object-store backends RemoveDirectory returned nil without doing
anything, so a directory delete synced to the remote as a successful
no-op and the objects under that prefix stayed there forever. Nothing
surfaced the divergence: the sync logged rmdir, advanced its offset,
and the local namespace looked clean.
Deleting a bucket-level directory on a filer store that can drop a
whole bucket emits no per-child delete events at all, so the single
rmdir event was the only chance to clean up the remote.
Each backend now lists the prefix and deletes what it finds: S3 in
DeleteObjects batches of one listing page, GCS and Azure per object.
The prefix always ends with a slash so a sibling like dir2 survives
deleting dir, and errors propagate so a failed delete is retried
instead of silently skipped. A directory that maps to the bucket root
is left alone: wiping every object in the bucket from one namespace
event is too destructive, and bucket removal already has its own path.
* gcs remote: wrap the per-object delete error
The listing error in the same function already wraps, so the delete
error should stay inspectable with errors.Is as well.
* s3 remote: name the empty-listing test for what it checks
The prefix in that test is a normal directory; what is empty is the
listing. The bucket-root guard has its own test.
* s3 remote: report the scope of a failed delete batch
A DeleteObjects response can carry per-key errors for up to a
thousand keys. Surfacing only the first hid how much of the batch
failed, and surfacing all of them would build an unbounded error
string, so report the count with the first failure as the sample.
* remote storage: carry Content-Encoding into mounted entries
A RemoteEntry now records the remote object's Content-Encoding, and every
path that materializes a local entry from remote metadata (lazy fetch, lazy
listing, remote.mount, remote.meta.sync, remote.cache) stamps it into the
entry extended attributes, so HTTP and S3 HeadObject/GetObject return the
header. GCS and Azure populate it on listing and stat; S3 only exposes it
via HeadObject, so listings leave it empty.
* remote storage: set Content-Encoding when uploading to the remote
An entry carrying Content-Encoding in its extended attributes (a native S3
upload, or a value pulled from the remote) now keeps it when
filer.remote.sync or remote.copy.local writes the object to GCS, S3, or
Azure, instead of silently dropping it.
* gcs: read remote objects without decompressive transcoding
GCS transparently decompresses gzip-encoded objects on download, which
ignores range requests and returns byte counts that disagree with the
tracked RemoteSize. Request the stored bytes instead; chunked reads of
gzip-encoded objects then behave like any other object.
* remote storage: track Content-Encoding presence so removals propagate
A listing that does not report encodings (S3) leaves the field unset and
the local header untouched, while an authoritative report of no encoding
(GCS, Azure, any stat) now clears a previously stamped header instead of
leaving it stale. remote.cache also schedules a metadata update when only
the reported encoding changes.
* remote storage: propagate Content-Encoding on metadata-only updates
filer.remote.sync routes same-content changes through UpdateFileMetadata,
which only touched custom metadata (GCS, Azure) or tags (S3), so a
Content-Encoding change in the extended attributes never reached the
remote object's real header. GCS now patches contentEncoding alongside
the metadata, and Azure reissues the blob's HTTP headers with the new
value, carrying the others over since the call replaces the full set.
S3 stays tags-only: changing the header there means rewriting the
object, which the sync already does whenever content changes.
* remote.meta.sync: optional per-file stat for listing-omitted metadata
S3 listings carry no Content-Encoding, so entries synced from them never
learn it and the lazy-stat path never runs once an entry exists. With
-statFiles, each new or changed file whose listing left the encoding
unreported is stat-ed before reconciling, and the stat-derived value is
persisted so the next run only stats files that changed. Off by default:
it costs one remote request per file, and GCS and Azure listings already
carry the encoding.
* s3: apply metadata-only Content-Encoding changes with an in-place copy
Content-Encoding is S3 system metadata, so the tags-only metadata update
silently left the object's real header untouched. When the encoding
differs, reissue the object as a self-copy with replaced metadata,
carrying the content type and configured storage class like a fresh
write does. CopyObject caps at 5 GiB; beyond that the change is logged
and applies on the next content write.
* azure: skip the metadata call when user metadata is unchanged
An encoding-only change reissues the blob's HTTP headers; sending the
unchanged user metadata alongside it wastes a round trip and bumps the
blob's ETag once more than needed.
* s3: carry existing object metadata through the encoding copy
The replace directive drops everything not resent, and a mounted entry
usually has no local mime or user metadata, so the in-place copy wiped
the object's Content-Type, Cache-Control, user metadata, encryption
settings, and storage class. Read them back with a HeadObject first and
carry them over, overriding only what SeaweedFS manages: the encoding,
a locally set mime, and the configured storage class. S3 reports
Expires as a string while the copy input wants a time, so it is parsed
and skipped when malformed.
* s3: add streaming reader interface for remote storage
Add RemoteStorageStreamReader optional interface to support efficient
streaming of large remote objects without buffering entire file in memory.
This enables future stream-through caching where data can be served to
clients while simultaneously writing to volume servers.
Implement ReadFileAsStream() for S3, GCS, and Azure backends using their
native streaming APIs. This provides the foundation for improving TTFB
on large remote file access by serving data directly from remote storage
while background cache operation populates local chunks.
The streaming interface allows remote storage backends to return io.ReadCloser,
enabling efficient memory usage for multi-GB objects compared to the
current ReadFile() approach which buffers entire ranges in memory.
* s3: adaptive timeout for remote object caching to improve TTFB
Use size-aware cache polling timeout to balance cache-hit rate against
time-to-first-byte:
- Small files (<50MB): 10s timeout - more likely to complete caching
before timeout, improving subsequent request performance
- Medium files (50-500MB): 5s timeout - default balance
- Large files (>500MB): 2s timeout - fail-fast to improve initial TTFB
for very large downloads
This reduces waiting time for large remote files while maintaining
high cache-hit rate for smaller files that cache quickly.
* s3: address code review feedback for stream-through cache
- Move startBackgroundRemoteCache call after policy recheck to avoid
cache side effects for denied requests (authorization first)
- Make startBackgroundRemoteCache version-aware by accepting versionId
parameter and using buildVersionedRemoteObjectPath
- Add timeout (5 minutes) to background cache context to prevent
goroutine pile-up if RPC stalls under load
- Update cacheRemoteObjectForStreamingWithShortTimeout to return both
entry and error, allowing callers to distinguish transient errors
(timeout/cancellation) from permanent errors (not found, denied)
- Update streamFromVolumeServers to handle permanent cache errors with
appropriate HTTP status codes (404 for not found, 503 for transient)
* fix(remote): reject short reads when caching remote objects
A short read from the remote (stale listing size, truncated or flaky
response) was silently zero-padded: the S3 and Azure clients pre-size
the buffer and discard the downloaded byte count, and the chunk is
recorded with the requested size. The cached file then matched the
expected size but its tail was NULL, and the entry was marked cached
so it never re-fetched.
Check the byte count against the requested size in both clients, and
add a backend-agnostic guard in FetchAndWriteNeedle. The cache now
fails loudly and the entry stays remote-only for a later retry.
* fix(remote): match S3 default modes when syncing remote metadata
Remote object listings carry no POSIX mode, so synced entries were
created with a hardcoded 0644. Against a SeaweedFS remote, whose S3
layer writes objects as 0660 and auto-creates directories as 0771
(0660|0111), the mounted copy ended up 0644/0755 and the permissions
visibly diverged from the source.
Default to the S3 modes instead (files 0660, directories 0771). The
filer derives parent-dir modes from the child as fileMode|0111, so
fixing the file default also brings the directories into line.
Directory mtimes still reflect sync time: S3 listings don't enumerate
directories, so the remote's directory timestamps aren't available.
fix(remote_storage/s3): forward entry.Attributes.Mime as ContentType
filer.remote.sync was uploading every object without a Content-Type, so
S3-compatible backends (e.g. Backblaze B2) stored binary/octet-stream
and browsers refused to render HTML, CSS, etc.
Pass entry.Attributes.Mime through to UploadInput.ContentType, leaving
the header unset when no Mime is recorded so the remote keeps its own
default behavior.
volume: require admin auth and refuse loopback endpoints in FetchAndWriteNeedle
Gate the RPC behind checkGrpcAdminAuth for parity with the rest of the
destructive volume-server RPCs, and reject cluster-internal remote S3
endpoints (loopback / link-local / IMDS / RFC 1918 / CGNAT) before
dialing. Pin the validated address against DNS rebinding by routing the
AWS SDK through an HTTP transport whose DialContext re-resolves the host
and re-applies the deny list on every dial, so an endpoint that resolves
to a public IP at validate-time and then flips to 127.0.0.1 at connect
time is refused. Operators that legitimately fetch from private hosts
can opt out with -volume.allowUntrustedRemoteEndpoints.
* improve large file sync throughput for remote.cache and filer.sync
Three main throughput improvements:
1. Adaptive chunk sizing for remote.cache: targets ~32 chunks per file
instead of always starting at 5MB. A 500MB file now uses ~16MB chunks
(32 chunks) instead of 5MB chunks (100 chunks), reducing per-chunk
overhead (volume assign, gRPC call, needle write) by 3x.
2. Configurable concurrency at every layer:
- remote.cache chunk concurrency: -chunkConcurrency flag (default 8)
- remote.cache S3 download concurrency: -downloadConcurrency flag
(default raised from 1 to 5 per chunk)
- filer.sync chunk concurrency: -chunkConcurrency flag (default 32)
3. S3 multipart download concurrency raised from 1 to 5: the S3 manager
downloader was using Concurrency=1, serializing all part downloads
within each chunk. This alone can 5x per-chunk download speed.
The concurrency values flow through the gRPC request chain:
shell command → CacheRemoteObjectToLocalClusterRequest →
FetchAndWriteNeedleRequest → S3 downloader
Zero values in the request mean "use server defaults", maintaining
full backward compatibility with existing callers.
Ref #8481
* fix: use full maxMB for chunk size cap and remove loop guard
Address review feedback:
- Use full maxMB instead of maxMB/2 for maxChunkSize to avoid
unnecessarily limiting chunk size for very large files.
- Remove chunkSize < maxChunkSize guard from the safety loop so it
can always grow past maxChunkSize when needed to stay under 1000
chunks (e.g., extremely large files with small maxMB).
* address review feedback: help text, validation, naming, docs
- Fix help text for -chunkConcurrency and -downloadConcurrency flags
to say "0 = server default" instead of advertising specific numeric
defaults that could drift from the server implementation.
- Validate chunkConcurrency and downloadConcurrency are within int32
range before narrowing, returning a user-facing error if out of range.
- Rename ReadRemoteErr to readRemoteErr to follow Go naming conventions.
- Add doc comment to SetChunkConcurrency noting it must be called
during initialization before replication goroutines start.
- Replace doubling loop in chunk size safety check with direct
ceil(remoteSize/1000) computation to guarantee the 1000-chunk cap.
* address Copilot review: clamp concurrency, fix chunk count, clarify proto docs
- Use ceiling division for chunk count check to avoid overcounting
when file size is an exact multiple of chunk size.
- Clamp chunkConcurrency (max 1024) and downloadConcurrency (max 1024
at filer, max 64 at volume server) to prevent excessive goroutines.
- Always use ReadFileWithConcurrency when the client supports it,
falling back to the implementation's default when value is 0.
- Clarify proto comments that download_concurrency only applies when
the remote storage client supports it (currently S3).
- Include specific server defaults in help text (e.g., "0 = server
default 8") so users see the actual values in -h output.
* fix data race on executionErr and use %w for error wrapping
- Protect concurrent writes to executionErr in remote.cache worker
goroutines with a sync.Mutex to eliminate the data race.
- Use %w instead of %v in volume_grpc_remote.go error formatting
to preserve the error chain for errors.Is/errors.As callers.
When S3StorageClass is empty (the default), aws.String("") was passed
as the StorageClass in PutObject requests. While AWS S3 treats this as
"use default," S3-compatible providers (e.g. SharkTech) reject it with
InvalidStorageClass. Only set StorageClass when a non-empty value is
configured, letting the provider use its default.
Fixes#8644
* feat(filer): add lazy directory listing for remote mounts
Directory listings on remote mounts previously only queried the local
filer store. With lazy mounts the listing was empty; with eager mounts
it went stale over time.
Add on-demand directory listing that fetches from remote and caches
results with a 5-minute TTL:
- Add `ListDirectory` to `RemoteStorageClient` interface (delimiter-based,
single-level listing, separate from recursive `Traverse`)
- Implement in S3, GCS, and Azure backends using each platform's
hierarchical listing API
- Add `maybeLazyListFromRemote` to filer: before each directory listing,
check if the directory is under a remote mount with an expired cache,
fetch from remote, persist entries to the local store, then let existing
listing logic run on the populated store
- Use singleflight to deduplicate concurrent requests for the same directory
- Skip local-only entries (no RemoteEntry) to avoid overwriting unsynced uploads
- Errors are logged and swallowed (availability over consistency)
* refactor: extract xattr key to constant xattrRemoteListingSyncedAt
* feat: make listing cache TTL configurable per mount via listing_cache_ttl_seconds
Add listing_cache_ttl_seconds field to RemoteStorageLocation protobuf.
When 0 (default), lazy directory listing is disabled for that mount.
When >0, enables on-demand directory listing with the specified TTL.
Expose as -listingCacheTTL flag on remote.mount command.
* refactor: address review feedback for lazy directory listing
- Add context.Context to ListDirectory interface and all implementations
- Capture startTime before remote call for accurate TTL tracking
- Simplify S3 ListDirectory using ListObjectsV2PagesWithContext
- Make maybeLazyListFromRemote return void (errors always swallowed)
- Remove redundant trailing-slash path manipulation in caller
- Update tests to match new signatures
* When an existing entry has Remote != nil, we should merge remote metadata into it rather than replacing it.
* fix(gcs): wrap ListDirectory iterator error with context
The raw iterator error was returned without bucket/path context,
making it harder to debug. Wrap it consistently with the S3 pattern.
* fix(s3): guard against nil pointer dereference in Traverse and ListDirectory
Some S3-compatible backends may return nil for LastModified, Size, or
ETag fields. Check for nil before dereferencing to prevent panics.
* fix(filer): remove blanket 2-minute timeout from lazy listing context
Individual SDK operations (S3, GCS, Azure) already have per-request
timeouts and retry policies. The blanket timeout could cut off large
directory listings mid-operation even though individual pages were
succeeding.
* fix(filer): preserve trace context in lazy listing with WithoutCancel
Use context.WithoutCancel(ctx) instead of context.Background() so
trace/span values from the incoming request are retained for
distributed tracing, while still decoupling cancellation.
* fix(filer): use Store.FindEntry for internal lookups, add Uid/Gid to files, fix updateDirectoryListingSyncedAt
- Use f.Store.FindEntry instead of f.FindEntry for staleness check and
child lookups to avoid unnecessary lazy-fetch overhead
- Set OS_UID/OS_GID on new file entries for consistency with directories
- In updateDirectoryListingSyncedAt, use Store.UpdateEntry for existing
directories instead of CreateEntry to avoid deleteChunksIfNotNew and
NotifyUpdateEvent side effects
* fix(filer): distinguish not-found from store errors in lazy listing
Previously, any error from Store.FindEntry was treated as "not found,"
which could cause entry recreation/overwrite on transient DB failures.
Now check for filer_pb.ErrNotFound explicitly and skip entries or
bail out on real store errors.
* refactor(filer): use errors.Is for ErrNotFound comparisons
* feat: add statfile; add error for remote storage misses
* feat: statfile implementations for storage providers
* test: add unit tests for StatFile method across providers
Add comprehensive unit tests for the StatFile implementation covering:
- S3: interface compliance and error constant accessibility
- Azure: interface compliance, error constants, and field population
- GCS: interface compliance, error constants, error detection, and field population
Also fix variable shadowing issue in S3 and Azure StatFile implementations where
named return parameters were being shadowed by local variable declarations.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: address StatFile review feedback
- Use errors.New for ErrRemoteObjectNotFound sentinel
- Fix S3 HeadObject 404 detection to use awserr.Error code check
- Remove hollow field-population tests that tested nothing
- Remove redundant stdlib error detection tests
- Trim verbose doc comment on ErrRemoteObjectNotFound
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: address second round of StatFile review feedback
- Rename interface assertion tests to TestXxxRemoteStorageClientImplementsInterface
- Delegate readFileRemoteEntry to StatFile in all three providers
- Revert S3 404 detection to RequestFailure.StatusCode() check
- Fix double-slash in GCS error message format string
- Add storage type prefix to S3 error message for consistency
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: comments
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
