* s3 tier backend: bound connect and response-header waits
The tier client used the SDK default transport: a 30s dial timeout and no
limit on waiting for response headers. A blackholed endpoint cost about
two minutes per request with the SDK's retries, and an endpoint that
accepts connections but never answers held the request forever.
Dial with a 10s timeout and give up on a response whose headers do not
arrive within a minute.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: stop the disk scan from waiting on a stalled tier backend
A volume server registers with the master only after every volume on its
disks has loaded, and loading a tiered volume reads its superblock from the
remote object. With the S3 endpoint unreachable, each read waited out the
client's timeouts and retries on one of the loader workers, so start-up grew
with the number of tiered volumes and the local volumes were not served
until it finished.
A tiered volume already loads without its superblock when the read fails.
Give that read 10s, and once it stalls, let the backend's other volumes skip
it for a minute instead of each waiting out the timeout. The volume reads
from the backend as before once the endpoint answers again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: bound connect and read waits of the S3 tier client
Mirror the Go tier client: a 10s connect timeout instead of the SDK's
3.1s default, and a minute to receive a response, so an endpoint that
accepts connections but never answers cannot hold a read forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: stop volume loading from waiting on a stalled tier backend
Mirror the Go loader: the superblock read of a tiered volume gets 10s, and
once it stalls the backend's other volumes skip it for a minute. The
volume loads with its .vif version as before when the read fails, and
reads from the backend once the endpoint answers again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: keep the .vif replication when a remote superblock read fails
Go applies the .vif replication to a tiered volume whether or not its
superblock read succeeded, so a volume loaded while its backend is down
still reports its replica placement. The Rust loader applied it only on
success and reported the default placement instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: skip the .idx head repair while the tier backend is unreachable
The repair reads needle headers from the remote .dat, so after the
superblock read failed it would wait on the same backend again and hold
up the load. It runs on the next load that reaches the backend.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: skip the .idx head repair while the tier backend is unreachable
Mirrors the Go load: after the remote superblock read failed, the
repair's remote header reads would wait on the same backend again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 tier backend: give up a ranged read whose body stops arriving
The response-header timeout ends once headers arrive, so an endpoint
that answers and then sends no body held ReadAt, and a superblock
read's worker, forever. Cancel the request once the body makes no
progress for s3ReadTimeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: test that a tier read whose body stalls gives up
The SDK's stalled-stream protection already fails it, matching the Go
read's no-progress cancel.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): keep the S3 read timeout off upload_part
The SDK's read timeout runs from the start of the request, so a 64 MiB
part on a slow link would time out while its body is still being sent.
Go's ResponseHeaderTimeout only starts after the body is written.
upload_part now keeps the connect timeout and drops the read timeout;
stalled-stream protection still fails an upload that stops moving.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): bound the upload_part response wait from when the part is sent
Without a read timeout, an upload_part whose body was accepted but never
answered waited forever. The part body now goes to the connection in 64 KiB
chunks, and once the last chunk is taken the response must arrive within
S3_READ_TIMEOUT, like Go's ResponseHeaderTimeout. A retry rebuilds the body
and restarts that clock. The body keeps its in-memory contents, so signing
and the checksum header are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): keep a retried part's response clock from an earlier attempt
The send state was a single flag, and a retry body could be rebuilt and
fully sent before the waiter saw the flag drop, so the waiter kept the
earlier attempt's deadline. Each body now carries its attempt number, only
the latest attempt marks the part sent, and the deadline holds only while
no later attempt has started.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 backend: optional HTTP client settings for the cloud tier
The S3 tier backend builds its client from a bare aws.Config, so it always
uses Go's default transport: HTTP/2 wherever the endpoint offers it over TLS,
no cap on connections per host, and a 90s idle timeout. Against some
S3-compatible services that is a poor fit. Over HTTP/2 every upload and
range read of a tiered volume shares one connection and its flow-control
window; with a small window, tier uploads and concurrent reads of tiered
volumes become throughput-bound on that one connection, while HTTP/1.1 opens
a connection per concurrent request. Some services also limit parallel
connections per client address, or drop idle connections before 90s.
Add three optional settings to [storage.backend.s3.<id>], all off by
default so existing deployments keep the SDK's default client:
- disable_http2: keep the client on HTTP/1.1 (ForceAttemptHTTP2=false and an
empty TLSNextProto, the documented opt-out);
- max_connections: MaxConnsPerHost, 0 = no cap;
- idle_connection_timeout_seconds: IdleConnTimeout, 0 = Go's default.
The session cache key includes the options when any is set, so two backends
on the same endpoint with different settings do not share a client.
ToProperties reports the settings; the master.toml scaffold documents them.
Tests: TestS3BackendHTTPOptionsReachTheClient fails without the change (the
settings never reach the client); TestS3BackendDefaultHTTPClientUnchanged,
TestS3BackendSessionsDifferByHTTPOptions, TestParseNonNegative.
* s3 backend: reject an idle timeout too large for time.Duration
An idle_connection_timeout_seconds above math.MaxInt64/1e9 overflowed when
multiplied by time.Second. The negative result made the transport keep its
default timeout and ToProperties export a negative value. Warn and use the
default instead.
* s3 backend: parse the idle timeout as 64-bit seconds
parseIdleConnTimeout went through strconv.Atoi, so on 32-bit platforms
any value above 2147483647 seconds was rejected while the same value was
accepted on 64-bit, and the overflow test failed under GOARCH=386. Parse
with ParseInt(…, 64) and export the effective value with FormatInt so the
accepted range is the same on every platform.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
* pb: add multipart concurrency fields to RemoteConf and tier move requests
RemoteConf gains upload_concurrency/download_concurrency (0 = client
default); VolumeTierMoveDatToRemote/FromRemote requests gain a
concurrency field (0 = backend default).
* remote storage: honor RemoteConf upload/download concurrency in s3 and azure clients
s3 client: ReadFile passes conf download_concurrency to the downloader,
WriteFile uses upload_concurrency for the uploader; previously
hard-coded 1 upload / 5 download parts. 0 keeps defaults. Same for
azure client.
* storage: plumb concurrency through backend interface and tier upload/download
BackendStorage.CopyFile/DownloadFile take a concurrency hint (<=0 =
backend configured default); s3 backend reads
upload_concurrency/download_concurrency from scaffold config with
parseConcurrency fallback, rclone updated to the new signature. Tier
move gRPC handlers forward the request concurrency to the backend.
* shell: -upload_concurrency/-download_concurrency for remote.configure, -concurrent for volume.tier
remote.configure exposes upload/download concurrency persisted into
RemoteConf; volume.tier move/evict commands forward -concurrent to the
tier move requests. Documented in master-cloud.toml scaffold.
* test: cover concurrency propagation in remote tier integration test
* remote.configure: merge existing config on partial update
Load the stored RemoteConf before saving so a partial update (e.g. only
-upload_concurrency) preserves credentials, endpoints, and type instead
of replacing them with new-config defaults. Only treat a confirmed
ErrNotFound as a new configuration; propagate all other load errors so a
transient filer failure does not overwrite stored settings.
On a type transition, reset backend-specific fields to the destination
type's new-config defaults rather than inheriting the old backend's
empty values. Bound configured concurrency to a sane maximum.
* remote storage: honor configured download concurrency in S3 and Azure
ReadFileWithConcurrency now resolves a zero request override against the
client's configured download_concurrency (new downloadConcurrency()
helpers), so the remote-mount/cache read path honors
RemoteConf.DownloadConcurrency instead of the hard-coded default.
Azure also clamps the resolved value to math.MaxUint16 regardless of
whether the fallback was used, preventing uint16 wraparound when a
configured value exceeds 65535.
* shell: rename -concurrent to -concurrency and validate tier transfer bounds
Rename the -concurrent flag to -concurrency across volume.tier.upload,
volume.tier.download, and volume.tier.compact to match the proto field and
RemoteConf field names. Add validateTierConcurrency to reject values that
would wrap int32 or exceed a 1024 cap before constructing the request.
* server: clamp tier move concurrency in gRPC handlers
Add clampTierConcurrency to both VolumeTierMoveDatToRemote and
VolumeTierMoveDatFromRemote handlers so a direct gRPC caller cannot spawn
an unbounded number of network workers.
* trim verbose comments added with concurrency feature
Remove redundant doc comments on the backend interface, rclone backend,
s3_backend parseConcurrency, and test helpers that restated the obvious.
* remote.configure: apply type defaults before re-parse so explicit flags win
applyTypeDefaults ran after the second flag parse, overwriting explicit
destination flags (e.g. -s3.region=eu-west-1) with new-config defaults.
Move the type-transition default reset before the re-parse so user-supplied
flags override the destination defaults.
* remote.configure: only treat explicit -type as a type transition
The first parse defaults -type to s3, so a concurrency-only update on an
existing non-S3 config captured requestedType=s3 and wrongly triggered a
type transition, resetting the stored backend to S3. Use fs.Visit to
detect whether -type was explicitly supplied; an omitted -type keeps the
stored backend.
---------
Co-authored-by: Jack Meredith <9480542+jackusm@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix: report short S3 ReaderAt reads
Problem: S3BackendStorageFile.ReadAt returned a short buffer with a nil error, hiding truncated remote data.
Root cause: Every terminal io.EOF was cleared regardless of how many bytes were read.
Fix: Clear io.EOF only when the requested buffer was completely filled.
Validation: go test ./weed/storage/backend/...
Co-authored-by: Codex <noreply@openai.com>
* fix: validate S3 ReaderAt requests
Co-authored-by: Codex <noreply@openai.com>
* s3 ReadAt: simplify negative offset error
* s3 ReadAt: stop reading once the buffer is full
Skips the extra Read that only collected the terminal EOF, and bounds
the loop if the server ignores the Range header and returns more data
than requested, where Read on an empty slice can spin forever.
* s3 ReadAt: cover full reads that arrive with io.EOF
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* tiering: stop a shared remote object being deleted while replicas still point at it
A remote-tiered volume's .dat content lives only in one cloud object that all
N replica .vif files point at. Deleting that object while destroying any one
replica, or before a downloaded replica is durable, bricks the survivors.
- volume.tier.move cleanup now deletes old replicas with keepRemoteData=true so
surviving replicas keep the shared object. Document why the alreadyPlaced
anchor needs no replica sync (same-object replicas are byte-identical).
- VolumeTierMoveDatFromRemote now fsyncs the downloaded .dat, fsyncs the
containing directory, trims the .vif (fsynced) and swaps to the local DiskFile
BEFORE deleting the remote object, on both the keep-remote and delete paths.
Only the final DeleteFile is gated by keep_remote_dat_file, so a keep-remote
download leaves the replica served from local disk rather than the shared
object, and a crash before delete merely leaks the object.
- volume.tier.download keeps the shared object for every replica except the
last, which deletes it.
- s3 and rclone download paths fsync the .dat before close.
* storage: swap the volume data backend under the data lock
The tier-download swap closed v.DataBackend and assigned the new local DiskFile
without holding dataFileAccessLock, racing concurrent reads/writes (use of a
closed file / nil deref). Add an exported Volume.SwapDataBackend that performs
the close-and-replace under the lock, and call it from the tier download.
* server: skip directory fsync on Windows in the tier download path
os.Open(dir).Sync() is unsupported on Windows and returns an error, which would
fail VolumeTierMoveDatFromRemote entirely there. Skip the directory fsync on
Windows, matching how the storage-side helper tolerates the unsupported case.
* shell: make multi-replica tier.download resilient to already-local replicas
If a multi-replica download is interrupted and retried, a replica made local
in the prior attempt returns "already on local disk", which aborted the whole
command and left the remaining remote replicas dangling. Treat that case as a
skip-and-continue so a retry completes the rest.
* server: assert downloaded .dat content, not just length, in the tier test
A length-only check passes even if the bytes are corrupted; compare the full
content of the local .dat against the original.
* chore(weed/storage/backend/s3_backend): remove unused function
* fix(s3_backend): cache session under the composite region|endpoint key
createSession looked up sessions by region|endpoint but stored them by
region alone, so the cache never hit and a new session was built every
call. With getSession gone the lock can also drop to a plain Mutex.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* Replace removeDuplicateSlashes with NormalizeObjectKey
Use s3_constants.NormalizeObjectKey instead of removeDuplicateSlashes in most places
for consistency. NormalizeObjectKey handles both duplicate slash removal and ensures
the path starts with '/', providing more complete normalization.
* Fix double slash issues after NormalizeObjectKey
After using NormalizeObjectKey, object keys have a leading '/'. This commit ensures:
- getVersionedObjectDir strips leading slash before concatenation
- getEntry calls receive names without leading slash
- String concatenation with '/' doesn't create '//' paths
This prevents path construction errors like:
/buckets/bucket//object (wrong)
/buckets/bucket/object (correct)
* ensure object key leading "/"
* fix compilation
* fix: Strip leading slash from object keys in S3 API responses
After introducing NormalizeObjectKey, all internal object keys have a
leading slash. However, S3 API responses must return keys without
leading slashes to match AWS S3 behavior.
Fixed in three functions:
- addVersion: Strip slash for version list entries
- processRegularFile: Strip slash for regular file entries
- processExplicitDirectory: Strip slash for directory entries
This ensures ListObjectVersions and similar APIs return keys like
'bar' instead of '/bar', matching S3 API specifications.
* fix: Normalize keyMarker for consistent pagination comparison
The S3 API provides keyMarker without a leading slash (e.g., 'object-001'),
but after introducing NormalizeObjectKey, all internal object keys have
leading slashes (e.g., '/object-001').
When comparing keyMarker < normalizedObjectKey in shouldSkipObjectForMarker,
the ASCII value of '/' (47) is less than 'o' (111), causing all objects
to be incorrectly skipped during pagination. This resulted in page 2 and
beyond returning 0 results.
Fix: Normalize the keyMarker when creating versionCollector so comparisons
work correctly with normalized object keys.
Fixes pagination tests:
- TestVersioningPaginationOver1000Versions
- TestVersioningPaginationMultipleObjectsManyVersions
* refactor: Change NormalizeObjectKey to return keys without leading slash
BREAKING STRATEGY CHANGE:
Previously, NormalizeObjectKey added a leading slash to all object keys,
which required stripping it when returning keys to S3 API clients and
caused complexity in marker normalization for pagination.
NEW STRATEGY:
- NormalizeObjectKey now returns keys WITHOUT leading slash (e.g., 'foo/bar' not '/foo/bar')
- This matches the S3 API format directly
- All path concatenations now explicitly add '/' between bucket and object
- No need to strip slashes in responses or normalize markers
Changes:
1. Modified NormalizeObjectKey to strip leading slash instead of adding it
2. Fixed all path concatenations to use:
- BucketsPath + '/' + bucket + '/' + object
instead of:
- BucketsPath + '/' + bucket + object
3. Reverted response key stripping in:
- addVersion()
- processRegularFile()
- processExplicitDirectory()
4. Reverted keyMarker normalization in findVersionsRecursively()
5. Updated matchesPrefixFilter() to work with keys without leading slash
6. Fixed paths in handlers:
- s3api_object_handlers.go (GetObject, HeadObject, cacheRemoteObjectForStreaming)
- s3api_object_handlers_postpolicy.go
- s3api_object_handlers_tagging.go
- s3api_object_handlers_acl.go
- s3api_version_id.go (getVersionedObjectDir, getVersionIdFormat)
- s3api_object_versioning.go (getObjectVersionList, updateLatestVersionAfterDeletion)
All versioning tests pass including pagination stress tests.
* adjust format
* Update post policy tests to match new NormalizeObjectKey behavior
- Update TestPostPolicyKeyNormalization to expect keys without leading slashes
- Update TestNormalizeObjectKey to expect keys without leading slashes
- Update TestPostPolicyFilenameSubstitution to expect keys without leading slashes
- Update path construction in tests to use new pattern: BucketsPath + '/' + bucket + '/' + object
* Fix ListObjectVersions prefix filtering
Remove leading slash addition to prefix parameter to allow correct filtering
of .versions directories when listing object versions with a specific prefix.
The prefix parameter should match entry paths relative to bucket root.
Adding a leading slash was breaking the prefix filter for paginated requests.
Fixes pagination issue where second page returned 0 versions instead of
continuing with remaining versions.
* no leading slash
* Fix urlEscapeObject to add leading slash for filer paths
NormalizeObjectKey now returns keys without leading slashes to match S3 API format.
However, urlEscapeObject is used for filer paths which require leading slashes.
Add leading slash back after normalization to ensure filer paths are correct.
Fixes TestS3ApiServer_toFilerPath test failures.
* adjust tests
* normalize
* Fix: Normalize prefixes and markers in LIST operations using NormalizeObjectKey
Ensure consistent key normalization across all S3 operations (GET, PUT, LIST).
Previously, LIST operations were not applying the same normalization rules
(handling backslashes, duplicate slashes, leading slashes) as GET/PUT operations.
Changes:
- Updated normalizePrefixMarker() to call NormalizeObjectKey for both prefix and marker
- This ensures prefixes with leading slashes, backslashes, or duplicate slashes are
handled consistently with how object keys are normalized
- Fixes Parquet test failures where pads.write_dataset creates implicit directory
structures that couldn't be discovered by subsequent LIST operations
- Added TestPrefixNormalizationInList and TestListPrefixConsistency tests
All existing LIST tests continue to pass with the normalization improvements.
* Add debugging logging to LIST operations to track prefix normalization
* Fix: Remove leading slash addition from GetPrefix to work with NormalizeObjectKey
The NormalizeObjectKey function removes leading slashes to match S3 API format
(e.g., 'foo/bar' not '/foo/bar'). However, GetPrefix was adding a leading slash
back, which caused LIST operations to fail with incorrect path handling.
Now GetPrefix only normalizes duplicate slashes without adding a leading slash,
which allows NormalizeObjectKey changes to work correctly for S3 LIST operations.
All Parquet integration tests now pass (20/20).
* Fix: Handle object paths without leading slash in checkDirectoryObject
NormalizeObjectKey() removes the leading slash to match S3 API format.
However, checkDirectoryObject() was assuming the object path has a leading
slash when processing directory markers (paths ending with '/').
Now we ensure the object has a leading slash before processing it for
filer operations.
Fixes implicit directory marker test (explicit_dir/) while keeping
Parquet integration tests passing (20/20).
All tests pass:
- Implicit directory tests: 6/6
- Parquet integration tests: 20/20
* Fix: Handle explicit directory markers with trailing slashes
Explicit directory markers created with put_object(Key='dir/', ...) are stored
in the filer with the trailing slash as part of the name. The checkDirectoryObject()
function now checks for both:
1. Explicit directories: lookup with trailing slash preserved (e.g., 'explicit_dir/')
2. Implicit directories: lookup without trailing slash (e.g., 'implicit_dir')
This ensures both types of directory markers are properly recognized.
All tests pass:
- Implicit directory tests: 6/6 (including explicit directory marker test)
- Parquet integration tests: 20/20
* Fix: Preserve trailing slash in NormalizeObjectKey
NormalizeObjectKey now preserves trailing slashes when normalizing object keys.
This is important for explicit directory markers like 'explicit_dir/' which rely
on the trailing slash to be recognized as directory objects.
The normalization process:
1. Notes if trailing slash was present
2. Removes duplicate slashes and converts backslashes
3. Removes leading slash for S3 API format
4. Restores trailing slash if it was in the original
This ensures explicit directory markers created with put_object(Key='dir/', ...)
are properly normalized and can be looked up by their exact name.
All tests pass:
- Implicit directory tests: 6/6
- Parquet integration tests: 20/20
* clean object
* Fix: Don't restore trailing slash if result is empty
When normalizing paths that are only slashes (e.g., '///', '/'), the function
should return an empty string, not a single slash. The fix ensures we only
restore the trailing slash if the result is non-empty.
This fixes the 'just_slashes' test case:
- Input: '///'
- Expected: ''
- Previous: '/'
- Fixed: ''
All tests now pass:
- Unit tests: TestNormalizeObjectKey (13/13)
- Implicit directory tests: 6/6
- Parquet integration tests: 20/20
* prefixEndsOnDelimiter
* Update s3api_object_handlers_list.go
* Update s3api_object_handlers_list.go
* handle create directory
