* fix: report short S3 ReaderAt reads
Problem: S3BackendStorageFile.ReadAt returned a short buffer with a nil error, hiding truncated remote data.
Root cause: Every terminal io.EOF was cleared regardless of how many bytes were read.
Fix: Clear io.EOF only when the requested buffer was completely filled.
Validation: go test ./weed/storage/backend/...
Co-authored-by: Codex <noreply@openai.com>
* fix: validate S3 ReaderAt requests
Co-authored-by: Codex <noreply@openai.com>
* s3 ReadAt: simplify negative offset error
* s3 ReadAt: stop reading once the buffer is full
Skips the extra Read that only collected the terminal EOF, and bounds
the loop if the server ignores the Range header and returns more data
than requested, where Read on an empty slice can spin forever.
* s3 ReadAt: cover full reads that arrive with io.EOF
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix: reject overflowing needle ID deltas
Problem: Parsing a file ID with a delta can wrap a valid maximum needle ID back to zero without returning an error.
Root cause: Needle.ParsePath added the parsed uint64 delta without checking whether the sum exceeded the needle ID range.
Fix: Compare the delta with the remaining uint64 capacity before addition and return a contextual overflow error when it does not fit.
Validation: go test ./weed/storage/needle -run ^TestNeedleParsePathRejectsDeltaOverflow$ -count=1; go test ./weed/storage/needle -count=1; git diff --check 10cdaf381875492a2c752d1038797e96ff18208f..HEAD
Co-authored-by: Codex <noreply@openai.com>
* fix: propagate needle ID delta parse errors
Co-authored-by: Codex <noreply@openai.com>
* print the needle id in hex in the delta overflow error
* batch delete: keep processing after a cookie mismatch
* rust volume: reject overflowing needle id deltas
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* feat: add collection pattern to delete empty volumes
Co-authored-by: Codex <noreply@openai.com>
* shell: match collection pattern with wildcard matcher
Use wildcard.MatchesWildcard in the shared collection-pattern helper,
matching command_volume_fix_replication's matchCollectionPattern. The
flag only advertises '*' and '?', which is exactly what the matcher
supports.
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* feat: add collection.mark shell command
Add collection.mark to mark all existing normal volume replicas in a collection as readonly or writable. The command runs in preview mode by default and requires -apply to execute changes. It reuses existing volume mark RPCs, supports default collection aliases, skips EC shards, and adds unit tests for option parsing and target collection logic.
* Revert "feat: add collection.mark shell command"
This reverts commit 50c2bbf94c.
* feat: support marking volumes by collection
Add a -collection option to volume.mark so operators can mark every normal volume replica in a collection using existing topology data and volume mark RPCs.
The change keeps the single-volume path unchanged and adds tests for collection target selection, EC shard exclusion, and argument validation.
Co-authored-by: Codex <noreply@openai.com>
* volume.mark: reuse eachDataNode for collection traversal
* volume.mark: continue past per-volume failures and report progress
Collection marking aborted on the first failed RPC, leaving the
collection half-marked with no record of which volumes succeeded.
Mark every reachable volume, print per-volume progress to the writer,
and return an aggregated error naming the failures.
* volume.mark: let -collection _default target the unnamed collection
Other volume commands use the _default sentinel to match volumes with
no named collection; volume.mark could not reach them at all. Map
_default to the empty collection name in the filter.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(s3api): preserve requested AES256 copy encryption
Problem
CopyObject metadata processing ignored an explicit x-amz-server-side-encryption: AES256 request header. A destination copy could lose the requested SSE-S3 metadata even though KMS requests were handled.
Root cause
processMetadataBytes only wrote the destination SSE header when the requested algorithm was aws:kms. Any other explicit SSE algorithm fell through to the source-preservation branch.
Fix
Write the requested SSE algorithm whenever x-amz-server-side-encryption is present, and keep KMS-specific metadata handling limited to aws:kms.
Co-authored-by: Codex <noreply@openai.com>
* fix(s3api): reject unsupported copy encryption algorithms
A mistyped or unsupported x-amz-server-side-encryption value on a copy
request slipped past validation and got persisted as the destination's
algorithm header, advertising encryption that was never applied. Reject
anything other than AES256 or aws:kms up front.
* fix(s3api): write SSE key metadata for empty encrypted copies
A zero-byte source copied with an explicit SSE request took the
no-content branch and never ran the encryption path, leaving the object
with a bare algorithm header but no key. HEAD then advertised SSE while
the encryption-state machine saw the header as orphaned. Run the inline
encryption path when the destination requests encryption so the key
metadata is written too.
* s3api: use SSEAlgorithmKMS constant in copy metadata handling
* test(s3api): cover source SSE preservation on copy
* test(iam): allow the local client's real source IP in SourceIp tests
The aws:SourceIp allow policies hardcoded the loopback CIDRs, but a CI
runner reaching the server over localhost can be observed with one of the
host's RFC1918 addresses (the S3 endpoint is advertised on a 10.x
interface), so the positive-condition PutObject was denied and the allow
assertion flaked while the deny path passed trivially. Broaden the allow
list to loopback plus private ranges via a shared helper, and log the
denial on each failed attempt so any residual failure is diagnosable.
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(s3api): sort repeated SigV4 query values
Problem
SigV4 canonical query strings must sort repeated parameter values. SeaweedFS preserved the incoming value order, so a correctly signed request with repeated query parameters could fail verification when the request order differed from canonical order.
Root cause
getCanonicalQueryString delegated directly to url.Values.Encode(), which sorts parameter names but preserves each key's value slice order.
Fix
Sort every query value slice before encoding the canonical query string, while still excluding X-Amz-Signature for presigned URLs.
Reproduction
go test ./weed/s3api -run TestGetCanonicalQueryStringSortsRepeatedValues -count=1 failed before the fix with partNumber=2 before partNumber=10.
Co-authored-by: Codex <noreply@openai.com>
* test(s3api): expand canonical query coverage
Co-authored-by: Codex <noreply@openai.com>
---------
Co-authored-by: Codex <noreply@openai.com>
* fix(shell): return fs.verify topology errors
Problem: fs.verify can silently ignore master topology lookup failures before verifying files.
Root cause: commandFsVerify.Do returned parseErr after collectVolumeIds failed, but parseErr is nil once path parsing succeeds.
Fix: Return the actual collectVolumeIds error so VolumeList and master client failures stop the command.
Co-authored-by: Codex <noreply@openai.com>
* remove the tests
---------
Co-authored-by: Codex <noreply@openai.com>
* fix(command): preserve fuse option after writers
Problem: FUSE mount option parsing skipped the option immediately following concurrentWriters, so values such as concurrentReaders could be silently ignored.
Root cause: runFuse incremented the options loop index inside the concurrentWriters case in addition to the loop increment.
Co-authored-by: Codex <noreply@openai.com>
* test: save and restore mountOptions pointers, not their values
The fields are reassigned to fresh heap variables during runFuse, so
dereferencing to back up/restore mutated throwaways instead of the
flag-bound originals and could nil-panic on unset fields.
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Problem: Signature V4 SignedHeaders parsing accepted empty header name segments such as host; or ;host. Malformed Authorization headers could continue into signature verification instead of failing during header parsing.
Root cause: parseSignedHeader only checked that the SignedHeaders value was non-empty, then split it on semicolons without validating each element.
Fix: reject empty or whitespace-only signed header elements with ErrMissingFields before returning the parsed header list.
Reproduction: go test ./weed/s3api -run TestParseSignedHeaderRejectsEmptyHeaderNames -count=1 failed before the fix because SignedHeaders=host; returned ErrNone.
Validation: gofmt -w weed/s3api/auth_signature_v4.go weed/s3api/auth_signature_v4_test.go; git diff --check; go test ./weed/s3api -run TestParseSignedHeaderRejectsEmptyHeaderNames -count=1; go test ./weed/s3api -count=1
Co-authored-by: Codex <noreply@openai.com>
Problem: SigV4 chunked upload checksum trailer parsing rejected mixed-case checksum header names even though HTTP header field names are case-insensitive.
Root cause: extractChecksumAlgorithm compared the x-amz-trailer value and trailer header key against exact lowercase strings.
Fix: Trim and lowercase checksum trailer header names before matching supported checksum algorithms.
Reproduction: go test ./weed/s3api -run TestExtractChecksumAlgorithmIsCaseInsensitive -count=1 with X-Amz-Checksum-Crc32; before the fix it returned unsupported checksum algorithm.
Validation: gofmt -w weed/s3api/chunked_reader_v4.go weed/s3api/chunked_reader_v4_test.go; git diff --check; go test ./weed/s3api -run TestExtractChecksumAlgorithmIsCaseInsensitive -count=1; go test ./weed/s3api -count=1
Co-authored-by: Codex <noreply@openai.com>
Problem: Re-storing object-lock default retention with Days left a previous Years extended attribute in place, so later loads could see both Days and stale Years.
Root cause: StoreObjectLockConfigurationInExtended only wrote period fields that were set on the new configuration and did not delete old Days or Years keys before writing the replacement rule.
Fix: Clear stored default-retention Days and Years keys before writing the current default retention period fields.
Reproduction: go test ./weed/s3api -run TestStoreObjectLockConfigurationClearsStaleYears -count=1 failed before the fix because the stale years key remained.
Validation: go test ./weed/s3api -run TestStoreObjectLockConfigurationClearsStaleYears -count=1; go test ./weed/s3api -count=1; git diff --check; git diff --cached --check
Co-authored-by: Codex <noreply@openai.com>
Problem: RandomUint64 generated eight random bytes but returned int32, truncating the value before mount file and directory handles converted it to uint64. This reduced handle entropy to 32 bits and produced sign-extended handle values.\n\nRoot cause: the helper cast BytesToUint64 to int32 and exposed int32 as its return type.\n\nFix: make RandomUint64 return uint64 and return the full BytesToUint64 result.\n\nReproduction: go test ./weed/util -run TestRandomUint64ReturnsUint64 -count=1 failed before the fix because RandomUint64() had kind int32.\n\nValidation: gofmt -w weed/util/bytes.go weed/util/bytes_test.go; git diff --check; go test ./weed/util -run TestRandomUint64ReturnsUint64 -count=1; go test ./weed/util -count=1; go test ./weed/mount -count=1; git diff --cached --check
* fix(s3api): accept HTTP-date conditionals
Problem: Object conditional headers rejected valid HTTP-date values in RFC850 or ANSIC format for If-Modified-Since and If-Unmodified-Since.
Root cause: parseConditionalHeaders used time.Parse(time.RFC1123), accepting only one HTTP-date representation instead of the standard formats accepted by net/http.ParseTime.
Fix: Parse conditional date headers with http.ParseTime so RFC1123, RFC850, and ANSIC HTTP-date forms are accepted.
Reproduction: go test ./weed/s3api -run TestParseConditionalHeadersAcceptsHTTPDateFormats -count=1 failed before the fix with ErrInvalidRequest for RFC850 and ANSIC date values.
Validation: env GOCACHE=/private/tmp/seaweedfs-go-cache go test ./weed/s3api -run TestParseConditionalHeadersAcceptsHTTPDateFormats -count=1; env GOCACHE=/private/tmp/seaweedfs-go-cache go test ./weed/s3api -count=1; git diff --check; git diff --cached --check
* fix(s3api): accept HTTP-date copy-source conditionals
Mirror the put-path http.ParseTime switch onto the copy-source If-Modified-Since / If-Unmodified-Since headers, which still rejected valid RFC850 and ANSIC dates.
* fix(s3api): keep RFC1123 UTC-zone dates working alongside http.ParseTime
http.ParseTime rejects the "UTC" zone that Go clients emit via t.UTC().Format(time.RFC1123), which the old RFC1123 parser accepted. Add a parseHTTPDate helper that tries http.ParseTime first and falls back to RFC1123, so the put and copy-source conditional date headers accept the union of HTTP-date formats plus the UTC zone.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Problem: Default object-lock retention accepted an explicitly provided Years value of zero, even though a default retention period must be positive when present.
Root cause: validateDefaultRetention rejected zero Days but only rejected negative Years, leaving YearsSet with Years=0 as a successful validation path.
Fix: Treat an explicitly provided zero Years value as ErrInvalidRetentionPeriod, matching the existing Days validation.
Reproduction: go test ./weed/s3api -run TestValidateDefaultRetention -count=1 failed before the fix because the Zero years case returned nil.
Validation: go test ./weed/s3api -run TestValidateDefaultRetention -count=1; go test ./weed/s3api -count=1; git diff --check; git diff --cached --check
* fix(s3api): require space in v2 auth prefix
Problem: Signature V2 Authorization headers with a malformed algorithm token such as AWSX... are accepted as if they were AWS ... headers.
Root cause: validateV2AuthHeader checks HasPrefix("AWS") but then slices past an assumed trailing space, so an extra character after AWS is skipped and the rest is parsed as credentials.
Fix: Require the Authorization header to start with the exact AWS plus space prefix before parsing fields.
Reproduction: go test ./weed/s3api -run 'TestValidateV2AuthHeader/algorithm_prefix_without_space|TestDoesSignV2Match/malformed_auth_-_no_space_after_AWS' -count=1 fails before the fix because AWSXAKIA... is accepted.
Validation: go test ./weed/s3api -run 'TestValidateV2AuthHeader/algorithm_prefix_without_space|TestDoesSignV2Match/malformed_auth_-_no_space_after_AWS' -count=1; go test ./weed/s3api -count=1; git diff --check; git diff --cached --check
* Update weed/s3api/auth_signature_v2.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Problem: Multipart uploads where the first part was a form field and a later part contained the file used the first part's Content-Type for the file metadata.
Root cause: After finding a later part with a filename, parseUpload copied data and MD5 from part2 but read Content-Type from the original part variable.
Fix: Read Content-Type from the discovered file part.
Reproduction: go test ./weed/storage/needle -run TestParseUploadUsesDiscoveredFilePartContentType -count=1 failed before the fix because the parsed MIME type was text/plain instead of application/x-seaweed-test.
Validation: go test ./weed/storage/needle -run TestParseUploadUsesDiscoveredFilePartContentType -count=1; go test ./weed/storage/needle -count=1; git diff --check; git diff --cached --check
* fix: log meta backup offset errors
* fix: log meta backup offset errors
* fix: exit on meta backup offset errors
Exit with a non-zero status when the initial metadata backup offset cannot be persisted.
Classify offset-read failures during streaming so the backup process exits instead of retrying forever, allowing supervisors to restart and bootstrap from a missing checkpoint.
* meta backup: read offset in the loop, drop offset error type
Reading the saved offset inside the retry loop makes an offset read
failure a clean exit and a stream error a retry, without a typed error
to tell them apart. streamMetadataBackup now takes the start time.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Explain:
- problem: Delete and DeleteProxied could panic on malformed URLs when a JWT was provided.
- root cause: maybeAddAuth was called before checking the error returned by http.NewRequest, so req could be nil.
- fix: return the request construction error before adding the Authorization header.
- validation: go test ./weed/util/http -run 'TestDelete(ReturnsInvalidRequestErrorBeforeAddingAuth|ProxiedReturnsInvalidRequestErrorBeforeAddingAuth)' -count=1; git diff --check
* fix(http): handle invalid gzip stream errors
Explain:
- problem: ReadUrlAsStream could panic when a response claimed gzip encoding but the body was not a valid gzip stream.
- root cause: the gzip reader error was ignored and a nil reader was deferred and read from.
- fix: return the gzip.NewReader error before registering Close or reading.
- validation: go test ./weed/util/http -run TestReadUrlAsStreamReturnsGzipReaderError -count=1; git diff --check.
* test: avoid closing shared global HTTP client in unit test
