mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-23 06:20:48 +02:00
137d29633b2b5d62da7fa63b0da7b43caff08822
15653
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Lu
|
137d29633b | s3: fix prefix filtering and delimiter pagination (#11962) | ||
|
github-actions[bot]
|
ca0314ea4f | docs: regenerate star history chart | ||
|
Chris LuandJavier de la Torre
|
82f0a82564 |
filer+volume: make acknowledged writes durable under the fsync rule (#11943)
* volume: never compact a volume tiered to remote storage A tiered volume's data backend is the remote object, but compaction reads the local .dat path. When tiering removed the local .dat, every vacuum of a tiered volume with garbage fails with "no such file or directory", and the master's sweep and the worker's vacuum task keep selecting it. When the local .dat was kept, the compaction succeeds against the stale copy and the commit reloads the volume to read the remote object (old layout) through the compacted index, so live needles stop reading. Mirror the Rust volume server (#11545): - CompactByIndex and CompactByVolumeData refuse a volume with a remote file before creating .cpd/.cpx ("volume N is tiered to remote storage, cannot compact"). - CommitCompact refuses under the data-file locks when the volume became tiered during the copy, and removes .cpd/.cpx. - Store.CheckCompactVolume reports no garbage for a tiered volume, so the master's vacuum and volume.vacuum do not select it. - The worker's vacuum detection skips a volume with a remote copy, as balance and EC detection already do. Tests (each fails on master): TestCompactRefusesTieredVolume, TestCommitCompactRefusesVolumeTieredDuringCopy (on master the commit succeeds and a surviving needle no longer reads), TestPickPrimaryReplicaSkipsTieredVolumes. * volume: skip tiered replicas in vacuum selection, keep their real ratio Reporting a zero garbage ratio for a tiered volume hid it from volume.tier.compact, whose preflight reads that ratio while the volume is still remote, and did not exclude it from a vacuum with a zero threshold. Report the real ratio again, as the Rust volume server does, and have the master's vacuum check skip a replica whose volume info names remote storage, whatever the threshold or request. When CommitCompact refuses a volume tiered during the copy, also close the temporary needle map and remove .cpldb: nothing else would, since no later vacuum selects the volume. The tiered-compaction test no longer depends on random needle sizes reaching a 0.3 garbage ratio. * volume tier: do not attach a copy of a .dat a compaction replaced The upload copies the .dat, then records the remote file in the .vif and switches the backend. A compaction commit in between swaps in a compacted .dat and index; the upload then served the old-layout remote object through the new index, so live needles stopped reading. CommitCompact's tiered check cannot see this: the volume is not tiered until the switch. AttachRemoteDatFile records the remote file and switches the backend under the data-file locks, and refuses when the backend is no longer the file that was copied. The upload then removes the unused remote copy. * volume tier: keep a committed .vif entry when only the dir fsync fails AttachRemoteDatFile rolled back the in-memory remote entry on any SaveVolumeInfo error. A NotCrashDurableError comes after the rename, so the .vif on disk already names the copy; with the entry gone, the upload handler deleted that object, and a reload would serve from a missing remote file. Keep the entry and switch to the copy as the .vif says, and still return the error so the caller keeps the local .dat. * vacuum worker: filter tiered replicas instead of skipping the volume A partially tiered volume is the leftover of an interrupted volume.tier.upload; skipping the whole volume left garbage on its local replicas unreclaimed. Per-replica filtering matches the master's batchVacuumVolumeCheck and is safe: a tiered replica refuses both compaction and commit. * filer: sync leveldb entry writes when the path rule asks for fsync The fsync storage rule only reached the volume upload; the metadata writes behind an acknowledged create, update or delete still went to the leveldb journal with Sync:false, so a power loss could roll back an acknowledged bucket or object entry. A write-side flag on the entry context carries the request: the filer sets it from the path's storage rule in CreateEntry, UpdateEntry, EnsureDirectoryEntry and DeleteEntryMetaAndData, and the HTTP write path sets it for an explicit ?fsync=true. The leveldb stores write with Sync:true when the flag is present. * volume: flush the .idx before acknowledging a durable write A write with fsync synced only the .dat: the index row written on publish stayed in the page cache, so a power loss could leave the needle's bytes durable with nothing indexing them. The durable paths now run one .idx sync after publish, before the requests complete — one per syncWrite and one per batch run, paired with the .dat sync, the same split the Rust volume server makes between flush_dat and flush_idx. A failed .idx sync cannot be rolled back, so it stops the volume taking writes and fails the requests instead of acknowledging an index entry that may not survive. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
|
7603528aff |
s3api: support s3:RequestObjectTag/<key> in bucket policies (#11910)
* s3api: support s3:RequestObjectTag/<key> in bucket policies The tags come from X-Amz-Tagging only where the handler stores them from there: not on PutObjectTagging, which reads the body, and not on a copy that keeps the source's tags. For a repeated key the last value counts, since that is what CopyObject stores. * s3api: extract RequestObjectTag only where the handler stores it The header was honored on any request carrying it, so a tag attached to a GET, DELETE, part upload, multipart completion, directory marker or a PUT subresource satisfied a tag-conditioned policy the object never receives. Extraction now follows the handlers: PutObject, a copy under X-Amz-Tagging-Directive: REPLACE, and CreateMultipartUpload store the header; a plain object PUT is identified by carrying only X-Amz-* signing parameters. * s3api: exclude multipart part PUTs before the copy-source check A PUT carrying X-Amz-Copy-Source plus a numeric partNumber and uploadId routes to CopyObjectPartHandler, which stores a part and never reads X-Amz-Tagging. Checking the copy source first let such a request satisfy an s3:RequestObjectTag condition under REPLACE with tags that are never stored. Mirror the router: the part handlers match only when both partNumber (digits only) and uploadId are present, so a lone or non-numeric partNumber still falls through to the storing handlers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: gate tag extraction on the routed subresource, not query key shape Rejecting every PUT whose query held a key outside X-Amz- dropped X-Amz-Tagging from ordinary object PUTs that carry arbitrary parameters — Signature V2 presigns (AWSAccessKeyId/Signature/Expires) among them — so a tag-conditioned Deny could be bypassed by appending an unrelated query parameter while putToFiler still stored the tags. Exclude the object PUT subresources the router actually dispatches (tagging, renameObject, acl, retention, legal-hold) ahead of the copy-source check instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: cover the routed-subresource and part-copy tag extraction cases Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
cec96c02ea |
s3: reject requests carrying more than one routed subresource (#11942)
* s3: reject requests carrying more than one routed subresource Routing picks the first matching subresource route while authorization resolves a single action from the query, and the two can disagree. A bucket-level PUT ?tagging&policy is authorized as s3:PutBucketTagging but served by PutBucketPolicyHandler, letting a tagger rewrite the bucket policy; the same shape lets a list grant read the policy via ?list-type=2&policy or ?versions&policy, and lets an object ACL writer retag via PUT ?acl&tagging. Declining ambiguous requests keeps the authorized action and the served handler identical. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: reuse hasAmbiguousSubresource in the routed-subresource guard The standalone key list dropped versioning, so ?versioning&policy still reached route matching and auth with two operations. The bucket-level check already keys off operationSubresources plus bucketQueryActions; running the same predicate keeps the lists from drifting apart. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Junker der Provinzandzouyixiong
|
ee2718b14c |
filer: reduce forwarded lock log noise (#11933)
Forwarding a lock to its owner filer is routine in a multi-filer setup, so the forward and acquire messages move from V(0) to V(3). The acquire message is only logged when the forwarded response holds a renew token and no application error. Transport failures still log at V(0). Co-authored-by: zouyixiong <3390062+zouyixiong@users.noreply.github.com> |
||
|
Chris LuandDevin
|
7ff77a4f3b |
s3api: skip error response body when http forbids it (#11931)
WriteResponse wrote the payload even for HEAD requests and statuses that disallow a body (1xx, 204, 304), which made net/http return ErrBodyNotAllowed and logged 'write err: http: request method or response status code does not allow body' for ordinary requests. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
0b197867a1 |
filer leveldb2: tunable cache, buffer, block and table sizes (#11927)
* filer leveldb2: read cache and buffer sizes from filer.toml The leveldb2 store hardcoded its block cache (32MiB) and write buffer (16MiB) and offered no way to tune the table block size or compaction table size. Optional block_cache_size_mb, write_buffer_size_mb, block_size_kb and compaction_table_size_mb keys under [leveldb2] now override them; zero or negative values keep the previous defaults. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer leveldb2: document tuning options in scaffold filer.toml Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer leveldb2: saturate size options instead of overflowing int The option fields take int, so a MiB config value past math.MaxInt/MiB (e.g. 2048 on 32-bit) wrapped negative and goleveldb would panic sizing the memtable. Clamp the product instead of the input. * filer leveldb2: reject size options too large for int bytes Clamping a MiB setting at math.MaxInt silently hands goleveldb a different size than configured; fail Initialize on values that cannot fit instead. * filer leveldb3: same tunable cache, buffer, block and table sizes Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer leveldb2/3: exercise oversized-option rejection on 64-bit too The tests skipped whenever int was 64-bit, but write_buffer_size_mb values above MaxInt/1MiB still overflow there, so the rejection path had no coverage on the primary platform. Use MaxInt/1MiB + 1 so the rejection triggers on every platform. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
1451096c17 |
volume.fsck: delete missing filer entries through a worker pool (#11924)
* volume.fsck: delete missing filer entries through a worker pool -reallyDeleteFilerEntries issued one synchronous HTTP DELETE per orphan path inside the BFS consumer, so a million-entry purge ran one request at a time. Queue the confirmed paths onto a bounded pool (default 16, -tunable via -purgeConcurrency) so deletes run in parallel while the traversal continues. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.fsck: serialize command output across purge workers * volume.fsck: serialize purges that share a hard link id * volume.fsck: cover the purge worker pool * volume.fsck: take the purge shard remainder in unsigned arithmetic Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Junker der Provinz
|
4854c6a0f4 |
s3: skip the local filer socket on Windows (#11921)
The filer only listens on its unix socket when GOOS is not windows, so with a filer socket path configured on Windows the S3 server dialed a socket that never exists. Fixes #7278 |
||
|
|
74da88fc29 |
s3, iceberg: record the access key a request was signed with (#11901)
* s3, iceberg: record the access key a request was signed with Several access keys can belong to one identity, but nothing downstream of authentication says which one signed a request: the audit log carries the identity name only, and every table credential the Iceberg catalog vends for that identity gets the same session name. When an identity is shared (one key per person or per service), the log cannot tell the holders apart. The verifier that matched a credential now records that credential's access key id in the request's identity holder (verifyV4Signature for header, presigned, streaming and STS requests; both SigV2 verifiers). Nothing re-reads the request for a key, so a second, unverified key elsewhere in the request (a junk X-Amz-Credential on a SigV2 request, a junk SigV4 header on a SigV2 presigned URL) is never recorded. - s3err.AccessLog gains access_key_id (omitempty): a static key's id, or an STS session's temporary key id. Every other field is unchanged. - The Iceberg and Lance catalogs install the holder before authenticating a SigV4 request; a Bearer token records the key it was minted from (the token is verified with a key derived from it); a Lance x-api-key records its key. - VendTableCredentials ends the session name with "-k" and the first 12 hex digits of SHA-256 of the key id (the table part gives way within the 64-character limit; the tag is never cut), and carries the key id in the session token (src_ak, SessionInfo.SourceAccessKeyId). Tests: auth_access_key_attribution_test.go and auth_access_key_shapes_test.go (two keys on one identity, the deny path, SigV2 and presigned SigV2 with a junk key elsewhere, streaming seed, STS session, failed signatures record nothing), s3api_iceberg_session_name_test.go, iceberg and lance Auth tests, sts source_access_key_test.go. * s3: record the access key of a verified POST-policy form and of a retargeted signature Two verified signatures left access_key_id empty in the audit log: - A browser POST upload is verified in PostPolicyBucketHandler from its form fields by doesPolicySignatureV2Match / doesPolicySignatureV4Match, which did not record the key. Both now take the request context and record the credential's key once its signature verifies; a form that fails verification records nothing. - verifyV4Signature refuses a signature that verified for one virtual-hosted bucket but was routed to another. The key was recorded only after that check, so the refusal carried none. It is now recorded as soon as a host candidate verifies. Test: TestPostPolicyUploadRecordsTheSigningKey (SigV4 and SigV2 forms, and a form with a wrong signature) and TestSignedVirtualHostRetargetRecordsTheSigningKey; both fail before this change. * s3: test VendTableCredentials end to end for the caller's access key TestVendTableCredentialsRecordsTheCallersKey mints sessions through VendTableCredentials with a configured STS service, the key recorded the way the catalog's authentication records it, then validates each session token: SourceAccessKeyId and the session name's key tag match the key, and a caller without a key gets an untagged session. It fails if the vendor stops passing either field. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Jaehoon KimandChris Lu
|
744ab586c5 |
fix: volume.fsck purges live chunks of files renamed during its filer walk (SQL stores) (#11883)
* shell: volume.fsck reads the filer from one store snapshot when it can volume.fsck collects the filer's chunk references by walking directory by directory, so each directory is seen at a different time. A file renamed from a directory not yet listed into one already listed is never seen, its chunks look like orphans, and -reallyDeleteFromVolume deletes live data. Checkpoint rotation and write-to-tmp-then-rename hit this. Add an optional filer store capability, EntryScanner, that visits every entry below a directory from one consistent snapshot. The SQL stores implement it with one REPEATABLE READ read-only transaction read in primary-key pages, so a concurrent rename is seen under exactly one of its names. Key-value rows are skipped by their dirhash, a bucket directory whose table is missing fails the scan (from the snapshot it cannot be told from one dropped a moment ago), and hard links are resolved in the same transaction. SQLite offers it only in WAL mode, on its own connection. TraverseBfsMetadata serves the snapshot when asked via request metadata sw-traverse-mode=snapshot and reports the mode in a response header; without the metadata it walks as before, so peer bootstrap and admin export are unchanged. Snapshot responses carry no entry_ts_ns, whose per-entry lookup would cost more than the scan. volume.fsck requests the snapshot and falls back to the walk on an older filer or a store that cannot scan. * filer, shell: fail a scan that cannot start, not fall back to the walk A transient BeginTx failure and a preliminary GetEntry error were treated like a missing capability or missing root, so fsck would take the directory walk - the very path that misses renamed-away entries. Only a genuinely unsupported scan or a confirmed-absent root may fall back. * filer: only snapshot-listed bucket names keep a missing table fatal A cached-only name, whose bucket may have been deleted between copying the cache and opening the snapshot, is skipped when its table is gone. * filer: roll back to a savepoint before skipping a cached-only table On PostgreSQL a failed statement aborts the snapshot transaction, so ignoring a missing-table error without the savepoint still left every later cached-only table unreadable. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
364e19a493 |
remote_storage/s3: use the AWS default credential chain when a source is configured (#11881)
* remote_storage/s3: use the AWS default credential chain when a source is configured Type s3 forced anonymous credentials whenever no access/secret key was set, so IRSA, container credentials, profiles and env keys were never consulted. Keep anonymous (public bucket) access only when the environment names no AWS credential source. * remote_storage/s3: also recognize shared-config and default-profile env sources * remote_storage/s3: resolve env credentials on their own transport A guarded HTTP client is for the S3 endpoint; the remote credential providers it would also serve reach link-local addresses that guard must not dial. Profiles load shared config, and a test resolves env keys. * remote_storage/s3: clear every detected credential source in the test * remote_storage/s3: resolve the full default chain on the guarded client The eager credential session used NewSession, whose chain covers only env, shared-credentials and ECS/EC2 providers. Web identity and assume-role profiles resolve through shared config, so IRSA never worked when a custom HTTP client was pinned. Enable shared config and carry the configured region so STS can sign assume-role requests. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
df33e81915 |
volume: report the needle read error with a failed read (#11934)
* volume: report the needle read error with a failed read A read that fails on the volume server, for example on a CRC mismatch, returned a bare 500, so mount and filer logs only showed the status. The volume server now writes the error into the response body, and the read helpers add it to the error they return. Co-authored-by: Bruce <82158894+half-life666@users.noreply.github.com> * volume: keep the 500 bare on HEAD requests writeJsonError's body write fails with ErrBodyNotAllowed under net/http when the request method is HEAD, adding an error-writing log line on top of the failed read. Send only the status, matching the no-error path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * util/http: decode gzip-encoded error bodies in statusError Full-chunk reads set Accept-Encoding: gzip themselves, so the transport does not transparently decompress; a gzip-encoded error body reached statusError as compressed bytes and produced an unreadable message. statusError now takes the response and decodes Content-Encoding before extracting the bounded message. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: log the read error when the 500 stays bare On HEAD requests the body write is skipped, which also skipped the V(0) response log inside writeJson — a failed HEAD read left no cause in normal volume-server logs. Log the error in the same format before the bare 500. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * util/http: check the status before decoding gzip in GetAuthenticated The gzip reader eagerly consumes the header when it is created, so statusError saw a drained or corrupt body on a gzipped error response and the server message was lost. Match the other read helpers and check the status first. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * util/http: export StatusError Other volume read paths outside this file need the same error body. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: redact jwt query param when logging failed-read URLs The new read-failure logs print the request URL, and volume reads accept ?jwt= as a credential, so a failed authenticated read would write the bearer token to the volume server log. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * util/http: surface the error body on the remaining chunk-read paths * volume: redact duplicate jwt params in failed-read logs * operation: pass the lookup jwt on the chunk needle read --------- Co-authored-by: Bruce <82158894+half-life666@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
867a537802 |
volume.configure: keep the superblock in sync with the .vif (#11930)
* volume.configure: keep the superblock in sync with the .vif ConfigureVolume only rewrote the .vif's replication field, leaving the .dat superblock stale. A mounted volume also kept reporting the old placement in heartbeats, and its next compaction would rewrite the stale placement into a fresh .dat. Parse the placement once, then update the live volume when mounted: the in-memory .vif copy and superblock, and the .dat header through the volume's own backend under the data-file locks. For unmounted volumes rewrite the header directly; remote-only volumes skip it since the .vif carries the placement. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.configure: update the .dat superblock in the Rust volume server Mirror the Go ConfigureVolume change: after rewriting the .vif, rewrite the replica placement in the .dat header too, so a remounted volume's superblock, heartbeats, and later compactions agree with the .vif. A missing .dat (remote-only volume) is skipped since the .vif carries the placement. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
85c1b990b1 |
shell: bound the initial master connection wait behind -connectTimeout (#11922)
* shell: bound the initial master connection wait behind -connectTimeout weed shell waited on WaitUntilConnected with a background context, so an unreachable master hung the command forever. The wait now runs on a 30s (configurable) timeout and reports the failure instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: stop the master reconnect loop when RunShell returns KeepConnectedToMaster ran on context.Background(), so once RunShell returned — including the new connect-timeout early return — the reconnect goroutine kept dialing forever, and repeated RunShell calls accumulated one each. Give it its own cancellable context, separate from connectCtx whose deadline must not cut reconnects under a live shell. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: bound filer discovery by -connectTimeout too The discovery WithClient and ListClusterNodes ran on the unbounded background context, so a master disconnect after the initial connect — or a persistently failing ListClusterNodes — could block startup past the deadline. Reusing connectCtx bounds the whole startup phase. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: report filer discovery failures instead of silently losing the filer target * shell: give filer discovery its own budget after master connects Discovery shared connectCtx, so a connection that landed near the -connectTimeout deadline left ListClusterNodes almost no time and no room for WithClient to retry. A fresh timeout keeps discovery bounded without starving it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: cover the -connectTimeout master wait with a test Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: note filer discovery in the -connectTimeout help text Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
ac319fe318 |
volume: never compact a volume tiered to remote storage (#11895)
* volume: never compact a volume tiered to remote storage A tiered volume's data backend is the remote object, but compaction reads the local .dat path. When tiering removed the local .dat, every vacuum of a tiered volume with garbage fails with "no such file or directory", and the master's sweep and the worker's vacuum task keep selecting it. When the local .dat was kept, the compaction succeeds against the stale copy and the commit reloads the volume to read the remote object (old layout) through the compacted index, so live needles stop reading. Mirror the Rust volume server (#11545): - CompactByIndex and CompactByVolumeData refuse a volume with a remote file before creating .cpd/.cpx ("volume N is tiered to remote storage, cannot compact"). - CommitCompact refuses under the data-file locks when the volume became tiered during the copy, and removes .cpd/.cpx. - Store.CheckCompactVolume reports no garbage for a tiered volume, so the master's vacuum and volume.vacuum do not select it. - The worker's vacuum detection skips a volume with a remote copy, as balance and EC detection already do. Tests (each fails on master): TestCompactRefusesTieredVolume, TestCommitCompactRefusesVolumeTieredDuringCopy (on master the commit succeeds and a surviving needle no longer reads), TestPickPrimaryReplicaSkipsTieredVolumes. * volume: skip tiered replicas in vacuum selection, keep their real ratio Reporting a zero garbage ratio for a tiered volume hid it from volume.tier.compact, whose preflight reads that ratio while the volume is still remote, and did not exclude it from a vacuum with a zero threshold. Report the real ratio again, as the Rust volume server does, and have the master's vacuum check skip a replica whose volume info names remote storage, whatever the threshold or request. When CommitCompact refuses a volume tiered during the copy, also close the temporary needle map and remove .cpldb: nothing else would, since no later vacuum selects the volume. The tiered-compaction test no longer depends on random needle sizes reaching a 0.3 garbage ratio. * volume tier: do not attach a copy of a .dat a compaction replaced The upload copies the .dat, then records the remote file in the .vif and switches the backend. A compaction commit in between swaps in a compacted .dat and index; the upload then served the old-layout remote object through the new index, so live needles stopped reading. CommitCompact's tiered check cannot see this: the volume is not tiered until the switch. AttachRemoteDatFile records the remote file and switches the backend under the data-file locks, and refuses when the backend is no longer the file that was copied. The upload then removes the unused remote copy. * volume tier: keep a committed .vif entry when only the dir fsync fails AttachRemoteDatFile rolled back the in-memory remote entry on any SaveVolumeInfo error. A NotCrashDurableError comes after the rename, so the .vif on disk already names the copy; with the entry gone, the upload handler deleted that object, and a reload would serve from a missing remote file. Keep the entry and switch to the copy as the .vif says, and still return the error so the caller keeps the local .dat. * vacuum worker: filter tiered replicas instead of skipping the volume A partially tiered volume is the leftover of an interrupted volume.tier.upload; skipping the whole volume left garbage on its local replicas unreclaimed. Per-replica filtering matches the master's batchVacuumVolumeCheck and is safe: a tiered replica refuses both compaction and commit. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
164a7d8b56 |
upload: upload -dir files concurrently behind a new -c flag (#11918)
* upload: upload -dir files concurrently behind a new -c flag Each file under -dir was submitted serially inside filepath.Walk, so every small file paid a sequential master-assign plus upload round trip. A bounded pool now runs those submissions concurrently while keeping results in walk order; failures are reported per file and the command exits non-zero when any upload failed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * submit: close opened readers when a batch assign fails SubmitFiles returns early when the assign RPC fails, before FilePart.Upload gets a chance to close each opened file. Close the readers on that path so repeated assign failures do not leak handles. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * upload: stream -dir results as each file completes Batching the output until every upload finished meant an interrupted run lost the file ids of completed uploads. A per-index ready channel lets a printer emit each result in walk order as soon as it lands. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * submit: close the opened file when Stat fails newFilePart returned the still-open handle with the error, and the concurrent upload path records the error without ever reaching SubmitFiles, leaking one descriptor per failing file. * upload: still upload files collected before a walk error A walk that aborts partway used to leave the already-uploaded files behind; collecting paths first must not drop them, so upload what was gathered and still exit non-zero. * upload: submit batched files concurrently SubmitFiles fetched all file ids in one assign RPC but then uploaded the files one after another, so a multi-file list still spent its time in serial per-file round trips even with -dir concurrency. Uploads now run through a small worker pool while results keep their input order. Verified end to end: a 5-file list uploads with consecutive file keys and each stored needle matches its source bytes. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * upload: print per-file results before exiting on a partial batch failure --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
chuanying chenandChris Lu
|
d35f4c1c6b |
Missing data read in super block extra parsing (#8392)
* storage: fail on short super block extra read even without an error Some backends (e.g. s3_backend) clear io.EOF after a short read, returning n < len(p) with a nil error. Checking n only when the error is non-nil lets truncated extra data reach proto.Unmarshal with a zero-filled tail. * storage: apply the same short-read check to the super block header A backend that returns n < SuperBlockSize with a nil error previously slipped through and parsed a partially zero-filled header. * storage: test super block extra reads Covers a round trip through ReadSuperBlock, a truncated extra region, a backend returning io.EOF alongside a full read, and a backend returning a short read with a nil error. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
1aad4cfe70 |
iam: optionally require an iam: grant for self-service access-key actions (#11937)
* iam: optionally require an iam: grant for self-service access-key actions
AuthorizeIamAction lets every authenticated identity run CreateAccessKey,
DeleteAccessKey, UpdateAccessKey, ListAccessKeys and GetUser on itself with
no grant. On AWS these actions need an IAM policy (the usual one grants them
on arn:aws:iam::*:user/${aws:username}); nothing is allowed implicitly.
Where identities hold data-plane rights only, the implicit carve-out lets a
leaked key mint further keys for its own identity, with no cap, that keep
working after the operator deletes the key that leaked; it can also
deactivate or delete the keys the operator issued.
A new option, off by default, makes these actions need the iam:<Action> grant
like every other IAM action: -s3.iam.selfServiceRequiresGrant on weed
server, filer and mini, -iam.selfServiceRequiresGrant on weed s3. With it
set, the grant is authorized against the target user's ARN, the caller's own
when UserName is empty, so a policy can confine it to the caller's own keys
(Resource arn:aws:iam:::user/${aws:username}), the same resource scoping
CreateServiceAccount already uses. The admin and iam:* grants are unchanged;
with the option off nothing changes.
Tests (s3api_embedded_iam_authz_test.go): with the option set, an identity
holding only s3:* and a bare identity are refused all five actions on an
empty, own and other target (fails without the change); a grant on the
caller's own user ARN, literal or ${aws:username}, allows them on itself and
not on another user; the admin and an iam:* grant keep every action; with the
option off self-service is allowed as before.
* iam: wire selfServiceRequiresGrant on the standalone iam server
Every other command that runs the embedded IAM API accepts the flag; the
deprecated weed iam could not enable the enforcement at all.
* iam: wire selfServiceRequiresGrant on weed server and filer -iam
The IamOptions used by 'weed server -iam' and 'weed filer -iam' never
received the flag, so startIamServer dereferenced a nil pointer at
startup. Share the -s3.iam.selfServiceRequiresGrant pointer with them.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
|
||
|
Chris LuandDevin
|
ea859a0844 |
sftp: reject empty passwords in PasswordAuthenticator (#11938)
* sftp: reject empty passwords in PasswordAuthenticator FileStore.ValidatePassword already refuses a zero-length stored or supplied password, but the guard belongs at the authenticator too: an empty password is never valid, and checking before the store call keeps any user.Store implementation without its own empty-secret check from matching on zero-length input. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * sftp: keep the brute-force delay on empty-password failures The early rejection returned before the shared failure path, so an empty password failed instantly while other failures slept 100-199 ms. Folding the length check into the store call keeps the store untouched and every failed attempt equally slow. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
c0152e6be1 |
s3: keep '\' a literal object-key character behind -s3.allowBackslashInKey (#11911)
* s3: keep '\' a literal object-key character when AllowBackslashInKey is set NormalizeObjectKey rewrites every S3 object key: '\' becomes '/' and duplicate slashes collapse. On AWS S3 both are ordinary key characters, so keys that are distinct there alias one object here — writing probe/back/slash.txt silently overwrites probe/back\slash.txt. The filer only treats '/' as a path separator, so a verbatim '\' stays inside one entry name. Gate the backslash rewrite and the matching '\' handling in IsValidObjectKey behind a process-wide mode the S3 server sets from the new S3ApiServerOption. Duplicate-slash collapsing stays: an entry name cannot contain an empty segment. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * command: add -s3.allowBackslashInKey flag for AWS-compatible key handling Registers the flag on all four S3 entry points (s3, server -s3, filer -s3, mini) and passes it through S3Options into S3ApiServerOption. Default off, so existing normalization is unchanged. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: correct IsValidObjectKey/NormalizeObjectKey docs under allowBackslashInKey Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: normalize DeleteObjects body keys the same as path keys DeleteObjects keys arrive in the XML body, so they bypassed the normalization applied to route-var keys. The lock, the authorization checks, the table-bucket layout check and the filer delete itself all addressed the raw key while the conditional check resolved the normalized one, so a key spelled with '\' or '//' could be reported deleted while the stored entry survived. Echo the client-sent key in the result while operating on the normalized form. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: report a normalized-empty DeleteObjects key as a delete no-op A multi-delete key of only slashes normalizes to the empty string. Left to reach the filer delete, the empty name resolves to the bucket entry itself, so an unauthorized caller could remove a bucket through the body-key path. Such a key can never name a stored entry, so like a missing key the delete is an idempotent no-op reported before any permission check or filer call. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: drop unused AllowBackslashInObjectKey getter * s3: answer conditional delete of a normalized-empty key with NoSuchKey * s3: normalize the probed key in authorizeAnonymousBucketList The masked-miss probe read the raw mux var, so a spelling like folder// or folder\ was authorized against a key that can never exist while the request itself targets the normalized folder/. Probe the normalized key so the answer matches what the read path authorizes. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: cover anonymous authz of a literal-backslash key under allowBackslashInKey A public grant on the slash key must not authorize an anonymous read of the distinct backslash object when the flag keeps '\' literal; grant the backslash key's own ACL and it reads back. Default mode still folds the spelling onto the covered slash key. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: publish the key mode only after the S3 server is constructed A failed NewS3ApiServerWithStore could previously flip the process-wide key mode out from under an already-running gateway. * s3: normalize the copy-source key before validation and authorization pathToBucketObjectAndVersion returned the raw header spelling, so an equivalent spelling (a//b, a\b) was authorized and compared against the normalized destination while resolveCopySourceEntry looked up the normalized key — a deny on the canonical key could be bypassed and self-copy detection missed. Normalize once at parse so the CopyObject and UploadPartCopy paths agree. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
e057244b0a |
storage: read the super block extra before unmarshaling it (#11932)
* storage: read the super block extra before unmarshaling it ReadSuperBlock allocated a buffer of ExtraSize bytes but never filled it, so any volume with a super block extra failed to load with an invalid wire-format error. Co-authored-by: chuanying chen <18412672+ccy00808@users.noreply.github.com> * storage: test super block extra read error paths --------- Co-authored-by: chuanying chen <18412672+ccy00808@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Javier de la TorreandJavier de la Torre
|
763e2a2090 |
iceberg: record geometry bounds on the data files compaction writes (#11898)
* iceberg: record geometry bounds on the data files compaction writes Compaction collected column statistics for every primitive column except geometry: the bound converter has no geometry case, so a compacted file's manifest entry carried value and null counts for a geometry column but no lower or upper bound. Readers that skip files by these bounds for a spatial filter (for example DuckDB for `geom && <box>`) had to open every compacted file, where the files before compaction could be skipped. The bound follows the spec's single-value serialization for geometry: the lower bound is the point (xmin, ymin), the upper bound (xmax, ymax), each an 8-byte little-endian double; z and m are not written. It is the union of the written file's row-group geospatial statistics, which parquet-go computes from every non-null value. A bound that is too small makes a reader skip rows, so any doubt gives none: a row group with non-null values but no geospatial statistics (parquet-go omits them when a value does not parse as WKB), a box that is not finite, an inverted box, or a chunk whose statistics cannot be read. A row group of nulls only constrains nothing. Geography gets no bound, as iceberg-go writes none for it (geodesic edges, boxes across the antimeridian). Tests: points and polygons far apart through a bin-packed compaction (the bound equals the union of every geometry), a sorted compaction (any bound covers every geometry), a column of nulls only, an unparseable value in a second row group (no bound; without the rule the file got the bound of the first row group only), geography, and the aggregation rules. * iceberg: do not order compaction bins by a geometry column When a table declares no usable sort order, compaction orders a bin's files by the first column every file has bounds for. A v3 table whose first column is a geometry qualifies as soon as its files carry geometry bounds (DuckDB writes them, and compacted files now do too). Those bounds decode into geo literals that compareLiterals cannot compare, so the order sorted nothing, yet an oversized bin was still split into "ordered" runs by splitOrderedBin. Geometry and geography columns are now skipped in both the declared sort order and the inferred one, so such a table is ordered by its next column with bounds. Test: TestResolveCompactionOrderSkipsGeometry, a geometry column first and an id column second; on master the order is the geometry column. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
|
917b6f4afd |
iceberg: authorize a commit before staging its metadata file (#11891)
* iceberg: authorize a commit before staging its metadata file Both commit paths (CommitTable and CommitTransaction) write the new vN.metadata.json into the table's metadata/ directory and only then reach UpdateTable, whose authorization refuses a caller that may read the table but not commit to it. The refused commit then deletes the staged file on the way out, so such a caller could make the server write and delete a file under the table's prefix on every attempt. The same shape was closed for creates in #10991. Both paths now ask Manager.AuthorizeUpdateTable right after loading the table and answer 403 ForbiddenException before anything is staged. UpdateTable still decides under the policies in force when the pointer flips, so a policy changed mid-commit is caught as before. Test: TestCommitDeniedBeforeStagingMetadata records every file the handlers create; on master both the single-table and the transaction commit by a GetTable-only identity write v2.metadata.json before answering 403, with the change nothing is written. TestCommitAllowedStagesMetadata is the positive control (the owner, and an identity whose own actions include UpdateTable). * iceberg: authorize a view commit before staging its metadata handleUpdateView has the same write-before-authorize gap the table commit paths had: it stages the new metadata file and only reaches the UpdateView authorization at the pointer flip, so a caller who may only load the view could still make the server write and delete files under it. Gate it on AuthorizeUpdateView, mirroring authorizeCommit. * iceberg: seed committed view metadata in the denied-commit test Without stored view metadata the request fails on the missing-metadata check before it could stage a file, so the zero-write assertion could not catch a write-before-denial regression. Seed the same metadata the allowed-path test uses so the denied request would otherwise stage. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
106fd31af1 |
s3: resolve versionId ACL and attributes requests to the version-scoped actions (#11916)
* s3: resolve versionId ACL requests to the version ACL actions A request for an object's ACL carrying versionId acted on the historical version but authorized against the current-object action (s3:GetObjectAcl/s3:PutObjectAcl), because the acl block in resolveFromQueryParameters returned before versionId was considered - the same gap closed for tagging. An identity granted only current-object ACL access - or explicitly denied s3:*ObjectVersionAcl - could still read and modify ACLs on noncurrent versions. ?acl&versionId=<id> on an object now resolves to s3:GetObjectVersionAcl/s3:PutObjectVersionAcl; ?acl alone keeps the current-object actions and bucket ACL is unaffected. s3:PutObjectVersionAcl is mapped onto the WriteAcp identity action so policies can name it. * s3: resolve versionId attributes requests to the version attributes action GET /bucket/key?attributes&versionId=<id> fetched the historical version's attributes but authorized against s3:GetObjectAttributes, so a grant limited to the current-object attributes action could reach noncurrent versions. The request now resolves to s3:GetObjectVersionAttributes, and both attributes actions map onto the Read identity action so policies can name them (s3:GetObjectAttributes was previously rejected as an unmapped action). * s3api: recheck the version action on versioned ACL/attributes requests The phase-2 policy recheck inside GetObjectAclHandler, PutObjectAclHandler, and GetObjectAttributesHandler passed the current-object action even when the request carried versionId and phase 1 had resolved the version action. A Deny scoped to s3:*VersionAcl or s3:GetObjectVersionAttributes could never fire in the recheck, and an Allow limited to the version action went unevaluated. Select the action the same way the resolver does — versioned whenever versionId is present. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: recheck the version action on versioned tagging requests The same phase-2 gap exists in the tagging handlers left by the earlier version-tagging resolution change: versioned GetObjectTagging / PutObjectTagging / DeleteObjectTagging rechecks evaluated the current-object tagging action, so a Deny on s3:*VersionTagging could not fire. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: resolve HEAD ?acl as the head-object action, not an ACL action No HEAD acl route exists, so HEAD /bucket/key?acl lands on HeadObjectHandler — which serves object headers, not the ACL. Resolving it to an ACL action denied readers who hold only GetObject(Version) and let ACL grants reach a head-object read. Dropping HEAD from the acl branch lets the request fall through to GetObject/GetObjectVersion, matching how the tagging branch already treats HEAD. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: require the version read grant for versioned attributes AWS asks for s3:GetObjectVersion alongside s3:GetObjectVersionAttributes when a GetObjectAttributes request names a versionId; the resolved version-scoped action on its own left a lone s3:GetObjectVersion Deny unable to cover the attributes read. Check the version read grant too: the identity side through VerifyActionPermission, the bucket-policy side through the same recheck that evaluates the scoped action. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: check the version read grant on a synthesized versioned GET The versioned attributes check passed the literal s3:GetObjectVersion action, but both policy paths re-resolve the action from the request, so the attributes query parameter mapped it back to s3:GetObjectVersionAttributes and a lone version-read Deny never fired. The literal action also reached the legacy CanDo path, where a plain Read grant no longer matched. Evaluate the grant on a request shaped like a plain versioned GET instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: reject a named versionId on buckets without versioning Tagging, ACL, and attributes handlers answered versionId requests on never-versioned buckets as the current object while evaluating the version-scoped action, so a grant limited to the version action could authorize changes to the live entry. Reject the request up front through the existing checkVersionIdConfigured gate. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: carry presign params onto synthesized policy requests objectKeyPolicyRequest rebuilt the query with only versionId and the STS token, so a presigned request's s3:authType condition value silently changed from REST-QUERY-STRING to unset during phase-2 rechecks (version read grant, copy source, delete keys, post-policy writes). A policy conditioned on s3:authType would then deny a request it already allowed. Carry the SigV2/SigV4 presign keys through; none of them steer ResolveS3Action. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
137f23a229 |
ec distribution: implement the documented shard distribution API (#11915)
* ec distribution: parse EC and replication configs into a shard plan NewECConfig/DefaultECConfig and NewReplicationConfig(FromString) turn an EC ratio and a replica placement into ECConfig/ReplicationConfig values, and CalculateDistribution derives the target and maximum shard counts per data center, rack, and node. ECDistribution.Summary and FaultToleranceAnalysis describe, per topology level, whether losing one failure domain leaves enough shards to reconstruct the volume, so the EC durability guarantees of a given setup can be answered directly instead of only documented informally. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: analyze topology and plan shard moves TopologyAnalysis records nodes and shard locations per DC/rack/node, and Rebalancer.PlanRebalance emits the moves that level the layout: data centers first, then racks within each DC, then nodes within each rack, moving parity shards before data shards and honoring per-node free slots. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: cover the plan math and rebalancing with tests Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: keep AddShardLocation and node ShardIDs in sync AddShardLocation updated the count maps but never the node's ShardIDs — yet every rebalancer count reads len(node.ShardIDs), so the documented AddNode-then-AddShardLocation flow produced an empty rebalance plan. It now appends to the node's shard list and skips shards the node already has, making the two registration styles equivalent and idempotent. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: key rack-scoped maps by dc/rack Rack names are only unique within a data center — two DCs can both hold a "rack1" — so ShardsByRack, RackToShards, and RackToNodes keyed by the bare rack name merged distinct domains and corrupted both the DC-level shard counts and rack-level balancing. Rack lookups now use the dc/rack pair; DCToRacks still lists bare rack names. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: never move a shard onto a node already holding it The move picker selected the destination node before the shard, so a node already holding the chosen shard could be targeted. Pick the shard first (parity-first) and choose among destination nodes that lack it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: tolerate zero-value replication configs A ReplicationConfig{} with zeroed minimums made maxShare divide by zero; clamp each level's unit count to at least one. Covers the new behaviors with tests. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: keep rack keys distinct when names contain "/" rackKey joined data center and rack with "/", so dc "a"/rack "b/c" and dc "a/b"/rack "c" shared one key and were counted as the same failure domain. Separate the parts with a NUL byte, which cannot appear in a name. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: plan rebalances on a copy of the topology recordMove mutated the caller's TopologyAnalysis while planning, so an interrupted or partial execution left the input describing moves that never happened, and replanning on it returned nothing. PlanRebalance now simulates on a cloned analysis; plan.Analysis reports the layout the moves would produce and the caller's input is left untouched. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: balance nodes against the rack's current shard count The node-level target still read the original analysis, so a rack that had just sent or received shards during DC/rack balancing aimed at a stale total and left its nodes unbalanced. * ec distribution: register each node id once Adding the same node twice appended it to RackToNodes twice, so every count that walks rack nodes double-counted its shards. * ec distribution: normalize replication counts in the stored plan A zero-value ReplicationConfig computed targets against one domain per level while Summary still reported zero DCs, racks, and nodes. * ec distribution: cap a survivable level's max at the loss budget * ec distribution: skip shards already inside the destination domain * ec distribution: register nodes implied by shard locations * ec distribution: merge repeat node registrations A shard location now auto-registers an unknown node with no free slots; a later AddNode carrying the real capacity was then dropped entirely. Repeat registration now updates the node's capacity and adopts any new shards, while the first registration keeps the topology placement. * ec distribution: try other overloaded sources before stopping When the most overloaded domain could not shed any shard, balance returned early and skipped every other source that still had movable shards. A source that cannot move now gets set aside for the rest of the pass. * ec distribution: collapse chained shard moves to the final destination Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: schedule collapsed moves behind the departures that free slots Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Junker der ProvinzandDan Foster
|
e645449872 |
fix(storage): count preallocated space when auto-sizing max volumes (#11919)
The max volume count subtracted each volume's remaining room from the free disk space, except when the master had preallocation turned on, in which case it subtracted nothing. That is wrong for a disk that mixes preallocated and plain volumes, and on platforms where preallocation is not supported. Use the larger of the .dat file's size and its allocated blocks, in the Go and the Rust volume server, so a preallocated volume counts as full and a plain one keeps its remaining room reserved. Fixes #5778 Co-authored-by: Dan Foster <4570306+danfoster@users.noreply.github.com> |
||
|
Junker der Provinz
|
9b642d4636 | volume: use FALLOC_FL_KEEP_SIZE and log preallocation failures (#11920) | ||
|
Javier de la TorreandJavier de la Torre
|
a882565cd7 |
iceberg: keep a default-CRS geometry column without a CRS after compaction (#11899)
* iceberg: keep a default-CRS geometry column without a CRS after compaction Writers such as DuckDB write a GEOMETRY column whose Iceberg type has the default CRS (OGC:CRS84) with no CRS in the Parquet logical type. parquet-go spells the default out when it writes such a column (geometryType's LogicalType), even when the column it read declared none, so every compacted file declared GEOMETRY(crs=OGC:CRS84) where its inputs declared GEOMETRY. Both mean the same to the Parquet spec, but a reader may treat the file column as another type than the table column: DuckDB casts it to the table's GEOMETRY, and a cast column is no longer pruned by its row-group statistics, so a spatial filter read every row group of a compacted file. After a bin is merged, the footer's schema elements of every GEOMETRY column whose Iceberg type has the default CRS (top-level or nested) are rewritten to declare no CRS. The data, the page indexes and every offset stay byte for byte; only the footer is re-encoded. An explicit CRS is left as it is. Retyping the writer's column instead does not work: parquet-go refuses to convert between two GEOMETRY types that both mean the default, and its sorting writer converts. Tests: geometry_crs_test.go, a bin-packed and a sorted compaction of default-CRS inputs (one of them spelling the default out) write no CRS, with a control showing parquet-go spells it out on its own; an explicit CRS is kept; a nested geometry. On master the binpack and sort cases fail with CRS "OGC:CRS84". * iceberg: compress the row lineage columns of compacted v3 files A v3 compaction appends _row_id and _last_updated_sequence_number to the merged file. The two leaves were built without a codec, so they were written uncompressed, 16 bytes a row, whatever codec the data columns use; on a table of points the compacted files grew by about a third. The lineage leaves now take the codec of the reference file's first leaf that declares one, ZSTD when none does. Test: TestCompactCompressesLineageColumns; on master both columns are written UNCOMPRESSED next to ZSTD data columns. * iceberg: match list and map geometry by Iceberg name in files without field ids A Parquet footer path runs through the repeated group of a LIST or MAP (shapes.list.element, attrs.key_value.value), which Iceberg leaves out of its field names (shapes.element, attrs.value). The name lookup for a file without field ids therefore missed every geometry inside a list or a map and left the default CRS spelled out. Name the paths as Iceberg does, including the legacy two-level list forms. * iceberg: rewrite the geometry CRS footer inside the merged file's buffer Removing a CRS shortens the footer, so the new one fits where the old one was. Copying the whole merged file to replace a few hundred bytes of footer doubled the memory of a compaction near the target file size. * iceberg: take the lineage codec from the data columns, not an input's _row_id An input that already carries _row_id may list it before its data columns, and write it uncompressed; the new lineage columns then copied that codec instead of the data columns'. Choose from the columns the layout keeps. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
Javier de la TorreandJavier de la Torre
|
de2fc03c7d |
s3tables: accept the external-volume verification object at a table bucket's root (#11900)
* s3tables: accept the external-volume verification object at a table bucket's root Snowflake verifies an S3-compatible external volume before it creates or writes Iceberg tables in it (SYSTEM$VERIFY_EXTERNAL_VOLUME, also run implicitly): it writes, reads, lists and deletes one small object, <base>/verify_<epoch milliseconds>_<number>, a 21-byte body. A table bucket refuses every key outside the Iceberg layout, so the check failed and Snowflake could not use a table bucket as an external volume. PUT, GET, HEAD, DELETE and DeleteObjects now accept exactly that key in a table bucket, narrowly enough that it cannot be used to store data: - at the bucket root only, so nothing is encoded in a prefix, written into a hidden tree or placed on a namespace or table name; - one shape: verify_<13 digits of epoch ms within one hour of the server's clock>_<1-10 digits>; - a PUT must declare a Content-Length of at most 1 KiB (no chunked or undeclared body) and carry no user metadata, tagging, Content-Disposition, Cache-Control, Expires or website redirect. Multipart, copy, rename, ACL, tagging and POST policy writes keep the plain layout rule. Buckets that are not table buckets are unaffected. Tests: table_bucket_verify_object_test.go, the key accepted (with the Iceberg layout as the positive control and the plain rule still refusing it), 15 look-alikes refused (prefixed, hidden tree, wrong shape, outside the window, ordinary keys), and seven PUT refusals through PutObjectHandler. * s3tables: keep the verification key off namespaces and narrow its PUT A namespace may be named like a verification key (verify_<ms>_<n>), and the key then names the namespace's directory. A PUT replaced that directory with a prefix object, dropping the namespace's catalog metadata; a DELETE removed an empty namespace. The key now gets the plain layout rule when it names a directory or its entry cannot be read, and a PUT of it never falls back to a prefix object, so a namespace created after the check is not replaced either. The time window applies to PUT only: GET, HEAD and DELETE take a key of any date, so an object a check left behind for more than an hour can still be read and removed. A verification PUT also refuses an extraction request and the stored Content-Language, Response-Content-Disposition and Content-Encoding headers (aws-chunked framing alone is still accepted). The integration test runs the check's PUT, HEAD, GET, list and DELETE against a table bucket, and the PUT and DELETE over a namespace of the key's shape. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
Javier de la TorreandJavier de la Torre
|
21add33b6e |
s3 backend: optional HTTP client settings for the cloud tier (#11897)
* s3 backend: optional HTTP client settings for the cloud tier The S3 tier backend builds its client from a bare aws.Config, so it always uses Go's default transport: HTTP/2 wherever the endpoint offers it over TLS, no cap on connections per host, and a 90s idle timeout. Against some S3-compatible services that is a poor fit. Over HTTP/2 every upload and range read of a tiered volume shares one connection and its flow-control window; with a small window, tier uploads and concurrent reads of tiered volumes become throughput-bound on that one connection, while HTTP/1.1 opens a connection per concurrent request. Some services also limit parallel connections per client address, or drop idle connections before 90s. Add three optional settings to [storage.backend.s3.<id>], all off by default so existing deployments keep the SDK's default client: - disable_http2: keep the client on HTTP/1.1 (ForceAttemptHTTP2=false and an empty TLSNextProto, the documented opt-out); - max_connections: MaxConnsPerHost, 0 = no cap; - idle_connection_timeout_seconds: IdleConnTimeout, 0 = Go's default. The session cache key includes the options when any is set, so two backends on the same endpoint with different settings do not share a client. ToProperties reports the settings; the master.toml scaffold documents them. Tests: TestS3BackendHTTPOptionsReachTheClient fails without the change (the settings never reach the client); TestS3BackendDefaultHTTPClientUnchanged, TestS3BackendSessionsDifferByHTTPOptions, TestParseNonNegative. * s3 backend: reject an idle timeout too large for time.Duration An idle_connection_timeout_seconds above math.MaxInt64/1e9 overflowed when multiplied by time.Second. The negative result made the transport keep its default timeout and ToProperties export a negative value. Warn and use the default instead. * s3 backend: parse the idle timeout as 64-bit seconds parseIdleConnTimeout went through strconv.Atoi, so on 32-bit platforms any value above 2147483647 seconds was rejected while the same value was accepted on 64-bit, and the overflow test failed under GOARCH=386. Parse with ParseInt(…, 64) and export the effective value with FormatInt so the accepted range is the same on every platform. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
Chris LuandDevin
|
9a03c09396 |
s3api: cover the inline-content UploadPartCopy route at handler level (#11914)
s3api: exercise the inline-content route through CopyObjectPartHandler Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Javier de la TorreandJavier de la Torre
|
f40267db44 |
s3: answer InvalidBucketName to a denied request on a malformed bucket name (#11893)
* s3: answer InvalidBucketName to a denied request on a malformed bucket name The store authorizes a request before it looks at the bucket, so a caller denied on a bucket whose name breaks the S3 naming rules (upper case, an underscore, too short, two adjacent periods, a non-ASCII letter) gets 403 AccessDenied, where AWS S3 answers 400 InvalidBucketName. Clients and compatibility suites that check the error code read the 403 as a permission problem. Auth and AuthPostPolicy now turn an AccessDenied for a resolved identity into InvalidBucketName when the bucket name fails VerifyS3BucketName or uses a character outside a-z, 0-9, '.' and '-' (VerifyS3BucketName checks with unicode.IsLower/IsNumber, which accept letters such as 'ü'). The answer depends on the name alone, so it reveals nothing about which buckets exist. An authentication failure, an allowed request and a denial on a well-formed name answer as before. Test: TestAuthDeniedInvalidBucketName drives iam.Auth with signed and anonymous requests; on master every malformed name answers 403, with the change 400, while the identity's own bucket stays allowed and a well-formed foreign name stays 403. TestRefineDenialForBucketName covers the rule table. * s3: refine POST-policy and deferred anonymous-read denials on a malformed bucket name Two denial paths bypass the refinement in Auth and AuthPostPolicy: - A browser POST upload carries its credentials in the form, so AuthPostPolicy delegates it to PostPolicyBucketHandler, which verifies the form and writes AccessDenied itself when the signer may not write the bucket or the per-key authorization denies the object. Both denials now go through refineDenialForBucketName with the verified identity. The coarse Write check used to run inside doesPolicySignatureV2Match and doesPolicySignatureV4Match before the signature was compared, so a form with a wrong signature was told whether the key may write the bucket. It now runs in doesPolicySignatureMatch after the signature verifies, as on the header-signed path; a form that fails verification answers SignatureDoesNotMatch (or MissingFields) and is never refined. - Unsigned GET/HEAD object reads are deferred by AuthWithPublicRead to the object handlers, where writeAnonymousAwareNotFound masks a miss as AccessDenied. That denial is refined with the anonymous identity, so a missing malformed bucket answers InvalidBucketName when an anonymous identity is configured, as Auth does. A denial on an entry that was found is left alone: the bucket exists. Test: TestPostPolicyDeniedInvalidBucketName drives AuthPostPolicy with signed forms (Write denied, per-key policy deny, bad signature, well-formed name); TestAnonymousReadDeniedInvalidBucketName drives AuthWithPublicRead with GET and HEAD, with and without an anonymous identity. Both fail before this change. * s3: refine a denied anonymous read of an existing key on a malformed bucket name A deferred anonymous GET/HEAD refined only its masked miss to InvalidBucketName, so in a bucket whose name breaks the S3 rules (one VerifyS3BucketName still accepts, such as a non-ASCII name) a missing key answered 400 and an existing private key 403, telling the caller which keys exist. The entry check now refines its denial the same way, so both answer alike. Without an anonymous identity no identity resolves and both stay 403, as in Auth. The deny-precedence case in TestAnonymousObjectACLConfiguredIdentity runs on the fixture bucket "b", shorter than S3's three-character minimum, so its denial now answers InvalidBucketName. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
Chris LuandDevin
|
6ee7c4c5c7 |
s3: no empty-directory marker in a "<dir>/" self-prefix listing (#11906)
An out-of-band empty directory listed under its own trailing-slash prefix answered the probe with a fake entry of itself: a Contents key on 4.48 and, since recently, a CommonPrefix. That listing enumerates children, where S3 returns nothing for an empty prefix, and clients that read every returned key (Pulumi lock listing) choked on the marker. The prefix object and directory key object paths are unchanged: a real "dir/" object still lists as its own key. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
9c427b265f |
volume tier: accept deletes on a volume as soon as it is tiered (#11894)
* volume tier: accept deletes on a volume as soon as it is tiered
volume.tier.upload marks the volume read-only without deletes before the
upload (markVolumeReplicasWritable(..., false, false), i.e. noWriteOrDelete),
and nothing clears that until the volume server restarts, when load() gives
every tiered volume noWriteCanDelete. Every delete in between is refused with
"volume N is read only": the needle stays live in the index, and its bytes in
the remote object, until someone notices.
After the remote file is loaded, the upload handler now gives the volume the
flags a restart would (MarkTieredDeletable: writes refused, deletes accepted),
with two exceptions that keep the current behaviour:
- a replicated volume: the shell copies the uploaded replica's .idx and .vif
to the other replicas after the upload, and a tombstone written before that
copy would reach one replica only;
- a volume in I/O quarantine: that flag is not the upload's to lift.
Tests: TestTierUploadAcceptsDeletesOnASingleCopyVolume fails without the
change ("volume 1 is read only") and checks writes stay refused;
TestTierUploadKeepsRefusingDeletesOnAReplicatedVolume; TestMarkTieredDeletable
(quarantine kept).
* volume tier: check the I/O quarantine under noWriteLock
MarkTieredDeletable read the quarantine state before taking noWriteLock.
CollectHeartbeat could quarantine the volume and set noWriteOrDelete in
between, and the upload then cleared that refusal, so a quarantined volume
accepted deletes until the next heartbeat. Take noWriteLock first: the
heartbeat marks the quarantine before it takes noWriteLock, so either the
check sees it or the heartbeat's refusal lands afterwards and stays.
Also leave alone a volume past IoErrorTolerance that the next heartbeat
will quarantine, as the heartbeat itself does.
* volume: read noWriteOrDelete under noWriteLock in DeleteVolumeNeedle
DeleteVolumeNeedle checked v.noWriteOrDelete without the lock that
MarkVolumeReadonly, CollectHeartbeat and now MarkTieredDeletable hold when
they write it, so a delete racing a tier upload was a data race. Copy the
flag under the read lock; the delete path is otherwise unchanged.
* volume tier: mirror the delete acceptance rules in the Rust volume server
The Rust tier upload already promoted a freshly tiered volume to
no-write-can-delete via refresh_remote_write_mode, and did so for every
volume -- including the replicated and I/O-quarantined ones the Go
MarkTieredDeletable deliberately refuses. A live tier-up now goes
through mark_tiered_deletable (replicated volumes and quarantines keep
their upload-time mark until the next load), while load() keeps
promoting every tiered volume unconditionally, as Go does.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep a tiered volume delete-only when marked writable
MarkVolumeWritable preserved noWriteCanDelete on a remote-tiered volume
only when it was already set. A tiered volume still holding the
upload-time mark (a replicated volume, or one whose tier upload ran
before MarkTieredDeletable existed) had it clear, so marking writable
published the volume as fully writable while its .dat lives remotely --
writes then failed against a backend that cannot append, after the
master was told the volume is writable again.
load() gives every remote-tiered volume noWriteCanDelete; deriving the
same flag here keeps mark-writable consistent with a restart: "writable"
on a remote volume resolves to can-delete.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep a tiered volume delete-only when marked writable (rust)
set_writable preserved the remote delete-only mode only when the flag was
already set, so a tiered volume holding the upload-time read-only mark
(replicated, or quarantined under mark_tiered_deletable) was published
fully writable with its .dat remote. Derive no_write_can_delete from
has_remote_file() instead, matching load() and the Go MarkVolumeWritable.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume tier: serialize the quarantine check with the flag swap (rust)
A heartbeat refusal landing between should_quarantine() and the store
could be clobbered until the next heartbeat re-asserted it — the same
race the Go side closed by checking under noWriteLock. A
compare-exchange gives the same ordering: the swap fails, and the retry
sees the quarantine mark the heartbeat sets first.
* volume tier: MarkTieredDeletable refuses a non-tiered volume
A stray call on a local single-copy volume would pin it
read-only-can-delete. The only caller promotes right after
LoadRemoteFile, but the guard makes the method safe to reuse; the Rust
mark_tiered_deletable gets the same check, though its caller already
only reaches it under has_remote.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Javier de la TorreandJavier de la Torre
|
97f8ca13ff |
s3tables, iceberg: log client refusals at V(1), not ERROR (#11896)
* s3tables, iceberg: log client refusals at V(1), not ERROR Requests the S3 Tables and Iceberg REST APIs correctly refuse with a 4xx (access denied, not authorized, no such namespace, already exists, not empty, a commit conflict) were logged with glog.Errorf, so a client probing for what it may do, or retrying an optimistic commit, filled the ERROR log and hid real faults from anyone alerting on it. - s3tables: HandleRequest wraps the ResponseWriter in a statusRecorder (with Unwrap for http.ResponseController) and logs a handler error by the status it answered: 4xx at V(1) as "<op> refused (<status>)", 5xx or no status at ERROR as before. - s3tables: the "handleCreateNamespace called" trace moves from ERROR to V(3). - iceberg: writeManagerError's mapping is split into classifyManagerError, shared with a new logManagerError that logs 4xx-mapped errors at V(1) and everything else at ERROR; used by CreateNamespace, CommitTable and CommitTransaction. - storage: loading a remote-tier volume logged "readSuperBlock remote volume N: <nil>" at ERROR even when the read succeeded; it is now logged only when there is an error. Tests: TestRequestRefusedWith4xxIsNotLoggedAsError, TestLogRequestErrorByStatus, TestStatusRecorderKeepsTheFirstStatus, TestStatusRecorderUnwrapsForResponseController, TestNoTraceLoggedAtErrorLevel (fails on master on the CreateNamespace trace), TestLogManagerErrorRefusalsAreNotErrors, TestRemoteSuperBlockLogOnlyOnError. * s3tables: do not log a malformed CreateNamespace body at ERROR handleCreateNamespace logged a body it could not decode at ERROR before answering 400; the dispatcher already logs that refusal once at V(1). Also drop statusRecorder.Flush: it made the recorder an http.Flusher even over a writer that cannot flush, so http.NewResponseController reported success instead of ErrNotSupported. Unwrap already lets the controller reach the wrapped writer. * iceberg: require exactly one log line per classified manager error The test accepted a refusal that produced no log, or both logs. Require one V(1) refusal and no ERROR for a 4xx, and the reverse for a fault. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
Chris LuandDevin
|
780760fccc |
s3: report the copied bytes' MD5 in UploadPartCopy ETags (#11884)
* s3api: regression test for UploadPartCopy ETag on ranged copies Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: hash the copied bytes for a partial-chunk UploadPartCopy ETag A ranged UploadPartCopy that lands inside a source chunk stored the source chunk's ETag on the destination chunk, so the part reported the MD5 of the wrong bytes. streamCopyChunkRange now tees the forwarded plaintext into an optional digest writer; copySingleChunkForRange feeds it an md5 only when the range is not the whole source chunk, since a whole-chunk copy's stored bytes (and ETag) are unchanged.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: stamp the copied range's MD5 on UploadPartCopy part entries The part entry got no Attributes.Md5, so copyEntryETag fell back to filer.ETagChunks: a single-chunk part inherited the source chunk's MD5 and a multi-chunk part reported a <md5>-N composite of source chunks. copyChunksForRange now returns the MD5 of the copied bytes — a single chunk's own digest, or for multi-chunk parts a second ordered stream of the source range hashed beside the concurrent chunk copies (md5 cannot combine per-chunk digests). The handler stores it on Attributes.Md5 and mirrors putToFiler's ExtETagKey, so CopyPartResult, ListParts and the completed object's composite ETag use each part's own MD5 (#11855).\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: drop the source chunk's gzip flag on partial-range copies A ranged UploadPartCopy fetches raw (decompressed) bytes from the source volume and stores them uncompressed, but the destination chunk kept the source's IsCompressed flag, so reads of a partial copy of a gzip-stored chunk would try to decompress raw bytes.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: hash the chunk range, not the entry, for the part MD5 getEncryptedStreamFromVolumes prefers inline Content over chunks, but copyChunksForRange copies the chunks — on an entry carrying both, the hash would describe different bytes than the copy.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: cover sparse ranges in the part MD5 and fail on hash errors A single chunk's digest is the part MD5 only when it supplies the whole range; a sparse source zero-fills the rest, so gaps are hashed via the ordered range stream, and a fully chunkless range hashes zeros locally. A failed hash pass now fails the part copy instead of storing a chunk-composite ETag that describes the wrong bytes.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: rank copied-range destination chunks by source timestamps The concurrent chunk copies stamped each destination chunk with an arbitrary finish time, but reads resolve overlapping chunks by ModifiedTsNs — a copy of overlapping source chunks could expose different bytes than the source (and than the part MD5 just hashed). Destination chunks now get timestamps ranked by the source's, preserving its overlap precedence.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: cancel the part-MD5 stream when a chunk copy fails The hashing goroutine ran on context.Background(), so an early chunk-copy error return left it streaming and hashing the whole source range for a discarded result. A cancellable context now aborts it on every exit path.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: hash the copied range when the covering chunk has no usable ETag TUS and mount writes leave chunks without an ETag, so a whole-range UploadPartCopy would otherwise save the part with an empty Md5 and no ExtETagKey, and multipart completion rejects the part. * s3api: build the md5 source entry without copying the proto message * s3api: drop the issue reference from the part-copy etag test * s3api: route inline-content sources through the plaintext part copy The raw-byte path walks entry.GetChunks() only, so a ranged UploadPartCopy of an inline-Content source (filer HTTP or mount writes) produced a zero-filled part. Send those through copyObjectPartViaReencryption, which streams the real bytes and stamps their MD5 on the part. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: store the empty-part MD5 on writeEmptyCopyPart entries The slow path's empty part returned the right CopyPartResult ETag but saved no Attributes.Md5 or ExtETagKey, so ListParts and the completed object's composite ETag still fell back to a chunk-derived value. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Jay HemnaniandChris Lu
|
a3e97ec709 |
s3: apply Range to HeadObject (#11856)
* s3: apply Range to HeadObject * s3: apply Range to a HEAD of a stored directory key Directory-key entries answer through serveDirectoryContent before HeadObjectHandler's range branch, so a ranged HEAD returned 200 and the full length. * s3: keep the missing-directory check for plain directories only A stored directory-key marker is an object, so a range on its empty body answers 416 like on any other zero-byte object. * s3: let partNumber keep its priority over Range on a directory key HEAD * s3: serve ranged GETs of a stored directory key The same branch now applies the validated range to the inline content on GET, so a directory-key object answers 206 with the selected bytes like any other object. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
23d8c62844 |
volume: route lookups to the leader master only (#11888)
* volume: route lookups to the leader master only (#11887) After a leader change, the volume server kept sending volume and EC shard lookups to masters that were not the leader: * currentMaster was set as soon as a heartbeat stream opened, before the peer said whether it was leader, and kept a follower through the redirect and its 3s sleep. A follower answers LookupVolume from its own vid cache, which right after an election can be empty, so reads went 404 rather than failing over. * EC lookups used Store.MasterAddress, a second, unlocked copy that the heartbeat loop set before each dial and cleared on errors, so EC and plain reads could disagree on the master or have none. * the receive goroutine called stream.Send itself when the volume options changed, racing the send loop, and exited silently on failure, after which leader redirects were no longer read. Publish currentMaster only once a master is known to be leader: the peer answered a new stream without naming a leader, which the leader always does, or a follower named the leader. Compare redirects against the connected peer. Replace Store.MasterAddress with MasterFn, backed by getCurrentMaster. Have the receiver ask the send loop for the extra heartbeat instead of sending it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: ask the masters for the leader after losing the heartbeat When the heartbeat stream to the leader failed, currentMaster kept the lost master until a new stream answered, so lookups on a cache miss kept dialing it through the retry delay even with a new leader already elected. On a regular heartbeat error, ask the configured masters for the raft leader and publish it, within the retry delay rather than on top of it. Without an answer the last known leader stays: clearing it would only trade a dead address for an empty one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: stop the heartbeat on a failed assertion The leader test closed stopChan only once every assertion had passed, so a failure left the stream open and the fake master's GracefulStop waited for the whole test run's timeout. Start the heartbeat through a helper whose cleanup stops it, safe to call twice, and waits for it to exit. It runs before the cleanups registered earlier, so the stream is closed before the store and the fake master go away. Both heartbeat tests use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: clear the leader when no master names one When the heartbeat lost its master and no configured master named a leader, refreshLeaderFromPeers kept the lost one. That master is dead or no longer leader, and a follower answers LookupVolume from its own cache instead of refusing, so lookups could return locations no leader vouched for. Clear currentMaster instead. Lookups then fail fast ("missing address") until a leader is known again, rather than reading a follower's cache or waiting on a dial to a dead master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: keep volumeSizeLimit 64-bit aligned on 32-bit builds MasterFn (a func, 4 bytes on 386) replaced MasterAddress (a string, 8 bytes) ahead of volumeSizeLimit and moved it to a 4-byte-aligned offset, where atomic.LoadUint64 panics with an unaligned 64-bit operation. The atomic.Uint64 type carries the alignment itself. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: publish the answering leader before a duplicate-uuid retry Read the leader field first on each heartbeat reply. A follower reply carries nothing else, so it no longer runs through the option updates; and a leader that rejects the registration over duplicate uuids still answers lookups, so it stays the lookup target through the retry wait instead of leaving the previous master in place. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: publish the master only once it is known leader (rust) * volume: ask the masters for the leader after losing the heartbeat (rust) * volume: read the leader field first on each heartbeat reply (rust) * volume: drop the seed fallback from ec shard lookups (rust) cached_lookup_ec_shard_locations fell back to the static first seed when no leader was published, so a mid-election EC lookup could land on a follower and read its stale shard map. Error out like Go's masterAddress() returning "" instead, via a current_master() getter matching getCurrentMaster. * volume: send volume lookups to the confirmed leader, not the seed (rust) Every HTTP lookup path — proxy/redirect targets, replicated writes and deletes, batch chunk deletes, remote chunk reads — dialed the static first seed, so after a failover they kept asking a follower whose vid map can be stale. Go routes all of these through GetMaster; switch the lookup calls and their leader-known gates to current_master(). * volume: mark volumes readonly on the leader, retrying the named one (rust) notify_master_volume_readonly fell back to the seed when no leader was published, which the peer answers with "Not current leader". It now uses only the confirmed leader, skips the call in masterless mode, and on NotLeader asks the seed masters who leads and retries there — the same path Go's notifyMasterVolumeReadonly takes. The peer leader query is shared with refresh_leader_from_peers as lookup_leader_from_peers. * volume: ask the confirmed leader for copy prealloc options (rust) VolumeCopy's preallocate query went to the static seed; Go issues it through GetMaster. Route it through current_master() too — when no leader is known the lookup fails fast and the copy proceeds without preallocating, as before. * volume: keep the announced master port forms, not just http (rust) publish_current_master stripped to http form, so a leader configured with an explicit gRPC port (host:9333.19999) was redialed on the default offset. Store the announced host:port[.grpcPort] form like Go's currentMaster; callers pick to_http_address / to_grpc_address / parse_grpc_address per use, and Ping admission compares on the http form. * volume: fail replicated writes and deletes when no leader is known (rust) Gating replication on a known leader meant a write could be acknowledged on fewer copies than required, and a delete could leave the data readable on the other replicas. Go fails the request when the replica-location lookup cannot run; drop the gate so the empty-leader lookup errors the same way. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
4bdbfbaaa8 |
s3tables: judge an absent tag target by its canonical ARN (#11912)
* s3tables: judge an absent tag target by its canonical ARN authorizeAbsentTarget matched the bucket policy's Deny statements against the request's raw resource ARN. The ARN parsers accept any account and region but resolve the target by name, so a caller with an identity grant could reformat the ARN, miss the Deny match, and learn whether a denied name exists — the oracle the absent-target check was meant to close. Rebuild the ARN from the resolved names and the real bucket owner and evaluate the Deny against that, matching what the existing-target path checks. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: cover a foreign-account ARN on an absent denied table Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: keep the absent-target deny check behind a supplied ARN Callers that pass no resource ARN (ListTables, ListViews, CreateNamespace) must reach CheckTableOrBucketPermission's admin shortcut unchanged; running the bucket-policy Deny match first would deny the admin lists those callers take. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mohd Quamar TyagiandChris Lu
|
e73c23a4ee |
fix(s3): set Access-Control-Allow-Credentials on CORS responses (#11886)
* fix(s3): set Access-Control-Allow-Credentials on CORS responses (fixes #11885) * fix(s3): expand wildcard exposed headers for credentialed CORS responses * fix(s3): keep wildcard in exposed headers for non-credentialed CORS requests * fix(s3): expose x-amz-tagging-count on credentialed CORS responses --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
8c3d8fcea0 |
s3tables: keep absent-target answers identical to denied ones (#11905)
* s3tables: authorize before revealing a list or tag target exists ListTables, ListViews and the tag operations read the target's catalog metadata before any permission check, so a denied caller could tell an existing name (403 or the resource's own error) from an absent one (an empty list, or 404). Evaluate the operation's permission with whatever bucket context could be read and answer a denied caller the same way for absent, damaged and denied targets. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: answer bucket reads uniformly when the entry is not a table bucket A plain S3 bucket directory answers GetTableBucket and ListNamespaces with 500 attribute-not-found while a missing name gets 404, and a denied CreateNamespace is 403 while a missing bucket is 404. Run the permission check before the absence answer and read not-a-table-bucket as NoSuchTableBucket. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: cover absent-target answers matching denied ones Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
bb1ba8b247 |
iam: take the implicit user name from the verified identity (#11903)
* iam: take the implicit user name from the verified identity For self-service actions sent without UserName, the implicit user was resolved by re-parsing the raw Authorization header for a SigV4 credential and looking that access key up. The header is not what authentication verified: a request can be authenticated by one mechanism while carrying a header that names another user's access key, and the action then ran against that other user. The auth middleware already records the verified identity in the request context, so use it, on both the standalone iam server and the embedded IAM surface. When no verified identity is present the name is left unset rather than guessed. * iam: test implicit user name resolution Cover a request authenticated by presigned credentials that also carries a junk SigV4 header naming another user: the self-service actions must run as the verified signer. Update the unit tests for the context-based lookup. * iam: resolve the implicit user for standalone GetUser too The embedded surface fills UserName for GetUser; the standalone server skipped it, so an implicit GetUser on `weed iam` answered NoSuchEntity. Same verified-identity source as the other self-service actions. * iam: cover all implicit-username actions in the regression test |
||
|
Javier de la TorreandJavier de la Torre
|
593cfee85b |
s3tables: accept Puffin statistics files in a table's metadata/ (#11892)
Table and partition statistics are Puffin files that the table metadata lists in its statistics entries; orphan cleanup keeps them (#11651), and its test places one at metadata/stats-1.puffin. The table bucket layout rule accepted only *.stats among them, so an engine that names a statistics file *.puffin had the upload refused with 403 and could not write statistics at all. The metadata patterns now admit a *.puffin file directly under metadata/, with the same safe-character rule as the other catch-all patterns. Nothing else widens: a Puffin file in a subdirectory of metadata/, with a trailing extension, or outside <namespace>/<table>/ is still refused. Test: the new metadata/*.puffin cases in TestIcebergLayoutValidator_AcceptsRealWorldManifestNames and TestTableBucketFileValidator_PuffinStatisticsOnlyInTableMetadata fail on master ("invalid metadata file format") and pass with the change; the rejected cases pass on both. Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
Daniel García
|
03065155db |
filer.remote.sync: do not upload an entry whose only content is the remote object (#11890)
* filer.remote.sync: do not upload an entry whose local copy was dropped remote.uncache and the filer's eviction under storage pressure drop a synced entry's chunks and clear LastLocalSyncTsNs; remote.meta.sync does the same when it pulls a newer remote version. When the remote mtime falls in the entry's mtime second, which is the common case for a small file synced promptly, shouldSendToRemote falls back to the cleared stamp and reads the entry as dirty. The dropped chunks make the update a content change, so the chunkless entry was uploaded and a zero-byte object replaced the remote one, the entry's only content. Skip an update that leaves the entry at the same key with no local data and a sized RemoteEntry: the object at that key already is the content. The stamp stays cleared, so remote.cache and eviction still see the entry as uncached. Renames keep going through completeRemoteOnlyRename, metadata-only edits of a remote-only entry still reach the remote, and a rewrite in the remote mtime's second holds local data and is uploaded. * filer.remote.sync: upload a truncate that leaves the entry remote-only A FUSE truncate of a cached, remote-backed file to zero (truncateEntry, or SetAttr with size 0) drops the chunks and keeps the sized RemoteEntry, so the entry reads as remote-only and the dropped-copy skip left the remote object with its old content. Skip only while the entry's size is still the remote object's. Uncache and eviction keep the size and remote.meta.sync sets it from the object, so those stay skipped; a truncate changes the size and is uploaded, in the remote mtime's second too. |
||
|
jsasandjames
|
f8cfea0dcf |
gcs: size the upload buffer from the entry instead of reserving the default 16 MiB chunk per writer (#11854)
Co-authored-by: james <james@medable.com> |
||
|
Jay Hemnani
|
c90da999f2 | s3: answer a conditional write on a missing object with 404, not 412 (#11882) | ||
|
Chris LuandDevin
|
379abbe278 |
volume: repair torn tails on canDelete volumes; reject damaged 1-3 byte records (#11876)
- Go/Rust: run the unindexed .dat tail repair on noWriteCanDelete volumes too — deletes still append tombstone records, so the same crash window applies and they previously degraded to fully read-only. - Go readTailNeedle: reject a positive size of 1-3 for v2+ as corrupted before ReadBytes slices the body's 4-byte data-size field and panics; a torn last record is now cut instead of crashing the load. - Tests: TestVolumeLoadRepairsTailOnCanDeleteVolume and TestVolumeLoadCutsDamagedSmallSizeTailOnCanDeleteVolume (Go), and the Rust mirrors volume_load_repairs_tail_on_can_delete_volume / volume_load_cuts_damaged_small_size_tail. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
GAURIandGAURI
|
c601712cc4 |
README: describe OLake Go and fix its wiki link (#11880)
README: describe OLake Go and point its link at the renamed wiki page Co-authored-by: GAURI <174253719+gauriiiiiiiiiiii@users.noreply.github.com> |