mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-24 23:10:41 +02:00
0d4e11a6e5b0c7fb855388936d6bf79ad5001a2a
10239
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Lu
|
0d4e11a6e5 |
filer: bootstrap a timestamp-less peer entry onto a missing local path (#11975)
A zero local commit time conflated 'no local state' with 'state at the epoch', so a missing entry was skipped whenever the sender reported no commit time and the entry mtime was at or before the epoch. Only hold the copy off when a local entry actually exists. |
||
|
Chris LuandDevin
|
679835e4fc |
volume.fsck: run all filer-entry purges through the shared worker pool (#11960)
* volume.fsck: queue per-volume orphan deletes through the shared purge pool The worker pool added for -reallyDeleteFilerEntries only covered entries whose volume was missing entirely; the per-volume check that finds filer entries pointing at absent needles still issued one synchronous HTTP DELETE per orphan, which is the bulk of the work the report measured. One pool now serves both purge phases and is drained before the empty- directory sweep. * volume.fsck: keep hard link ids in the filer fid file Orphans queued by the per-volume check must hash into the pool by their hard link id just like the missing-volume path, so two paths sharing an id still serialize on one worker. The fid file now stores the id beside each path and readFilerFileIdFile hands it to the caller. * volume.fsck: cover orphan purge queuing and hard link id round-trip * volume.fsck: fail on a truncated hard link id in the fid file A .fid record cut short inside the hard link id length was emitted with a nil id, so its purge could hash to a different worker than the entries it shares the link with. Read the length strictly like the header and path. * volume.fsck: drain the save consumer before returning a root genFn error The early return left the save goroutine blocked on outputChan while the caller ran finishPurge, so a late queuePurge could send on a closed channel and panic. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * storage: skip EC shard-size encoding tests when disk space is low TestCalculateExpectedShardSizeWithRealEncoding writes a .dat file plus 14 EC shards per subtest (~1.3GB peak for the 512MB case) into the temp dir. Shared CI runners can run low on space while other test packages write to /tmp in parallel, and the encode then dies with ENOSPC mid-write, surfacing as a test failure. Check free space up front and skip the subtest instead. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
3810fb1123 |
s3: url-encode ListMultipartUploads results for encoding-type=url (#11998)
* s3: url-encode ListMultipartUploads results for encoding-type=url The response echoed EncodingType=url but returned raw upload keys, and XML 1.0 cannot carry control characters, so the encoder replaced them with U+FFFD. The listed key then matched no upload: AbortMultipartUpload with it answered 404 NoSuchUpload, leaving such uploads uncleanable. Encode the key-named elements AWS encodes for this API (Key, Prefix, Delimiter, KeyMarker, NextKeyMarker) the way ListObjects encodes its keys when encoding-type=url is requested. Upload ids stay opaque, and responses without encoding-type=url are unchanged. Fixes #11991 * s3: echo EncodingType only when encoding-type=url is honored A request with an unrecognized encoding-type echoed the value back in <EncodingType> while the keys stayed raw, matching neither AWS (which omits the element unless url encoding applies) nor the ListObjects handlers, which gate the echo on the parsed url flag. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
baf0dd680e |
ec.encode: require shard placement to meet -shardReplicaPlacement (#11947)
* ec: measure shard placement against replica-placement caps ecShardPlacementViolations reports where a volume's newest-generation shard placement exceeds the per-node and per-rack caps a -shardReplicaPlacement requests, using the same digit mapping the balance planner enforces (SameRackCount = shards/node, DiffRackCount = shards/rack, dc-qualified rack keys). * ec.encode: -strictPlacement keeps originals when placement breaches caps -shardReplicaPlacement tells the balancer the requested per-node and per-rack shard limits, but nothing verifies the outcome. With -strictPlacement the pre-delete shard verification also measures the final placement against those caps: a volume still over the node or rack limit aborts before its source is deleted, so the encode only completes when the requested failure-domain spread is actually met. * ec.encode: cover the strict-placement delete gate with a test The gate in verifyEcShardsBeforeDelete now runs against a stubbed topology: a compliant spread passes while an over-cap placement is refused before source deletion. The retry knobs become package vars so the test does not wait out the heartbeat-propagation backoff. * ec.encode: enforce -shardReplicaPlacement without a flag Drop the -strictPlacement option: the requested per-node and per-rack caps are always requirements at the pre-delete verification, so the encode only completes when the placement actually honors them. |
||
|
|
f5e319ae8e |
iam: GetUserPolicy returns NoSuchEntity for a policy name the user does not have (#11944)
* iam: GetUserPolicy returns NoSuchEntity for a policy name the user does not have When the user has stored inline policies, an unknown name fell through to the policy rebuilt from the user's actions, so a deleted policy still read back. The standalone IAM server already returns NoSuchEntity here. * iam: return service failure when listing inline policies fails A ListUserInlinePolicies error in GetUserPolicy fell through to action-based reconstruction, masking a store failure as a policy document. Propagate it as ServiceFailureException instead, matching the standalone iam server and the other callers in this file. * iam: return service failure when the inline policy store read fails --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
f8579bf728 |
s3: gate POST Object acl fields behind PutObjectAcl (#11939)
* s3: gate POST Object acl fields behind PutObjectAcl The POST Object handler promotes acl and x-amz-grant-* form fields onto the upload's ACL headers, and resolveFileMode honors the promoted canned ACL. The PUT path only applies such headers after an s3:PutObjectAcl grant; mirror that on POST by running the shared per-key authorization with the WriteAcp action whenever the request carries ACL headers. * s3: cover the POST Object acl gate Signed POST requests carrying an acl form field are denied for a writer without the WriteAcp action and for a writer a bucket policy denies s3:PutObjectAcl on the target key; the authorization boundary is exercised directly as well. * s3: cover x-amz-grant-* POST fields and the no-ACL case * s3: encode POST policy conditions as JSON in the acl-gate test |
||
|
Chris LuandDevin
|
168789d5f3 |
master: cap writable volumes per collection via -maxWritableVolumes (#11923)
* master: cap writable volumes per collection via -maxWritableVolumes Some deployments serialize disk writes (e.g. an Unraid array where concurrent writes to multiple disks collapse throughput). With more than one writable volume in a collection, the master's weighted pick spreads assignments over all of them, so several disks get written at once. -maxWritableVolumes=N narrows the pick candidates to the N fullest writable volumes, so a collection's writes concentrate on one (or few) volumes at a time. 0 keeps the current spread-everywhere behavior. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: give master.follower a maxWritableVolumes default * master: cap writable volumes inside the requested location * master: pick the capped volumes without sorting the writable list * master: assert the capped candidate set instead of random pick outcomes * master: describe -maxWritableVolumes as a per-layout cap --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
cd97c93deb |
command: keep filer, s3, and sftp up when the local unix socket cannot bind (#11999)
* pb: share the stale-socket tolerant unix listener ServeGrpcOnLocalSocket already treats the local socket as optional; pull the remove-and-listen block out so the filer, s3, and sftp http sockets can use it too. * command: keep filer, s3, and sftp up when the local socket cannot bind A stale socket file owned by another uid (e.g. left in /tmp by a containerized job) made os.Remove fail with EPERM and the Fatalf took the whole server down. The gRPC local socket already treats the unix listener as optional; do the same for the http/sftp sockets. * filer: clear shared localSocket flag when the unix socket cannot bind s3.localFilerSocket and (in weed server) the sftp local socket share the filer's -localSocket flag pointer. If the filer cannot bind the socket it now keeps running without it, so consumers still holding the configured path would dial a dead or foreign-owned socket. * command: keep sftp on its own localSocket flag under weed server sftpOptions.localSocket was overwritten with the filer's -localSocket flag, so -sftp.localSocket was ignored and an explicit filer socket path made the sftp server try to bind the same socket the filer already held. * s3: fall back to TCP when the filer local socket cannot be dialed The embedded S3 server copies the filer's -localSocket flag before the filer tries to bind it, so a socket that never comes up left every filer HTTP request dialing a dead path. Try the unix socket first and fall back to the request's TCP address on dial failure. * filer: keep the shared localSocket flag read-only after startup Clearing *fo.localSocket raced with the embedded s3 goroutine reading the same flag pointer and could not help a reader that had already copied it. The s3 filer client's TCP fallback covers the unbound-socket case without mutating shared flag state. |
||
|
Chris Lu
|
6c87fb70f9 |
server: skip local unix sockets that cannot be bound (#12000)
server: skip default local unix sockets that cannot be bound weed server fatally exited during S3, filer, or SFTP startup when the default /tmp/seaweedfs-<svc>-<port>.sock path could not be removed or bound. On shared hosts (e.g. CI runners where another user's weed left a socket in the sticky /tmp directory) a stale socket file turns into 'operation not permitted' and kills the whole server, even though the TCP listeners are unaffected. Warn and skip the unix socket listener instead, matching how the gRPC local socket already degrades. An explicitly configured -localSocket path still fails fast, since a co-located S3 server may be configured to dial the filer through it. |
||
|
Chris Lu
|
29b81a827d |
volume: also listen on localhost like other components (#11995)
* volume: also listen on localhost like other components Master, filer and s3 bind both the detected host ip and localhost via NewIpAndLocalListeners, but the volume server used a plain NewListener on bindIp, so a volume bound to the host ip never answers on loopback. Any tooling that polls localhost (e.g. the java test workflow's readiness curl on localhost:8080/status) then waits out its full timeout even though the volume is up and registered with the master. Use NewIpAndLocalListeners for the volume grpc, public and cluster http listeners, and add a small multiHttpdownServer composite so graceful shutdown still covers both listeners. * volume: stop and wait dual listeners concurrently A serial multiHttpdownServer.Stop let the localhost listener keep accepting requests while the bound-ip listener drained, and a serial Wait hid a localhost serving failure until the bound-ip listener ended. Run both waits and both stops in parallel: Wait reports the first error promptly, Stop closes every listener when shutdown begins. |
||
|
Chris Lu
|
c925869dcc |
shell: retry filer discovery until a filer registers (#11994)
* shell: retry filer discovery until a filer registers When -filer is not given, the shell discovers a filer with a single ListClusterNodes call right after connecting to the master. A filer can take several seconds to register after its port reports ready, so the list routinely comes back empty and the shell proceeds with an empty FilerAddress. Commands then fail with "dial tcp: missing address", which is how the s3tables Dremio integration test flaked in CI. Treat an empty list as retryable and keep polling ListClusterNodes until a filer appears or -connectTimeout expires. On timeout, print a clear "no filer registered" notice instead of failing later inside a command; master-only shell commands keep working either way. * shell: report the last filer discovery attempt failure A successful empty ListClusterNodes followed by a master outage left the diagnostic claiming no filer registered, hiding the real RPC failure. Track the outcome of the latest attempt instead: if the final list call failed, report that error; only when the last response was genuinely empty report that no filer registered within the timeout. * shell: test filer discovery retries against a fake master Extract the discovery loop into listFilers and cover the retry behavior with a fake master_pb.SeaweedServer: an empty cluster turning populated, an empty cluster exhausting the window without error, the last attempt's RPC failure being reported, and master-only commands still running after the discovery timeout. |
||
|
|
ad89f003a5 |
s3: end a catalog-vended session with the access key it was vended for (#11945)
* s3: end a catalog-vended session with the access key it was vended for A table credential the Iceberg catalog vends is a self-contained STS session: it stays valid until it expires (an hour by default) even after the access key that loaded the table is deleted or deactivated, or its identity is removed or disabled. An operator revoking a leaked key cannot cut off the table credentials that key already obtained. Sessions are stateless JWTs, so there is no revocation list to consult. This change follows the same pattern as role sessions, which are already refused once their role's ID no longer matches (checkSessionRoleBinding): a vended session carries the access key it was vended for (src_ak, recorded by the previous change), and validateSTSSessionToken checks it against the identity table held in memory. A session of provider iceberg-catalog that carries a source key is refused (InvalidAccessKeyId, as for an unknown key) once its identity is gone or disabled, or the key is no longer one of that identity's active, unexpired credentials. Scope: - Only catalog-vended sessions are checked; web-identity, AssumeRole and every other STS session are validated as before. - The vendor binds a session to the caller's key only when the caller is a stored identity holding that key. A caller that signed with another STS session's key, an OIDC user, or a request that recorded no key gets an unbound session, as today; binding it would refuse every request made with it. A session minted before this change carries no key and is unbound. - The check is one map lookup under the read lock and a scan of one identity's keys (about 11 ns, 0 allocations, with 1,000 identities). Tests (s3api_iceberg_vended_session_test.go) mint real sessions through the catalog vendor and validate them the way a data-path request is validated: deactivating, expiring or deleting key A ends A's session while the session vended through key B on the same identity keeps working; disabling or deleting the identity ends it; callers that are not a stored identity holding the key get unbound sessions that keep working; other STS sessions are not checked. The first two tests and the third fail without this change. * s3: keep the vended-session key binding when the key dies before vend vendedSessionKeyLive gated the binding, so a key revoked between the request's authentication and the vend skipped it and minted an unbound session that stayed usable until it expired. Every catalog auth path pairs the identity name in context with the key that authenticated, so a stored-identity principal always means the request signed with that identity's own key. Bind whenever the principal is a stored identity: a session bound to a now-dead key is refused on its first request, which is what the binding is for. * s3: gate the vended-session key binding on the authenticated caller type lookupByIdentityName(principal) named only that some identity exists, not that the request signed with a stored key: an STS session or OIDC principal that collides with an identity name got a session bound to its temporary key, which vendedSessionKeyLive then refused on every request. Verifiers now record the verified key as a stored credential, and the vendor binds only when that marker is present — keeping the binding even when the key was revoked between authentication and the vend. * s3: enforce the vended key binding under a dedicated claim Sessions minted before the binding existed carry the caller's key in the audit-only SourceAccessKeyId — including temporary STS keys that can never resolve in the identity table. Enforcing on that field would end live table credentials at upgrade; the binding now travels under BoundAccessKeyId, leaving legacy sessions unbound as before. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: bind vended sessions to a credential fingerprint The key id alone cannot identify a credential: an operator can delete a key and re-create it under the same id with a different secret, reviving sessions minted for the original. Verifiers now record a fingerprint of the secret at authentication; the session dies unless it still matches. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: keep recording the caller's key on every catalog session SourceAccessKeyId was unconditional audit metadata before the binding existed; gating it on the stored-credential marker dropped the record for callers that authenticated another way. With the binding under its own claim there is no ambiguity left, so the field records the caller's key again regardless of how it authenticated. * s3: key the vended-session credential fingerprint with the signing key The bound-credential fingerprint travels inside the vended session token. As a plain SHA-256 of the secret it gave any token holder an offline guessing oracle for weak secrets, and CodeQL flags it twice. Store the verified secret in the request marker instead and HMAC it with the STS signing key at vend and at validation. The claim can no longer be mined, and the per-request auth path stops hashing secrets. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
bf3081e23a |
s3 lifecycle: route version-file writes to current-version expiration (#11956)
* s3lifecycle: schedule ExpirationDays from version-file writes On a versioned bucket a PUT lands in the meta-log as the version file plus the .versions/ directory update — there is no bare-key event, so the version-folder skip dropped every version-file event for all kinds but ExpiredDeleteMarker. ExpirationDays therefore only fired via the recovery walker's cold-start / rule-change pass: once a shard had a saved cursor, no daily run ever scheduled another delete marker. A version-file write is the latest for its logical key at write time, so route it through the same event-driven evaluation the regular path runs. The match carries the file's identity with an empty VersionID — the dispatcher resolves the current latest at due time, and a fresher PUT meanwhile resolves to STALE_IDENTITY there. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lifecycle test: versioned Expiration fires on the second daily run Establishes the shard cursor with a first pass, then PUTs and backdates a new versioned object and runs again. The delete marker must appear without a rule change forcing a recovery walk. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: run TestLifecycleVersionedBucketExpiresOnSecondRun in the lifecycle matrix --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
2e7e691bed |
volume: drop stale index entries that resolve to a different needle (#11948)
* volume: drop index entries that resolve to a different needle An index row whose offset lands on a needle with a different id and a different size cannot have been produced by any write path: every appender indexes a key at the offset and size of the record it wrote, and WriteNeedleBlob keeps its indexed size consistent with the blob header. Such rows are corruption (for example an offset replayed with stale high bytes) and until now made the volume reject every write for the key with "mismatching cookie" while reads resolved to the wrong needle. When the duplicate check finds the indexed record belongs to a different key and size, drop the stale entry and let the append re-index the key. Same-key conflicts still fail the cookie check, and entries staying consistent with a re-keyed blob keep their existing semantics. * volume: verify the indexed needle header before streaming reads The buffered read path re-parses the record and rejects a needle whose stored size disagrees with the index, but the streamed path serves data bytes without ever looking at the header, so a stale index row would stream another needle's payload. Probe the header once up front and report a miss when the record is neither the requested key nor the indexed size, the same staleness test the write path applies. * rust volume: drop index entries that resolve to a different needle Mirror of the Go volume change: when the duplicate check finds the indexed record belongs to a different key and size, delete the stale entry and let the append re-index the key. * volume: overwrite a stale index entry on commit, not up front Deleting the stale row during the duplicate check published a tombstone before the replacement append and, for durable writes, before its fsync. A failed append or sync then rolled back the data but left the key deleted. The publish step already overwrites any older offset, so leave the entry in place and let it be replaced once the record commits. * volume: report streaming header-probe failures to the io error tracker A failed header read returned before the tracked body-read loop, so repeated EIO on the probe never reached checkReadWriteError. Skip io.EOF the way the loop does: it is a truncated record, not a disk fault. * rust volume: overwrite a stale index entry on commit, not up front Mirror of the Go volume change: the staged delete published a tombstone before the replacement append and its .dat flush, so a failed append or sync left the key deleted. publish_write already overwrites any older offset once the record commits. * rust volume: report needle meta read failures to the io error tracker read_meta is the header probe ahead of a streamed body: an EIO there escaped check_read_write_error entirely, so a failing disk could not trip the tolerance. Report the outcome the way read_full already does. * volume: retry the wrapped offset before judging an index entry stale A 4-byte index value can index past the 32 GiB wrap: the low offset may hold another needle while the real record sits at +MaxPossibleVolumeSize, which is where reads already retry on a size mismatch. Skipping the cookie check on the low-offset record alone would accept a foreign-cookie write over a live key. Retry the wrapped header read first, the same condition the read path uses; a wrapped read that fails for reasons other than EOF stops the write like the low-offset read does. * volume: probe the streamed needle header under the access lock With HasSlowRead the access lock is dropped after the index lookup, so a compaction commit could swap the backend between the lookup and the header probe and fail a valid download on the old offset. Take the read lock for the probe and resolve the compaction revision first, the way the streaming loop re-resolves it on every page. * rust volume: retry the wrapped offset before judging an index entry stale Mirror of the Go volume change: probe offset + MAX_POSSIBLE_VOLUME_SIZE under the 4-byte-offset build before treating a mismatched record as stale, so a live wrapped entry still gets its cookie checked. * rust volume: do not clear the io error streak on a successful meta read read_meta runs before the payload read in read_needle_for_get, so recording success there reset the streak before every failing download and the volume could never reach quarantine. Report failures only; read_full still records success after a complete read. * volume: record a meta-only read's success against the io error streak A meta read no longer clears the streak because a payload read still follows on GETs, but requests the meta read ends outright — HEAD answers and cookie mismatches — never reached the body path, so their success left earlier failures counted. Note the success on those terminal paths; mirrored in the Rust volume server. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: report streamed reads to the io error streak Streamed GETs never touched the tracker: a failing chunk read went uncounted and a completed download never cleared the streak, unlike readNeedleDataInto which does both. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: reset the io error streak only at terminal meta reads A streamed GET's metadata read runs before its payload read, so clearing the streak there erased the count before every failing download and the volume could never reach quarantine. Record the meta-read success only where the request actually ends: must-meta-only reads, HEAD responses, and cookie mismatches — matching the Rust volume server's placement. * rust volume: record the streamed read's success at the final frame Hyper can complete a Content-Length response without polling the body again, so a success recorded only when the stream is next polled could be skipped, leaving an earlier io error streak in place after a successful download. * volume: record a meta-only read's success on conditional 304 replies * rust volume: record a meta-only read's success on conditional 304 replies * volume: reset the io error streak only after the checksum passes A streamed read that ends in a CRC mismatch still cleared the EIO streak, so corruption could mask progress toward quarantine. The Rust side records success only after the final frame's checksum verifies; move the reset below the CRC check to match. * volume: satisfy codespell in the meta-read comment * rust volume: report range reads to the io error streak * rust volume: cover the range read's io error reporting --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
27fa285d48 |
filer: do not stall startup waiting out the leader-settle window (#11972)
* filer: do not stall startup waiting out the leader-settle window ListExistingPeerUpdates re-polled the leader for up to 10s when it reported zero filers. A fresh single-filer cluster can never gain peers, so every filer start paid the full settle timeout before listening — past the 6s health window in weed mini and short probe budgets. Take the leader's first answer instead; the fallback fullest-answer scan below still covers a leader that is mid-convergence. * filer: bound leader-settle re-poll below the startup health window * filer: update ListExistingPeerUpdates doc for empty-leader fallback * filer: re-check empty membership at the announce gate before registering A fresh filer that scans empty membership during a master restart or re-election skipped bootstrap and announced immediately, serving missing files until a peer happened to reconnect — and a late peer only got an incremental subscription, so pre-existing metadata was never backfilled. The announce gate now re-polls ListExistingPeerUpdates while the scan can still be disproved (10s), backfills from any late peers before their streams start, and then waits for their catch-up like startup peers. A genuinely empty cluster pays a delayed master listing rather than a stalled startup, since this runs after the ports already serve. * filer: overlap the late-peer copy cutoff and stream start A peer event stamped just behind this filer's clock could land after the traversal read but carry a timestamp below the stream start — the startup scan margins both cutoffs by LogFlushInterval for exactly this skew; mirror it for peers discovered at the announce gate. * filer: bound the late-peer bootstrap copy by the shared announce deadline MaybeBootstrapFromOnePeer now takes the caller's context so a stalled peer metadata copy cannot out-wait the announce cap, and the bootstrap re-check shares one announceCatchUpCap deadline with the catch-up wait instead of each spending a full minute. * filer: merge bootstrap entries by commit time instead of blind insert The late-peer copy runs while gRPC already serves, so a blind InsertEntry could replace a write the local filer just accepted. Routing each entry through applyTraversedEntry orders it against the local commit time under the replay locks — a fresher local write survives the backfill. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
b65832bdc9 |
iam/oidc: reject reused jti claims with a bounded replay cache (#11936)
* iam/oidc: reject reused jti claims with a bounded replay cache * iam/sts: pass jti replay settings through convertToOIDCConfig * iam/oidc: keep consumed jtis and stop the cleanup worker across re-init Re-initializing a provider reset the JTI store, so an already-used token could be presented again after a config refresh; the store is now allocated once and survives Initialize. The cleanup worker is stopped on every re-init (previously only when protection stayed enabled), is started only after the rest of Initialize has succeeded, and Shutdown no longer consumes a permanent sync.Once, so a re-initialized worker can be stopped again. * iam/oidc: enforce the jti replay cache bound for every token Tokens without a usable jti claim skipped the cache entirely and stayed reusable; they are now keyed by the token's own fingerprint so a repeat presentation is still a replay. Tokens without exp previously failed as expired on first use; their entry now gets a bounded retention. The replay test-and-set now runs before any capacity work so replayed tokens cannot each trigger a full-cache scan, the entry count is incremented as part of the insert so concurrent validations cannot overshoot the bound, and a guaranteed serialized sweep runs before a capacity rejection while opportunistic sweeps are rate limited and confined to one at a time. Raw token identifiers are no longer written to logs or errors, a stable fingerprint is logged instead. * iam/sts: shut down providers dropped by the managed-provider swap Each managed OIDC provider runs a background JTI cleanup worker, but SetIAMManagedOIDCProviders replaced the provider map without stopping the old providers, so every refresh leaked goroutines. Providers absent from the new map are now shut down; providers that stay keep running. * iam/integration: reuse providers across refreshes for unchanged records Every refresh rebuilt providers from their records, so a refreshed provider forgot the JTIs it had already consumed and an already-used token could be presented again. Records are now fingerprinted and an unchanged record reuses its provider, keeping the consumed-token cache; changed records are rebuilt, removed records are dropped, and the swap in STS releases the workers of whatever is no longer referenced. * iam/oidc: consume replay entries only on the single-use exchange path Bearer authentication validates the same token on every request, so consuming it in shared ValidateToken/Authenticate made every token single-use and rejected all but the first request. The replay check now runs only for the STS exchange via AuthenticateSingleUse; bearer paths validate without consuming. A token without exp also no longer gets a bounded replay entry that ages out while the token stays valid: its entry is kept until the capacity bound evicts it, so it can never become usable again. * iam/oidc: keep the mock provider on the mock path for single-use calls * iam/oidc: require exp on single-use tokens A cache slot frees only by expiry, so an expiry-less entry would hold it forever; a burst of distinct tokens could exhaust the bound permanently and block every later exchange. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: give each web-identity test token a unique jti Subtests minted tokens with identical claims inside one second, so the exchange saw the same token fingerprint twice and the second call read as a replay. Each exchange now carries its own jti. * iam/oidc: carry the consumed-token replay store across provider rebuilds A record change rebuilds the managed provider with a fresh JTI store, so a token consumed before the rebuild could be presented again until it expired. The new provider adopts the replaced provider's replay store before it is published, and the replaced provider is shut down so only one cleanup worker sweeps the shared store. * iam/oidc: stop the cleanup worker of providers dropped from the store A refresh removed the provider object from the managed cache but left its JTI cleanup goroutine running for the life of the process. * iam/oidc: merge a provider's consumed replay entries into another's store MergeReplayStateFrom folds one provider's consumed-token entries into another's, skipping already-expired ones. Unlike AdoptReplayStateFrom it swaps no pointers, so it is safe on a provider still serving validators. * iam/integration: share one provider and replay store across identical issuer records Records for the same issuer URL with identical validation settings now reuse a single provider, so a token consumed through one account's record is rejected through the other's — previously each record owned a separate jti store and a spent token could be replayed cross-account. A superseded provider's consumed entries merge into the shared store so no rejection is lost, and a removed or rebuilt provider is only shut down once no remaining record references it. * iam/oidc: hand the superseded provider's live replay store to the shared one * iam/oidc: carry merged replay entries without charging the capacity bound * iam/oidc: key the jti-less replay fingerprint on the signed input Hashing the whole token lets ECDSA signature malleability (s -> N-s) mint a second cache key for identical claims, so a consumed token could be replayed for another exchange. Key the fallback on header.payload instead. * iam/oidc: publish the jti cache bound before the enabled flag Initialize stored jtiEnabled ahead of jtiMaxSize, so a provider flipped from disabled to enabled exposed enabled=true with a zero bound for a window; a single-use validation landing there would compute newCount > maxSize and reject with "JTI cache capacity exceeded". Store the bound first so the flag is the last thing published. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
186bc5cf7f |
s3: bound ListMultipartUploads memory to the page size (#11971)
* s3: bound ListMultipartUploads memory to the page size listMultipartUploads loaded every open upload in the bucket into a slice and selectMultipartUploads filtered and sorted a second full copy, so one request cost O(total open uploads) heap regardless of prefix, key-marker, or max-uploads; concurrent calls multiplied it and could OOM the gateway. The response is ordered by the object key kept in extended attributes, not the upload id the filer sorts by, so every entry still has to be seen. Streaming the listing into a bounded top-K collector keeps at most maxUploads (capped at maxUploadsList) candidates, bounding heap by the page instead of the folder. A caller-supplied max-uploads above the server cap now also paginates rather than accumulating unboundedly. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: cover bounded ListMultipartUploads accumulation Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: report the effective max-uploads cap in ListMultipartUploads * s3: cover ListMultipartUploads retry through the collector --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
7154732b84 |
worker/ec: retry the pinned volume copy after a mid-copy compaction (#11929)
* worker/ec: retry the pinned volume copy after a mid-copy compaction The EC task pins .idx/.dat copies to the CompactionRevision read from ReadVolumeFileStatus so the pair comes from one volume generation. When vacuum compaction lands between that read and either CopyFile, the server rejects the copy with "volume N is compacted" and the whole task fails — on clusters with queued maintenance this makes most EC tasks die to a race they cannot avoid. On a compaction mismatch the copy now restarts under a freshly read revision, up to a bounded number of attempts; other errors still fail immediately. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * worker/ec: read the retried copy's expiry from its own file status A mid-copy compaction makes the source reload the volume, and the fresh ReadVolumeFileStatus can carry a different expireAtSec than the pre-compaction read. Keeping the attempt-0 expiry could let the EC volume outlive its source TTL. * worker/ec: cover the compaction retry with a stub volume server * worker/ec: also observe revised .idx bytes in the compaction retry test * worker/ec: read the fake source under its mutex in the retry test --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
f2d5c27f65 |
operation: renew the write JWT when a fan-out rollback delete is rejected (#11970)
* operation: renew the cleanup JWT when a fan-out rollback delete is rejected A stalled holder can out-wait the assignment JWT's 10s expiry, so the type=replicate delete that rolls back landed copies comes back 401 and the needle leaks. Retry the delete once with a fresh token minted through the new LookupJwtFunc hook when a holder answers unauthorized. * s3api: wire a cleanup-JWT renewer into chunked uploads Keep the master client on the server and use it to mint fresh volume JWTs for rollback deletes that run past the assignment token's expiry. * admin: wire a cleanup-JWT renewer into file-browser uploads * operation: bound the cleanup-JWT renewal with a per-attempt deadline A rollback delete that hits 401 renewed the token through a background context, so a master outage could park the wait for a connection and the lookup RPC inside the failed upload's cleanup path. Thread a context through LookupJwtFunc and LookupJwt, and bound each renewal with a 10s deadline covering both the master wait and the lookup call. * operation: let every holder's 401 retry with a renewed cleanup JWT A single renewed flag meant only the first rejecting holder refreshed the token; a holder that stalled past the renewed JWT's lifetime left later copies stranded with a 401 they could not resolve. The retry limit is now per holder — each delete is attempted once with the current token and once more after a bounded renewal — so a slow middle holder cannot strand the rest. * operation: log the volume server's error body on a failed cleanup delete Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * operation: count test upload attempts atomically The chunk upload loop fans AssignFunc out to parallel goroutines, so the plain int counter in the partial-results test raced under -race. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
b97ed0f101 | 4.49 | ||
|
Mohd Quamar Tyagi
|
94dd96a322 | weed filer.remote.sync: ignore .system directory under buckets (#11966) (#11969) | ||
|
Chris Lu
|
00e64441c6 |
fix: clone protobuf snapshots without copying internal locks (#11968)
* fix: clone protobuf snapshots without copying internal locks * fix: reuse cloned snapshot fields when filtering |
||
|
Chris Lu
|
8f6121694a |
test: fix HEAD bucket assertions and FUSE failover startup timing (#11967)
test: fix HEAD bucket assertions and warm up failover clusters |
||
|
Chris Lu
|
137d29633b | s3: fix prefix filtering and delimiter pagination (#11962) | ||
|
Chris LuandJavier de la Torre
|
82f0a82564 |
filer+volume: make acknowledged writes durable under the fsync rule (#11943)
* volume: never compact a volume tiered to remote storage A tiered volume's data backend is the remote object, but compaction reads the local .dat path. When tiering removed the local .dat, every vacuum of a tiered volume with garbage fails with "no such file or directory", and the master's sweep and the worker's vacuum task keep selecting it. When the local .dat was kept, the compaction succeeds against the stale copy and the commit reloads the volume to read the remote object (old layout) through the compacted index, so live needles stop reading. Mirror the Rust volume server (#11545): - CompactByIndex and CompactByVolumeData refuse a volume with a remote file before creating .cpd/.cpx ("volume N is tiered to remote storage, cannot compact"). - CommitCompact refuses under the data-file locks when the volume became tiered during the copy, and removes .cpd/.cpx. - Store.CheckCompactVolume reports no garbage for a tiered volume, so the master's vacuum and volume.vacuum do not select it. - The worker's vacuum detection skips a volume with a remote copy, as balance and EC detection already do. Tests (each fails on master): TestCompactRefusesTieredVolume, TestCommitCompactRefusesVolumeTieredDuringCopy (on master the commit succeeds and a surviving needle no longer reads), TestPickPrimaryReplicaSkipsTieredVolumes. * volume: skip tiered replicas in vacuum selection, keep their real ratio Reporting a zero garbage ratio for a tiered volume hid it from volume.tier.compact, whose preflight reads that ratio while the volume is still remote, and did not exclude it from a vacuum with a zero threshold. Report the real ratio again, as the Rust volume server does, and have the master's vacuum check skip a replica whose volume info names remote storage, whatever the threshold or request. When CommitCompact refuses a volume tiered during the copy, also close the temporary needle map and remove .cpldb: nothing else would, since no later vacuum selects the volume. The tiered-compaction test no longer depends on random needle sizes reaching a 0.3 garbage ratio. * volume tier: do not attach a copy of a .dat a compaction replaced The upload copies the .dat, then records the remote file in the .vif and switches the backend. A compaction commit in between swaps in a compacted .dat and index; the upload then served the old-layout remote object through the new index, so live needles stopped reading. CommitCompact's tiered check cannot see this: the volume is not tiered until the switch. AttachRemoteDatFile records the remote file and switches the backend under the data-file locks, and refuses when the backend is no longer the file that was copied. The upload then removes the unused remote copy. * volume tier: keep a committed .vif entry when only the dir fsync fails AttachRemoteDatFile rolled back the in-memory remote entry on any SaveVolumeInfo error. A NotCrashDurableError comes after the rename, so the .vif on disk already names the copy; with the entry gone, the upload handler deleted that object, and a reload would serve from a missing remote file. Keep the entry and switch to the copy as the .vif says, and still return the error so the caller keeps the local .dat. * vacuum worker: filter tiered replicas instead of skipping the volume A partially tiered volume is the leftover of an interrupted volume.tier.upload; skipping the whole volume left garbage on its local replicas unreclaimed. Per-replica filtering matches the master's batchVacuumVolumeCheck and is safe: a tiered replica refuses both compaction and commit. * filer: sync leveldb entry writes when the path rule asks for fsync The fsync storage rule only reached the volume upload; the metadata writes behind an acknowledged create, update or delete still went to the leveldb journal with Sync:false, so a power loss could roll back an acknowledged bucket or object entry. A write-side flag on the entry context carries the request: the filer sets it from the path's storage rule in CreateEntry, UpdateEntry, EnsureDirectoryEntry and DeleteEntryMetaAndData, and the HTTP write path sets it for an explicit ?fsync=true. The leveldb stores write with Sync:true when the flag is present. * volume: flush the .idx before acknowledging a durable write A write with fsync synced only the .dat: the index row written on publish stayed in the page cache, so a power loss could leave the needle's bytes durable with nothing indexing them. The durable paths now run one .idx sync after publish, before the requests complete — one per syncWrite and one per batch run, paired with the .dat sync, the same split the Rust volume server makes between flush_dat and flush_idx. A failed .idx sync cannot be rolled back, so it stops the volume taking writes and fails the requests instead of acknowledging an index entry that may not survive. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
|
7603528aff |
s3api: support s3:RequestObjectTag/<key> in bucket policies (#11910)
* s3api: support s3:RequestObjectTag/<key> in bucket policies The tags come from X-Amz-Tagging only where the handler stores them from there: not on PutObjectTagging, which reads the body, and not on a copy that keeps the source's tags. For a repeated key the last value counts, since that is what CopyObject stores. * s3api: extract RequestObjectTag only where the handler stores it The header was honored on any request carrying it, so a tag attached to a GET, DELETE, part upload, multipart completion, directory marker or a PUT subresource satisfied a tag-conditioned policy the object never receives. Extraction now follows the handlers: PutObject, a copy under X-Amz-Tagging-Directive: REPLACE, and CreateMultipartUpload store the header; a plain object PUT is identified by carrying only X-Amz-* signing parameters. * s3api: exclude multipart part PUTs before the copy-source check A PUT carrying X-Amz-Copy-Source plus a numeric partNumber and uploadId routes to CopyObjectPartHandler, which stores a part and never reads X-Amz-Tagging. Checking the copy source first let such a request satisfy an s3:RequestObjectTag condition under REPLACE with tags that are never stored. Mirror the router: the part handlers match only when both partNumber (digits only) and uploadId are present, so a lone or non-numeric partNumber still falls through to the storing handlers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: gate tag extraction on the routed subresource, not query key shape Rejecting every PUT whose query held a key outside X-Amz- dropped X-Amz-Tagging from ordinary object PUTs that carry arbitrary parameters — Signature V2 presigns (AWSAccessKeyId/Signature/Expires) among them — so a tag-conditioned Deny could be bypassed by appending an unrelated query parameter while putToFiler still stored the tags. Exclude the object PUT subresources the router actually dispatches (tagging, renameObject, acl, retention, legal-hold) ahead of the copy-source check instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: cover the routed-subresource and part-copy tag extraction cases Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
cec96c02ea |
s3: reject requests carrying more than one routed subresource (#11942)
* s3: reject requests carrying more than one routed subresource Routing picks the first matching subresource route while authorization resolves a single action from the query, and the two can disagree. A bucket-level PUT ?tagging&policy is authorized as s3:PutBucketTagging but served by PutBucketPolicyHandler, letting a tagger rewrite the bucket policy; the same shape lets a list grant read the policy via ?list-type=2&policy or ?versions&policy, and lets an object ACL writer retag via PUT ?acl&tagging. Declining ambiguous requests keeps the authorized action and the served handler identical. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: reuse hasAmbiguousSubresource in the routed-subresource guard The standalone key list dropped versioning, so ?versioning&policy still reached route matching and auth with two operations. The bucket-level check already keys off operationSubresources plus bucketQueryActions; running the same predicate keeps the lists from drifting apart. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Junker der Provinzandzouyixiong
|
ee2718b14c |
filer: reduce forwarded lock log noise (#11933)
Forwarding a lock to its owner filer is routine in a multi-filer setup, so the forward and acquire messages move from V(0) to V(3). The acquire message is only logged when the forwarded response holds a renew token and no application error. Transport failures still log at V(0). Co-authored-by: zouyixiong <3390062+zouyixiong@users.noreply.github.com> |
||
|
Chris LuandDevin
|
7ff77a4f3b |
s3api: skip error response body when http forbids it (#11931)
WriteResponse wrote the payload even for HEAD requests and statuses that disallow a body (1xx, 204, 304), which made net/http return ErrBodyNotAllowed and logged 'write err: http: request method or response status code does not allow body' for ordinary requests. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
0b197867a1 |
filer leveldb2: tunable cache, buffer, block and table sizes (#11927)
* filer leveldb2: read cache and buffer sizes from filer.toml The leveldb2 store hardcoded its block cache (32MiB) and write buffer (16MiB) and offered no way to tune the table block size or compaction table size. Optional block_cache_size_mb, write_buffer_size_mb, block_size_kb and compaction_table_size_mb keys under [leveldb2] now override them; zero or negative values keep the previous defaults. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer leveldb2: document tuning options in scaffold filer.toml Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer leveldb2: saturate size options instead of overflowing int The option fields take int, so a MiB config value past math.MaxInt/MiB (e.g. 2048 on 32-bit) wrapped negative and goleveldb would panic sizing the memtable. Clamp the product instead of the input. * filer leveldb2: reject size options too large for int bytes Clamping a MiB setting at math.MaxInt silently hands goleveldb a different size than configured; fail Initialize on values that cannot fit instead. * filer leveldb3: same tunable cache, buffer, block and table sizes Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer leveldb2/3: exercise oversized-option rejection on 64-bit too The tests skipped whenever int was 64-bit, but write_buffer_size_mb values above MaxInt/1MiB still overflow there, so the rejection path had no coverage on the primary platform. Use MaxInt/1MiB + 1 so the rejection triggers on every platform. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
1451096c17 |
volume.fsck: delete missing filer entries through a worker pool (#11924)
* volume.fsck: delete missing filer entries through a worker pool -reallyDeleteFilerEntries issued one synchronous HTTP DELETE per orphan path inside the BFS consumer, so a million-entry purge ran one request at a time. Queue the confirmed paths onto a bounded pool (default 16, -tunable via -purgeConcurrency) so deletes run in parallel while the traversal continues. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.fsck: serialize command output across purge workers * volume.fsck: serialize purges that share a hard link id * volume.fsck: cover the purge worker pool * volume.fsck: take the purge shard remainder in unsigned arithmetic Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Junker der Provinz
|
4854c6a0f4 |
s3: skip the local filer socket on Windows (#11921)
The filer only listens on its unix socket when GOOS is not windows, so with a filer socket path configured on Windows the S3 server dialed a socket that never exists. Fixes #7278 |
||
|
|
74da88fc29 |
s3, iceberg: record the access key a request was signed with (#11901)
* s3, iceberg: record the access key a request was signed with Several access keys can belong to one identity, but nothing downstream of authentication says which one signed a request: the audit log carries the identity name only, and every table credential the Iceberg catalog vends for that identity gets the same session name. When an identity is shared (one key per person or per service), the log cannot tell the holders apart. The verifier that matched a credential now records that credential's access key id in the request's identity holder (verifyV4Signature for header, presigned, streaming and STS requests; both SigV2 verifiers). Nothing re-reads the request for a key, so a second, unverified key elsewhere in the request (a junk X-Amz-Credential on a SigV2 request, a junk SigV4 header on a SigV2 presigned URL) is never recorded. - s3err.AccessLog gains access_key_id (omitempty): a static key's id, or an STS session's temporary key id. Every other field is unchanged. - The Iceberg and Lance catalogs install the holder before authenticating a SigV4 request; a Bearer token records the key it was minted from (the token is verified with a key derived from it); a Lance x-api-key records its key. - VendTableCredentials ends the session name with "-k" and the first 12 hex digits of SHA-256 of the key id (the table part gives way within the 64-character limit; the tag is never cut), and carries the key id in the session token (src_ak, SessionInfo.SourceAccessKeyId). Tests: auth_access_key_attribution_test.go and auth_access_key_shapes_test.go (two keys on one identity, the deny path, SigV2 and presigned SigV2 with a junk key elsewhere, streaming seed, STS session, failed signatures record nothing), s3api_iceberg_session_name_test.go, iceberg and lance Auth tests, sts source_access_key_test.go. * s3: record the access key of a verified POST-policy form and of a retargeted signature Two verified signatures left access_key_id empty in the audit log: - A browser POST upload is verified in PostPolicyBucketHandler from its form fields by doesPolicySignatureV2Match / doesPolicySignatureV4Match, which did not record the key. Both now take the request context and record the credential's key once its signature verifies; a form that fails verification records nothing. - verifyV4Signature refuses a signature that verified for one virtual-hosted bucket but was routed to another. The key was recorded only after that check, so the refusal carried none. It is now recorded as soon as a host candidate verifies. Test: TestPostPolicyUploadRecordsTheSigningKey (SigV4 and SigV2 forms, and a form with a wrong signature) and TestSignedVirtualHostRetargetRecordsTheSigningKey; both fail before this change. * s3: test VendTableCredentials end to end for the caller's access key TestVendTableCredentialsRecordsTheCallersKey mints sessions through VendTableCredentials with a configured STS service, the key recorded the way the catalog's authentication records it, then validates each session token: SourceAccessKeyId and the session name's key tag match the key, and a caller without a key gets an untagged session. It fails if the vendor stops passing either field. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Jaehoon KimandChris Lu
|
744ab586c5 |
fix: volume.fsck purges live chunks of files renamed during its filer walk (SQL stores) (#11883)
* shell: volume.fsck reads the filer from one store snapshot when it can volume.fsck collects the filer's chunk references by walking directory by directory, so each directory is seen at a different time. A file renamed from a directory not yet listed into one already listed is never seen, its chunks look like orphans, and -reallyDeleteFromVolume deletes live data. Checkpoint rotation and write-to-tmp-then-rename hit this. Add an optional filer store capability, EntryScanner, that visits every entry below a directory from one consistent snapshot. The SQL stores implement it with one REPEATABLE READ read-only transaction read in primary-key pages, so a concurrent rename is seen under exactly one of its names. Key-value rows are skipped by their dirhash, a bucket directory whose table is missing fails the scan (from the snapshot it cannot be told from one dropped a moment ago), and hard links are resolved in the same transaction. SQLite offers it only in WAL mode, on its own connection. TraverseBfsMetadata serves the snapshot when asked via request metadata sw-traverse-mode=snapshot and reports the mode in a response header; without the metadata it walks as before, so peer bootstrap and admin export are unchanged. Snapshot responses carry no entry_ts_ns, whose per-entry lookup would cost more than the scan. volume.fsck requests the snapshot and falls back to the walk on an older filer or a store that cannot scan. * filer, shell: fail a scan that cannot start, not fall back to the walk A transient BeginTx failure and a preliminary GetEntry error were treated like a missing capability or missing root, so fsck would take the directory walk - the very path that misses renamed-away entries. Only a genuinely unsupported scan or a confirmed-absent root may fall back. * filer: only snapshot-listed bucket names keep a missing table fatal A cached-only name, whose bucket may have been deleted between copying the cache and opening the snapshot, is skipped when its table is gone. * filer: roll back to a savepoint before skipping a cached-only table On PostgreSQL a failed statement aborts the snapshot transaction, so ignoring a missing-table error without the savepoint still left every later cached-only table unreadable. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
364e19a493 |
remote_storage/s3: use the AWS default credential chain when a source is configured (#11881)
* remote_storage/s3: use the AWS default credential chain when a source is configured Type s3 forced anonymous credentials whenever no access/secret key was set, so IRSA, container credentials, profiles and env keys were never consulted. Keep anonymous (public bucket) access only when the environment names no AWS credential source. * remote_storage/s3: also recognize shared-config and default-profile env sources * remote_storage/s3: resolve env credentials on their own transport A guarded HTTP client is for the S3 endpoint; the remote credential providers it would also serve reach link-local addresses that guard must not dial. Profiles load shared config, and a test resolves env keys. * remote_storage/s3: clear every detected credential source in the test * remote_storage/s3: resolve the full default chain on the guarded client The eager credential session used NewSession, whose chain covers only env, shared-credentials and ECS/EC2 providers. Web identity and assume-role profiles resolve through shared config, so IRSA never worked when a custom HTTP client was pinned. Enable shared config and carry the configured region so STS can sign assume-role requests. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
df33e81915 |
volume: report the needle read error with a failed read (#11934)
* volume: report the needle read error with a failed read A read that fails on the volume server, for example on a CRC mismatch, returned a bare 500, so mount and filer logs only showed the status. The volume server now writes the error into the response body, and the read helpers add it to the error they return. Co-authored-by: Bruce <82158894+half-life666@users.noreply.github.com> * volume: keep the 500 bare on HEAD requests writeJsonError's body write fails with ErrBodyNotAllowed under net/http when the request method is HEAD, adding an error-writing log line on top of the failed read. Send only the status, matching the no-error path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * util/http: decode gzip-encoded error bodies in statusError Full-chunk reads set Accept-Encoding: gzip themselves, so the transport does not transparently decompress; a gzip-encoded error body reached statusError as compressed bytes and produced an unreadable message. statusError now takes the response and decodes Content-Encoding before extracting the bounded message. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: log the read error when the 500 stays bare On HEAD requests the body write is skipped, which also skipped the V(0) response log inside writeJson — a failed HEAD read left no cause in normal volume-server logs. Log the error in the same format before the bare 500. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * util/http: check the status before decoding gzip in GetAuthenticated The gzip reader eagerly consumes the header when it is created, so statusError saw a drained or corrupt body on a gzipped error response and the server message was lost. Match the other read helpers and check the status first. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * util/http: export StatusError Other volume read paths outside this file need the same error body. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: redact jwt query param when logging failed-read URLs The new read-failure logs print the request URL, and volume reads accept ?jwt= as a credential, so a failed authenticated read would write the bearer token to the volume server log. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * util/http: surface the error body on the remaining chunk-read paths * volume: redact duplicate jwt params in failed-read logs * operation: pass the lookup jwt on the chunk needle read --------- Co-authored-by: Bruce <82158894+half-life666@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
867a537802 |
volume.configure: keep the superblock in sync with the .vif (#11930)
* volume.configure: keep the superblock in sync with the .vif ConfigureVolume only rewrote the .vif's replication field, leaving the .dat superblock stale. A mounted volume also kept reporting the old placement in heartbeats, and its next compaction would rewrite the stale placement into a fresh .dat. Parse the placement once, then update the live volume when mounted: the in-memory .vif copy and superblock, and the .dat header through the volume's own backend under the data-file locks. For unmounted volumes rewrite the header directly; remote-only volumes skip it since the .vif carries the placement. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.configure: update the .dat superblock in the Rust volume server Mirror the Go ConfigureVolume change: after rewriting the .vif, rewrite the replica placement in the .dat header too, so a remounted volume's superblock, heartbeats, and later compactions agree with the .vif. A missing .dat (remote-only volume) is skipped since the .vif carries the placement. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
85c1b990b1 |
shell: bound the initial master connection wait behind -connectTimeout (#11922)
* shell: bound the initial master connection wait behind -connectTimeout weed shell waited on WaitUntilConnected with a background context, so an unreachable master hung the command forever. The wait now runs on a 30s (configurable) timeout and reports the failure instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: stop the master reconnect loop when RunShell returns KeepConnectedToMaster ran on context.Background(), so once RunShell returned — including the new connect-timeout early return — the reconnect goroutine kept dialing forever, and repeated RunShell calls accumulated one each. Give it its own cancellable context, separate from connectCtx whose deadline must not cut reconnects under a live shell. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: bound filer discovery by -connectTimeout too The discovery WithClient and ListClusterNodes ran on the unbounded background context, so a master disconnect after the initial connect — or a persistently failing ListClusterNodes — could block startup past the deadline. Reusing connectCtx bounds the whole startup phase. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: report filer discovery failures instead of silently losing the filer target * shell: give filer discovery its own budget after master connects Discovery shared connectCtx, so a connection that landed near the -connectTimeout deadline left ListClusterNodes almost no time and no room for WithClient to retry. A fresh timeout keeps discovery bounded without starving it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: cover the -connectTimeout master wait with a test Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: note filer discovery in the -connectTimeout help text Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
ac319fe318 |
volume: never compact a volume tiered to remote storage (#11895)
* volume: never compact a volume tiered to remote storage A tiered volume's data backend is the remote object, but compaction reads the local .dat path. When tiering removed the local .dat, every vacuum of a tiered volume with garbage fails with "no such file or directory", and the master's sweep and the worker's vacuum task keep selecting it. When the local .dat was kept, the compaction succeeds against the stale copy and the commit reloads the volume to read the remote object (old layout) through the compacted index, so live needles stop reading. Mirror the Rust volume server (#11545): - CompactByIndex and CompactByVolumeData refuse a volume with a remote file before creating .cpd/.cpx ("volume N is tiered to remote storage, cannot compact"). - CommitCompact refuses under the data-file locks when the volume became tiered during the copy, and removes .cpd/.cpx. - Store.CheckCompactVolume reports no garbage for a tiered volume, so the master's vacuum and volume.vacuum do not select it. - The worker's vacuum detection skips a volume with a remote copy, as balance and EC detection already do. Tests (each fails on master): TestCompactRefusesTieredVolume, TestCommitCompactRefusesVolumeTieredDuringCopy (on master the commit succeeds and a surviving needle no longer reads), TestPickPrimaryReplicaSkipsTieredVolumes. * volume: skip tiered replicas in vacuum selection, keep their real ratio Reporting a zero garbage ratio for a tiered volume hid it from volume.tier.compact, whose preflight reads that ratio while the volume is still remote, and did not exclude it from a vacuum with a zero threshold. Report the real ratio again, as the Rust volume server does, and have the master's vacuum check skip a replica whose volume info names remote storage, whatever the threshold or request. When CommitCompact refuses a volume tiered during the copy, also close the temporary needle map and remove .cpldb: nothing else would, since no later vacuum selects the volume. The tiered-compaction test no longer depends on random needle sizes reaching a 0.3 garbage ratio. * volume tier: do not attach a copy of a .dat a compaction replaced The upload copies the .dat, then records the remote file in the .vif and switches the backend. A compaction commit in between swaps in a compacted .dat and index; the upload then served the old-layout remote object through the new index, so live needles stopped reading. CommitCompact's tiered check cannot see this: the volume is not tiered until the switch. AttachRemoteDatFile records the remote file and switches the backend under the data-file locks, and refuses when the backend is no longer the file that was copied. The upload then removes the unused remote copy. * volume tier: keep a committed .vif entry when only the dir fsync fails AttachRemoteDatFile rolled back the in-memory remote entry on any SaveVolumeInfo error. A NotCrashDurableError comes after the rename, so the .vif on disk already names the copy; with the entry gone, the upload handler deleted that object, and a reload would serve from a missing remote file. Keep the entry and switch to the copy as the .vif says, and still return the error so the caller keeps the local .dat. * vacuum worker: filter tiered replicas instead of skipping the volume A partially tiered volume is the leftover of an interrupted volume.tier.upload; skipping the whole volume left garbage on its local replicas unreclaimed. Per-replica filtering matches the master's batchVacuumVolumeCheck and is safe: a tiered replica refuses both compaction and commit. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
164a7d8b56 |
upload: upload -dir files concurrently behind a new -c flag (#11918)
* upload: upload -dir files concurrently behind a new -c flag Each file under -dir was submitted serially inside filepath.Walk, so every small file paid a sequential master-assign plus upload round trip. A bounded pool now runs those submissions concurrently while keeping results in walk order; failures are reported per file and the command exits non-zero when any upload failed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * submit: close opened readers when a batch assign fails SubmitFiles returns early when the assign RPC fails, before FilePart.Upload gets a chance to close each opened file. Close the readers on that path so repeated assign failures do not leak handles. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * upload: stream -dir results as each file completes Batching the output until every upload finished meant an interrupted run lost the file ids of completed uploads. A per-index ready channel lets a printer emit each result in walk order as soon as it lands. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * submit: close the opened file when Stat fails newFilePart returned the still-open handle with the error, and the concurrent upload path records the error without ever reaching SubmitFiles, leaking one descriptor per failing file. * upload: still upload files collected before a walk error A walk that aborts partway used to leave the already-uploaded files behind; collecting paths first must not drop them, so upload what was gathered and still exit non-zero. * upload: submit batched files concurrently SubmitFiles fetched all file ids in one assign RPC but then uploaded the files one after another, so a multi-file list still spent its time in serial per-file round trips even with -dir concurrency. Uploads now run through a small worker pool while results keep their input order. Verified end to end: a 5-file list uploads with consecutive file keys and each stored needle matches its source bytes. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * upload: print per-file results before exiting on a partial batch failure --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
chuanying chenandChris Lu
|
d35f4c1c6b |
Missing data read in super block extra parsing (#8392)
* storage: fail on short super block extra read even without an error Some backends (e.g. s3_backend) clear io.EOF after a short read, returning n < len(p) with a nil error. Checking n only when the error is non-nil lets truncated extra data reach proto.Unmarshal with a zero-filled tail. * storage: apply the same short-read check to the super block header A backend that returns n < SuperBlockSize with a nil error previously slipped through and parsed a partially zero-filled header. * storage: test super block extra reads Covers a round trip through ReadSuperBlock, a truncated extra region, a backend returning io.EOF alongside a full read, and a backend returning a short read with a nil error. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
1aad4cfe70 |
iam: optionally require an iam: grant for self-service access-key actions (#11937)
* iam: optionally require an iam: grant for self-service access-key actions
AuthorizeIamAction lets every authenticated identity run CreateAccessKey,
DeleteAccessKey, UpdateAccessKey, ListAccessKeys and GetUser on itself with
no grant. On AWS these actions need an IAM policy (the usual one grants them
on arn:aws:iam::*:user/${aws:username}); nothing is allowed implicitly.
Where identities hold data-plane rights only, the implicit carve-out lets a
leaked key mint further keys for its own identity, with no cap, that keep
working after the operator deletes the key that leaked; it can also
deactivate or delete the keys the operator issued.
A new option, off by default, makes these actions need the iam:<Action> grant
like every other IAM action: -s3.iam.selfServiceRequiresGrant on weed
server, filer and mini, -iam.selfServiceRequiresGrant on weed s3. With it
set, the grant is authorized against the target user's ARN, the caller's own
when UserName is empty, so a policy can confine it to the caller's own keys
(Resource arn:aws:iam:::user/${aws:username}), the same resource scoping
CreateServiceAccount already uses. The admin and iam:* grants are unchanged;
with the option off nothing changes.
Tests (s3api_embedded_iam_authz_test.go): with the option set, an identity
holding only s3:* and a bare identity are refused all five actions on an
empty, own and other target (fails without the change); a grant on the
caller's own user ARN, literal or ${aws:username}, allows them on itself and
not on another user; the admin and an iam:* grant keep every action; with the
option off self-service is allowed as before.
* iam: wire selfServiceRequiresGrant on the standalone iam server
Every other command that runs the embedded IAM API accepts the flag; the
deprecated weed iam could not enable the enforcement at all.
* iam: wire selfServiceRequiresGrant on weed server and filer -iam
The IamOptions used by 'weed server -iam' and 'weed filer -iam' never
received the flag, so startIamServer dereferenced a nil pointer at
startup. Share the -s3.iam.selfServiceRequiresGrant pointer with them.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
|
||
|
Chris LuandDevin
|
ea859a0844 |
sftp: reject empty passwords in PasswordAuthenticator (#11938)
* sftp: reject empty passwords in PasswordAuthenticator FileStore.ValidatePassword already refuses a zero-length stored or supplied password, but the guard belongs at the authenticator too: an empty password is never valid, and checking before the store call keeps any user.Store implementation without its own empty-secret check from matching on zero-length input. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * sftp: keep the brute-force delay on empty-password failures The early rejection returned before the shared failure path, so an empty password failed instantly while other failures slept 100-199 ms. Folding the length check into the store call keeps the store untouched and every failed attempt equally slow. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
c0152e6be1 |
s3: keep '\' a literal object-key character behind -s3.allowBackslashInKey (#11911)
* s3: keep '\' a literal object-key character when AllowBackslashInKey is set NormalizeObjectKey rewrites every S3 object key: '\' becomes '/' and duplicate slashes collapse. On AWS S3 both are ordinary key characters, so keys that are distinct there alias one object here — writing probe/back/slash.txt silently overwrites probe/back\slash.txt. The filer only treats '/' as a path separator, so a verbatim '\' stays inside one entry name. Gate the backslash rewrite and the matching '\' handling in IsValidObjectKey behind a process-wide mode the S3 server sets from the new S3ApiServerOption. Duplicate-slash collapsing stays: an entry name cannot contain an empty segment. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * command: add -s3.allowBackslashInKey flag for AWS-compatible key handling Registers the flag on all four S3 entry points (s3, server -s3, filer -s3, mini) and passes it through S3Options into S3ApiServerOption. Default off, so existing normalization is unchanged. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: correct IsValidObjectKey/NormalizeObjectKey docs under allowBackslashInKey Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: normalize DeleteObjects body keys the same as path keys DeleteObjects keys arrive in the XML body, so they bypassed the normalization applied to route-var keys. The lock, the authorization checks, the table-bucket layout check and the filer delete itself all addressed the raw key while the conditional check resolved the normalized one, so a key spelled with '\' or '//' could be reported deleted while the stored entry survived. Echo the client-sent key in the result while operating on the normalized form. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: report a normalized-empty DeleteObjects key as a delete no-op A multi-delete key of only slashes normalizes to the empty string. Left to reach the filer delete, the empty name resolves to the bucket entry itself, so an unauthorized caller could remove a bucket through the body-key path. Such a key can never name a stored entry, so like a missing key the delete is an idempotent no-op reported before any permission check or filer call. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: drop unused AllowBackslashInObjectKey getter * s3: answer conditional delete of a normalized-empty key with NoSuchKey * s3: normalize the probed key in authorizeAnonymousBucketList The masked-miss probe read the raw mux var, so a spelling like folder// or folder\ was authorized against a key that can never exist while the request itself targets the normalized folder/. Probe the normalized key so the answer matches what the read path authorizes. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: cover anonymous authz of a literal-backslash key under allowBackslashInKey A public grant on the slash key must not authorize an anonymous read of the distinct backslash object when the flag keeps '\' literal; grant the backslash key's own ACL and it reads back. Default mode still folds the spelling onto the covered slash key. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: publish the key mode only after the S3 server is constructed A failed NewS3ApiServerWithStore could previously flip the process-wide key mode out from under an already-running gateway. * s3: normalize the copy-source key before validation and authorization pathToBucketObjectAndVersion returned the raw header spelling, so an equivalent spelling (a//b, a\b) was authorized and compared against the normalized destination while resolveCopySourceEntry looked up the normalized key — a deny on the canonical key could be bypassed and self-copy detection missed. Normalize once at parse so the CopyObject and UploadPartCopy paths agree. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
e057244b0a |
storage: read the super block extra before unmarshaling it (#11932)
* storage: read the super block extra before unmarshaling it ReadSuperBlock allocated a buffer of ExtraSize bytes but never filled it, so any volume with a super block extra failed to load with an invalid wire-format error. Co-authored-by: chuanying chen <18412672+ccy00808@users.noreply.github.com> * storage: test super block extra read error paths --------- Co-authored-by: chuanying chen <18412672+ccy00808@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Javier de la TorreandJavier de la Torre
|
763e2a2090 |
iceberg: record geometry bounds on the data files compaction writes (#11898)
* iceberg: record geometry bounds on the data files compaction writes Compaction collected column statistics for every primitive column except geometry: the bound converter has no geometry case, so a compacted file's manifest entry carried value and null counts for a geometry column but no lower or upper bound. Readers that skip files by these bounds for a spatial filter (for example DuckDB for `geom && <box>`) had to open every compacted file, where the files before compaction could be skipped. The bound follows the spec's single-value serialization for geometry: the lower bound is the point (xmin, ymin), the upper bound (xmax, ymax), each an 8-byte little-endian double; z and m are not written. It is the union of the written file's row-group geospatial statistics, which parquet-go computes from every non-null value. A bound that is too small makes a reader skip rows, so any doubt gives none: a row group with non-null values but no geospatial statistics (parquet-go omits them when a value does not parse as WKB), a box that is not finite, an inverted box, or a chunk whose statistics cannot be read. A row group of nulls only constrains nothing. Geography gets no bound, as iceberg-go writes none for it (geodesic edges, boxes across the antimeridian). Tests: points and polygons far apart through a bin-packed compaction (the bound equals the union of every geometry), a sorted compaction (any bound covers every geometry), a column of nulls only, an unparseable value in a second row group (no bound; without the rule the file got the bound of the first row group only), geography, and the aggregation rules. * iceberg: do not order compaction bins by a geometry column When a table declares no usable sort order, compaction orders a bin's files by the first column every file has bounds for. A v3 table whose first column is a geometry qualifies as soon as its files carry geometry bounds (DuckDB writes them, and compacted files now do too). Those bounds decode into geo literals that compareLiterals cannot compare, so the order sorted nothing, yet an oversized bin was still split into "ordered" runs by splitOrderedBin. Geometry and geography columns are now skipped in both the declared sort order and the inferred one, so such a table is ordered by its next column with bounds. Test: TestResolveCompactionOrderSkipsGeometry, a geometry column first and an id column second; on master the order is the geometry column. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
|
917b6f4afd |
iceberg: authorize a commit before staging its metadata file (#11891)
* iceberg: authorize a commit before staging its metadata file Both commit paths (CommitTable and CommitTransaction) write the new vN.metadata.json into the table's metadata/ directory and only then reach UpdateTable, whose authorization refuses a caller that may read the table but not commit to it. The refused commit then deletes the staged file on the way out, so such a caller could make the server write and delete a file under the table's prefix on every attempt. The same shape was closed for creates in #10991. Both paths now ask Manager.AuthorizeUpdateTable right after loading the table and answer 403 ForbiddenException before anything is staged. UpdateTable still decides under the policies in force when the pointer flips, so a policy changed mid-commit is caught as before. Test: TestCommitDeniedBeforeStagingMetadata records every file the handlers create; on master both the single-table and the transaction commit by a GetTable-only identity write v2.metadata.json before answering 403, with the change nothing is written. TestCommitAllowedStagesMetadata is the positive control (the owner, and an identity whose own actions include UpdateTable). * iceberg: authorize a view commit before staging its metadata handleUpdateView has the same write-before-authorize gap the table commit paths had: it stages the new metadata file and only reaches the UpdateView authorization at the pointer flip, so a caller who may only load the view could still make the server write and delete files under it. Gate it on AuthorizeUpdateView, mirroring authorizeCommit. * iceberg: seed committed view metadata in the denied-commit test Without stored view metadata the request fails on the missing-metadata check before it could stage a file, so the zero-write assertion could not catch a write-before-denial regression. Seed the same metadata the allowed-path test uses so the denied request would otherwise stage. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
106fd31af1 |
s3: resolve versionId ACL and attributes requests to the version-scoped actions (#11916)
* s3: resolve versionId ACL requests to the version ACL actions A request for an object's ACL carrying versionId acted on the historical version but authorized against the current-object action (s3:GetObjectAcl/s3:PutObjectAcl), because the acl block in resolveFromQueryParameters returned before versionId was considered - the same gap closed for tagging. An identity granted only current-object ACL access - or explicitly denied s3:*ObjectVersionAcl - could still read and modify ACLs on noncurrent versions. ?acl&versionId=<id> on an object now resolves to s3:GetObjectVersionAcl/s3:PutObjectVersionAcl; ?acl alone keeps the current-object actions and bucket ACL is unaffected. s3:PutObjectVersionAcl is mapped onto the WriteAcp identity action so policies can name it. * s3: resolve versionId attributes requests to the version attributes action GET /bucket/key?attributes&versionId=<id> fetched the historical version's attributes but authorized against s3:GetObjectAttributes, so a grant limited to the current-object attributes action could reach noncurrent versions. The request now resolves to s3:GetObjectVersionAttributes, and both attributes actions map onto the Read identity action so policies can name them (s3:GetObjectAttributes was previously rejected as an unmapped action). * s3api: recheck the version action on versioned ACL/attributes requests The phase-2 policy recheck inside GetObjectAclHandler, PutObjectAclHandler, and GetObjectAttributesHandler passed the current-object action even when the request carried versionId and phase 1 had resolved the version action. A Deny scoped to s3:*VersionAcl or s3:GetObjectVersionAttributes could never fire in the recheck, and an Allow limited to the version action went unevaluated. Select the action the same way the resolver does — versioned whenever versionId is present. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: recheck the version action on versioned tagging requests The same phase-2 gap exists in the tagging handlers left by the earlier version-tagging resolution change: versioned GetObjectTagging / PutObjectTagging / DeleteObjectTagging rechecks evaluated the current-object tagging action, so a Deny on s3:*VersionTagging could not fire. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: resolve HEAD ?acl as the head-object action, not an ACL action No HEAD acl route exists, so HEAD /bucket/key?acl lands on HeadObjectHandler — which serves object headers, not the ACL. Resolving it to an ACL action denied readers who hold only GetObject(Version) and let ACL grants reach a head-object read. Dropping HEAD from the acl branch lets the request fall through to GetObject/GetObjectVersion, matching how the tagging branch already treats HEAD. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: require the version read grant for versioned attributes AWS asks for s3:GetObjectVersion alongside s3:GetObjectVersionAttributes when a GetObjectAttributes request names a versionId; the resolved version-scoped action on its own left a lone s3:GetObjectVersion Deny unable to cover the attributes read. Check the version read grant too: the identity side through VerifyActionPermission, the bucket-policy side through the same recheck that evaluates the scoped action. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: check the version read grant on a synthesized versioned GET The versioned attributes check passed the literal s3:GetObjectVersion action, but both policy paths re-resolve the action from the request, so the attributes query parameter mapped it back to s3:GetObjectVersionAttributes and a lone version-read Deny never fired. The literal action also reached the legacy CanDo path, where a plain Read grant no longer matched. Evaluate the grant on a request shaped like a plain versioned GET instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: reject a named versionId on buckets without versioning Tagging, ACL, and attributes handlers answered versionId requests on never-versioned buckets as the current object while evaluating the version-scoped action, so a grant limited to the version action could authorize changes to the live entry. Reject the request up front through the existing checkVersionIdConfigured gate. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: carry presign params onto synthesized policy requests objectKeyPolicyRequest rebuilt the query with only versionId and the STS token, so a presigned request's s3:authType condition value silently changed from REST-QUERY-STRING to unset during phase-2 rechecks (version read grant, copy source, delete keys, post-policy writes). A policy conditioned on s3:authType would then deny a request it already allowed. Carry the SigV2/SigV4 presign keys through; none of them steer ResolveS3Action. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
137f23a229 |
ec distribution: implement the documented shard distribution API (#11915)
* ec distribution: parse EC and replication configs into a shard plan NewECConfig/DefaultECConfig and NewReplicationConfig(FromString) turn an EC ratio and a replica placement into ECConfig/ReplicationConfig values, and CalculateDistribution derives the target and maximum shard counts per data center, rack, and node. ECDistribution.Summary and FaultToleranceAnalysis describe, per topology level, whether losing one failure domain leaves enough shards to reconstruct the volume, so the EC durability guarantees of a given setup can be answered directly instead of only documented informally. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: analyze topology and plan shard moves TopologyAnalysis records nodes and shard locations per DC/rack/node, and Rebalancer.PlanRebalance emits the moves that level the layout: data centers first, then racks within each DC, then nodes within each rack, moving parity shards before data shards and honoring per-node free slots. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: cover the plan math and rebalancing with tests Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: keep AddShardLocation and node ShardIDs in sync AddShardLocation updated the count maps but never the node's ShardIDs — yet every rebalancer count reads len(node.ShardIDs), so the documented AddNode-then-AddShardLocation flow produced an empty rebalance plan. It now appends to the node's shard list and skips shards the node already has, making the two registration styles equivalent and idempotent. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: key rack-scoped maps by dc/rack Rack names are only unique within a data center — two DCs can both hold a "rack1" — so ShardsByRack, RackToShards, and RackToNodes keyed by the bare rack name merged distinct domains and corrupted both the DC-level shard counts and rack-level balancing. Rack lookups now use the dc/rack pair; DCToRacks still lists bare rack names. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: never move a shard onto a node already holding it The move picker selected the destination node before the shard, so a node already holding the chosen shard could be targeted. Pick the shard first (parity-first) and choose among destination nodes that lack it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: tolerate zero-value replication configs A ReplicationConfig{} with zeroed minimums made maxShare divide by zero; clamp each level's unit count to at least one. Covers the new behaviors with tests. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: keep rack keys distinct when names contain "/" rackKey joined data center and rack with "/", so dc "a"/rack "b/c" and dc "a/b"/rack "c" shared one key and were counted as the same failure domain. Separate the parts with a NUL byte, which cannot appear in a name. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: plan rebalances on a copy of the topology recordMove mutated the caller's TopologyAnalysis while planning, so an interrupted or partial execution left the input describing moves that never happened, and replanning on it returned nothing. PlanRebalance now simulates on a cloned analysis; plan.Analysis reports the layout the moves would produce and the caller's input is left untouched. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: balance nodes against the rack's current shard count The node-level target still read the original analysis, so a rack that had just sent or received shards during DC/rack balancing aimed at a stale total and left its nodes unbalanced. * ec distribution: register each node id once Adding the same node twice appended it to RackToNodes twice, so every count that walks rack nodes double-counted its shards. * ec distribution: normalize replication counts in the stored plan A zero-value ReplicationConfig computed targets against one domain per level while Summary still reported zero DCs, racks, and nodes. * ec distribution: cap a survivable level's max at the loss budget * ec distribution: skip shards already inside the destination domain * ec distribution: register nodes implied by shard locations * ec distribution: merge repeat node registrations A shard location now auto-registers an unknown node with no free slots; a later AddNode carrying the real capacity was then dropped entirely. Repeat registration now updates the node's capacity and adopts any new shards, while the first registration keeps the topology placement. * ec distribution: try other overloaded sources before stopping When the most overloaded domain could not shed any shard, balance returned early and skipped every other source that still had movable shards. A source that cannot move now gets set aside for the rest of the pass. * ec distribution: collapse chained shard moves to the final destination Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec distribution: schedule collapsed moves behind the departures that free slots Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Junker der ProvinzandDan Foster
|
e645449872 |
fix(storage): count preallocated space when auto-sizing max volumes (#11919)
The max volume count subtracted each volume's remaining room from the free disk space, except when the master had preallocation turned on, in which case it subtracted nothing. That is wrong for a disk that mixes preallocated and plain volumes, and on platforms where preallocation is not supported. Use the larger of the .dat file's size and its allocated blocks, in the Go and the Rust volume server, so a preallocated volume counts as full and a plain one keeps its remaining room reserved. Fixes #5778 Co-authored-by: Dan Foster <4570306+danfoster@users.noreply.github.com> |