mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-11 16:57:45 +02:00
59db4ddc6cf40d817f8e5a394a22ab9faafe95a4
15444
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 homeofe
|
59db4ddc6c | build(deps): bump golang.org/x/net from 0.58.0 to 0.61.0 (#11705) | ||
|
Bharat Kaurav
|
e7ae0d6049 |
s3api: accept max-keys up to 2147483647 and cap list pages at 1000 (#11702)
* fix(s3api): accept max-keys up to 2147483647 and cap list pages at 1000 * add tests for ListObjects max-keys range and 1000 cap * refactor(s3api): return uint16 from parseMaxKeys to avoid narrowing casts |
||
|
|
fac021544e |
s3api: register the azurekms KMS provider under its build tag (#11691)
* s3api: register the azurekms KMS provider under its build tag weed/kms/azure registers itself with weed/kms from an init(), but the blank import that triggers it was commented out in auth_credentials.go (TODO: Fix Azure SDK compatibility issues). Nothing else imports the package, so building with -tags azurekms still ends with "KMS provider 'azure' not registered" at startup for any config using "type": "azure". Move the import into weed/s3api/kms_azure.go behind //go:build azurekms so default builds stay free of the Azure SDK, and drop the dead comment. Closes #11686 * ci: compile and test azurekms-gated code weed/kms/azure is excluded from every default build, so nothing in CI compiled it; that is how the provider shipped unregistered. Build the tree and run the kms tests with -tags azurekms on every Go change. --------- Co-authored-by: Yi-111-a <> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
448de42b05 |
worker: reject balance moves onto servers without the volume's disk type (#11695)
* worker: reject balance moves onto servers without the volume's disk type Proposals are bucketed by disk type at detection time, but queued or out-of-band jobs can still target a server that lacks the volume's disk. The copy then reads the source disk only to be rejected by the target's VolumeCopy, and the failure is retried on every scheduling cycle. Fetch the master's topology once per job and check each move's target against the volume's disk type before executing, so an incompatible move fails before any data is copied. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * worker: cover the balance disk-type guard Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * worker: match old-topology nodes in the balance disk-type guard Topology entries from older masters omit Address and carry a plain host:port Id, so checkMoveDiskType never matched grpc-suffixed move nodes and silently skipped the disk-type check. Also compare each node via pb.NewServerAddressFromDataNode, which reconstructs the same host:port.grpcPort form from Id and GrpcPort. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
d050464316 |
s3api: apply bucket default encryption when volume data encryption is enabled (#11681)
* s3api: apply bucket default encryption when volume data encryption is enabled When -s3.encryptVolumeData (s3a.cipher) is enabled, putToFiler skipped checking and applying bucket default encryption due to a '!s3a.cipher' guard. Volume-level data encryption and object-level Server-Side Encryption (SSE-S3 / SSE-KMS) operate at different layers, and explicit SSE headers already work alongside volume encryption. Remove the '!s3a.cipher' guard so PutObject without explicit SSE headers inherits bucket default encryption regardless of volume data encryption. Add regression test TestPutObjectAppliesBucketDefaultEncryptionWithVolumeCipher. Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com> * s3api: decrypt the volume cipher on direct SSE chunk reads fetchFullChunk, fetchChunkViewData, and createEncryptedChunkReader fetched raw bytes over HTTP, so volume-encrypted chunks reached SSE-S3/KMS/C decryptors still ciphered. Route them through fetchChunkData: encrypted or compressed chunks go through RetriedFetchChunkData (cipher-aware, slices plaintext space for views); plain chunks keep the streaming range read. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: cover cipher-aware chunk reads with a fake volume server The fake volume now serves stored GETs with Range support, and TestFetchChunkDataDecryptsVolumeCipher verifies full-chunk and view reads return plaintext for ciphered chunks while plain chunks still slice via HTTP ranges. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: guard fake volume server stored map with its mutex The HTTP handler goroutine read v.stored while test goroutines wrote it, a data race go test -race can flag. Lock v.mu around the map read and the test writes. --------- Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
c96eb7b8ee |
kms/azure: fix encrypt/decrypt round trip for Azure Key Vault (#11692)
* kms/azure: fix encrypt/decrypt round trip for Azure Key Vault The provider stored the wrapped data key as string(encryptResult.Result) in the JSON envelope, sent the encryption context as AAD to an RSA-OAEP key, and passed a full key URL to the azkeys client in the name position. Each of those breaks a round trip on its own. - split the key URL into the (name, version) pair azkeys expects before Encrypt, Decrypt and GetKey; a plain name still resolves to the latest version, and decrypt keeps the stored version so objects stay readable after a key rotation - store the wrapped key base64-encoded and decode it strictly, so the JSON envelope no longer replaces the raw bytes with U+FFFD - stop sending AAD: Key Vault rejects it on RSA-OAEP with BadParameter, so the encryption context is logged and dropped instead * kms/azure: reject a key URL that names another vault splitKeyID dropped the host of a full Key Vault URL, so a key ID from a different vault silently resolved to this vault's same-named key. Return an error instead of encrypting under a key the caller did not ask for. * kms/azure: bind encryption context via an envelope digest RSA-OAEP rejects AAD, so removing it left the encryption context unauthenticated: a wrapped key could be decrypted under a different object's context. Record a sha256 digest of the marshaled context in the envelope's provider_specific field on encrypt and verify it before calling Decrypt, restoring the binding without AAD. * kms/azure: treat an explicit :443 port as the same vault * kms/azure: accept an absent context digest only for empty contexts * kms/azure: normalize both hosts when comparing key URLs to the vault splitKeyID stripped :443 and a trailing dot from the configured vault but only :443 from the key URL, so a key URL naming the same vault with a trailing DNS dot was rejected before Azure was ever called. Compare both sides through vaultHost so they are normalized identically. * kms/azure: reject non-key vault URLs in splitKeyID, document digest limits A Key Vault URL that does not name a key under /keys/, has an empty key name, or carries extra path segments now fails fast instead of being passed to the client as a key name, where it would surface as a confusing vault-side error. Also note that the envelope context digest is a client-side mismatch check, not vault-authenticated AAD, and only allocate providerSpecific when a context is present. * ci: compile and test azurekms-gated code weed/kms/azure is excluded from every default build, so nothing in CI compiled it; that is how the provider shipped unregistered. Build the tree and run the kms tests with -tags azurekms on every Go change. --------- Co-authored-by: Yi-111-a <> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
b2eefb8a73 |
filer: fail the request, not the process, when chunks cannot be resolved (#11685)
* filer: fail the request, not the process, when chunks cannot be resolved CompactFileChunks and ViewFromChunks discarded the error from NonOverlappingVisibleIntervals and iterated the nil interval list it returns on failure. A CreateEntry or UpdateEntry whose context was cancelled therefore panicked in SeparateGarbageChunks and took the whole filer down, losing the metadata log it had not yet flushed. Both functions now return the error. cleanupChunks fails the request with it, the mount flush paths return it, the replication sinks and the S3 SSE range reader return it, and a ChunkStreamReader that cannot build its view fails every read with it and reports it as its SourceError instead of reading as an empty file. Fixes #11682 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * filer: propagate chunk resolution failures in scan paths NonOverlappingVisibleIntervals can fail without a cancelled context; the query engine and log-store callers discarded the error and dereferenced nil intervals. * query: propagate chunk resolution errors instead of skipping files When a manifest cannot be resolved, ReadParquetStatistics, countLiveLogRowsExcludingParquetSources, and computeLiveLogMinMax logged a warning and skipped the failed file, so queries returned understated counts and wrong min/max values. Surface the errors instead: the fast-path aggregation collector now returns a DataSourceError so the engine falls back to a full scan, and the row-count helpers return the error rather than a partial total. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Bharat Kaurav
|
774c9927a7 |
s3api: validate ListObjectVersions max-keys and honor max-keys=0 (#11703)
* fix(s3api): accept max-keys up to 2147483647 and cap list pages at 1000 * add tests for ListObjects max-keys range and 1000 cap * refactor(s3api): return uint16 from parseMaxKeys to avoid narrowing casts * fix(s3api): validate ListObjectVersions max-keys and honor max-keys=0 * add test for ListObjectVersions max-keys=0 * fix(s3api): widen uint16 max-keys for ListObjectVersions |
||
|
Chris LuandDevin
|
c3d13c60e9 |
filer: fail writes when the existing-entry lookup fails (#11699)
* filer: fail writes when the existing-entry lookup fails CreateEntry only returned a FindEntry error for exclusive creates; a regular write treated a transient store error as "entry not found" and, on upsert stores, replaced the entry without loading its old chunks — leaving them out of the cleanup process. Return any lookup error except ErrNotFound for both paths. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: test that a lookup failure fails a regular write Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
4bf5e92b64 |
storage: unify needle-map counters across index loaders and warn on copy-count drift (#11701)
* storage: count index-load metrics only for live-key removals The in-memory and leveldb offset loaders counted a deletion for every tombstone row replayed, including re-deletes of already-deleted keys (runtime logDelete only counts a live removal), and added the returned old size to the byte counters even when it was a tombstone sentinel — uint64(-1) wraps DeletionByteCounter. The leveldb offset loader also counted every index row as a file and its bytes unconditionally. Gate the counters the same way the runtime paths do so a reload reports the same numbers the runtime counters hold. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * storage: derive index metrics from latest-row state in the metric loader The leveldb metric loader walked the index in reverse and incremented FileCounter for every row (inflating FileCount with tombstones) and DeletionCounter for every superseded row plus every deleted-key row (double counting both). Per key, the runtime deleted count is exactly (valid rows) - (1 when the latest row is live), so count live-latest rows via the same bloom filter and derive both deletion counters; this matches the runtime logPut/logDelete counters exactly instead of drifting with tombstone history. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * server: warn instead of failing volume copies on counter drift checkCopyCounts compared the source's in-memory counters against the target's fresh-load counters, but counters computed by different loaders drifted apart for identical .idx data, so a good copy was rejected (issue #11678). The file-size checks already gate integrity; log the mismatch instead of failing the copy. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * storage: assert MaxFileKey parity in the metric loader test MaxFileKey derives from row order alone, so unlike the bloom-filtered deletion counters it must always match the runtime counters exactly. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Peter Dodd
|
5f1a742726 |
fix: return NoSuchKey and drop stale entries when a remote-mounted object is gone from the remote (#11680)
A remote-only entry whose object was deleted from the remote storage outside the filer answered GET with 500 and stayed in the filer. The remote's not-found was lost on the way: the backends' ReadFile returned it as an untyped error, so FetchAndWriteNeedle failed with codes.Unknown and nothing downstream could tell it from any other failure. - remote_storage: GCS, S3 (NoSuchKey) and Azure (BlobNotFound) reads return ErrRemoteObjectNotFound. GCS reports a missing bucket the same way as a missing object, so it confirms the bucket with a listing. - volume server and filer: the not-found crosses gRPC as codes.NotFound carrying the sentinel's text, and the filer's cache RPC returns codes.NotFound, which the S3 gateway already maps to NoSuchKey. - s3api: the origin fallback answers NoSuchKey on a confirmed not-found. - filer: a confirmed not-found removes the stale entry, so the lazy remote-metadata cache converges on the remote. Only remote-only files outside .versions and without an active object lock are removed, only if unchanged since the fetch (checked on the object's write owner, under the lock S3 object writes take), and with a metadata-only delete: the filer skips its inline remote delete, and the delete events' entries carry a marker that makes filer.remote.sync and filer.remote.gateway skip their remote delete, while filer.sync still replicates it. The replicated DeleteEntryRequest carries keep_remote_object, so the destination's delete events are marked too. The store drops the marker from every write, so clients cannot plant it. |
||
|
|
d3dc03c85a |
s3api: resolve volume data encryption in CopyObject SSE flows (#11646) (#11683)
* s3api: resolve volume data encryption in CopyObject SSE flows (#11646) * s3api: honor bucket-default KMS key on copy and fix transformed-upload metadata - Synthesize the destination bucket's default encryption as request headers before any SSE evaluation, so the configured KMS key ID and bucket-key setting reach the copy paths instead of only a boolean. - uploadTransformedChunkData returns the upload result so callers record the uploader's cipher key AND compression decision; a wrongly cleared IsCompressed made transformed copies unreadable. - decompressChunkVolumeCipher fails loudly when a compressed chunk does not decompress, instead of uploading still-compressed bytes marked uncompressed. - copyMultipartSSECChunk now strips the volume cipher and re-encrypts on upload like the other transform paths. - The ciphered inner upload now uses the caller's private BytesBuffer. * s3api: extract copy bucket-default header synthesis for coverage Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: test bucket-default encryption header synthesis on copy Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: validate copy encryption headers before bucket defaults and resolve empty KMS key * s3api: name the AWS-managed SSE-KMS default key once --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
ea65746947 |
fix(filer): end metadata subscriptions before gRPC GracefulStop on shutdown (#11663)
* fix(filer): end metadata subscriptions before gRPC GracefulStop on shutdown GracefulStop waits for every open stream. The filer's own MetaAggregator subscription and, under -s3, the S3 gateway's IAM subscription live in the same process and only end when the filer shuts down, which happens after GracefulStop - so every shutdown sat out the full 15s timeout. StopSubscriptions ends them (and any started later) right after leaving the lock ring; the handlers derive their context from the stream and this signal. Measured on weed server -s3: stop 25.6s -> 10.7s, and 0.6s with -volume.preStopSeconds=0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FTq6bDpgdQfqQagQsUUvdw * fix(filer): end stopped subscriptions with Unavailable, not cleanly A subscription that StopSubscriptions cut short returned nil. The client reads that as io.EOF, "caught up, done", and util.RetryUntil - which the S3 gateway and mount follow with - stops on nil. A separate S3 gateway therefore never resubscribed after its filer restarted (measured: 0 subscriptions in 25s after the restart; upstream master and this fix: it is back within a second). endOfSubscription now turns that clean end into codes.Unavailable, which is what a dropped connection looks like, so followers reconnect. Errors pass through, and so does the end of a stream the client closed. A subscriber that has stopped reading is still left to GracefulStop's timeout: its handler sits in stream.Send, which only the end of the stream releases, and grpc-go does not allow Send after the handler returns. Documented at StopSubscriptions. Shutdown under weed server -s3 is unchanged at 10.7s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FTq6bDpgdQfqQagQsUUvdw * fix(filer): interrupt in-flight subscription reads and end them Unavailable StopSubscriptions only reached the wait points: a handler replaying the ring or the persisted log kept streaming until the pass ended, and a pass cut short at a ctx.Err() boundary (the replay semaphore, chunk ref collection, ref sends) returned a wrapped context.Canceled, which reads as codes.Canceled on the wire - a code transient-error classifiers treat as non-retryable. eachLogEntryFn and sendRefsBatched now check the subscription context per entry/batch, and endOfSubscription reports a context.Canceled cut short by the stop signal as codes.Unavailable, same as a clean end it interrupted. * filer: honor the subscription context in the pipelined sender A subscriber that stops reading wedges sendLoop in stream.Send; Send and Close then blocked past StopSubscriptions, so GracefulStop still waited out its timeout. Both now return when the subscription context ends, letting the handler return and gRPC tear the stream down. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: stop sendLoop from starting sends after the subscription ends With Send and Close released by the subscription context, sendLoop could still issue a stream.Send after the handler returned. Gate every send on the context so at most the in-flight Send overlaps teardown - and the stream's own teardown is what unblocks it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: assert sendLoop exits in the cancel test Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
c023493cf0 |
s3api: tolerate session tokens on statically configured credentials (#11694)
* s3api: consolidate session token extraction into extractSessionToken Three call sites duplicated the same header/header/query lookup; share one helper named after the existing s3tables equivalent. * s3api: tolerate session tokens on statically configured credentials Credential vendors like Unity Catalog emit a session token with every vended credential, including static ones (UC's StaticAwsCredentialGenerator only engages when s3.sessionToken is set). Requests signed by a configured access key were routed to STS validation and rejected, so static credential vending never worked against SeaweedFS. Resolve the access key first: when it maps to a configured credential the signature alone authenticates the request, and the attached token is marked ignored so authorization does not route it into the STS session-policy path. STS-issued access keys are never in the static map, so temporary credentials still validate their token exactly as before. * s3api: cover static credentials carrying a session token * test: exercise UC static credential vending against SeaweedFS s3.sessionToken.0 selects UC's StaticAwsCredentialGenerator, which vends the configured keys verbatim. The vended session token is foreign to SeaweedFS and previously failed SigV4; now it round-trips through temporary-table-credentials into real S3 I/O. * s3api: reject temporary credentials in GetFederationToken after auth Token presence alone cannot distinguish a temporary credential from a statically configured one carrying a vended token. Move the check behind verifyV4Signature and key it on the operative session token so tolerated tokens keep the caller eligible. * s3api: test GetFederationToken with authenticated temporary credentials Sign the rejection cases with real session credentials so they reach the post-auth check, and cover a vended static credential being accepted. * test: check vended-credential delete error in UC integration test |
||
|
Eliah RusinandChris Lu
|
924a683397 |
volume server: answer chunk-manifest reads from the sum of chunk sizes (#11628)
* volume: answer chunk-manifest HEAD from the chunk-size sum * volume: size chunk-manifest reads by the sum of chunk sizes * volume: saturate the sum of chunk-manifest Range lengths * volume: fetch remote chunk-manifest ranges with a bounded Range * volume: fmt chunk-manifest handlers * volume: cap whole-window chunk reads at the visible window * volume: reject an over-cap chunk span and an illegal manifest MIME * volume: take the manifest transform extension from the URL first Go's tryHandleChunkedFile keeps the extension parsed from the URL — including a fid suffix like /vid,fid.png — and only falls back to the resolved filename's extension when the URL has none. The manifest path now does the same, so a transform requested on a fid-suffixed URL runs as it does in Go. Also: compute the chunk-size sum once, drop an unused remote_read_for parameter, share the crop/resize predicate with parse_read_request, and settle a couple of clippy lints. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Paolo Valletta
|
4d306e1279 |
s3api: revoke the identities a config reload no longer declares (#11679)
* s3api: revoke the identities a config reload no longer declares A static config file reload merges with isFullState=false, so an identity deleted from -s3.config kept authenticating with all of its access keys until the process restarted. An operator revoking a leaked key got no error and no revocation. The file is the source of truth for the identities it declares, so a reload now drops the ones that have left it, together with their access keys and those of their service accounts. staticIdentityNames was only ever added to, which kept a removed name protected as well; the names the file itself declares are tracked separately from the AWS environment credentials, so a reload never revokes what the file never declared. * s3api: keep a file reload from replacing the dynamic store Removing the last identity from a config file emptied staticIdentityNames, so useStaticConfig turned false and the next reload of that file took the replace path: every filer-managed identity and its access keys disappeared until a dynamic reload brought them back. A static config file is authoritative for the identities it declares and never for the dynamic store, so a file load now always merges. The first file load at startup takes the merge path as well, from an empty state. Reported by the Devin and Greptile reviews of this PR. TestReloadStaticConfigWithoutIdentitiesKeepsDynamic reloads an emptied file twice and asserts that a filer-managed identity and its access key survive both; the existing test now also checks that the service account key was loaded before the reload and that the environment identity's key still works after it. * s3api: clear the environment credentials in the emptied-file test The AWS environment identity is static and is re-added after every merge, so on a runner that has AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set it kept hasStaticConfig true once the file was emptied, and the next reload still took the merge path: the regression this test guards passed unnoticed. Clear both variables whatever the runner has set. With the pre-fix condition (hasStaticConfig alone) and the variables present in the environment, the test now fails with "reload 2: a filer-managed identity must survive a reload of an emptied file". Reported by the Greptile review of this PR. |
||
|
github-actions[bot]
|
c4f2f514fa | docs: regenerate star history chart | ||
|
Chris LuandDevin
|
5d80ac39b8 |
s3api: verify under the write lock before re-committing an ambiguous routed PUT (#11649)
* s3api: verify under the write lock before re-committing an ambiguous routed PUT A routed PUT whose response was lost after the owner committed, or whose transaction returned a store error, fell straight into the lock path and re-sent the same entry. A concurrent PUT that had superseded the commit in the meantime had already deleted this entry's chunks as old, so the re-commit restored metadata pointing at dead needles and the object read back 404 permanently. The lock path now resolves the routed attempt's outcome inside the write lock first: a stored entry with the uploaded chunks means the route landed; a different stored entry or an unresolved lookup refuses the re-commit with ServiceUnavailable so the client retries with a fresh upload; only a proven-absent entry falls through to the normal create. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: tighten ambiguous routed PUT recovery - a match found only after an unanswered filer is not authoritative; the unreachable filer may hold a newer entry, so refuse instead of finalizing on it (both recovery paths) - a failed post-recovery finalization now rolls the recovered entry back, matching the create path's undo - a proven-absent entry is only re-committed after probing that the uploaded chunks' needles are still alive; a committed-and-deleted PUT would otherwise write back an entry pointing at reclaimed needles - the .versions latest pointer no longer flips back to an older version when a newer one committed while the write was uncertain Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: stamp late versions noncurrent when a newer latest pointer wins The keep-newer early return in updateLatestVersionInDirectory left the version just stored without ExtNoncurrentSinceNsKey, so the lifecycle engine could never age it out. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: only a failure past mutation 0 makes a routed PUT ambiguous A deterministic refusal at the PUT mutation (e.g. "existing entry is a directory") applied nothing, but marking it ambiguous sent the lock-path fallback through conservative recovery, which found the directory entry and refused with 503 instead of the correct 409. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: keep all transaction errors ambiguous; route directory conflicts through the lock path * s3: verify uploaded chunks before re-committing over a directory A stored directory does not prove the routed PUT never committed: the route can land, a delete can remove the entry and its chunks, and a nested write can recreate the directory before recovery takes the lock. Re-committing then stores an entry pointing at dead needles. Probe the uploaded chunks first and refuse with ServiceUnavailable when they can no longer be verified. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
e337443176 |
filer: cut ReaderCache mutex contention on small reads (#11677)
* filer: skip cache lock when a cacher cannot be removed yet SingleChunkCacher.readChunkAt defers removeConsumed on every read, and every unpin retries removal too. Each call acquired the ReaderCache mutex even though the removal conditions are plain atomics, so a busy cache paid a lock acquisition per read just to discover readers>0. Check the atomics before locking: when a cacher is not yet consumable the removal is impossible and the lock round trip is pure contention. Removals still run under the lock via removeConsumedLocked, so the attach-vs-remove race keeps its existing serialization. Ref #11676 * filer: guard stream position with a per-stream mutex, not the cache lock chunkStream.cacher is only ever shared by concurrent ReadAt calls on one ChunkReadAt, yet pin, unpin, releaseIfFinished, and releaseStream all mutated it under the ReaderCache mutex that serializes every reader in the process. Each read that switched chunks took that global lock to pin the new chunk, released it, then reacquired it in unpin() just to drop the old chunk's pin and retry removal; releaseIfFinished and releaseStream did the same lock-detach-unlock-relock dance. At high small-GET concurrency that turned per-request stream bookkeeping into a global convoy (#11676). Give chunkStream its own mutex and drop the cache lock from the pin lifecycle entirely: pin/detach serialize on stream.mu, the pin counter and consumable checks are already atomics, and removeConsumed only takes the cache lock when a cacher is actually removable. The map lock now guards only map membership and read registration. * filer: look up cached chunks under the read lock readChunkAt serialized every small read on the write lock even though the common paths are read-only: an existing downloader just needs its read registered, and a chunkCache hit needs no map access at all. Each GET also paid the lock a second time to reach the chunkCache check. Use the read lock for the downloader lookup and registration; the registered read keeps the buffer alive against a concurrent destroy, and only error eviction, insertion, and removal need the write lock. The chunkCache probe runs lock-free, and the insert path re-checks the map under the write lock to cover a downloader registered in between. * filer: start the chunk download outside the map lock The insert path held the ReaderCache write lock across goroutine spawn and the cacheStartedCh handshake, so every downloader miss serialized against the startup of a fetch goroutine. Register the cacher in the map under the lock, then start the download after releasing it; a fetch that fails early still lands in the map and is evicted by the next reader's completed-error check. * filer: test that stream pin lifecycle stays off the cache lock Regression coverage for the contention fix: pin and releaseStream on a non-removable cacher must complete while the ReaderCache lock is held by another goroutine. * filer: pin the stream under the map lock Between read registration and stream.pin the cacher showed zero pins, so a budget eviction in that gap could pick a chunk the stream was just attaching to and the stream's next slice refetched it. The pin counter is an atomic and stream.mu is never held while acquiring the cache lock, so pinning inside the map hold is deadlock-free and closes the window. |
||
|
|
838c554e33 |
s3api: reject SSE headers PutObject cannot honor, as CopyObject does (#11642)
* s3api: reject SSE headers PutObject cannot honor, as CopyObject does PutObject and CreateMultipartUpload did not check the server-side encryption headers they were given. An x-amz-server-side-encryption value that names no method, such as "aes:kms", matched none of the SSE paths, so the object was stored unencrypted and the request answered 200. SSE-C together with x-amz-server-side-encryption was also accepted, and one of the two silently won. CopyObject already rejects both through validateEncryptionCompatibility. Run the same check, with the same error codes, before PutObject and CreateMultipartUpload store anything. * s3api: answer rejected SSE headers with InvalidArgument, as S3 does S3 rejects an unknown x-amz-server-side-encryption value and SSE-C combined with another method with InvalidArgument. PutObject and CreateMultipartUpload now return that code, with S3's messages, through two new error codes; CopyObject keeps its own. Also run ceph/s3-tests' test_put_obj_enc_conflict_c_s3, _c_kms and _bad_enc_kms in CI, which check both handlers' responses end to end. * s3api: close the remaining ways a PUT could skip requested encryption - A repeated x-amz-server-side-encryption header is rejected: the encryption paths apply only the first value, so extra values could hide the method the client asked for. - KMS options (key id, encryption context, bucket key) are rejected unless the method is aws:kms, matching the S3 InvalidArgument error. - CreateMultipartUpload validates before auto-create, so a refused upload cannot leave a bucket behind. - Directory markers encrypt their inline content through the shared SSE path and record the same entry metadata as regular objects, instead of storing requested-encrypted bytes in plaintext. * s3api: read back what the SSE marker write stores - Repeated SSE-C and KMS option headers are rejected alongside a repeated x-amz-server-side-encryption, closing the same first-value bypass for customer-key and KMS fields. - Algorithm validity is checked before KMS options so an unsupported value keeps the "not supported" error. - serveDirectoryContent decrypts marker content with the stored SSE metadata and returns the SSE headers, so an encrypted marker reads back what was written; its Content-Length now reflects the bytes actually served. * s3api: HEAD of a marker skips decryption and keeps the stored size HEAD returns no body, so it now runs only the SSE-C key check instead of decrypting — matching HeadObjectHandler and avoiding a KMS round-trip — and chunk-backed directory entries report Attributes.FileSize again rather than the length of their (empty) inline content. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: keep GET Content-Length to the bytes serveDirectoryContent writes The FileSize override described chunk-backed markers on HEAD, but GET sends only the inline content, so it promised bytes it never wrote. * s3api: stream a chunk-backed directory on GET like any object A directory promoted over an uploaded object keeps the object chunks with empty inline content, so serving only entry.Content made GET deliver nothing while HEAD reported FileSize. GET now routes those entries through the regular volume-server stream, keeping the two methods consistent. * s3api: answer a busy volume read with RequestBytesExceed --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Younsung Lee
|
68af030e38 |
helm: add Gateway API HTTPRoute for master, volume, filer, s3 and admin (#11625)
* helm: add Gateway API HTTPRoute for master, volume, filer, s3 and admin Each component that offers an Ingress can now be exposed through a Gateway API HTTPRoute instead, via <component>.httpRoute. It is disabled by default, so existing renders are unchanged. parentRefs, hostnames, labels and annotations pass through as given. Each rule may set matches, filters, timeouts and backendRefs, and a rule without backendRefs routes to the component's own service and port, following all-in-one mode the same way the Ingress templates do. An empty rules list yields a single rule sending all traffic there. Signed-off-by: younsl <cysl@kakao.com> * helm: fix HTTPRoute backend selection for S3 on filer and all-in-one - s3: with S3 on the filer the route targets filer.s3.port, the port the s3 Service exposes. The all-in-one Service is chosen only when allInOne.s3 is enabled, so a standalone S3 next to all-in-one routes to the s3 Service. - master: render in all-in-one mode and route to the all-in-one Service, like the volume and filer routes. - Name routes with seaweedfs.componentName, keeping them within 63 characters like the Services they point to. - CI: cover the filer S3 port and the all-in-one master route. Signed-off-by: younsl <cysl@kakao.com> * helm: document the Gateway API HTTPRoute values in the chart README Signed-off-by: younsl <cysl@kakao.com> * helm: link the Gateway API docs from the chart README Signed-off-by: younsl <cysl@kakao.com> * helm: scope the filer S3 port note in the chart README Signed-off-by: younsl <cysl@kakao.com> --------- Signed-off-by: younsl <cysl@kakao.com> |
||
|
Chris LuandDevin
|
e3fadc6e04 |
filer: keep the proxy-JWT test from hanging on a failed fetch (#11664)
The handler sends the header non-blocking and the test stops on fetch errors instead of waiting on a channel that may never be fed. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
440249a270 |
test: stop the compat server even when startup fails (#11659)
* test: stop the compat server even when startup fails test-with-server depended on start-server as a prerequisite, so a failed startup skipped the recipe and the spawned weed mini kept the port. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: verify the recorded PID is weed before stopping it A prerequisite failure leaves a stale weed-server.pid in place; the PID may have been reused by an unrelated process. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
9a9bd67efe |
s3api: do not re-run permission checks in the aws-chunked reader (#11655)
calculateSeedSignature called verifyV4Signature with shouldCheckPermissions=true, which runs VerifyActionPermission — a check that does not consult bucket policies. Every caller of newChunkedReader is already behind the Auth middleware, which does evaluate them, so a principal allowed only by the bucket policy was authorized upstream and then denied when the handler built the body reader: aws-chunked PutObject/UploadPart (botocore's default shape over TLS, PyArrow's over HTTP as well) failed with AccessDenied. The reader now verifies only the signature; a wrong secret is still SignatureDoesNotMatch. The non-streaming paths already work this way. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
f206d021f1 |
iceberg maintenance: write position-delete files with spec field ids and dictionary paths (#11652)
rewrite_position_delete_files emitted a schema-less parquet file whose file_path column was DELTA_LENGTH_BYTE_ARRAY. PyIceberg reads delete files with file_path as a dictionary column, which PyArrow cannot decode from that encoding, so every table the worker rewrote failed scans outright; readers that resolve columns by field id found none. The row struct now declares the spec's reserved field ids (2147483546 file_path, 2147483545 pos) and dictionary-encodes file_path, which is also the compact choice for a column repeating one data file's path. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
4111aa4dd8 |
iceberg maintenance: keep statistics files the table metadata references (#11651)
Orphan cleanup built its referenced set from snapshot manifests plus the current and previous metadata files, then walked all of metadata/ and data/. Table statistics (Puffin) and partition statistics files are listed in the table metadata's statistics and partition-statistics lists, not in any snapshot, so once they passed the safety window remove_orphans deleted them while the metadata still pointed at them. Mark the files in both lists referenced as well. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
9a365549c3 |
catalog: refused writes answer 403 to callers that can read the entry (#11650)
* s3tables: answer a denied write with 403 when the caller can read the entry Update/Delete/Rename answered every authorization refusal as not-found so the denial leaked no existence signal. For a caller allowed to GetTable (or GetView) the same entry, the veil hides nothing it could not load — yet a refused write was still answered 404, so clients saw a table they just loaded reported as missing. When the write check fails, re-check read permission on the same entry: readable entries get 403 AccessDenied; invisible ones keep the not-found answer. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg: map UpdateTable errors through writeManagerError on commit paths CommitTable and CommitTransaction answered any non-conflict UpdateTable failure, including AccessDenied and NoSuchTable, with a bare 500. A 500 reads as outcome-unknown to clients (PyIceberg raises CommitStateUnknownException) where a refused commit is a plain ForbiddenException, matching what the drop and rename handlers already emit through the same mapper. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: keep the not-found veil over views and match the real read check on renames UpdateTable and DeleteTable read shared metadata without checking the entry kind, so a denied write on a view reported 403 to a GetTable-only caller where a missing name reports 404. The rename visibility check also fed resource tags into GetView evaluation that the real GetView path never supplies. * s3tables: gofmt handler_table.go Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
37465237a9 |
build(deps): bump rustls from 0.23.43 to 0.23.45 in /seaweed-common (#11675)
Bumps [rustls](https://github.com/rustls/rustls) from 0.23.43 to 0.23.45. - [Release notes](https://github.com/rustls/rustls/releases) - [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md) - [Commits](https://github.com/rustls/rustls/compare/v/0.23.43...v/0.23.45) --- updated-dependencies: - dependency-name: rustls dependency-version: 0.23.45 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
0f01610403 | docs: regenerate star history chart | ||
|
|
1df165d514 |
fix(volume): keep the TTL clock across a vacuum commit instead of rescanning (#11630)
* fix(volume): keep the TTL clock across a vacuum commit instead of rescanning CommitCompact reloads the swapped files while holding dataFileAccessLock, and for a vacuumed TTL volume that reload re-derived lastModifiedTsSeconds by reading every live needle's append timestamp from the .dat: two random reads per needle, with every read of the volume blocked behind them. The in-memory clock is already current at that point. Every write since the volume loaded moved it, and makeupDiff only replays writes that went through that path. Carry it across the reload instead. This also stops an over-budget scan from falling back to the new .dat's mtime and restarting an expiring volume's TTL at the commit. * volume: carry the append watermark as the TTL clock across a vacuum commit The running append watermark is the clock the reload's recovery scan recomputes, so the commit can keep it directly. Client-supplied needle modified times can run ahead of or behind the append time; keeping lastModifiedTsSeconds itself would let a forged or stale timestamp move expiry through a vacuum, where the scan it replaces used server-side append timestamps. * storage: test that a vacuum commit keeps the append clock A write's client supplied modified time can lie ahead of or behind its append time; the commit must land the TTL clock on the append watermark, the same value the recovery scan would have recomputed. * volume: carry the append watermark as the TTL clock across a vacuum commit Mirrors the Go volume server: the running append watermark is the clock the reload's recovery scan recomputes, so the commit keeps it instead of rescanning live needles under the write lock. * volume: commit carries the last-write append time, not the latest append lastAppendAtNs counts tombstone appends and is reseeded from the .dat tail at every load, so it can sit ahead of the last write -- a delete freshens the commit clock -- or behind it: a restarted vacuumed volume's tail needle is not its newest write, and the commit would move the TTL clock backward into premature expiry. Track lastWriteAppendAtNs instead, bumped only on needle appends and seeded by the recovery scan, so the commit lands the clock on the same live-write maximum the rescan would have recomputed. * volume: rescan at commit when the newest write was deleted lastWriteAppendAtNs can hold a write the index no longer holds, so carrying it extends the TTL clock past what recovery over the compacted index would compute. Remember the key behind the watermark so its tombstone or index rollback can send the reload back through recoverLastModifiedTs, landing on the newest surviving write. * volume: a tombstone retires the write rows beneath it in the last-write scan A needle deleted after the compaction copy leaves its write row followed by a tombstone in the committed index. The reverse scan skipped the tombstone row but then counted the dead write, reseeding the watermark and flag as if it were alive. Track keys whose latest row is a tombstone so their earlier write rows stop counting, and exercise the delete-inside-the-commit-window ordering in the tests. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
0305e837fd |
iceberg maintenance: keep compacted files prunable (stats, bound order, row groups) (#11654)
* iceberg maintenance: record column statistics on compacted files A compacted file's manifest entry was built with no column_sizes, value_counts, null_value_counts, lower_bounds, upper_bounds or split_offsets, so no reader could skip a compacted file on any predicate. parquet-go already writes exact per-chunk min/max and null counts into the footer; read that footer back after the merge and record it on the data file, bounds as the spec's single-value serialization with string and binary truncated as truncate(16). A column whose bounds cannot be converted exactly gets none, and a statistics failure is logged while the compaction commits anyway: metrics are an optimization, not a correctness requirement. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: merge bins in bound order so compacted files stay prunable A bin's files were concatenated in manifest order, or largest-first when a partition was split under the target size, so inputs disjoint on a column came out as outputs that overlapped on it. Order each bin's files by their bounds on one column before merging and split an oversized partition into runs of consecutive files, so every output covers one contiguous range. The column is the first identity field of the table's sort order when it declares one (a descending order sorts by upper bound), otherwise the first schema column every candidate file has bounds for; detection resolves the same order so it plans the bins execution builds. Files without bounds keep the old behavior. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: cap compacted files' row groups from table config Neither merge writer set a row-group limit (parquet-go's default is unlimited rows), so every compacted file was a single row group and readers could not skip inside it either. Rows per row group now come from the table's write.parquet.row-group-limit and write.parquet.row-group-size-bytes, defaulting to PyIceberg's 1 048 576 rows and Iceberg's 128 MiB, with the byte size turned into rows from the bin's inputs' compressed bytes per row and a floor of 1 024 rows. Both writers take the cap, and the statistics the entry records list one split offset per row group. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg: keep compaction order eligibility per group, rescue stranded runs The merge order resolved over all candidates, so one oversized or non-Parquet file without bounds disabled ordering for files that could participate. Resolve it per partition group over the eligible entries. Ordered runs too short to merge were dropped entirely. Runs from an inferred bounds order now fall back to size-based packing — ordering is a preference there — while runs under a declared sort order are still left for later passes so the sort contract holds. An explicit write.parquet.row-group-limit is a cap, not a floor: values below the estimate floor are now honored instead of being raised. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg: only use the declared sort order when its bounds are complete An entry without bounds on the sort column sorted to the tail and merged into an output claiming an order it cannot verify. Fall back to bound inference instead of ordering by a later sort field alone. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: keep ordered runs when the full repack yields nothing The bestEffort leftover fallback removed the ordered runs before checking whether repacking the whole bin produced any bins, discarding valid compaction work. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
8f80dac30f |
ec: strict_placement option so encode only runs while guarantees hold (#11656)
* ec: strict_placement option so encode only runs while guarantees hold Shard placement during encode was best-effort (PlaceDurabilityFirst): when the cluster could not satisfy the per-disk caps, anti-affinity, replica-placement or per-rack caps, the constraints were relaxed and the volume was encoded anyway, weaker than configured. A strict_placement option on the erasure coding task switches planning to PlaceStrict so the volume's planning fails instead, and the encode is retried when capacity allows the guarantee. Also documents the resilience rule in ec.encode help: a volume survives losing any nodes or racks holding at most parity-shards shards between them, and how -shardReplicaPlacement's rack and node digits bound that loss. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec: expose strict_placement through the plugin form and persisted task policy The admin UI, the admin.toml maintenance mapping, and the TaskPolicy serialization all dropped the new flag; add the bool field to ErasureCodingTaskConfig, the worker config form, and both conversion directions. * shell: describe shardReplicaPlacement as requested limits, not guarantees ec.encode places shards best-effort, so the configured rack/node caps only bound shard loss when the final placement actually satisfies them. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
30cf53265b |
deps: pin seaweedfs/goexif at v1.0.3 and stop dependabot re-bumping it (#11648)
* deps: pin github.com/seaweedfs/goexif back to v1.0.3 The fork's newest tagged release is v1.0.3; the v2.0.0+incompatible requirement resolved to older, untagged code and breaks isolated builds that fetch it from the proxy. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: stop dependabot bumping seaweedfs/goexif past its latest tag Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test/kafka: settle goexif at v1.0.3 too The kafka test module recorded v2.0.0+incompatible in its own requires, so MVS kept selecting it over the pinned v1.0.3 in the root module. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
4d1f49c638 |
filer.sync: sign proxied chunk I/O from the per-side security file (#11645)
* filer.sync: sign proxied chunk I/O from the per-side security file The -a.security / -b.security files were used for gRPC TLS and the HTTPS client but not for jwt.filer_signing, so filer-proxied chunk reads and writes carried a token signed with the process-wide key and failed authorization whenever the two clusters' keys differ. LoadFilerJwtFromFile returns a FilerJwtProvider for each side's file, which FilerSource and FilerSink now accept for proxied chunk reads and writes. With no keys in the file or no flag, both fall back to the process-wide jwt.filer_signing configuration as before. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: use the side filer read key for manifest downloads and fall back per access level Manifest chunk resolution still signed proxied downloads with the process-wide read key, so a source filer requiring its own key 401'd on manifest-bearing files. A side security file that set only one access level also produced empty tokens for the other instead of inheriting the process-wide key, and the side file loader ignored the WEED_ environment overrides the filer itself honors. ResolveChunkManifest/ResolveOneChunkManifest keep their signatures; FilerJwt-aware variants thread the provider down to fetchWholeChunk, which prefers it on proxy URLs. The side loader now applies the same environment precedence and falls back to the process-wide signer per missing access level. * security: verify the configured filer token lifetimes * security: reject negative filer token lifetimes A negative expires_after_seconds reached GenJwtForFilerServer and produced a token with no expiration claim. Also synchronize the Authorization-header capture in the proxy test and restore the prior viper key on cleanup. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
14fdd61aea |
filer: stop the aggregated metadata subscribe loop rescanning an exhausted persisted log (#11644)
* fix(filer): gate the aggregated metadata disk pass on real change A subscriber whose start position is past the end of the local persisted log re-ran the whole persisted-log pass - store listings, file opens, readahead - on every loop iteration. Each iteration is paced only by the shortest wake (the 20ms hold floor on a busy watermark), so one parked subscriber kept a full CPU core busy for the life of the stream. The aggregated loop now mirrors the local loop's gate: the disk pass runs on the first pass and afterwards only when something it cannot miss changed - a local flush landed, the peers' flush low-watermark advanced (more content admitted, or new files in a shared store), the cursor moved, or a disk hold is pending (the ring read that follows an empty pass parks internally, so skipping there would strand a held entry). Regression test: a subscriber parked past the persisted-log tail holds the listing rate near zero and still delivers once peers report progress. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: re-arm the aggregated disk pass on unobserved change Review found three staleness classes the gate could not see: the flush low-watermark only catching rises (a joining peer lowers the minimum and invalidates an earlier pass's proof), a peer past the minimum landing a file without moving it, and a chunk subscriber's refs-stop bound advancing with wall time. Re-read when the low-watermark moves in either direction, when the chunk listing bound admits more files, and on a slow re-probe cadence for files no watermark can signal. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: unwind the parked ring read so the disk re-probe runs, and re-read on cursor rewinds A caught-up subscriber parks inside LoopProcessLogData's wait loop, so the re-probe interval in the outer disk gate could never elapse there; the callback now unwinds the read once the cadence is due so the gate re-evaluates. The cursor trigger also needs to notice rewinds, not just advances, since ResumeFromDiskError moves the cursor backward. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
49c25890ad | docs: regenerate star history chart | ||
|
Chris LuandDevin
|
56fbc3deb5 |
s3api: align S3/IAM error responses with AWS (#11632)
* s3err: add InvalidArgument and AuthorizationHeaderMalformed codes Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: answer unrecognized bucket PUT sub-resources with 501 A PUT on a bucket carrying an unrecognized query (logging, metrics, intelligent-tiering, ...) fell through to the bare CreateBucket route and returned BucketAlreadyOwnedByYou or re-created the bucket. AWS answers these with NotImplemented. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: reject malformed copy-source and multipart PUT parameters A malformed X-Amz-Copy-Source or a non-numeric partNumber fell through to the plain PutObject route and stored the body as a regular object. Answer them with InvalidArgument-class errors instead of writing data. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: verify x-amz-content-sha256 against the streamed body A PUT carrying a hex or base64 payload hash now streams through a verifier that reports a mismatch once the stream is exhausted, instead of storing an object that does not match its declared hash. The error is deferred so intermediate reads that drop (n>0, err) results cannot silently swallow it. Sentinel values (unsigned/streaming payloads) remain exempt and malformed values fail fast. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: map truncated SigV4 headers to the error for the missing field AWS answers an Authorization header missing Credential= with InvalidArgument and one missing or malformed Signature= with AuthorizationHeaderMalformed, instead of a generic MissingFields. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: answer IAM/STS failures in the query-protocol envelope Embedded IAM and STS routes now report authentication, form-parse and authorization failures with the IAM ErrorResponse body instead of the S3 Error envelope, so IAM SDK clients can parse them. Requests signed for s3 keep the S3 envelope, keyed off the credential scope. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: return 403 AccessDenied when the request has no Date Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: answer throttling rejections as SlowDown ErrTooManyRequest and ErrRequestBytesExceed reported made-up codes; AWS serves these throttling rejections as SlowDown. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: reject versionId requests on buckets that never had versioning GET, HEAD and DELETE carrying a non-empty versionId on an unversioned bucket now fail with InvalidArgument instead of being answered as a plain object request. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: answer DeleteObjects over 1000 keys with MalformedXML Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: InvalidArgument for non-numeric or out-of-range part numbers partNumber=abc, 0 and >10000 all resolve to InvalidArgument, matching AWS, instead of InvalidPart. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iam: correct LimitExceeded, InvalidAction and ServiceFailure mappings LimitExceeded is a conflict (409), an unknown Action is InvalidAction (404) rather than NotImplemented, and internal failures report the IAM receiver fault type. Applies to both the embedded IAM endpoint and the standalone iamapi server. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iam: refuse DeleteUser while access keys remain Deleting a user with live credentials orphaned its access keys; AWS answers DeleteConflict until they are removed first. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test/s3: add S3/IAM error-response compatibility harness * s3: return InvalidArgument for malformed x-amz-content-sha256 A header value that decodes to neither 32-byte hex nor base64 is a malformed argument, not a hash mismatch. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: stop spawned mini when readiness times out A slow-starting server otherwise survives the failure path and keeps the S3 port occupied for the next run. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: register atexit cleanup before setup A failed setup previously skipped cleanup, leaking the bucket and IAM user on persistent servers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: reject unrouted subresources on the DELETE bucket catch-all PutBucketHandler gained the same guard when the route-level check moved into the handlers; DeleteBucketHandler was missed, so an authorized DELETE /bucket?logging could delete the bucket instead of answering NotImplemented. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: tolerate unset fixture variables in cleanup and cover DELETE ?logging Cleanup now runs its IAM/multipart steps only when setup reached them, so an early setup failure still removes the bucket. Added a DELETE bucket-subresource case asserting NotImplemented and bucket survival. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: fail a case when its side-effect check reports a regression A non-empty check note now fails the case, so a deleted bucket or an object created by a malformed request cannot slip through behind a passing status check. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Joel Town RoadandChris Lu
|
079a7d8ba3 |
s3: give each prefix its own hidden-prefix probe budget (#11626)
* s3: give each prefix its own hidden-prefix probe budget The shared cursor.probedEntries counter in dirHoldsOnlyHiddenEntries was exhausted by one large all-deleted subtree, causing every later prefix in the same request to be treated as visible. The fix allocates a fresh budget (hiddenProbePerPrefixBudget, default 1000) for each top-level call and threads it down to recursive calls via a pointer, so cross-prefix budget bleed is impossible. Fixes #10847. * s3: keep 10000 probe budget, now per prefix Restore hiddenProbeBudget = 10000 as a package const (not a mutable var) applied per-prefix instead of per-request. The override hook moves to an unexported probeBudget field on ListingCursor; zero means use the package default. Tests set probeBudget: 3 on the cursor, keeping them fast without touching package state. Also revert the unrelated uint32 cast on the ListEntries Limit field. * s3: cap total hidden-prefix probe work per listing request The per-prefix budget resets for every candidate prefix, but deleted prefixes do not spend maxKeys, so a page can walk an unbounded number of them. Keep a request-wide probedEntries ceiling (hiddenProbeTotalBudget, 10x the per-prefix budget) so total probe work stays bounded. * s3api: pin the probe-budget cutoff and the request-wide ceiling --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Eliah RusinandClaude Opus 5.5
|
0bcebf708c |
ecbalancer: cap total shards per rack in Plan (#11623)
* ecbalancer: cap total shards per rack in Plan Plan caps data and parity per rack separately (ceil(data/racks) and ceil(parity/racks)), so with 10+4 over 8 racks a rack can legally hold 2 data + 1 parity. When a rack is one disk, losing two such racks loses 6 of 14 shards and the volume can't be read. The cross-rack phase now also caps each rack's TOTAL shards of a volume, sized with Place's rackTotalCap: ceil(shards/racks) unless the racks lack room, counting a rack's own shards of the volume as room since Plan can move them. - A rack above the cap sheds parity until it fits. Those shards may go to a data-bearing rack, and when no rack is under the parity cap they fall back to a rack under the total cap. #11438's non-overflow candidates keep moving only to data-free racks. - No cross-rack move lands on a rack at the cap. - A rack above the cap triggers balancing regardless of the imbalance threshold. - The fallback applies only while the source rack is above the cap; otherwise the next Plan moves the shard back. - Options.RackTotalCapRaised reports volumes whose cap had to be raised above the even share; the worker and the shell log it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * ecbalancer: finish the rack cap in one Plan, count SameRackCount room Review follow-ups. - The data pass can run out of destinations before the parity pass frees slots elsewhere, which left a data-heavy rack above the cap after one Plan (a one-shot shell balance stops there). The cross-rack phase now repeats while a rack is above the cap and the last round moved something. A shard moves at most once per plan, since each move runs as its own task. - planRackTotalCap bounds each node's room by what SameRackCount still allows, as Place does. Counting the raw free slots sized the cap too low, so RackTotalCapRaised missed volumes it should report. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
576837f1b2 |
fix(s3): make self-heal pointer persist CAS-bound against concurrent writers (#11627)
* fix(s3): make self-heal pointer persist CAS-bound against concurrent writers Follow-up to #11618: pointerless reads of a slash key whose regular-path entry is a physical parent (or a bare-key object) now fall through to healStaleLatestVersionPointer, which rescans .versions and persists a repaired pointer. The persist was an unconditional upsert off the pre-scan snapshot, so a PUT or delete that atomically advanced the pointer on the owner filer while the heal was rescanning could be rolled back, making older content or ACLs current again. Mirror the CAS discipline clearStaleLatestVersionPointer already applies: re-fetch the live .versions entry, require its pointer fields to still match the ones the heal observed, and abandon the persist (still returning the rescanned entry) when a concurrent writer has moved them. Write the live Extended map so concurrently updated fields are preserved. * fix(s3): close the check-then-act window in the self-heal pointer persist The CAS re-fetch added in the previous commit narrows the race but leaves a gateway-side window: after the live .versions entry is re-read and the pointer compared, the repair is still written back through an unconditional RPC, so a PUT or delete committing between the re-fetch and the persist still ends up rolled back by the stale repair. Bind the persist to the live image the heal just re-read with an IF_ENTRY_EQUAL precondition, the same discipline routedSelfCopy applies to stale self-copies: the filer evaluates the condition under the entry's path lock and conditional writes route to the owner filer, so a writer committing inside the window fails the precondition and the winner's pointer stands. FailedPrecondition and NotFound are authoritative replies and are not replayed by the failover layer. The test now also covers a writer committing during the persist, which reverts the pointer on the previous unconditional write-back. * s3api: CAS-bind the stale-pointer clear against concurrent writers The pointer clear re-read the live .versions entry and then wrote it back unconditionally through mkFile, so a writer committing between the re-fetch and the persist was rolled back to a cleared pointer. Persist through the same IF_ENTRY_EQUAL conditional update as the repair path. * s3api: test the CAS contract on the stale-pointer clear * s3api: never clear a pointer the clear did not observe as stale The CAS clear skipped its live-pointer match when the caller's snapshot carried an empty latest-version id, so a writer promoting a version between the clear's rescan and its re-fetch had the fresh pointer CAS-cleared away (expected = the writer's own live entry), briefly making the just-written version appear absent. With an empty observed id, reaching the persist at all implies a concurrent promotion (an idle key short-circuits as already-clear), so make the pointer match unconditional and abort instead. Extend TestClearStaleLatestVersionPointerConcurrentWriter with pointerless-snapshot cases: a post-rescan promotion must survive, and an idle pointerless key must short-circuit as already-clear. The fake filer's proto round-trip drops empty Extended maps, so the snapshot is padded the way real callers do. --------- Co-authored-by: zhaoyuchen <yc.zhao@yinzon.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
github-actions[bot]
|
3288d90b21 | docs: regenerate star history chart | ||
|
|
199d78539a |
fix(cosi): add get/list/watch on bucketclasses to enable static provi… (#11620)
* fix(cosi): add get/list/watch on bucketclasses to enable static provisioning * fix(cosi): bind provisioner deployment to the chart-created service account The deployment referenced a release-prefixed service account name while the chart creates and binds "seaweedfs-objectstorage-provisioner", so the RBAC grants never reached the provisioner pod. --------- Co-authored-by: Jonas Onshuus <jons@dips.no> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
ec261c5fbc |
S3: quote ETag in CopyObject and UploadPartCopy XML responses (#11624)
* s3api: extract quoteETag from setEtag Consolidate ETag quoting so the XML response builders can share it. * s3api: quote ETag in CopyObjectResult XML AWS returns the ETag quoted in the copy result body, matching the ETag header. Fixes seaweedfs/seaweedfs#11622 * s3api: quote ETag in CopyPartResult XML UploadPartCopy returned the raw ETag in the XML body while the response header and other APIs return it quoted. Fixes seaweedfs/seaweedfs#11622 * s3api: test ETag quoting in copy responses * s3api: assert quoted ETag on the wire in copy response tests * s3api: use strconv.Quote in quoteETag Addresses CodeQL 'potentially unsafe quoting' on string concatenation. |
||
|
dependabot[bot]
|
cac6cd4b16 |
build(deps): bump rustls from 0.23.43 to 0.23.45 in /seaweed-worker (#11617)
Bumps [rustls](https://github.com/rustls/rustls) from 0.23.43 to 0.23.45. - [Release notes](https://github.com/rustls/rustls/releases) - [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md) - [Commits](https://github.com/rustls/rustls/compare/v/0.23.43...v/0.23.45) --- updated-dependencies: - dependency-name: rustls dependency-version: 0.23.45 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
zhao-ycandChris Lu
|
d829275de6 |
fix(s3): recheck cache metadata and validate null objects (#11618)
* fix(s3): recheck cache metadata and read null directory markers Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com> * fix(s3): preserve null object identity and bound test cache Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com> * s3: trim comments on the anonymous read cache path Signed-off-by: Chris Lu <chris.lu@gmail.com> --------- Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com> Signed-off-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Joel Town RoadandChris Lu
|
23893eb378 |
volume: return error instead of panicking when .dat open fails (#11619)
* volume: return error instead of panicking when .dat open fails When backend.OpenVolumeFile returns an error (e.g. disk below -minFreeSpace), dataFile is nil. Calling backend.NewDiskFile(nil) immediately after caused a nil-pointer panic in f.Stat()/f.Name(). Move the existing error check to run right after OpenVolumeFile, before NewDiskFile is called, so the error is returned cleanly. Fixes seaweedfs/seaweedfs#11615 * volume: share .dat load error handling Extract datFileLoadError helper so open and create paths share one check. * volume: trim dat-open-fail test comments * volume(rust): cover unopenable .dat load path --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
32e77ff980 |
Secure Weed Mini Admin Listeners by Default (#11613)
* securing admin Signed-off-by: Subhadeep Maity <smaity@slb.com> * updated readme Signed-off-by: Subhadeep Maity <322813880+deepnemesis@users.noreply.github.com> * fixed pr comments Signed-off-by: Subhadeep Maity <322813880+deepnemesis@users.noreply.github.com> * docs: tidy weed mini admin bind notes Drop the new single-entry CHANGELOG.md since changes are documented via GitHub releases, and rewrap the README paragraph to match the surrounding one-line style without self-referential issue/PR links. * review comments Signed-off-by: Subhadeep Maity <322813880+deepnemesis@users.noreply.github.com> --------- Signed-off-by: Subhadeep Maity <smaity@slb.com> Signed-off-by: Subhadeep Maity <322813880+deepnemesis@users.noreply.github.com> Co-authored-by: Subhadeep Maity <smaity@slb.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
dependabot[bot]andChris Lu
|
de75450655 |
build(deps): bump github.com/apache/iceberg-go from 0.6.1-0.20260817192109-c2105090c9e2 to 0.7.0 (#11609)
* build(deps): bump github.com/apache/iceberg-go Bumps [github.com/apache/iceberg-go](https://github.com/apache/iceberg-go) from 0.6.1-0.20260817192109-c2105090c9e2 to 0.7.0. - [Release notes](https://github.com/apache/iceberg-go/releases) - [Commits](https://github.com/apache/iceberg-go/commits/v0.7.0) --- updated-dependencies: - dependency-name: github.com/apache/iceberg-go dependency-version: 0.7.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * iceberg: write delete manifests with NewManifestWriter iceberg-go 0.7.0 validates that delete entries only land in delete-content manifests, so the WriteManifest + byte-level content patch workaround no longer works: WriteManifest always creates a data-content writer and rejects delete entries outright. Use NewManifestWriter with WithManifestWriterContent instead and dispatch entries through Add/Existing/Delete by status. The returned ManifestFile now carries writer-computed counts, partitions and min-sequence-number, so the manual ManifestFile rebuild is dropped. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Upayan
|
b14dd1cee4 |
helm: drop fromToml dependency in security-configmap.yaml (fixes #11611) (#11614)
* helm: drop fromToml dependency in security-configmap.yaml (fixes #11611) fromToml was added to Helm in v3.17.0 (helm/helm#12026, merged 2024-09-12, one day after v3.16.0 was cut). This chart declares no minimum Helm version (no Chart.yaml kubeVersion, nothing in the README), and the call in security-configmap.yaml:21 is an unconditional *parse*-time failure on Helm < v3.17.0 - Go's text/template parses a file's entire body before evaluating any {{if}}, so this breaks the chart (any topology, any values) even when securityConfigEnabled is false and the ConfigMap would render nothing. Replaces the fromToml-based dig lookup with a small regex-based helper (seaweedfs.existingTomlKey) that reads the same four "key = ..." JWT signing-key values out of a previously-rendered security.toml, preserving the existing fallback-to-random behavior exactly. Verified: - helm lint (v3.16.3 and v4.3.0): clean - helm template with chart defaults: byte-identical output to the unpatched chart rendered via Helm v4 (which has fromToml) - the disabled/default path is untouched - helm template with security enabled, no prior ConfigMap: identical structure to the unpatched chart (helm v4), modulo the expected random key - Real helm install + helm upgrade round trip (live lookup, since "helm template" never evaluates lookup, even under the original fromToml code): the JWT signing key is identical across both releases - confirms key persistence across upgrades is preserved, not just "renders without erroring" - helm template with chart defaults, Helm v3.16.3: previously failed with a parse error naming fromToml as undefined; now renders successfully Fixes #11611. * helm: harden existingTomlKey against commented key lines and CRLF Addresses two review findings from greptile-apps on PR #11614: - The key-line regex matched the first "key = ..." anywhere in the section block, including a commented-out "# key = ..." line, which would shadow a real active key on a hand-edited or otherwise non-chart-generated ConfigMap. Anchored to line start with the Go regexp multiline flag ((?m)^key...), which a line starting with "#" cannot match. - The section-header match required an exact "]\n", so a ConfigMap with CRLF line endings would fail to match the block at all and regenerate the key instead of reusing it. Changed to "]\r?\n". Also adds a CI test ("Verify JWT signing key persistence across upgrades") exercising all of this end to end with a real helm install -> edit the live ConfigMap -> helm upgrade cycle, matching the existing "Verify SFTP host key secret lifecycle" test's shape: both edge cases are reproduced against a real ConfigMap and asserted on the post-upgrade rendered security.toml. Verified locally (same commands as the new CI step) against a real cluster before pushing. * helm: preserve JWT keys across supported TOML layouts * ci: use setup-python interpreter for JWT upgrade checks * helm: preserve keys under quoted TOML section headers * helm: ignore unrelated quoted TOML section headers |
||
|
dependabot[bot]
|
3a65365f6e |
build(deps): bump org.apache.spark:spark-core_2.12 from 3.5.7 to 3.5.8 in /test/java/spark (#11616)
build(deps): bump org.apache.spark:spark-core_2.12 in /test/java/spark Bumps org.apache.spark:spark-core_2.12 from 3.5.7 to 3.5.8. --- updated-dependencies: - dependency-name: org.apache.spark:spark-core_2.12 dependency-version: 3.5.8 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |