mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-25 06:20:42 +01:00
428af1c0eb74f9da96260e6c63ec5189a7040e5e
15772
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 user.emailandDevin
|
428af1c0eb |
filer: keep chunks a partially-committed format entry still references
A failed CreateEntry can persist the entry before erroring on parent creation; deleting every uploaded chunk then leaves the stored entry pointing at gone data. Filter the cleanup through the same inline-append reference check, and apply it to repack's UpdateEntry failure path too. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
user.email
|
fa872adb7f | hlsts: test playlist segment URLs carry the jwt credential | ||
|
user.email
|
b95b7f5761 | filer: also honor request-level Content-Md5 in format ingest | ||
|
user.email
|
19d7c0224b | Merge remote-tracking branch 'origin/master' into format-adapters | ||
|
user.email
|
a5ac780e66 |
filer: verify the media part Content-Md5 during format ingest
A supplied checksum that mismatches now rejects with BadDigest and the uploaded chunks are deleted, matching the normal upload path. |
||
|
user.email
|
291353b1e3 |
hlsts: carry the jwt query credential into generated segment URLs
Players fetching the playlist with ?jwt=... authenticate each segment through the same query parameter; without it every segment 401s. |
||
|
user.email
|
84e227dbc3 |
filer: hand uploaded chunks back on WORM rejection in saveMetaData
The locked recheck discarded uncommittedChunks, so a rejected write left its uploaded chunks behind; return them so callers delete them. |
||
|
user.emailandDevin
|
e144503caf |
format: gofmt hlsts test
Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
user.email
|
327495006a |
Merge remote-tracking branch 'origin/master' into format-adapters
# Conflicts: # weed/filer/filechunk_manifest.go # weed/server/filer_server_handlers_write.go # weed/server/filer_server_handlers_write_upload.go |
||
|
Chris LuandDevin
|
95d8a709e8 |
ci: run s3-go-tests on remote_storage changes (#12043)
The S3 Remote Cache tests exercise the remote-storage S3 client, but the path filter did not include weed/remote_storage, so a tagging change there can regress the suite without running it on the PR. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
e432b75822 |
remote_storage/s3: send only real object tags to the remote (#12039)
* remote_storage/s3: send only real object tags to the remote WriteFile and toTagging copied every entry.Extended attribute into the remote object's tag set, including Seaweed-internal keys such as Seaweed-X-Amz-Acl whose values are not legal S3 tag values. Remotes that validate x-amz-tagging now reject these uploads with InvalidTag. Only X-Amz-Tagging-* attributes are S3 tags: filter to that prefix, strip it, and encode through s3TaggingHeader so keys and values are properly escaped. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * remote_storage/s3: skip remote tagging call when the tag set is unchanged UpdateFileMetadata fired PutObjectTagging on any Extended diff, but most attribute changes are not tag changes. Compare the effective tag sets and return early when they match. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * remote_storage/s3: assert WriteFile errors in tag-header tests Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
81c90cb446 |
filer.backup: retry a failed checkpoint read instead of replaying from the epoch (#12036)
* filer.backup: retry a failed checkpoint read instead of replaying from the epoch Fixes #12035 * test: reset sink.local.directory in backup offset test cleanup The new test sets sink.local.directory on the shared viper but only reset the enabled flag, leaking a key that points at a deleted t.TempDir() to later tests in the package. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover the retried attempt re-reading the checkpoint The fake filer now fails only its first KvGet, and the test runs a second doFilerBackup attempt asserting the checkpoint is re-read and the attempt proceeds past the offset stage. LookupDirectoryEntry is stubbed so the SSE-S3 KEK probe returns a miss instead of retrying 10x2s. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
4c2c0ff21c |
s3: fix wildcard matcher skipping '*' targets in policy patterns (#12034)
* s3: check pattern '*' before literal equality in matchWildcardString When the target string carried a '*' at the offset of the pattern's '*', the literal-equality case consumed it as a one-character match and no backtrack point was recorded, so the comparison returned false. A Deny Resource carrying a policy variable, or an Allow carrying NotResource, could then be sidestepped by an object key containing '*' at the aligned offset. * test: cover literal asterisks in wildcard match targets --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
1d2d9dfed2 |
test(fuse): raise mini/filer and mount readiness waits to 90s (#12037)
Same startup-timing flake class as the Makefile waits: volume/filer readiness at 30s is marginal on loaded runners — observed failing on a github-hosted runner too (TestPOSIXCompliance + TestExternalPOSIXSuites). The process-death early return still reports real startup failures quickly. |
||
|
|
c4d743a69f |
s3: make ListObjectsV2 continuation tokens round-trip XML-unsafe keys (#11996)
* s3: keep ListObjectsV2 continuation tokens round-tripping XML-unsafe keys A page ending on a key XML 1.0 cannot carry (for example one containing \x01) produced a NextContinuationToken from that raw key. The XML encoder replaced the control characters with U+FFFD, so a client resuming at the token restarted after the mangled name and skipped every key sorting between it and the real one. encoding-type=url encoded the keys but not the token, so it hit the same skip. Encode the token as base64url behind a prefix when the key is not valid XML text; keys that survive XML unchanged keep the bare key as the token, so their tokens are byte-identical to before. The handler decodes the prefix back to the raw key before listing. Fixes #11990 * s3: keep undecodable base64: list tokens resuming at the raw key * s3: round-trip ListObjectsV1 markers over XML-unsafe keys * s3: round-trip ListObjectVersions key markers over XML-unsafe keys * s3: round-trip ListMultipartUploads key markers over XML-unsafe keys * s3: decode only tokens whose key could not have stayed bare * s3: document and pin the encoded-wins rule for ambiguous list tokens A token like base64:AQ reads as both a legacy bare key and the encoded form of an XML-unsafe key. The encoded reading already won in code; state why and cover it so the choice is not reverted as a false simplification. * s3: keep request-marker echoes in the response's wire encoding The decoded marker/key-marker is a raw key; echoing it writes XML-unsafe bytes straight into the response, where the encoder turns them into U+FFFD. Re-encode each echo through encodeListContinuationToken so the client reads back the token it sent, and apply urlPathEscape for V1 when encoding-type=url is active, matching the listed keys. * s3: wire-encode the V1 NextMarker echo like the request marker * s3: pin resuming an XML-unsafe key past a sibling directory * s3: wire-encode ListMultipartUploads markers and keys under encoding-type=url Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: unescape url-encoded marker echoes on list resume Under encoding-type=url the V1 and ListMultipartUploads responses emit marker echoes through urlPathEscape, and clients pass them back verbatim. The incoming marker was used as the resume position without undoing that escaping, so resuming after a url-unsafe-but-XML-safe key like "a b" (marker "a%20b") restarted the listing at the literal escaped text. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: assert the full decoded V1 sequence in the marker round-trip test The test only checked the page count, so a regression that skipped one key while repeating another still passed. Requesting encoding-type=url keeps the XML-unsafe keys legible for the decoded comparison and exercises resuming with the escaped marker echo. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: only unescape markers in the canonical urlPathEscape form A url-encoded request marker like a%41 cannot be encoder output, since urlPathEscape never emits %41 for 'A'. Unescaping it anyway resumed at aA and skipped every key sorting between a%41 and aA, including a%42. Require the unescape to re-escape to the received marker so only verbatim echoes decode; anything else stays the raw key the client sent. * s3: pin a raw-key marker that looks like an escape sequence * s3: cover the marker-echo decode guard with unit cases * s3: wire-encode the ListObjectsV2 StartAfter echo under encoding-type=url AWS encodes Delimiter, Prefix, Key, and StartAfter in V2 responses when url encoding is requested, but StartAfter was echoed raw. The echo now takes urlPathEscape and the incoming start-after param runs through decodeMarkerEcho, so a client resuming from the emitted form lands on the raw key. * Merge master and reconcile ListMultipartUploads marker encoding Resolve the overlap with the merged encoding-type=url pass: keys and markers now go through the single post-pass so nothing is url-escaped twice. Markers are still token-encoded when the response is not url-encoded, and the key-marker decode checks the query-escaped wire form the endpoint actually emits. * s3: echo the decoded start-after under encoding-type=url The escaped echo of the raw input is not a fixpoint: a client resubmitting the echoed StartAfter decoded it to the literal escape text and resumed at the wrong position. * s3: keep url-mode multipart key-marker decodes off the token path Under encoding-type=url the emitted KeyMarker/NextKeyMarker are the url-escaped raw key, never a base64-prefixed token, so the request decode must unescape only. Running the token decode afterward misread a literal "base64:..." key as an encoded token and resumed at the decoded bytes, skipping the keys in between. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
fcd28031e7 |
mq/kafka: wait for a filer to register instead of panicking on an empty list (#12019)
* mq/kafka: wait for a filer to register instead of panicking on an empty list NewSeaweedMQBrokerHandler queried the masters for filers once, ~2 s after start, and handed an empty list to NewFilerClientAccessor, which panics. A fresh filer registers only after its empty-membership confirm window (10 s), so a gateway started with a fresh cluster crashed (Kafka Quick Test, kafka-gateway exit 2). Retry while the list is empty for up to 60 s, then return an error. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * mq/kafka: wait out a transient-empty broker listing too A fresh SMQ broker registers with the masters on the same announce delay as a filer, so the one-shot broker lookup could fail startup the same way. Generalize waitForFilers into waitForClusterNodes and bound both discoveries by the same window; the empty-broker check becomes unreachable and drops away. * mq/kafka: wait out the broker health check like node discovery A broker announces to the masters before it has acquired its balancer lock, so the gateway's three-attempt probe could still fail fast on a fresh cluster. Retry FindBrokerLeader for the same window the caller waits out empty listings. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
34c7bd03ad |
ci: clean up stale FUSE mounts before running tests (#12020)
* ci(s3tables): tolerate stale FUSE mounts in disk diagnostics * ci(s3tables): bound df diagnostics with timeout, explain once A stale FUSE mount can also hang statfs, not just fail it, so wrap df in timeout 15; one comment at the first site covers all sixteen. * ci: tolerate stale FUSE mounts in df diagnostics elsewhere The same self-hosted runners serve the other workflow files; a bare df -h there can stall or fail a job the same way. * ci: unmount disconnected FUSE test mounts before jobs start * ci: skip FUSE cleanup when the variant checkout is skipped * ci: reclaim the crashed run's test dir once its stale FUSE mount is detached * ci: stop tracking python bytecode for the FUSE cleanup action The helper's __pycache__ slipped into the tree; ignore it so local test runs cannot re-add it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: run the FUSE cleanup action's unit tests on action changes test_cleanup.py was never wired into CI, so a regression in cleanup.py would ship silently. A path-filtered workflow runs it on every change under .github/actions/. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: match every FUSE test mount layout in the cleanup action The candidates list only matched seaweedfs_fuse_test_*/mount, so stale mounts left by fuse_failover and fuse_p2p runs (mountN leaves under seaweedfs_fuse_* dirs) and the posix-compliance workflow's fixed /tmp/seaweedfs/mount were never reclaimed and could still wedge df. * ci: run the FUSE mount cleanup in the fuse_failover and fuse_p2p workflows --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
18f9d9a0b6 |
feat: Free Volume Deadband (M) for EC shard placement (#12029)
* feat: Free Volume Deadband (M) for EC Place nodes and racks |ΔfreeSlots| ≤ M is a tie; remainder is liveShards. M=0 stays strict freeSlots. M is free volume slots (× dataShards). pickNodeInRackEligible and pickRackEligible share the rule. Worker plugin config free_slots_deadband, default 0. Not on the write path. * fix: anchor EC freeSlots deadband on class maxFree Pairwise |Δfree|≤M vs the current winner is not transitive: 120→110→100 at M=10 slots picked 100. After evenness, rank by maxFree, keep |free-maxFree|≤M, then min liveShards. Shared by node and rack. M=0 is still only maxFree then live. * worker: add free_slots_deadband to ErasureCodingTaskConfig * erasure_coding: persist free_slots_deadband through TaskPolicy ApplyMaintenanceConfigFromToml accepted the key but the admin save path round-trips config through ToTaskPolicy, so the value was dropped on reload; serialize it like the other EC settings. * erasure_coding: bound free_slots_deadband at its declared maximum The TOML loader and the plugin worker config only clamped negatives, so a value above the spec's 100000 could reach the int32 TaskPolicy field and wrap; cap at 100000 on both entry paths and declare MaxValue on the plugin field. * erasure_coding: document free_slots_deadband in scaffold admin.toml --------- Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
|
52e05ccda4 |
s3: accept a self-copy that changes the storage class (#12031)
* s3: accept a self-copy that changes the storage class or encryption AWS changes an object's storage class or encryption by copying the object onto itself with the new x-amz-storage-class or SSE headers, and only rejects a self-copy that changes none of metadata, storage class, website redirect location or encryption. CopyObject rejected every self-copy without REPLACE metadata or tagging directives as InvalidRequest, so these requests failed. * s3: reject an invalid x-amz-storage-class on CopyObject A self-copy with a bad storage class used to fail as InvalidRequest; once such copies are admitted the value must be validated like PutObject does, or it would be stored as opaque metadata. * test: cover the invalid storage class rejection on self-copy * s3: only a storage class change admits a self-copy Admitting encryption changes on a self-copy is unsafe: the metadata-only path compares only KMS key IDs, so a new encryption context or bucket-key setting is advertised over the old data key, and SSE-KMS key rotation onto the same entry keeps the old data key. Count only a requested storage class that differs from the object's (STANDARD when none is stored) as a change, so a self-copy to the current class or with only encryption headers is rejected as before. Such a self-copy updates the object in place and has to keep its encryption; one that would re-encrypt it or uses SSE-KMS is rejected too. --------- Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
514c0c1e99 |
topology: stop a stale duplicate-vid report from re-quarantining the survivor (#12032)
* topology: drop a duplicate-vid report whose own copy is already gone flagDuplicateVolumeId runs on a goroutine after registration, so by the time it executes the reporter's copy may have been unregistered (a finished move drops the losing copy). The surviving copies are then legitimate, yet the stale report still quarantined them and stored a phantom (node, vid) key for the departed reporter. Skip the marks when the reporter no longer serves the volume under its own layout key. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(topology): cover a stale duplicate report against a lone survivor The existing tests only flag the conflict before the losing copy is unregistered. Run the delayed duplicate check after the reporter's copy is already gone and assert nothing is quarantined and the survivor stays writable, so removing the stale-report guard fails the test outright instead of needing a lucky schedule. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(topology): cover the enforceQuarantine-driven release retry A failed writable push re-arms the record, and the next volume report drains that retry through enforceQuarantine itself - a different entry point than the pulse-driven pass the other tests exercise. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
8841eb6c70 |
volume: check If-None-Match before If-Modified-Since and match ETag lists (#12027)
* filer: match If-Match and If-None-Match against "*" and ETag lists checkPreconditions compared the whole header as one ETag, so "*", a list of ETags and a weak W/ tag in If-None-Match never matched (RFC 9110 13.1.1, 13.1.2). * volume: check If-None-Match before If-Modified-Since and match ETag lists A volume GET answered 304 from If-Modified-Since even when If-None-Match named a different ETag, and compared If-None-Match as one exact string, so "*", lists and weak tags never matched (RFC 9110 13.1.2, 13.2.2). * filer: split etag lists only outside quoted tags A comma inside a quoted opaque tag is a legal etagc, not a list separator; strings.Split cut such tags into fragments that CanonicalizeETag could turn into a false match. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: match repeated If-Match and If-None-Match header lines Header.Get reads only the first field line, so a match on a later repeated header was missed; join all field values like any other comma-separated list. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: match repeated If-None-Match header lines Header.Get reads only the first field line; join all field values so a match on a later repeated header is seen. Also covers a comma inside a quoted opaque tag. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: send the etag on a 304 like the filer does RFC 9110 15.4.5 lists ETag among the fields a 304 must carry when a 200 would send it; SetEtag now runs before the notModified check. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: mirror the conditional-GET fixes in the rust server not_modified_response now checks If-None-Match before If-Modified-Since, matches "*", weak tags, quoted commas and repeated field lines through the same list rules, and sends the etag on a 304. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: treat an empty If-None-Match like the Go server does * filer: ignore empty elements in etag precondition lists * volume: ignore empty elements in etag lists like the filer * filer: let a present-but-empty precondition line suppress the date check * volume: gate If-Modified-Since on a present If-None-Match line * volume: gate If-Modified-Since on header presence like the Go server --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
59cccc788c |
master: read the writable-volumes cap from master.toml (#12009)
* master: read the writable-volumes cap from master.toml -maxWritableVolumes becomes the master.max_writable_volumes config key, so the new master limit is set the same way as other master options instead of a flag wired into every command that embeds a master. * master: name the writable cap master.max_writable_volumes_per_layout The cap applies per volume layout (collection + replica placement + ttl + disk type), so the key names its scope instead of reading as a global cap. * master: keep an embedder's MaxWritableVolumes when the key is unset NewMasterServer overwrote option.MaxWritableVolumes with the viper value, which reads as 0 whenever master.toml omits the key, silently dropping a cap a Go caller supplied through MasterOption. Seed the default from the option instead so the field survives an absent key; an explicit config entry still wins. * master: drop stray nested-checkout gitlinks Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
9e7fd2eb90 |
filer: match If-Match and If-None-Match against "*" and ETag lists (#12026)
* filer: match If-Match and If-None-Match against "*" and ETag lists checkPreconditions compared the whole header as one ETag, so "*", a list of ETags and a weak W/ tag in If-None-Match never matched (RFC 9110 13.1.1, 13.1.2). * filer: split etag lists only outside quoted tags A comma inside a quoted opaque tag is a legal etagc, not a list separator; strings.Split cut such tags into fragments that CanonicalizeETag could turn into a false match. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: match repeated If-Match and If-None-Match header lines Header.Get reads only the first field line, so a match on a later repeated header was missed; join all field values like any other comma-separated list. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: ignore empty elements in etag precondition lists * filer: let a present-but-empty precondition line suppress the date check --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
d6a68468ad |
volume: keep a below-minimum fsync batch open for stragglers (#12016)
* storage: hold a below-minimum fsync batch open for stragglers When the durable-write queue drained, the worker flushed whatever it held, so low-concurrency fsync writes collapsed to one sync per request -- a seek plus a rotation on rotational media no matter the payload. Once the queue empties with the batch still below the configured minimum bytes and requests, the batch now stays open up to SyncBatchOption.Wait, armed once per batch so the added latency stays bounded. DiskLocation carries the option and defaults it; Store exposes a setter the server calls once at startup. * volume server: expose the fsync batch window as -fsync* flags -fsyncMinBatchBytes, -fsyncMinBatchRequests and -fsyncBatchWait carry the operator's batching floor into every disk's SyncBatchOption. -fsyncBatchWait=0 restores the previous flush-on-drain behavior. * server/mini: register the fsync batch window flags weed server and weed mini build their own VolumeServerOptions and left the three new fsync flag pointers nil, so startVolumeServer panicked dereferencing them at startup. * storage: bound the fsync batch window by Wait while the queue refills Once the straggler timer was armed, a steady trickle of queued writes kept taking the len(requests)>0 continue path, so the first write could wait past Wait until the size or count cap flushed the batch. Track the armed deadline and flush once it passes even with requests buffered. * storage: sync the straggler test on the worker taking the first write The fixed sleep let both writes queue together (passing under the old flush-on-drain behavior) or the straggler arrive after the window closed (flaky failure). Poll the drained queue instead so the second write can only land inside the armed window or a later batch. * volume server: default the fsync batch window when its options are unset weed server and weed mini build VolumeServerOptions without the fsync flag pointers, and outside embedders can too. Resolve the batch window from the defaults so startVolumeServer never dereferences nil options. * storage: drive the batch-window test through an injectable timer The window is armed by the worker now, so the straggler provably joins an open batch and expiry is fired by hand instead of racing a sleep. * seaweed-volume: accept -fsyncMinBatch/-fsyncBatchWait flags Same flags and defaults as weed volume so the Rust server's durable write batching can be tuned identically. * seaweed-volume: hold a below-minimum fsync batch open for stragglers Mirrors the Go worker's batchWait: once the queue drains under the minimum batch, the window stays open for stragglers and an expired window still bounds the batch while the queue refills. Paused-time coverage replaces any sleep-based assertion. * seaweed-volume: sync the straggler test on the armed batch window The test fired both writes together, so the straggler could already be queued when the worker drained — a regression that drops mid-window writes from the open batch would still pass. Wait until the worker arms the window before sending the last write (Go's test polls the drained channel for the same guarantee). Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * seaweed-volume: reject fsyncMinBatchBytes that does not fit usize --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
b0e6f7a096 |
s3: allow up to 50 tags in a bucket tag set (#12030)
* s3: allow up to 50 tags in a bucket tag set PutBucketTagging validated the tag set with the object tag rules, which allow 10 tags. AWS allows a bucket up to 50 tags, so tag sets of 11 to 50 tags were rejected with InvalidTag. Validate bucket tags against 50 and keep the limit of 10 for objects and S3 Tables. A tag set that repeats a key is rejected with InvalidTag, as AWS does, instead of keeping one of the values, so 51 tags with a repeated key no longer pass as 50. * s3: reject duplicate tag keys in PutObjectTagging The tag set collapses to a map, so a repeated key was silently kept once.\nMatch the bucket tagging check and answer InvalidTag instead. * s3: reject empty tag keys AWS requires a tag key to be at least one character; the shared\nvalidator only checked the 128-char upper bound. * s3: reject duplicate tag keys in the tagging header parser parseTagsHeader (the x-amz-tagging path used by CopyObject with TaggingDirective=REPLACE) silently kept the last value when a key repeated, while the PutObject and multipart header paths and the new XML checks return InvalidTag for the same input. A repeated key after URL decoding now fails the same way. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: validate x-amz-tagging tags on PutObject before storing the entry The header path only rejected duplicate keys, so an upload could store more than ten tags or keys and values that fail S3 tag rules. Run the parsed set through ValidateTags like the XML tagging handlers do. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: validate x-amz-tagging tags on multipart initiate ParseS3Metadata stored header tags with no tag-set validation, the same gap putToFiler had. Multipart uploads now reject oversize or malformed tag sets with InvalidTag. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: count tag length in characters, not bytes * s3: validate object tags before uploading chunks * s3: reuse parseTagsHeader and drop raw header from warnings --------- Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
de02158212 |
filer: let NewChunkStreamReaderFromFiler fall back to the master on a cache miss (#11964)
refreshUrls invalidates every cached location of a volume after a failed read and re-looks it up through the reader's lookup function. With the cache-only masterClient.LookupFileId, one failed read to one replica left the volume missing from the filer's cache until restart, so metadata-log replay skipped it. Use GetLookupFileIdFunction like the other read paths. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
8844d9823f |
iam/oidc: wrap the jti replay cache in its own type (#12017)
* iam/oidc: wrap the jti replay cache in its own type The replay-protection state lived as ten fields and nine methods spread across OIDCProvider. jtiReplayCache now owns the store, count, bound, sweep scheduling, and cleanup worker; the provider keeps a single jti field and delegates configure, consume, stop, adopt, and merge. * iam/oidc: lazily allocate the jti replay cache A zero-value OIDCProvider left jti nil, so Initialize, single-use validation, Shutdown, and the replay-state handoffs panicked on a nil cache. Allocate the cache through a sync.Once-guarded getter instead. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
3e5aa8c3b3 |
topology: mark replica quarantined before recording the pair (#11983)
* topology: serialize quarantine records with their layout marks A quarantine publishes a (node, vid) record and marks the replica read-only at the layout level; release and heartbeat flag updates must observe that pair as one state. Store-then-mark let a concurrent release delete the record before the mark landed, stranding the surviving replica read-only; mark-then-store let a heartbeat clear the mark before the record was visible, returning a conflicting replica to the writable list. quarantineLock now covers every record<->mark pair: flagDuplicateVolumeId's quarantine, releaseResolvedQuarantine's undo, the isQuarantined-gated marks on registration, and the quarantine-folded read-only flag writes on heartbeat paths. Remote server pushes stay outside the lock; a push that lands after the record was released is undone, and a failed release push re-adds the record for a later retry. The release test now waits for in-flight async flags to settle instead of asserting on a single read. * topology: fold quarantine into RegisterVolume's flag rebuild RegisterVolume rewrites every replica's location read-only flag from what its node last reported, and reported flags deliberately exclude quarantine. Any registration of a shared vid -- a reconnected replica, a new copy, the self-heal and delta-heartbeat re-register paths -- therefore cleared the quarantine mark on every replica in the location, while the gate afterwards only re-marked the registering node. Two call sites had no gate at all. RegisterVolume now takes an isQuarantined predicate and evaluates it under accessLock for every replica it touches, so the mark is never dropped in the first place: a release unmarks after the rebuild, and a quarantine mark re-applies on the next rebuild, with no window between. * topology: keep quarantine when a vacuum commit re-adds a replica SetVolumeAvailable re-adds the replica into the location list; a re-added slot starts with a cleared read-only flag, so a quarantined replica came back flag-free and the next ensureCorrectWritables could return a conflicting volume to the writable list. Pass the quarantine state in and set the flag back under the same accessLock, with the caller holding quarantineLock so release cannot interleave between the check and the mark. * topology: merge quarantine records instead of overwriting them enforceQuarantine wrote back the state copy it loaded before its remote push, so a concurrent writer's enforced=true was lost: the server stayed read-only while the record said unenforced, and a later release skipped the writable push it owed. The write-back now reloads the record under quarantineLock and merges -- enforced wins from either side, reported is taken from the newer pre-enforcement observation only while no push has landed yet. All three compensating writable pushes had the same hole in reverse: a failed un-quarantine left the server read-only with no record to retry from, and release's restore blindly overwrote any record a fresh flag pass had created in between. They now go through restoreQuarantineRecord, which adds the record only when absent and otherwise sets enforced on the current one, preserving its reported flag. * topology: hold quarantineLock over registration's quarantine check-and-mark RegisterVolume reads the quarantine record and marks the replica read-only under accessLock, but the pair ran without quarantineLock. A release could push writable, watch the record get re-read, and then delete it: the replica stayed read-only with no record left to clear, and unchanged writable heartbeats skip SetReplicaReadOnlyFlag so the mark never repaired. Holding quarantineLock around the check-and-mark in all three production callers serializes it against the release clear-and-delete. * topology: serialize remote quarantine pushes behind a record recheck releaseResolvedQuarantine deletes the record under quarantineLock and sends the writable request after unlock. A conflict flagged while that request is in flight creates a fresh record and marks the replica read-only again, and the older writable request can still land last, leaving the server writable under an active quarantine. The reverse flap strands a stale read-only mark the same way. Route every QuarantineVolumeFn call through pushQuarantine, which holds quarantinePushLock across the remote call and re-checks the record under quarantineLock right before sending: a read-only push requires the pair to exist, a writable push requires it to be gone. A skipped push means the decision already moved on, so the final server state always follows the newest record. * topology: serialize quarantine pushes per replica, not globally A single quarantinePushLock held across the remote mark call let one slow volume server (up to the 10s client timeout) delay pushes for unrelated replicas. Keying the lock by (node, vid) keeps pushes to one replica ordered in record-decision order while pushes to others run in parallel. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: restore the layout read-only mark with a released record A failed writable push re-armed the quarantine record but left the replica marked writable in the layout, so PickForWrite could route to a replica the server still holds read-only. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: wait out async duplicate checks in the release-retry test TestFailedReleasePushRestoresQuarantineMark asserts layout marks around a release pass, but the two RegisterVolumeLayout calls each start a background flagDuplicateVolumeId that can still be mid-flight. A late mark could land after the final release and fail the writable assertion even though the code behaved correctly. Track the async passes in a WaitGroup and drain them before the explicit flag call so the rest of the test is deterministic. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: keep quarantine when a replica's layout key is unreadable * topology: keep a failed writable push's restore ordered under the push lock * topology: retry the quarantine release on every duplicate check * ci: isolate Java and S3 test servers on persistent runners * ci: drop the duplicated -ip flags in the lifecycle test server The bind flags were already on the command; the isolation pass added a second pair. * topology: retry a failed quarantine writable push on the next report * topology: retry pending quarantine releases off the leader pulse A failed writable push re-arms the quarantine record with a retry that only enforceQuarantine fires, and that runs only for volumes a heartbeat carries. Once the server has reported the replica read-only it drops out of the delta reports, and restoreQuarantineRecord keeps the stored flag matching the server's so the digests agree and no full list is requested - the pending release then never retries and the idle volume stays read-only forever. Run the same release pass from the leader's pulse loop so a pending retry no longer waits on a report. * topology: exercise the pulse-driven quarantine release retry --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
5700aa5e91 |
s3: evaluate bucket CORS for path-style OPTIONS on the bucket root (#12028)
* s3: evaluate bucket CORS for path-style OPTIONS on the bucket root
A path-style preflight for /bucket (no trailing slash) missed the bucket
OPTIONS route, whose template is /{bucket}/, and was answered by the global
OPTIONS handler from the global AllowedOrigins instead of the bucket CORS
configuration. Register the bucket OPTIONS route without a path matcher, and
keep OPTIONS for the health probe paths on the service-level handler ahead of
each path-style router.
Fixes #12025
* test: drop the issue reference from the bucket-root preflight test comment
---------
Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Chris LuandDevin
|
9281ed8007 |
master: never auto-admit raft peers on a single-node configuration (#12021)
* master: never auto-admit raft peers on a single-node configuration A master started without cluster peers (peers=none, or no peers so the list holds only itself) declares a one-node cluster, yet any foreign master that registered over KeepConnected was admitted as a second voter. The resulting two-voter cluster cannot reach quorum when the foreign peer is unreachable, wedging a previously healthy single master. Gate raft admission on having been configured with cluster peers; masters started with a peer list keep the announce-to-join flow used to grow the cluster. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: source SingleMaster from -peers=none, not the peer count A master started with an empty peer list ends up with a one-peer map containing itself, and NewMasterServer inferred declared single-master mode from that count. The master still mints a raft cluster through the bootstrap path, but AdmitRaftPeer then ignores every later joiner, so a deployment that grows from one master to several can never form a quorum. isSingleMasterMode already spells out the declared case (-peers=none); populate MasterOption.SingleMaster from it instead. * master: cover clustered raft admission with a real in-memory leader The previous clustered test had no raft leader and asserted nothing, so an AdmitRaftPeer that always returned early still passed. Bootstrap a single-node hashicorp raft over the in-memory transport, wait for leadership, and assert the leader reports and admits a missing peer. * master: return the raft admission result and assert committed join AdmitRaftPeer ran AddVoter on a detached goroutine, so the only observable state was an in-flight marker deleted on success and failure alike, and the clustered test could never tell whether the join worked. Admission now hands back a result channel; the test runs a connected second in-memory node and asserts the peer sees itself a committed voter. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
fd3b437e04 |
s3api: write the null version's object-lock metadata to the plain entry (#12033)
* s3api: write the null version's lock metadata to the plain entry PutObjectRetention and PutObjectLegalHold with versionId=null loaded the pre-versioning entry but wrote the update under <key>.versions/v_null, so the acknowledged lock never bound the object it was read from. Resolve the entry and its write path in one helper that applies the same null-version rule as getSpecificObjectVersion. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: cover the null-version lock write path Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: persist object-lock metadata with an in-place entry update mkFile rebuilds the target entry from chunks alone, so a lock write on a plain (null-version) entry discarded attributes, inline content, and the remote reference. UpdateEntry keeps the whole fetched entry, matching how the tagging and ACL writers persist metadata. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
689ea6ea3e |
release: digest-bound provenance and SBOM attestations for container images (#12024)
* release: emit digest-bound provenance and SBOM attestations Per-platform builds now record SLSA provenance at mode=max and a BuildKit SBOM, so the builder identity and resolved dependencies are captured instead of the empty predicate the 4.48 build record carried. After the multi-arch index is assembled and signed, the merge job publishes GitHub artifact attestations bound to the merged digest and a syft SPDX SBOM of the image, giving digest-pinning consumers a verifiable record on the exact index they pull. * release: keep unfixed findings in the trivy SARIF upload ignore-unfixed dropped every finding without an available patch, so the uploaded report understated the image's exposure. The scan stays report-only; only the completeness of what is uploaded changes. * release: generate and attest SBOMs per platform digest An index-wide syft scan only describes its default platform, so the SBOM attested to the merged multi-arch digest misrepresented every other manifest. Scan and attest each platform digest instead, which also drops the sbom-action upload defaults that raced release-asset permissions and artifact names across the merge matrix. * release: resolve per-platform index to its image manifest before syft/cosign push-by-digest records point at OCI indexes (image + attestation manifests), so scanning or attesting the index digest picks the runner's platform and fails outright for the other arches under set -e. Resolve the non-attestation manifest inside each index and bind the SBOM attestation to the image digest a same-platform pull of the merged index actually returns. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
be081b49b7 |
notification: named webhook targets for per-path event routing (#11925)
* notification: honor the config prefix in the multi-queue guard validateOneEnabledQueue checked "<name>.enabled" without the notification prefix, so the fatal-on-two-queues guard never saw the real keys and could never fire. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: support named webhook targets under [notification.webhook.<name>] The webhook queue only accepted one endpoint, so events could not be routed to different services by path prefix. Each enabled [notification.webhook.<name>] subsection now spins up its own target — own filter, http client, retry middleware and worker budget — published on a per-target topic so every matching endpoint gets each event exactly once. The flat [notification.webhook] section keeps working as the unnamed target, and named subsections alone now activate the queue. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: require enabled for the flat webhook target buildTargets included the flat section whenever an endpoint was set, so a disabled [notification.webhook] block — which the scaffold ships with a placeholder endpoint — would start receiving events as soon as any named target activated the queue. * notification: resolve named target enabled flags through config namedTargetPrefixes read the raw map value, which never sees environment variable overrides; GetBool resolves WEED_* env bindings, so a named target disabled in the file can now be switched on by WEED_NOTIFICATION_WEBHOOK_<NAME>_ENABLED. * notification: cancel the webhook queue only after all workers stop Each target's router goroutine deferred the shared cancel, so the first worker to exit tore down the queue context for every other target and the dead letter logger. * notification: isolate webhook targets behind per-target queues SendMessage published to each target's topic inline, so a subscriber channel that filled up blocked every later target and the filer's notification path with it. Each target now drains its own inbox through a forwarder goroutine; SendMessage fails fast on a full inbox, matching the "throw message" contract the filer already applies to errors. * notification: name the webhook target in failure and dead letter logs Every target dead-letters to the same topic, so without the name there is no way to tell which endpoint failed or needs a replay. * notification: assert delivery in the named targets test The test only exercised SendMessage; it now records requests on both endpoints and checks matching, non-matching, and unrouted keys reach the right servers exactly once. * notification: wait for the webhook router before accepting events * notification: apply backpressure instead of dropping a full webhook inbox * notification: bound queued and in-flight webhook deliveries per target * notification: release the pending token when shutdown races the enqueue A SendMessage that wins the pending select as the queue is canceling could block forever on a full, undrained inbox. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: stop waiting for a router that failed to start A router that exits before reaching Running() would leave initialize blocked on a context nothing had canceled yet. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: route webhook events past saturated targets A full pending reservation made SendMessage wait on the slow target before it ever reached the healthy ones after it, coupling every endpoint's latency to the most congested one. Targets that can take a reservation now get the event immediately; saturated targets defer to a second pass that still blocks for a slot, so backpressure is kept and no event is dropped. * notification: retry webhook deliveries before dead-lettering PoisonQueue must wrap the retry middleware: added after it, the poison middleware intercepts the first handler error, republishes to the dead letter topic, and returns nil — the retry chain never sees a failure, so max_retries/backoff settings were dead code and every delivery failed straight to dead letter on the first attempt. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
fb3fb9dcb8 |
s3api: support s3:RequestObjectTagKeys in bucket policies (#11993)
* s3api: support s3:RequestObjectTagKeys in bucket policies Condition operators accept the ForAllValues: and ForAnyValue: qualifiers, and s3:RequestObjectTagKeys holds the keys of the tags taken from X-Amz-Tagging wherever s3:RequestObjectTag/<key> is set. CopyObject no longer stores an empty-key tag for an empty segment in X-Amz-Tagging, so it keeps the key set a policy has checked. Fixes #7679 * s3: parse tagging header like the tag handlers, not url.ParseQuery url.ParseQuery stops at 10,000 parameters, so an X-Amz-Tagging header padded with empty & segments made requestObjectTags return nil: the copy handler still stored the denied tag while every s3:RequestObjectTag* condition saw nothing. Scan non-empty segments the way parseTagsHeader does so the policy sees the same tags the object gets. * test: gofmt s3tables test files after #11997 The format check fails on these files on master as merged, so every PR's S3 Tables Format Check goes red until they are re-formatted. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
7a9a9f10af |
ci: invoke pip via python -m in s3tests clone steps (#12010)
* ci: invoke pip via python -m in s3tests clone steps The self-hosted runner reaches the Clone s3-tests step with the setup-python toolcache interpreter on PATH but no pip shim, so the bare 'pip install' exits 127. python -m pip resolves through the interpreter that is already on PATH, matching the helm_ci workflow. * ci: invoke pip via python3 -m in s3tests clone steps The adjacent fix_s3_tests_bucket_conflicts step already runs python3, so python3 is the interpreter guaranteed on the self-hosted PATH; a missing python alias would fail the same way the missing pip shim did. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
ecd778a4a0 |
iceberg maintenance: a merged file with null row ids names no first_row_id (#11973)
* iceberg maintenance: a merged file with null row ids names no first_row_id
Compaction stamps a merged v3 data file with the _row_id of its first row as
first_row_id (mergedFileFirstRowID). When the bin mixed rows that carry a
materialized _row_id with rows that carry none (a data file with no
first_row_id, as a v2 table's files are after an upgrade to v3, gives null
ids), the null rows then inherit first_row_id + their position in the file:
ids that other rows already hold, so two rows answer to the same row id.
A merged file now names a first_row_id only when its footer shows no null
_row_id in any row group. Otherwise the commit assigns the file a fresh range
from the table's next-row-id: the null rows get new, unused ids and the
materialized ones keep theirs (a materialized value overrides inheritance).
Test: TestMergedFileFirstRowIDOnlyWithoutNullIDs ("materialized first, a null
later" fails without the fix).
* iceberg maintenance: name no first_row_id when the merged file has a null _row_id
mergedFileFirstRowID stamped the merged file with its first row's id
without checking later rows. A null _row_id in a merged file inherits
first_row_id+position, an id another row already holds, so the commit
produced two rows answering to the same row id.
Check the footer's null counts for the _row_id column across row groups
first: any null means the file is committed without a first_row_id, so its
nulls take fresh ids from the table's next-row-id while materialized
values still override inheritance.
* shell: retry filer discovery until master knows a filer
A shell started against a freshly booted cluster (weed mini, the
s3tables/lance test harnesses) can list cluster nodes before the filer's
first heartbeat lands on the master. The empty list left FilerAddress
unset and the next command died with "dial tcp: missing address". Poll
ListClusterNodes until a filer appears or the connect timeout expires.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: discover the filer on first use instead of at startup
The startup retry loop made every weed shell invocation wait out the
connect timeout on clusters that have no filer, even for commands that
only talk to the master. Discovery now runs once, on the first command
that actually needs a filer, keeping the retry-until-registered fix for
the fresh-cluster race without stalling master-only commands.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: stop filer discovery retries on non-transient errors
A master that keeps answering ListClusterNodes with a permanent error
(permission denied, unimplemented) cannot gain a filer by retrying, so
discovery returns instead of polling until the deadline.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
|
||
|
Chris LuandDevin
|
6a34d82f05 |
shell: retry filer discovery until master knows a filer (#11989)
* shell: discover the filer on first use instead of at startup The startup retry loop made every weed shell invocation wait out the connect timeout on clusters that have no filer, even for commands that only talk to the master. Discovery now runs once, on the first command that actually needs a filer, keeping the retry-until-registered fix for the fresh-cluster race without stalling master-only commands. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: stop filer discovery retries on non-transient errors A master that keeps answering ListClusterNodes with a permanent error (permission denied, unimplemented) cannot gain a filer by retrying, so discovery returns instead of polling until the deadline. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: explain an empty filer list when discovery times out * shell: retry filer discovery while the address is still empty Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
dependabot[bot]anduser.email
|
b86b2f3cee |
build(deps): bump github.com/go-chi/chi/v5 from 5.3.1 to 5.4.0 (#12004)
Bumps [github.com/go-chi/chi/v5](https://github.com/go-chi/chi) from 5.3.1 to 5.4.0. - [Release notes](https://github.com/go-chi/chi/releases) - [Changelog](https://github.com/go-chi/chi/blob/master/CHANGELOG.md) - [Commits](https://github.com/go-chi/chi/compare/v5.3.1...v5.4.0) --- updated-dependencies: - dependency-name: github.com/go-chi/chi/v5 dependency-version: 5.4.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
|
57a122fbcf |
filer.remote.sync: dispatch an update event by the side of the mount it touches instead of slicing the mount prefix off a path outside it (#11958)
* filer.remote.sync: dispatch an update event by the side of the mount it touches instead of slicing the mount prefix off a path outside it The daemon subscribes to its mount prefix and to /etc/remote. An update event whose path lies outside the mount still reaches processUpdateEvent: an update of the /etc/remote directory entry itself (which fs.meta.load of a mapping dump re-creates) through the /etc/remote prefix, or a rename across the mount boundary through the path on the inside. processUpdateEvent then maps both paths to remote keys by slicing the mount prefix off, which panics with "slice bounds out of range [20:11]" on a path shorter than the mount and names a key in the wrong tree on a longer one. The event sits in the replay window, so every restart dies on it and the persisted offset never moves. Before dispatch, an update event is now mapped onto the side the mount covers: a rename into the mount is the creation of the new entry, a rename out of it the deletion of the old entry's object (kept when the entry is remote-only and the object holds its only copy), an event with neither path inside is not the mount's and is dropped, and an update with both paths inside is unchanged. * filer.remote.sync: extract processCreateEvent Moves the create branch of eachEntryFunc into a function, matching processDeleteEvent and processUpdateEvent, so the create path can be exercised directly in tests. No behavior change. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.remote.sync: upload a create reshaped from a rename into the mount A file synced under the mount keeps its RemoteEntry across renames. The rename out deletes the remote object, and the rename back in reshapes to a create whose inherited stamp still matches the entry, so shouldSendToRemote skipped the upload and the new key stayed empty. The reshaped create now sends unconditionally: the stamp describes the old key, not this one. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.remote.sync: a create reshaped from a rename into the mount writes an empty entry as a zero-byte object * filer.remote.sync: a create reshaped from a rename sends a stamped entry truncated since its sync A stamped entry still reports remote-only once truncated to zero (the stamp's RemoteSize stays positive), so the remote-only exemption skipped it and the new key got no zero-byte object. Gate the exemption on isDroppedLocalCopy: only an entry whose size still matches its stamp has its content already at the new key via the rename's remote copy. * filer.remote.sync: recover a superseded create reshaped from a rename-in A create reshaped from a rename into the mount whose write is superseded was skipped like a plain create, but no later event covers this destination: remote.uncache accepts the rename's inherited RemoteEntry stamp and drops the local chunks, and the update it logs skips the upload through isDroppedLocalCopy, while the source mount's sync still deletes the source object. The destination object is then missing and the content exists nowhere. recoverSupersededCreate finishes the event from the entry the filer holds now: a rewritten entry is left to its own event's upload or uploaded here on the stamp edge, and a remote-only entry is completed by copying the object its stamp describes -- resolved through a pending-rename marker under this mount, or through the filer's mount mappings to the source mount's remote location. --------- Co-authored-by: james <james@medable.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
5e01caa522 |
master: cap parallel vacuums per rack via master.vacuum.max_parallel_per_rack (#11935)
* master: cap parallel vacuums per rack via -maxParallelVacuumPerRack Hosts that run one volume server per disk under a single rack label could not bound host-level vacuum I/O: maxParallelVacuumPerServer limits each process, so every disk on the machine compacted at once. A second quota map keyed on rack id debits alongside the per-server one, so at most N volume copies compact in parallel within a rack. Defaults to 0 (no limit); the flag is wired on master, server, mini and master.follower. * master: resolve a data node's rack key once per vacuum sweep A volume server that unregisters mid-sweep loses its parent node, so the quota credit path calling dn.GetRack() panicked and crashed the master. Rack names are also local to their data center, so same-named racks in different centers shared one quota bucket. Each node's rack key is now captured when the limiters are built and carries the data center id, so checks, debits and credits never touch the live parent pointer. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: reserve vacuum quota per needed copy count, not per copy Each copy of a volume checked the shared server/rack quota independently, so several same-rack replicas all passed against one remaining slot and the debits drove the bucket negative. Needs are now counted per server and per rack first; a volume whose own copies exceed a cap reserves the whole bucket once it is free instead of pending forever. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master.follower: register -maxParallelVacuumPerRack as a flag It was only a fixed default, so passing the flag on the command line failed flag parsing. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: key the rack vacuum quota by rack node identity Joining data center and rack names with "/" still let two different racks share one bucket when the names themselves contained slashes. Key the limiter by the rack node instead, which is unique per rack regardless of naming. * master: keep the tiered-replica vacuum test through the rebase * master: document 0 as no limit for -maxParallelVacuumPerServer The clamped per-server need makes a non-positive cap mean "no limit", matching the new rack flag; say so in the flag help. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: key the rack-key cache by data node identity Node ids are only unique within a rack, so two servers that share an id in different racks would have reused the first server's cached rack and escaped the per-rack vacuum cap. * master: read the per-rack vacuum cap from master.toml -maxParallelVacuumPerRack becomes the master.max_parallel_vacuum_per_rack config key, so the new master limit is set the same way as other master options instead of a flag wired into every command that embeds a master. * master: let master.max_parallel_vacuum_per_server override the flag The released -maxParallelVacuumPerServer flag keeps working, but a value in master.toml now wins when set, matching how master.mdir already overrides -mdir. * master: group the vacuum caps under master.vacuum in master.toml Rename master.max_parallel_vacuum_per_rack to master.vacuum.max_parallel_per_rack and master.max_parallel_vacuum_per_server to master.vacuum.max_parallel_per_server, so the keys sit in one subsystem table like master.volume_growth and master.maintenance. * master: deprecate -maxParallelVacuumPerServer for master.toml The flag still works - and is overridden by master.vacuum.max_parallel_per_server when set - but the help text now points operators at the config key, matching the codebase style of help-text-only deprecation. * topology: read node parent atomically UnlinkChildNode clears a data node's parent under the rack's lock while readers like the vacuum sweep's rack capture reach it with no shared lock, so a torn interface read could crash the master. Store the parent in an atomic.Pointer and nil-guard the detached walk in DataNode.GetTopology. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
a1b3131bf6 |
test: handle unsupported catalog clients in S3 Tables CI (#12023)
test: distinguish unsupported catalog clients from S3 Tables failures |
||
|
Chris Lu
|
aba0cb7b2a | ci: isolate S3 lifecycle mini from LAN clusters (#12022) | ||
|
Chris Lu
|
81e0859c12 |
test(s3 remote cache): wait for the master to list the filer before shell commands (#12018)
A fresh single-filer mini registers its filer only after the ~10s empty-membership confirm window, and weed shell locates the filer via the master's listing, so remote.configure sent right after the S3 health check races it. Ports the wait from the enterprise test to the OSS copy. |
||
|
Chris LuandDevin
|
0d00e92178 |
iceberg maintenance: match percent-decoded keys when checking file references (#11978)
* iceberg maintenance: match percent-decoded keys when checking file references Writers may percent-escape partition directory names in the file paths they record in manifests (e.g. "part%3D1"), while the S3 layer decodes the key before storing the object. remove_orphans compared the escaped reference to the decoded filer path, counted the live file as an orphan, and deleted it once it passed the age window — readers then got NoSuchKey on the next scan. Match both spellings of every referenced path in remove_orphans, apply the same coverage to the kept-file set in expire_snapshots, and delete expired files at their decoded key so the object the S3 layer stored is the one removed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: retry filer lookups at the percent-decoded key A manifest reference carrying an escaped segment resolves to a filer name that does not exist, because the S3 layer decoded the key on write. When a lookup misses and the reference contains escapes, retry once at the decoded spelling so maintenance can still read the file. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: resolve referenced files verbatim-first, decoded fallback filer_pb.LookupEntry reports misses as ErrNotFound rather than a gRPC NotFound, so the escaped-reference retry never fired against a real filer. Both maintenance reads and expired-snapshot deletes now resolve through one verbatim-first/decoded-fallback lookup, and expiry refuses to delete a resolved path outside the table's data directory. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: test expiry of a file stored at a literal percent key * iceberg maintenance: resolve referenced files in one filer lookup lookupFileByIcebergPath re-fetched the entry resolveFilerPath had just looked up, doubling the RPC per referenced file during compaction and expiration. The resolver now returns the entry alongside the path so a hit costs one lookup and a decoded fallback two. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: keep every decoded spelling of a referenced path The kept-reference sets recorded a path plus a single decode of it, but lookupResolvedEntry decodes recursively until the spelling is stable. A doubly-escaped reference could resolve to a spelling absent from the set and be deleted while still referenced. addFilerPathSpellings now walks the same decode chain the resolver does. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: keep every decoded spelling of a referenced path * iceberg maintenance: test expiry keeps a doubly-escaped kept reference * iceberg maintenance: clean decoded spellings in the referenced-path set * iceberg maintenance: test expiry keeps a kept reference resolved via decoded parent segment * iceberg maintenance: decode only the reference in the spelling set Decoding the composed filer path also decoded the table directory, so a table at a literal percent-escaped path produced a kept set missing the spelling the resolver returns, and expiry could delete a live file. * test: gofmt the s3tables cluster helpers Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
8541f959a9 |
filer.conf: merge fsync on presence like worm (#11953)
* filer.conf: merge fsync on presence like worm fsync is a pure preference, so OR-merging it meant a nested rule could never turn a parent's fsync off. Make PathConf.fsync optional and merge it most-specific-wins, matching worm: an unset rule inherits, an explicit value overrides in either direction. readOnly and disableChunkDeletion stay OR'ed — they are fail-closed locks quota enforcement depends on. Pre-version-2 configs were written with EmitUnpopulated, so every rule carries an explicit fsync=false that meant nothing; it is dropped to unset on load so it does not read as an opt-out. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.conf: test fsync override survives a save/reload round trip --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
673d1a02e0 |
operation: euthanize pooled upload conns that stall far below a fast host (#12003)
* operation: euthanize pooled upload conns that stall far below a fast host A pooled keep-alive conn can wedge and drain the request body at a trickle while fresh conns to the same volume run at full rate; every later upload routed onto it stalls for tens of seconds, indefinitely. Track per-host healthy-rate uploads (>=1MiB/s on a >=256KiB body) and, on a host that was fast within the last minute, cancel an upload still in flight past max(10s, size/256KiB/s): the transport discards the wedged conn and the existing retry dials a fresh one. A watchdog kill also clears the fast mark so the retried attempt runs unbounded, and hosts that never prove fast keep today's unbounded behavior - a chronically slow link is never penalized. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * operation: keep the stall watchdog armed through the response read uploadDone was set as soon as Do() returned, so a conn that accepted the request body then wedged on the reply escaped the watchdog and could park the upload for hours — the same failure this watchdog exists to kill. Deferring the done-store keeps the timer live until the function returns, so a stalled io.ReadAll is cancelled and retried like a stalled write. The redundant post_err recheck goes with it. * operation: pin watchdog coverage of a stalled response read * operation: disarm the stall watchdog for an upload's remaining attempts Once the watchdog has cancelled an attempt, a concurrent healthy upload to the same host can restore the fast mark during the retry backoff. The re-armed watchdog would then cancel the retry too, and a large upload that merely exceeds the floor could burn every attempt. Track the watchdog kill on uploadStatusError and carry a stallWatchdogDisarmed flag through retriedUploadData so the rest of that upload's attempts run unbounded; other uploads to the host keep their watchdog. * operation: pin watchdog disarm on retry, fresh-dial, and slow-upload cases * operation: drop the send-stall watchdog once the request body drains A wedged pooled conn is only blameworthy while it is still trickling the request body; once the body is on the wire the wait is server-side commit, which the transport's response header timeout already bounds. Cancelling then would fail a healthy write whose replication or fsync is simply slow. Track body EOF through a wrapping reader, gate the send-side timer on it, and re-arm the watchdog only around the response-body read, which the header timeout does not cover. * operation: prove a host fast only on a verified upload A quick rejection moves bytes no faster than the wire did, so mark the host from the body-send duration and only after the response checks out. * operation: count an abandoned request body as sent The transport closes a request body it never finished draining when the server answers early, so EOF alone under-reports the send boundary and the stall watchdog could still blame the conn during the reply wait. * operation: scope the body-sent mark to the current attempt A lagging Close or EOF from the first attempt's body could stamp the shared mark after the inner retry reset it, disarming the watchdog for a send that never happened; each attempt now points the watchdog at its own sent timestamp. * operation: drop only the fast mark the stall timer observed A concurrent upload can re-prove the host while a stalled attempt is in flight; deleting the key outright would erase that newer mark, so the watchdog now removes only the value it armed against. * operation: keep the stall disarm on a per-call upload option retriedUploadData marked the caller's UploadOption, so a caller reusing it across UploadData calls would start the next upload with the watchdog off and lose the cancellation this adds. * test: stall watchdog retry must fail if the conn gets cancelled The fake retry slept through any cancellation, so a re-armed watchdog firing mid-retry still let the upload succeed — the guard was untested. Park on the request context like a real wedged conn so a re-armed watchdog cancels the attempt and fails the test. * operation: prove the fast-host mark with elapsed-to-response Body-drain time only measures when the transport consumed the reader: kernel socket buffers can swallow a small body instantly on a slow link, inflating the rate and marking the host fast. The next upload then gets its conn cancelled mid-send despite making progress. Elapsed-to-response bounds remote speed from below — the server cannot answer before the body lands — so the mark can only under-mark, never over-mark. * operation: scale the stall budget off the host's measured rate Fixed bps limits fit no link well: 256KiB/s never fires on a slow link and waits far too long on a fast one. Record the wire rate of each sizeable upload and let a later upload run for slowFactor times the body's span at that rate — a wedged conn shows as an order-of-magnitude drop on any link speed. The recorded rate uses the wire body size so a pre-compressed payload no longer inflates the mark. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
6e8a2cdb14 |
filer.backup: process metadata events through the concurrent processor (#11951)
* filer.backup: process metadata events through the concurrent processor Each event was handled inline on the subscription loop, so one fetch+write at a time capped the stream at a few entries per second and a busy source kept building lag. Route events through the same MetadataProcessor filer.sync uses: it fans jobs out behind -concurrency while serializing same-path and subtree-barrier conflicts, and the checkpoint now advances on the processor's contiguous watermark via GetResumeTsNs, with Resubscribe replaying anything a failure pins. -disableErrorRetry keeps its log-and-skip meaning by returning nil after logging, since the processor would otherwise pin the checkpoint forever. * filer.backup: drain in-flight jobs before the stream retry loop reconnects FollowMetadata returns as soon as the stream ends while admitted jobs keep writing; the next doFilerBackup builds a fresh processor that cannot order its replayed writes against them, so an older upload could finish last and clobber a newer file. * filer.backup: checkpoint the completed watermark on a ticker and at stream end The offset callback only fires while events arrive, so work draining on a quiet stream stayed uncheckpointed until the next event and a restart recopied it. Persist the processed watermark every 3s when it advances, and once more after the post-stream drain. * filer.backup: bound the checkpoint wait before reconnecting setOffset has no context, so a stalled KvPut would hold persistDone and block the retry loop forever. The persist goroutine now performs the final write itself on stopPersist, and the caller waits at most 30s. * filer.backup: carry the processed position across retry attempts A follow attempt that ended with a pinned failure recomputed startFrom on retry, so a -timeAgo window could move past the failed event and skip it. doFilerBackup now returns the failure pin (or processed watermark) and the retry loop clamps startFrom to it. * filer.backup: resume retries at the previous attempt's pinned position The retry loop threaded resumeTsNs through but never consulted it, so a -timeAgo run recomputed startFrom as now-minus-the-window on every attempt — a failed event older than the window was skipped on reconnect. Use the pinned position when one exists; -timeAgo and the persisted checkpoint only choose the first starting point. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: resume below the failure pin so the failed event replays The metadata reader treats the resume position as exclusive — events at or below it are skipped — so resuming at the failed event's own timestamp would skip it. Back one tick off the pin. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: label the progress line as the received-event rate The counter increments when a job is admitted, not when its copy finishes, so the old wording showed backup throughput during a burst of slow copies while the processed watermark had not moved. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: bound the offset checkpoint RPC setOffset called KvPut on the background context, so a stalled write could pin a caller's drain or persistence loop indefinitely. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: keep the resume pin when filer-source setup fails A retried attempt carries resumeTsNs so the failed event replays from the same position, but filer-source and SSE initialization failures returned 0, so the next attempt recomputed -timeAgo and could move the window past the pinned event. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: serialize checkpoint writes per key A checkpoint write can outlive its caller's wait and land beside the next subscription attempt's writes, moving saved progress backward. Serialize writes per key inside setOffset and drop any value older than one already written from this process. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: gofmt * replication: serialize checkpoint writes per filer and key A single process-wide lock made a stalled checkpoint RPC block every stream's checkpoint, including the opposite direction of a bidirectional filer.sync. Keeping the last-written watermark per persisted key also let one stream's offset shadow another stream checkpointing the same key to a different target filer. Lock and track progress per (filer, key). * replication: keep the resume position behind a cleared failure pin A redelivery that clears failedTs lets the processed watermark advance past the earliest failure, but events dropped while the processor was stopped — possibly sharing the failure's timestamp — have to replay. Latched earliestFailedTsNs floors the resume position one tick below that failure so reconnects still cover them. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: persist floored checkpoints through the write ordering setOffset drops a value older than the last written, which also swallows a deliberate regression: the resume floor, an operator-seeded offset, or a snapshot high-water mark. allowRegress marks those callers so the checkpoint can move backward when a failure forces replay. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: order checkpoint writes by issuance, not value The value-based stale guard could never fire — every caller passed allowRegress because floors and seeds are deliberate regressions — and ordering by value cannot tell a stale high watermark apart from real progress: a write that outlives its attempt could still land after a floored resume and skip events owed a replay. Order by issuance instead: a write issued before the last one to land under the key is dropped, so late writes from a superseded attempt can no longer move the checkpoint in either direction while deliberate regressions still apply. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: keep the resume position one tick below the frontier * remote.sync: resume and persist through the floored watermark * test: concurrent metadata jobs fan out up to the limit Distinct-path events must all be in flight before any finishes — a serial loop could only ever have one running — while the limit still bounds the pool. Pins the property behind the backup-throughput fix. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
cd562b8514 | docs: regenerate star history chart | ||
|
Chris LuandDevin
|
617599859b |
docker: build RocksDB v11.1.2 for grocksdb v1.11.1 (#12002)
The rocksdb release image fails to compile: grocksdb v1.11.1 calls C.rocksdb_block_based_options_set_block_align, C.rocksdb_block_based_options_set_index_block_search_type, and C.rocksdb_block_based_options_set_separate_key_value_in_data_block, all of which only exist in RocksDB v11.x headers. grocksdb v1.11.1's own build script pins 11.1.2, so move every default from v10.10.1 to v11.1.2 to match. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
ce4c1fe31e |
ci: install fuse3 when runners lack fusermount3 (#12013)
Self-hosted runners do not ship fuse3, so the setuid repair step failed with 'no fusermount3 in PATH' before the FUSE jobs could run. Install the package instead of failing outright. |