mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-07 04:27:49 +01:00
39ef551bcfed8db8ae6476649e8ca42ba824fcc5
1685
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
39ef551bcf |
filer.remote.sync: keep the resume position for the retry when its write fails
The retry in the same process now resumes from the position the ended subscription returned, as filer.backup does, so a failed offset write followed by an empty read cannot recompute -timeAgo past a failed event. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a0d60ee00f |
filer.remote.sync: say that any saved offset wins over the creation time
The creation time in seconds was compared with the offset in milliseconds, which made any saved offset win. That is the behaviour to keep: remote.unmount recreates the directory and then saves the offset, usually in the same second, so a seconds comparison would start at the creation second and replay the purge. The comparison now states it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
7be919ded1 |
filer.remote.sync: save the resume position when a subscription ends
A failed event ends the subscription with ErrResubscribe, and the retry loop runs startup again, recomputing -timeAgo from a later now. The periodic checkpoint only runs as events arrive, so a failure before the first one could start the retry after the event it failed on. The resume position is now saved whenever the subscription returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
f535fca8f6 |
filer.remote.sync: take the start time before the mount lookup
A bucket created between a not-found lookup and the end of the offset read had its creation before a start taken afterwards, so the gateway never created its remote bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
c31497a9a3 |
filer.remote.sync: a directory not created yet starts now, as before
A gateway started before the first bucket looked up a missing buckets directory, had no saved offset, and returned an error to the retry loop on every attempt until a bucket appeared. A missing directory with no saved offset has nothing to replay, so it starts now as it did before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
1f332e5866 |
filer.remote.sync: stamp a copy of the event entry
updateLocalEntry set the RemoteEntry and dropped the rename marker on the caller's entry before the RPC. A transient stamp failure retries the same event in place, which then read as already synced and skipped both the upload and the stamp. The stub filer now applies a successful update the way the filer does, since tests had relied on that shared object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
0a1b6c2944 |
filer.remote.sync: an unread offset never resolves to the mount's creation time
With the offset read now bounded, a timed-out read fell through to the directory's creation time and replayed every event already synced. The error goes back to the startup retry loop, as with -timeAgo; a missing offset still reads as zero without an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
c7338fb68f |
filer.remote.sync: bound the stamp written after each upload
updateLocalEntry's UpdateEntry was the last filer call on the sync path without a deadline. A filer that never answers the stamp keeps the job and its path in the conflict index the same way an unanswered lookup did. The stamp carries IF_ENTRY_EQUAL, so a retried event after a timeout either stamps again or skips a stamp that already landed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
51f603b5ec |
filer.remote.sync: isSuperseded returns the lookup error itself
Its only production callers needed the error, so the boolean wrapper was left to the tests. The tests now assert the error on a failed lookup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
952d80c4d3 |
filer.remote.sync: keep doc comments on the functions they describe
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
james
|
da377157ef |
filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting
resolveLastSyncOffset returns the lookup or offset-read error; both callers (directory and buckets) hand it to util.RetryUntil, which retries with backoff instead of the process exiting. |
||
|
james
|
2bcb748a67 |
filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo
After a failed upload attempt the supersession lookup's error ended in isSuperseded's boolean and the loop could upload the snapshot again unverified. The lookup error now ends the loop (errLookupUnanswered, not retried) and the processor retries the event. With -timeAgo set, a failed read of the saved offset resolved to now minus timeAgo, skipping everything older; the daemon exits instead so a restart retries. (cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566) |
||
|
james
|
db4ea51fd2 | filer.remote.sync test: keep the slices import the other gateway tests use | ||
|
james
|
e4c0acfa61 |
filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read
retriedWriteFile returns the supersession lookup's error instead of treating an unanswered lookup as "not superseded": in a replay an older snapshot could otherwise overwrite a newer remote object. The processor retries the event. isSuperseded keeps its boolean form for the other call sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls back to the saved offset cannot hang on it. TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed lookup; it now asserts the withheld upload and the lookup error. (cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa) |
||
|
james
|
4f8f817461 |
filer.remote.sync: a failed mount lookup at startup never resolves to now
collectLastSyncOffset returned time.Now() when the mounted directory lookup failed, which skips every event since the saved offset. The saved offset is used instead; without one the daemon exits so a restart retries. (cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae) |
||
|
james
|
3b8c4702bd |
filer.remote.sync: bounded filer lookups, one chunk cache per process
The filer lookups on the sync path (currentEntry, isSuperseded, uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no deadline. A lookup that never answers keeps its job alive, the job keeps its path in the MetadataProcessor conflict index, AddSyncJob waits on the condition and the daemon stops consuming events with no log line. Each lookup now carries a 30 s deadline, so the job fails and the processor's stop-and-resubscribe path runs. remoteUploadReader built a chunk cache per upload; the cache's worker goroutine is never stopped, so each upload leaked one goroutine and its cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory limit). One chunk cache per process. (cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78) |
||
|
|
80f32bc074 |
weed/command: skip remote bucket delete when bucket is unmounted (#12418) (#12425)
* weed/command: skip remote bucket delete when bucket is unmounted * remote_storage: count deleting an already-missing bucket as done filer.remote.gateway can see the same bucket delete more than once, for example once per filer when several filers peer their metadata. A copy that still finds the mount mapping calls DeleteBucket again, and S3 (NoSuchBucket), Azure (ContainerNotFound, ContainerBeingDeleted) and GCS (404) all failed it, which held the sync offset on the delete. A missing bucket now counts as deleted, the way deleting a missing object already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: keep the mapping table writable after mount.mapping is deleted Deleting mount.mapping replaced the gateway's mappings with a table whose map was nil, so the next bucket create panicked writing its mount into it. Reset to an empty map instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: count a delete of an absent object as done filer.remote.sync already treats ErrRemoteObjectNotFound from DeleteFile as deleted, since GCS reports a missing object that way where S3 and Azure succeed. The gateway returned it, so a delete event seen twice, as it is when several filers peer their metadata, held the sync offset on GCS. Use the same deleteRemoteFile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: drop cached remote clients after the new gateway tests GetRemoteStorage caches clients by configuration, so a repeated run (-count=2) got the first run's recorder back and saw no calls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: keep the default storage across mount.mapping changes The default storage for new buckets is derived at startup and kept only in the gateway's copy of the mappings; mount.mapping never stores it. Replacing or deleting that file dropped it, so a later bucket create with no -createBucketAt skipped the remote bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
c971252871 |
admin: make the cluster lock's fairness window configurable (#12426)
* admin: make the cluster lock's fairness window configurable Scheduled jobs hold the cluster admin lock for their whole run. Once the lock had been held for a minute, every new job waited until all running jobs had released it, so with hour-long EC jobs dispatch ran in waves and most workers sat idle between them. -adminLock.fairnessWindow keeps the one-minute default; 0 lets new jobs join the shared hold, and shell clients get the lock when the scheduler next drains. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * admin: wait for the blocked acquire before disabling the fairness window The wake-up test slept 50 ms before SetFairnessWindow(0), so a late acquire that started after the change joined without exercising the Broadcast. Wait until it is registered as a waiter instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
120d17a092 |
filer.remote.sync: stop a touched file from being restamped forever (#12419)
* filer.remote.sync: stop a touched file from being restamped forever A metadata-only update that moves the mtime past RemoteMtime (touch, or an O_TRUNC open of an empty lock file) goes through UpdateFileMetadata, which changes nothing on the remote, and the entry is stamped with the same RemoteEntry and a new LastLocalSyncTsNs. That stamp is an update event of its own. shouldSendToRemote only looks at LastLocalSyncTsNs when RemoteMtime and the mtime share a second, so it still reads the entry as dirty, and the event is stamped again. Nothing is uploaded, but the daemon keeps rewriting the entry as fast as it reads its own events back. Skip the restamp when the backend left the object alone and the event's entry already carries a stamp taken after its mtime. The touch itself is still stamped once. * filer.remote.sync: log the skipped restamp The other skips in processUpdateEvent log at V(2); this one did not, so with -v=2 a touched file showed an "update meta" line with nothing after it. * filer.remote.sync: skip the restamp of a stamp's own event, whatever the clocks The skip compared the entry's stamp with its mtime, so an mtime ahead of the sync's clock (a skewed writer, touch -d) stayed past every stamp and kept the loop going until the clock caught up. An event that changed only the RemoteEntry is the sync's own stamp, which needs no other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: ignore the filer's commit timestamp when spotting a stamp The filer gives every stored update a new CommitTsNs, so a stamp's event never compared equal outside the RemoteEntry and a touched file kept being restamped against a real filer. The test now assigns one per update. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
eeb8aefe70 |
weed-vector: gRPC TLS to the masters and for the gateway (#12368)
* weed-vector: read the cluster's security.toml The engine finds security.toml where weed processes do (., ~/.seaweedfs, /usr/local/etc/seaweedfs, /etc/seaweedfs) with WEED_* environment variables over it, and builds gRPC client and server credentials by the rules of security.LoadClientTLS and LoadServerTLS: a dedicated client certificate is preferred, both halves of a pair must be set, and grpc.ca is required once either is. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: gRPC TLS to the masters and from the gateway The masters are dialed with [grpc.client]'s credentials when the cluster has them, the master's certificate checked against the host dialed or --tls-server-name. The engine's own listener takes its identity from [grpc.vector] and then requires a client certificate chaining to grpc.ca, which is how the gateway, dialing with [grpc.client], reaches it. The rustls provider is pinned before the first TLS use, as the other binaries linking both aws-lc-rs and ring do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * scaffold: document [grpc.vector] Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: put every gRPC hop under mutual TLS in the hardened leg VECTOR_TLS=1 now also writes [grpc.master], [grpc.volume], [grpc.filer], [grpc.s3], [grpc.client] and [grpc.vector], and probes the master's and the engine's gRPC ports: plaintext HTTP/2 and a client without a certificate are refused, the client certificate gets a SETTINGS frame. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: give the test volume server 64 volume slots Every collection grows 7 volumes and test_vector_bucket_encryption opens two buckets while the previous test's bucket is still being torn down, so 16 slots were marginal: the Go legs failed CreateIndex now and then with "failed to find writable volumes" and master.log showing "create 7 volume, created 4: Not enough data nodes found!". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: reload the listener's certificates and enforce allowed_commonNames The listener now runs the way weed servers run theirs. The certificate, key and CA files are re-read every WEED_TLS_CERT_REFRESH_INTERVAL (weed's default of 5h) and on SIGHUP, and new handshakes use the changed files without a restart; a file that no longer parses is logged and the working configuration stays. [grpc.vector] allowed_commonNames and grpc.allowed_wildcard_domain restrict which client certificates are served, checked against the subject common name as security.Authenticator and the Rust volume server's CommonNameVerifier do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * scaffold: allowed_commonNames and the refresh cadence for [grpc.vector] Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the engine's allowlist keeps an outsider certificate out The hardened leg sets [grpc.vector] allowed_commonNames = "localhost" and probes the engine with a second certificate the same CA signed for CN=outsider: the handshake must be refused, while the gateway's certificate still gets an HTTP/2 SETTINGS frame. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: take the master's TLS name from the parsed endpoint A master given without a port (https://master.example) had the whole URL checked against its certificate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: bound the master TLS handshake connect_timeout covers only the TCP connect, so a master that accepted the connection and never answered the handshake held up the other masters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: look up security.toml keys the way viper does An empty WEED_* variable no longer hides the file's value, and keys match regardless of case, as for weed processes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: refuse master allowlists it cannot enforce weed checks a dialed server against allowed_server_commonNames and allowed_server_wildcard_domain on top of grpc.ca. The engine's master dial accepted every certificate the CA signed even when one was set; it now fails at startup and names the key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: read the whole frame header in the gRPC TLS probe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: probe the volume, filer and S3 gRPC listeners too Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read the master dial credentials only when given masters A standalone engine never dials with grpc.client, so its settings no longer stop it from starting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: run each listener TLS handshake on its own task The accept loop awaited every handshake before accepting the next connection, so a peer that connected and sent nothing stopped every gateway connection after it. Handshakes now run concurrently, each under a 10s timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read the master dial's TLS files at every dial The client certificate, key and CA were read once at startup, so after the files rotated a reconnect still presented the old pair. Each dial now reads them, as the listener re-reads its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read every unit Go durations accept in the refresh interval WEED_TLS_CERT_REFRESH_INTERVAL values in ns, us or a leading + fell back to five hours in the engine while weed servers honored them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: fall back on a refresh interval too long to hold An oversized WEED_TLS_CERT_REFRESH_INTERVAL panicked the engine at startup; weed servers reject it and keep the default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: refuse allowed_commonNames without the listener's certificate With [grpc.vector] allowed_commonNames set but no cert and key, the listener served plaintext and checked no client at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read the master dial's credentials inside its connect timeout The CA, certificate and key were read synchronously before the timeout started, so a stalled credential path held up every other master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: let a set key win over an empty spelling of it With allowed_server_commonNames = "" beside allowed_server_commonnames = "master", the lookup stopped at the empty exact match and the allowlist went unnoticed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
c2ca83c0bf |
filer: say jwt.filer_signing.key gates only the IAM gRPC service (#12353)
* filer: say jwt.filer_signing.key gates only the IAM gRPC service The key never covered the rest of the filer gRPC port, which can read and write the IAM store under /etc/iam. Point operators at [grpc.filer] mTLS for a port reachable beyond a trusted network instead of the key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: note the IAM gRPC OIDC provider and role RPCs are refused without the key Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b7b2a5947f |
admin: configurable file browser upload size limit (#12348)
* admin: carry a file browser upload size limit The admin server keeps the total size limit of one file browser upload, 500 MB by default as the UI enforced before, and hands it to the file browser page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: reject file browser uploads over the size limit The 500 MB limit was only checked in the browser, so any client could post larger uploads and fill the temp dir with spooled parts. Cap the request body at the limit plus room for the multipart framing, and answer 413 when the files add up to more than the limit. Parse the form before reading the path field. FormValue used to parse it first with the default 32 MB memory, which made the later 1 GB ParseMultipartForm a no-op; keep 32 MB explicitly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: show and check the configured upload limit in the file browser The upload dialog reads the limit from the page instead of a hard-coded 500 MB, lists it under the file picker, and skips the check when uploads are unlimited. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: add -uploadMaxSizeMB to set the file browser upload limit weed admin -uploadMaxSizeMB and weed mini -admin.uploadMaxSizeMB set the total size limit of one file browser upload; 0 removes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: note why the upload framing allowance is enough Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: test an upload whose files only exceed the limit together Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: remove spooled upload parts when the upload handler returns Parts over 32 MB spool to temp files. The router passes the handler a copy of the request, so net/http never sees the parsed form and never removes those files. Every large upload, and every one rejected for size after parsing, left them on disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
2eb219df61 |
s3vectors: encrypt the objects of a vector bucket declared AES256 (#12324)
* s3api: encrypt the objects of a vector bucket declared AES256 CreateVectorBucket recorded encryptionConfiguration sseType AES256 and returned it, but nothing on the write path read it: the index's data files, manifests and transactions were stored without SSE. A LANCE table bucket's record now supplies the bucket's default encryption whenever the bucket has none of its own, so PutObject, CreateMultipartUpload and CopyObject apply SSE-S3 to every object the engine writes. The S3 bucket metadata itself is unchanged. ICEBERG table buckets keep recording the declaration only: their catalog reads the manifests clients upload straight from the filer, which would see ciphertext. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3tables: refuse an AES256 vector bucket on a gateway without an SSE-S3 key A vector bucket's AES256 is now applied to every index write, and SSE-S3 fails each write when the gateway has no KEK. Creating the bucket there succeeded and the first CreateIndex came back as InternalFailure. CreateTableBucket now refuses a LANCE bucket declaring AES256 when the gateway reports no SSE-S3 key, naming s3.sse.kek and s3.sse.key; CreateVectorBucket returns it as ValidationException. ICEBERG buckets, whose declaration is only recorded, and tooling that sets no checker are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3api: refuse DeleteBucketEncryption on a bucket's declared encryption GetBucketEncryption reports a vector bucket's declared AES256, but DeleteBucketEncryption answered NoSuchBucketEncryptionConfiguration and the declaration kept applying. The declaration is fixed when the bucket is created, so the delete now returns MethodNotAllowed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): keep an inherited WEED_S3_SSE_KEK out of the test cluster The gateway refuses to start SSE-S3 with both s3.sse.kek and s3.sse.key set, so a developer shell exporting WEED_S3_SSE_KEK made the AES256 bucket test fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b9bc42c217 |
s3vectors: vend write credentials to index data only to the vector engine and Lance worker (#12316)
* security: share the filer admin token check CheckGrpcAdminPeer verifies an admin Bearer token inline. Move the check into IsFilerAdminToken so an HTTP endpoint can accept the same token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: recognize the cluster's own components by a signed token A request whose X-Seaweedfs-Cluster-Token header carries an admin token signed with jwt.filer_signing.key comes from a component that holds the cluster key: the vector engine or the Lance maintenance worker. Mark such a request as trusted. It still authenticates as an identity, which stays the principal for vended credentials. A dedicated header leaves the Authorization header to Lance OAuth clients. A gateway with no key trusts no one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: vend write credentials to a vector bucket's indexes only to trusted callers An index in a vector bucket's vector namespace is also a Lance table, so a caller granted s3tables:UpdateTable could describe it, or declare it again, and get credentials that write the index data without passing the vector bucket policy. In that namespace both now vend read-only credentials unless the caller is trusted, and a declare from anyone else no longer writes the dataset markers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance worker: prove the cluster key to the Lance namespace The gateway now vends write credentials to a vector bucket's indexes only to callers that send a token signed with jwt.filer_signing.key. The worker already takes that key for the admin stream; its namespace client now mints a token from it for every catalog request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: describe the index the gateway created and prove the cluster key CreateIndex declared a table the gateway had just created, which only worked because declare is idempotent. Describe it instead. The Lance namespace vends write credentials to a vector bucket's indexes only to callers that send a token signed with jwt.filer_signing.key, so take the key with --filer-signing-key, the same flag the Lance worker has. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance worker: keep the cluster key out of Debug output Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: answer a declare of an existing table with its own location CreateTable succeeds for a name that already exists without looking at the location requested, and declare kept using that location for the dataset markers and the vended credentials. Declaring an existing table with another table's location, such as a vector index, got credentials to that table's data. Use the location the catalog holds, which CreateTable now also reports for a new table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: sign cluster tokens with a key derived for the catalog A cluster token was a filer admin token signed with jwt.filer_signing.key and, by default, never expired. Sent to the Lance namespace on every request, often over plain HTTP, a captured one would also authorize the filer's IAM service. Sign it with a key derived from the cluster key for this one use, which the admin services do not accept, and mint it with a five minute lifetime. weed-vector drops its token lifetime flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: follow a reloaded jwt.filer_signing.key The Lance namespace copied the cluster key at startup. A SIGHUP reload refreshes the gateway's filer guard but not that copy, so after a key rotation the restarted vector engine and worker were no longer trusted. Read the key from the guard on every check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: keep a vector bucket's indexes read-only An index in a vector bucket is also a catalog table the Iceberg REST catalog loads, and LoadTable vended credentials that could write it to any caller granted s3tables:UpdateTable. A commit staged its metadata file in the index directory on the same grant. Apply the vector namespace rule to both, after the caller's own UpdateTable check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: confine a created table's location to its namespace before writing Stage-create skips the registration that confines a location to the table's namespace, yet vends write credentials for that location. A caller allowed to create a table in any namespace of a bucket could name another namespace's prefix, a vector index included, and get write credentials to it. The non-staged create also wrote its first metadata file there before the registration refused the location. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: a declare that finds an existing table answers like describe Creating a name that already exists is idempotent, so a caller allowed only to create tables could declare someone else's table and get write credentials to it, and clear its deregistered marker. Describe would have vended that caller read-only credentials. A declare now looks the name up first; when it only finds the table, writing needs UpdateTable, as it does for describe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
2ea63bad48 |
filer.backup: keep local sink writes inside the sink directory (#12310)
* localsink: skip keys that resolve outside the sink directory Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: reject entry paths with dot segments Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test renames onto dot-segment targets Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: keep the dot-segment rename test inside one bucket scope Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * localsink: resolve keys through an os.Root on the sink directory The lexical check alone lets a symlinked directory that already exists under the sink directory carry writes and deletes outside it. Create, rename and delete now go through an os.Root opened on the sink directory, so a symlink can only resolve inside it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * localsink: follow absolute symlinks that resolve inside the sink directory os.Root rejects every absolute symlink target, so a backup through an absolute link that points back into the sink failed, and a delete through it left the file behind. Resolve the parent of the key first and use the resolved path when it stays under the sink directory. Operations still go through os.Root, so a link swapped after the check cannot escape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
e118ba7127 |
vector: keep S3 Vectors requests off the master-discovery lock (#12291)
* vector: throttle worker discovery from lookup end cli() set nextResolve before resolve() ran, so a lookup that takes as long as the throttle interval (an unreachable first master costs the full 5s dial timeout) left the next lookup already due when it returned. Discovery then ran back to back and, since it holds d.mu, every vector request queued behind it. resolve() now sets nextResolve when the lookup lands, using the short recheck window only while no client is cached. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: resolve workers outside the discovery lock resolve() held d.mu across the master lookup, so while it waited out an unreachable master every vector request blocked on cli(), even with a healthy worker already cached. The lookup now runs in a goroutine without d.mu, and d.resolving keeps it to one at a time. Callers with a cached client use it while the lookup is in flight; callers without one wait for that lookup and share its result. Throttling, the sticky worker and reset(c) are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: keep the cached worker connection when a redial fails resolve() closed d.conn before dialing the replacement worker. A synchronous dial error then left d.client bound to a closed connection. Requests got codes.Canceled, which never triggers reset, and later refreshes saw the unchanged d.worker and never redialed. The old connection is now closed only after the new dial succeeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandDevin
|
e98adfd3bb |
security: fail closed on gRPC TLS errors, verify dialed server identity (#12263)
* security: fail closed on misconfigured gRPC TLS LoadServerTLS/LoadClientTLS returned plaintext credentials on every error path once TLS was configured: a typo in tls.min_version or tls.cipher_suites, cert and key in different directories, or cert/key set without grpc.ca all left the component serving or dialing plaintext. A partially configured TLS setup now stops the process at startup instead; leaving cert/key unset still means no TLS, as before. The loaders also stat cert/key/ca files up front since the pemfile providers read lazily and would otherwise only fail at handshake time. Mirrored in the Rust volume server: partial grpc.volume cert/key, a missing grpc.ca, or a rustls build failure now abort startup rather than warn-and-continue on plaintext. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * security: optional server-identity allowlist for gRPC clients The mTLS client accepts any certificate the configured CA signs, so a per-workload CA lets one component stand in for any other: a filer identity can answer as the master. The server side already screens callers via allowed_commonNames and allowed_wildcard_domain; nothing screened the server. LoadClientTLS now wires AdditionalPeerVerification when the caller configures a server allowlist: grpc.<component>.allowed_server_commonNames (falling back to a shared grpc.allowed_server_commonNames) matched against the peer cert's subject CN, DNS SANs, or URI SANs such as a SPIFFE id, plus a grpc.allowed_server_wildcard_domain suffix match. Leaving them unset keeps the previous behavior. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * security: URI SANs are exact-match only in the server identity allowlist A URI identity whose path happens to end in the wildcard domain would otherwise pass the DNS suffix check. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * security: tighten gRPC TLS fail-closed and server-identity paths - URI SANs match the server-identity allowlist exactly; the wildcard domain suffix check applies only to CN and DNS SANs. - A partially set client_cert/client_key pair is an error when the cert/key fallback is empty, instead of dialing plaintext. - Cert/key/ca files are loaded and parsed at startup (LoadX509KeyPair and CA PEM validation) since pemfile providers read lazily. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: error on a partial grpc client_cert/client_key pair Mirror the Go loadClientTLS path: when client_cert or client_key is set without its mate and the cert/key fallback is empty, fail instead of dialing plaintext. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * security: wildcard domain match requires a valid hostname A CA-signed identity like ".example.com" satisfied the bare suffix check for wildcard ".example.com" despite not being a hostname. Require a real subdomain: non-empty label before the suffix, a '.' boundary when the configured wildcard has no leading dot, and DNS-name syntax. A leading "*." label still matches so wildcard certs keep working. Applied to both the Go server/client authenticators and the Rust volume's client-cert verifier. * security: ignore a trailing root dot in wildcard domain matching * tls: reject a partial dedicated client cert/key pair instead of falling back When exactly one of component.client_cert/client_key is set, warn-and- fallback silently dialed with the serving identity. Fail closed like the other TLS misconfigurations in this change; mirrored in the Rust loader. Regression tests cover a partial pair with a valid fallback. * rust volume: harden partial client cert pair test with real pem files * security: verify dialed server identity per target kind on the filer The filer reused one grpc.filer-verifying dial option for master, volume, filer and s3 targets, so a broad allowlist accepted the wrong component at the wrong port and a narrow one rejected legitimate peers. Outbound calls now pick grpc.<target>.allowed_server_commonNames (falling back to the client component's and the global list) while still presenting the grpc.filer client identity. * security: scope per-target server allowlists under the client component The filer read grpc.volume.allowed_server_commonNames when dialing a volume, but that key already denotes the volume's own outbound allowlist (e.g. the master identity). A deployment restricting volume->master calls would make the filer reject real volume servers. Per-target policy now lives under the client component: grpc.filer.volume_* falls back to grpc.filer.* then grpc.*. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
ad6257ebbc |
filer.backup: add -s3.flattenVersions to mirror the current S3 view of versioned buckets (#12255)
* filer.backup: add -s3FlattenVersions to mirror the current S3 view Versioned buckets are stored as <key>.versions/v_<versionId> files plus a latest-version pointer on the .versions directory, so a backup mirror held internal version files and could never be restored through the S3 API. With -s3FlattenVersions the backup maps events on .versions directories onto the flattened object key: a new latest version uploads to <key>, a latest delete marker or a cleared pointer deletes <key> when -doDeleteFiles is set, and the suspended-versioning null object resolves to the regular path. Version files themselves are never mirrored. The initial snapshot seeds only the version the pointer names. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: serialize flattened writes per object key Same-path non-barrier dir updates run concurrently in the metadata processor, so two pointer updates for one object -- or a pointer update and a regular-path write to it -- could overlap and let the older upload finish last. Lock on the flattened key: versions events lock the object key, every other event its own path, via a striped mutex. * filer.backup: apply exclusions to the flattened object key File-name and path excludes ran against the .versions entry, so an excluded object was still copied flat and its deletion still removed the destination copy. Match the rules against the flattened key on both the apply and the versions-dir-removal paths. * filer.backup: re-resolve the pointer on version-file events A tag or attribute update on the current version touches the v_* file, not the .versions pointer, so dropping every version-file event left the mirror with stale tags until the next pointer flip. Re-resolve the pointer on each version-file event; noncurrent versions still stay out. * filer.backup: tolerate vanished versions dirs in flattened snapshot The flattened seed path returned source errors before the existing missing-source policy ran, so one version deleted mid-walk could force a full re-walk. Apply the same 404/superseded skip as the generic seed branch, and cover the review fixes with tests. * filer.backup: lock both flattened keys an event touches A rename or delete can retire one flattened key while writing another, so serialize on the old- and new-side keys together. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: refresh the flattened key only for the still-current child A v_* event on a non-pointed version cannot move the flattened key, so skip the re-upload; tag and attribute updates on the current version still flow through. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: let resolved-latest writes bypass the source staleness check A filer sink skips a CreateEntry whose source path no longer holds the entry, but a flattened write lands on a key whose source path is not the one it was read from, so the check dropped every flattened upload. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: reclaim destination chunks on flattened deletes Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: route regular-path events for versioned keys through the pointer A null-object write or delete at the regular path can coexist with a .versions pointer that names a live version; the pointer decides the flattened key, so deleting the null object must not remove it. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: make the checkpoint identity cover the flattening mode Resuming a checkpoint taken in the other mode would skip the flattened re-seed and leave mirrored version files behind; a mode change now starts a fresh snapshot. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover flattened regular-path routing and per-mode checkpoints Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: resolve both sides of a flattened rename * test: cover flattened renames with one versioned side * replication: let resolved-latest writes overwrite a newer destination The stale-destination skip trusted a newer destination mtime, but a resolved-latest write can legitimately install an older version over the copy of a since-deleted latest. Skip only on provably identical content (matching chunk provenance, or equal ETag with full byte coverage). * filer.backup: rescan .versions when the latest null object is gone Deleting ?versionId=null removes the regular-path object without touching the null-is-latest pointer, so the flattened key kept its stale copy. Fall back to the newest remaining version (or a flat delete), the same rescan the S3 read path applies. * replication: compare inline and remote content in the destination-holds checks A chunk-less entry keeps its bytes in Content and a remote-only entry in the object RemoteEntry points at, but the identical-content shortcuts only compared chunk lists and chunk-derived ETags. Two different inline files looked identical, so a flattened re-copy or a same-mtime rewrite kept the previous version's bytes. Compare Content bytes and remote object identity wherever the checks claim the destination already holds the reference's content. * filer.backup: paginate the null-latest .versions rescan A limit=0 listing is capped at DirListingLimit server-side, and old-format version names sort oldest-first, so a history longer than one page left the flattened copy on an older version after the null object was removed. Page through the directory like the S3 read path does. * replication: require matching writable metadata for resolved-latest skips A skip keeps the destination's stored metadata, so identical bytes were not enough: stale attributes or extended entries (e.g. a superseded version id) survived under equal content. Compare the fields a rewrite controls — attributes minus filer-assigned inode/crtime, with the store's atime/ctime zero-defaults applied — plus the extended map, in both the provenance and ETag fast paths. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: require the extended key to exist when comparing metadata * replication: ignore a stored rule TTL when the resolved write carries none * filer.backup: rename -s3FlattenVersions to -s3.flattenVersions Match the -s3.* flag naming convention used by the other s3 options. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Denis Kieftanduser.email
|
a2f595b207 |
backup: don't Close a volume after Destroy (#12225)
Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
349576b63e |
backup: resume incremental copies by index position instead of appendAtNs (#12228)
* volume server: add last-entry cursor fields to VolumeIncrementalCopy * backup: resume incremental copies by index position instead of appendAtNs appendAtNs is stamped by whichever replica received the write, so the same needle carries a different timestamp on every replica. A backup that resumes from a different replica than last time can append the last needle again or skip needles written inside the clock gap. Send the local copy's index entry count and last entry identity (key, size, cookie) with VolumeIncrementalCopy. The replica verifies the same record sits at that position and streams from the next index entry, so the resume no longer depends on any replica's clock. When the boundary record does not match, or the local copy is ahead, the replica returns FailedPrecondition and the backup rebuilds the local copy from scratch. Old servers ignore the new fields and keep the timestamp behavior. * rust volume server: mirror positional resume in VolumeIncrementalCopy * volume server: add entries_hash prefix checksum to VolumeIncrementalCopy * backup: verify the whole index prefix, not just the boundary entry * rust volume server: verify the index prefix hash in resume_by_last_entry * backup: keep the source's compaction revision on rebuilt copies * backup: diverge on a zero-offset boundary entry A remote-tier tombstone lives in the .idx with no .dat record, so readNeedleAt(0) parses the superblock and its error bypassed the ErrIncrementalCopyDiverged mapping; the backup got a generic failure instead of FailedPrecondition and never rebuilt. * rust volume server: mirror zero-offset boundary divergence * rust volume server: gate the empty-dat early return on entry_count With a positional resume the Go handler still reaches ResumeByLastEntry, so an empty replica reports divergence; the Rust early return answered a clean empty stream and weed backup kept a stale copy. * backup: hash the index entry's size field, not trailing offset bytes In a 5BytesOffset build the offset occupies bytes 8-12 and the size starts at byte 13, so the hash folded in one byte of the offset. Matching records at different offsets then hashed differently and a valid backup was reported diverged and rebuilt. * backup: read the index in whole-entry chunks for the prefix hash When entries are 17 bytes a 1 MiB read buffer does not end on an entry boundary: the partial tail entry was skipped by the loop but still counted as consumed, so every later chunk was hashed at a misaligned position. * rust volume server: hash the index entry's size field after the offset Mirrors the Go hashIndexEntries change: with the 5bytes feature the offset occupies bytes 8-12, so starting the size slice at byte 12 folded one offset byte into the hash and matching records at different offsets compared diverged. * rust volume server: read the index in whole-entry chunks for the prefix hash Mirrors the Go change. With 17-byte entries a 1 MiB buffer does not end on an entry boundary, so slicing the partial tail entry panicked once a backup held 61681 entries. * rust volume server: default the new incremental copy request fields in tests |
||
|
Chris LuandDevin
|
8a2fec8610 |
filer.sync: skip superseded replays before fetching any chunks (#12212)
* filer.sync: skip superseded replays before fetching any chunks On a directory under heavy overwrite, most entries the sync replays are already superseded on the source. The sink only learned that after chunk reads failed — or worse, after fetching and uploading bytes a later event replaces — so the backlog drained slowly and the log filled with 404s. CreateEntry and UpdateEntry now check sourceMovedOn before touching a chunk: one entry lookup skips a stale replay entirely. A mid-replication supersession is still caught by the existing retry gates. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: bound the source liveness probes with a timeout The supersession checks added a source entry lookup to the replication path, and every lookup ran on context.Background() with no deadline. A source that accepts the call but whose metadata store stalls would hold the sync forever — before a single chunk fetch, even when every chunk is still readable. Bound the probe at 30s; an unanswered lookup proves nothing, so replication proceeds and the existing retry gates decide. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: sync the live version a superseded replay points at Skipping a stale replay assumed a later event always re-carries the current content. It does not when the superseding write was itself replicated in from another cluster: those events are dropped before they reach a filer sink, so the destination could be left missing the file. When the source reports a live entry newer than the replay, commit that version through the normal path instead — equivalent to applying the later event, with no dependence on its delivery. A gone source entry still skips, since the destination already matches the source. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: reconcile moved entry content after a native rename AtomicRenameEntry relocates whatever version the destination held. If the preceding update replay was skipped because the source had already moved to the new name, the moved entry keeps the old bytes and — with no further writes — stays stale forever. MoveEntry now runs the same reconciliation as an in-place update against the rename's snapshot, so the moved entry picks up pending content changes. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: repair from the live snapshot a superseded replay resolves to An incremental diff against the replayed oldEntry can resolve manifests the overwrite already vacuumed, which fails every retry of the event. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
9df7d62b4b |
mount, s3: clarify allowOthers requirement and bucket name error (#12197)
* mount, s3: clarify allowOthers requirement and bucket name error
`weed mount -allowOthers` defaults to true. On Linux, a non-root
user's mount then needs `user_allow_other` in /etc/fuse.conf, but the
help text only said "allows other users to access the file system".
Say so in the help, and point at -allowOthers=false.
Also fix the grammar of the bucket name length error ("must between
[3, 63]" -> "must be between 3 and 63").
Help and error text only; no behaviour change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: normalize the remaining VerifyS3BucketName error messages
The length message was clarified; the rest of the function still had
inconsistent phrasing ("name must start", "cannot be ip addresses",
"prefix xn-- ... in bucket prefix"). Align them on "bucket name ..."
wording matching the AWS bucket naming rules.
---------
Co-authored-by: pingqiu <pingqiu999@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
|
||
|
Chris Lu
|
43626431c3 |
vector: recheck master discovery quickly while no worker is cached (#12199)
The vectors gateway backs off discovery for 5s between resolutions, but the backoff also suppressed rediscovery after an empty lookup. A worker that landed its KeepConnected registration a moment after the first query stayed invisible for the whole window — longer than a client's retry budget — so the S3 Vectors e2e flaked: CreateIndex raced the registration, got InternalFailure, and every boto retry fell inside the same 5s fast-fail window. With no cached backend, recheck every 500ms instead; the 5s interval now only bounds membership refreshes against a live client, and an absent worker still cannot flood the master with lookups. The empty-membership error also becomes codes.Unavailable so the API answers 503 rather than InternalFailure. cluster.ps lists registered vector nodes like the other node types, and the e2e harness polls it for the engine's registration instead of only checking the gRPC port — a listening engine is not discoverable until the registration lands. |
||
|
|
6cd14f06c9 |
filer.remote.sync: upload a holed entry zero-filled instead of skipping it (#12178)
* filer.remote.sync: skip an event the filer has written past, not only rewritten
isSuperseded only looked for chunks the event named that the filer no
longer holds. A file copied in through a mount logs a create while the
copy is still running: the mount flushes metadata with chunk uploads in
flight, so that chunk list can have holes. ChunkStreamReader stops at
the first hole, the multipart upload sends a short part ("ContentLength=
8388608 with Body length 4194304"), and since every chunk the event
names is still live the event was never superseded. The daemon retried
it forever and the offset never moved past it.
Treat chunks the filer has added since the event the same way. The
event that added them follows in the log and uploads the finished file.
* filer.remote.sync: skip a holed snapshot even while the filer still holds it
A mid-write snapshot the writer never finished is neither deleted nor
written past: the filer holds it unchanged, so the superseded check never
fires and every retry reads short at the same gap, pinning the sync offset
forever. The upload can never succeed for that entry — skip it like a
superseded one. A finishing write logs its own event and uploads the full
content; a dead write has nothing readable to lose.
The check also covers a chunkless entry that is not a dropped local copy
(e.g. a truncated remote-only entry): its reader yields zero bytes against
a nonzero size, which can silently clobber the remote object with an empty
body where the backend does not verify the length.
* filer.remote.sync: upload a holed entry zero-filled instead of skipping it
Skipping the event left a completed sparse file absent or stale on the
remote, and the coverage check on the raw chunk list missed both gaps a
manifest's bounding extent hides and a FileSize smaller than the chunk
extent. ChunkReadAt already resolves manifests the way the upload reader
does and zero-fills every uncovered range up to the effective size, so
route the upload through it: a sparse file, a mid-write snapshot, and a
chunkless entry all send the bytes a read of the entry returns, and a
hole can neither wedge the event nor publish a truncated object.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: skip a superseded entry on reader-construction error too
---------
Co-authored-by: Alex Russell <alex@sengine.cloud>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Alex K
|
fffdb7cc75 |
filer.remote.sync: skip an event the filer has written past, not only rewritten (#12176)
isSuperseded only looked for chunks the event named that the filer no
longer holds. A file copied in through a mount logs a create while the
copy is still running: the mount flushes metadata with chunk uploads in
flight, so that chunk list can have holes. ChunkStreamReader stops at
the first hole, the multipart upload sends a short part ("ContentLength=
8388608 with Body length 4194304"), and since every chunk the event
names is still live the event was never superseded. The daemon retried
it forever and the offset never moved past it.
Treat chunks the filer has added since the event the same way. The
event that added them follows in the log and uploads the finished file.
|
||
|
Chris Lu
|
511727cf3a |
vector: DeleteIndex format guard, sticky discovery, advertise normalize (#12171)
* vector: DeleteIndex format guard, sticky discovery, advertise normalize - handleDeleteIndex verifies the table carries the Lance format before DeleteTable — an Iceberg table under the same namespace is not an index and must not be removed by it. - discovery keeps the current worker while it is still registered — membership-order churn can no longer cancel in-flight calls, and the resolver closes each master conn inside a 5s deadline. - advertised_addr strips scheme and .grpcPort so the UDP probe reaches every supported master form. * vector: empty-follower retry, format check without read grant - discoverVectorWorkers keeps looking past an empty master reply — followers list nothing when workers registered with the leader. - the DeleteIndex format check uses the internal catalog read, so DeleteTable-only callers don't gain a GetTable requirement, and the format must be Lance exactly — format-less tables no longer pass. |
||
|
Chris LuandDevin
|
18691e7b5b |
server: keep the first write on a fresh weed server from stalling (#12149)
* master: end the post-mint allocation hold on the embedded reporter's clean inventory DeferAllocation armed a warmup-length hold whenever the leader minted a fresh topology id, and on weed server the hold ran the whole window: the embedded volume server's join kept refreshing the quiet evidence clock, so the two-pulse early release still cost the first write 10-15s on every fresh start. weed mini already releases on its single pinned reporter's first inventory, but pinning refuses outside volume servers, which weed server must keep admitting. Add a non-exclusive variant: the topology names one expected reporter whose first clean full inventory lifts the hold, while other reporters stay admitted and still feed prior-state evidence and the topology-id freeze. A single-master weed server points it at its embedded volume server — the one reporter guaranteed to arrive quickly; a veteran's heartbeat carries prior state and keeps the window through the regular checks, so a wiped -dir with surviving external volume servers still trips the freeze on first report. Restarted clusters are untouched: a recovered topology id means no mint and no hold. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: announce on the startup peer scan for embedded single-master setups A fresh filer whose startup peer scan came back empty re-polled for the full confirm window before registering on the master, so the lock ring had no member and routed traffic such as S3 PUTs spun on "no lock server found" for ~10s on every fresh weed server / weed mini start. For setups that start with exactly one filer the scan is already final: a single-master weed server and weed mini announce as soon as it completes. A late external filer still joins and replays through the normal path — the confirm window only ever caught peers arriving inside it, and the residual risk mirrors the accepted volume-side trade-off: an embedded filer announcing a few seconds earlier can own the lock ring while a surviving external filer's backlog is still on its way. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: retry a failed heartbeat promptly until first registration A heartbeat stream refused before the leader is known — the common race when a volume server starts while an election is still running — always fell back to sleeping out the full pulse, so a fresh volume's first registration could wait ~5s behind a leader elected a second later. The peer refresh already identifies the leader during the wait, so cap the retry pause at a second once a leader is named; while no leader is known the same short retry applies until the first stream is confirmed, since an unregistered server has nothing else to do. Once the server was registered the pulse backoff is unchanged, so a lost leader still paces reconnects the way it did before. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: retry a failed heartbeat promptly until first registration (rust) Mirror the Go heartbeat retry: cap the post-refresh wait at a second when a leader is named, and keep the short retry until the first stream is confirmed so a fresh volume registers promptly when it races an election. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: leave admitted reporters a confirm pulse when the embedded report ends the hold The single-master weed server releases its post-mint allocation hold on the embedded volume server's clean report, but outside volume servers stay admitted and one may still carry prior cluster state — an inventory from the embedded server alone cannot prove none is due. Clearing the hold outright let a rebuilt master re-issue a volume id a reconnecting veteran still owned. Collapse the released hold to one heartbeat interval instead: late reporters land their evidence or a topology-id freeze inside the pulse, and a re-mint re-arms that pulse rather than the full warmup. The pinned single-reporter release (weed mini) still drops the hold outright, since no other reporter can be admitted. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: keep the empty-membership confirm on embedded single-master weed server A single-master weed server still admits external filers, so an empty startup peer scan is not final: after a master restart the existing filers simply have not reconnected yet. Announcing on the scan let the embedded filer take lock-ring ownership and serve writes that a peer's replayed backlog could then overwrite. Only weed mini keeps the skip, where the embedded filer is the only member its dev cluster can have. The confirm window only delays the master listing for a fresh store; the filer serves while it runs. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: tolerate an expired heartbeat retry deadline (rust) refresh_leader_from_peers can consume the whole pulse budget, leaving retry_at in the past; subtracting a later Instant panics and kills the heartbeat task instead of retrying. Match the Go side, where time.Until of a past deadline just sleeps zero. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: retry the seed list without a second pulse sleep (rust) The error path already slept its retry delay; an extra full pulse after losing the round delayed the next attempt past what Go's heartbeat() does, which paces retries only inside the per-master attempt. * master: keep a configured allocation hold shorter than the confirm pulse Once the embedded report lands, the collapsed hold replaced the configured deferral with a full pulse even when the operator set a shorter one, including zero (disabled). Cap at the pulse instead. * filer: end the empty-membership confirm early on a fresh-minted embedded master On a fresh single-master weed server the announce gate still held the filer for the full confirm window, so the master's lock ring stayed empty and the first write retried 'no lock server found' for ~7s on top of the collapsed allocation pulse. A fresh TopologyId mint proves the master's persisted state was empty, making the empty peer scan final; skip the rest of the confirm only then, and keep it whenever the master recovers a prior id. * filer: keep a short reconnect window on a fresh-minted embedded master A fresh TopologyId mint proves the master's persisted state was empty, not that no external filer survived its loss — one can still be mid-reconnect on its keepalive cycle when the confirm starts. Ending the empty-scan confirm on the first poll let the fresh filer register and serve routed reads before that peer was found and backfilled, so existing files could read as missing. The fresh-mint path now shortens the confirm to a few keepalive cycles instead of skipping it: a reconnecting filer still lands in time, while a genuinely fresh cluster announces after ~3s rather than ~10s. * volume: skip the heartbeat loop when no seed masters are configured An empty seed list made the outer loop spin with no await between passes and no path to observe the stop flag. A masterless volume server has nothing to register with, so the heartbeat exits after a warning. * volume: skip the heartbeat loop when no master addresses are given (rust) Mirrors the Go heartbeat: an empty seed list spun the outer loop with no await and no shutdown checks. Exit after a warning so library callers get the same behavior as the binary path that validates the list up front. * filer: re-check the fresh-mint hint on each confirm pass The topology id is minted after raft election, which can land after the empty-scan confirm has already started — a one-time check at entry missed the mint and held the full window. Evaluate the hint on every poll and shorten the confirm from the first observed mint, still capped by the normal confirm deadline. * master: mark a fresh topology mint only when the volume bound is zero Recovery restores the max volume id unconditionally but the topology id only when present, so an upgraded or partially rebuilt master can mint a new id while still holding prior volume state. Marking fresh there would let the embedded filer shortcut its peer confirm on a cluster that is not empty; only mark the mint fresh when no bound survived. * filer: drop the fresh-mint shortcut once a peer has been observed --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
190ab32cfb |
vector: weed-vector registers to master; s3 discovers it there (#12146)
* vector: weed-vector registers with the weed master weed-vector-server --masters holds a KeepConnected stream to the weed master, the same mechanism filer and s3 use via wdclient — the server shows up in ListClusterNodes(client_type=vector). Resends registration and rediscovers on failure. * vector: s3 discovers weed-vector through the master With -vectors.grpc unset the vectors gateway resolves one registered weed-vector via ListClusterNodes on the masters it already derives from filer config — discovery replaces the static dial for the single-worker OSS path. The conformance harness registers the engine to mini's master and drops -vectors.grpc. * vector: default-fill registration fields ..Default::default() keeps the source compiling against proto variants that carry extra fields, so the same file works in either tree. * vector: serve the Vectors API on the main s3 port The vectors router sits on the main -port listener behind a dispatch that checks the SigV4 credential scope for /s3vectors/ — no separate port needed. -port.vectors stays for explicit dedicated-port deployments; the conformance harness now hits vectors through :8334. * vector: drop -port.vectors entirely It was new — no compat to preserve. The main -port listener serves the API whenever a weed-vector backend is reachable. * vector: drop -vectors.grpc; discovery is the only locator weed-vector always registers to master, the gateway always resolves it through ListClusterNodes — a static address just forks membership. * vector: binary is weed-vector Matches the enterprise binary; package and lib names are unchanged. * vector: review fixes — cluster type, grpc port, lazy resolve, leader hint, advertise - cluster: register 'vector' as a node type so ListClusterNodes returns weed-vector members - s3: dial masters via ToGrpcAddress (HTTP port + 10000) - s3: backend resolves lazily through vectorDiscovery — workers joining after gateway startup work on first query, and address changes re-dial - master_client: convert master HTTP addr to grpc +10000, honor http(s) scheme, follow the leader hint in KeepConnectedResponse, and --advertise covers wildcard/unreachable listen addresses * vector: no --advertise flag; derive the outbound address A wildcard listen resolves to the source IP the master sees via a route lookup, so registration always advertises something reachable without another flag. * vector: review fixes — plaintext dispatch, conn leaks, discovery backoff The two plaintext http servers now serve the vectors dispatch too — signed s3vectors calls on the main port reach the right router. discoverVectorWorker closes each master conn and bounds lookups to 5s; a dead or absent worker no longer re-hits master on every RPC — the resolver backs off for 5s between rediscovery attempts. * vector: 1 GiB grpc cap, dead-worker invalidation, CLI rename Valid PutVectors batches exceed the tonic 4 MiB default — match the gateway's 1 GiB cap. A replaced worker no longer sticks: any RPC error resets the cached address and the next call rediscovers through the master. The core CLI bin renames to weed-vector-cli so the two crates do not collide on the same output path. * vector: retry-only-transport reset, periodic re-resolve, cli rename - reset() takes the failed client and runs only on transport errors — an InvalidArgument on a healthy conn no longer interrupts unrelated requests, and a concurrent re-resolve wins the race. - cli() re-resolves once the interval passes even with a live client, so a worker that moves addresses is picked up. - verify.rs spike docs use weed-vector-cli after the bin rename. * vector: only Unavailable counts as a transport failure A request deadline or an application error can fire on a healthy connection — resetting the shared conn on those churned reconnects. |
||
|
Chris Lu
|
5f26ef6ad1 |
filer,webdav,iam,sftp: add -idleTimeout flag for connection idle timeout (#12136)
* filer: make the HTTP connection idle timeout configurable The filer applied a hardcoded 10s activity timeout to its HTTP listener. A buffering proxy in front of the filer can stall upstream reads longer than that when the end client drains slowly, so large downloads through it were truncated (#8062). Expose the value as -idleTimeout (default 10, unchanged), matching the existing -idleTimeout flags on the volume and s3 servers, and wire filer.idleTimeout through 'weed server' and 'weed mini' too. A nil option keeps the 10s default so embedders constructing FilerOptions directly keep working. * filer: apply the idle timeout to the readonly port too The -port.readonly listener ran without any connection timeout while the main API port timed out. Give it the same -idleTimeout so both ports behave identically. * webdav: make the HTTP connection idle timeout configurable The WebDAV server used the same hardcoded 10s activity timeout. Expose it as -idleTimeout (default 10, unchanged), including -webdav.idleTimeout when embedded via 'weed filer -webdav', 'weed server -webdav', and 'weed mini'. * iam: make the HTTP connection idle timeout configurable The IAM API server used the same hardcoded 10s activity timeout. Expose it as -idleTimeout (default 10, unchanged), including -iam.idleTimeout when embedded via 'weed filer -iam' and 'weed server -iam'. * sftp: make the connection idle timeout configurable The SFTP server used the same hardcoded 10s activity timeout. Expose it as -idleTimeout (default 10, unchanged), including -sftp.idleTimeout when embedded via 'weed filer -sftp' and 'weed server -sftp'. * command: clamp idle timeout seconds before converting to duration A seconds value past time.Duration's range would overflow on multiplication and silently disable the timeout. Bound it in one shared helper, which also deduplicates the nil-default handling at the four call sites. * command: return zero for non-positive idle timeout seconds A large negative seconds value could wrap on multiplication and come out positive. Guard before converting so non-positive input always disables the timeout. * command: apply idle timeout to local unix socket listeners The unix-socket listeners in filer, s3, and sftp served accepted connections with no activity deadline while the TCP listeners already wrapped conns in util.Listener. Wrap the socket listeners the same way so a positive idleTimeout also bounds idle socket connections. * command: clamp s3 idle timeout seconds before multiplying s3 still multiplied the flag inline, so an out-of-range value could overflow time.Duration. Route it through idleTimeoutSeconds like the other commands; non-positive keeps meaning disabled. * volume: clamp idle connection timeout seconds before converting * command: cover idleTimeoutSeconds conversion semantics * command: gate idleTimeoutSeconds clamp check on 64-bit int |
||
|
Chris LuandDevin
|
ee315527cb |
vector: address remaining s3vectors review findings (#12115)
* vector: address remaining s3vectors review findings Catalog and permission model: - Vector buckets are declared Lance table buckets end to end: create sets format=lance, get/list/delete reject or skip ICEBERG buckets, and the format lookup runs on an internal trusted catalog so callers need no GetTableBucket grant. - The implicit "vector" index namespace is created through the same internal manager, so a CreateTable grant alone authorizes CreateIndex; typed S3TablesError codes replace string matching on err.Error(). - CreateIndex pre-checks existence after the authorize, tags the record with a create nonce, and rolls back on engine failure under context.WithoutCancel with a versionToken CAS, so a retry never deletes a table it did not create. - DeleteIndex drops the catalog record directly; DropIndex is unimplemented on the shared VectorService. - DeleteVectorBucket removes the empty internal namespace on BucketNotEmpty and retries once. - PutVectors/DeleteVectors require the update grant, and engine gRPC errors map to typed API errors instead of generic 500s. API correctness: - CreateVectorBucket/CreateIndex return their resource ARNs. - ARNs reject empty bucket segments; vectorBucketArn must be an ARN. - ListIndexes returns summaries built only from catalog fields. - ListVectors honors maxResults/nextToken inside a segment via "seg:<pageStart>:<nextIndex>" resume tokens and validates segmentCount/segmentIndex. - QueryVectors requires positive topK; PutVectors rejects empty keys. - CreateVectorBucket rejects a supplied encryptionConfiguration rather than silently dropping it. - Metadata filters emit Lance's real JSONB UDFs: json_get_string/ json_get_float/json_get_bool picked by operand type, json_exists with a bracketed JSONPath, and per-member accessors for $in/$nin. Engine: - weed-vector-server stores the metadata column as Lance JSONB (lance.json extension) so the json_* UDFs apply; reads accept legacy Utf8 JSON columns. Vectors port TLS: - The TLS config is built before binding listeners, loads -cacert.file into ClientCAs when client-cert verification is on, fails startup on an empty/invalid CA file, and closes the gRPC conn on every failure path. * s3vectors: authorize index create before writing the implicit namespace A denied caller could still leave a persistent namespace behind because ensureIndexNamespace wrote through the trusted catalog before the caller's create grant ran. AuthorizeCreateTable now tolerates the not-yet-declared implicit namespace (bucket policy alone decides) and runs before any write; real table creates keep requiring the namespace. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: keep engine error details out of the API response Unavailable now maps to 503 ServiceUnavailableException and unclassified codes return a generic message; the real gRPC error stays in the log. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: exact-type metadata filters + NULL-safe $nin The jsonb extractors convert numeric-looking strings through json_get_float and 'true'/'false' through json_get_bool, so a stored '3' could satisfy a filter for the number 3. Numeric and boolean comparisons now also require the stored value not be a string, and each $in/$nin group coalesces NULL so NOT cannot invert NULL into NULL. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: validate the vectors TLS key pair before binding The reloading certificate provider reads the files asynchronously, so a missing or mismatched pair would only surface as failed handshakes on an announced HTTPS port. Load it once up front. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: make a token-checked table delete atomic The version token was read in one filer call and enforced in another, so a delete+recreate slipping between them let the delete erase the replacement. When a token is supplied the name path now carries the observed mtime as an if_not_modified_after precondition (evaluated under the path lock), and a decoupled delete strips the catalog marker under a compare of the captured attributes before touching the data path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: compare filter operands against the stored JSON type jsonb's strict converters make json_get_string return text for numbers and booleans too, so a get_string IS NULL guard is never true for scalar metadata — numeric and boolean comparisons silently matched nothing. Numbers and booleans now compare through TRY_CAST of the serialized json_extract text, which yields NULL on any other stored type; string comparisons stay on json_get_string behind a CASE guard so the extractor only runs on rows storing a JSON string. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: allocate implicit namespace metadata before use A first table create in an undeclared namespace took the allowMissingNamespace branch with a nil record and panicked on the owner assignment. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: read the delete target's metadata and snapshot in one lookup Two separate lookups let a delete+recreate in between validate the caller token against the old record while the conditional delete aimed at the replacement. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: assert the captured record before a conditional table delete An xattr update keeps Mtime, so the IfNotModifiedAfterNs guard alone could not see a metadata write that had already rotated the version token, and a recreate preserving the mtime could slip the mtime check. The delete now first proves the entry still carries the captured attributes under the filer's path lock, then keeps the mtime precondition. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: match only JSON boolean literals in metadata filters * filer,s3tables: delete versioned tables under one attribute precondition * s3vectors: compare serialized JSON for string filter operands json_get_string converts numeric-looking strings and booleans, so it cannot serve exact-typed string comparisons even behind a guard. String operands now compare the serialized json_extract text directly, which is exact-typed and never errors; a Rust test pins the semantics against a mixed-type dataset. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: compare boolean filter operands as serialized text Arrow's boolean cast accepts '1' and '0', so a TRY_CAST column let a stored number satisfy a boolean filter. Booleans now compare the serialized json_extract text like strings do — a JSON boolean only ever serializes as 'true' or 'false'. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: claim a catalog name's markers in one compare-checked write A create wrote its markers attribute-by-attribute after checking the name was free. A delete's restore landing in the check-to-write gap put the old markers back first, and the create then overwrote only the ones it sets — leaving the old tags, policy, and maintenance settings on the new table. CreateTable, RegisterTable, and CreateView now write all their markers in a single update that refuses to run while any catalog marker is present, so the second writer loses with a conflict instead of inheriting them. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: test the attribute-preconditioned table delete and name claim Adds a BeforeDelete hook plus ExpectedExtended evaluation to the in-memory filer so tests can land a competing write in a handler's check-to-delete window, and covers two losing races: a token delete whose record changed since the read, and a create colliding with a delete's marker restore. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: reserve a catalog record under one compare-checked claim claimCatalogRecord writes a deleting marker onto the record while it still holds exactly the caller's captured attributes, so a delete or rename can hold the record through a multi-step operation and a losing writer backs out before any data or destination entry is written. The claim key joins the markers a name-claim asserts absent, so the name cannot be reused while the reservation stands. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: claim the record before purging or publishing past it A token delete of a decoupled table purged the data first and checked the marker afterwards, so a rename that finished in between left the moved record pointing at purged data while the delete reported a conflict. The delete now claims the record under the captured attributes before any data is removed and releases the claim if the purge fails, keeping the record intact for a retry. The rename claims the source before publishing the destination, so a losing rename backs out while no destination entry exists instead of leaving one behind pointing at data a delete went on to remove. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: test the record claim across delete and rename races A rename completing in the token delete's check-to-claim window turns the delete into a conflict with the data intact, and a delete's claim on the source turns the rename into a conflict before the destination entry is published. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
67240f538d |
filer.remote.gateway: stop zero-byte uploads of remote-only entries (#12110)
* filer.remote.gateway: skip uploading a dropped local copy on update The bucketed update path uploaded the chunkless entry whenever the update was not metadata-only. remote.uncache, eviction, and remote.meta.sync drop a synced entry's chunks and clear its sync stamp, which reads as dirty when the remote mtime shares the entry's second; the upload then replaced the object that is the entry's only content with zero bytes. Skip the upload the same way the mounted-directory sync does. * filer.remote.gateway: copy a remote-only entry's object on rename A rename deleted the source object and uploaded the chunkless entry, replacing the entry's only content with zero bytes — the same loss remote.uncache caused on plain updates. Complete the rename on the remote the way the mounted-directory sync does: copy the source object over (or take the destination when it already describes the entry), restamp so reads stop resolving through the pending-source marker, and remove the source only while it is still the object the stamp describes. A remote-only entry renamed out of the gateway's coverage keeps its object, and the plain rename's old-key delete only removes the object the event's stamp still describes. * filer.remote.gateway: scope the sync-stamp skip to same-path updates shouldSendToRemote reads the entry's sync stamp, which still covers a rename's unchanged mtime, so a synced or pending-stamped rename event was skipped before its copy or upload could run. Only updates in place may skip on the stamp. * filer.remote.gateway: keep the source a pending rename moved on from A rename event whose pending destination has since been renamed again still names the source a later event reads through its own marker; deleting the old key here loses the only copy. Mirror the remote.sync check: leave the source when the event's entry carries the pending-source marker and no entry holds the destination any more. * filer.remote.gateway: release a rename's pending source once complete A rename whose old entry read through a pending-source marker leaves that earlier source behind: the destination is released by its own key, never the marker's. Mirror releaseRenamedSource — once the renamed entry no longer reads through a marker, delete the pending source guarded by the old entry's stamp, keeping it while a remote-only entry holds its path. * filer.remote.gateway: finalize a complete rename before failing its source A rename source outside the gateway's coverage errored before the destination was checked, and the recorded failure held the event offset. Stat the destination first: when it already holds the content the entry describes, refresh the stamp and leave the unmanaged source untouched; fail only when nothing holds the content. * filer.remote.sync: document the source-of-truth protocol both processors share --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
dcc94f818d |
vector: S3 Vectors API gateway (P1) (#12103)
* vector: S3 Vectors API gateway over VectorService weed/s3api/s3vectors serves the AWS S3 Vectors JSON API: bucket CRUD maps to S3 Tables table buckets, index CRUD to Lance table records plus the VectorService CreateIndex/DropIndex RPCs, and the data plane (Put/Get/List/Delete/QueryVectors) calls the shared service. The AWS filter document translates to a Lance SQL predicate over the metadata JSON column; unsupported operators fail loudly rather than overmatch. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: -port.vectors and -vectors.grpc on weed s3 Mirror the Lance server startup: the S3 Vectors API binds its own port and dials the weed-vector-server gRPC address; both disabled by default. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: resolve index ARNs, authorize index ops, fix catalog fields - vector buckets declare FormatLance so a new bucket accepts index tables instead of defaulting to ICEBERG. - ListIndexes decodes the `tables` response field; `tableSummaries` hid every index. - resolveIndexARNs honors indexArn alone; handlers no longer treat indexName as the only carrier. - data-plane RPCs fetch the index record first, so callers authorize against the index resource and a missing index 404s before touching the engine; QueryVectors reports the index's configured metric. - IndexArn added to the index-level request types. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: serve the vectors port over TLS when configured When -key.file/-cert.file are set, the vectors listener mirrors the main API and serves TLS rather than a second plaintext endpoint. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: harden the s3vectors gateway handlers - $ne no longer shadows the operator: exclusion filters emit <> instead of equality. - a null element in a PutVectors list fails the request instead of panicking. - requests are capped at 32 MiB; distances return via *float32 so an exact match still reports 0 when returnDistance is set. - CreateIndex declares the `vector` namespace first and validates dimension (1..=4096) before the catalog write; a failed VectorService.CreateIndex rolls the table record back. - non-filterable metadata keys persist as JSON rather than a comma-joined string, and index records carry their createdAt. - ListVectors honors segmentIndex/segmentCount by slicing the full listing in the gateway. - catalog errors map to AccessDeniedException/ConflictException instead of a blanket ValidationException, and malformed ARNs 400 instead of slicing. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: /{Operation} routes and vectors-port TLS, conn cleanup Path-dispatched clients (POST /QueryVectors) reach the same handlers as X-Amz-Target calls. A failed listener bind now closes the vector gRPC connection instead of leaving it open with the endpoint disabled. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
72ac424198 |
filer: deliver meta-log events under their stream position, not their stamp (#12084)
* filer: deliver meta-log events under their stream position, not their stamp A mutation's event timestamp is stamped before the filer store write so the entry and its event share a commit time, while the log-buffer append happens after the write. A write delayed behind later-stamped mutations keeps its early stamp in the payload while AddDataToBuffer bumps LogEntry.TsNs to keep the stream monotone - so subscribers could receive events whose payload TsNs arrives out of order. Consumers treat the delivered TsNs as a stream position (cursors, watermarks, replay windows). The s3 lifecycle daily-run drains events in order and pins its per-shard cursor behind the first not-yet-due match; an event delivered with a stale stamp after the cursor already passed it is filtered out forever and its pending match silently never fires - the flaky test_lifecyclev2_expiration signature of exactly one object of a same-rule pair surviving, with errors=0 on every pass. Normalize event.TsNs to logEntry.TsNs at delivery (and for the skeleton-filtered heartbeat) so delivered timestamps are monotone by construction. The commit-time stamp remains available on the entries themselves via CommitTsNs. * pb,filer: stamp metadata events with a mutation timestamp TsNs on the wire now carries the delivery stream position, so the mutation's commit stamp needs its own field. Add mutation_ts_ns to SubscribeMetadataResponse and set it when the event is created. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: carry the mutation stamp through event delivery Delivery rewrites TsNs to the stream position; the payload stamp survives in MutationTsNs, backfilled from TsNs for payloads written before the field existed. The notification callback forwards it to the wire message. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * pb: normalize chunk-replay timestamps like eachLogEntryFn ReadLogFileRefs bypasses the server decode path, so a delayed append still arrived under its stale payload stamp and could hide behind a subscriber cursor. Deliver the stream position and preserve the stamp in MutationTsNs. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: order replay and tombstones by the mutation stamp Replay treated the delivered TsNs as the commit time, so a peer write delayed in its own log could overwrite a newer local entry. Compare and stamp with MutationTsNs, falling back to TsNs for older peers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * remote.sync: carry the mutation stamp through reshaped renames Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: test replay ordering on the mutation stamp * filer: tombstone remote deletes by delivery position, order by commit stamp * filer: keep the peer commit stamp when restamping aggregated events * pb: keep the log cursor on the entry position past callback restamps --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
3abf8b2572 |
filer,pb,mount: keep per-chunk cipher keys out of logs (#12085)
* filer: format ChunkView without its cipher key readChunkSliceAt logs a failed chunk fetch with %+v, which printed the per-chunk AES-256-GCM key into error-level logs on every failed read. Give ChunkView a String() that omits CipherKey so %v and %+v stay safe. * pb: scrub cipher keys when an entry must be logged whole ScrubCipherKeys clones an Entry with chunk CipherKey fields cleared, so a warning that needs the full entry still cannot leak the keys. * log entry identifiers instead of whole metadata messages Several log lines formatted whole protobuf messages — SubscribeMetadataResponse, EventNotification, Entry-carrying requests — whose chunks include cipher_key under -encryptVolumeData, so the keys escaped into logs (some at default verbosity). Log the directory, entry name and event timestamp instead; where a chunk list was printed, print file ids via FileIdsString. * log queue: unwrap SubscribeMetadataResponse and never print raw protos The fallback printed any message with %+v, so a SubscribeMetadataResponse carrying entries would still put cipher keys in the log. Unwrap its event notifications (batched events included) and reduce unknown messages to their type name. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * remote sync logs: name the event's real path Rename targets logged under resp.Directory while the entry lives under NewParentPath; route them through MetadataEventTargetDirectory. Sync failure logs printed only the directory, so same-dir failures shared an identifier — log the source and destination paths instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * log queue: log the entry's destination path on moves A cross-directory rename printed "file" -> "file" under the source path, hiding where the entry landed. Use NewParentPath so the log shows the destination directory next to the new name. * admin: install the go.mod-pinned templ, not @latest install-templ fetched templ@latest, which now emits templruntime.NewAttributeExpression - a symbol absent from the runtime pinned in go.mod - so every fresh 'make' regenerated uncompilable _templ.go files. Install the go.mod version and reinstall when the binary on PATH reports a different one. * admin: drop stray Makefile change belonging to the templ pin PR Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: join the move destination path in log queue output A NewParentPath of "/" printed //name for moves into the root; path.Join keeps a single leading slash. * log queue: skip a nil event notification instead of logging an empty move Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * pb: tolerate nil chunk entries while scrubbing cipher keys for logs Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * subscription and backup failure logs name the event's file The process/marshal failure logs printed only the event directory and timestamp, so a failed sync left operators guessing which file failed. Use MetadataEventSourceFullPath/MetadataEventTargetFullPath so each failure names the source and target paths without printing the entries. * replication: identify entry-less events without printing the proto Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: scrub cipher keys from the bucket-metadata debug dump buildBucketMetadata marshals the whole filer entry to JSON for a V(3) log line; a chunked entry would emit each chunk's cipher_key base64. Marshal the scrubbed copy instead. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
6ebc055a94 |
master, volume, filer: start on a read-only mounted directory and serve existing data read-only (#12081)
* master, volume, filer: start on a read-only mounted directory and serve existing data read-only A read-only mount was treated as writable (permission bits only), so every service failed on its first write: the master panicked writing raft conf.tmp, the filer exited opening the leveldb LOCK, and the volume server skipped every volume while still offering the directory as free capacity. - util.IsReadOnlyMount: access(W_OK) == EROFS. - volume: on a read-only -dir take the existing O_RDONLY/noWriteOrDelete load, index .idx in memory when no .sdx exists, offer no free slots (same branches as isDiskSpaceLow), skip the vid stamp there. - master: on a read-only -mdir run raft from a private copy and refuse new volume ids. - filer: leveldb2/leveldb3 set their existing ReadOnly option. * volume: skip read-only mounts when picking EC shard copy targets A shard copy landing on a read-only mount fails at write time; offer only writable locations like the other placement paths do. * volume: mirror the read-only mount handling in the Rust volume server The .dat open only fell back to read-only on EACCES, so a real read-only mount (EROFS) still failed the load. The index side already degrades once no_write_or_delete is set. Add is_read_only_mount via access(W_OK) == EROFS, mark DiskLocation.read_only_mount, and mirror the Go checks: no free slots, heartbeat max = used, no vid-stamp writes, no EC shard copy targets, and a read-only index dir forces the volume read-only. * volume: write vid stamps while only the index dir is read-only * filer: skip the write-bit check when a leveldb store opens read-only * filer: serve reads of buckets with no leveldb3 dir on a read-only store * volume: derive a stable dir uuid when a read-only mount cannot store one * master: keep the read-only raft copy across restarts so votes survive * volume: index a missing .idx in memory on a read-only mount * volume: release the .idx handle after an in-memory read-only load * volume: mount EC shards read-only when their index dir is on a read-only mount * volume: mark the index dir read-only before the tail repair opens .idx * volume: write vid stamps while only the index dir is read-only (Rust) * volume: derive a stable dir uuid and report no free slots on a read-only mount (Rust) * volume: index a missing .idx in memory on a read-only mount (Rust) * volume: refuse an EC copy whose shard owner is on a read-only disk * master: key the read-only raft copy by the absolute -mdir Two masters sharing the same relative -mdir text from different working directories would have collided on one temp copy and shared raft state. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: mix device and host into the read-only dir uuid Deriving the uuid from the path alone let independent servers at the same mount point report one uuid, which RegisterUuids rejects. Hash in the device id and hostname so restarts stay stable but disks differ. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: vic6996-web <320442181+vic6996-web@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
53834117a6 |
mini: lift the post-mint allocation hold on first inventory report (#12061)
* mini: lift the post-mint allocation hold on first inventory report DeferAllocation armed a fixed warmup window whenever the leader minted a fresh topology id, so the first write on a brand-new weed mini blocked for the whole window even though no prior cluster state could exist. The window still has to cover a rebuilt master's veteran servers, which report on their first post-reconnect heartbeat. Mini has exactly one possible reporter — the embedded volume server, whose first heartbeat carries the full inventory — so hold allocation only until that report is applied. The bound raise and topology-id freeze checks run while the heartbeat is processed, so a wiped master state still raises the bound or trips the freeze before the hold lifts. Standalone weed master and weed server keep the fixed window: volume servers outside the process can still be on their way. * master: apply queued topology-id reports before opening the allocation gate SetTopologyInitPending(false) cleared the pending flag before replaying topology ids stashed while the local id was still being minted. A NextVolumeId landing in the drain window saw no gate and no freeze yet. The post-mint allocation hold used to cover the gap by delaying the winner past the freeze; once the hold can release on first inventory, the gate must stay closed until the queued checks have run. * mini: gate the first-report release on single-master mode releaseAllocOnFirstReport assumes the embedded volume server is the only possible reporter of prior cluster state. A mini started with -master.peers joins other masters, whose own volume servers can still be on their way when the first report lands, so keep the regular quiet-pulse hold there. * master: check reported topology id on empty full reports A HasNoVolumes heartbeat carries the same LastSeenTopologyId a populated list does, but the mismatch freeze only ran for non-empty reports. An EC-only veteran could therefore release the post-mint allocation hold through NoteInventoryApplied while its stale topology id never reached CheckReportedTopologyId. * master: drain the topology-id stash inside the init-gate lock CheckReportedTopologyId only held reportedTopologyIdsLock while stashing, so a report in flight during SetTopologyInitPending(false) could land its freeze after topologyInitPending cleared, letting a NextVolumeId slip through the gap. The whole check now runs under the lock, and the gate is released inside the same critical section as the drain. * mini, master: pin and enforce the single volume-server reporter isSingleMasterMode only checks master peers; SendHeartbeat still accepted any volume server, so a late foreign report could carry prior cluster state after the first-report release dropped the hold. Mini now pins the embedded volume server's address and the master refuses registration from any other reporter while ReleaseAllocOnFirstReport is set. * master: admit the volume-server reporter before applying heartbeat state * test: cover SendHeartbeat admission ordering A refused reporter must not freeze growth or raise the id bound, and the pinned reporter's stale topology id still must. * mini: re-derive the single-reporter release on every run The package-level master options survive in-process reruns, so a multi-master mini following a single-master one must not inherit the first-report release or a stale reporter pin. * master: admit addressless first heartbeats under the reporter pin An unregistered heartbeat with no Ip skipped the single-reporter admission check but still reached the seen-bound raise and the topology-id freeze before the delta-heartbeat skip. Admission now runs for every unregistered heartbeat; an empty address can never equal the pin. |
||
|
Chris LuandDevin
|
88b91cccba |
command: no extra localhost listener when -ip.bind is specified (#12055)
* util: add listenOnLocalhost parameter to NewIpAndLocalListeners Nil keeps the current behavior of also binding localhost when the host is a specific address; all callers pass nil for now. * command: add -ip.bind.localhost flag to opt out of the extra localhost listener Binding a command to a specific -ip.bind address also opened a second listener on localhost for every port, with no way to turn it off. Each command that calls NewIpAndLocalListeners (master, master.follower, filer, s3, iam, volume, sftp, mq.broker, mq.agent) now accepts -ip.bind.localhost=false to bind only the requested address. Embedded services get prefixed flags (s3.ip.bind.localhost, sftp.ip.bind.localhost, iam.ip.bind.localhost), and the command-level flag also disables them so a single switch turns off every extra listener the command opens. * mini: only require all-interface port availability while the loopback listener is on With -ip.bind.localhost=false each service binds just -ip.bind, so a port held on another address no longer collides; the all-interface precheck would reject or relocate usable ports. WebDAV, the Admin UI, and the Admin worker gRPC (which binds -admin.worker.ip, not -ip.bind) never open a loopback companion listener, so their checks now use the address they actually bind. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mq.broker: bind the gRPC listener to -ip.bind instead of all interfaces The listener used an empty host so -ip was ignored for binding and the localhost companion was never created, leaving -ip.bind.localhost with nothing to control. Give the broker the same -ip/-ip.bind pair the other servers use; embedded mode binds to -ip.bind like its peers. * server: add -sftp.ip.bind.localhost and -iam.ip.bind.localhost switches weed filer already exposes the prefixed opt-outs for its embedded sftp and iam; give weed server the same pair so each extra listener can be disabled individually while -ip.bind.localhost=false still turns them all off. * mq.broker: keep the default all-interface bind unless localhost is off An unset -ip.bind used to mean the standalone broker listened on every interface; defaulting it to -ip silently cut off multi-homed clients. Only fall back to -ip when -ip.bind.localhost=false, where a specific bind is needed for the opt-out to take effect. * mini: validate the Admin HTTP port on the address it will bind The port check always used -ip.bind while startMiniAdminWithWorker falls back to loopback when Admin has no credential, so a loopback-held port passed validation and killed the admin at startup. * mq.broker: keep the all-interface default bind when -ip.bind is unset -ip only becomes the fallback once -ip.bind.localhost=false needs a specific address for the listener. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mini: validate the Admin HTTP port on its effective bind IP When the Admin listener falls back to loopback (no credential or mTLS configured), the pre-flight port check now probes that same address instead of -ip.bind. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * util: keep NewIpAndLocalListeners source-compatible for 3-arg callers * command: drop the -ip.bind.localhost flag; an explicit -ip.bind drops the extra listener Instead of an opt-out flag, the loopback companion listener now simply follows whether the bind address was chosen: it is created only while -ip.bind is unset and the bind address falls back to -ip. Passing -ip.bind=<addr> binds each port to exactly that address. Each service options struct carries a listenOnLocalhost *bool that embedders (weed server, weed mini, embedded s3/sftp/iam in weed filer) set from the parent's -ip.bind state, captured before the bind flag is repointed or filled in by the -ip fallback; standalone commands derive it from their own flag before the fallback runs. iam and mq.agent gain a real -ip.bind flag so they have a way to pin a single address. mq.broker keeps its new -ip.bind flag but drops the -ip.bind.localhost fallback hack: empty still listens on all interfaces. * server: default embedded mq.agent brokers to the broker's bind address With an explicit -ip.bind the embedded broker no longer listens on localhost, so an unset -mq.agent.brokers defaulting to localhost:17777 could not reach it. Default to <serverBindIp>:<mq.broker.port> (the address the embedded broker binds), falling back to localhost when the bind address is a wildcard. An explicit -mq.agent.brokers still wins. Also fix the NewIpAndLocalListeners skip message to say localhost. * server: bind embedded mq.agent to serverBindIp The embedded agent's ipBind was never set, so startQueueAgent fell back to the advertised -ip address and an explicit -ip.bind did not pin the agent listener. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
fefac028ec |
feat: send EC shards in parallel per destination disk (#12050)
* feat: send EC shards in parallel per destination disk One stream per destination disk. max_shard_transfers unset means no global cap, 0 sends shards sequentially, and N is a global cap that still keeps one stream per disk. The value is persisted through TaskPolicy. The maintenance form is a checkbox plus Max transfers because unset and 0 differ. The key is intentionally omitted from GetConfigSpec. * ec shard distribution: treat negative max_shard_transfers as sequential The config layer (normalizedMaxShardTransfers) clamps negative values to 0, which selects sequential distribution. shardTransferCap diverged: a negative cap that reached ShardDistributionOptions directly would have selected uncapped parallel mode, the most aggressive setting. Align the options layer with the same normalization so invalid input degrades to the conservative mode. * ec: serialize shard transfers to a server when a target leaves disk choice to the receiver disk_id 0 is auto-placement on the receiving volume server, so an auto-placed stream can land on a disk another queued stream is pinned to. Grouping by (destNode, disk_id) alone let both run at once; collapse all of that server's transfers under the destNode key instead. * worker: clamp max_shard_transfers to the int32 wire range ReadIntConfig returned the 0 fallback for values above MaxInt32, turning a large positive cap into sequential mode. Read int64 and clamp into range instead. * worker: drop unused shard-transfer form helper The maintenance form renders and collects max_shard_transfers in plugin.templ JavaScript; the Go copy had no callers. * worker: bound max_shard_transfers conversion under an in-range guard --------- Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
da1d7444b8 |
master: shorten the first-write hold after a fresh topology id (#12059)
* master: release the post-mint allocation hold after a quiet pulse DeferAllocation armed a fixed warmup window whenever the leader minted a fresh topology id, so the first write on a brand-new cluster blocked for the whole window even though no prior state could exist. The window still has to cover a rebuilt master's veteran servers, which report on their first post-reconnect heartbeat. Track the last heartbeat that could carry prior cluster state — a server joining, held volumes or shards, a nonzero id bound, or a foreign topology id — and lift the hold one pulse after it goes quiet, capped by the original window. * master: make the post-mint allocation hold configurable master.volume_growth.allocation_deferral overrides the warmup window a leader holds volume id allocation for after minting a fresh topology id. Unset keeps the existing window; "0s" disables the hold for setups that know the cluster is fresh (test harnesses, single-node labs); any other duration replaces the cap while the quiet-pulse early release still applies. * master: re-check the allocation deadline while it is held NextVolumeId slept for the whole computed delay, so a heartbeat landing mid-sleep that pushed the quiet deadline out was ignored — the pending allocation could still run ahead of a straggler's report. Wait in short slices, re-evaluating the deadline and the growth freeze each round. * master: count only ids above the issued bound as prior-state evidence Heartbeat echoes of volumes and shards this master allocated during the hold refreshed the quiet window, so each growth allocation re-extended the deferral until the hard cap. Evidence now requires an id beyond the bound this leader has issued, a higher reported bound, or a foreign topology id; a joining server (dn == nil) still counts once. * master: reject an unparseable allocation_deferral instead of disabling the hold Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: do not count a skipped heartbeat as a joining server Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: assert the allocation hold extends past the original quiet deadline Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: release the post-mint hold after two quiet pulses One pulse proved short for a veteran that reconnects on a staggered restart: with only five seconds of quiet, allocation could resume and reuse ids before a slower server landed its first heartbeat. Two pulses still halves the fresh-cluster first-write stall versus the full warmup cap, and known-fresh setups keep the "0s" override. * master: pick once more when assign times out as growth concludes The pick loop's final attempt can predate the growth it triggered concluding: growth registers its volumes in the layout before clearing the pending flag, so a timeout landing on that boundary used to shed a hard 406 over volumes that had just arrived. Pick once more after the loop in both the http and grpc assign paths before deciding how to shed. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
81c90cb446 |
filer.backup: retry a failed checkpoint read instead of replaying from the epoch (#12036)
* filer.backup: retry a failed checkpoint read instead of replaying from the epoch Fixes #12035 * test: reset sink.local.directory in backup offset test cleanup The new test sets sink.local.directory on the shared viper but only reset the enabled flag, leaking a key that points at a deleted t.TempDir() to later tests in the package. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover the retried attempt re-reading the checkpoint The fake filer now fails only its first KvGet, and the test runs a second doFilerBackup attempt asserting the checkpoint is re-read and the attempt proceeds past the offset stage. LookupDirectoryEntry is stubbed so the SSE-S3 KEK probe returns a miss instead of retrying 10x2s. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |