mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-30 00:40:44 +01:00
eec2452e30dbec1108b9c4b0af79932f61d0fd9e
15846
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandDevin
|
eec2452e30 |
master: replicate the recovery-wait growth unfreeze through raft (#12165)
* master: replicate the recovery-wait growth unfreeze through raft state_loss_recovery_wait lifted the volume-growth freeze only in memory, while the freeze itself was replicated. A restart replayed the frozen flag and growth stayed refused until the wait expired again, on every restart, unless an operator ran volume.growth.resume. The lift now applies GrowthFrozen=false like ResumeVolumeGrowth, from a goroutine since IsGrowthFrozen is a read path that cannot block on raft. The apply runs under reportedTopologyIdsLock so a freeze landing after the lift keeps its command last in the log; the stale unfreeze skips itself when the flag is frozen again. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: cover raft replication of the recovery-wait unfreeze Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: retry and re-arm a growth unfreeze raft failed to record applyGrowthFreeze swallowed the raft error, so a rejected lift during a leader change left the flag clear while the log still held the freeze; nothing retried and a restart re-froze growth anyway. The apply now returns the error, replicateGrowthUnfreeze retries it, and on persistent failure re-arms the flag so the next IsGrowthFrozen attempts the lift again instead of diverging from the log. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: release the topology-id lock before the freeze raft write CheckReportedTopologyId and SetTopologyInitPending held reportedTopologyIdsLock across the blocking raft apply, so a slow apply stalled every later report — including the heartbeat that should trigger the next freeze. The lock now covers only the mismatch decision; the replicate step runs after it is released. * master: bind the recovery unfreeze to the freeze instant it lifts An unfreeze submitted while a newer freeze sat committed but not yet applied — e.g. after this node won an election — could land behind that freeze in the log and erase it. The command now carries the freeze instant cluster-wide (GrowthFrozenAt) and the lift is bound to the instant its submitter observed (GrowthUnfreezeFor): the state machine no-ops a lift while a strictly newer freeze exists, which also lets the unfreeze run without holding the topology-id lock at all. Concurrent freeze writes keep their own mutex so log order cannot regress the instant. * master: a superseded unfreeze retry leaves the resumed flag alone replicateGrowthUnfreeze broke out of its retry loop when a newer\nfreeze superseded the lift, but the break fell through to the\nre-arm: an operator resume in between was silently undone. Return\ninstead — the newer freeze generation owns the flag. * master: replayed freeze entries without an instant stay ageless Freeze commands written before the instant field existed carry zero.\nSubstituting the replay time made each restart stamp the freeze as\nnew, so a committed bound unfreeze always looked older and was\nrejected — the freeze came back on every restart. Keep the zero\ninstant: the recovery window reads as elapsed and bound lifts apply. * master: pack the growth freeze flag and generation into one atomic Separate flag and timestamp atomics left three holes: a resume's\nload-and-clear could lift a generation stamped in between, a bound\nlift compared stamps numerically although masters' clocks need not\nbe monotonic, and an ageless generation had no identity a bound lift\ncould match without colliding with the unconditional encoding.\n\nOne value now carries the state — 0 unfrozen, -1 an ageless freeze,\nelse the freeze instant — so every transition is a single atomic\nswap or compare-and-swap bound to an exact generation. * master: a repeat mismatch report keeps the armed freeze generation Re-stamping while already frozen never reaches the raft log, so the local generation diverges from the replicated one and every bound unfreeze becomes a no-op on replay — the freeze returns after restart or failover. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: snapshots read the freeze flag and generation in one load Save, Snapshot, and PersistMaxVolumeId read IsGrowthFrozen and the generation separately, so a resume landing between them persisted frozen=true with stamp 0 — restoring an ageless freeze a committed bound lift cannot clear. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Nguyễn Lộc Phúcandchrislusf
|
b62bbb99fc |
fix(iceberg): serve a registered table's metadata on load and commit (#12159)
* iceberg: serve a registered table's metadata on load and commit RegisterTable saved only the metadata location, and loads and commits never read the file there. They served a placeholder with no schema, no snapshots and a new random table UUID on every load, so a registered table read as empty, an INSERT failed against the empty schema, and every other commit failed its assert-table-uuid requirement. Metadata uploaded through the S3 gateway, as a migrated table's is, is stored in chunks that registration did not read, so it registered an empty table. Read the file following its chunks, reject one that is not Iceberg metadata with a 400, and keep its content and table UUID in the record the way CreateTable does. Records registered before this change hold only the location. Loads and commits now read the file there, and the first commit stores it in the record under the file's table UUID. * iceberg: transactions read a location-only registration's metadata file * iceberg: cover transactions on a location-only registration * iceberg: refresh the default name mapping on transaction commits * iceberg: cover name-mapping refresh in transactions * iceberg: roll a transaction back to a registration's metadata file A transaction that failed after flipping a table registered before the record kept its metadata could not undo the flip. The rollback restored the record's prior metadata, which for such a table is only a location, and UpdateTable keeps the full metadata a flip stored when the restore sends none. Loads went on serving the change the client was told had failed, and a client retrying the transaction could apply it twice. Restore the metadata loads served from the file instead, under the file's table UUID. * iceberg: register oversized metadata by its location Registration sends the metadata file's content to the S3 Tables manager, which refuses a request over its 10MB body limit, so a table whose metadata is larger could no longer be registered at all. Register such a file by its location and table UUID alone; loads read it from that location, as they do for a record that never kept its metadata. Export the limit as s3tables.MaxRequestBodySize so the catalog can check against it. A commit to such a table still sends the whole metadata to UpdateTable and stays over the limit. * iceberg: commit oversized metadata by its location * iceberg: let maintenance follow a location-only table record * admin: read a table's metadata file when the record keeps none * iceberg: bound the metadata file a location-only record reads * iceberg: bound only chunked metadata reads in the worker Inline content is what the catalog itself wrote and the REST load path accepts it at any size, so rejecting it in loadMetadata stranded tables over 64 MiB from every maintenance scan. Cap only the chunked read, where the recorded location can name any object in the bucket. --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> |
||
|
Jay Hemnanianduser.email
|
a816c6202c |
sftp: fail rm and rmdir of a missing path, and mkdir of an existing one (#12168)
* sftp: fail rm and rmdir of a missing path, and mkdir of an existing one * sftp: create directories exclusively --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
3fd8d0cf09 |
master: let location-constrained assigns wait for pending growth (#12167)
* topology: add AvailableSpaceWithinLocation for location-scoped free slots AvailableSpaceFor answers for the whole topology; a location-constrained assign needs to know whether the requested data center, rack, or data node still has room before deciding growth can satisfy it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: let location-constrained assigns wait for pending growth An assign carrying a data center or rack constraint failed on the first unsatisfied pick, even when the same request had just queued volume growth for that location — so the first write into a fresh collection errored out instead of picking up the volume growth landed a moment later. Keep the fail-fast path only when growth cannot help: growth disabled, no growth pending, or no free slots left inside the constraint. Otherwise the request takes the same wait/shed path as an unconstrained assign. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: sum leaf free slots in AvailableSpaceWithinLocation Rolled-up rack and data center counters keep a maintenance-mode server's capacity, and one overcommitted node can cancel a sibling's slots. Sum the matching data nodes directly so the location check agrees with what growth can actually use. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: let constrained growth initiators reach the shed path When a constrained assign's own growth concluded unfulfilled, the fast-fail check saw only that no grow was pending and returned the plain pick error — a code the assign client does not retry, so the first write died instead of re-triggering growth. Exclude the request's own initiated grow from the no-pending-growth condition so it reaches the existing re-pick and ResourceExhausted shed like an unconstrained assign. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: judge targeted grow requests by the location they name A grow request carrying a data center, rack, or data node constraint was discarded whenever the layout's writables elsewhere were healthy, so a location with free slots but no writable could never receive the volume its assign was waiting for. Add ShouldGrowVolumesWithinLocation and use it for constrained requests; the layout-wide check still governs untargeted ones. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: only positive leaf slots count in AvailableSpaceWithinLocation An over-committed node reports negative free slots; summed plainly it\ncancelled a sibling's room and a constrained assign could fail fast\nwhile the location still had space to grow into. * master: fail constrained assigns fast when growth cannot place A constrained assign with free slots in its location still waited out\ngrowth that could never satisfy the replica placement — a single-rack\ndata center asked for 010 sheds the same doomed grow on every client\nretry. Run the grow-time node search as a dry run and return its\nplacement error instead of a retryable shed. * master: judge constrained growth by layout, not free slots A slot-level feasibility probe races with the grow it judges: a\npending grow consuming the last slot in one rack reports the\nplacement impossible while it mints the request's own volume, and\nthe hard error skips the re-pick that would find it. GrowthBlocker\nnow counts racks and data nodes inside the named location — shape a\ngrow cannot consume — so only genuinely unplaceable layouts fail\nfast. * topology: GrowthBlocker mirrors the grow selector's feasibility The location filter bound every replica tier, so a 010 request naming one rack needed two racks inside it while the selector constrains only the first placement and counts the rest by room. And counting nodes without free slots let a full location queue a grow it could never serve, so the assign shed ResourceExhausted at timeout instead of failing fast. GrowthBlocker now applies the option's location to the primary placement chain only and leaf-sums free slots the way the selector does, so it reports a blocker exactly when the grow can never place. * master: repick once before surfacing a growth blocker An in-flight grow can spend the last free slot minting the request's own volume: the blocker's slot counts predate that registration by a heartbeat, so returning the placement error on the first failed pick could reject a write whose volume lands a moment later. A pending grow now earns one repick grace before the blocker error is returned. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
18691e7b5b |
server: keep the first write on a fresh weed server from stalling (#12149)
* master: end the post-mint allocation hold on the embedded reporter's clean inventory DeferAllocation armed a warmup-length hold whenever the leader minted a fresh topology id, and on weed server the hold ran the whole window: the embedded volume server's join kept refreshing the quiet evidence clock, so the two-pulse early release still cost the first write 10-15s on every fresh start. weed mini already releases on its single pinned reporter's first inventory, but pinning refuses outside volume servers, which weed server must keep admitting. Add a non-exclusive variant: the topology names one expected reporter whose first clean full inventory lifts the hold, while other reporters stay admitted and still feed prior-state evidence and the topology-id freeze. A single-master weed server points it at its embedded volume server — the one reporter guaranteed to arrive quickly; a veteran's heartbeat carries prior state and keeps the window through the regular checks, so a wiped -dir with surviving external volume servers still trips the freeze on first report. Restarted clusters are untouched: a recovered topology id means no mint and no hold. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: announce on the startup peer scan for embedded single-master setups A fresh filer whose startup peer scan came back empty re-polled for the full confirm window before registering on the master, so the lock ring had no member and routed traffic such as S3 PUTs spun on "no lock server found" for ~10s on every fresh weed server / weed mini start. For setups that start with exactly one filer the scan is already final: a single-master weed server and weed mini announce as soon as it completes. A late external filer still joins and replays through the normal path — the confirm window only ever caught peers arriving inside it, and the residual risk mirrors the accepted volume-side trade-off: an embedded filer announcing a few seconds earlier can own the lock ring while a surviving external filer's backlog is still on its way. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: retry a failed heartbeat promptly until first registration A heartbeat stream refused before the leader is known — the common race when a volume server starts while an election is still running — always fell back to sleeping out the full pulse, so a fresh volume's first registration could wait ~5s behind a leader elected a second later. The peer refresh already identifies the leader during the wait, so cap the retry pause at a second once a leader is named; while no leader is known the same short retry applies until the first stream is confirmed, since an unregistered server has nothing else to do. Once the server was registered the pulse backoff is unchanged, so a lost leader still paces reconnects the way it did before. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: retry a failed heartbeat promptly until first registration (rust) Mirror the Go heartbeat retry: cap the post-refresh wait at a second when a leader is named, and keep the short retry until the first stream is confirmed so a fresh volume registers promptly when it races an election. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: leave admitted reporters a confirm pulse when the embedded report ends the hold The single-master weed server releases its post-mint allocation hold on the embedded volume server's clean report, but outside volume servers stay admitted and one may still carry prior cluster state — an inventory from the embedded server alone cannot prove none is due. Clearing the hold outright let a rebuilt master re-issue a volume id a reconnecting veteran still owned. Collapse the released hold to one heartbeat interval instead: late reporters land their evidence or a topology-id freeze inside the pulse, and a re-mint re-arms that pulse rather than the full warmup. The pinned single-reporter release (weed mini) still drops the hold outright, since no other reporter can be admitted. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: keep the empty-membership confirm on embedded single-master weed server A single-master weed server still admits external filers, so an empty startup peer scan is not final: after a master restart the existing filers simply have not reconnected yet. Announcing on the scan let the embedded filer take lock-ring ownership and serve writes that a peer's replayed backlog could then overwrite. Only weed mini keeps the skip, where the embedded filer is the only member its dev cluster can have. The confirm window only delays the master listing for a fresh store; the filer serves while it runs. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: tolerate an expired heartbeat retry deadline (rust) refresh_leader_from_peers can consume the whole pulse budget, leaving retry_at in the past; subtracting a later Instant panics and kills the heartbeat task instead of retrying. Match the Go side, where time.Until of a past deadline just sleeps zero. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: retry the seed list without a second pulse sleep (rust) The error path already slept its retry delay; an extra full pulse after losing the round delayed the next attempt past what Go's heartbeat() does, which paces retries only inside the per-master attempt. * master: keep a configured allocation hold shorter than the confirm pulse Once the embedded report lands, the collapsed hold replaced the configured deferral with a full pulse even when the operator set a shorter one, including zero (disabled). Cap at the pulse instead. * filer: end the empty-membership confirm early on a fresh-minted embedded master On a fresh single-master weed server the announce gate still held the filer for the full confirm window, so the master's lock ring stayed empty and the first write retried 'no lock server found' for ~7s on top of the collapsed allocation pulse. A fresh TopologyId mint proves the master's persisted state was empty, making the empty peer scan final; skip the rest of the confirm only then, and keep it whenever the master recovers a prior id. * filer: keep a short reconnect window on a fresh-minted embedded master A fresh TopologyId mint proves the master's persisted state was empty, not that no external filer survived its loss — one can still be mid-reconnect on its keepalive cycle when the confirm starts. Ending the empty-scan confirm on the first poll let the fresh filer register and serve routed reads before that peer was found and backfilled, so existing files could read as missing. The fresh-mint path now shortens the confirm to a few keepalive cycles instead of skipping it: a reconnecting filer still lands in time, while a genuinely fresh cluster announces after ~3s rather than ~10s. * volume: skip the heartbeat loop when no seed masters are configured An empty seed list made the outer loop spin with no await between passes and no path to observe the stop flag. A masterless volume server has nothing to register with, so the heartbeat exits after a warning. * volume: skip the heartbeat loop when no master addresses are given (rust) Mirrors the Go heartbeat: an empty seed list spun the outer loop with no await and no shutdown checks. Exit after a warning so library callers get the same behavior as the binary path that validates the list up front. * filer: re-check the fresh-mint hint on each confirm pass The topology id is minted after raft election, which can land after the empty-scan confirm has already started — a one-time check at entry missed the mint and held the full window. Evaluate the hint on every poll and shorten the confirm from the first observed mint, still capped by the normal confirm deadline. * master: mark a fresh topology mint only when the volume bound is zero Recovery restores the max volume id unconditionally but the topology id only when present, so an upgraded or partially rebuilt master can mint a new id while still holding prior volume state. Marking fresh there would let the embedded filer shortcut its peer confirm on a cluster that is not empty; only mark the mint fresh when no bound survived. * filer: drop the fresh-mint shortcut once a peer has been observed --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Jay Hemnanianduser.email
|
9059cc2a07 |
s3: allow PutObjectAcl when auth is off (#12155)
* s3: allow PutObjectAcl when auth is off * s3: ownerless objects claim the admin account when auth is off An object written through the filer carries no owner metadata, so objectOwner fell back to an empty account id and ExtractAcl rejected the canned ACL's canonical grantee. Extend TestPutObjectAclWithoutAuth to cover the ownerless case. * s3: GetObjectAcl reports the admin owner for ownerless objects when auth is off The same empty-owner fallback produced an ACL response with a blank Owner ID and a blank canonical grantee. * s3: ignore the client-supplied account id when auth is off Auth does not strip AmzAccountId on the iam-disabled path, so the ownerless fallback could persist a forged owner. requestAccountId substitutes the admin account, matching the idiom already used in the upload ACL path. --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
b3d0caabe2 |
vector: only follow a leader hint that points elsewhere (#12164)
* vector: only follow a leader hint that points elsewhere The master sends volume_location on every KeepConnected response — the leader always names itself, so the worker exited the stream on the first response and flapped every 3s, leaving ListClusterNodes to see the node intermittently. Only redirect when the hint points at a different master. * vector: leader hints carry host:http.grpc — normalize before comparing A self-hint arrives as host:9333.19333 while --masters gives host:9333, so the raw comparison still redirected and grpc_addr could not parse the dotted port. grpc_addr now prefers the explicit .grpcPort, and the hint comparison normalizes both sides to host:httpPort first. |
||
|
Chris Lu
|
c2863d9a4f |
vector: owner-prefixed lease tags stop cross-server reaping (#12158)
* vector: owner-prefixed lease tags; foreign leases get a long backstop
reap_expired exempted only process-local LIVE_TAGS — server B could
reap server A's live query lease at the 1h TTL. Tags now carry a
per-process instance id: own leases use the short TTL, foreign leases
only the 24h backstop, so a live query elsewhere keeps its pin while
dead-process tags still collect eventually.
* vector: pick the lease-tag format by segment count, not numeric-ness
An owner that is all decimal digits parsed as the legacy timestamp, so
its real timestamp was ignored and the lease never expired. The third
segment is what distinguishes {instance}-{ts}-{uuid} from {ts}-{uuid}.
|
||
|
Chris Lu
|
7a25983266 |
vector: cover index names in named_segment_fragments (#12161)
vector: cover the index name in named_segment_fragments The planner groups tasks by index name — assert the helper actually returns the logical name so a regression dropping it fails. |
||
|
Chris Lu
|
d5433ae745 |
vector: cover the identity object in s3vectors auth (#12162)
vector: cover the identity object in the s3vectors auth context The auth middleware must carry the resolved identity object — only the name survives otherwise and authorization loses actions, account, and claims. |
||
|
Jay Hemnanianduser.email
|
9b3f09d0a5 |
sftp: refuse mkdir and uploads under a missing folder (#12156)
* sftp: refuse mkdir and uploads under a missing folder * sftp: re-check the parent directory when a buffered upload commits The open-time check cannot catch a parent removed while the upload buffered; the filer would recreate it when the entry lands. * sftp: cover mkdir and upload commit against a missing parent Extend the test filer with CreateEntry so makeDir can be exercised end to end. * sftp: refuse a rename into a missing folder The filer's rename path creates the target entry through CreateEntry, which recreates absent parents the same way uploads did. --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
afb8472c5e |
s3vectors: cover non-filterable key round-trip and nested filter rejection (#12160)
* s3vectors: cover non-filterable key round-trip and nested filter rejection The catalog writes the key list as a JSON string while older records may still carry the array form; indexFromRecord must decode both. The rejection path extracts field names through nested $and/$or filters, so a blocked key deep in the tree still fails the query. * s3vectors: factor the non-filterable rejection into a testable gate Pull the blocked-key check out of handleQueryVectors into nonFilterableField so the rejection itself is covered without standing up the catalog stack; the handler now 400s on whatever the gate names. * s3vectors: report the non-filterable match separately from the key An empty metadata key in nonFilterableMetadataKeys collided with the "no match" sentinel, so a filter on that key skipped rejection. Return a match flag and cover the empty-key case in the test. |
||
|
Chris Lu
|
7d823265e5 |
wdclient: regression test for the elapsed caller-deadline lookup path (#12154)
* wdclient: cover a lapsed caller deadline that Err() has not observed yet * wdclient: reuse isCanceledErr for the elapsed-deadline check * wdclient: reinterpret only deadline errors as caller-gave-up isCanceledErr also matches gRPC Canceled, which a filer can return for its own reasons; treating it as an elapsed caller deadline hid the failure from the filer's health record. The fallback now matches only context and gRPC DeadlineExceeded, so filer-side cancellations keep counting as filer failures. |
||
|
dependabot[bot]andchrislusf
|
2b7c65048d |
build(deps): bump com.google.guava:guava from 32.0.0-jre to 33.7.1-jre in /other/java/client (#12147)
* build(deps): bump com.google.guava:guava in /other/java/client Bumps [com.google.guava:guava](https://github.com/google/guava) from 32.0.0-jre to 33.7.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-version: 33.7.1-jre dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * other/java/client: bump guava to 33.7.2-jre --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
237db29dca |
vector: round-trip and enforce non-filterable metadata keys (#12157)
vector: non-filterable metadata round-trips and filters reject it - GetIndex omitted nonFilterableMetadataKeys — the catalog stores the list as a JSON string but the reader only accepted an array; both forms now decode. - QueryVectors now rejects filters referencing non-filterable keys instead of silently scanning them. |
||
|
Chris Lu
|
190ab32cfb |
vector: weed-vector registers to master; s3 discovers it there (#12146)
* vector: weed-vector registers with the weed master weed-vector-server --masters holds a KeepConnected stream to the weed master, the same mechanism filer and s3 use via wdclient — the server shows up in ListClusterNodes(client_type=vector). Resends registration and rediscovers on failure. * vector: s3 discovers weed-vector through the master With -vectors.grpc unset the vectors gateway resolves one registered weed-vector via ListClusterNodes on the masters it already derives from filer config — discovery replaces the static dial for the single-worker OSS path. The conformance harness registers the engine to mini's master and drops -vectors.grpc. * vector: default-fill registration fields ..Default::default() keeps the source compiling against proto variants that carry extra fields, so the same file works in either tree. * vector: serve the Vectors API on the main s3 port The vectors router sits on the main -port listener behind a dispatch that checks the SigV4 credential scope for /s3vectors/ — no separate port needed. -port.vectors stays for explicit dedicated-port deployments; the conformance harness now hits vectors through :8334. * vector: drop -port.vectors entirely It was new — no compat to preserve. The main -port listener serves the API whenever a weed-vector backend is reachable. * vector: drop -vectors.grpc; discovery is the only locator weed-vector always registers to master, the gateway always resolves it through ListClusterNodes — a static address just forks membership. * vector: binary is weed-vector Matches the enterprise binary; package and lib names are unchanged. * vector: review fixes — cluster type, grpc port, lazy resolve, leader hint, advertise - cluster: register 'vector' as a node type so ListClusterNodes returns weed-vector members - s3: dial masters via ToGrpcAddress (HTTP port + 10000) - s3: backend resolves lazily through vectorDiscovery — workers joining after gateway startup work on first query, and address changes re-dial - master_client: convert master HTTP addr to grpc +10000, honor http(s) scheme, follow the leader hint in KeepConnectedResponse, and --advertise covers wildcard/unreachable listen addresses * vector: no --advertise flag; derive the outbound address A wildcard listen resolves to the source IP the master sees via a route lookup, so registration always advertises something reachable without another flag. * vector: review fixes — plaintext dispatch, conn leaks, discovery backoff The two plaintext http servers now serve the vectors dispatch too — signed s3vectors calls on the main port reach the right router. discoverVectorWorker closes each master conn and bounds lookups to 5s; a dead or absent worker no longer re-hits master on every RPC — the resolver backs off for 5s between rediscovery attempts. * vector: 1 GiB grpc cap, dead-worker invalidation, CLI rename Valid PutVectors batches exceed the tonic 4 MiB default — match the gateway's 1 GiB cap. A replaced worker no longer sticks: any RPC error resets the cached address and the next call rediscovers through the master. The core CLI bin renames to weed-vector-cli so the two crates do not collide on the same output path. * vector: retry-only-transport reset, periodic re-resolve, cli rename - reset() takes the failed client and runs only on transport errors — an InvalidArgument on a healthy conn no longer interrupts unrelated requests, and a concurrent re-resolve wins the race. - cli() re-resolves once the interval passes even with a live client, so a worker that moves addresses is picked up. - verify.rs spike docs use weed-vector-cli after the bin rename. * vector: only Unavailable counts as a transport failure A request deadline or an application error can fire on a healthy connection — resetting the shared conn on those churned reconnects. |
||
|
Chris Lu
|
81879fba7b |
vector: preserve typed error chains in lease pin/release (#12153)
vector: keep typed error chains in lease pin/release Error::msg discarded the source chain — denied() could not downcast to object_store::PermissionDenied, so try_pin surfaced the error instead of degrading to an unleased query and read-only credentials broke GetVectors/QueryVectors. Plain ? preserves the chain. |
||
|
Chris Lu
|
a2863bf83b |
wdclient: count an elapsed caller deadline as caller-gave-up in LookupVolumeIds (#12151)
* wdclient: treat an elapsed caller deadline as caller-gave-up on lookup errors gRPC can surface its own DeadlineExceeded a tick before ctx.Err() becomes visible, so a lookup that died on the caller's deadline was sporadically recorded as a filer failure (seen in the linux/386 job). * wdclient: reinterpret only deadline errors against an elapsed caller deadline |
||
|
Chris Lu
|
67ce9a5f6d |
vector: expose index names in segment planning (#12150)
vector: named_segment_fragments carries the logical index name Sharded builds create one named index per shard and a with_index_segments scan must not mix names — a fanout planner needs the name to group tasks, so expose it alongside uuid and coverage. |
||
|
Chris Lu
|
10ba1b782f |
vector: carry the full identity through s3vectors auth (#12152)
vector: keep the full identity in the request context Auth discarded the resolved identity object — the s3tables manager request then re-resolved by name and denied authenticated users their actions, policies, account, and claims. Store the object via SetIdentityInContext so downstream authorization sees it. |
||
|
dependabot[bot]
|
b6851e561f |
build(deps): bump github.com/go-jose/go-jose/v4 from 4.1.5 to 4.1.6 (#12148)
Bumps [github.com/go-jose/go-jose/v4](https://github.com/go-jose/go-jose) from 4.1.5 to 4.1.6. - [Release notes](https://github.com/go-jose/go-jose/releases) - [Commits](https://github.com/go-jose/go-jose/compare/v4.1.5...v4.1.6) --- updated-dependencies: - dependency-name: github.com/go-jose/go-jose/v4 dependency-version: 4.1.6 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]anduser.email
|
3b65169d6c |
build(deps): bump com.google.guava:guava from 32.1.3-jre to 33.7.1-jre in /test/java/spark (#12141)
* build(deps): bump com.google.guava:guava in /test/java/spark Bumps [com.google.guava:guava](https://github.com/google/guava) from 32.1.3-jre to 33.7.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-version: 33.7.1-jre dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * test/java/spark: bump guava to 33.7.2-jre 33.7.1 is covered by a published advisory for uncontrolled memory allocation during CompactHashMap/CompactHashSet deserialization; 33.7.2 is the patched release. * test/random_access: bump guava to 33.7.2-jre Same deserialization advisory; the pin predates the fix. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
dependabot[bot]
|
bb5375d2b9 |
build(deps): bump rustls from 0.23.45 to 0.23.46 in /seaweed-volume (#12144)
Bumps [rustls](https://github.com/rustls/rustls) from 0.23.45 to 0.23.46. - [Release notes](https://github.com/rustls/rustls/releases) - [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md) - [Commits](https://github.com/rustls/rustls/compare/v/0.23.45...v/0.23.46) --- updated-dependencies: - dependency-name: rustls dependency-version: 0.23.46 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Ege Zambelliandchrislusf
|
89cc910a86 |
helm: render the filer's mysql credentials once when the user sets them (#12139)
* helm: render the filer's mysql credentials once when the user sets them With the MySQL filer store, the filer reads WEED_MYSQL_USERNAME and WEED_MYSQL_PASSWORD from the chart's db secret. Setting either one in extraEnvironmentVars or secretExtraEnvironmentVars, as helm_ci.yml does for the password, added a second entry with the same name: Helm 4 rejects the install (env: duplicate entries for key), and Helm 3 deploys with a warning that one definition may be dropped on apply. The db secret's entry is now left out for a key the user sets, so each key is rendered once and the user's value wins, the rule #8488 applied to a key set in both global and filer extraEnvironmentVars. Without an override the filer StatefulSet is unchanged. * helm: render the selected filer mysql credential ahead of the merged vars A credential the user sets now renders in the early position the db secret's entries used to occupy, and is excluded from the merged and secret loops. Kubernetes resolves a $(VAR) reference only against names declared earlier, so a later entry such as WEED_MYSQL_DSN can again reference the active credential. Secret-backed values win over plain ones; the db secret's entry remains the fallback. * helm: let a secret-backed filer env key replace the plain one secretExtraEnvironmentVars now wins over extraEnvironmentVars for the same name, rendering once at the merged key's sorted position instead of twice. The values.yaml WEED_MYSQL_* defaults no longer duplicate when overridden through the secret map - e.g. enabling the store via a secret WEED_MYSQL_ENABLED also rendered the "false" default before, which Helm 4's server-side apply rejects as a duplicate. * helm: keep a credential that references another env var in sorted position A user-supplied credential whose value itself holds a $(...) reference stays in the merged vars' sorted order so its own references still resolve; only reference-free values, valueFrom entries, and the db-secret defaults move to the early slot. * helm: render vars a filer mysql credential references ahead of it Hoisting the selected WEED_MYSQL_USERNAME/PASSWORD entries above the merged extra vars let WEED_MYSQL_DSN-style $(...) references resolve, but it broke the opposite direction: a user-set credential such as WEED_MYSQL_USERNAME: "$(DB_USER)" was emitted before DB_USER, so Kubernetes left the reference literal. Pull every merged or secret var a credential value transitively references out of the sorted set and emit them first, dependencies before dependents, so both directions resolve. * helm: emit filer mysql credentials after the chart's own env vars The dependency-ordered credential block rendered before SEAWEEDFS_FULLNAME and the s3 credentials, so a user-supplied WEED_MYSQL_USERNAME could not reference them via $(...). Move the block below those fixed entries so references in either direction resolve. * helm: bound filer mysql credential ordering passes by var count Chains deeper than the fixed eight passes forced a credential out ahead of the vars it references. Cap discovery at the number of merged/secret vars and emit passes at the ordered list length, so any resolvable chain finishes before the last-pass fallback emits cyclic leftovers. --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> |
||
|
Ege ZambelliandChris Lu
|
3a0af308ce |
helm: keep the filer db credentials across uninstall and install (#12138)
* helm: keep the filer db credentials across uninstall and install The db secret is kept on uninstall (`helm.sh/resource-policy: keep`), but it is a pre-install hook, and Helm deletes an existing hook resource before creating it again. The next install therefore wrote a new random password and reset the user to YourSWUser, so the filer could no longer log in to a database set up with the old ones, including credentials edited into the kept Secret. The template now reads the existing Secret with lookup and reuses its user and password; a first install still generates them as before. A kind step in helm_ci.yml installs, edits the Secret, uninstalls, installs again and checks both values. * helm: README, the db secret is kept and reused across a reinstall * helm: make the db secret hook-delete-policy explicit before-hook-creation is Helm's default for hook resources, but spell it out so the reuse contract is obvious: lookup reads the kept Secret at render time, the old copy is deleted before the hook recreates it, and the stored credentials carry over. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
dependabot[bot]
|
1e9cc44697 |
build(deps): bump rustls from 0.23.45 to 0.23.46 in /seaweed-worker (#12142)
Bumps [rustls](https://github.com/rustls/rustls) from 0.23.45 to 0.23.46. - [Release notes](https://github.com/rustls/rustls/releases) - [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md) - [Commits](https://github.com/rustls/rustls/compare/v/0.23.45...v/0.23.46) --- updated-dependencies: - dependency-name: rustls dependency-version: 0.23.46 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
f16f234c53 |
build(deps): bump org.xerial.snappy:snappy-java from 1.1.10.4 to 1.1.10.9 in /test/java/spark (#12143)
build(deps): bump org.xerial.snappy:snappy-java in /test/java/spark Bumps [org.xerial.snappy:snappy-java](https://github.com/xerial/snappy-java) from 1.1.10.4 to 1.1.10.9. - [Release notes](https://github.com/xerial/snappy-java/releases) - [Commits](https://github.com/xerial/snappy-java/compare/v1.1.10.4...v1.1.10.9) --- updated-dependencies: - dependency-name: org.xerial.snappy:snappy-java dependency-version: 1.1.10.9 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
7b0e57eada |
build(deps): bump github.com/go-jose/go-jose/v4 from 4.1.5 to 4.1.6 in /test/kafka (#12140)
build(deps): bump github.com/go-jose/go-jose/v4 in /test/kafka Bumps [github.com/go-jose/go-jose/v4](https://github.com/go-jose/go-jose) from 4.1.5 to 4.1.6. - [Release notes](https://github.com/go-jose/go-jose/releases) - [Commits](https://github.com/go-jose/go-jose/compare/v4.1.5...v4.1.6) --- updated-dependencies: - dependency-name: github.com/go-jose/go-jose/v4 dependency-version: 4.1.6 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
a1f6621553 | docs: regenerate star history chart | ||
|
Ege Zambelli
|
ef16b4ed02 |
helm: render numbers and booleans in extraEnvironmentVars as values (#12137)
Only a string rendered as `value`; anything else went to `valueFrom`. An unquoted number or boolean in a values file or `--set`, such as `WEED_MYSQL_PORT: 3306` or `WEED_MYSQL_ENABLED=true`, rendered as `valueFrom: 3306` and the API server rejected the workload. A map still renders as `valueFrom`, a null still renders an empty variable, and strings are unchanged. Numbers go through toJson because values files decode them as float64, and quote would print 1000000 as 1e+06. |
||
|
Chris Lu
|
5f26ef6ad1 |
filer,webdav,iam,sftp: add -idleTimeout flag for connection idle timeout (#12136)
* filer: make the HTTP connection idle timeout configurable The filer applied a hardcoded 10s activity timeout to its HTTP listener. A buffering proxy in front of the filer can stall upstream reads longer than that when the end client drains slowly, so large downloads through it were truncated (#8062). Expose the value as -idleTimeout (default 10, unchanged), matching the existing -idleTimeout flags on the volume and s3 servers, and wire filer.idleTimeout through 'weed server' and 'weed mini' too. A nil option keeps the 10s default so embedders constructing FilerOptions directly keep working. * filer: apply the idle timeout to the readonly port too The -port.readonly listener ran without any connection timeout while the main API port timed out. Give it the same -idleTimeout so both ports behave identically. * webdav: make the HTTP connection idle timeout configurable The WebDAV server used the same hardcoded 10s activity timeout. Expose it as -idleTimeout (default 10, unchanged), including -webdav.idleTimeout when embedded via 'weed filer -webdav', 'weed server -webdav', and 'weed mini'. * iam: make the HTTP connection idle timeout configurable The IAM API server used the same hardcoded 10s activity timeout. Expose it as -idleTimeout (default 10, unchanged), including -iam.idleTimeout when embedded via 'weed filer -iam' and 'weed server -iam'. * sftp: make the connection idle timeout configurable The SFTP server used the same hardcoded 10s activity timeout. Expose it as -idleTimeout (default 10, unchanged), including -sftp.idleTimeout when embedded via 'weed filer -sftp' and 'weed server -sftp'. * command: clamp idle timeout seconds before converting to duration A seconds value past time.Duration's range would overflow on multiplication and silently disable the timeout. Bound it in one shared helper, which also deduplicates the nil-default handling at the four call sites. * command: return zero for non-positive idle timeout seconds A large negative seconds value could wrap on multiplication and come out positive. Guard before converting so non-positive input always disables the timeout. * command: apply idle timeout to local unix socket listeners The unix-socket listeners in filer, s3, and sftp served accepted connections with no activity deadline while the TCP listeners already wrapped conns in util.Listener. Wrap the socket listeners the same way so a positive idleTimeout also bounds idle socket connections. * command: clamp s3 idle timeout seconds before multiplying s3 still multiplied the flag inline, so an out-of-range value could overflow time.Duration. Route it through idleTimeoutSeconds like the other commands; non-positive keeps meaning disabled. * volume: clamp idle connection timeout seconds before converting * command: cover idleTimeoutSeconds conversion semantics * command: gate idleTimeoutSeconds clamp check on 64-bit int |
||
|
Chris Lu
|
6536fac9bd |
s3: answer NoSuchBucket for encryption and policy writes to a missing bucket (#12133)
* s3: answer NoSuchBucket for encryption and policy writes to a missing bucket PutBucketEncryption, DeleteBucketEncryption, and PutBucketPolicy surfaced the filer's bucket-miss as InternalError; GetBucketEncryption masked it as ServerSideEncryptionConfigurationNotFoundError. Map the miss to NoSuchBucket instead, matching AWS. Internal readers still treat a missing bucket as "no encryption configured" so uploads keep their behavior. * s3: wrap ErrBucketNotFound only for verified misses in bucket policy ops The filer lookup helpers in get/set/deleteBucketPolicy marked every LookupDirectoryEntry failure as bucket-not-found, so a transient filer outage answered NoSuchBucket for a live bucket. Route them through filer_pb.LookupEntry, which normalizes only true misses to ErrNotFound. * s3: answer NoSuchBucket when the policy delete's own write loses the bucket * s3: classify a filer NotFound on the policy UpdateEntry as ErrBucketNotFound * s3: require the bucket entry to exist on config patch transactions * s3: answer NoSuchBucket when tagging and CORS writes lose the bucket mid-request * s3: keep the existence check on no-op bucket config patches An unchanged config write produced an empty patch that early-returned before the transaction, so a bucket deleted between the read and the write still reported success. Ship the condition-only transaction so IF_EXISTS runs on its own. * s3: guard the iam step and cover failure-vs-miss classification Nil-guard s3a.iam in the bucket policy handlers so the handlers do not panic when no IAM server is wired (unit tests, embedded servers), treat a "precondition failed" response string the same as the error code when mapping a failed patch to ErrNotFound, and add handler-level tests that pin the miss/failure distinction: missing buckets and vanished mid-write buckets answer 404 NoSuchBucket while transient filer failures stay 500 InternalError. |
||
|
Chris Lu
|
4c9e62177d |
s3: keep multipart part encryption on the initiate-time decision (#12134)
* s3: keep multipart part encryption on the initiate-time decision putToFiler applied the *current* bucket default encryption when writing a staged part, so a PutBucketEncryption landing between two UploadPart calls produced an object whose parts disagree: some chunks ciphertext, some plaintext. Completion then marks the object by its majority and GetObject fails decrypting the rest. Parts staged under .uploads now skip the bucket-default lookup; the decision snapshotted into the upload entry at CreateMultipartUpload is replayed through request headers instead, which matches AWS (default encryption applies at initiation). * s3: mark multipart part writes via request context, not the staged path Inferring "is a part" from the .uploads file path let an ordinary PutObject named .uploads/x.part skip bucket default encryption entirely. The part decision now travels on the request context from the two callers that stage parts (UploadPart, UploadPartCopy); a client-chosen key cannot reach it. |
||
|
Chris LuandDevin
|
660a826eab |
remote_storage/s3: send Content-MD5 on RemoveDirectory DeleteObjects (#12129)
S3DisableContentMD5Validation skips the request header that DeleteObjects requires, so AWS rejects every directory removal and the filer remote sync offset stalls. Issue the delete through DeleteObjectsRequest and re-enable the flag on that request's config copy so the SDK's own checksum handler adds the header without changing any other operation. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
7a47c63d54 |
volume: replicate FetchAndWriteNeedle writes over the configured TLS scheme (#12128)
* volume: replicate FetchAndWriteNeedle writes with the configured TLS scheme The guarded upload path hardcoded an http:// URL and used a bare *http.Client, so a peer fronted by https rejected the replica write and read-through of a remote-evicted object failed whenever the volume had replicas. Normalize the upload URL through the shared global client and let the guarded transport borrow its https.client TLS configuration, the same settings the regular replication path resolves. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: keep the global HTTPS downgrade guard on replica uploads Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: cover TLS replicate uploads and the redirect downgrade guard Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: pin the downgrade guard against a dialable plaintext target The previous redirect pointed at 169.254.169.254, which the guarded dialer already refuses, so the test passed even with CheckRedirect removed. Redirect to a live plaintext server on a dialable peer address and assert it never receives the resent body. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume-rs: keep the HTTPS downgrade redirect guard on outbound writes Mirrors the Go peer upload client: the shared outgoing HTTP client now stops redirects that would downgrade an https request to plaintext, so replicate uploads to TLS peers cannot leak data or auth headers via a 307 to http. * volume-rs: fail rejected redirects instead of returning the 30x response Go's rejectHttpsDowngrade uses ErrUseLastResponse, which surfaces to the caller as a non-2xx status it rejects; attempt.stop() returned the redirect response as Ok, so a refused downgrade could masquerade as a completed write to any caller that does not check the status. * volume-rs: fail replica writes on a non-2xx status fetch_and_write_needle discarded the replica response without checking its status, so a peer rejection — or a redirect the client refused to follow — was reported back as a successful copy while no replica was written. Go's doUploadData fails on any non-success status. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Hector Peeters
|
c46a42694b |
Fix s3 cancelled read relookup (#12130)
* filer: keep cached volume locations when the reader cancels * wdclient: do not count a caller's cancelled lookup as a filer failure |
||
|
Chris LuandDevin
|
af77406cf9 |
vector: index-segment restriction + partition helper (P3 seam) (#12124)
* vector: allow restricting ANN search to index segments Query gains an optional index_segments list; when set and the index path is used, the scanner restricts the vector index search to those segments via with_index_segments. This is the seam the Enterprise planner uses to fan one query out across per-segment workers; OSS keeps passing None. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: enumerate and partition index segments for fanout segments::index_segments lists the physical segment UUIDs covering the vector field at the pinned version; partition() assigns them round-robin to workers. The parity test proves two workers scanning disjoint shares merge back to the full-dataset top-K — the P3 exit gate, in-process. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: let a query carry fragment ids for the unindexed sweep Lance only applies the index-segment restriction to indexed rows — unindexed fragments outside the selected segments are skipped. For a partitioned query to stay correct, the planner needs to hand the unindexed sweep to a worker too, so Query gains a fragments list that the scanner resolves to manifest fragments via with_fragments. A task carrying both searches its segments via the index and its fragments with flat KNN in the same scan. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: empty scope returns no hits; cover the sweep in tests Some([]) fragments or index_segments fell through to an unrestricted scan — a worker assigned nothing would have searched everything. empty_scope short-circuits those queries, and the segment restriction only calls with_index_segments on a non-empty list. segments::segment_fragments also returns each segment's live fragment bitmap, and the parity test now leaves a fragment unindexed so the worker carrying it exercises the flat sweep — the contract the Enterprise planner relies on. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: scope the scan to listed ∪ segment-covered fragments Lance intersects with_fragments with the selected index segments, so a worker given its segment share plus the unindexed sweep lost the share entirely — only the listed fragments were scanned. Expand the fragment list with the live coverage of the selected segments. The list is now also applied when it resolves empty: stale ids or an empty assignment must scan nothing, not fall through to an unrestricted scan. The merge test now leaves a fragment unindexed so the combined path is exercised. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: partition assigns nothing when there are no workers n.max(1) returned every segment in one phantom partition for a nonexistent worker, silently dropping coverage. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
a5036f568f |
webdav: refuse MKCOL and PUT under a missing folder with 409 (#12122)
* webdav: refuse MKCOL and PUT under a missing folder with 409 RFC 4918 9.3 and 9.7.1 say a MKCOL or PUT whose parent collection does not exist must fail with 409 Conflict. The filer creates missing parents, so both succeeded and made the intermediate folders. Check the parent in Mkdir and in OpenFile with O_CREATE and return os.ErrNotExist, which x/net/webdav turns into 409. * webdav: check the PUT parent before an overwrite deletes the target OpenFile deleted an existing destination before the new parent lookup, so a PUT whose parent check failed left the old file gone. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * webdav: report a missing rename source as not-exist Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * webdav: refuse MOVE to a missing destination folder AtomicRenameEntry grows any missing parents itself, so a MOVE to /missing/file created /missing; check the destination parent first. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * webdav: answer 409 for COPY and MOVE under a missing collection The webdav handler reports a failed directory Mkdir or Rename as 403, so the destination parent is checked at the request layer instead. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * webdav: defer foreign-host MOVE and COPY destinations to the 502 handler * webdav: only check the destination parent for absolute paths * webdav: cover foreign-host and relative MOVE destinations in the guard test --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Jay Hemnani
|
5c14e566bc |
s3: allow bucket ownership controls when auth is off (#12121)
checkAccessByOwnership compared the caller's account with the bucket owner even with no identities configured, so Get/Put/Delete BucketOwnershipControls answered AccessDenied to every request. With auth off there is no owner to compare against, as the other auth-off checks already assume. |
||
|
Chris Lu
|
8557771fc2 |
build(deps): bump google.golang.org/grpc past the #9330 deadlock fix (#12123)
The pinned dev snapshot (v1.85.0-dev.0.20260915183914) predates grpc/grpc-go#9563, which fixes the control-buffer throttle deadlock (grpc/grpc-go#9330) that wedges weed mini under concurrent S3 writes over unix sockets. Verified with the issue's standalone repro (one ClientConn over a unix socket, concurrent unary RPCs): the old snapshot deadlocks at c=512 after 743 RPCs; the new snapshot completes 3.06M RPCs/20s at c=512 and 1.97M/15s at c=2048 with no stall -- above the v1.82.0 pre-regression baseline of ~112k ops/s. |
||
|
|
80f3cc0613 |
topology: keep oversized volumes out of the writable list across re-registration (#12120)
* topology: keep the oversized verdict after the location list is gone The per-replica oversized marks live in the volume's location list, which is deleted when its last location leaves and recreated flagless on the next registration. Between that recreation and the next heartbeat that re-measures the volume, ensureCorrectWritables saw no mark and re-admitted an oversized volume to the writable list; two other paths (quarantine release, mark-writable) never consulted the marks at all. VolumeLayout now remembers the oversized verdict per vid, written only by heartbeat sizes and forgotten only when a report measures the volume under the limit. The writable admission gate itself refuses a marked volume, and the vacuum commit path drops a verdict its live size check has just disproved. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: mark oversized at registration when the report carries a size RegisterVolumeLayout ran EnsureCorrectWritables while the recreated location list still held no per-replica marks. When the registering report knows the volume's size, refresh the oversized state before the writable check instead of waiting for the next heartbeat pass. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: test that oversized volumes stay out of writables across re-registration Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: remember the oversized verdict per replica A shared per-volume verdict was cleared by any single replica's under-limit report, so a still-oversized replica whose location entry had died could re-register from a size-less report and put the volume back on the writable list. Keep one verdict per reporting replica: an under-limit report clears only its own, and a replica that never returns cannot pin its volume out of writables forever because the verdict ages out. The vacuum commit now feeds the node's fresh answer through the same per-replica bookkeeping instead of erasing the whole verdict. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: count a volume at the size limit as full on vacuum commit The layout's oversized test is >= the limit, but the vacuum commit and status checks used >, so a replica sitting exactly at the limit passed as not-full and the commit path could re-admit it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: cover per-replica oversized verdicts in tests One replica shrinking must not clear a still-oversized replica's remembered verdict; cover that plus a size-less re-registration before the second replica is re-measured. The vacuum-commit case now shrinks the stored record first, since the commit re-derives the mark from the node's own answer. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: judge the vacuum commit by the size it just measured SetVolumeAvailable rebuilt the oversized verdict from the cached VolumeInfo, which still carries the pre-compaction size until the next heartbeat. A commit that shrank a volume under the limit re-marked the replica oversized and never restored writes. The commit path already asks every replica for its fresh size; pass it through per replica and record it on the mark, the verdict, and the size tracking seed. * topology: hold an expired oversized verdict while its replica is back The TTL forgives a replica that never returns, but a remount announces the volume without a size and registration skips the oversized check. Deleting the verdict on expiry then re-admitted a still-oversized volume until the next size-bearing heartbeat. Keep counting an expired verdict while its replica sits in the live location list, and keep sweeping verdicts whose replicas stayed gone. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chrislusf@gmail.com> |
||
|
Chris LuandDevin
|
d182be6a9e |
s3api: cap SOSAPI capacity by the reported physical disk bytes (#12118)
calculateClusterCapacity multiplies advertised volume slots by the volume size limit. Slot counts are an admission bound rather than a capacity measurement — an explicit -max can be set higher than the volumes a disk actually fills — so a small disk can report more total and available capacity than it physically has. Bound both numbers by the per-disk filesystem counters the volume server already reports in its heartbeat, falling back to the slot estimate only when they are absent. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
0ea818a44f |
mount: drain OFD locks when a file description is released (#12117)
* mount: drain OFD locks when a file description is released The kernel scopes OFD fcntl locks to the open file description but reports no lock_owner for them at RELEASE — it only sends one for flock holders — so an OFD write lock taken on a file handle outlived the holder and blocked every later open of that file (e.g. etcd failing to reopen its WAL after a restart). The mount now records the owner of each granted non-flock lock under its file handle and releases those owners on RELEASE, which also covers any POSIX locks left over when FLUSH could not run. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: give every open its own fh value over the shared handle AcquireFileHandle reuses one FileHandle per inode, so every open reported the same fh to the kernel. A RELEASE for one description could then not be told apart from a sibling still holding the inode open — draining lock owners recorded under the shared fh dropped locks belonging to descriptions that stay open. Each open now registers its own fh value in fh2inode, all resolving to the shared FileHandle, and RELEASE removes exactly the id that is closing. This also stops a duplicated RELEASE on one id from double-decrementing the open count. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: forget flushed lock owners so a later release cannot re-drop locks POSIX lock owners are process-scoped: Flush releases every lock the process holds on the inode, but the owner stayed recorded under each open's fh for the Release-time drain. If the process took a fresh lock through another open before an old description finished closing, the stale record made that RELEASE drop the new lock too. Flush now drops the released owner from every recorded open of the inode, so the Release drain only ever fires for owners whose locks were never flushed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: keep close-time lock owner records when the release fails A routed POSIX owner release that errors keeps the flush hint for a retry, but Flush also dropped the owner from every open's release records unconditionally — so the final Release skipped the retry and the owner filer's lock was stranded until lease reaping. Only forget the owner once its locks were actually released. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
b747e0d3b1 |
cluster: bound each DistributedLock attempt with a timeout (#12113)
* cluster: bound each DistributedLock attempt with a timeout An unreachable lock owner blocked the attempt on the connection timeout. Bound it so the write fails fast; a timed-out attempt keeps no renew token. * cluster: keep renewToken across failed lock attempts A request may have renewed the lease server-side before its response was lost, so the last token can still be valid on the next attempt. Clearing it on any transport failure strands a live lease until its TTL expires. A definitive rejection still clears the token through the response's empty RenewToken, and falling back to the seed filer no longer abandons the lease either. * cluster: bound each lock attempt by the remaining route window The 15s unavailability window started only after the first failure, so three 10s attempts plus sleeps outlasted it: the third failure selected the seed fallback at the same moment the loop gave up, and a reachable seed route was never tried. Start the clock before the first attempt and cap each attempt at what the window has left. When the window is spent on a non-seed target, fall back to the seed filer — it can forward to the real owner — and give that route its own window, so the fallback is always exercised before acquisition fails. * cluster: bound DistributedUnlock by the lock RPC timeout The unlock used context.Background(), so Stop against an unreachable lock server could wait on the TCP connect timeout far past the renewal goroutine's bounded shutdown wait. A timed-out unlock is harmless: the lease expires via its TTL regardless. * dlm: replay the current token on a lost renewal's retry A renewal rotates the token server-side before the response is sent. When that response is lost the client only knows the superseded token: retries mismatched, the token was cleared, and the holder conflicted with its own lease until it expired. The lock now remembers the superseded token, so a retry from the same owner replays the current token, and a superseded token can still release its own lock. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * cluster: give the seed lock fallback one fresh window per acquisition The seed can redirect back to a dead primary via LockHostMovedTo, so refilling the retry window on every return to the seed made the primary->seed->primary cycle unbounded. The first seed fallback keeps its fresh window; later returns to the seed keep the running deadline. * cluster: advance the lock seq when a lost-renewal replay extends the lease A retry of a renewal whose response was lost updates ExpiredAtNs in place but returned the existing Seq, so the replicated extension was rejected by InsertBackupLock as a stale mutation. A promoted backup would then grant the lock when the old expiry passed even though the holder still believes the lease is renewed. Bump Seq on the replay so the backup accepts the new expiry. * cluster: carry the superseded lock token through replication and transfer A renewal rotates the lock token and keeps the superseded one in PrevToken so a holder whose renewal response was lost can still recover. That recovery broke as soon as the lock moved: neither the replicated backup copy nor a transferred lock carried PrevToken, so a new primary rejected retries of the old token until the lease expired. Add prev_renew_token to the Lock and ReplicateLockRequest messages and thread it through ReplicateLock, TransferLocks, InsertBackupLock and InsertLock. * cluster: count clean lock redirects against the retry window A bare LockHostMovedTo response (no error, no token) returns nil from attemptToLockWithin, and retryUntilLocked reset the retry window on every nil result. Two ring-disagreeing filers forwarding to each other therefore bounced the client in an unbounded loop; the spent-window exit could also return a nil error with no token, handing callers an unacquired lock. * cluster: do not mark a lock held or hot-loop on tokenless attempts A bare LockHostMovedTo redirect returns nil with no renewToken; the attempt previously flipped isLocked to 1 anyway and retried immediately. Gate the flag on a granted token and pace the retry like other outcomes. * cluster: keep renewToken across bare lock redirects A LockHostMovedTo response carries no token and no error; overwriting renewToken with it dropped a held token, so the next renewal re-acquired fresh and could contend against its own still-live lease until TTL. * cluster: poll for replicated backup state in lock tests Replication runs in a goroutine, so a fixed sleep followed by an immediate backup read is racy. Wait for the expected backup state with require.Eventually instead. * cluster: mark a lock held only on a grant verdict A bare LockHostMovedTo redirect with a saved token left isLocked set without the redirected filer granting anything, and retryUntilLocked could exit on a stale token with no attempt made. doLock now reports whether the response was a grant, and both the retry loop and isLocked follow that verdict. --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
ee315527cb |
vector: address remaining s3vectors review findings (#12115)
* vector: address remaining s3vectors review findings Catalog and permission model: - Vector buckets are declared Lance table buckets end to end: create sets format=lance, get/list/delete reject or skip ICEBERG buckets, and the format lookup runs on an internal trusted catalog so callers need no GetTableBucket grant. - The implicit "vector" index namespace is created through the same internal manager, so a CreateTable grant alone authorizes CreateIndex; typed S3TablesError codes replace string matching on err.Error(). - CreateIndex pre-checks existence after the authorize, tags the record with a create nonce, and rolls back on engine failure under context.WithoutCancel with a versionToken CAS, so a retry never deletes a table it did not create. - DeleteIndex drops the catalog record directly; DropIndex is unimplemented on the shared VectorService. - DeleteVectorBucket removes the empty internal namespace on BucketNotEmpty and retries once. - PutVectors/DeleteVectors require the update grant, and engine gRPC errors map to typed API errors instead of generic 500s. API correctness: - CreateVectorBucket/CreateIndex return their resource ARNs. - ARNs reject empty bucket segments; vectorBucketArn must be an ARN. - ListIndexes returns summaries built only from catalog fields. - ListVectors honors maxResults/nextToken inside a segment via "seg:<pageStart>:<nextIndex>" resume tokens and validates segmentCount/segmentIndex. - QueryVectors requires positive topK; PutVectors rejects empty keys. - CreateVectorBucket rejects a supplied encryptionConfiguration rather than silently dropping it. - Metadata filters emit Lance's real JSONB UDFs: json_get_string/ json_get_float/json_get_bool picked by operand type, json_exists with a bracketed JSONPath, and per-member accessors for $in/$nin. Engine: - weed-vector-server stores the metadata column as Lance JSONB (lance.json extension) so the json_* UDFs apply; reads accept legacy Utf8 JSON columns. Vectors port TLS: - The TLS config is built before binding listeners, loads -cacert.file into ClientCAs when client-cert verification is on, fails startup on an empty/invalid CA file, and closes the gRPC conn on every failure path. * s3vectors: authorize index create before writing the implicit namespace A denied caller could still leave a persistent namespace behind because ensureIndexNamespace wrote through the trusted catalog before the caller's create grant ran. AuthorizeCreateTable now tolerates the not-yet-declared implicit namespace (bucket policy alone decides) and runs before any write; real table creates keep requiring the namespace. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: keep engine error details out of the API response Unavailable now maps to 503 ServiceUnavailableException and unclassified codes return a generic message; the real gRPC error stays in the log. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: exact-type metadata filters + NULL-safe $nin The jsonb extractors convert numeric-looking strings through json_get_float and 'true'/'false' through json_get_bool, so a stored '3' could satisfy a filter for the number 3. Numeric and boolean comparisons now also require the stored value not be a string, and each $in/$nin group coalesces NULL so NOT cannot invert NULL into NULL. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: validate the vectors TLS key pair before binding The reloading certificate provider reads the files asynchronously, so a missing or mismatched pair would only surface as failed handshakes on an announced HTTPS port. Load it once up front. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: make a token-checked table delete atomic The version token was read in one filer call and enforced in another, so a delete+recreate slipping between them let the delete erase the replacement. When a token is supplied the name path now carries the observed mtime as an if_not_modified_after precondition (evaluated under the path lock), and a decoupled delete strips the catalog marker under a compare of the captured attributes before touching the data path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: compare filter operands against the stored JSON type jsonb's strict converters make json_get_string return text for numbers and booleans too, so a get_string IS NULL guard is never true for scalar metadata — numeric and boolean comparisons silently matched nothing. Numbers and booleans now compare through TRY_CAST of the serialized json_extract text, which yields NULL on any other stored type; string comparisons stay on json_get_string behind a CASE guard so the extractor only runs on rows storing a JSON string. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: allocate implicit namespace metadata before use A first table create in an undeclared namespace took the allowMissingNamespace branch with a nil record and panicked on the owner assignment. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: read the delete target's metadata and snapshot in one lookup Two separate lookups let a delete+recreate in between validate the caller token against the old record while the conditional delete aimed at the replacement. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: assert the captured record before a conditional table delete An xattr update keeps Mtime, so the IfNotModifiedAfterNs guard alone could not see a metadata write that had already rotated the version token, and a recreate preserving the mtime could slip the mtime check. The delete now first proves the entry still carries the captured attributes under the filer's path lock, then keeps the mtime precondition. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: match only JSON boolean literals in metadata filters * filer,s3tables: delete versioned tables under one attribute precondition * s3vectors: compare serialized JSON for string filter operands json_get_string converts numeric-looking strings and booleans, so it cannot serve exact-typed string comparisons even behind a guard. String operands now compare the serialized json_extract text directly, which is exact-typed and never errors; a Rust test pins the semantics against a mixed-type dataset. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: compare boolean filter operands as serialized text Arrow's boolean cast accepts '1' and '0', so a TRY_CAST column let a stored number satisfy a boolean filter. Booleans now compare the serialized json_extract text like strings do — a JSON boolean only ever serializes as 'true' or 'false'. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: claim a catalog name's markers in one compare-checked write A create wrote its markers attribute-by-attribute after checking the name was free. A delete's restore landing in the check-to-write gap put the old markers back first, and the create then overwrote only the ones it sets — leaving the old tags, policy, and maintenance settings on the new table. CreateTable, RegisterTable, and CreateView now write all their markers in a single update that refuses to run while any catalog marker is present, so the second writer loses with a conflict instead of inheriting them. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: test the attribute-preconditioned table delete and name claim Adds a BeforeDelete hook plus ExpectedExtended evaluation to the in-memory filer so tests can land a competing write in a handler's check-to-delete window, and covers two losing races: a token delete whose record changed since the read, and a create colliding with a delete's marker restore. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: reserve a catalog record under one compare-checked claim claimCatalogRecord writes a deleting marker onto the record while it still holds exactly the caller's captured attributes, so a delete or rename can hold the record through a multi-step operation and a losing writer backs out before any data or destination entry is written. The claim key joins the markers a name-claim asserts absent, so the name cannot be reused while the reservation stands. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: claim the record before purging or publishing past it A token delete of a decoupled table purged the data first and checked the marker afterwards, so a rename that finished in between left the moved record pointing at purged data while the delete reported a conflict. The delete now claims the record under the captured attributes before any data is removed and releases the claim if the purge fails, keeping the record intact for a retry. The rename claims the source before publishing the destination, so a losing rename backs out while no destination entry exists instead of leaving one behind pointing at data a delete went on to remove. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: test the record claim across delete and rename races A rename completing in the token delete's check-to-claim window turns the delete into a conflict with the data intact, and a delete's claim on the source turns the rename into a conflict before the destination entry is published. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
8379ee9251 |
vector: snapshot lease + maintenance safety tests (P2) (#12107)
* vector: version-lease helpers + maintenance safety tests Pinned queries must survive Lance maintenance. maintenance::pin tags the current dataset version so cleanup refuses to reclaim it; release drops the tag. Tests prove the P2 gate: a pinned query returns identical hits across compact_files, a tagged version survives cleanup_old_versions (error_if_tagged protects it, the non-strict run skips it), and a missing data file fails the query instead of returning partial hits. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: lease the query's dataset version in the VectorService query_vectors pins the resolved version with a query tag for the duration of the RPC and releases it after, so concurrent maintenance cleanup cannot reclaim the files out from under an in-flight read. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: release the version lease on every query exit path The lease tag was only released on query_vectors' success path — get_vectors never released and query_vectors leaked its tag on early returns. pin now returns a guard whose drop deletes the tag, so every path frees the version. * vector: unique lease tag names + reap abandoned leases Nanos-since-epoch tag names can collide across concurrent readers and processes, so each lease now carries a unique suffix. pin() also drops query tags older than an hour, since a killed or cancelled request leaves its lease behind and would otherwise block cleanup forever. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: keep live lease tags + throttle abandoned-lease scans An expired tag is only reaped once no live lease in this process owns it, so a query running past LEASE_TTL keeps its pin. Tag scans run at most once per 60s per dataset instead of on every request. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: answer read-only queries without a version lease A namespace may hand the server credentials that can read a table but not update it, so the tag write in pin() fails and the whole query errors out. try_pin() degrades a denied tag write to an unleased query instead of failing the request. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: fail a lease whose pinned version is already gone Cleanup lists tags once at the start, so a tag created mid-cleanup does not protect the version it names. After the tag lands, pin now verifies the version still exists and drops the lease when it does not, failing the read at pin time instead of mid-query. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
bfb6e40fb4 |
test: assert the resumed segment token in ListVectors paging (#12108)
test: assert the resumed segment token The second listSegmentVectors result was never checked (staticcheck SA4006); assert it resumes scanning at row 19 after matches 10..18. |
||
|
Chris LuandDevin
|
67240f538d |
filer.remote.gateway: stop zero-byte uploads of remote-only entries (#12110)
* filer.remote.gateway: skip uploading a dropped local copy on update The bucketed update path uploaded the chunkless entry whenever the update was not metadata-only. remote.uncache, eviction, and remote.meta.sync drop a synced entry's chunks and clear its sync stamp, which reads as dirty when the remote mtime shares the entry's second; the upload then replaced the object that is the entry's only content with zero bytes. Skip the upload the same way the mounted-directory sync does. * filer.remote.gateway: copy a remote-only entry's object on rename A rename deleted the source object and uploaded the chunkless entry, replacing the entry's only content with zero bytes — the same loss remote.uncache caused on plain updates. Complete the rename on the remote the way the mounted-directory sync does: copy the source object over (or take the destination when it already describes the entry), restamp so reads stop resolving through the pending-source marker, and remove the source only while it is still the object the stamp describes. A remote-only entry renamed out of the gateway's coverage keeps its object, and the plain rename's old-key delete only removes the object the event's stamp still describes. * filer.remote.gateway: scope the sync-stamp skip to same-path updates shouldSendToRemote reads the entry's sync stamp, which still covers a rename's unchanged mtime, so a synced or pending-stamped rename event was skipped before its copy or upload could run. Only updates in place may skip on the stamp. * filer.remote.gateway: keep the source a pending rename moved on from A rename event whose pending destination has since been renamed again still names the source a later event reads through its own marker; deleting the old key here loses the only copy. Mirror the remote.sync check: leave the source when the event's entry carries the pending-source marker and no entry holds the destination any more. * filer.remote.gateway: release a rename's pending source once complete A rename whose old entry read through a pending-source marker leaves that earlier source behind: the destination is released by its own key, never the marker's. Mirror releaseRenamedSource — once the renamed entry no longer reads through a marker, delete the pending source guarded by the old entry's stamp, keeping it while a remote-only entry holds its path. * filer.remote.gateway: finalize a complete rename before failing its source A rename source outside the gateway's coverage errored before the destination was checked, and the recorded failure held the event offset. Stat the destination first: when it already holds the content the entry describes, refresh the stamp and leave the unmanaged source untouched; fail only when nothing holds the content. * filer.remote.sync: document the source-of-truth protocol both processors share --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
0f1e73cc3a | docs: regenerate star history chart | ||
|
Chris LuandDevin
|
e2d899ac25 |
test/vectors: fix the S3 Vectors e2e harness (#12112)
* test/vectors: give the vector server static storage credentials The e2e harness ran weed-vector-server without credentials, so lance's AWS object store fell back to IMDS and CreateIndex died on the runner's 411 from the token endpoint. Point it at the test credentials the s3 gateway now reads from s3.json. * vector: take the list item field from the schema when batching vectors Lance stores the fixed-size list item as nullable even when the write schema declared it non-null, so PutVectors built batches that no longer matched: expected FixedSizeList(4 x Float32) but found FixedSizeList(4 x non-null Float32). * vector: store index metadata as lance jsonb so filters resolve The metadata field was plain Utf8, but the filter functions lance's planner registers (json_get_string, json_get_float, json_exists) take a LargeBinary jsonb column, so QueryVectors with a metadata filter failed to plan. Declare the field with the arrow.json extension — lance encodes it to jsonb on write and back on read — and encode the batch's strings when the stored column is LargeBinary. Also rename the filter's json_get_str call to json_get_string, the name lance registers. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |