mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-30 17:00:48 +01:00
2a1afadcd079148d8f14ea5f37145e7a3e290cbc
22
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandDevin
|
d0441bbdbb |
s3vectors: validate index ARNs and expand live integration coverage (#12222)
* test: align vector bucket policy coverage with vector actions * s3vectors: validate ARN service and resource shape before resolving indexes * test: exercise signed vector bucket policies and catalog lifecycle * ci: require the expanded vector integration matrix and all gateway tests * test: reject index ARNs in bucket-level API requests * s3vectors: scope ARN resolution to the gateway namespace * s3vectors: reject name fields that conflict with resolved ARNs bucketARNs and resolveIndexARNs already reject an indexArn that names a different bucket than vectorBucketArn, but a vectorBucketName or indexName disagreeing with the ARNs was silently dropped. Validate the plain names against the resolved ARN values the same way. * test: cover conflicting vector names and ARNs in live requests --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
cc5912b8b8 |
s3vectors: e2e conformance for vector bucket policy evaluation (#12221)
* s3vectors: reject empty boolean-operator lists and sub-objects in filters An empty $and/$or list or a nested empty object rendered '()'' — invalid SQL surfaced as an engine error instead of a clean ValidationException. * s3vectors: e2e conformance for vector bucket policy evaluation The handler-level test covers statements; this runs the whole stack a client sees — HTTP dispatch, authentication, policy evaluation, catalog, engine — and walks the full lifecycle: a non-owner is denied without a grant, an Allow on the bucket ARN grants every index under it, a Deny vetoes an Allow on the same action, and deleting the policy restores owner-only access. * s3vectors: prove deny precedence and grant removal in policy e2e * s3vectors: reject field operator objects with no operators * test: use an s3vectors action in the vector policy round-trip |
||
|
Chris Lu
|
9d5929a73f |
s3vectors: reject empty boolean-operator lists and sub-objects in filters (#12220)
* s3vectors: reject empty boolean-operator lists and sub-objects in filters An empty $and/$or list or a nested empty object rendered '()'' — invalid SQL surfaced as an engine error instead of a clean ValidationException. * s3vectors: reject field operator objects with no operators |
||
|
Chris Lu
|
ac7b4f4cf6 |
s3vectors: bound ListVectorBuckets and ListIndexes maxResults (#12218)
* s3vectors: bound ListVectorBuckets and ListIndexes maxResults Both pass maxResults to the catalog unbounded; AWS caps these list pages at 100, the same way ListVectors is bounded and topK is bounded. * s3vectors: raise ListVectorBuckets/ListIndexes maxResults cap to 500 AWS documents a valid range of 1-500 for both list APIs; the shared limit of 100 rejected legitimate page sizes. * s3vectors: reject explicit maxResults of zero on list calls AWS requires maxResults >= 1, but a plain int field cannot tell an omitted field from an explicit zero. Making MaxResults a pointer keeps the catalog default for omitted fields while a supplied zero fails validation. * s3vectors: test max ListIndexes page against an existing bucket |
||
|
Chris LuandDevin
|
98fe70a247 |
s3vectors: keep bucket policies in vector form and evaluate them in the gateway (#12214)
* s3vectors: translate vector bucket policies to the catalog form The document is stored verbatim on the shared table bucket and evaluated against s3tables actions and ARNs, so a policy written for the vector API was a dead rule. Actions now map to the catalog operation the request authorizes against, and an arn:aws:s3vectors: resource becomes the bucket/.../table/vector/... ARN the index's catalog entry answers to. * s3tables: expose PrincipalFor and PolicyContext for façade evaluation A façade authorizing against its own action namespace needs the same identity resolution and evaluation context the catalog uses internally. * s3vectors: keep bucket policies in vector form and evaluate them in the gateway The earlier approach translated the document to catalog actions and ARNs, which lost the caller's original policy for GetVectorBucketPolicy and could not cover data-plane calls that never touch the catalog. The document now stays in s3vectors form under a dedicated bucket attribute and every handler evaluates it through authorizeVector against the action and ARN the request names; a dedicated key also keeps a vector grant from ever authorizing a catalog call. * s3vectors: close policy-mutation races and keep vector grants visible - Authorize Put/DeleteVectorBucketPolicy inside the policy slot's compare-and-swap, so a grant revoked between reads cannot authorize the pending write. - Assert the policy attributes stay absent in the update precondition, so a concurrent policy or tag write trips the retry instead of being silently reverted. - Migrate a raw vector document out of the shared policy slot on replace or delete, keeping its catalog rules. - Pass the unresolved identity to PolicyContext so open-by-default deployments still work. - Filter ListVectorBuckets on the vector policy itself, so buckets a caller was granted under s3vectors actions stay listed. * s3vectors: reuse SnapshotExtended for the vector-policy CAS precondition * s3vectors: skip buckets that vanish mid-list instead of failing the page * s3vectors: reject non-vector bucket formats inside authorizeVector * s3vectors: check bucket format after authorization in authorizeVector An unauthorized caller probing an ICEBERG bucket got NoSuchBucket,\nleaking that the bucket exists. The format check now runs after the\npermission check so denied callers always see AccessDenied. * s3vectors: cover legacy shared-slot policy removal on delete * s3tables: resolve the forwarded identity name like getAccountID in PrincipalFor A synthesized manager request resolves its caller through the identity- name branch of getAccountID, which normalizes the name (ARNs collapse to their trailing segment, whitespace trims) before falling back to the handler's account ID. PrincipalFor returned the raw name, so ARN-shaped identities never matched their own grants. * s3tables: assert the vector policy key in update preconditions s3vectors-policy is stored on table bucket entries but was missing from s3tablesExtendedKeys, so a catalog update that read the entry before the first vector policy was saved could overwrite it without tripping the compare-and-swap. Asserting the key makes such a race fail instead of silently erasing the policy. * s3vectors: reject rules the evaluator cannot honor in vector bucket policies The shared Statement shape has no negated fields, the vector evaluation supplies no condition keys, and a document that fails the typed decode is discarded wholesale at evaluation — including denies the writer believed active. Reject NotPrincipal, Condition, foreign-service actions and resource ARNs, and require the typed PolicyDocument decode. Also cover that an admin-named identity still reaches its own buckets. * s3vectors: read vector policy fields the way the evaluator decodes them Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: match ARN policy principals through caller normalization Caller principals are normalized ids, so a statement naming a full IAM ARN could never match: an ARN-scoped deny was silently dead and an ARN-scoped allow unreachable. Compare arn: specs against the caller under the same normalizePrincipalID the ownership path uses, while an ARN caller keeps verbatim wildcard matching. * s3tables: grant the admin bypass on the caller's actions, not the name A principal that only normalizes to the admin id skipped every policy deny. When the request carries an identity, the bypass now requires the identity's own admin action — the same gate getAccountID applies — while name-only transports with no action list keep it. PolicyContext takes the request so it can carry IdentityActions, and the vector context helpers route through the shared identity keys so the resolved name and actions agree. * s3tables: match ARN policy principals only for deny statements Normalization drops the ARN's account and path, so an allow matched on the trailing name could grant a same-named caller the statement never verified. Denies keep the normalized match so the rule cannot be ducked through the same collapse. * s3tables: scope service wildcards to their own service s3tables:* counted as blanket admin inside the shared permission check, so an identity holding only catalog grants administered s3vectors operations too. hasAdminActionFor now requires the wildcard to name the operation's service; the global admin markers and s3:* stay service-agnostic. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
4ef123d764 |
s3vectors: fix shared API behavior and test Go/Rust volume parity (#12216)
* s3vectors: reuse shared authenticated identity context * s3vectors: encode empty results as arrays * test: cover S3 Vectors with Go and Rust volume servers * ci: require both volume implementations in vector integration tests * s3vectors: keep identityOf as a shim over the shared identity context Concurrent branches add identityOf call sites; deleting the helper would break the build when they merge in either order. Route it through GetIdentityNameFromContext so every caller resolves the same identity. * test: wait for the vector engine to register before running The gateway resolves weed-vector lazily through the master cluster list, so a listening port does not mean the engine is reachable yet; the first vector write or query could race KeepConnected and fail. Restore the cluster.ps registration wait the previous harness had, and ignore the pytest cache directory the suite creates. * test: wait for master registration across OSS and enterprise Use the exact engine registration event after AddClusterNode completes. The current enterprise cluster.ps does not list vector nodes, so waiting on its output prevents the shared fixture from starting there. * s3vectors: keep contextWithIdentity as a shim alongside identityOf In-flight callers still write the caller name through it; route the write through SetIdentityNameInContext so both helpers agree on the shared key whichever lands first. |
||
|
Chris Lu
|
15066c8a9f |
vector: implement drop_index and have DeleteIndex drop the data (#12204)
* vector: implement drop_index in the engine The RPC was a stub; it now resolves the table's vended location and removes every object under it through the same credential path open uses. A table the catalog can no longer describe is already dropped, so the call stays idempotent. * s3vectors: drop index data after the catalog delete DeleteTable carries the delete authorization, so it runs first; the engine DropIndex then removes the data under the index's location. NotFound and Unimplemented are tolerated so an index whose data went away with the catalog entry, or an older engine, still answers 200. * s3vectors: let the catalog delete carry the index data removal A vector index is colocated: the catalog entry sits at the dataset root, so DeleteTable's recursive delete already removes entry and data. The engine DropIndex call afterwards could only ever describe a missing or recreated record — a no-op in the common case, and a delete of a replacement index's files when the name was reused in between. |
||
|
Chris Lu
|
c7d77eb9a8 |
s3vectors: accept and return encryptionConfiguration on vector buckets (#12203)
* s3tables: carry encryptionConfiguration in table bucket metadata * s3vectors: accept and return encryptionConfiguration on vector buckets The configuration is validated (sseType AES256 or aws:kms) and stored on the bucket record, so the API round-trips what the caller declares. * s3vectors: reject caller-chosen encryption keys on vector buckets * s3tables: reject caller-chosen encryption keys on table buckets * s3tables: reject unknown sseType in bucket encryptionConfiguration The bucket check rejected aws:kms and kmsKeyArn but stored any other sseType — a made-up or mistyped value would round-trip through GetTableBucket as an encryption claim the storage does not keep. The validator now admits only the AES256 declaration, shared so the vector layer can drop its copy. * s3vectors: delegate encryption validation to the shared catalog check * s3tables: accept the sseAlgorithm name in bucket encryptionConfiguration S3 Tables declares the algorithm as sseAlgorithm while S3 Vectors uses sseType; the shared check now accepts either spelling, still admits only AES256, and still refuses any caller-chosen key. * s3vectors: call the shared encryption validator directly * s3vectors: require each API's own encryption field name The shared validator accepted sseAlgorithm on the vector path, so a CreateVectorBucket carrying the s3tables field stored a configuration GetVectorBucket could never echo back in the vectors shape (and vice versa). ValidateEncryptionConfiguration now takes the wire name its caller expects; the vectors create path normalizes to sseAlgorithm for the catalog and GetVectorBucket renames it back to sseType. |
||
|
Chris Lu
|
fc9be77dbb |
s3vectors: bound QueryVectors topK at the AWS limit (#12202)
* s3vectors: bound QueryVectors topK at the AWS limit of 30 topK passed through unbounded; AWS caps a query at 30 results, so a larger request is a ValidationException before it reaches the engine. * s3vectors: raise QueryVectors topK bound to the AWS limit of 10,000 * s3vectors: validate QueryVectors topK before the catalog lookup * s3vectors: cover the QueryVectors topK bounds |
||
|
|
eecc461b39 |
s3vectors: put, get and delete vector bucket policies (#12201)
* util: do not bind outbound dials to a loopback -ip.bind `weed server -ip.bind=127.0.0.1` keeps every listener local, but the same address became the source of outbound connections, so the filer dialled its remote mount from 127.0.0.1 and failed with "connect: invalid argument". A loopback source cannot reach off the host; leave the source to the OS, as for a wildcard bind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: put, get and delete vector bucket policies The AWS S3 Vectors policy operations had no handlers; the policy document is stored on the shared table bucket, so the handlers delegate to the catalog's PutTableBucketPolicy, GetTableBucketPolicy and DeleteTableBucketPolicy and keep its guard. * s3vectors: treat ICEBERG buckets as absent in policy handlers GetVectorBucketPolicy/PutVectorBucketPolicy/DeleteVectorBucketPolicy delegated straight to the table bucket, so a bucket created for ICEBERG tables — invisible to every other vectors operation — could still have its policy read and written through the vectors API. Gate all three on lanceBucket, the same format check DeleteVectorBucket and CreateIndex already apply. * test/vectors: cover vector bucket policy put/get/delete Round-trips a policy document through the new ops and asserts a get on the deleted policy surfaces NotFoundException, so the boto3 conformance run exercises the added handlers end to end. * s3vectors: validate the bucket ARN and policy document Bucket-level operations now reject an index ARN and an ARN minted for another service, and PutVectorBucketPolicy refuses a document carrying s3vectors actions — the evaluator only honors s3tables actions, so such a rule would be a dead grant. * s3vectors: validate the bucket ARN on the remaining bucket-level operations * s3vectors: cover bucket ARN and policy document validation TestBucketARNs pins the accepted forms and the index/foreign-service rejections; TestValidatePolicyDocument covers a valid document, a malformed one, and the dead s3vectors action namespace. * s3vectors test: use an enforceable action in the bucket policy test The catalog refuses s3vectors actions in a stored policy, so the put call rejected the document before the get and delete assertions ran. --------- Co-authored-by: Ólafur Bjarki Bogason <olafur@genkiinstruments.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandDevin
|
8e44178dc6 |
s3vectors: cap PutVectors at 500 items and check data against the index dimension (#12200)
* s3vectors: cap PutVectors at 500 items and check data against the index dimension A batch could carry any number of vectors and a vector of the wrong dimensionality reached the engine: AWS caps a PutVectors call at 500 items and requires each vector to match the index's declared dimension. * s3vectors: cover the PutVectors batch and dimension checks * s3vectors: reject query vectors that do not match the index dimension Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: cover the PutVectors batch size bounds Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: cover the catalog-backed PutVectors handler path Run PutVectors through the real handlers over the in-memory catalog: a matching vector reaches the engine, a mismatching one is rejected with ValidationException before PutVectors is called, and a full vectorsMaxPutBatch batch is accepted. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
511727cf3a |
vector: DeleteIndex format guard, sticky discovery, advertise normalize (#12171)
* vector: DeleteIndex format guard, sticky discovery, advertise normalize - handleDeleteIndex verifies the table carries the Lance format before DeleteTable — an Iceberg table under the same namespace is not an index and must not be removed by it. - discovery keeps the current worker while it is still registered — membership-order churn can no longer cancel in-flight calls, and the resolver closes each master conn inside a 5s deadline. - advertised_addr strips scheme and .grpcPort so the UDP probe reaches every supported master form. * vector: empty-follower retry, format check without read grant - discoverVectorWorkers keeps looking past an empty master reply — followers list nothing when workers registered with the leader. - the DeleteIndex format check uses the internal catalog read, so DeleteTable-only callers don't gain a GetTable requirement, and the format must be Lance exactly — format-less tables no longer pass. |
||
|
Chris Lu
|
d5433ae745 |
vector: cover the identity object in s3vectors auth (#12162)
vector: cover the identity object in the s3vectors auth context The auth middleware must carry the resolved identity object — only the name survives otherwise and authorization loses actions, account, and claims. |
||
|
Chris Lu
|
afb8472c5e |
s3vectors: cover non-filterable key round-trip and nested filter rejection (#12160)
* s3vectors: cover non-filterable key round-trip and nested filter rejection The catalog writes the key list as a JSON string while older records may still carry the array form; indexFromRecord must decode both. The rejection path extracts field names through nested $and/$or filters, so a blocked key deep in the tree still fails the query. * s3vectors: factor the non-filterable rejection into a testable gate Pull the blocked-key check out of handleQueryVectors into nonFilterableField so the rejection itself is covered without standing up the catalog stack; the handler now 400s on whatever the gate names. * s3vectors: report the non-filterable match separately from the key An empty metadata key in nonFilterableMetadataKeys collided with the "no match" sentinel, so a filter on that key skipped rejection. Return a match flag and cover the empty-key case in the test. |
||
|
Chris Lu
|
237db29dca |
vector: round-trip and enforce non-filterable metadata keys (#12157)
vector: non-filterable metadata round-trips and filters reject it - GetIndex omitted nonFilterableMetadataKeys — the catalog stores the list as a JSON string but the reader only accepted an array; both forms now decode. - QueryVectors now rejects filters referencing non-filterable keys instead of silently scanning them. |
||
|
Chris Lu
|
10ba1b782f |
vector: carry the full identity through s3vectors auth (#12152)
vector: keep the full identity in the request context Auth discarded the resolved identity object — the s3tables manager request then re-resolved by name and denied authenticated users their actions, policies, account, and claims. Store the object via SetIdentityInContext so downstream authorization sees it. |
||
|
Chris LuandDevin
|
ee315527cb |
vector: address remaining s3vectors review findings (#12115)
* vector: address remaining s3vectors review findings Catalog and permission model: - Vector buckets are declared Lance table buckets end to end: create sets format=lance, get/list/delete reject or skip ICEBERG buckets, and the format lookup runs on an internal trusted catalog so callers need no GetTableBucket grant. - The implicit "vector" index namespace is created through the same internal manager, so a CreateTable grant alone authorizes CreateIndex; typed S3TablesError codes replace string matching on err.Error(). - CreateIndex pre-checks existence after the authorize, tags the record with a create nonce, and rolls back on engine failure under context.WithoutCancel with a versionToken CAS, so a retry never deletes a table it did not create. - DeleteIndex drops the catalog record directly; DropIndex is unimplemented on the shared VectorService. - DeleteVectorBucket removes the empty internal namespace on BucketNotEmpty and retries once. - PutVectors/DeleteVectors require the update grant, and engine gRPC errors map to typed API errors instead of generic 500s. API correctness: - CreateVectorBucket/CreateIndex return their resource ARNs. - ARNs reject empty bucket segments; vectorBucketArn must be an ARN. - ListIndexes returns summaries built only from catalog fields. - ListVectors honors maxResults/nextToken inside a segment via "seg:<pageStart>:<nextIndex>" resume tokens and validates segmentCount/segmentIndex. - QueryVectors requires positive topK; PutVectors rejects empty keys. - CreateVectorBucket rejects a supplied encryptionConfiguration rather than silently dropping it. - Metadata filters emit Lance's real JSONB UDFs: json_get_string/ json_get_float/json_get_bool picked by operand type, json_exists with a bracketed JSONPath, and per-member accessors for $in/$nin. Engine: - weed-vector-server stores the metadata column as Lance JSONB (lance.json extension) so the json_* UDFs apply; reads accept legacy Utf8 JSON columns. Vectors port TLS: - The TLS config is built before binding listeners, loads -cacert.file into ClientCAs when client-cert verification is on, fails startup on an empty/invalid CA file, and closes the gRPC conn on every failure path. * s3vectors: authorize index create before writing the implicit namespace A denied caller could still leave a persistent namespace behind because ensureIndexNamespace wrote through the trusted catalog before the caller's create grant ran. AuthorizeCreateTable now tolerates the not-yet-declared implicit namespace (bucket policy alone decides) and runs before any write; real table creates keep requiring the namespace. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: keep engine error details out of the API response Unavailable now maps to 503 ServiceUnavailableException and unclassified codes return a generic message; the real gRPC error stays in the log. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: exact-type metadata filters + NULL-safe $nin The jsonb extractors convert numeric-looking strings through json_get_float and 'true'/'false' through json_get_bool, so a stored '3' could satisfy a filter for the number 3. Numeric and boolean comparisons now also require the stored value not be a string, and each $in/$nin group coalesces NULL so NOT cannot invert NULL into NULL. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: validate the vectors TLS key pair before binding The reloading certificate provider reads the files asynchronously, so a missing or mismatched pair would only surface as failed handshakes on an announced HTTPS port. Load it once up front. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: make a token-checked table delete atomic The version token was read in one filer call and enforced in another, so a delete+recreate slipping between them let the delete erase the replacement. When a token is supplied the name path now carries the observed mtime as an if_not_modified_after precondition (evaluated under the path lock), and a decoupled delete strips the catalog marker under a compare of the captured attributes before touching the data path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: compare filter operands against the stored JSON type jsonb's strict converters make json_get_string return text for numbers and booleans too, so a get_string IS NULL guard is never true for scalar metadata — numeric and boolean comparisons silently matched nothing. Numbers and booleans now compare through TRY_CAST of the serialized json_extract text, which yields NULL on any other stored type; string comparisons stay on json_get_string behind a CASE guard so the extractor only runs on rows storing a JSON string. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: allocate implicit namespace metadata before use A first table create in an undeclared namespace took the allowMissingNamespace branch with a nil record and panicked on the owner assignment. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: read the delete target's metadata and snapshot in one lookup Two separate lookups let a delete+recreate in between validate the caller token against the old record while the conditional delete aimed at the replacement. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: assert the captured record before a conditional table delete An xattr update keeps Mtime, so the IfNotModifiedAfterNs guard alone could not see a metadata write that had already rotated the version token, and a recreate preserving the mtime could slip the mtime check. The delete now first proves the entry still carries the captured attributes under the filer's path lock, then keeps the mtime precondition. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: match only JSON boolean literals in metadata filters * filer,s3tables: delete versioned tables under one attribute precondition * s3vectors: compare serialized JSON for string filter operands json_get_string converts numeric-looking strings and booleans, so it cannot serve exact-typed string comparisons even behind a guard. String operands now compare the serialized json_extract text directly, which is exact-typed and never errors; a Rust test pins the semantics against a mixed-type dataset. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: compare boolean filter operands as serialized text Arrow's boolean cast accepts '1' and '0', so a TRY_CAST column let a stored number satisfy a boolean filter. Booleans now compare the serialized json_extract text like strings do — a JSON boolean only ever serializes as 'true' or 'false'. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: claim a catalog name's markers in one compare-checked write A create wrote its markers attribute-by-attribute after checking the name was free. A delete's restore landing in the check-to-write gap put the old markers back first, and the create then overwrote only the ones it sets — leaving the old tags, policy, and maintenance settings on the new table. CreateTable, RegisterTable, and CreateView now write all their markers in a single update that refuses to run while any catalog marker is present, so the second writer loses with a conflict instead of inheriting them. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: test the attribute-preconditioned table delete and name claim Adds a BeforeDelete hook plus ExpectedExtended evaluation to the in-memory filer so tests can land a competing write in a handler's check-to-delete window, and covers two losing races: a token delete whose record changed since the read, and a create colliding with a delete's marker restore. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: reserve a catalog record under one compare-checked claim claimCatalogRecord writes a deleting marker onto the record while it still holds exactly the caller's captured attributes, so a delete or rename can hold the record through a multi-step operation and a losing writer backs out before any data or destination entry is written. The claim key joins the markers a name-claim asserts absent, so the name cannot be reused while the reservation stands. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: claim the record before purging or publishing past it A token delete of a decoupled table purged the data first and checked the marker afterwards, so a rename that finished in between left the moved record pointing at purged data while the delete reported a conflict. The delete now claims the record under the captured attributes before any data is removed and releases the claim if the purge fails, keeping the record intact for a retry. The rename claims the source before publishing the destination, so a losing rename backs out while no destination entry exists instead of leaving one behind pointing at data a delete went on to remove. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: test the record claim across delete and rename races A rename completing in the token delete's check-to-claim window turns the delete into a conflict with the data intact, and a delete's claim on the source turns the rename into a conflict before the destination entry is published. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
bfb6e40fb4 |
test: assert the resumed segment token in ListVectors paging (#12108)
test: assert the resumed segment token The second listSegmentVectors result was never checked (staticcheck SA4006); assert it resumes scanning at row 19 after matches 10..18. |
||
|
Chris LuandDevin
|
e2d899ac25 |
test/vectors: fix the S3 Vectors e2e harness (#12112)
* test/vectors: give the vector server static storage credentials The e2e harness ran weed-vector-server without credentials, so lance's AWS object store fell back to IMDS and CreateIndex died on the runner's 411 from the token endpoint. Point it at the test credentials the s3 gateway now reads from s3.json. * vector: take the list item field from the schema when batching vectors Lance stores the fixed-size list item as nullable even when the write schema declared it non-null, so PutVectors built batches that no longer matched: expected FixedSizeList(4 x Float32) but found FixedSizeList(4 x non-null Float32). * vector: store index metadata as lance jsonb so filters resolve The metadata field was plain Utf8, but the filter functions lance's planner registers (json_get_string, json_get_float, json_exists) take a LargeBinary jsonb column, so QueryVectors with a metadata filter failed to plan. Declare the field with the arrow.json extension — lance encodes it to jsonb on write and back on read — and encode the batch's strings when the stored column is LargeBinary. Also rename the filter's json_get_str call to json_get_string, the name lance registers. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
a9d6bfc51f |
test: boto3 s3vectors conformance harness (P1) (#12105)
* vector: boto3 s3vectors conformance harness test/vectors spins a mini cluster, an s3 gateway with -port.vectors and -vectors.grpc, and a weed-vector-server, then exercises the AWS client end to end: bucket and index lifecycle, put/get/list/delete vectors, and a filtered query asserting distances and metadata. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: run the s3vectors conformance subset An e2e job in rust-vector-tests: builds weed and weed-vector-server, then runs test/vectors against a mini cluster plus the vectors gateway. The workflow now also watches vector.proto, its stubs, and test/vectors. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: fix e2e protoc and codespell findings The e2e job needed the same vendored-protoc step as the unit-test job — lance's build scripts locate protoc themselves. Codespell flags the AWS `$nin` operator literal and the `lits` variable: exempt `nin` in .codespellrc like the existing word exemptions, rename the variable, and fold the negation into the IN/NOT IN symbol directly. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: classify catalog errors by code, not message text * s3: drop the implicit vector namespace on vector-bucket delete * test: point the vector engine at the gateway lance port * test: bind the vectors gateway listeners to loopback * test: make the filter and distance assertions load-bearing * test: rebuild the harness binaries on every run * ci: fix the vectors end-to-end job * ci: run the vector tests on gateway and command changes --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
c55b7c5a29 |
s3: fix CreateIndex metadata panic and ListVectors segmented paging (#12106)
* s3: nil-check metadataConfiguration before reading its keys * s3: honor maxResults/nextToken in segmented ListVectors * s3: exclude non-Lance buckets from ListVectorBuckets * s3: test segmented ListVectors paging token math * ci: silence codespell on the "$nin" operator and rename lits |
||
|
Chris LuandDevin
|
dcc94f818d |
vector: S3 Vectors API gateway (P1) (#12103)
* vector: S3 Vectors API gateway over VectorService weed/s3api/s3vectors serves the AWS S3 Vectors JSON API: bucket CRUD maps to S3 Tables table buckets, index CRUD to Lance table records plus the VectorService CreateIndex/DropIndex RPCs, and the data plane (Put/Get/List/Delete/QueryVectors) calls the shared service. The AWS filter document translates to a Lance SQL predicate over the metadata JSON column; unsupported operators fail loudly rather than overmatch. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: -port.vectors and -vectors.grpc on weed s3 Mirror the Lance server startup: the S3 Vectors API binds its own port and dials the weed-vector-server gRPC address; both disabled by default. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: resolve index ARNs, authorize index ops, fix catalog fields - vector buckets declare FormatLance so a new bucket accepts index tables instead of defaulting to ICEBERG. - ListIndexes decodes the `tables` response field; `tableSummaries` hid every index. - resolveIndexARNs honors indexArn alone; handlers no longer treat indexName as the only carrier. - data-plane RPCs fetch the index record first, so callers authorize against the index resource and a missing index 404s before touching the engine; QueryVectors reports the index's configured metric. - IndexArn added to the index-level request types. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: serve the vectors port over TLS when configured When -key.file/-cert.file are set, the vectors listener mirrors the main API and serves TLS rather than a second plaintext endpoint. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: harden the s3vectors gateway handlers - $ne no longer shadows the operator: exclusion filters emit <> instead of equality. - a null element in a PutVectors list fails the request instead of panicking. - requests are capped at 32 MiB; distances return via *float32 so an exact match still reports 0 when returnDistance is set. - CreateIndex declares the `vector` namespace first and validates dimension (1..=4096) before the catalog write; a failed VectorService.CreateIndex rolls the table record back. - non-filterable metadata keys persist as JSON rather than a comma-joined string, and index records carry their createdAt. - ListVectors honors segmentIndex/segmentCount by slicing the full listing in the gateway. - catalog errors map to AccessDeniedException/ConflictException instead of a blanket ValidationException, and malformed ARNs 400 instead of slicing. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: /{Operation} routes and vectors-port TLS, conn cleanup Path-dispatched clients (POST /QueryVectors) reach the same handlers as X-Amz-Target calls. A failed listener bind now closes the vector gRPC connection instead of leaving it open with the endpoint disabled. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |