mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-03 18:47:41 +01:00
ec968aa0165ce0df0959fd4482b61d50e2eb006e
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
ec968aa016 |
s3vectors: support TagResource, UntagResource and ListTagsForResource (#12335)
* s3vectors: tag vector buckets and indexes
TagResource, UntagResource and ListTagsForResource answer on
/tags/{resourceArn} (POST, DELETE, GET) and by name. Each authorizes its
own s3vectors action on the bucket or index ARN, then reads or writes the
catalog record's tags. A resource holds at most 50 tags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: apply the tags passed to CreateVectorBucket and CreateIndex
The tags field was accepted and dropped. It is now stored on the new
bucket or index, and creating with tags needs s3vectors:TagResource in
addition to the create action.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: cover resource tagging end to end
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: keep tag operations to a bound table bucket
TagResource, UntagResource and ListTagsForResource run through Execute
with WithTableBucketCreatedAt now answer not found when the bucket name
holds another bucket, as the bound creates and deletes already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bind tagging to the authorized bucket and to vector indexes
The catalog tag calls carry the creation time of the bucket authorizeVector
approved, so a bucket created again under the name is not touched. An index
ARN must name a Lance table: another table in the vector namespace of a
format-less bucket is refused, as DeleteIndex does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: match a TagResource grant scoped to the bucket being created
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: merge tag writes into the tags the record holds at the write
TagResource and UntagResource merged into the tags they read before the
compare-and-swap and wrote that map on every attempt, so a tag write that
landed in between was undone. The merge now runs inside the mutation.
WithMaxTags lets an in-process caller cap the merged set there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: check the 50-tag limit inside the catalog's tag write
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: check a tag operation's bound bucket after its reads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: validate each tag key UntagResource removes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: validate untag keys one at a time
UntagResource sized a map by the number of submitted tag keys before
validating them, so a large body of repeated keys allocated far more than
the request itself, before authorization ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: authorize UntagResource before validating its keys
Validating a near-32 MiB body of tag keys costs about 13x its JSON
decode, so an unauthorized caller could make the gateway spend that
work on a request it then refuses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Chris LuandClaude Opus 5.5
|
2eb219df61 |
s3vectors: encrypt the objects of a vector bucket declared AES256 (#12324)
* s3api: encrypt the objects of a vector bucket declared AES256 CreateVectorBucket recorded encryptionConfiguration sseType AES256 and returned it, but nothing on the write path read it: the index's data files, manifests and transactions were stored without SSE. A LANCE table bucket's record now supplies the bucket's default encryption whenever the bucket has none of its own, so PutObject, CreateMultipartUpload and CopyObject apply SSE-S3 to every object the engine writes. The S3 bucket metadata itself is unchanged. ICEBERG table buckets keep recording the declaration only: their catalog reads the manifests clients upload straight from the filer, which would see ciphertext. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3tables: refuse an AES256 vector bucket on a gateway without an SSE-S3 key A vector bucket's AES256 is now applied to every index write, and SSE-S3 fails each write when the gateway has no KEK. Creating the bucket there succeeded and the first CreateIndex came back as InternalFailure. CreateTableBucket now refuses a LANCE bucket declaring AES256 when the gateway reports no SSE-S3 key, naming s3.sse.kek and s3.sse.key; CreateVectorBucket returns it as ValidationException. ICEBERG buckets, whose declaration is only recorded, and tooling that sets no checker are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3api: refuse DeleteBucketEncryption on a bucket's declared encryption GetBucketEncryption reports a vector bucket's declared AES256, but DeleteBucketEncryption answered NoSuchBucketEncryptionConfiguration and the declaration kept applying. The declaration is fixed when the bucket is created, so the delete now returns MethodNotAllowed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): keep an inherited WEED_S3_SSE_KEK out of the test cluster The gateway refuses to start SSE-S3 with both s3.sse.kek and s3.sse.key set, so a developer shell exporting WEED_S3_SSE_KEK made the AES256 bucket test fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d72f761d83 |
seaweed-vector: work on its defaults, and without identities or credentials (#12323)
* seaweed-vector: default --namespace to the gateway's Lance port The old default, http://localhost:8888/lance, is a path nothing serves: the Lance Namespace listens on its own port, 9101 by default on weed s3, weed server, weed filer and weed mini. With it, the filer answered the engine's catalog calls with a 500 and every CreateIndex failed with InternalFailure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: default --listen to a port no SeaweedFS server takes 127.0.0.1:8080 is the volume server's default port, so weed-vector next to weed server exited with "Address already in use (os error 98)". 8090 is not among the defaults of master, volume, filer, S3 and its Iceberg and Lance catalogs, IAM, WebDAV, SFTP, the message queue broker and agent, the admin server or weed mini, nor any of their gRPC ports (HTTP + 10000). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance: take any key while the S3 server runs without identities Without identities the S3 port accepts any signature and treats the caller as the admin. The Lance catalog still checked an x-api-key or Bearer token against identities that did not exist, so weed-vector started with --storage-option credentials on such a server got 401 on every catalog call. The catalog now skips both checks when S3 auth is off and leaves the caller to the S3 authenticator, which admits it as the admin, as it already did for a request with no key. With identities configured nothing changes: an unknown key is still 401. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance: tell clients to skip signing while S3 auth is off A client with no credentials of its own could not reach a table's data on a server without identities, though the S3 port takes unsigned requests there: lance fell back to the AWS default credential chain and failed with CredentialsNotLoaded. weed-vector without --storage-option failed every CreateIndex that way. The catalog now adds aws_skip_signature to the storage options it hands out while S3 auth is off, and leaves it out once identities exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: bind the listener before registering with the master The engine spawned its master registration and then bound --listen, so an engine whose port was taken could reach the master before it exited: on weed server with 18080 taken the master logged "vector node 127.0.0.1:18080 added" and removed it in the same millisecond, and the gateway noticed it. Bind first, and name the address in the error instead of "transport error". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-worker: keep an unsigned store off the AWS default chain A store the catalog says takes unsigned requests needs no credentials, but lance builds the AWS default credential chain for any store without them, and the chain probes instance metadata on every dataset open. With no credentials of its own, weed-vector spent a second on each query (1030-1066 ms against 32-46 ms with --storage-option ones). The merge of the operator's fallback and the vended options, which the engine and the Lance worker each did in their own copy, moves into table_options, and gives such a store empty credentials. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): serve an index with no identities and no engine credentials A second cluster runs weed s3 without -config and weed-vector without --storage-option, with no AWS credentials in either environment: weed s3 would turn them into an admin identity, and the engine would find them. On master its CreateIndex fails with InternalFailure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
ff7045a4e5 |
s3vectors: report the Euclidean distance, not its square (#12318)
A euclidean index answered QueryVectors with Lance's L2 distance, which is the squared Euclidean distance: [3, 4] was 25 from the origin, not 5. AWS defines the metric as the straight-line distance between the vectors. The gateway now returns its square root, so the single engine and the enterprise coordinator, which both hand back the raw Lance value, report the same distance. Ordering is unchanged. Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
06b9d01423 |
s3vectors: match a filter value against the elements of an array (#12321)
* vector: compare json_array_contains elements as typed JSON values Lance's json_array_contains compares each element's text with the value, or with the value wrapped in quotes, so 1 matches "1", true matches "true", and 1.0 misses a stored 1. The filter swap now replaces it like json_extract and json_exists: the value is parsed as JSON once per batch and compared with jsonb's typed equality, and only literal arguments are swapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: match a filter value against the elements of an array A filter never looked inside an array: {"tags": "space"} and {"tags": {"$in": ["space"]}} both missed "tags": ["space", "classic"]. AWS documents that $eq, and so a bare value, matches an array metadata value when any element equals the input. $eq, a bare value and each $in member now also match through json_array_contains; $ne and $nin remain their negations. Range operators, $exists and nested objects are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
5a60ce04e3 |
vector: stop re-parsing a filter's JSONPath for every scanned row (#12302)
* vector: parse a filter's JSONPath once per batch instead of per row
Lance's json_extract and json_exists parse their JSONPath argument again
for every row they see. Every translated S3 Vectors filter goes through
them, so a flat scan paid about 1 µs per row before the predicate
compared anything, and a 10%-selective {"category": {"$eq": "c3"}} ran
about 50% slower than the same query unfiltered.
The path is always a literal, so the scanner now parses the filter as
before and swaps in copies of the two functions that parse it once per
batch. They call the same jsonb selector and serializer, so results and
errors do not change. Calls whose path is not a literal keep Lance's
functions. The filter SQL is unchanged, so the gateway, existing tables
and the distributed workers (which run query::ann) need no change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: read the field once per extractor in $in and $nin
$in and $nin rendered one IN per member, each with its own json_extract,
so an N-member list parsed the row's metadata N times. Members that share
an extractor (strings and booleans, or numbers) now go into one IN list,
so a list reads the field at most twice. Matching is unchanged: each
group still only sees its own JSON type and still coalesces NULL before
$nin negates it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: rename the $in literal map so codespell passes
Codespell reads "lits" as a misspelling of "list".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Chris LuandClaude Opus 5.5
|
9a09506be6 |
s3vectors: bound ListVectors parameters and lowercase distanceMetric (#12290)
* s3vectors: bound ListVectors maxResults and segment parameters segmentCount had no upper bound, so segmentCount=1000000, segmentIndex=999999 made the segment emulation scan the whole index to return nothing. Apply the AWS ListVectors windows before authorization and any engine call: maxResults 1..1000, segmentCount 1..16, segmentIndex in [0, segmentCount), and segmentCount and segmentIndex given together. The fields become pointers so an explicit 0 is rejected while an omitted one still takes the default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: store and return distanceMetric in lowercase CreateIndex stored distanceMetric exactly as sent, so an index created with "COSINE" came back as "COSINE" from GetIndex and QueryVectors. AWS returns the lowercase enum values. Lowercase the metric when the index is created, and again when the record is read so indexes already stored in mixed case render the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris Lu
|
74f727d987 |
test: move the s3vectors pytest suite under test/s3vectors (#12230)
* test: move the s3vectors pytest suite under test/s3vectors The vectors API tests lived in test/vectors next to the new s3vectors integration home; the rust-vector-tests workflow's paths and working directory move with it. * gitignore: cover the binaries the s3vectors Makefile builds |