mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-03 10:37:35 +01:00
20c8876a524ec585a3e4336e6584180571f003dd
15990
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 dependabot[bot]
|
20c8876a52 |
build(deps): bump github.com/aws/aws-sdk-go-v2/service/sts from 1.51.1 to 1.51.6 (#12346)
build(deps): bump github.com/aws/aws-sdk-go-v2/service/sts Bumps [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) from 1.51.1 to 1.51.6. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.51.1...service/sts/v1.51.6) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/sts dependency-version: 1.51.6 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
a670553813 |
build(deps): bump google.golang.org/api from 0.297.0 to 0.310.0 (#12343)
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.297.0 to 0.310.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.297.0...v0.310.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-version: 0.310.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
9ae31fa425 |
build(deps): bump actions/setup-python from 5 to 7 (#12345)
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5 to 7. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v5...v7) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
e166629f9b |
build(deps): bump go.etcd.io/etcd/client/pkg/v3 from 3.7.1 to 3.7.3 (#12344)
Bumps [go.etcd.io/etcd/client/pkg/v3](https://github.com/etcd-io/etcd) from 3.7.1 to 3.7.3. - [Release notes](https://github.com/etcd-io/etcd/releases) - [Commits](https://github.com/etcd-io/etcd/compare/v3.7.1...v3.7.3) --- updated-dependencies: - dependency-name: go.etcd.io/etcd/client/pkg/v3 dependency-version: 3.7.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris LuandClaude Opus 5.5
|
b7b2a5947f |
admin: configurable file browser upload size limit (#12348)
* admin: carry a file browser upload size limit The admin server keeps the total size limit of one file browser upload, 500 MB by default as the UI enforced before, and hands it to the file browser page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: reject file browser uploads over the size limit The 500 MB limit was only checked in the browser, so any client could post larger uploads and fill the temp dir with spooled parts. Cap the request body at the limit plus room for the multipart framing, and answer 413 when the files add up to more than the limit. Parse the form before reading the path field. FormValue used to parse it first with the default 32 MB memory, which made the later 1 GB ParseMultipartForm a no-op; keep 32 MB explicitly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: show and check the configured upload limit in the file browser The upload dialog reads the limit from the page instead of a hard-coded 500 MB, lists it under the file picker, and skips the check when uploads are unlimited. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: add -uploadMaxSizeMB to set the file browser upload limit weed admin -uploadMaxSizeMB and weed mini -admin.uploadMaxSizeMB set the total size limit of one file browser upload; 0 removes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: note why the upload framing allowance is enough Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: test an upload whose files only exceed the limit together Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: remove spooled upload parts when the upload handler returns Parts over 32 MB spool to temp files. The router passes the handler a copy of the request, so net/http never sees the parsed form and never removes those files. Every large upload, and every one rejected for size after parsing, left them on disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a074538c8f |
admin: manage S3 vector buckets (#12334)
* s3vectors: run an operation for trusted local tooling Like s3tables.Manager.Execute, Server.Execute runs one S3 Vectors operation through its handler as a given identity, without a signed request. The admin console can then manage vector buckets through the same validation, authorization and catalog steps as the API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * admin: list, inspect, create and delete vector buckets over the API /api/s3vectors runs the S3 Vectors operations through Server.Execute as the admin account: list vector buckets with their encryption, read one bucket with its policy and each index's data type, dimension, distance metric and non-filterable keys, create and delete a bucket, and delete an index. Going through the operations keeps the API's rules, such as ICEBERG buckets not being vector buckets and the implicit index namespace not blocking a delete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: export the rule for which table buckets are vector buckets IsVectorBucketFormat names the check ListVectorBuckets and isVectorBucket already make, so the admin console can tell the same buckets apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * admin: vector bucket pages Object Store > Vector Buckets lists the vector buckets with their ARN, encryption and creation time, creates and deletes them, and opens one bucket with its indexes (data type, dimension, distance metric, non-filterable keys) and its policy, where an index can be deleted. A LANCE row on the Table Buckets page links to the same bucket's vector page, since both APIs serve it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * admin: stop the vector bucket and index lists at the list limit A page cut short by filtered ICEBERG buckets let the next page carry the list past s3TablesAdminListLimit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: say when a vector bucket or index list is cut at the limit The lists stop at s3TablesAdminListLimit while more remain, so mark them truncated and point to an S3 Vectors client for the rest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: indent the vector bucket policy A policy stored as one line ran off the right edge of the details page. Policies that are not valid JSON are shown as stored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: show a deeply nested vector bucket policy as stored Indenting multiplies the size of a deeply nested document, so a policy past 32 levels of nesting is shown without indenting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
0176231ea4 |
volume: replicas keep the primary's compression and metadata (#12350)
* volume(rust): replicate the stored needle instead of the request body The primary forwarded the original request body and headers, so a replica re-derived compression from the raw bytes. Send what Go's ReplicatedWrite sends: a one-part multipart upload of the stored bytes, gzip-marked when compressed, with the name, mime, pairs and Content-MD5, and carry ts, ttl, cm and fsync in the query so form-field values reach the replica too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): keep the primary's compression on replica writes A type=replicate upload now stores the bytes as received, like Go's ParseUpload, so a replica no longer compresses data the primary stored uncompressed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): replicas keep the primary's name and mime Forward the primary's filename and mime even past the 255-byte needle fields, so a replica applies the same limit and sets the same flags, and skip the filename MIME guess on type=replicate, so an untyped upload stays untyped on every copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): keep the multipart part Content-Type as sent Parsing the part type as a mime value lowercased it and dropped values it could not parse, so a replica stored a different mime than the primary. Read the header verbatim, as Go's part.Header.Get does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: replicas keep the primary's mime as forwarded A Go replica guessed the mime from the file extension, so an upload the primary stored as application/octet-stream, or with no mime at all, got a different type on the replica. On type=replicate keep the part's Content-Type as sent, and have the replication sender leave an empty mime empty instead of filling it in from the extension. This matches what the Rust replica does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): pick the JPEG orientation fix by file name, as Go does Go checks the mime and the upload's file name, Rust checked the mime and the URL path, so the same upload could be rotated on one server and not on the other. With the name and mime forwarded as stored, a replica now reaches the primary's decision, and a rotated image has no EXIF tag left to act on, so the fix stays a no-op there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
ec968aa016 |
s3vectors: support TagResource, UntagResource and ListTagsForResource (#12335)
* s3vectors: tag vector buckets and indexes
TagResource, UntagResource and ListTagsForResource answer on
/tags/{resourceArn} (POST, DELETE, GET) and by name. Each authorizes its
own s3vectors action on the bucket or index ARN, then reads or writes the
catalog record's tags. A resource holds at most 50 tags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: apply the tags passed to CreateVectorBucket and CreateIndex
The tags field was accepted and dropped. It is now stored on the new
bucket or index, and creating with tags needs s3vectors:TagResource in
addition to the create action.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: cover resource tagging end to end
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: keep tag operations to a bound table bucket
TagResource, UntagResource and ListTagsForResource run through Execute
with WithTableBucketCreatedAt now answer not found when the bucket name
holds another bucket, as the bound creates and deletes already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bind tagging to the authorized bucket and to vector indexes
The catalog tag calls carry the creation time of the bucket authorizeVector
approved, so a bucket created again under the name is not touched. An index
ARN must name a Lance table: another table in the vector namespace of a
format-less bucket is refused, as DeleteIndex does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: match a TagResource grant scoped to the bucket being created
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: merge tag writes into the tags the record holds at the write
TagResource and UntagResource merged into the tags they read before the
compare-and-swap and wrote that map on every attempt, so a tag write that
landed in between was undone. The merge now runs inside the mutation.
WithMaxTags lets an in-process caller cap the merged set there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: check the 50-tag limit inside the catalog's tag write
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: check a tag operation's bound bucket after its reads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: validate each tag key UntagResource removes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: validate untag keys one at a time
UntagResource sized a map by the number of submitted tag keys before
validating them, so a large body of repeated keys allocated far more than
the request itself, before authorization ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: authorize UntagResource before validating its keys
Validating a near-32 MiB body of tag keys costs about 13x its JSON
decode, so an unauthorized caller could make the gateway spend that
work on a request it then refuses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Chris LuandClaude Opus 5.5
|
690aabda0e |
filer: reuse IsEqualOrUnder for the rename subdirectory check (#12338)
* filer: test the rename subdirectory check through CanRename Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: reuse IsEqualOrUnder for the rename subdirectory check A trailing slash on the old name made the source path end in a slash, so a child of the source was no longer seen as under it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d650abe0e7 |
compression: compress JSON uploads, and match file types by extension (#12339)
* compression: compress JSON uploads, and match file types by extension
IsCompressableFileType did not recognise JSON media types, so S3 uploads of application/json (which carry no file name) were never compressed. Its callers also passed filepath.Base of the file name where it expects the extension, so the .json/.js/.html/... extension rules never matched a real file name. Both callers now pass filepath.Ext; the function lower-cases the extension and media type, ignores media-type parameters ("application/json; charset=utf-8"), and treats */json and *+json types as compressible.
* compression: treat application/json-seq as JSON
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): compress JSON uploads and match media types without parameters
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: keep uploader gzip only when it saves at least 10%
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Pat White <5015863+patwhite@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Jiri PucandClaude Opus 5.5
|
1637941368 |
filer: allow renaming into a sibling that shares the name prefix (#12337)
CanRename rejected a move whenever the new parent directory's path started with the source path as a plain string, so moving /dir/a into /dir/a.d (or anything under /dir/ab) failed with "can not move directory to a subdirectory of itself". Check the path boundary instead: reject only when the new parent is the source itself or lies under it. Claude-Session: https://claude.ai/code/session_01VncHKTehVroSAzRMUHykEX Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
7298b4ed37 |
fix(s3tables,s3vectors): bind DeleteNamespace and the vector bucket cleanup to the authorized records (#12331)
* s3tables: bind DeleteNamespace to the authorized namespace The delete claimed whatever finished namespace held the name at claim time, so one deleted and created again by another owner after the authorization read was removed under the first one's authorization. The claim now requires the creation time that was authorized. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: bind DeleteVectorBucket's namespace cleanup to the authorized bucket After the bound DeleteTableBucket reported the bucket not empty, the index namespace cleanup ran unbound, so a bucket deleted and created again in between lost its empty index namespace under the first bucket's authorization. DeleteNamespace now honors the bucket binding, checking the bucket after reading the namespace, and the cleanup carries it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
github-actions[bot]
|
984eac6a79 | 4.52 4.52 | ||
|
Chris LuandClaude Opus 5.5
|
aa2d94cc50 |
ci: cancel a PR's runs when it is merged or closed (#12330)
A merged or closed PR kept every pull_request run going: concurrency groups cancel a run only when a newer push arrives. After #12329 merged, 16 of its runs were still queued or running. On pull_request_target closed, cancel the queued and running pull_request runs on the PR's head branch whose commit belongs to the PR. Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
66a6598fdd |
fix(s3tables,s3vectors): bind table and vector bucket writes to the bucket they were authorized against (#12329)
* s3tables: bind table bucket policy writes to the authorized bucket PutTableBucketPolicy compared only the policy bytes, and DeleteTableBucketPolicy removed the policy unconditionally. A bucket deleted and created again under the same name between authorization and the write took a policy change nobody authorized for it. Both writes now require the bucket's CreatedAt and the policy they were authorized against, answer 409 otherwise and 404 for a bucket that is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: bind bucket tag writes to the authorized bucket TagResource and UntagResource checked the creation time only for a table ARN. On a bucket ARN the merged tags landed on a bucket deleted and created again since authorization. extractResourceOwnerAndBucket now returns the resource's CreatedAt for both kinds, and the write requires it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: bind bucket maintenance writes to the authorized bucket PutTableBucketMaintenanceConfiguration wrote onto whatever bucket held the name at write time. authorizeMaintenanceBucket now returns the bucket's CreatedAt like authorizeMaintenanceTable, and putMaintenanceConfiguration takes the check and creation time from both callers instead of an optional closure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: bind vector bucket policy writes to the authorized bucket Put/DeleteVectorBucketPolicy authorized inside the compare-and-swap with the owner from an earlier bucket read, so a bucket deleted and created again by another account in between had its policy rewritten by the old owner. The mutation now requires the entry's createdAt to match the record it authorized against. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: bind DeleteVectorBucket to the authorized bucket DeleteVectorBucket authorized one bucket record, then ran the trusted catalog DeleteTableBucket on whatever bucket held the name by then, so an empty bucket deleted and created again by another account in between was deleted by the old owner. The catalog delete now accepts the authorized bucket's createdAt through an in-process context value and treats any other bucket as gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: bind CreateIndex and DeleteIndex to the authorized bucket Both authorized the bucket, then ran trusted CreateNamespace, CreateTable or DeleteTable on whatever bucket held the name by then, so an index could be created in, or deleted from, a bucket another account created again in between. Those catalog operations now honor the bucket binding DeleteTableBucket already takes, and the vector handlers pass the authorized bucket's createdAt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4850def831 |
fix(s3tables): bind table and view writes to the record they were authorized against (#12328)
* s3tables: bind table and view writes to the record they were authorized against * s3tables: bind UpdateNamespace to the namespace it was authorized against UpdateNamespace writes back the whole record it read, owner and creation time included. A namespace deleted and created again before the write took the first one's owner and CreatedAt, which also undoes the CreatedAt binding creates now rely on. Refuse the write with 409 when the namespace at the path has another CreatedAt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
ce292a4e5e |
s3: require Admin to put or delete a bucket's lifecycle configuration (#12327)
PUT and DELETE ?lifecycle were authorized as Write, so an identity granted only object writes on a bucket could add expiration rules or remove the ones the bucket administrator set. They now need Admin on the bucket, like the encryption, public access block and ownership controls writes. Identities authorized by IAM policies are unaffected: they are still checked against s3:PutLifecycleConfiguration. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d431c1b35a |
s3api: let the vectors route, not the credential scope, raise the body limit (#12326)
SigV4 verification picked the 32 MiB S3 Vectors body limit from the credential scope, which is not verified until the body is hashed. A request claiming the s3vectors scope on any other route could make the gateway buffer 32 MiB before authentication instead of 10 MiB. The vectors handler now sets the limit on the request context, and every other route keeps 10 MiB. Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
02d22800aa |
s3vectors: accept bodies up to the 32 MiB limit, report larger ones as a size error, log engine failures (#12320)
* s3api: hash an s3vectors body up to the Vectors API limit SigV4 verification of a non-S3 request hashed at most 10 MiB of the body and silently dropped the rest, so any S3 Vectors request over 10 MiB failed its signature check with 403, although the API accepts 32 MiB. The limit now follows the signing service, and a body over it is reported as ErrEntityTooLarge instead of being truncated; the embedded IAM answers that with InvalidInput, as it already does for a body over its own limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: answer an oversized request with a ValidationException A body over 32 MiB was refused as "403 AccessDenied: authentication failed". A declared Content-Length over the limit is now refused before authentication, and a streamed body over it, whether the signature check or the handler reads it, gets the same "request body exceeds 32 MiB" ValidationException. The limit is the one SigV4 verification uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: log engine and catalog failures at the default level An engine error reached the client as a bare "internal error", and the cause was logged only with -v=1, so a misconfigured engine left nothing in a default log. Engine and catalog failures are now logged as errors, and a reported unavailability as a warning, with the operation, bucket and index. The response still leaves the engine's message out, since it can name internal hosts and paths, and points at the gateway log instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3api: hash the signed body through readRequestBody streamHashRequestBody repeated readRequestBody's read-past-the-limit check; it now reads the body with readRequestBody and hashes the bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
2eb219df61 |
s3vectors: encrypt the objects of a vector bucket declared AES256 (#12324)
* s3api: encrypt the objects of a vector bucket declared AES256 CreateVectorBucket recorded encryptionConfiguration sseType AES256 and returned it, but nothing on the write path read it: the index's data files, manifests and transactions were stored without SSE. A LANCE table bucket's record now supplies the bucket's default encryption whenever the bucket has none of its own, so PutObject, CreateMultipartUpload and CopyObject apply SSE-S3 to every object the engine writes. The S3 bucket metadata itself is unchanged. ICEBERG table buckets keep recording the declaration only: their catalog reads the manifests clients upload straight from the filer, which would see ciphertext. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3tables: refuse an AES256 vector bucket on a gateway without an SSE-S3 key A vector bucket's AES256 is now applied to every index write, and SSE-S3 fails each write when the gateway has no KEK. Creating the bucket there succeeded and the first CreateIndex came back as InternalFailure. CreateTableBucket now refuses a LANCE bucket declaring AES256 when the gateway reports no SSE-S3 key, naming s3.sse.kek and s3.sse.key; CreateVectorBucket returns it as ValidationException. ICEBERG buckets, whose declaration is only recorded, and tooling that sets no checker are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3api: refuse DeleteBucketEncryption on a bucket's declared encryption GetBucketEncryption reports a vector bucket's declared AES256, but DeleteBucketEncryption answered NoSuchBucketEncryptionConfiguration and the declaration kept applying. The declaration is fixed when the bucket is created, so the delete now returns MethodNotAllowed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): keep an inherited WEED_S3_SSE_KEK out of the test cluster The gateway refuses to start SSE-S3 with both s3.sse.kek and s3.sse.key set, so a developer shell exporting WEED_S3_SSE_KEK made the AES256 bucket test fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d72f761d83 |
seaweed-vector: work on its defaults, and without identities or credentials (#12323)
* seaweed-vector: default --namespace to the gateway's Lance port The old default, http://localhost:8888/lance, is a path nothing serves: the Lance Namespace listens on its own port, 9101 by default on weed s3, weed server, weed filer and weed mini. With it, the filer answered the engine's catalog calls with a 500 and every CreateIndex failed with InternalFailure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: default --listen to a port no SeaweedFS server takes 127.0.0.1:8080 is the volume server's default port, so weed-vector next to weed server exited with "Address already in use (os error 98)". 8090 is not among the defaults of master, volume, filer, S3 and its Iceberg and Lance catalogs, IAM, WebDAV, SFTP, the message queue broker and agent, the admin server or weed mini, nor any of their gRPC ports (HTTP + 10000). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance: take any key while the S3 server runs without identities Without identities the S3 port accepts any signature and treats the caller as the admin. The Lance catalog still checked an x-api-key or Bearer token against identities that did not exist, so weed-vector started with --storage-option credentials on such a server got 401 on every catalog call. The catalog now skips both checks when S3 auth is off and leaves the caller to the S3 authenticator, which admits it as the admin, as it already did for a request with no key. With identities configured nothing changes: an unknown key is still 401. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance: tell clients to skip signing while S3 auth is off A client with no credentials of its own could not reach a table's data on a server without identities, though the S3 port takes unsigned requests there: lance fell back to the AWS default credential chain and failed with CredentialsNotLoaded. weed-vector without --storage-option failed every CreateIndex that way. The catalog now adds aws_skip_signature to the storage options it hands out while S3 auth is off, and leaves it out once identities exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: bind the listener before registering with the master The engine spawned its master registration and then bound --listen, so an engine whose port was taken could reach the master before it exited: on weed server with 18080 taken the master logged "vector node 127.0.0.1:18080 added" and removed it in the same millisecond, and the gateway noticed it. Bind first, and name the address in the error instead of "transport error". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-worker: keep an unsigned store off the AWS default chain A store the catalog says takes unsigned requests needs no credentials, but lance builds the AWS default credential chain for any store without them, and the chain probes instance metadata on every dataset open. With no credentials of its own, weed-vector spent a second on each query (1030-1066 ms against 32-46 ms with --storage-option ones). The merge of the operator's fallback and the vended options, which the engine and the Lance worker each did in their own copy, moves into table_options, and gives such a store empty credentials. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): serve an index with no identities and no engine credentials A second cluster runs weed s3 without -config and weed-vector without --storage-option, with no AWS credentials in either environment: weed s3 would turn them into an admin identity, and the engine would find them. On master its CreateIndex fails with InternalFailure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
ff7045a4e5 |
s3vectors: report the Euclidean distance, not its square (#12318)
A euclidean index answered QueryVectors with Lance's L2 distance, which is the squared Euclidean distance: [3, 4] was 25 from the origin, not 5. AWS defines the metric as the straight-line distance between the vectors. The gateway now returns its square root, so the single engine and the enterprise coordinator, which both hand back the raw Lance value, report the same distance. Ordering is unchanged. Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d6c0d728e4 |
s3vectors: resolve the caller like table buckets and honor bucket-scoped identity actions (#12319)
* s3vectors: resolve the caller the way table buckets do The vector checks used the identity name as principal, while the catalog records a bucket's owner through getAccountID: the account id, or the issuer-scoped user id of a federated identity. An identity with an account or a federated identity did not own the vector bucket it created, a Deny naming its account did not bind it, and an Admin identity in the admin account did not get the admin bypass it has on table buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: honor identity actions scoped to a vector bucket An identity action such as s3vectors:QueryVectors:team-a granted nothing, because the vector policy context never named the bucket. Table buckets match <action>:<bucket> through the same context field, so pass the bucket the check is about. Creating a bucket stays unscoped, as for table buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
54777d7fb9 |
vector: build weed-vector without a system protoc (#12317)
* vector: build without a system protoc lance's build scripts run $PROTOC and fail without one. Point it at the protoc that protoc-bin-vendored ships, at the version in Cargo.lock, so building inside seaweed-vector/ needs only the Rust toolchain. An explicit PROTOC still wins, and a registry without the crate falls back to PATH. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): build weed-vector inside its workspace Cargo reads seaweed-vector/.cargo/config.toml only when it runs there, so build from that directory to pick up the vendored protoc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * ci(vector): drop the PROTOC step the workspace config replaces Builds run inside seaweed-vector/, where the cargo config already points lance at the vendored protoc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): resolve CARGO_TARGET_DIR before changing directory A relative CARGO_TARGET_DIR given to make would point cargo, which now runs in seaweed-vector/, and the cp after it at different directories. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: pick the vendored protoc on every Linux host it ships for protoc-bin-vendored also carries ppc64le, s390x and x86_32 binaries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
80e0bf97dd |
shell: dial volume servers on their registered gRPC port in volume.scrub, ec.scrub, volume.server.state (#12315)
* shell: dial scrubbed volume servers on their registered gRPC port volume.scrub, ec.scrub and volume.server.state built each data node's address from its HTTP address alone, so a volume server whose gRPC port is not the HTTP port + 10000 was dialed on the wrong port. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: resolve -node volume servers to their registered gRPC port An explicit -node/-nodes list was dialed on HTTP port + 10000 too. Match each entry against the topology and take the data node's gRPC port; an address the master does not know is used as given. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep a gRPC port given in -node A node given as host:port.grpcPort was rewritten to the registered gRPC port; only a bare host:port takes it from the topology. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: dial -node volume servers when the topology is unavailable An explicit node list did not need the master before; keep using it as given if the volume list cannot be fetched. Selecting every node still requires the topology. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: bound the topology lookup for an explicit -node list With the master down, volumeServer.state -nodes waited on the master forever instead of reaching the given volume servers. Give the lookup a 30s deadline and fall back to the addresses as given. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: resolve volume server addresses from a topology-only listing Only the data nodes are needed to find each server's gRPC port, so ask the master for the topology without its volumes and ec shards. A large cluster no longer risks running past the -node lookup deadline and falling back to the HTTP port + 10000. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: skip the topology lookup when every -node names its gRPC port A host:port.grpcPort entry is dialed as given, so a list made only of those does not need the master and no longer waits on it during an outage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b9bc42c217 |
s3vectors: vend write credentials to index data only to the vector engine and Lance worker (#12316)
* security: share the filer admin token check CheckGrpcAdminPeer verifies an admin Bearer token inline. Move the check into IsFilerAdminToken so an HTTP endpoint can accept the same token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: recognize the cluster's own components by a signed token A request whose X-Seaweedfs-Cluster-Token header carries an admin token signed with jwt.filer_signing.key comes from a component that holds the cluster key: the vector engine or the Lance maintenance worker. Mark such a request as trusted. It still authenticates as an identity, which stays the principal for vended credentials. A dedicated header leaves the Authorization header to Lance OAuth clients. A gateway with no key trusts no one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: vend write credentials to a vector bucket's indexes only to trusted callers An index in a vector bucket's vector namespace is also a Lance table, so a caller granted s3tables:UpdateTable could describe it, or declare it again, and get credentials that write the index data without passing the vector bucket policy. In that namespace both now vend read-only credentials unless the caller is trusted, and a declare from anyone else no longer writes the dataset markers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance worker: prove the cluster key to the Lance namespace The gateway now vends write credentials to a vector bucket's indexes only to callers that send a token signed with jwt.filer_signing.key. The worker already takes that key for the admin stream; its namespace client now mints a token from it for every catalog request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: describe the index the gateway created and prove the cluster key CreateIndex declared a table the gateway had just created, which only worked because declare is idempotent. Describe it instead. The Lance namespace vends write credentials to a vector bucket's indexes only to callers that send a token signed with jwt.filer_signing.key, so take the key with --filer-signing-key, the same flag the Lance worker has. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance worker: keep the cluster key out of Debug output Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: answer a declare of an existing table with its own location CreateTable succeeds for a name that already exists without looking at the location requested, and declare kept using that location for the dataset markers and the vended credentials. Declaring an existing table with another table's location, such as a vector index, got credentials to that table's data. Use the location the catalog holds, which CreateTable now also reports for a new table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: sign cluster tokens with a key derived for the catalog A cluster token was a filer admin token signed with jwt.filer_signing.key and, by default, never expired. Sent to the Lance namespace on every request, often over plain HTTP, a captured one would also authorize the filer's IAM service. Sign it with a key derived from the cluster key for this one use, which the admin services do not accept, and mint it with a five minute lifetime. weed-vector drops its token lifetime flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: follow a reloaded jwt.filer_signing.key The Lance namespace copied the cluster key at startup. A SIGHUP reload refreshes the gateway's filer guard but not that copy, so after a key rotation the restarted vector engine and worker were no longer trusted. Read the key from the guard on every check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: keep a vector bucket's indexes read-only An index in a vector bucket is also a catalog table the Iceberg REST catalog loads, and LoadTable vended credentials that could write it to any caller granted s3tables:UpdateTable. A commit staged its metadata file in the index directory on the same grant. Apply the vector namespace rule to both, after the caller's own UpdateTable check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: confine a created table's location to its namespace before writing Stage-create skips the registration that confines a location to the table's namespace, yet vends write credentials for that location. A caller allowed to create a table in any namespace of a bucket could name another namespace's prefix, a vector index included, and get write credentials to it. The non-staged create also wrote its first metadata file there before the registration refused the location. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: a declare that finds an existing table answers like describe Creating a name that already exists is idempotent, so a caller allowed only to create tables could declare someone else's table and get write credentials to it, and clear its deregistered marker. Describe would have vended that caller read-only credentials. A declare now looks the name up first; when it only finds the table, writing needs UpdateTable, as it does for describe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
06b9d01423 |
s3vectors: match a filter value against the elements of an array (#12321)
* vector: compare json_array_contains elements as typed JSON values Lance's json_array_contains compares each element's text with the value, or with the value wrapped in quotes, so 1 matches "1", true matches "true", and 1.0 misses a stored 1. The filter swap now replaces it like json_extract and json_exists: the value is parsed as JSON once per batch and compared with jsonb's typed equality, and only literal arguments are swapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: match a filter value against the elements of an array A filter never looked inside an array: {"tags": "space"} and {"tags": {"$in": ["space"]}} both missed "tags": ["space", "classic"]. AWS documents that $eq, and so a bare value, matches an array metadata value when any element equals the input. $eq, a bare value and each $in member now also match through json_array_contains; $ne and $nin remain their negations. Range operators, $exists and nested objects are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
533f5060b9 |
filer: scope a prefix JWT to the directory a zip extract writes under (#12305)
* filer: scope a prefix JWT to the directory a zip extract writes under Without a trailing slash, an extract request writes members beside the named path, in its parent directory, while the JWT check only saw the request path. A token scoped to /tenant1 could therefore extract into / by posting to /tenant1?extract=zip. Add the extract root to the paths a prefix-restricted token must cover. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: do not keep an extracted archive under the name ".." A multipart file name of ".." passes path.Base unchanged, so extract.keep stored the archive as an entry literally named ".." in the target directory, a name that local sinks resolve to the parent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * leveldb3: keep a non-local bucket name out of the per-bucket folder findDB splits the bucket out of a /buckets/ path and uses it as a folder name under the store directory. The name cannot hold a slash and dot names already go to the default DB, so this only rejects an empty or non-local name, but it also lets static analysis see the folder cannot escape the store directory. * filer: clean the zip extract root before scoping and writing A percent-encoded dot segment survives the path cleaning in front of the mux, so a request path ending in "/" reached extractZipRoot with ".." in it. The JWT scope check cleans the path it compares, but the members were written under the raw root, so the two could disagree. * leveldb3: apply the bucket folder check to bucket create and delete events findDB already keeps dot and non-local bucket names out of the per-bucket folders, but OnBucketCreation and OnBucketDeletion used the name as is. A directory event for /buckets/.. opened a DB in the parent of the store directory, and its delete removed that parent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * leveldb3: keep a bucket delete event off the default DB A bucket named _main maps to the default DB in findDB, so deleting /buckets/_main closed the default DB and removed its folder, taking every non-bucket entry with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * leveldb3: keep bucket names with a path separator out of the store dir A bucket event named x/../_main passed the local-name check, and its delete removed the default DB folder once store.dir/x existed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * leveldb3: only reject the platform separator in a bucket name A backslash is an ordinary file name character on Unix, and a bucket folder named that way already exists for anyone who created one. Rejecting it on every platform would route those entries to the default DB after an upgrade. The slash can never appear in a bucket name here, and on Windows the backslash is the separator, so rejecting os.PathSeparator alongside the slash keeps the folder flat without changing Unix routing. * leveldb3: check for an escaped bucket db before its delete event --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d84ae4f2ab |
volume: don't let an unreachable tier backend hold up volume server startup (#12312)
* s3 tier backend: bound connect and response-header waits The tier client used the SDK default transport: a 30s dial timeout and no limit on waiting for response headers. A blackholed endpoint cost about two minutes per request with the SDK's retries, and an endpoint that accepts connections but never answers held the request forever. Dial with a 10s timeout and give up on a response whose headers do not arrive within a minute. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: stop the disk scan from waiting on a stalled tier backend A volume server registers with the master only after every volume on its disks has loaded, and loading a tiered volume reads its superblock from the remote object. With the S3 endpoint unreachable, each read waited out the client's timeouts and retries on one of the loader workers, so start-up grew with the number of tiered volumes and the local volumes were not served until it finished. A tiered volume already loads without its superblock when the read fails. Give that read 10s, and once it stalls, let the backend's other volumes skip it for a minute instead of each waiting out the timeout. The volume reads from the backend as before once the endpoint answers again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: bound connect and read waits of the S3 tier client Mirror the Go tier client: a 10s connect timeout instead of the SDK's 3.1s default, and a minute to receive a response, so an endpoint that accepts connections but never answers cannot hold a read forever. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: stop volume loading from waiting on a stalled tier backend Mirror the Go loader: the superblock read of a tiered volume gets 10s, and once it stalls the backend's other volumes skip it for a minute. The volume loads with its .vif version as before when the read fails, and reads from the backend once the endpoint answers again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: keep the .vif replication when a remote superblock read fails Go applies the .vif replication to a tiered volume whether or not its superblock read succeeded, so a volume loaded while its backend is down still reports its replica placement. The Rust loader applied it only on success and reported the default placement instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: skip the .idx head repair while the tier backend is unreachable The repair reads needle headers from the remote .dat, so after the superblock read failed it would wait on the same backend again and hold up the load. It runs on the next load that reaches the backend. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: skip the .idx head repair while the tier backend is unreachable Mirrors the Go load: after the remote superblock read failed, the repair's remote header reads would wait on the same backend again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 tier backend: give up a ranged read whose body stops arriving The response-header timeout ends once headers arrive, so an endpoint that answers and then sends no body held ReadAt, and a superblock read's worker, forever. Cancel the request once the body makes no progress for s3ReadTimeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: test that a tier read whose body stalls gives up The SDK's stalled-stream protection already fails it, matching the Go read's no-progress cancel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): keep the S3 read timeout off upload_part The SDK's read timeout runs from the start of the request, so a 64 MiB part on a slow link would time out while its body is still being sent. Go's ResponseHeaderTimeout only starts after the body is written. upload_part now keeps the connect timeout and drops the read timeout; stalled-stream protection still fails an upload that stops moving. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): bound the upload_part response wait from when the part is sent Without a read timeout, an upload_part whose body was accepted but never answered waited forever. The part body now goes to the connection in 64 KiB chunks, and once the last chunk is taken the response must arrive within S3_READ_TIMEOUT, like Go's ResponseHeaderTimeout. A retry rebuilds the body and restarts that clock. The body keeps its in-memory contents, so signing and the checksum header are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): keep a retried part's response clock from an earlier attempt The send state was a single flag, and a retry body could be rebuilt and fully sent before the waiter saw the flag drop, so the waiter kept the earlier attempt's deadline. Each body now carries its attempt number, only the latest attempt marks the part sent, and the deadline holds only while no later attempt has started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
965c6c06fb |
Ship weed-vector in release assets and Docker images (#12322)
* ci(release): publish weed-vector binaries
Tagged releases now carry weed-vector_linux_{amd64,arm64}.tar.gz, static
musl builds made the same way as weed-worker's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* docker: carry weed-vector in the image
The image takes a pre-built weed-vector the way it takes weed-worker, or an
empty placeholder where none was built. `vector` runs it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* ci(docker): build weed-vector for the images
The container workflows cross-compile weed-vector for amd64 and arm64
beside weed-worker and place it in the Docker context.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* install.sh: add the vector component
`--component vector` installs weed-vector from a release, and `all` adds it
when the release carries one, as it does for the worker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* ci(docker): skip weed-vector for source refs that predate it
The manual latest-image rebuild accepts any source_ref; one without the
engine's server crate now builds its image without weed-vector instead of
failing the Rust job.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* docker: have the vector engine listen beyond loopback
weed-vector listens on 127.0.0.1 by default, which a gateway in another
container cannot reach. `vector` now adds --listen 0.0.0.0:8090 unless a
listen address is given by flag or WEED_VECTOR_LISTEN. The engine registers
its container address for a wildcard listen.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Chris LuandClaude Opus 5.5
|
2ea63bad48 |
filer.backup: keep local sink writes inside the sink directory (#12310)
* localsink: skip keys that resolve outside the sink directory Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: reject entry paths with dot segments Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test renames onto dot-segment targets Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: keep the dot-segment rename test inside one bucket scope Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * localsink: resolve keys through an os.Root on the sink directory The lexical check alone lets a symlinked directory that already exists under the sink directory carry writes and deletes outside it. Create, rename and delete now go through an os.Root opened on the sink directory, so a symlink can only resolve inside it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * localsink: follow absolute symlinks that resolve inside the sink directory os.Root rejects every absolute symlink target, so a backup through an absolute link that points back into the sink failed, and a delete through it left the file behind. Resolve the parent of the key first and use the resolved path when it stays under the sink directory. Operations still go through os.Root, so a link swapped after the check cannot escape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
github-actions[bot]
|
88b76b5890 | docs: regenerate star history chart | ||
|
Chris LuandClaude Opus 5.5
|
5cf053c32e |
volume: read a manifest chunk from another replica when one fails (#12313)
* operation: read a manifest chunk from the next replica when one fails ChunkedFileReader read each chunk from one random location and returned the first error, so a chunk whose cached replica went down failed the whole file although another replica had it. Try the chunk's locations in turn (local-data replicas first) and resume each attempt at the byte the previous one reached. A failed write to the caller still ends the read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * operation: drop a chunk volume's cached locations after a failed read A chunk read that failed on a replica left that replica in the 10-minute location cache, so later reads kept trying it first. Drop the volume's cached locations after a failure, and when every cached replica has failed, ask the master once more and try the locations it now lists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * operation: don't continue a partly sent chunk on another replica A volume server streams a large needle before it has verified the checksum and aborts the response on a mismatch, so bytes that already reached the reader may be corrupt. Continuing such a chunk on another replica would hand back a mix of bad and good bytes as a complete read. Move to the next replica only while nothing of the chunk has been written; a mid-body failure still fails the read, and the dropped cache entry lets the next read pick a live replica. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * operation: treat a short chunk body as a failed replica read A replica that ends a chunk's body cleanly before the chunk's size made the read look successful, so the next replica was never tried and the following chunks landed at the wrong offsets. Read exactly the chunk's remaining length and report a short body as an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ab204a03e6 |
fix(s3): verify SigV4 signature with decoded path behind X-Forwarded-Prefix (#12299)
* fix(s3): verify SigV4 signature with decoded path behind X-Forwarded-Prefix When SeaweedFS is hosted behind a reverse proxy that strips a URL prefix (X-Forwarded-Prefix) and forwards the percent-encoded request path, signature verification fails for object keys containing non-ASCII characters. The client signs the canonical URI as the single percent-encoded path including the prefix, but the server verifies against the already-encoded forwarded path joined to the prefix, producing a mismatch (SignatureDoesNotMatch). Decode the forwarded path and rejoin the prefix before verifying, mirroring the existing un-prefixed decoded retry. ASCII keys are unaffected because the decoded path equals the original, so the retry is skipped. Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com> * s3: build the SigV4 candidate paths once and try them per host The forwarded-prefix and plain branches each repeated the decode-and-verify step for every host candidate. List the escaped and decoded paths once, the prefixed forms first, and verify them in the same order as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d6ec6bc4e7 |
fix(s3tables): let DeleteTableBucket remove empty leftover directories (#12307)
* s3tables: let DeleteTableBucket remove empty leftover directories Page through the bucket instead of reading one page, and check each bare directory a deleted namespace left behind for files: empty ones go with the bucket, one holding files refuses with 409 naming it. The delete claims the bucket before scanning, and CreateNamespace refuses a claimed bucket and confirms it again after its write, so a namespace created meanwhile is seen by the scan or turned away instead of deleted with the bucket. The bucket is removed only while it still carries that claim, otherwise 409. A delete that refuses lifts the claim, retrying a failed release, since a claim left behind refuses every create in the bucket. The new namespace stays marked unfinished until that check passes, so no table or view is created in one that gets backed out; a create takes over a namespace still carrying the mark. Both checks accept only the bucket the create was authorized against, by its creation time: one recreated under the same name answers 404 and gets nothing written into it. The removal is not recursive. Each leftover is claimed first, only while it is still the directory the scan saw, and goes only while it holds no file or catalog record; the bucket goes only while nothing is left in it. A file uploaded or a namespace taken back after the scan is refused with 409 instead of deleted. * s3tables: keep directory-backed S3 objects in DeleteTableBucket * s3tables: keep a file that replaced a leftover directory in DeleteTableBucket * s3tables: claim only the authorized bucket in DeleteTableBucket * s3tables: assert the prefix-object mark when claiming a leftover directory * s3tables: hide unfinished namespaces from readers and DeleteNamespace * s3tables: stop CreateNamespace on any failed bucket check * s3tables: retry lifting claims on leftover directories * s3vectors: declare the index namespace again after losing it to a concurrent declare CreateNamespace now writes a namespace marked unfinished and confirms it after checking the bucket. A second CreateIndex that declares the namespace in that window takes the marked record over, and the first one's confirmation answers 409, which failed its CreateIndex. Declare it again, a bounded number of times, when the create answers a conflict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: name an unfinished namespace that keeps a bucket from being deleted A CreateNamespace that dies between its write and its confirmation leaves a namespace marked unfinished. Listings and GetNamespace hide it and DeleteNamespace answers that it does not exist, while DeleteTableBucket counts it, so the bucket could not be deleted and nothing said why. Name it in the BucketNotEmpty answer; creating it again takes it over. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: read a refused empty-directory delete with filer.DeleteEntryError Use the filer's own reading of a non-recursive delete's answer instead of a copy of its message prefix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: release the held create when the concurrent-declare test fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: buffer the held create's result so it can exit after a failure Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: retry lifting a refused namespace delete's claim A claim left on a namespace refuses every create in it, the same harm the bucket's claim does, so its release retries too. With no one-shot caller left, the retrying release takes the releaseDeleteClaim name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: keep the data under a hidden directory on bucket delete The scan filed every hidden entry by name for a recursive delete, so .hidden/data went with the bucket. A hidden directory waits with the leftovers instead: the walk refuses one that holds a file or an object, and removeEmptyTree takes only one that is still bare and empty. A hidden file is removed by name as before. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandClaude Opus 5.5
|
55f1e1efc5 |
vector: cache reads of write-once Lance files across requests (#12303)
* lance worker: let dataset::open take an object store wrapper open_with is open with the dataset's object store wrapped, so a caller such as weed-vector can put a read cache in front of the table's files. open keeps its signature and opens unwrapped, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: cache reads of write-once Lance files across requests Every VectorService call opens the table afresh, and Lance keeps no data pages between opens, so each QueryVectors re-read the whole vector column through the S3 gateway: 83 MB and about 228 range GETs per query at 50k x 384. The gateway and volume server spent their CPU streaming it and throughput stayed at 6-8 queries/s whatever the client concurrency. weed-vector now wraps each table's object store with a bounded in-memory cache (--read-cache-mb, 512 MiB by default, 0 disables). It only serves data files, deletion files and index files: Lance writes each once under a fresh random name, so a cached read cannot go stale, even when an index is dropped and recreated at the same location. Manifests, tags and every other path still go to the store, and each call still describes the table, loads the latest version and takes the query lease, so reads see every write committed before them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: cache an owned copy of each fetched read A range from get_ranges can be a slice of a larger response buffer, such as a coalesced fetch, and caching the slice kept the whole buffer alive while the weigher counted only the slice. The cache now copies each fetched range and whole-object read into its own buffer, so --read-cache-mb bounds what it actually holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Mohd Quamar Tyagi
|
5f4d05b103 |
admin: measure volume server usage like its capacity (#12306)
The volume server views summed the logical size of the volumes into DiskUsage while DiskCapacity came from the filesystem statfs size. On a compressing filesystem the logical sum exceeds the physical capacity, so the server details popup derived a negative available space and rendered it as "NaN undefined". Pair each disk's capacity with the usage measured the same way, using the statfs numbers the volume server already heartbeats, as CollectTierStats does. Volume servers without statfs numbers keep the previous slot-based capacity and logical bytes, and EC shard bytes are only counted there instead of on top of the filesystem usage. |
||
|
Chris LuandClaude Opus 5.5
|
bcb3940b37 |
test(s3vectors): add an opt-in end-to-end perf suite for vector buckets (#12297)
* test(s3vectors): add an opt-in end-to-end perf suite perf_test.py runs on the same cluster fixture as the integration suite (Go or Rust volume, single or distributed queries) and times the data plane through the S3 Vectors API: PutVectors ingest and upsert, QueryVectors latency with and without metadata and a 10% filter, query throughput across client threads, GetVectors, segmented ListVectors, and DeleteVectors. Recall is checked against an exact client-side search and listings must be complete; timings only report. Sizes come from PERF_* variables, and results go to a JSON report and a printed summary. It runs through run_perf.sh or `make perf`. run_test.sh does not collect it, so CI is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): check every listed key in the perf suite, not just the count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): cap perf samples at PERF_VECTORS so small runs work Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): warm perf worker clients before starting throughput timers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): report the key count of perf GetVectors and DeleteVectors calls Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
5a60ce04e3 |
vector: stop re-parsing a filter's JSONPath for every scanned row (#12302)
* vector: parse a filter's JSONPath once per batch instead of per row
Lance's json_extract and json_exists parse their JSONPath argument again
for every row they see. Every translated S3 Vectors filter goes through
them, so a flat scan paid about 1 µs per row before the predicate
compared anything, and a 10%-selective {"category": {"$eq": "c3"}} ran
about 50% slower than the same query unfiltered.
The path is always a literal, so the scanner now parses the filter as
before and swaps in copies of the two functions that parse it once per
batch. They call the same jsonb selector and serializer, so results and
errors do not change. Calls whose path is not a literal keep Lance's
functions. The filter SQL is unchanged, so the gateway, existing tables
and the distributed workers (which run query::ann) need no change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: read the field once per extractor in $in and $nin
$in and $nin rendered one IN per member, each with its own json_extract,
so an N-member list parsed the row's metadata N times. Members that share
an extractor (strings and booleans, or numbers) now go into one IN list,
so a list reads the field at most twice. Matching is unchanged: each
group still only sees its own JSON type and still coalesces NULL before
$nin negates it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: rename the $in literal map so codespell passes
Codespell reads "lits" as a misspelling of "list".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Chris LuandClaude Opus 5.5
|
a1a208506e |
volume: serve a ranged GET without reading the whole needle (#12301)
* volume: serve a ranged GET without reading the whole needle A ranged GET read every needle up to 1MB in full, checksum and all, then copied out the requested bytes. Read the metadata and stream only the range instead, as the Go server already does above 1MB and the Rust server does at any size. A small range of a 790KB needle goes from ~340us to ~160us. The metadata read stays non-terminal: the range read that follows reports the io outcome, so a failing disk still builds its error streak. HEAD, compressed needles and chunk manifests keep their current reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * volume: read the whole needle for a range that may need an image transform Without a URL extension, the stored name can still make a ranged GET with width, height or crop parameters an image transform, which needs the payload. Keep the full read there instead of streaming the range. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * volume: stream a range from the record its metadata came from The streamed read looked the needle up again, so a replacement written after the metadata read sent new bytes under the old length, checksum and headers. Read the record the metadata read resolved, as the Rust server does, and after a compaction refuse a record whose size changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * rust volume: read the whole needle for a range that may need an image transform A ranged GET with no URL extension but width, height or crop parameters took the metadata-first path, so a stored image name never got its transform. Read a small needle whole there, as the Go server does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * volume: keep later parts of a multi-range reply on a moved record When a compaction moved the record before one part of a multi-range reply, that part re-resolved it, but the next part went back to the offset the metadata read saw. Remember the re-resolved offset too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * volume: check a compaction-moved record is the one the metadata came from After a compaction, a range re-resolved the needle and accepted any record of the same size, so a replacement written in between could send its bytes under the old headers. Compare the moved record's checksum and append time with the metadata already read, and fail otherwise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * volume: accept a re-encoded checksum and count read errors on a moved record Compaction rewrites a pre-3.09 checksum in the current encoding, so the moved-record check now accepts either. A failed read of the moved record's metadata now feeds the io-error streak and surfaces as that error instead of not-found. The pinned record is a fresh value rather than a write through the index's pointer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * volume: keep a moved record's wrapped offset when a range follows it With 4-byte offsets, a record past the 32GB wrap is read at offset plus MaxPossibleVolumeSize. The compaction refresh dropped that, so it checked and streamed the wrong location. Retry the wrapped offset in the moved record check, as the metadata read does, and apply it to the stream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
4be191173a |
vector: keep ListVectors pages and segments flat as an index grows; read only keys in GetVectors (#12300)
* vector: scan only the fragments a ListVectors page covers A page pushed its offset into the scan, but the scan still loaded every fragment first, reading each one's deletion file: about 105 S3 GETs per page at 50k rows against 25 at 10k. The page's fragments now come from the live row counts in the manifest, so a page reads only its own. Tokens and row order are unchanged; a manifest without the counts keeps the old scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: list a ListVectors segment in the engine The gateway paged through the whole index for every segment and kept one row in segmentCount, so a segment cost segmentCount engine pages and 16 segments listed slower than one. The engine now takes segment_count and segment_index: it reads keys across the stride, keeps the segment's rows and takes their data and metadata alone, one call per page. A segment is still the rows whose position modulo segmentCount is segmentIndex, and its token still the position to resume from. An engine that predates the fields does not set segmented, and the gateway then pages through it as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: read only the key column across the table in GetVectors The key lookup scanned every column. On S3, lance reads a column during the scan when a row of it is under 1000 bytes, so a 128-dimension vector was read for every row of the table to return 100 of them, and data and metadata were read even when the request did not ask for them. The scan now projects the requested columns and materializes everything but the key late, for the matching rows only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: start the legacy segment fallback from the page already read With an engine that ignores segment_count, the gateway dropped the page it had just read and fetched it again. The fallback now consumes that page first, so a one-row segment page still costs one engine call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: check metadata on segmented ListVectors pages The segment test listed data only. Its rows now carry metadata on two in three keys, and every segmented row, read through take_rows, must equal the same row in a plain listing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: lease the version while a segment takes its data A segmented page with data or metadata reads keys first and then takes the other columns from the same version, so maintenance could reclaim its files in between. That path now holds a lease like GetVectors. A keys-only page is one scan and stays unleased, as plain pages are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: leave GetVectors materialization to lance Forcing every column late made a 100-key GetVectors with data slower at 10k x 128 in the perf cluster: 54.7 ms against 46.3 ms. The take issued a read per row, where lance's heuristic reads a narrow vector column in one pass per fragment. The projection keeps the gain on its own: with neither data nor metadata requested, the call went from 38.6 ms to 25.3 ms and from 150 to 89 S3 GETs. At 50k x 384 the vector is already late under the heuristic, and a full lookup is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a7a6482752 |
vector: follow a leader hint at once and skip one that failed this pass (#12295)
* vector: follow a leader hint at once and skip one that failed this pass A follower naming a dead leader stalled master failover. Each pass broke off at the first hint, tried that leader first on the next pass, then got the same hint from the same follower, so the later configured masters were never tried. A pass now works like wdclient.tryAllMasters: it follows a hint at once, does not follow a hint to a master that already failed in that pass, and moves on to the rest. The hops are bounded by the master count, as the volume heartbeat bounds its hints, so two masters naming each other cannot spin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: test a hint naming a failed master in its gRPC form A pass keys failed masters by host:http, so a configured m1:9333 that failed must also stop a hint of m1:9333.19333. Cover that equivalence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
6cdddcd76d |
vector: confine DropIndex to the table's own path, bound master dials, check the gRPC port (#12294)
* vector: refuse to drop a location that is not the table's own DropIndex removed everything under whatever location the catalog described. That location is caller-supplied through the Lance namespace and only confined to the bucket, so a table registered at s3://bkt/vector, or at another index's directory, wiped every index in the bucket or that index. drop now accepts only the colocated path the gateway's tableLocation derives from the table id, after trimming a trailing slash, and the RPC answers FailedPrecondition otherwise. This mirrors the ancestor and live-entry guards on the catalog's DeleteTable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: bound the master dial and try a leader hint once A remembered leader hint was never cleared and the dial had no connect timeout, so a black-holed former leader added the kernel's SYN timeout (about two minutes) to every reconnect pass before the configured masters were tried, and registration lapsed. The dial now gives up after 5s, the same bound wdclient puts on a master call, and a hint is tried once ahead of the configured masters, as wdclient.tryAllMasters keeps nextHintedLeader for one cycle only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: refuse a master port whose gRPC port would pass 65535 grpc_addr derived the gRPC port as http + 10000 in u16, which panics in debug builds and in release silently dials a wrapped port for HTTP ports above 55535. Go's pb.ServerToGrpcAddress adds in int, so the port never wraps and the dial fails instead. The addition is now checked, and weed-vector refuses to start with an error asking for the explicit host:http.grpc form. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: refuse a numeric master port past 65535 A master given as m1:65536 or m1:9333.65536 failed both u16 parses and fell through to the raw address, so startup accepted it and every dial failed. Ports now parse as u64, as Go's ServerToGrpcAddress does with ParseUint, and one outside u16 is refused at startup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: bound the master's first keep-connected answer The connect timeout only covers TCP. A master that accepts the stream and never answers held stream_to forever: grpc-go sends the response headers with the first message, so the call never returned and later seeds were never tried. A healthy master always answers at once (a follower with its leader hint), so the stream open and the first response now share the same 5s bound; after that the long-lived stream stays untimed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
github-actions[bot]
|
455400ed4a | docs: regenerate star history chart | ||
|
Chris LuandClaude Opus 5.5
|
ab355b43b9 |
s3tables: keep the vector-namespace fence on a declare of an existing index (#12293)
s3tables: keep the fence on a declare of an existing index Deciding the existing-index exception before the handler ran left a window: an index deleted between that lookup and the handler's own check would be recreated, unfenced, by a caller without s3vectors:CreateIndex. Drop the up-front exception, so every CreateTable runs fenced and a fresh create is refused at its first write. The engine's declare still passes because the idempotent path writes nothing; for a build whose idempotent path re-applies state already in place, the fence answers an UpdateEntry that leaves the entry exactly as stored without writing. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
e2826c48bd |
s3tables: make the vector namespace of a vector bucket read-only (#12292)
* s3tables: make the vector namespace of a vector bucket read-only S3 Vectors authorizes index operations against the vector bucket policy, but an index is also a catalog table in the bucket's "vector" namespace, and the S3 Tables API only checks s3tables grants. A caller the vector policy denies DeleteIndex could still delete the index, and its data, with DeleteTable. Untrusted writes that name the vector namespace of a LANCE bucket now run behind a fence that passes filer reads and refuses filer writes. The operation's own checks still answer everything short of a write, so a caller without the grant gets its usual denial and learns nothing about the bucket, and a CreateTable that finds the index already there, which is what the vector engine's declare does, still succeeds. A refused write answers 403 AccessDenied. Trusted callers (the gateway, the shell) are unaffected, and so is a namespace named vector in an Iceberg bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: refuse register and deregister in a vector bucket's vector namespace Every catalog write the Lance namespace makes goes through the S3 Tables handler and its vector namespace fence, but register over an existing table at the same location and deregister only add or remove a marker file in the dataset directory. Check the vector namespace before those writes, after the table read that already gates them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: refuse deferred creates in a vector bucket's vector namespace Manager.AuthorizeCreateTable gates the Iceberg create paths that write metadata into the table bucket before the catalog registers the table. Those writes never pass the handler's fence, so the gate applies the same read-only rule once the caller's own create permission is confirmed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: let a declare of an existing index past the vector namespace fence The vector engine declares every index the gateway has just created, and that CreateTable only reaches its idempotent path. Upstream that path writes nothing, so the fence never trips, but a build whose idempotent path repairs table state would refuse the engine. Decide the exception up front instead: a CreateTable naming a finished index in the vector namespace runs unfenced, while a new name or an unfinished create stays fenced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
e118ba7127 |
vector: keep S3 Vectors requests off the master-discovery lock (#12291)
* vector: throttle worker discovery from lookup end cli() set nextResolve before resolve() ran, so a lookup that takes as long as the throttle interval (an unreachable first master costs the full 5s dial timeout) left the next lookup already due when it returned. Discovery then ran back to back and, since it holds d.mu, every vector request queued behind it. resolve() now sets nextResolve when the lookup lands, using the short recheck window only while no client is cached. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: resolve workers outside the discovery lock resolve() held d.mu across the master lookup, so while it waited out an unreachable master every vector request blocked on cli(), even with a healthy worker already cached. The lookup now runs in a goroutine without d.mu, and d.resolving keeps it to one at a time. Callers with a cached client use it while the lookup is in flight; callers without one wait for that lookup and share its result. Throttling, the sticky worker and reset(c) are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: keep the cached worker connection when a redial fails resolve() closed d.conn before dialing the replacement worker. A synchronous dial error then left d.client bound to a closed connection. Requests got codes.Canceled, which never triggers reset, and later refreshes saw the unchanged d.worker and never redialed. The old connection is now closed only after the new dial succeeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
9a09506be6 |
s3vectors: bound ListVectors parameters and lowercase distanceMetric (#12290)
* s3vectors: bound ListVectors maxResults and segment parameters segmentCount had no upper bound, so segmentCount=1000000, segmentIndex=999999 made the segment emulation scan the whole index to return nothing. Apply the AWS ListVectors windows before authorization and any engine call: maxResults 1..1000, segmentCount 1..16, segmentIndex in [0, segmentCount), and segmentCount and segmentIndex given together. The fields become pointers so an explicit 0 is rejected while an omitted one still takes the default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: store and return distanceMetric in lowercase CreateIndex stored distanceMetric exactly as sent, so an index created with "COSINE" came back as "COSINE" from GetIndex and QueryVectors. AWS returns the lowercase enum values. Lowercase the metric when the index is created, and again when the record is read so indexes already stored in mixed case render the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
21fc46692c |
s3vectors: authorize CreateVectorBucket as a vector action, close DeleteIndex race, fold policy service case (#12287)
* s3vectors: authorize CreateVectorBucket as an s3vectors action CreateVectorBucket went through the catalog's CreateTableBucket check, so an identity granted s3vectors:* was denied the create while s3tables:* was allowed it. Check s3vectors:CreateVectorBucket in the gateway, the same way the catalog checks a bucket create, and create the record on the internal manager under the caller's name so the caller stays the owner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: delete only the index record DeleteIndex checked DeleteIndex read the record, checked it is a Lance index, then deleted by name. A table that replaced the index in between was deleted too. Send the checked record's version token, so a replaced record fails the delete with a 409 ConflictException. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: reject s3vectors rules in any letter case validatePolicyDocument kept s3vectors actions and resources out of the shared policy slot with case-sensitive checks, so "S3Vectors:GetIndex" was stored as a rule that never takes effect. IAM service prefixes are case-insensitive; fold case in both checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Anton Rozanov
|
e7f5c6873a |
ec.balance: spread concurrent shard moves over target nodes (#12283)
* ec.balance: spread concurrent shard moves over target nodes The planner gives every freshly encoded volume the same target order, so running each volume's moves in plan order sends all concurrent copies to the same node while the other nodes idle. Pick, among the volumes not already moving, the pending move whose target receives the fewest moves right now. One volume still never has two moves at once, and moves of one shard keep their plan order. * ec.balance: let a move into a node wait for the move of its volume leaving it The planner counts the slot an outgoing shard frees as room for a later incoming shard of the same volume, and the receiver refuses a shard it has no slot for. Make the spread tests deterministic: hold the first moves until all have started, and check that errors of several volumes are all returned. * ec.balance: pin that the spread moves run concurrently and in plan order without parallelism |