Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-11-07 20:47:46 +01:00
Code Issues Packages Projects Releases Wiki Activity
16,091 Commits 227 Branches 365 Tags
195f8156de5bdfd1a71d88aeaac5eb01dbc26345
Commit Graph
11 Commits
Author SHA1 Message Date
Chris LuandClaude Opus 5.5 d0b5529a2d S3 Vectors: weed-vector, weed-worker and the CLI write tables to the filers and volume servers (#12470) 
* sw-client: the filer and volume client library

The client the Enterprise repository's VFS and vector engine use, with
the same API: filer gRPC for entries, listings, volume locations,
assignments and remote caching; volume HTTP, plain or TLS, for needle
bytes; chunk resolution for sparse, overlapping and manifest chunks; a
location cache; and the write side. Enterprise-only protocol fields
(license, conditional mutations, hard links, KV listing) are left out.
Compiled against weed/pb/filer.proto like the other worker crates.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-lance-worker: reads straight from the filers and volume servers

direct_read::DirectReads wraps the object store Lance opens so that
entries, listings and bytes come from the filers (gRPC) and the volume
servers (HTTP, signed with the read JWT), whatever the entry holds:
inline content, chunk manifests, sparse or overlapping chunks, the
filer's volume encryption, the gateway's SSE-S3 (sse.rs, with the KEK
from security.toml or the filer) and objects tiered to remote storage.
Writes still go to the inner store. The code and its tests are those of
the Enterprise repository's engine.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: read tables from the filers and volume servers

The engine asks the masters for the filers (and follows the list), takes
jwt.signing.read.key, grpc.client, https.client and the SSE-S3 keys from
security.toml, and opens every table with DirectReads innermost and the
read cache outside it. --masters is required: the gateway finds the
engine through them and the engine finds the filers. Writes still go to
the gateway.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sse: chunks out of a manifest keep their own SSE-S3 keys

A chunk expanded from a manifest carries its id as fid, so keying the
per-chunk key cache by file_id gave every such chunk the first one's key
and IV. Keyed by the chunk's id however it is carried.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct reads: a silent filer is passed over like an unreachable one

Each filer attempt is bounded, so a filer that accepts the connection and
never answers does not hold the read or the startup lookup.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sw-client: carry the assignment's fsync to the volume upload

The filer's storage rule may want a needle on disk before the volume
answers; the assignment says so and the upload now asks for it, as the
Go uploader does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: argument tests pass the required masters; connect timeout for filers

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct reads: a failed read is tried once more against fresh locations

A fetch that fails forgets the volume locations it was made against; the
request now looks them up again and retries instead of failing on a
volume that moved.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sw-client: an exclusive create that finds its entry, and needle deletion

create_entry marks ENTRY_ALREADY_EXISTS with EntryExists, so a caller can
tell a lost conditional create from a failure; the transport deletes a
needle whose entry was never created.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sse: encode chunks as the gateway stores them

EntryEncoder writes SSE-S3 under a fresh object key wrapped by the KEK,
serialized with the IV, key id and commitment the gateway's reader
expects, each chunk under the IV at its offset; and the filer's volume
encryption under a per-chunk key.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: writes, deletes and table removal without the gateway

A put uploads its chunks to the volume servers the filer assigns, at
the filer's chunk size, encoded as the bucket and cluster ask, and
creates the entry the gateway would: md5, mime, size, the ETag
attribute, the SSE-S3 attributes. A conditional create is the filer's
exclusive create, so Lance's manifest commit conflicts the same way;
the needles of a create that failed are discarded. A multipart upload
turns parts into chunks and derives its ETag from them. Deletes remove
the entry with its data, and remove_table a table directory. Copy and
rename, which Lance does not use on S3, are not offered.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector.proto: Table says whether its bucket encrypts with SSE-S3

The engine writes the table's files itself, so it needs what the S3 write
path knew: whether the bucket declares SSE-S3. storage_options go: the
engine reaches no S3 store.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: send the engine the bucket's SSE-S3 setting

The rule the S3 write path applies to a Lance bucket's files, factored out
of DeclaredBucketEncryption, decides it from the vector bucket's record.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: every table operation goes to the filers and volume servers

Opens, index creation and drops all go through the direct store, so the
engine has no S3 client of its own left to protect: --storage-option,
the endpoint pin and the request-option allowlist go, and lance's own S3
store is built unsigned and never dialed. The bucket's SSE-S3 setting
comes with the table; -encryptVolumeData encrypts chunks for the volume
servers as the gateway's flag does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors tests: the engine takes no credentials

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors e2e: the engine writes what the gateway would have

Every leg checks that the entries the engine wrote carry the gateway's
ETag attribute, the hardened leg that their chunks are encrypted for the
volume servers, and a new test that an SSE-S3 bucket's files are written
under keys the gateway unwraps when it serves them over S3.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* cluster plumbing moves to the worker crate for the engine, worker and CLI

security.toml, the gRPC TLS connector and acceptor, the master dialer,
the filer discovery and the storage connection built from them now live
in weed-lance-worker (cluster, security, grpc_tls), where the worker and
the CLI can use them too. The server crate re-exports security and tls
under their old paths.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker, weed-vector-cli: tables through the filers and volume servers

dataset::Storage replaces the S3 credentials every job and the CLI
carried: a table is opened and written through the cluster's storage,
encrypted as its bucket declares (read from the bucket's record on the
filer), and dropped there. Both take --masters and security.toml like the
engine; --access-key and --secret-key go, and with them the catalog's
x-api-key credential, which the cluster token replaces.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: the cluster token alone authenticates a cluster process

The engine, the maintenance worker and the CLI hold the cluster's filer
signing key and reach a table's files themselves. A request presenting
the cluster token and no other credential is theirs: it is authenticated
by the token, runs through the trusted manager, and is vended no S3
credentials.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* worker chart, README and test design follow the worker's new arguments

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors tests: the worker and the CLI reach the cluster through the masters

The cluster gets a filer signing key, which the catalog authenticates
the worker and the CLI by; both take --masters in place of S3 keys.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: encrypted chunks each start their own CTR stream

The gateway encrypts a multipart part from the IV derived for its offset,
from position zero; the encoder encrypted from the exact byte offset of
the object's stream instead, which agreed only at block boundaries. A
part of 3,000 bytes read back wrong; now tested.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: keep what a lost reply saved, discard what a failed put left

A create the filer saved but did not answer, or an exclusive create
retried onto this very upload, keeps its chunks and counts as written;
the needles of a put that failed midway are discarded. The report prints
reads as reads.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: the security.toml it is given also holds its signing key

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* worker chart: the Lance worker mounts grpc.client's certificate

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct reads: the filer deadline bounds a listing page, not the listing

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: security.toml booleans read as viper reads them

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: a filer the master lists badly does not hide the others

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: register with the masters only once ready to serve

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sse: every chunk after the first is written under a key of its own

Chunks of one object shared a key with IVs derived from their offsets, so
two chunks meeting inside a cipher block shared keystream. The first
chunk stays under the object's key and IV, which is how the gateway reads
an object of one chunk; every other chunk gets a fresh key and IV, as the
gateway's multipart parts have and as it reads an object of several.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* worker chart: the Lance worker is given the filer group

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors e2e: the gateway serves engine-written SSE-S3 files of one chunk and of several

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sw-client: the HTTP transport asks the volume to fsync when the assignment does

write_synced fell back to the trait default, which refuses fsync, so every
upload under a fsync storage rule failed before it was sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: https.client.ca alone vouches for volume servers

With a CA configured, reqwest still trusted the system roots next to it;
weed processes trust only the configured pool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: an empty volume range is read without a request

A zero length underflowed the Range end, and a range past u64::MAX
overflowed it; the first now returns no bytes, the second an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: a whole-needle answer to a ranged read is cut to the range

A server or proxy that ignores Range answers 200 with the whole needle;
the read returned its first bytes in place of the ones asked for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: a read size past u64::MAX reads to the end of the file

offset + size overflowed before the clamp to the file size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: an SSE-encrypted manifest is refused without a decoder

Only filer-encrypted manifests were; an SSE one went to the protobuf
decoder as ciphertext.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: an empty write leaves the staged file size alone

A zero-byte write past the end grew the size to its offset, as a
zero-byte pwrite does not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: the chunk cache keeps no empty chunks

They add no bytes, so nothing evicted them and the map grew with every
distinct empty needle, even at capacity zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: flushing with a zero chunk size is an error, not a panic

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: an empty master list fails at startup

"" split into one empty address that passed the check, and the engine
then waited two minutes for a filer before a generic error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance cluster: an empty master list fails at startup

Cluster::new took "" as one master, so weed-vector, weed-worker and the
CLI waited two minutes for a filer before a generic error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage: a failed create keeps its chunks when the filer cannot be asked

The lookup after a failed create is what tells a lost reply from a
refusal; when it fails too, the entry may be saved, so its chunks stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: a gateway-described table takes the shared unsigned store options

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector.proto: give sse_s3 a new field number and reserve storage_options

Field 4 held the storage_options map on master; a gateway from that build
would send a length-delimited value the engine cannot decode as a bool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector-cli: open s3:// URIs on the cluster's storage

A table named by its s3:// location skipped the store wrapper and opened
Lance's own S3 store, which has no credentials since the fallback flags
were dropped. It now opens like a namespace table and needs --masters;
local paths keep the plain builder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: an index id is plain path components, and drop trusts describe()

`resolved()` now refuses an id whose part is empty, `.`, `..` or holds a
`/`, since the colocated path would land elsewhere on the filer; and
`drop_index` no longer repeats the colocation check for the location
describe() already enforced for the configured schemes, which made a drop
on a memory:// table fail in the scheme test configuration.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: a file delete takes only its entry; a table path is plain names

The store's delete of one file no longer asks the filer for a recursive
delete, so a path that names a directory does not take its content with
it, and `remove_table` refuses a location whose bucket or path component
is empty, `.`, `..` or holds a separator, which the filer would resolve
outside the table.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: chunks are assigned to the bucket's collection

The filer assigns a chunk's volume from the collection the caller names;
the gateway names the bucket, prefixed by the filer group when the
cluster has one, so a plain assign landed the engine's chunks in the
default collection. The store now asks for `<bucket>` or
`<group>_<bucket>`, with the group taken from the cluster connection.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: a put's content type becomes the entry's mime

Lance sets no content type today, so every entry was written as
application/octet-stream; a put that does carry one now creates the
entry with it, as the gateway does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: an upload whose create went unanswered is done with

A create the filer took and never answered (a timeout, or a transport
that reports the filer unavailable) may still be applied, so its chunks
are kept, as they already were for a create whose reply was lost. The
upload then records that it is done: a second completion is refused
instead of creating an empty entry, and an abort discards nothing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: an upload dropped unfinished discards its chunks

A request cancelled mid-write drops its multipart upload with neither
complete nor abort called, leaving its needles on the volume servers.
Dropping an upload that still holds chunks now discards them on the
runtime; a completed upload, or one whose completion is in doubt, holds
none.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: a cluster process registers and deregisters in the vector namespace

The Lance catalog ran a cluster-token caller through the trusted manager
for every operation but the vector-namespace write check of register and
deregister, which still used the caller-checked manager and answered 403
for an index of a LANCE bucket. One `manager(r)` now picks the manager
for all three.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: the worker's flags take two dashes

The worker parses its flags with clap and, unlike the engine, does not
accept Go-style single-dash long flags, so the harness's hardened worker
leg and the README's `-encryptVolumeData` did not start it. The README's
commands also gain the `--masters` the worker now requires.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: the CLI takes --encryptVolumeData

An index build or optimize from the CLI wrote its chunks unencrypted for
the volume servers unless the filer itself asked for the cipher; the CLI
now takes the flag the engine and the worker have.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: a bucket's encryption is read when a table is opened

The setting was cached for a minute, so a bucket recreated with SSE-S3
within that minute had its first tables written plain, and a record that
could not be decoded was read as "no encryption". A table open now reads
the bucket's record from the filer, once per open, and an unreadable
record is an error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: a removed table's files leave the object cache

`remove_table` asked the cache to forget everything under the table, but
the cache was built without invalidation closures, so the call was refused
and its error dropped: a data file read before the drop was still served
afterwards. The cache now supports the invalidation, and a refusal is an
error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: an abort keeps the chunks it could not discard

`discard_all` stopped at the first chunk a volume server would not give
up and dropped the rest from the upload's state, so a retried abort found
nothing to do and the needles stayed. Every chunk is now attempted, the
ones left are kept for the next abort or the drop, and the upload can no
longer be completed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: a chunk write that fails gives its needle back

When the volume server took the chunk but its answer was lost, the needle
stayed under a file id nothing would ever reference. The assignment is
known before the write, so a failed write now discards it, best effort,
before the error is returned.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: the live worker tests require the cluster's masters

With WEED_LANCE_NAMESPACE set and WEED_MASTERS not, the tests seeded their
tables through lance's own S3 store, which reaches nothing, and failed
before a job ran. The missing variable is now named outright.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: only the filer's missing-entry error means not found

Matching any "not found" text read failures such as an unresolved remote
storage as a missing entry, so a delete or a listing reported success over
data that was still there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage test: compare the chunks' SSE-S3 IVs

Each chunk's metadata wraps its key under a fresh nonce, so the metadata
differed even if the key and IV were reused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage: a filer passed over unanswered keeps the outcome open

When one filer timed out on a create and the next refused it, the refusal
came back without the no-answer mark, so the put discarded chunks the
first filer may still reference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance test: start every concurrent writer before awaiting them

The lazy map spawned each writer only after the previous one finished, so
the commits never overlapped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage: count failed uploads and deletes

The storage report counted failed reads and lookups but not a volume
upload, an object delete or a table removal that failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* S3 Vectors tests: run weed-vector-cli as the cluster's processes run

The CLI ran without the fixture's environment and outside its directory,
so under VECTOR_READ_JWT it could not read the volumes and under
VECTOR_TLS it found no security.toml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage: a filer that fell away leaves the outcome open too

An unavailable error (connection reset, deadline, transport failure) can
arrive after the filer took the request, so a refusal from the next filer
no longer settles a create the first one may still apply: the chunks are
kept, as after a timeout.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: a chunk write abandoned in flight gives its needle back

The needle is assigned before the write, so a request cancelled while the
write is in flight left it on the volume server with nothing to name it.
`Source::upload` is now `assign` and `write`, and the upload holds the
assignment across the write in a guard: a write that fails discards it
and one abandoned discards it on the runtime, while a write that returns
hands it to the entry.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance tests: sign for the namespace as the worker does

The live tests reached the namespace anonymously once the S3 keys went,
and an anonymous caller lists no tables, so detection proposed nothing.
They now take the cluster key from where the worker reads it
(WEED_JWT_FILER_SIGNING_KEY or security.toml) and log at info, so a table
a handler skips says why in the harness output.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3tables lifecycle test: run the Lance worker against the cluster

The worker leg of TestLanceTableLifecycle still handed the Rust tests S3
credentials and no masters. The mini cluster now has a cluster key, and
the tests get WEED_MASTERS and that key; passes with
WEED_LANCE_MAINTENANCE=worker (3 fragments became 1, 4 versions removed).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: name no collection outside a filer group, as the gateway

Without a filer group the gateway leaves the collection to the filer, so a
filer.conf rule for the bucket's path picks it; naming the bucket
overrode that rule, and the chunks landed where a bucket delete does not
look.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage: a read reports the entry's content type

A put now stores its ContentType as the entry's mime, but get_opts
returned no attributes, so the type did not come back on a read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-11-07 10:51:52 -08:00
Chris LuandClaude Fable 5.1 2fe480a6e2 S3 Vectors: the gateway resolves the table it sends to weed-vector (#12468) 
* lance catalog: in-process DescribeTable for trusted callers

DescribeTable and DescribeTableInternal resolve a table's location and
storage options without the HTTP layer; the internal form runs the lookup
through a trusted manager so the caller's catalog grants do not gate a
resolution its own permission already authorized. Same code as the
Enterprise repository.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector.proto: Table carries its location and storage options

The gateway resolves the table it addresses, so the engine can open it
without a describe round trip of its own. Both fields are optional; an
empty location means the engine looks the table up as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: test the in-process describe

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: resolve the table before calling the engine

The gateway describes the table in process, through the trusted catalog
manager, and sends its location and storage options with every engine
call. Storage options are vended for the caller's principal, as the
Enterprise router does for queries. A catalog failure answers with the
code the engine would have given.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: open a table at the location the gateway sent

A request whose table carries a location is opened there, with the
storage options the gateway vended over the engine's fallback; the
catalog is described only for a table that comes without one, as from
an older gateway. dataset::open_at and dataset::drop_at take a
description the caller already holds.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors e2e: run the engine without a catalog

VECTOR_ENGINE_NO_CATALOG=1 gives the engine an unreachable catalog URL;
every CI leg sets it, so the suite proves the gateway resolves each table
it sends.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: an internal describe may write a vector index

With credential vending on, the options vended to the engine followed the
caller's s3tables grants, which never allow writing the vector namespace,
so every write the gateway sent would have run with read-only credentials
and reads would have lost their version lease. The engine is that
namespace's writer and the caller's vector permission authorized the
operation, so an internal describe of a vector table vends write-capable
options; other namespaces keep the UpdateTable check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: resolve tables for the engine only while the catalog runs

With -port.lance=0 the gateway has no catalog to resolve through; the
engine then keeps looking tables up itself, as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: a deregistered table describes as NoSuchTable

So the gateway answers NotFoundException for it, not InternalFailure.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: trust a sent location only for a colocated vector index

A caller that reaches the engine could otherwise name any storage the
engine's credentials cover. A location is taken from the request only
when the id names a vector index and the location is its colocated path,
which the engine writes already; anything else is described through the
catalog as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: scope the read cache by storage endpoint

The cache was keyed by bucket and path alone, so the same path reached
through another endpoint would have shared entries. Each open wraps the
store with the cache scoped to the endpoint its options name.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep a configured endpoint over one a request names

A caller that reaches the engine could otherwise send its fallback
credentials to an endpoint of the caller's choosing. When the engine's
--storage-option names an aws_endpoint, a table whose request names
another one is described through the catalog instead.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: use request options only where credentials cannot leave

A request's storage options are taken only when they hold the keys a
gateway's describe produces, so a proxy or an endpoint alias cannot route
the engine's credentials elsewhere, and an endpoint only when it is the
configured one or the engine holds no credentials to send. Anything else
sends the table to the catalog.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors e2e: give the engine its endpoint with its credentials

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: a drop checks the request's options like an open

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: an unpinned endpoint needs the request to sign for itself

Without configured credentials the S3 store still signs with the AWS
default chain, so an endpoint from a request is taken only when the
request skips signing or brings its own credentials.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: a trailing slash does not unpin the endpoint, and say so

A configured aws_endpoint written with a trailing slash matched nothing
the gateway sends, so every table fell back to the catalog in silence.
The comparison ignores the slash, and a fallback for a resolved table is
logged at debug.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: resolved by the gateway, or refused

The engine no longer has a catalog: --namespace and --filer-signing-key
are gone, a request whose table names no location is InvalidArgument, and
one whose location is not the index's own or whose storage options the
engine's credentials must not be used with is PermissionDenied. A drop
removes the colocated path it was given.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: the catalog resolves vector tables whether or not its port is served

newLanceServer builds the Lance catalog for the S3 Vectors API; the
listener on -port.lance is optional.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors tests: start the engine without a catalog

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: the aux listener test builds the catalog before serving it

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: an unpinned endpoint takes only credentials the request signs with alone

Lance fills an option a request leaves out from the engine's environment,
under any of object_store's names for it (TOKEN, SESSION_TOKEN, ...), and
asks the AWS default chain when the key pair is incomplete. A request that
named its own endpoint with only an access key id, or a key pair while the
engine's environment held a session token, would have sent the engine's
credentials there. Such a request now needs a full key pair, and its own
session token whenever the environment would supply one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: accept --namespace and --filer-signing-key, ignored

Both flags shipped in 4.52 and are documented, so dropping them made an
existing command line exit at startup with an unexpected-argument error.
They are kept hidden and ignored, with a warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors design: the engine is started without a catalog

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: the trust check takes the environment it reads

The options test read the process environment through signs_alone, so a
host exporting AWS_SESSION_TOKEN or TOKEN failed it. The caller now
passes the variable names in; describe passes the engine's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: log both endpoints when a request's options are refused

The gateway turns the PermissionDenied into a bare AccessDenied and logs
nothing, so a pinned aws_endpoint written in another form than the one the
gateway sends (host case, a default port, an IP for a name) failed every
call with no trace of which value differed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: the Lance catalog sets the credential role whether or not Iceberg runs

-iceberg.credentialRole reached the S3 server only while the Iceberg
listener started, so with -port.iceberg=0 the Lance catalog installed its
vendor but vending stayed off, and the vector engine got no credentials for
the tables the gateway resolves. Both catalogs now set the role through
one helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: set the credential role once, before either catalog serves

Both catalogs wrote the role as they were built; the second write landed
while the first was already serving requests that read it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-11-06 19:39:09 -08:00
Chris LuandClaude Fable 5.1 e465880ee8 weed-worker: build the vector index of S3 Vectors tables past a row threshold (ANN phase 2) (#12413) 
* seaweed-vector: build and extend the vector index

weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector-cli: index build and index optimize

Logs go to stderr so stdout stays the JSON a caller parses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_INDEX builds the index before the timed queries

The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: probe a tenth of the index's partitions, more under a filter

A query that names no probe count gets one from the index's partition
count. Lance searches at least that many partitions, and under a prefilter
keeps going until it has k rows; without one it stops there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: --nprobes overrides the probe policy

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that a build takes the table's metric

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make clean removes weed-vector-cli

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: measure recall with vectors the table does not hold

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep the read cache's doc comment on with_read_cache

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold

Detection keeps the tables a vector bucket created, with at least min_rows
rows (50,000 by default) and no index on their vector column. Execution
builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: optimize only the indices on the vector column

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: resolve the query metric with index::distance_metric

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test the L2 metric under the name CreateIndex stores

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: build the index through the admin server and the Lance worker

VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new
test sets the job's threshold, runs it through the plugin API, checks
coverage and recall, and that a row written after the build is found.
CI gains the worker leg for both volume servers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: open a table inside a caller's Lance session

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep opened indices between queries in one session

Every open used a fresh Lance session, so the index and metadata caches
died with the request and each query loaded the IVF index again.
--index-cache-mb sizes the shared cache, 512 MiB by default.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the index test draws embedding-like vectors

Independent Gaussian vectors have no locality for an IVF index to use, so
recall through one only tracks the fraction of partitions probed; points
around shared centres behave like embeddings. The test must also run
before the one that stops the engine.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that compaction drops the deleted key from tied hits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test that an append-only optimize serves the new row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: record PERF_NOISE in the perf report config

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: a filtered query widens to four times the probes, not without bound

With no upper bound Lance prepares every partition for the late search
even when the minimum probes already hold k rows, and a filtered query
through the index cost as much as the exact scan. A query that names its
own probe count gets exactly that count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: report recall through an index without asserting it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a hit tying the k-th best score toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: lance_vector_index runs in the Lance lane

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: check a table's index before counting its rows

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make build and clean cover the worker leg's binaries

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: allow float32 rounding at the recall cutoff

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a repeated hit key once toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: assert every filtered hit passes the filter, indexed or not

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: forget the shared session when a table was dropped and created again

The session keys index lists and transactions by dataset version, and a
table created again at the same location restarts its versions. An open
that finds an older version than the newest seen at that URI, or the same
version committed later, replaces the session and opens again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: test an open after another engine created the table again

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: replace the session in the same lock that records a recreated table

A second open of the recreated table could see the new version recorded
and the old session still shared, and keep the dropped table's index list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: skip a vector table whose rows cannot be counted

Detection returned on the first table whose count_rows failed, so one
unreadable table kept every other table from being proposed. It now skips
that table like an open or index failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: panel for vector indexes the Lance worker built

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: size default probes from the largest index on the column

A sharded build gives the column one index name per shard. The first
name alone could be a small shard, and its probe count would then be
spent on a larger one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: skip the probe lookup when a query is given no segments

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: open every table in its own Lance session again

One session shared by every open keeps a table's index loaded between
queries, but Lance keys what it caches by location and version, so a
table dropped and created again at the same location could be served
the old table's index list. Telling the two apart from version and
commit time needs a map that only grows, a retry loop and clocks that
agree across engines. A private session per open, as before, has no
such state. Measured at 50k x 384 through an index, eight clients: 164
queries/s with the shared session, 137 without; one client is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: reject --nprobes 0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: read the partition count from the opened index

Lance's index statistics took 3.6 ms a query on a 200k-row table with
447 partitions. Opening the index and asking it takes 0.4 ms, and the
scan that follows finds the index already open.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* ci: give the S3 Vectors worker legs the vendored protoc

The worker workspace has no protoc of its own for lance's build
scripts, so the worker legs failed at their first build.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: give a vector index build an hour before the admin times it out

The job set no execution timeout, so the admin's default of 90 seconds
applied, and a build that trains on a large table would be reported as
failed and retried while it was still running.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: lance_vector_index proposes at most the admin's max_results

Every qualifying table went into one message however many there were.
The detector stops at the limit the request carries and sets has_more,
as the Go detectors do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a vector index job re-checks the table it was proposed for

A table emptied since detection completes without a build, and one
without a vector column fails instead of reporting it already indexed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a vector index run window as long as one build

An attempt is capped at the end of the job type's run window plus the
drain grace, and the default window is 30 minutes, so a build could be
cut off before its own hour was up. The window is an hour too, as the
Go iceberg handler sets both.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-11-04 22:16:31 -08:00
Chris LuandClaude Fable 5.1 f0f0ba85ae seaweed-vector: build a vector index from the CLI and measure it in the perf suite (ANN phase 0) (#12411) 
* seaweed-vector: build and extend the vector index

weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector-cli: index build and index optimize

Logs go to stderr so stdout stays the JSON a caller parses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_INDEX builds the index before the timed queries

The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that a build takes the table's metric

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make clean removes weed-vector-cli

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: optimize only the indices on the vector column

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: resolve the query metric with index::distance_metric

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test the L2 metric under the name CreateIndex stores

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that compaction drops the deleted key from tied hits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test that an append-only optimize serves the new row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: record PERF_NOISE in the perf report config

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: report recall through an index without asserting it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a hit tying the k-th best score toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: allow float32 rounding at the recall cutoff

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a repeated hit key once toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: assert every filtered hit passes the filter, indexed or not

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-11-04 19:25:38 -08:00
Chris LuandClaude Fable 5.1 de4a71665e test/s3vectors: wait for the S3 port before handing out the cluster (#12383) 
The fixture waited for the Lance catalog port and the engine, but weed s3
opens its HTTP port last, about half a second after the catalog here. The
open cluster's engine now registers within that window, so
test_engine_without_credentials met a connection refused on the go and rust
single legs. Wait for the S3 port like the other listeners.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-11-03 14:08:42 -08:00
Chris LuandClaude Opus 5.5 5063aa9be2 weed-vector: check master certificates against the server allowlists, read -config_dir (#12374) 
* weed-vector: look for security.toml in -config_dir first

weed processes search the directory -config_dir names before ., so a
cluster keeping security.toml there left the engine in plaintext.
--config-dir (also -config_dir, WEED_VECTOR_CONFIG_DIR) does the same.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: dial the masters the way weed clients do

The master's certificate must chain to grpc.ca and is not checked against
the host dialed, as weed clients verify (advancedtls.CertVerification), so
masters sharing one workload certificate, or dialed by IP, register as
they do from weed. Instead, grpc.client.master_allowed_server_commonNames,
grpc.client.allowed_server_commonNames or grpc.allowed_server_commonNames
(and the wildcard domain likewise) restrict which server identities are
accepted: the common name or a DNS SAN, or an exact URI SAN, as
security.Authenticator.AuthenticateServer does. These used to be refused
at startup; --tls-server-name is gone with the host check.

The TLS runs in a connector that reads the credentials asynchronously at
every dial, so the connect timeout still covers the reads, the TCP connect
and the handshake together. An IPv6 master's brackets are stripped before
the dial.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: the hardened leg allowlists server identities for grpc.client

The gateway and the engine then accept only servers named localhost, which
exercises the engine's allowed_server_commonNames check on the master dial.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: parse the refresh interval with time.ParseDuration's arithmetic

The interval was summed in floating-point seconds, so 0.9ns rounded up to
1ns where Go truncates it to 0 and falls back to the 5h default, 0.3s was
off by a nanosecond, and 3000000h was accepted where Go rejects anything
past the longest time.Duration. Each part is now an integer count of
nanoseconds plus the fraction Go adds, with Go's overflow bounds, and the
value is no longer trimmed, as Go does not trim it either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: test a master dial whose credential read stalls

The bounded-dial tests passed no credentials or a prebuilt endpoint, so
moving the credential read out of the connect timeout would have failed
none of them. The new test points grpc.ca at a FIFO nobody writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: dial a closed port in the stalled credential read test

Against a silent listener, a dial that never read the credentials timed
out with the same message, so the test passed without the read. Nothing
listens on the port now, so only the stalled read can time out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: test that Security::load searches the config dir

Only candidates() was tested, so load dropping config_dir on the way
would have passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: expand a leading ~ in -config_dir like weed's ResolvePath

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: hold the port in the stalled credential read test

A released port can be taken before the dial, and a dial that skipped the
read would then time out and pass. A listener that drops every connection
keeps the port and still fails such a dial at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-11-03 12:41:02 -08:00
Chris LuandClaude Opus 5.5 eeb8aefe70 weed-vector: gRPC TLS to the masters and for the gateway (#12368) 
* weed-vector: read the cluster's security.toml

The engine finds security.toml where weed processes do (., ~/.seaweedfs,
/usr/local/etc/seaweedfs, /etc/seaweedfs) with WEED_* environment
variables over it, and builds gRPC client and server credentials by the
rules of security.LoadClientTLS and LoadServerTLS: a dedicated client
certificate is preferred, both halves of a pair must be set, and grpc.ca
is required once either is.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: gRPC TLS to the masters and from the gateway

The masters are dialed with [grpc.client]'s credentials when the cluster
has them, the master's certificate checked against the host dialed or
--tls-server-name. The engine's own listener takes its identity from
[grpc.vector] and then requires a client certificate chaining to grpc.ca,
which is how the gateway, dialing with [grpc.client], reaches it. The
rustls provider is pinned before the first TLS use, as the other binaries
linking both aws-lc-rs and ring do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* scaffold: document [grpc.vector]

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: put every gRPC hop under mutual TLS in the hardened leg

VECTOR_TLS=1 now also writes [grpc.master], [grpc.volume], [grpc.filer],
[grpc.s3], [grpc.client] and [grpc.vector], and probes the master's and
the engine's gRPC ports: plaintext HTTP/2 and a client without a
certificate are refused, the client certificate gets a SETTINGS frame.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: give the test volume server 64 volume slots

Every collection grows 7 volumes and test_vector_bucket_encryption opens
two buckets while the previous test's bucket is still being torn down, so
16 slots were marginal: the Go legs failed CreateIndex now and then with
"failed to find writable volumes" and master.log showing "create 7
volume, created 4: Not enough data nodes found!".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: reload the listener's certificates and enforce allowed_commonNames

The listener now runs the way weed servers run theirs. The certificate, key
and CA files are re-read every WEED_TLS_CERT_REFRESH_INTERVAL (weed's
default of 5h) and on SIGHUP, and new handshakes use the changed files
without a restart; a file that no longer parses is logged and the working
configuration stays. [grpc.vector] allowed_commonNames and
grpc.allowed_wildcard_domain restrict which client certificates are served,
checked against the subject common name as security.Authenticator and the
Rust volume server's CommonNameVerifier do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* scaffold: allowed_commonNames and the refresh cadence for [grpc.vector]

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the engine's allowlist keeps an outsider certificate out

The hardened leg sets [grpc.vector] allowed_commonNames = "localhost" and
probes the engine with a second certificate the same CA signed for
CN=outsider: the handshake must be refused, while the gateway's certificate
still gets an HTTP/2 SETTINGS frame.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: take the master's TLS name from the parsed endpoint

A master given without a port (https://master.example) had the whole URL
checked against its certificate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: bound the master TLS handshake

connect_timeout covers only the TCP connect, so a master that accepted the
connection and never answered the handshake held up the other masters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: look up security.toml keys the way viper does

An empty WEED_* variable no longer hides the file's value, and keys match
regardless of case, as for weed processes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: refuse master allowlists it cannot enforce

weed checks a dialed server against allowed_server_commonNames and
allowed_server_wildcard_domain on top of grpc.ca. The engine's master dial
accepted every certificate the CA signed even when one was set; it now
fails at startup and names the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: read the whole frame header in the gRPC TLS probe

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: probe the volume, filer and S3 gRPC listeners too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read the master dial credentials only when given masters

A standalone engine never dials with grpc.client, so its settings no longer
stop it from starting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: run each listener TLS handshake on its own task

The accept loop awaited every handshake before accepting the next
connection, so a peer that connected and sent nothing stopped every
gateway connection after it. Handshakes now run concurrently, each under a
10s timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read the master dial's TLS files at every dial

The client certificate, key and CA were read once at startup, so after the
files rotated a reconnect still presented the old pair. Each dial now reads
them, as the listener re-reads its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read every unit Go durations accept in the refresh interval

WEED_TLS_CERT_REFRESH_INTERVAL values in ns, us or a leading + fell back
to five hours in the engine while weed servers honored them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: fall back on a refresh interval too long to hold

An oversized WEED_TLS_CERT_REFRESH_INTERVAL panicked the engine at
startup; weed servers reject it and keep the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: refuse allowed_commonNames without the listener's certificate

With [grpc.vector] allowed_commonNames set but no cert and key, the
listener served plaintext and checked no client at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read the master dial's credentials inside its connect timeout

The CA, certificate and key were read synchronously before the timeout
started, so a stalled credential path held up every other master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: let a set key win over an empty spelling of it

With allowed_server_commonNames = "" beside allowed_server_commonnames
= "master", the lookup stopped at the empty exact match and the
allowlist went unnoticed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-11-03 11:42:49 -08:00
Chris LuandClaude Opus 5.5 e5bc658eb7 test/s3vectors: a hardened cluster with TLS volume servers, read JWTs and encrypted chunks (#12356) 
* test/s3vectors: a hardened cluster, with TLS volume servers, read JWTs and encrypted chunks

Three switches on the test cluster: VECTOR_TLS=1 puts the volume servers
behind HTTPS requiring client certificates and enables https.client for
every process; VECTOR_READ_JWT=1 gives every process a jwt.signing.read.key,
so volume reads need a token; VECTOR_ENCRYPT_VOLUMES=1 starts the gateway
with -encryptVolumeData. The integration matrix gains a hardened leg per
volume server with all three on.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the hardened cluster proves its enforcement

Once the gateway is up, the harness checks that the hardening holds, not
only that it is configured: a TLS volume server refuses plain HTTP (Go
with HTTP 400, Rust with a TLS alert) and a connection without the client
certificate, and answers with it; under a read key an unsigned read of a
real chunk, written through the gateway and located through the filer,
gets 401.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the hardened cluster proves its chunks are encrypted

With VECTOR_ENCRYPT_VOLUMES the probe reads the stored chunk straight
from the volume server, signing the read when a read key is set, and
requires a recorded cipher key and bytes that differ from the plaintext.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: the signed probe read must return the chunk

An empty 200 passed both branches; an unencrypted read now has to return
the bytes written, and an encrypted one has to return something else.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: a read key in the caller's environment does not leak into the cluster

Only VECTOR_READ_JWT turns read JWTs on; an inherited
WEED_JWT_SIGNING_READ_KEY made the volume servers refuse the probe's
unsigned read in an encrypt-only run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: quote certificate paths as TOML strings in security.toml

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: keep non-ASCII certificate paths as UTF-8 in security.toml

TOML has no surrogate-pair escapes, and the Rust volume server reads the
quoted value without decoding escapes at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: an encrypted probe chunk must not be a readable gzip stream

The gateway may gzip the body before encrypting it, so a chunk uploaded
compressed but unencrypted would differ from the body and still pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-11-03 04:56:15 -08:00
Chris LuandClaude Opus 5.5 2eb219df61 s3vectors: encrypt the objects of a vector bucket declared AES256 (#12324) 
* s3api: encrypt the objects of a vector bucket declared AES256

CreateVectorBucket recorded encryptionConfiguration sseType AES256 and
returned it, but nothing on the write path read it: the index's data
files, manifests and transactions were stored without SSE.

A LANCE table bucket's record now supplies the bucket's default
encryption whenever the bucket has none of its own, so PutObject,
CreateMultipartUpload and CopyObject apply SSE-S3 to every object the
engine writes. The S3 bucket metadata itself is unchanged.

ICEBERG table buckets keep recording the declaration only: their
catalog reads the manifests clients upload straight from the filer,
which would see ciphertext.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3tables: refuse an AES256 vector bucket on a gateway without an SSE-S3 key

A vector bucket's AES256 is now applied to every index write, and SSE-S3
fails each write when the gateway has no KEK. Creating the bucket there
succeeded and the first CreateIndex came back as InternalFailure.

CreateTableBucket now refuses a LANCE bucket declaring AES256 when the
gateway reports no SSE-S3 key, naming s3.sse.kek and s3.sse.key;
CreateVectorBucket returns it as ValidationException. ICEBERG buckets,
whose declaration is only recorded, and tooling that sets no checker
are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: refuse DeleteBucketEncryption on a bucket's declared encryption

GetBucketEncryption reports a vector bucket's declared AES256, but
DeleteBucketEncryption answered NoSuchBucketEncryptionConfiguration and
the declaration kept applying. The declaration is fixed when the bucket
is created, so the delete now returns MethodNotAllowed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): keep an inherited WEED_S3_SSE_KEK out of the test cluster

The gateway refuses to start SSE-S3 with both s3.sse.kek and s3.sse.key
set, so a developer shell exporting WEED_S3_SSE_KEK made the AES256
bucket test fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-01 23:57:40 -08:00
Chris LuandClaude Opus 5.5 d72f761d83 seaweed-vector: work on its defaults, and without identities or credentials (#12323) 
* seaweed-vector: default --namespace to the gateway's Lance port

The old default, http://localhost:8888/lance, is a path nothing serves:
the Lance Namespace listens on its own port, 9101 by default on weed s3,
weed server, weed filer and weed mini. With it, the filer answered the
engine's catalog calls with a 500 and every CreateIndex failed with
InternalFailure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: default --listen to a port no SeaweedFS server takes

127.0.0.1:8080 is the volume server's default port, so weed-vector next
to weed server exited with "Address already in use (os error 98)". 8090
is not among the defaults of master, volume, filer, S3 and its Iceberg
and Lance catalogs, IAM, WebDAV, SFTP, the message queue broker and
agent, the admin server or weed mini, nor any of their gRPC ports
(HTTP + 10000).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance: take any key while the S3 server runs without identities

Without identities the S3 port accepts any signature and treats the
caller as the admin. The Lance catalog still checked an x-api-key or
Bearer token against identities that did not exist, so weed-vector
started with --storage-option credentials on such a server got 401 on
every catalog call. The catalog now skips both checks when S3 auth is
off and leaves the caller to the S3 authenticator, which admits it as
the admin, as it already did for a request with no key. With
identities configured nothing changes: an unknown key is still 401.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance: tell clients to skip signing while S3 auth is off

A client with no credentials of its own could not reach a table's data
on a server without identities, though the S3 port takes unsigned
requests there: lance fell back to the AWS default credential chain and
failed with CredentialsNotLoaded. weed-vector without --storage-option
failed every CreateIndex that way. The catalog now adds
aws_skip_signature to the storage options it hands out while S3 auth is
off, and leaves it out once identities exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: bind the listener before registering with the master

The engine spawned its master registration and then bound --listen, so
an engine whose port was taken could reach the master before it exited:
on weed server with 18080 taken the master logged "vector node
127.0.0.1:18080 added" and removed it in the same millisecond, and the
gateway noticed it. Bind first, and name the address in the error
instead of "transport error".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-worker: keep an unsigned store off the AWS default chain

A store the catalog says takes unsigned requests needs no credentials,
but lance builds the AWS default credential chain for any store without
them, and the chain probes instance metadata on every dataset open.
With no credentials of its own, weed-vector spent a second on each
query (1030-1066 ms against 32-46 ms with --storage-option ones). The
merge of the operator's fallback and the vended options, which the
engine and the Lance worker each did in their own copy, moves into
table_options, and gives such a store empty credentials.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): serve an index with no identities and no engine credentials

A second cluster runs weed s3 without -config and weed-vector without
--storage-option, with no AWS credentials in either environment: weed s3
would turn them into an admin identity, and the engine would find them.
On master its CreateIndex fails with InternalFailure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-01 23:52:30 -08:00
Chris Lu 74f727d987 test: move the s3vectors pytest suite under test/s3vectors (#12230) 
* test: move the s3vectors pytest suite under test/s3vectors

The vectors API tests lived in test/vectors next to the new s3vectors
integration home; the rust-vector-tests workflow's paths and working
directory move with it.

* gitignore: cover the binaries the s3vectors Makefile builds
 2026-10-29 19:34:48 -07:00