mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-07 20:47:46 +01:00
195f8156de5bdfd1a71d88aeaac5eb01dbc26345
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
d0b5529a2d |
S3 Vectors: weed-vector, weed-worker and the CLI write tables to the filers and volume servers (#12470)
* sw-client: the filer and volume client library The client the Enterprise repository's VFS and vector engine use, with the same API: filer gRPC for entries, listings, volume locations, assignments and remote caching; volume HTTP, plain or TLS, for needle bytes; chunk resolution for sparse, overlapping and manifest chunks; a location cache; and the write side. Enterprise-only protocol fields (license, conditional mutations, hard links, KV listing) are left out. Compiled against weed/pb/filer.proto like the other worker crates. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-lance-worker: reads straight from the filers and volume servers direct_read::DirectReads wraps the object store Lance opens so that entries, listings and bytes come from the filers (gRPC) and the volume servers (HTTP, signed with the read JWT), whatever the entry holds: inline content, chunk manifests, sparse or overlapping chunks, the filer's volume encryption, the gateway's SSE-S3 (sse.rs, with the KEK from security.toml or the filer) and objects tiered to remote storage. Writes still go to the inner store. The code and its tests are those of the Enterprise repository's engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: read tables from the filers and volume servers The engine asks the masters for the filers (and follows the list), takes jwt.signing.read.key, grpc.client, https.client and the SSE-S3 keys from security.toml, and opens every table with DirectReads innermost and the read cache outside it. --masters is required: the gateway finds the engine through them and the engine finds the filers. Writes still go to the gateway. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sse: chunks out of a manifest keep their own SSE-S3 keys A chunk expanded from a manifest carries its id as fid, so keying the per-chunk key cache by file_id gave every such chunk the first one's key and IV. Keyed by the chunk's id however it is carried. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: a silent filer is passed over like an unreachable one Each filer attempt is bounded, so a filer that accepts the connection and never answers does not hold the read or the startup lookup. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: carry the assignment's fsync to the volume upload The filer's storage rule may want a needle on disk before the volume answers; the assignment says so and the upload now asks for it, as the Go uploader does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: argument tests pass the required masters; connect timeout for filers Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: a failed read is tried once more against fresh locations A fetch that fails forgets the volume locations it was made against; the request now looks them up again and retries instead of failing on a volume that moved. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: an exclusive create that finds its entry, and needle deletion create_entry marks ENTRY_ALREADY_EXISTS with EntryExists, so a caller can tell a lost conditional create from a failure; the transport deletes a needle whose entry was never created. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sse: encode chunks as the gateway stores them EntryEncoder writes SSE-S3 under a fresh object key wrapped by the KEK, serialized with the IV, key id and commitment the gateway's reader expects, each chunk under the IV at its offset; and the filer's volume encryption under a per-chunk key. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: writes, deletes and table removal without the gateway A put uploads its chunks to the volume servers the filer assigns, at the filer's chunk size, encoded as the bucket and cluster ask, and creates the entry the gateway would: md5, mime, size, the ETag attribute, the SSE-S3 attributes. A conditional create is the filer's exclusive create, so Lance's manifest commit conflicts the same way; the needles of a create that failed are discarded. A multipart upload turns parts into chunks and derives its ETag from them. Deletes remove the entry with its data, and remove_table a table directory. Copy and rename, which Lance does not use on S3, are not offered. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector.proto: Table says whether its bucket encrypts with SSE-S3 The engine writes the table's files itself, so it needs what the S3 write path knew: whether the bucket declares SSE-S3. storage_options go: the engine reaches no S3 store. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: send the engine the bucket's SSE-S3 setting The rule the S3 write path applies to a Lance bucket's files, factored out of DeclaredBucketEncryption, decides it from the vector bucket's record. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: every table operation goes to the filers and volume servers Opens, index creation and drops all go through the direct store, so the engine has no S3 client of its own left to protect: --storage-option, the endpoint pin and the request-option allowlist go, and lance's own S3 store is built unsigned and never dialed. The bucket's SSE-S3 setting comes with the table; -encryptVolumeData encrypts chunks for the volume servers as the gateway's flag does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors tests: the engine takes no credentials Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: the engine writes what the gateway would have Every leg checks that the entries the engine wrote carry the gateway's ETag attribute, the hardened leg that their chunks are encrypted for the volume servers, and a new test that an SSE-S3 bucket's files are written under keys the gateway unwraps when it serves them over S3. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * cluster plumbing moves to the worker crate for the engine, worker and CLI security.toml, the gRPC TLS connector and acceptor, the master dialer, the filer discovery and the storage connection built from them now live in weed-lance-worker (cluster, security, grpc_tls), where the worker and the CLI can use them too. The server crate re-exports security and tls under their old paths. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker, weed-vector-cli: tables through the filers and volume servers dataset::Storage replaces the S3 credentials every job and the CLI carried: a table is opened and written through the cluster's storage, encrypted as its bucket declares (read from the bucket's record on the filer), and dropped there. Both take --masters and security.toml like the engine; --access-key and --secret-key go, and with them the catalog's x-api-key credential, which the cluster token replaces. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: the cluster token alone authenticates a cluster process The engine, the maintenance worker and the CLI hold the cluster's filer signing key and reach a table's files themselves. A request presenting the cluster token and no other credential is theirs: it is authenticated by the token, runs through the trusted manager, and is vended no S3 credentials. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * worker chart, README and test design follow the worker's new arguments Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors tests: the worker and the CLI reach the cluster through the masters The cluster gets a filer signing key, which the catalog authenticates the worker and the CLI by; both take --masters in place of S3 keys. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: encrypted chunks each start their own CTR stream The gateway encrypts a multipart part from the IV derived for its offset, from position zero; the encoder encrypted from the exact byte offset of the object's stream instead, which agreed only at block boundaries. A part of 3,000 bytes read back wrong; now tested. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: keep what a lost reply saved, discard what a failed put left A create the filer saved but did not answer, or an exclusive create retried onto this very upload, keeps its chunks and counts as written; the needles of a put that failed midway are discarded. The report prints reads as reads. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: the security.toml it is given also holds its signing key Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * worker chart: the Lance worker mounts grpc.client's certificate Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: the filer deadline bounds a listing page, not the listing Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: security.toml booleans read as viper reads them Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a filer the master lists badly does not hide the others Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: register with the masters only once ready to serve Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sse: every chunk after the first is written under a key of its own Chunks of one object shared a key with IVs derived from their offsets, so two chunks meeting inside a cipher block shared keystream. The first chunk stays under the object's key and IV, which is how the gateway reads an object of one chunk; every other chunk gets a fresh key and IV, as the gateway's multipart parts have and as it reads an object of several. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * worker chart: the Lance worker is given the filer group Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: the gateway serves engine-written SSE-S3 files of one chunk and of several Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: the HTTP transport asks the volume to fsync when the assignment does write_synced fell back to the trait default, which refuses fsync, so every upload under a fsync storage rule failed before it was sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: https.client.ca alone vouches for volume servers With a CA configured, reqwest still trusted the system roots next to it; weed processes trust only the configured pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an empty volume range is read without a request A zero length underflowed the Range end, and a range past u64::MAX overflowed it; the first now returns no bytes, the second an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: a whole-needle answer to a ranged read is cut to the range A server or proxy that ignores Range answers 200 with the whole needle; the read returned its first bytes in place of the ones asked for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: a read size past u64::MAX reads to the end of the file offset + size overflowed before the clamp to the file size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an SSE-encrypted manifest is refused without a decoder Only filer-encrypted manifests were; an SSE one went to the protobuf decoder as ciphertext. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an empty write leaves the staged file size alone A zero-byte write past the end grew the size to its offset, as a zero-byte pwrite does not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: the chunk cache keeps no empty chunks They add no bytes, so nothing evicted them and the map grew with every distinct empty needle, even at capacity zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: flushing with a zero chunk size is an error, not a panic Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: an empty master list fails at startup "" split into one empty address that passed the check, and the engine then waited two minutes for a filer before a generic error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance cluster: an empty master list fails at startup Cluster::new took "" as one master, so weed-vector, weed-worker and the CLI waited two minutes for a filer before a generic error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: a failed create keeps its chunks when the filer cannot be asked The lookup after a failed create is what tells a lost reply from a refusal; when it fails too, the entry may be saved, so its chunks stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: a gateway-described table takes the shared unsigned store options Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector.proto: give sse_s3 a new field number and reserve storage_options Field 4 held the storage_options map on master; a gateway from that build would send a length-delimited value the engine cannot decode as a bool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector-cli: open s3:// URIs on the cluster's storage A table named by its s3:// location skipped the store wrapper and opened Lance's own S3 store, which has no credentials since the fallback flags were dropped. It now opens like a namespace table and needs --masters; local paths keep the plain builder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: an index id is plain path components, and drop trusts describe() `resolved()` now refuses an id whose part is empty, `.`, `..` or holds a `/`, since the colocated path would land elsewhere on the filer; and `drop_index` no longer repeats the colocation check for the location describe() already enforced for the configured schemes, which made a drop on a memory:// table fail in the scheme test configuration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a file delete takes only its entry; a table path is plain names The store's delete of one file no longer asks the filer for a recursive delete, so a path that names a directory does not take its content with it, and `remove_table` refuses a location whose bucket or path component is empty, `.`, `..` or holds a separator, which the filer would resolve outside the table. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: chunks are assigned to the bucket's collection The filer assigns a chunk's volume from the collection the caller names; the gateway names the bucket, prefixed by the filer group when the cluster has one, so a plain assign landed the engine's chunks in the default collection. The store now asks for `<bucket>` or `<group>_<bucket>`, with the group taken from the cluster connection. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a put's content type becomes the entry's mime Lance sets no content type today, so every entry was written as application/octet-stream; a put that does carry one now creates the entry with it, as the gateway does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: an upload whose create went unanswered is done with A create the filer took and never answered (a timeout, or a transport that reports the filer unavailable) may still be applied, so its chunks are kept, as they already were for a create whose reply was lost. The upload then records that it is done: a second completion is refused instead of creating an empty entry, and an abort discards nothing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: an upload dropped unfinished discards its chunks A request cancelled mid-write drops its multipart upload with neither complete nor abort called, leaving its needles on the volume servers. Dropping an upload that still holds chunks now discards them on the runtime; a completed upload, or one whose completion is in doubt, holds none. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: a cluster process registers and deregisters in the vector namespace The Lance catalog ran a cluster-token caller through the trusted manager for every operation but the vector-namespace write check of register and deregister, which still used the caller-checked manager and answered 403 for an index of a LANCE bucket. One `manager(r)` now picks the manager for all three. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: the worker's flags take two dashes The worker parses its flags with clap and, unlike the engine, does not accept Go-style single-dash long flags, so the harness's hardened worker leg and the README's `-encryptVolumeData` did not start it. The README's commands also gain the `--masters` the worker now requires. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: the CLI takes --encryptVolumeData An index build or optimize from the CLI wrote its chunks unencrypted for the volume servers unless the filer itself asked for the cipher; the CLI now takes the flag the engine and the worker have. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a bucket's encryption is read when a table is opened The setting was cached for a minute, so a bucket recreated with SSE-S3 within that minute had its first tables written plain, and a record that could not be decoded was read as "no encryption". A table open now reads the bucket's record from the filer, once per open, and an unreadable record is an error. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a removed table's files leave the object cache `remove_table` asked the cache to forget everything under the table, but the cache was built without invalidation closures, so the call was refused and its error dropped: a data file read before the drop was still served afterwards. The cache now supports the invalidation, and a refusal is an error. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: an abort keeps the chunks it could not discard `discard_all` stopped at the first chunk a volume server would not give up and dropped the rest from the upload's state, so a retried abort found nothing to do and the needles stayed. Every chunk is now attempted, the ones left are kept for the next abort or the drop, and the upload can no longer be completed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a chunk write that fails gives its needle back When the volume server took the chunk but its answer was lost, the needle stayed under a file id nothing would ever reference. The assignment is known before the write, so a failed write now discards it, best effort, before the error is returned. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: the live worker tests require the cluster's masters With WEED_LANCE_NAMESPACE set and WEED_MASTERS not, the tests seeded their tables through lance's own S3 store, which reaches nothing, and failed before a job ran. The missing variable is now named outright. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: only the filer's missing-entry error means not found Matching any "not found" text read failures such as an unresolved remote storage as a missing entry, so a delete or a listing reported success over data that was still there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage test: compare the chunks' SSE-S3 IVs Each chunk's metadata wraps its key under a fresh nonce, so the metadata differed even if the key and IV were reused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: a filer passed over unanswered keeps the outcome open When one filer timed out on a create and the next refused it, the refusal came back without the no-answer mark, so the put discarded chunks the first filer may still reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance test: start every concurrent writer before awaiting them The lazy map spawned each writer only after the previous one finished, so the commits never overlapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: count failed uploads and deletes The storage report counted failed reads and lookups but not a volume upload, an object delete or a table removal that failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * S3 Vectors tests: run weed-vector-cli as the cluster's processes run The CLI ran without the fixture's environment and outside its directory, so under VECTOR_READ_JWT it could not read the volumes and under VECTOR_TLS it found no security.toml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: a filer that fell away leaves the outcome open too An unavailable error (connection reset, deadline, transport failure) can arrive after the filer took the request, so a refusal from the next filer no longer settles a create the first one may still apply: the chunks are kept, as after a timeout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: a chunk write abandoned in flight gives its needle back The needle is assigned before the write, so a request cancelled while the write is in flight left it on the volume server with nothing to name it. `Source::upload` is now `assign` and `write`, and the upload holds the assignment across the write in a guard: a write that fails discards it and one abandoned discards it on the runtime, while a write that returns hands it to the entry. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance tests: sign for the namespace as the worker does The live tests reached the namespace anonymously once the S3 keys went, and an anonymous caller lists no tables, so detection proposed nothing. They now take the cluster key from where the worker reads it (WEED_JWT_FILER_SIGNING_KEY or security.toml) and log at info, so a table a handler skips says why in the harness output. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3tables lifecycle test: run the Lance worker against the cluster The worker leg of TestLanceTableLifecycle still handed the Rust tests S3 credentials and no masters. The mini cluster now has a cluster key, and the tests get WEED_MASTERS and that key; passes with WEED_LANCE_MAINTENANCE=worker (3 fragments became 1, 4 versions removed). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: name no collection outside a filer group, as the gateway Without a filer group the gateway leaves the collection to the filer, so a filer.conf rule for the bucket's path picks it; naming the bucket overrode that rule, and the chunks landed where a bucket delete does not look. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: a read reports the entry's content type A put now stores its ContentType as the entry's mime, but get_opts returned no attributes, so the type did not come back on a read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandDevin
|
d15396fc4d |
s3tables: scope the anonymous identity's grant on the table catalogs (#12056)
* s3tables: scope the resolved anonymous identity instead of defaulting to allow Catalog authorization fell back to the open default whenever the request resolved to the configured anonymous identity, ignoring its grant. The fallback now applies only when no principal was resolved (zero-config), so anonymous requests are checked against the identity's configured actions like any other principal. The S3 Tables API and the Iceberg and Lance catalogs share this handler, so all three are covered. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: authenticate the catalog clients in the s3tables suites Anonymous calls into the Lance and Iceberg catalogs are now scoped by the anonymous identity's grant, so the suites that used to rely on the anonymous pass authenticate like their real deployments do: x-api-key for the Lance probes and REST drivers, client-credentials OAuth for pyiceberg. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * worker(lance): authenticate namespace calls with the storage credential Anonymous catalog calls are now scoped by the anonymous grant, so a worker that holds an S3 credential sends it as x-api-key on the namespace requests instead of relying on the anonymous pass. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: authenticate the pushdown catalog connection in catalog_lancedb Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: follow catalog redirects only within the same host reqwest forwards custom headers on a follow, so a redirect to another host would carry the x-api-key credential to a server it was never meant for. Keep same-host hops for proxies that normalize the path or upgrade the scheme. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: warn when the catalog api key travels in cleartext The x-api-key pair is a real storage credential: over plain http to a non-loopback host the network can read it. Warn at client build so the operator sees it; https and loopback stay quiet, and the key still sends since an http catalog is a valid deployment. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: require the same origin on catalog redirects A same-host hop can still downgrade https to http and put the api key on the wire; matching scheme, host, and port keeps the credential on the listener it was issued for. * lance: bound catalog redirect hops so loops fail fast The custom redirect policy drops reqwest's default 10-hop limit, so a catalog that redirects back to itself would hold each call open until the request timeout. Keep the bound alongside the origin check. * lance: gofmt the redirect helpers --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
462e4445e2 |
test: skip catalog client images that cannot execute on the host CPU (#12014)
New upstream releases build for x86-64-v3 (AVX2) - ClickHouse since 26.5, plus the DuckDB iceberg extension and pylance wheels - so on a runner whose CPU lacks AVX2 the container dies on SIGILL before the test reaches any SeaweedFS code. Probe each third-party image first and skip on the illegal-instruction exit instead of failing; other errors still surface normally. |
||
|
Chris Lu
|
aeb5db1b13 |
tests: pin weed mini admin ports and disable unused aux services (#11997)
* tests: pin weed mini admin ports and disable unused aux services Every weed mini that does not configure them still binds the fixed default ports for the admin server (12535 + derived 22535 gRPC), its legacy compat listener (23646 + 33646), WebDAV (7333), Lance (9101) and the Iceberg REST catalog (8181). Sequential or concurrent minis on one host then collide on them: the port check warns and shifts, and the admin's late bind can land on a stolen ephemeral port, adding noise and flakiness to test logs (e.g. run 40017181958, job 120659267479). Add testutil.MiniAdminArgs to pin the admin server to an allocated mini-style port (HTTP + port+10000 gRPC, both below the Linux ephemeral port floor since admin binds late) and disable the legacy compat listener. Every test that spawns weed mini now also disables the aux services it does not use: WebDAV, Lance and the Iceberg REST endpoint get port=0 unless the test actually exercises them, in which case their ports stay pinned as before. * tests: take mini admin ports from the below-ephemeral range * tests: reuse MiniAdminArgs for the remote-s3 fetch mini * tests: fully disable webdav and co-allocate mini admin ports - -webdav.port=0 only makes the service bind a random port; add -webdav=false so minis that do not test WebDAV skip it entirely. - Spark, RisingWave and s3 Spark launchers allocated the admin port in a second MustFreeMiniPort call, which could pick a port already handed out by the first batch. Fold 'Admin' into the same MustFreeMiniPorts call. - test/fuse_integration is a separate Go module and cannot import testutil; allocate the admin ports with its local freePort helper instead. * test/fuse: derive the mini admin gRPC port from its HTTP port Two independent freePort calls could return the same port, making admin.port equal admin.port.grpc (or collide with the filer port). Allocate once and derive the gRPC port with the +10000 convention. * test/fuse: pin the mini admin port in the fuse_test framework copy The fuse_test package framework spawns weed mini too; without a pinned admin port it still binds the fixed defaults 12535/23646 and collides with other minis. * tests: drop the redundant -webdav.port=0 flag -webdav=false alone disables the service; the port flag is dead config. |
||
|
Chris Lu
|
0ca1c19821 |
s3api: unify auth error handling across s3tables, iceberg and lance (#11381)
* s3api: fail closed when S3 Tables signature verification fails * s3api: avoid nil Account dereference in S3 Tables auth log * iceberg: return auth error instead of falling back to DefaultAllow * lance: return auth error instead of falling back to DefaultAllow * s3api: stop trusting client-supplied s3-account-id The header is set by the server after successful authentication; scrub inbound values alongside the other internal headers, and apply the same admin guard to the header fallback branch of getAccountID that the identity branch already has. * test: cover table-catalog auth wrappers and principal resolution * test: configure anonymous identity where catalog clients do not sign * s3api: scrub s3-account-id after signature verification |
||
|
Nguyễn Đăng Minh Lực
|
c968084b34 |
iceberg: fix OAuth token expiry handling (401 + token-exchange + configurable TTL) (#11242)
* iceberg: return 401 for invalid or expired Bearer tokens BUG-0001: when the OAuth JWT expired, Server.Auth fell through to the S3 SigV4 authenticator, which rejects the "Authorization: Bearer" scheme with NotImplemented — a 501. Iceberg clients (Java OAuth2Manager, pyiceberg) only refresh tokens on 401, so they retried the dead token forever: RisingWave sinks stalled and Doris catalog queries failed every token TTL (1h) until the client process was restarted. A request carrying a Bearer header is an Iceberg REST client: answer 401 (+ WWW-Authenticate: Bearer, RFC 6750) when the token fails, and only fall through to the S3 authenticator when no Bearer header is present. * iceberg: make OAuth token TTL configurable via ICEBERG_OAUTH_TOKEN_EXPIRY BUG-0001 follow-up: production evidence shows Iceberg Java 1.10.x clients (RisingWave connector node, Doris FE) never re-fetch tokens on 401 — the sink stalled again on token expiry even with the 501→401 fix, and no POST /v1/oauth/tokens appeared in server logs across dozens of retries. 401 is necessary but not sufficient for these clients. The TTL was hardcoded to 3600 with no knob. Read the expiry (seconds) from ICEBERG_OAUTH_TOKEN_EXPIRY, defaulting to 3600, so deployments can issue longer-lived tokens (e.g. 86400) to survive client restart cycles. * iceberg: support OAuth token exchange (RFC 8693) for client refresh Decompiling the Iceberg Java 1.10.1 client bundled with Doris FE showed the missing half of BUG-0001: OAuth2Manager refreshes via token-exchange (AuthConfig.exchangeEnabled defaults to true — the client_credentials re-fetch branch only runs with exchange disabled), so a server that only accepts client_credentials leaves Iceberg clients unable to ever refresh their token, regardless of 401 correctness. Accept grant_type=urn:ietf:params:oauth:grant-type:token-exchange on POST /v1/oauth/tokens: verify the subject_token signature against the issuing credential, allow exchange within a recovery grace window (max(2*TTL, 1h), capped 24h) so clients holding tokens that expired while the grant was unsupported recover without a restart, and mint a fresh access token with the configured TTL. * iceberg: harden OAuth token exchange and Bearer matching per review - match the Bearer scheme case-insensitively (RFC 7235), like authenticateBearer already does - accept optional client authentication on the token-exchange grant (Basic or form credentials, bound to the subject token's client); expired subject tokens now require it. Iceberg Java's proactive refresh sends Bearer-only headers, so the grant cannot require it - reject subject tokens without an exp claim, and re-check the issuer on the verified claims - unauthenticated exchange cannot extend the lifetime past the subject token's own expiry (no chain-refresh from a leaked token) - return 400 invalid_grant per RFC 6749 §5.2 (was 401) - include issued_token_type on exchange responses (RFC 8693) - clamp ICEBERG_OAUTH_TOKEN_EXPIRY to 365d so Duration math cannot overflow into already-expired tokens * iceberg: give authenticated token exchanges a fresh full TTL The remaining-lifetime cap only guards unauthenticated (Bearer-only) exchanges; an authenticated client renewing a live token must get the full configured TTL, matching client_credentials. * iceberg: reject token exchange when no lifetime remains A Bearer-only exchange with under a second of subject lifetime would mint a token with expires_in: 0. Reject with invalid_grant instead. * iceberg: pin near-expiry test token to the next second boundary jwt/v5 serializes exp at one-second precision, so a 300 ms offset can round into the current second and route the test through the expired branch instead of the ttlSeconds<=0 guard. Mint the subject with the next whole-second expiry: live at exchange time, deterministically under a second of remaining lifetime. * iceberg: drop internal ticket reference from comments * iceberg: clamp oversized OAuth TTLs on 32-bit platforms strconv.Atoi on an int-sized value fails with ErrRange on 386, so an oversized ICEBERG_OAUTH_TOKEN_EXPIRY silently fell back to the default instead of clamping. Parse in 64-bit space and clamp, then narrow. * iceberg: make OAuth TTL narrowing explicit * iceberg: disable legacy OAuth in PyIceberg integration tests |
||
|
Chris Lu
|
0dfaa103d0 |
test: take a table through its whole life, for Iceberg and Lance (#10862)
* lance worker: share the integration tests' scaffolding The recorder that keeps what a handler sent, the config builder and the storage-option fallback all lived inside compaction.rs, so a second test binary would have had to copy them. They move to tests/common. The fallback now reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_ENDPOINT_URL from the environment, defaulting to what it used before. A harness can then point these tests at a gateway that checks what it is given rather than one that accepts anything. * lance worker: maintain one named table, for a harness to drive Compacts and cleans up whatever WEED_LANCE_TABLE names, through the handlers' own detect-then-execute path: a proposal the worker would not have made is not one worth running. The existing tests seed the tables they check. This one deliberately does not, so a harness that has already written a table and knows what is in it can have the real handlers maintain it and then read it back. * test: take a table through its whole life, for Iceberg and Lance Created in the catalog, filled by a real client, maintained by the worker, read again, dropped. The step nothing was checking is the read after maintenance: compaction once rewrote every dictionary-encoded column onto a single value and shipped, because the maintenance tests were thorough about sequence numbers, manifest entries and metadata versions and none of them opened the parquet file the worker had just written. So the assertion is a tally - row count, the cardinality of each dictionary-encoded column, and an md5 over whole rows - taken before maintenance and again after, required to be equal. The cardinalities name the failure that happened; the digest catches a rewrite that keeps every column's cardinality and hands the values to the wrong rows. A compaction that merged nothing fails rather than passes, or the read afterwards is checking a file the worker never wrote. The Iceberg half runs two clients. DuckDB is the one the corruption was reported against and the only one here that writes the deprecated PLAIN_DICTIONARY encoding, which parquet-go normalizes away on write, so a Go writer cannot produce it. PyIceberg writes the modern spelling. Pinning parquet-go back to v0.30.1 fails the DuckDB half and passes the PyIceberg one, which is why both are here. Lance maintenance lives in the Rust worker, so it runs there where cargo is installed and through the two lance calls those handlers wrap where it is not. WEED_LANCE_MAINTENANCE picks one instead of letting the test guess. * ci: run the table lifecycle tests CI maintains the Lance table through the lance library rather than the worker: a cold build of the lance crate costs more than the glue it would be checking, and the worker's own tests cover its handlers. The suite drives the Iceberg maintenance worker, so a change to it now triggers this workflow too. * test: let the lifecycle harness fail instead of skipping Setup failures all exited zero, so a cluster that would not come up, or a port allocation that lost, reported a green run for code nothing had executed. That is the failure mode this whole directory exists to close, and it was in the harness itself. Only a checkout without a weed binary skips now, and it runs the tests so each one says so rather than the package quietly passing. Everything else fails. The filer existence probe gets a deadline while I am here: it ran without one, so an unresponsive filer would hang the suite past every timeout the clients have. * test: make the lifecycle checks check what they claim to Three of them could pass without having looked. The DuckDB skip matched "syntax error", "not implemented" and "Failed to load" anywhere in the output, in any phase. A parse error in the SQL this test generates, or a refusal from our own catalog, would have taken the only coverage of the PLAIN_DICTIONARY encoding out of CI and left it green. It now matches the extension failing to install, and only in the phase that installs it. Everything past LOAD is ours and fails. The digests covered id, category and value. Compaction rewrites the whole row, so a defect confined to ts, or to a Lance vector, changed nothing either side of maintenance. Every persisted column goes in now, ts as microseconds so no timezone sits between the two runs. The Lance drop check caught every exception as proof the dataset was gone. pylance turns credential and transport failures into the same ValueError, so it only accepts the message that means not found. * docs: say up front which maintenance path the Lance half takes The opening summary said the worker maintains both tables. It maintains the Iceberg one always and the Lance one only where cargo is installed, which is not what CI does. |