mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-07 20:47:46 +01:00
S3 Vectors: weed-vector reads tables from the filers and volume servers (#12469)
* sw-client: the filer and volume client library The client the Enterprise repository's VFS and vector engine use, with the same API: filer gRPC for entries, listings, volume locations, assignments and remote caching; volume HTTP, plain or TLS, for needle bytes; chunk resolution for sparse, overlapping and manifest chunks; a location cache; and the write side. Enterprise-only protocol fields (license, conditional mutations, hard links, KV listing) are left out. Compiled against weed/pb/filer.proto like the other worker crates. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-lance-worker: reads straight from the filers and volume servers direct_read::DirectReads wraps the object store Lance opens so that entries, listings and bytes come from the filers (gRPC) and the volume servers (HTTP, signed with the read JWT), whatever the entry holds: inline content, chunk manifests, sparse or overlapping chunks, the filer's volume encryption, the gateway's SSE-S3 (sse.rs, with the KEK from security.toml or the filer) and objects tiered to remote storage. Writes still go to the inner store. The code and its tests are those of the Enterprise repository's engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: read tables from the filers and volume servers The engine asks the masters for the filers (and follows the list), takes jwt.signing.read.key, grpc.client, https.client and the SSE-S3 keys from security.toml, and opens every table with DirectReads innermost and the read cache outside it. --masters is required: the gateway finds the engine through them and the engine finds the filers. Writes still go to the gateway. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sse: chunks out of a manifest keep their own SSE-S3 keys A chunk expanded from a manifest carries its id as fid, so keying the per-chunk key cache by file_id gave every such chunk the first one's key and IV. Keyed by the chunk's id however it is carried. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: a silent filer is passed over like an unreachable one Each filer attempt is bounded, so a filer that accepts the connection and never answers does not hold the read or the startup lookup. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: carry the assignment's fsync to the volume upload The filer's storage rule may want a needle on disk before the volume answers; the assignment says so and the upload now asks for it, as the Go uploader does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: argument tests pass the required masters; connect timeout for filers Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: a failed read is tried once more against fresh locations A fetch that fails forgets the volume locations it was made against; the request now looks them up again and retries instead of failing on a volume that moved. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: the filer deadline bounds a listing page, not the listing Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: security.toml booleans read as viper reads them Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a filer the master lists badly does not hide the others Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: register with the masters only once ready to serve Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: the HTTP transport asks the volume to fsync when the assignment does write_synced fell back to the trait default, which refuses fsync, so every upload under a fsync storage rule failed before it was sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: https.client.ca alone vouches for volume servers With a CA configured, reqwest still trusted the system roots next to it; weed processes trust only the configured pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an empty volume range is read without a request A zero length underflowed the Range end, and a range past u64::MAX overflowed it; the first now returns no bytes, the second an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: a whole-needle answer to a ranged read is cut to the range A server or proxy that ignores Range answers 200 with the whole needle; the read returned its first bytes in place of the ones asked for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: a read size past u64::MAX reads to the end of the file offset + size overflowed before the clamp to the file size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an SSE-encrypted manifest is refused without a decoder Only filer-encrypted manifests were; an SSE one went to the protobuf decoder as ciphertext. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an empty write leaves the staged file size alone A zero-byte write past the end grew the size to its offset, as a zero-byte pwrite does not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: the chunk cache keeps no empty chunks They add no bytes, so nothing evicted them and the map grew with every distinct empty needle, even at capacity zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: flushing with a zero chunk size is an error, not a panic Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: an empty master list fails at startup "" split into one empty address that passed the check, and the engine then waited two minutes for a filer before a generic error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Chris LuandClaude Opus 5.5 authored and GitHub committed
1 parent
2fe480a6e2
commit
b650303aee
23 files changed
+8862
-46
No files matched your search
Generated
+582
-7
@@ -8,6 +8,41 @@ version = "2.0.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
|
||||
|
||||
[[package]]
|
||||
name = "aead"
|
||||
version = "0.5.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d122413f284cf2d62fb1b7db97e02edb8cda96d769b16e443a4f6195e35662b0"
|
||||
dependencies = [
|
||||
"crypto-common 0.1.7",
|
||||
"generic-array",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "aes"
|
||||
version = "0.8.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cipher",
|
||||
"cpufeatures 0.2.17",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "aes-gcm"
|
||||
version = "0.10.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "831010a0f742e1209b3bcea8fab6a8e149051ba6099432c8cb2cc117dec3ead1"
|
||||
dependencies = [
|
||||
"aead",
|
||||
"aes",
|
||||
"cipher",
|
||||
"ctr",
|
||||
"ghash",
|
||||
"subtle",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ahash"
|
||||
version = "0.8.12"
|
||||
@@ -661,10 +696,10 @@ dependencies = [
|
||||
"bytes",
|
||||
"form_urlencoded",
|
||||
"hex",
|
||||
"hmac",
|
||||
"hmac 0.13.0",
|
||||
"http",
|
||||
"percent-encoding",
|
||||
"sha2",
|
||||
"sha2 0.11.1",
|
||||
"time",
|
||||
"tracing",
|
||||
]
|
||||
@@ -929,6 +964,12 @@ dependencies = [
|
||||
"tokio",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "base16ct"
|
||||
version = "0.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4c7f02d4ea65f2c1853089ffd8d2787bdbc63de2f0d29dedbcf8ccdfa0ccd4cf"
|
||||
|
||||
[[package]]
|
||||
name = "base64"
|
||||
version = "0.22.1"
|
||||
@@ -951,6 +992,12 @@ dependencies = [
|
||||
"vsimd",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "base64ct"
|
||||
version = "1.8.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06"
|
||||
|
||||
[[package]]
|
||||
name = "bigdecimal"
|
||||
version = "0.4.11"
|
||||
@@ -1144,6 +1191,16 @@ dependencies = [
|
||||
"phf",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cipher"
|
||||
version = "0.4.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad"
|
||||
dependencies = [
|
||||
"crypto-common 0.1.7",
|
||||
"inout",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "clap"
|
||||
version = "4.6.7"
|
||||
@@ -1251,6 +1308,12 @@ dependencies = [
|
||||
"crossbeam-utils",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "const-oid"
|
||||
version = "0.9.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8"
|
||||
|
||||
[[package]]
|
||||
name = "const-oid"
|
||||
version = "0.10.2"
|
||||
@@ -1400,6 +1463,18 @@ version = "0.2.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5"
|
||||
|
||||
[[package]]
|
||||
name = "crypto-bigint"
|
||||
version = "0.5.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0dc92fb57ca44df6db8059111ab3af99a63d5d0f8375d9972e319a379c6bab76"
|
||||
dependencies = [
|
||||
"generic-array",
|
||||
"rand_core 0.6.4",
|
||||
"subtle",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "crypto-common"
|
||||
version = "0.1.7"
|
||||
@@ -1407,6 +1482,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
|
||||
dependencies = [
|
||||
"generic-array",
|
||||
"rand_core 0.6.4",
|
||||
"typenum",
|
||||
]
|
||||
|
||||
@@ -1450,6 +1526,15 @@ dependencies = [
|
||||
"linktime-proc-macro",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ctr"
|
||||
version = "0.9.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0369ee1ad671834580515889b80f2ea915f23b8be8d0daa4bbaf2ac5c7590835"
|
||||
dependencies = [
|
||||
"cipher",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ctutils"
|
||||
version = "0.4.6"
|
||||
@@ -1459,6 +1544,33 @@ dependencies = [
|
||||
"cmov",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "curve25519-dalek"
|
||||
version = "4.1.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "97fb8b7c4503de7d6ae7b42ab72a5a59857b4c937ec27a3d4539dba95b5ab2be"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cpufeatures 0.2.17",
|
||||
"curve25519-dalek-derive",
|
||||
"digest 0.10.7",
|
||||
"fiat-crypto",
|
||||
"rustc_version",
|
||||
"subtle",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "curve25519-dalek-derive"
|
||||
version = "0.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f46882e17999c6cc590af592290432be3bce0428cb0d5f8b6715e4dc7b383eb3"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 2.0.119",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "darling"
|
||||
version = "0.24.1"
|
||||
@@ -1833,7 +1945,7 @@ dependencies = [
|
||||
"num-traits",
|
||||
"rand 0.9.5",
|
||||
"regex",
|
||||
"sha2",
|
||||
"sha2 0.11.1",
|
||||
"uuid",
|
||||
]
|
||||
|
||||
@@ -2151,6 +2263,17 @@ dependencies = [
|
||||
"thiserror 2.0.21",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "der"
|
||||
version = "0.7.10"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb"
|
||||
dependencies = [
|
||||
"const-oid 0.9.6",
|
||||
"pem-rfc7468",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "der-parser"
|
||||
version = "9.0.0"
|
||||
@@ -2181,6 +2304,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
|
||||
dependencies = [
|
||||
"block-buffer 0.10.4",
|
||||
"const-oid 0.9.6",
|
||||
"crypto-common 0.1.7",
|
||||
"subtle",
|
||||
]
|
||||
@@ -2192,7 +2316,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2"
|
||||
dependencies = [
|
||||
"block-buffer 0.12.1",
|
||||
"const-oid",
|
||||
"const-oid 0.10.2",
|
||||
"crypto-common 0.2.2",
|
||||
"ctutils",
|
||||
]
|
||||
@@ -2250,12 +2374,71 @@ version = "1.0.20"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555"
|
||||
|
||||
[[package]]
|
||||
name = "ecdsa"
|
||||
version = "0.16.9"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ee27f32b5c5292967d2d4a9d7f1e0b0aed2c15daded5a60300e4abb9d8020bca"
|
||||
dependencies = [
|
||||
"der",
|
||||
"digest 0.10.7",
|
||||
"elliptic-curve",
|
||||
"rfc6979",
|
||||
"signature",
|
||||
"spki",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ed25519"
|
||||
version = "2.2.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "115531babc129696a58c64a4fef0a8bf9e9698629fb97e9e40767d235cfbcd53"
|
||||
dependencies = [
|
||||
"pkcs8",
|
||||
"signature",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ed25519-dalek"
|
||||
version = "2.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "70e796c081cee67dc755e1a36a0a172b897fab85fc3f6bc48307991f64e4eca9"
|
||||
dependencies = [
|
||||
"curve25519-dalek",
|
||||
"ed25519",
|
||||
"serde",
|
||||
"sha2 0.10.9",
|
||||
"subtle",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "either"
|
||||
version = "1.19.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0e9c71c2167ca323c882b99918929403426e2373ea17242ff5653e0d5e1058be"
|
||||
|
||||
[[package]]
|
||||
name = "elliptic-curve"
|
||||
version = "0.13.8"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b5e6043086bf7973472e0c7dff2142ea0b680d30e18d9cc40f267efbf222bd47"
|
||||
dependencies = [
|
||||
"base16ct",
|
||||
"crypto-bigint",
|
||||
"digest 0.10.7",
|
||||
"ff",
|
||||
"generic-array",
|
||||
"group",
|
||||
"hkdf",
|
||||
"pem-rfc7468",
|
||||
"pkcs8",
|
||||
"rand_core 0.6.4",
|
||||
"sec1",
|
||||
"subtle",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "encoding_rs"
|
||||
version = "0.8.42"
|
||||
@@ -2324,6 +2507,22 @@ version = "2.5.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223"
|
||||
|
||||
[[package]]
|
||||
name = "ff"
|
||||
version = "0.13.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c0b50bfb653653f9ca9095b427bed08ab8d75a137839d9ad64eb11810d5b6393"
|
||||
dependencies = [
|
||||
"rand_core 0.6.4",
|
||||
"subtle",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "fiat-crypto"
|
||||
version = "0.2.9"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "28dea519a9695b9977216879a3ebfddf92f1c08c05d984f8996aecd6ecdc811d"
|
||||
|
||||
[[package]]
|
||||
name = "find-msvc-tools"
|
||||
version = "0.1.15"
|
||||
@@ -2526,6 +2725,7 @@ checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
|
||||
dependencies = [
|
||||
"typenum",
|
||||
"version_check",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -2567,6 +2767,16 @@ dependencies = [
|
||||
"wasm-bindgen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ghash"
|
||||
version = "0.5.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f0d8a4362ccb29cb0b265253fb0a2728f592895ee6854fd9bc13f2ffda266ff1"
|
||||
dependencies = [
|
||||
"opaque-debug",
|
||||
"polyval",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "glob"
|
||||
version = "0.3.4"
|
||||
@@ -2585,6 +2795,17 @@ dependencies = [
|
||||
"wasm-bindgen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "group"
|
||||
version = "0.13.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f0f9ef7462f7c099f518d754361858f86d8a07af53ba9af0fe635bbccb151a63"
|
||||
dependencies = [
|
||||
"ff",
|
||||
"rand_core 0.6.4",
|
||||
"subtle",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "h2"
|
||||
version = "0.4.20"
|
||||
@@ -2673,6 +2894,24 @@ version = "0.4.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"
|
||||
|
||||
[[package]]
|
||||
name = "hkdf"
|
||||
version = "0.12.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7b5f8eb2ad728638ea2c7d47a21db23b7b58a72ed6a38256b8a1849f15fbbdf7"
|
||||
dependencies = [
|
||||
"hmac 0.12.1",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "hmac"
|
||||
version = "0.12.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e"
|
||||
dependencies = [
|
||||
"digest 0.10.7",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "hmac"
|
||||
version = "0.13.0"
|
||||
@@ -3030,6 +3269,15 @@ dependencies = [
|
||||
"serde_core",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "inout"
|
||||
version = "0.1.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01"
|
||||
dependencies = [
|
||||
"generic-array",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "io-uring"
|
||||
version = "0.7.15"
|
||||
@@ -3214,6 +3462,30 @@ dependencies = [
|
||||
"zmij",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "jsonwebtoken"
|
||||
version = "10.4.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "eba32bfb4ffdeaca3e34431072faf01745c9b26d25504aa7a6cf5684334fc4fc"
|
||||
dependencies = [
|
||||
"base64 0.22.1",
|
||||
"ed25519-dalek",
|
||||
"getrandom 0.2.17",
|
||||
"hmac 0.12.1",
|
||||
"js-sys",
|
||||
"p256",
|
||||
"p384",
|
||||
"pem",
|
||||
"rand 0.8.8",
|
||||
"rsa",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha2 0.10.9",
|
||||
"signature",
|
||||
"simple_asn1",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "lance"
|
||||
version = "10.0.0"
|
||||
@@ -3774,6 +4046,9 @@ name = "lazy_static"
|
||||
version = "1.5.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "20870f649af7073d53e38067b2a84312175d56ea15217e1b15bc83506ec50afb"
|
||||
dependencies = [
|
||||
"spin",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "lexical-core"
|
||||
@@ -4140,6 +4415,22 @@ dependencies = [
|
||||
"num-traits",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "num-bigint-dig"
|
||||
version = "0.8.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e661dda6640fad38e827a6d4a310ff4763082116fe217f279885c97f511bb0b7"
|
||||
dependencies = [
|
||||
"lazy_static",
|
||||
"libm",
|
||||
"num-integer",
|
||||
"num-iter",
|
||||
"num-traits",
|
||||
"rand 0.8.8",
|
||||
"smallvec",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "num-complex"
|
||||
version = "0.4.6"
|
||||
@@ -4164,6 +4455,16 @@ dependencies = [
|
||||
"num-traits",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "num-iter"
|
||||
version = "0.1.46"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c92800bd69a1eac91786bcfe9da64a897eb72911b8dc3095decbd07429e8048b"
|
||||
dependencies = [
|
||||
"num-integer",
|
||||
"num-traits",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "num-traits"
|
||||
version = "0.2.19"
|
||||
@@ -4260,6 +4561,12 @@ version = "1.70.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
|
||||
|
||||
[[package]]
|
||||
name = "opaque-debug"
|
||||
version = "0.3.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381"
|
||||
|
||||
[[package]]
|
||||
name = "opendal"
|
||||
version = "0.57.0"
|
||||
@@ -4404,6 +4711,30 @@ version = "0.5.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1a80800c0488c3a21695ea981a54918fbb37abf04f4d0720c453632255e2ff0e"
|
||||
|
||||
[[package]]
|
||||
name = "p256"
|
||||
version = "0.13.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c9863ad85fa8f4460f9c48cb909d38a0d689dba1f6f6988a5e3e0d31071bcd4b"
|
||||
dependencies = [
|
||||
"ecdsa",
|
||||
"elliptic-curve",
|
||||
"primeorder",
|
||||
"sha2 0.10.9",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "p384"
|
||||
version = "0.13.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "fe42f1670a52a47d448f14b6a5c61dd78fce51856e68edaa38f7ae3a46b8d6b6"
|
||||
dependencies = [
|
||||
"ecdsa",
|
||||
"elliptic-curve",
|
||||
"primeorder",
|
||||
"sha2 0.10.9",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "parking"
|
||||
version = "2.2.1"
|
||||
@@ -4451,6 +4782,25 @@ dependencies = [
|
||||
"stfu8",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pem"
|
||||
version = "3.0.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1d30c53c26bc5b31a98cd02d20f25a7c8567146caf63ed593a9d87b2775291be"
|
||||
dependencies = [
|
||||
"base64 0.22.1",
|
||||
"serde_core",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pem-rfc7468"
|
||||
version = "0.7.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "88b39c9bfcfc231068454382784bb460aae594343fb030d46e9f50a645418412"
|
||||
dependencies = [
|
||||
"base64ct",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "percent-encoding"
|
||||
version = "2.3.2"
|
||||
@@ -4519,12 +4869,45 @@ version = "0.2.17"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd"
|
||||
|
||||
[[package]]
|
||||
name = "pkcs1"
|
||||
version = "0.7.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c8ffb9f10fa047879315e6625af03c164b16962a5368d724ed16323b68ace47f"
|
||||
dependencies = [
|
||||
"der",
|
||||
"pkcs8",
|
||||
"spki",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pkcs8"
|
||||
version = "0.10.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f950b2377845cebe5cf8b5165cb3cc1a5e0fa5cfa3e1f7f55707d8fd82e0a7b7"
|
||||
dependencies = [
|
||||
"der",
|
||||
"spki",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pkg-config"
|
||||
version = "0.3.34"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548"
|
||||
|
||||
[[package]]
|
||||
name = "polyval"
|
||||
version = "0.6.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9d1fe60d06143b2430aa532c94cfe9e29783047f06c0d7fd359a9a51b729fa25"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cpufeatures 0.2.17",
|
||||
"opaque-debug",
|
||||
"universal-hash",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "portable-atomic"
|
||||
version = "1.15.0"
|
||||
@@ -4576,6 +4959,15 @@ dependencies = [
|
||||
"syn 2.0.119",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "primeorder"
|
||||
version = "0.13.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "353e1ca18966c16d9deb1c69278edbc5f194139612772bd9537af60ac231e1e6"
|
||||
dependencies = [
|
||||
"elliptic-curve",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "proc-macro2"
|
||||
version = "1.0.107"
|
||||
@@ -4852,13 +5244,24 @@ version = "0.7.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "dc33ff2d4973d518d823d61aa239014831e521c75da58e3df4840d3f47749d09"
|
||||
|
||||
[[package]]
|
||||
name = "rand"
|
||||
version = "0.8.8"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e058c7de0b26af77780c769414d6257830bb240f3c38477dbc2c16e5f54d6d4c"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"rand_chacha 0.3.1",
|
||||
"rand_core 0.6.4",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rand"
|
||||
version = "0.9.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41"
|
||||
dependencies = [
|
||||
"rand_chacha",
|
||||
"rand_chacha 0.9.0",
|
||||
"rand_core 0.9.5",
|
||||
]
|
||||
|
||||
@@ -4873,6 +5276,16 @@ dependencies = [
|
||||
"rand_core 0.10.1",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rand_chacha"
|
||||
version = "0.3.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88"
|
||||
dependencies = [
|
||||
"ppv-lite86",
|
||||
"rand_core 0.6.4",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rand_chacha"
|
||||
version = "0.9.0"
|
||||
@@ -4883,6 +5296,15 @@ dependencies = [
|
||||
"rand_core 0.9.5",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rand_core"
|
||||
version = "0.6.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c"
|
||||
dependencies = [
|
||||
"getrandom 0.2.17",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rand_core"
|
||||
version = "0.9.5"
|
||||
@@ -5079,13 +5501,13 @@ dependencies = [
|
||||
"base64 0.23.1",
|
||||
"bytes",
|
||||
"hex",
|
||||
"hmac",
|
||||
"hmac 0.13.0",
|
||||
"http",
|
||||
"jiff",
|
||||
"log",
|
||||
"percent-encoding",
|
||||
"sha1 0.11.0",
|
||||
"sha2",
|
||||
"sha2 0.11.1",
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
@@ -5184,6 +5606,16 @@ dependencies = [
|
||||
"web-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rfc6979"
|
||||
version = "0.4.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f8dd2a808d456c4a54e300a23e9f5a67e122c3024119acbfd73e3bf664491cb2"
|
||||
dependencies = [
|
||||
"hmac 0.12.1",
|
||||
"subtle",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ring"
|
||||
version = "0.17.14"
|
||||
@@ -5208,6 +5640,26 @@ dependencies = [
|
||||
"byteorder",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rsa"
|
||||
version = "0.9.10"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b8573f03f5883dcaebdfcf4725caa1ecb9c15b2ef50c43a07b816e06799bb12d"
|
||||
dependencies = [
|
||||
"const-oid 0.9.6",
|
||||
"digest 0.10.7",
|
||||
"num-bigint-dig",
|
||||
"num-integer",
|
||||
"num-traits",
|
||||
"pkcs1",
|
||||
"pkcs8",
|
||||
"rand_core 0.6.4",
|
||||
"signature",
|
||||
"spki",
|
||||
"subtle",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rust-ini"
|
||||
version = "0.21.3"
|
||||
@@ -5446,6 +5898,20 @@ dependencies = [
|
||||
"seaweed-worker-core",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "sec1"
|
||||
version = "0.7.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d3e97a565f76233a6003f9f5c54be1d9c5bdfa3eccfb189469f11ec4901c47dc"
|
||||
dependencies = [
|
||||
"base16ct",
|
||||
"der",
|
||||
"generic-array",
|
||||
"pkcs8",
|
||||
"subtle",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "security-framework"
|
||||
version = "3.7.0"
|
||||
@@ -5622,6 +6088,17 @@ dependencies = [
|
||||
"digest 0.11.3",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "sha2"
|
||||
version = "0.10.9"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cpufeatures 0.2.17",
|
||||
"digest 0.10.7",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "sha2"
|
||||
version = "0.11.1"
|
||||
@@ -5658,6 +6135,16 @@ dependencies = [
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "signature"
|
||||
version = "2.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de"
|
||||
dependencies = [
|
||||
"digest 0.10.7",
|
||||
"rand_core 0.6.4",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "simd-adler32"
|
||||
version = "0.3.10"
|
||||
@@ -5680,6 +6167,18 @@ version = "0.1.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e"
|
||||
|
||||
[[package]]
|
||||
name = "simple_asn1"
|
||||
version = "0.6.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0d585997b0ac10be3c5ee635f1bab02d512760d14b7c468801ac8a01d9ae5f1d"
|
||||
dependencies = [
|
||||
"num-bigint",
|
||||
"num-traits",
|
||||
"thiserror 2.0.21",
|
||||
"time",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "siphasher"
|
||||
version = "1.0.4"
|
||||
@@ -5729,6 +6228,22 @@ dependencies = [
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "spin"
|
||||
version = "0.9.9"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3763264f6b73151db08c50ff20d7d8a0b8796e021cdea7ceedad07b80155fa0e"
|
||||
|
||||
[[package]]
|
||||
name = "spki"
|
||||
version = "0.7.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d"
|
||||
dependencies = [
|
||||
"base64ct",
|
||||
"der",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "sqlparser"
|
||||
version = "0.62.0"
|
||||
@@ -5811,6 +6326,25 @@ version = "2.6.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
|
||||
|
||||
[[package]]
|
||||
name = "sw-client"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
"bytes",
|
||||
"log",
|
||||
"prost",
|
||||
"protoc-bin-vendored",
|
||||
"reqwest 0.12.28",
|
||||
"rustls",
|
||||
"thiserror 2.0.21",
|
||||
"tokio",
|
||||
"tonic",
|
||||
"tonic-prost",
|
||||
"tonic-prost-build",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "syn"
|
||||
version = "2.0.119"
|
||||
@@ -6347,6 +6881,16 @@ version = "0.2.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254"
|
||||
|
||||
[[package]]
|
||||
name = "universal-hash"
|
||||
version = "0.5.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "fc1de2c688dc15305988b563c3854064043356019f97a4b46276fe734c4f07ea"
|
||||
dependencies = [
|
||||
"crypto-common 0.1.7",
|
||||
"subtle",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "untrusted"
|
||||
version = "0.7.1"
|
||||
@@ -6583,18 +7127,31 @@ dependencies = [
|
||||
name = "weed-lance-worker"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"aes",
|
||||
"aes-gcm",
|
||||
"anyhow",
|
||||
"arrow-cast",
|
||||
"arrow-schema",
|
||||
"async-trait",
|
||||
"base64 0.22.1",
|
||||
"bytes",
|
||||
"chrono",
|
||||
"clap",
|
||||
"ctr",
|
||||
"flate2",
|
||||
"futures",
|
||||
"hex",
|
||||
"hkdf",
|
||||
"hmac 0.12.1",
|
||||
"jsonwebtoken",
|
||||
"lance",
|
||||
"lance-datafusion",
|
||||
"lance-index",
|
||||
"lance-linalg",
|
||||
"lance-table",
|
||||
"md-5 0.10.6",
|
||||
"moka",
|
||||
"object_store",
|
||||
"prometheus",
|
||||
"reqwest 0.12.28",
|
||||
"roaring",
|
||||
@@ -6603,9 +7160,12 @@ dependencies = [
|
||||
"seaweed-worker-sort",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha2 0.10.9",
|
||||
"sw-client",
|
||||
"tokio",
|
||||
"tracing",
|
||||
"tracing-subscriber",
|
||||
"zstd",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -6661,6 +7221,7 @@ dependencies = [
|
||||
"rustls",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sw-client",
|
||||
"tempfile",
|
||||
"tokio",
|
||||
"tokio-rustls",
|
||||
@@ -6955,6 +7516,20 @@ name = "zeroize"
|
||||
version = "1.9.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e13084392c5e4bc371903e2935a5eaeed24905a7511356b883835e18a78f6879"
|
||||
dependencies = [
|
||||
"zeroize_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zeroize_derive"
|
||||
version = "1.5.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3c50655cbb0fe3fc43170059e702f1ce5e19b84cec58dc87b037a09935c2f328"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 2.0.119",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zerotrie"
|
||||
|
||||
@@ -15,6 +15,7 @@ path = "src/main.rs"
|
||||
[dependencies]
|
||||
weed-vector-core = { path = "../core" }
|
||||
weed-lance-worker = { path = "../../../seaweed-worker/crates/lance" }
|
||||
sw-client = { path = "../../../seaweed-worker/crates/sw-client" }
|
||||
lance = { version = "10", default-features = false, features = ["aws"] }
|
||||
lance-arrow = "10"
|
||||
lance-linalg = "10"
|
||||
|
||||
@@ -1,10 +1,14 @@
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use clap::Parser;
|
||||
use tokio_stream::wrappers::TcpListenerStream;
|
||||
use tonic::transport::Server;
|
||||
use weed_lance_worker::dataset::FallbackOptions;
|
||||
use weed_lance_worker::direct_read::{Connection, DirectReads, FilerDialer};
|
||||
use weed_vector_server::master_client;
|
||||
use weed_vector_server::pb::vector::vector_service_server::VectorServiceServer;
|
||||
use weed_vector_server::read_cache::ReadCache;
|
||||
use weed_vector_server::security::Security;
|
||||
@@ -22,10 +26,18 @@ struct Args {
|
||||
/// only with the aws_endpoint given here.
|
||||
#[arg(long = "storage-option", alias = "storage.option", value_parser = parse_option)]
|
||||
storage_options: Vec<(String, String)>,
|
||||
/// Comma-separated master addresses to register this server with; the
|
||||
/// gateway discovers it via ListClusterNodes(client_type=vector).
|
||||
/// Comma-separated master addresses. The engine registers with them, so
|
||||
/// the gateway finds it, and asks them for the filers it reads through.
|
||||
#[arg(long, alias = "master", env = "WEED_VECTOR_MASTERS")]
|
||||
masters: Option<String>,
|
||||
masters: String,
|
||||
/// The filer group whose filers the engine reads through.
|
||||
#[arg(
|
||||
long,
|
||||
alias = "filer.group",
|
||||
env = "WEED_VECTOR_FILER_GROUP",
|
||||
default_value = ""
|
||||
)]
|
||||
filer_group: String,
|
||||
/// Memory for caching reads of write-once table files, in MiB; 0 disables.
|
||||
#[arg(long, env = "WEED_VECTOR_READ_CACHE_MB", default_value_t = 512)]
|
||||
read_cache_mb: u64,
|
||||
@@ -85,6 +97,68 @@ fn normalize_args(args: Vec<String>) -> Vec<String> {
|
||||
|
||||
/// A wildcard listen address is not reachable for peers; derive the outbound
|
||||
/// IP toward the first master instead — the address the master sees anyway.
|
||||
/// How often the masters are asked for the filers.
|
||||
const FILER_REFRESH: Duration = Duration::from_secs(30);
|
||||
/// How long startup waits for a filer to be registered.
|
||||
const FILER_WAIT: Duration = Duration::from_secs(120);
|
||||
|
||||
/// The filers the masters list, waiting for the first to register.
|
||||
async fn discover_filers(
|
||||
masters: &[String],
|
||||
filer_group: &str,
|
||||
tls: Option<&weed_vector_server::security::GrpcTls>,
|
||||
) -> Result<Vec<String>> {
|
||||
let deadline = tokio::time::Instant::now() + FILER_WAIT;
|
||||
loop {
|
||||
match master_client::list_filers(masters, filer_group, tls).await {
|
||||
Ok(filers) if !filers.is_empty() => return Ok(filers),
|
||||
Ok(_) => tracing::warn!("no filer is registered with the masters yet"),
|
||||
Err(err) => tracing::warn!("listing filers: {err:#}"),
|
||||
}
|
||||
if tokio::time::Instant::now() >= deadline {
|
||||
anyhow::bail!("no filer registered with the masters within {FILER_WAIT:?}");
|
||||
}
|
||||
tokio::time::sleep(Duration::from_secs(3)).await;
|
||||
}
|
||||
}
|
||||
|
||||
async fn follow_filers(
|
||||
reads: DirectReads,
|
||||
masters: Vec<String>,
|
||||
filer_group: String,
|
||||
tls: Option<weed_vector_server::security::GrpcTls>,
|
||||
) {
|
||||
loop {
|
||||
tokio::time::sleep(FILER_REFRESH).await;
|
||||
match master_client::list_filers(&masters, &filer_group, tls.as_ref()).await {
|
||||
Ok(filers) if !filers.is_empty() => {
|
||||
if let Err(err) = reads.set_filers(&filers) {
|
||||
tracing::warn!("filers {}: {err:#}", filers.join(","));
|
||||
}
|
||||
}
|
||||
// Keep the last known filers while the masters list none.
|
||||
Ok(_) => tracing::warn!("the masters list no filer; keeping the known ones"),
|
||||
Err(err) => tracing::warn!("listing filers: {err:#}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn master_addresses(masters: &str) -> Result<Vec<String>> {
|
||||
let masters: Vec<String> = masters
|
||||
.split(',')
|
||||
.map(str::trim)
|
||||
.filter(|s| !s.is_empty())
|
||||
.map(str::to_string)
|
||||
.collect();
|
||||
if masters.is_empty() {
|
||||
anyhow::bail!("--masters (WEED_VECTOR_MASTERS) lists no master address");
|
||||
}
|
||||
for master in &masters {
|
||||
master_client::grpc_addr(master)?;
|
||||
}
|
||||
Ok(masters)
|
||||
}
|
||||
|
||||
fn advertised_addr(listen: &str, masters: &[String]) -> String {
|
||||
let Some((host, _)) = listen.rsplit_once(':') else {
|
||||
return listen.to_string();
|
||||
@@ -129,10 +203,7 @@ async fn main() -> Result<()> {
|
||||
if let Some(path) = security.path() {
|
||||
tracing::info!("security settings from {}", path.display());
|
||||
}
|
||||
let master_tls = match &args.masters {
|
||||
Some(_) => security.grpc_client_tls("grpc.client", "grpc.master")?,
|
||||
None => None,
|
||||
};
|
||||
let master_tls = security.grpc_client_tls("grpc.client", "grpc.master")?;
|
||||
let server_tls = security
|
||||
.grpc_server_tls("grpc.vector")?
|
||||
.map(ServerTls::load)
|
||||
@@ -142,21 +213,56 @@ async fn main() -> Result<()> {
|
||||
let listener = tokio::net::TcpListener::bind(&args.listen)
|
||||
.await
|
||||
.with_context(|| format!("listen on {}", args.listen))?;
|
||||
if let Some(masters) = &args.masters {
|
||||
let masters: Vec<String> = masters.split(',').map(|s| s.trim().to_string()).collect();
|
||||
for master in &masters {
|
||||
weed_vector_server::master_client::grpc_addr(master)?;
|
||||
}
|
||||
let advertise = advertised_addr(&args.listen, &masters);
|
||||
tokio::spawn(weed_vector_server::master_client::keep_connected(
|
||||
masters,
|
||||
advertise,
|
||||
master_tls.clone(),
|
||||
));
|
||||
}
|
||||
let masters = master_addresses(&args.masters)?;
|
||||
// The filers are dialed as the masters are: grpc.client's credentials
|
||||
// and server allowlist, through the engine's own connector.
|
||||
let filer_dialer: Option<FilerDialer> = security
|
||||
.grpc_client_tls("grpc.client", "grpc.filer")?
|
||||
.map(|tls| {
|
||||
Arc::new(move |addr: &str| {
|
||||
let channel = tonic::transport::Endpoint::new(format!("http://{addr}"))?
|
||||
.connect_timeout(Duration::from_secs(10))
|
||||
.connect_with_connector_lazy(tls::connector(tls.clone()));
|
||||
Ok(sw_client::FilerClient::from_channel(addr, channel))
|
||||
}) as FilerDialer
|
||||
});
|
||||
let conn = Connection {
|
||||
filers: discover_filers(&masters, &args.filer_group, master_tls.as_ref()).await?,
|
||||
read_jwt_key: security.jwt_read_key(),
|
||||
sse: security.sse(),
|
||||
grpc_tls: None,
|
||||
https_tls: security.https_client_tls()?,
|
||||
filer_dialer,
|
||||
};
|
||||
let reads = DirectReads::connect(&conn).await.context("direct reads")?;
|
||||
tracing::info!(
|
||||
"reading tables through {reads:?}; filer gRPC {}, volume servers {}; SSE-S3 objects {}",
|
||||
if conn.filer_dialer.is_some() {
|
||||
"over TLS"
|
||||
} else {
|
||||
"plaintext"
|
||||
},
|
||||
if conn.https_tls.is_some() {
|
||||
"over HTTPS"
|
||||
} else {
|
||||
"over HTTP"
|
||||
},
|
||||
if reads.sse_s3_configured() {
|
||||
"are readable"
|
||||
} else {
|
||||
"need s3.sse.kek or s3.sse.key"
|
||||
},
|
||||
);
|
||||
tokio::spawn(follow_filers(
|
||||
reads.clone(),
|
||||
masters.clone(),
|
||||
args.filer_group.clone(),
|
||||
master_tls.clone(),
|
||||
));
|
||||
let fallback: FallbackOptions = args.storage_options.into_iter().collect();
|
||||
let mut service =
|
||||
VectorServer::new(fallback).with_nprobes(args.nprobes.map(std::num::NonZeroUsize::get));
|
||||
let mut service = VectorServer::new(fallback)
|
||||
.with_nprobes(args.nprobes.map(std::num::NonZeroUsize::get))
|
||||
.with_direct_reads(reads);
|
||||
if args.read_cache_mb > 0 {
|
||||
service = service.with_read_cache(ReadCache::new(args.read_cache_mb << 20));
|
||||
}
|
||||
@@ -172,6 +278,14 @@ async fn main() -> Result<()> {
|
||||
},
|
||||
if master_tls.is_some() { "over TLS" } else { "in plaintext" },
|
||||
);
|
||||
// Registered only now, with storage connected and the listener bound,
|
||||
// so the gateway never finds an engine that cannot serve yet.
|
||||
let advertise = advertised_addr(&args.listen, &masters);
|
||||
tokio::spawn(master_client::keep_connected(
|
||||
masters.clone(),
|
||||
advertise,
|
||||
master_tls.clone(),
|
||||
));
|
||||
let router = Server::builder()
|
||||
// Match the gateway's 1 GiB cap — put batches over the tonic 4 MiB
|
||||
// default otherwise get rejected before they reach the handler.
|
||||
@@ -200,8 +314,11 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn nprobes_must_be_positive() {
|
||||
assert!(Args::try_parse_from(["weed-vector", "--nprobes", "0"]).is_err());
|
||||
let args = Args::try_parse_from(["weed-vector", "--nprobes", "12"]).unwrap();
|
||||
let parse = |nprobes: &str| {
|
||||
Args::try_parse_from(["weed-vector", "--masters", "m:9333", "--nprobes", nprobes])
|
||||
};
|
||||
assert!(parse("0").is_err());
|
||||
let args = parse("12").unwrap();
|
||||
assert_eq!(args.nprobes.map(std::num::NonZeroUsize::get), Some(12));
|
||||
}
|
||||
|
||||
@@ -210,6 +327,8 @@ mod tests {
|
||||
let args = Args::try_parse_from(normalize_args(
|
||||
[
|
||||
"weed-vector",
|
||||
"-master",
|
||||
"m:9333",
|
||||
"--namespace",
|
||||
"http://localhost:9101",
|
||||
"-filer-signing-key",
|
||||
@@ -221,4 +340,15 @@ mod tests {
|
||||
.unwrap();
|
||||
assert!(args.namespace.is_some() && args.filer_signing_key.is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_masters_are_required() {
|
||||
assert!(Args::try_parse_from(["weed-vector"]).is_err());
|
||||
assert!(master_addresses("").is_err());
|
||||
assert!(master_addresses(" , ").is_err());
|
||||
assert_eq!(
|
||||
master_addresses(" m1:9333,,m2:9333 ").unwrap(),
|
||||
["m1:9333", "m2:9333"]
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -9,7 +9,9 @@ use anyhow::{Context, Result};
|
||||
use tokio::time::sleep;
|
||||
use tonic::transport::{Channel, Endpoint};
|
||||
|
||||
use crate::master_pb::{KeepConnectedRequest, seaweed_client::SeaweedClient};
|
||||
use crate::master_pb::{
|
||||
KeepConnectedRequest, ListClusterNodesRequest, seaweed_client::SeaweedClient,
|
||||
};
|
||||
use crate::security::GrpcTls;
|
||||
|
||||
fn registration(client_addr: &str) -> KeepConnectedRequest {
|
||||
@@ -340,3 +342,74 @@ mod tests {
|
||||
assert!(grpc_addr("m1:9333.65536").is_err());
|
||||
}
|
||||
}
|
||||
|
||||
/// The filers of `filer_group` registered with the masters, as gRPC
|
||||
/// `host:port`s, in registration order; the masters are all asked, since a
|
||||
/// follower may know fewer than the leader. An error means no master answered.
|
||||
pub async fn list_filers(
|
||||
masters: &[String],
|
||||
filer_group: &str,
|
||||
tls: Option<&GrpcTls>,
|
||||
) -> Result<Vec<String>> {
|
||||
let mut filers = Vec::new();
|
||||
let mut answered = false;
|
||||
let mut last = None;
|
||||
for master in masters {
|
||||
match filers_of(master, filer_group, tls).await {
|
||||
Ok(found) => {
|
||||
answered = true;
|
||||
for filer in found {
|
||||
if !filers.contains(&filer) {
|
||||
filers.push(filer);
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(err) => {
|
||||
tracing::warn!("master {master}: listing filers: {err:#}");
|
||||
last = Some(err);
|
||||
}
|
||||
}
|
||||
}
|
||||
match (answered, last) {
|
||||
(false, Some(err)) => Err(err),
|
||||
(false, None) => Err(anyhow::anyhow!("no master given")),
|
||||
_ => Ok(filers),
|
||||
}
|
||||
}
|
||||
|
||||
async fn filers_of(master: &str, filer_group: &str, tls: Option<&GrpcTls>) -> Result<Vec<String>> {
|
||||
let channel = connect(master, tls).await?;
|
||||
let request = ListClusterNodesRequest {
|
||||
client_type: "filer".into(),
|
||||
filer_group: filer_group.into(),
|
||||
..Default::default()
|
||||
};
|
||||
let nodes = tokio::time::timeout(
|
||||
CONNECT_TIMEOUT,
|
||||
SeaweedClient::new(channel).list_cluster_nodes(request),
|
||||
)
|
||||
.await
|
||||
.context("master did not answer")??
|
||||
.into_inner()
|
||||
.cluster_nodes;
|
||||
Ok(nodes
|
||||
.iter()
|
||||
.filter_map(|node| match filer_grpc_addr(&node.address) {
|
||||
Ok(addr) => Some(addr),
|
||||
Err(err) => {
|
||||
tracing::warn!("master {master} lists filer {:?}: {err:#}", node.address);
|
||||
None
|
||||
}
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// A filer registers as `host:port.grpc`, or `host:port` with its gRPC at
|
||||
/// port + 10000.
|
||||
pub fn filer_grpc_addr(address: &str) -> Result<String> {
|
||||
let addr = grpc_addr(address)?;
|
||||
Ok(addr
|
||||
.split_once("://")
|
||||
.map_or(addr.as_str(), |(_, rest)| rest)
|
||||
.to_string())
|
||||
}
|
||||
@@ -5,6 +5,8 @@
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use anyhow::{Context, Result, bail};
|
||||
use sw_client::HttpsTls;
|
||||
use weed_lance_worker::sse::SseConfig;
|
||||
|
||||
const FILE_NAME: &str = "security.toml";
|
||||
|
||||
@@ -191,8 +193,11 @@ impl Security {
|
||||
self.raw(key).filter(|v| !v.is_empty())
|
||||
}
|
||||
|
||||
/// A boolean as weed reads one: viper accepts what Go's strconv.ParseBool
|
||||
/// does, `1`, `t`, `T`, `TRUE`, `true` and `True`.
|
||||
pub fn get_bool(&self, key: &str) -> bool {
|
||||
self.raw(key).is_some_and(|v| v.trim() == "true")
|
||||
self.raw(key)
|
||||
.is_some_and(|v| matches!(v.trim(), "1" | "t" | "T" | "TRUE" | "true" | "True"))
|
||||
}
|
||||
|
||||
/// The file a key names, read whole.
|
||||
@@ -287,6 +292,47 @@ impl Security {
|
||||
}
|
||||
}
|
||||
|
||||
impl Security {
|
||||
/// `https.client` for volume servers; `None` unless enabled.
|
||||
pub fn https_client_tls(&self) -> Result<Option<HttpsTls>> {
|
||||
if !self.get_bool("https.client.enabled") {
|
||||
return Ok(None);
|
||||
}
|
||||
let (cert, key) = match (self.get("https.client.cert"), self.get("https.client.key")) {
|
||||
(Some(cert), Some(key)) => (
|
||||
Some(self.read("https.client.cert", &cert)?),
|
||||
Some(self.read("https.client.key", &key)?),
|
||||
),
|
||||
(None, None) => (None, None),
|
||||
_ => bail!("https.client.cert and https.client.key must both be set"),
|
||||
};
|
||||
Ok(Some(HttpsTls {
|
||||
ca_pem: match self.get("https.client.ca") {
|
||||
Some(ca) => Some(self.read("https.client.ca", &ca)?),
|
||||
None => None,
|
||||
},
|
||||
cert_pem: cert,
|
||||
key_pem: key,
|
||||
insecure_skip_verify: self.get_bool("https.client.insecure_skip_verify"),
|
||||
}))
|
||||
}
|
||||
|
||||
/// `jwt.signing.read.key`, which volume reads must be signed with.
|
||||
pub fn jwt_read_key(&self) -> Option<String> {
|
||||
self.get("jwt.signing.read.key")
|
||||
}
|
||||
|
||||
/// The S3 gateway's SSE-S3 settings.
|
||||
pub fn sse(&self) -> SseConfig {
|
||||
SseConfig {
|
||||
kek_hex: self.get("s3.sse.kek"),
|
||||
key: self.get("s3.sse.key"),
|
||||
passphrase: self.get("s3.sse.kek.passphrase"),
|
||||
require_commitment: self.get_bool("s3.require_key_commitment"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
@@ -21,6 +21,7 @@ use std::sync::Arc;
|
||||
use tonic::{Request, Response, Status};
|
||||
use weed_lance_worker::catalog::TableDescription;
|
||||
use weed_lance_worker::dataset::{self, FallbackOptions};
|
||||
use weed_lance_worker::direct_read::{DirectReads, Wrappers};
|
||||
use weed_lance_worker::jobs::vector_index::is_vector_table;
|
||||
use weed_vector_core::index;
|
||||
use weed_vector_core::query::{self, Query};
|
||||
@@ -455,6 +456,7 @@ fn resolved(table: &pb::Table, schemes: &[&str]) -> Option<TableDescription> {
|
||||
|
||||
pub struct VectorServer {
|
||||
fallback: FallbackOptions,
|
||||
direct_reads: Option<DirectReads>,
|
||||
read_cache: Option<ReadCache>,
|
||||
nprobes: Option<usize>,
|
||||
/// URI schemes a table's location may have.
|
||||
@@ -465,6 +467,7 @@ impl VectorServer {
|
||||
pub fn new(fallback: FallbackOptions) -> Self {
|
||||
Self {
|
||||
fallback,
|
||||
direct_reads: None,
|
||||
read_cache: None,
|
||||
nprobes: None,
|
||||
schemes: &["s3"],
|
||||
@@ -484,6 +487,12 @@ impl VectorServer {
|
||||
self
|
||||
}
|
||||
|
||||
/// Read tables from the filers and volume servers instead of the gateway.
|
||||
pub fn with_direct_reads(mut self, reads: DirectReads) -> Self {
|
||||
self.direct_reads = Some(reads);
|
||||
self
|
||||
}
|
||||
|
||||
/// Serve repeated reads of write-once table files from `cache`.
|
||||
pub fn with_read_cache(mut self, cache: ReadCache) -> Self {
|
||||
self.read_cache = Some(cache);
|
||||
@@ -526,17 +535,25 @@ impl VectorServer {
|
||||
|
||||
async fn open(&self, table: &pb::Table) -> Result<Dataset, Status> {
|
||||
let description = self.describe(table)?;
|
||||
// The options name the store the bytes come from; the cache is scoped
|
||||
// by it so the same bucket and path on another endpoint never share.
|
||||
let cache = self.read_cache.as_ref().map(|cache| {
|
||||
// Direct reads innermost; the cache sits outside them, so a hit never
|
||||
// reaches storage. The options name the store the bytes come from
|
||||
// when reads go through the gateway, so the cache is scoped by it and
|
||||
// the same bucket and path on another endpoint never share.
|
||||
let mut wrappers: Vec<Arc<dyn WrappingObjectStore>> = Vec::new();
|
||||
if let Some(reads) = &self.direct_reads {
|
||||
wrappers.push(Arc::new(reads.clone()));
|
||||
}
|
||||
if let Some(cache) = &self.read_cache {
|
||||
let options = dataset::table_options(&description, &self.fallback);
|
||||
let endpoint = options
|
||||
.get(ENDPOINT_OPTION)
|
||||
.map(String::as_str)
|
||||
.unwrap_or("");
|
||||
Arc::new(cache.scoped(endpoint)) as Arc<dyn WrappingObjectStore>
|
||||
});
|
||||
let mut dataset = dataset::open_at(&description, &self.fallback, cache)
|
||||
wrappers.push(Arc::new(cache.scoped(endpoint)));
|
||||
}
|
||||
let wrapper = (!wrappers.is_empty())
|
||||
.then(|| Arc::new(Wrappers(wrappers)) as Arc<dyn WrappingObjectStore>);
|
||||
let mut dataset = dataset::open_at(&description, &self.fallback, wrapper)
|
||||
.await
|
||||
.map_err(open_status)?;
|
||||
if table.version != 0 {
|
||||
|
||||
Reference in new issue
Block a user