mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-02 10:17:10 +01:00
Chris LuandClaude Opus 5.5
4be191173a
* vector: scan only the fragments a ListVectors page covers A page pushed its offset into the scan, but the scan still loaded every fragment first, reading each one's deletion file: about 105 S3 GETs per page at 50k rows against 25 at 10k. The page's fragments now come from the live row counts in the manifest, so a page reads only its own. Tokens and row order are unchanged; a manifest without the counts keeps the old scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: list a ListVectors segment in the engine The gateway paged through the whole index for every segment and kept one row in segmentCount, so a segment cost segmentCount engine pages and 16 segments listed slower than one. The engine now takes segment_count and segment_index: it reads keys across the stride, keeps the segment's rows and takes their data and metadata alone, one call per page. A segment is still the rows whose position modulo segmentCount is segmentIndex, and its token still the position to resume from. An engine that predates the fields does not set segmented, and the gateway then pages through it as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: read only the key column across the table in GetVectors The key lookup scanned every column. On S3, lance reads a column during the scan when a row of it is under 1000 bytes, so a 128-dimension vector was read for every row of the table to return 100 of them, and data and metadata were read even when the request did not ask for them. The scan now projects the requested columns and materializes everything but the key late, for the matching rows only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: start the legacy segment fallback from the page already read With an engine that ignores segment_count, the gateway dropped the page it had just read and fetched it again. The fallback now consumes that page first, so a one-row segment page still costs one engine call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: check metadata on segmented ListVectors pages The segment test listed data only. Its rows now carry metadata on two in three keys, and every segmented row, read through take_rows, must equal the same row in a plain listing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: lease the version while a segment takes its data A segmented page with data or metadata reads keys first and then takes the other columns from the same version, so maintenance could reclaim its files in between. That path now holds a lease like GetVectors. A keys-only page is one scan and stays unleased, as plain pages are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: leave GetVectors materialization to lance Forcing every column late made a 100-key GetVectors with data slower at 10k x 128 in the perf cluster: 54.7 ms against 46.3 ms. The take issued a read per row, where lance's heuristic reads a narrow vector column in one pass per fragment. The projection keeps the gain on its own: with neither data nor metadata requested, the call went from 38.6 ms to 25.3 ms and from 150 to 89 S3 GETs. At 50k x 384 the vector is already late under the heuristic, and a full lookup is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>